I have the dataframe as follow
val employees = sc.parallelize(Array[(String, Int, BigInt)]( ("Rafferty", 31, 222222222), ("Jones", 33, 111111111), ("Heisenberg", 33, 222222222), ("Robinson", 34, 111111111), ("Smith", 34, 333333333), ("Williams", 15, 222222222) )).toDF("LastName", "DepartmentID", "Code") employees.show() +----------+------------+---------+ | LastName|DepartmentID| Code| +----------+------------+---------+ | Rafferty| 31|222222222| | Jones| 33|111111111| |Heisenberg| 33|222222222| | Robinson| 34|111111111| | Smith| 34|333333333| | Williams| 15|222222222| +----------+------------+---------+ I want to create another column as personal_id as concentrate DepartmentId and Code. Example: Rafferty => 31222222222
So I write code as follow:
val anotherdf = employees.withColumn("personal_id", $"DepartmentID".cast("String") + $"Code".cast("String")) +----------+------------+---------+------------+ | LastName|DepartmentID| Code| personal_id| +----------+------------+---------+------------+ | Rafferty| 31|222222222|2.22222253E8| | Jones| 33|111111111|1.11111144E8| |Heisenberg| 33|222222222|2.22222255E8| | Robinson| 34|111111111|1.11111145E8| | Smith| 34|333333333|3.33333367E8| | Williams| 15|222222222|2.22222237E8| +----------+------------+---------+------------+ But I got personal_id at double.
anotherdf.printSchema root |-- LastName: string (nullable = true) |-- DepartmentID: integer (nullable = false) |-- Code: decimal(38,0) (nullable = true) |-- personal_id: double (nullable = true) 