I have a dataframe which I want to modify in a way that each row will containt the column name . for example :
FirstName LastName Jhon Doe David Lue to create the follwing
(FirstName=Jhon,LastName=Doe) (FirstName=David,LastName=Lue) I managed to do for df with 2 columns
val x = df.map { row => (names(0) + "=" +row(0) , names(1)+"="+rows(1)} but how can I do it with for loop for any number of columns?
Thanks
One option is to use foldLeft on the column names:
import org.apache.spark.sql.functions._ import org.apache.spark.sql.DataFrame import sqlContext.implicits._ val df = Seq( ("John", "Doe"), ("David", "Lue") ).toDF("first_name", "last_name") val x = df.columns.foldLeft(df) { (acc: DataFrame, colName: String) => acc.withColumn(colName, concat(lit(colName + "="), col(colName))) } x.show() Resulting in:
+----------------+-------------+ | first_name| last_name| +----------------+-------------+ | first_name=John|last_name=Doe| |first_name=David|last_name=Lue| +----------------+-------------+ If you then want to convert it to an RDD of tuples, you can call a map on it:
x.rdd.map(r => (r.getString(0), r.getString(1))) or even with Spark SQL's typed API:
x.as[(String, String)].rdd 
