In Spark dataframe how to Transpose rows to columns?

Question

this may be a very simple question. I want to transpose all the rows of dataframe to columns. I want to convert this df as shown below output DF. What are the ways in spark to achieve this?

Note : I have single column in input DF

import sparkSession.sqlContext.implicits._ val df = Seq(("row1"), ("row2"), ("row3"), ("row4"), ("row5")).toDF("COLUMN_NAME") df.show(false) Input DF: +-----------+ |COLUMN_NAME| +-----------+ |row1 | |row2 | |row3 | |row4 | |row5 | +-----------+ Output DF +----+----+----+----+----+ |row1|row2|row3|row4|row5| +----+----+----+----+----+

Does this answer your question? How to pivot Spark DataFrame? — mazaneicha
– mazaneicha, Commented Jun 20, 2020 at 12:02
Think this answer will help: stackoverflow.com/a/49393080/1125159. It's not clear if you want row1, row2. etc to be column names in the output DataFrame. I'm guessing not, so you should update your question to include the desired column names. — Powers
– Powers, Commented Jun 20, 2020 at 13:03
Without groupBy, pivot, agg, & first ..check this - stackoverflow.com/questions/61686883/… — s.polam
– s.polam, Commented Jun 20, 2020 at 13:48

abc_spark · Accepted Answer · 2020-06-20 13:15:47Z

0

Does this help you ?

df.withColumn("group",monotonicallyIncreasingId ).groupBy("group").pivot("COLUMN_NAME").agg(first("COLUMN_NAME")).show

answered Jun 20, 2020 at 13:15

abc_spark

3935 silver badges20 bronze badges

Sign up to request clarification or add additional context in comments.

1 Comment

rajesh Over a year ago

This is creating row entries in the output DF

Collectives™ on Stack Overflow

In Spark dataframe how to Transpose rows to columns?

1 Answer 1

1 Comment

Linked

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

1 Comment

Linked

Related