1

I'm trying to pivot a Spark DF columns into rows like the example below.

scala> df.show() +----+--------------------+ |year| String| +----+--------------------+ |ina|List(Apple, 136, Train ...)...| |inb|List(Orange, 4.36, car ...)...| |ina|List(Apple,34, plane ...)...| +----+--------------------+

And create an output DF as:

+----+-------------+-------------+ |year|key|String| +----+-------------+-------------+ |ina|Apple |136 | |inb|Car |4.36 | |ina|Orange |34 |

How can I get the desired output? using explode?

Many thanks!

Improve this question

1 Answer 1

Reset to default
1

If you consider your value column as a json:

val mappingSchema = MapType(StringType, FloatType) originalDF.withColumn("map", from_json($"value", mappingSchema)).select($"year", explode($"map")).show()

gives:

+----+------+---------+ |year| key| value| +----+------+---------+ |2020| Apple|1064.3667| |2020| Car| 1434.2| |2020|Orange| 104.3667| |2020| Plane| 145.2| |2020| Apple| 1064.37| |2020| Train| 134.2| +----+------+---------+

Rename your columns if it is necessary

Improve this answer
Sign up to request clarification or add additional context in comments.

1 Comment

it "explodes" the intermediate MapType which is created from the json

Start asking to get answers

Find the answer to your question by asking.

Ask question

Explore related questions

See similar questions with these tags.