Link between scala object, dataset and dataframe [duplicate]

Comments are helpful, but the generic flattening function may not output columns in the desired order, and/or handle putting array elements into their own separate columns.

Supposing your JSON file contains lines such as:

{"ads": {"subject": "abc", "body": "doing something", "price": 13, "location": {"city": "Houston", "zipcode": 39014}, "attribut": [{"key_label": "a", "value_label": "b"}]}} 

If the file is fairly consistent and you're already including Spark as a dependency, you probably do not need to use a separate library to parse the JSON.

You will need to use the explode function to handle the fact that the "attribut" column is a list. Use the explode_outer function instead, if the lists may be empty but you would like to preserve other columns' values.

import org.apache.spark.sql.functions._ // assuming spark is the Spark Session val df = spark.read.json("mydata.json") val df1 = df.select(col("ads.subject").alias("Subject"), col("ads.body").alias("Body"), col("ads.location.city").alias("City"), col("ads.location.zipcode").alias("Zipcode"), explode(col("ads.attribut"))) val resultDF = df1.select(col("Subject"), col("Body"), col("City"), col("Zipcode"), col("col.key_label"), col("col.value_label")) 

resultDF.show would output:

+-------+---------------+-------+-------+---------+-----------+ |Subject| Body| City|Zipcode|key_label|value_label| +-------+---------------+-------+-------+---------+-----------+ | abc|doing something|Houston| 39014| a| b| +-------+---------------+-------+-------+---------+-----------+ 

To output as a single CSV file in the specified directory, with headers:

resultDF.repartition(1).write.option("header", "true").csv("/tmp/my-output-dir/")