Create a dataframe from column of dictionaries in pyspark

Question

I want to create a new dataframe from existing dataframe in pyspark. The dataframe "df" contains a column named "data" which has rows of dictionary and has a schema as string. And the keys of each dictionary are not fixed.For example the name and address are the keys for the first row dictionary but that would not be the case for other rows they may be different. following is the example for that;

........................................................ data ........................................................ {"name": "sam", "address":"uk"} ........................................................ {"name":"jack" , "address":"aus", "occupation":"job"} .........................................................

How do I convert into the dataframe with individual columns like following.

 name address occupation sam uk jack aus job

Possible duplicate of How to convert list of dictionaries into Spark DataFrame — pault
– pault, Commented Nov 9, 2018 at 12:34
Or a dupe of Pyspark: explode json in column to multiple columns. It's hard to tell from your question — pault
– pault, Commented Nov 9, 2018 at 12:44
@pault Its not duplicate of above both these links. I referred it before asking the query. The question is properly understood. The dataframe "df" has a column named "data" which contains rows of dictionary. Its not a list of dictionary. — amol desai
– amol desai, Commented Nov 11, 2018 at 5:20
Your question is still unclear. You can't have "rows of dictionaries" in a pyspark DataFrame. Is df a pandas DataFrame? Or is the data column actually of type StringType() or MapType()? Edit your question with the output of df.select('data').printSchema(). Better yet, provide a reproducible example. Maybe you're looking for this answer. — pault
– pault, Commented Nov 13, 2018 at 15:55

cs95 · Accepted Answer · 2018-11-09 04:48:35Z

Convert data to an RDD, then use spark.read.json to convert the RDD into a dataFrame with the schema.

data = [ {"name": "sam", "address":"uk"}, {"name":"jack" , "address":"aus", "occupation":"job"} ] spark = SparkSession.builder.getOrCreate() df = spark.read.json(sc.parallelize(data)).na.fill('') df.show() +-------+----+----------+ |address|name|occupation| +-------+----+----------+ | uk| sam| | | aus|jack| job| +-------+----+----------+

I have tried this method its giving py4j.Py4JException: Method __getnewargs__([]) does not exist. The data is a column name of dataframe df.

Ali AzG · Accepted Answer · 2018-11-09 08:24:20Z

If the order of rows is not important, this is another way you can do this:

from pyspark import SparkContext sc = SparkContext() df = sc.parallelize([ {"name":"jack" , "address":"aus", "occupation":"job"}, {"name": "sam", "address":"uk"} ]).toDF() df = df.na.fill('') df.show() +-------+----+----------+ |address|name|occupation| +-------+----+----------+ | aus|jack| job| | uk| sam| | +-------+----+----------+

you have to have all the columns in the first row all the time =\

Collectives™ on Stack Overflow

Create a dataframe from column of dictionaries in pyspark

2 Answers 2

1 Comment

1 Comment

Linked

Hot Network Questions

Collectives™ on Stack Overflow

2 Answers 2

1 Comment

1 Comment

Linked

Related