Creating dataframe with complex schema that includes MapType in pyspark

Question

I'm trying to create a dataframe with the following schema:

|-- data: struct (nullable = true) | |-- id: long (nullable = true) | |-- keyNote: struct (nullable = true) | | |-- key: string (nullable = true) | | |-- note: string (nullable = true) | |-- details: map (nullable = true) | | |-- key: string | | |-- value: string (valueContainsNull = true)

This is the best I managed to do:

schema = StructType([ StructField("id",LongType(), True), StructField("keyNote",StructType([ StructField("key",StringType(),True), StructField("note",StringType(),True) ])), StructField("details",MapType(StringType, StringType, True)) ]) df = spark\ .createDataFrame([("idd",("keyy","notee"),("keyy","valuee")),schema])

But I'm getting an exception:

AssertionError: keyType should be DataType

missing () after StringType in details?

Vamsi Prabhala
– Vamsi Prabhala

2020-01-27 01:36:44 +00:00
Commented Jan 27, 2020 at 1:36 — Vamsi Prabhala
– Vamsi Prabhala, Commented Jan 27, 2020 at 1:36
Always share the entire error message.

AMC
– AMC

2020-01-27 02:26:54 +00:00
Commented Jan 27, 2020 at 2:26 — AMC
– AMC, Commented Jan 27, 2020 at 2:26

Enayat · Accepted Answer · 2020-01-27 02:56:16Z

Seems you should write the correct syntax for the MapType:

MapType(StringType(), StringType(), True)

Instead of StringType(), you wrote StringType without parentheses.

Collectives™ on Stack Overflow

Creating dataframe with complex schema that includes MapType in pyspark

1 Answer 1

Comments

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

Comments

Related