How to enumerate columns in Spark's DataFrame? What if columns are nested?

Question

Apperently it doesn't have items() method. By how then?

I am trying to send Row to database with the code:

def write_row(table_name, cur, row): data = [] for key, value in row.items(): data.append((key, value)) data = zip(*data) columns = ", ".join(data[0]) values = data[1] questionmarks = ", ".join(["?"] * len(columns)) query = f"INSERT INTO {table_name} ({columns}) VALUES ({questionmarks})" cur.execute(query, values) def write_data_frame(df, epoch1): conn = mariadb.connect(**config["mariadb"]) cur = conn.cursor() table_name = "pysparktest" rows = df.collect() for row in rows: write_row(table_name, cur, row) conn.commit()

It swears

AttributeError: items

What if rows are nested?

root |-- track: struct (nullable = true) | |-- name: string (nullable = true) | |-- version: string (nullable = true) |-- car: struct (nullable = true) | |-- name: string (nullable = true) | |-- version: string (nullable = true) |-- cnt: long (nullable = false) |-- minBestLapTime: double (nullable = true)

try row.asDict().items()?

mck
– mck

2021-04-07 19:24:45 +00:00
Commented Apr 7, 2021 at 19:24 — mck
– mck, Commented Apr 7, 2021 at 19:24

Nihad TP · Accepted Answer · 2021-04-07 19:43:05Z

Like the compiler swears - there is no method called "items()" in Row class.

What you need to do is use "asDict" method. It outputs key, values in Row as python dict.

In the case of nested columns, there is an argument in asDict function called recursive, set that to True. By default, it is set as False.

For instance:

row = Row(name="Alice", age=11) row_as_dict = row.asDict() row_as_dict

Output:

{'name': 'Alice', 'age': 11}

For iterating:

for key in row_as_dict: print("{} : {}".format(key, row_as_dict[key]))

Output:

name : Alice age : 11

In case of nested columns

row = Row(key=1, value=Row(name='a', age=2)) row_as_dict = row.asDict(recursive=True) row_as_dict

Output:

{'key': 1, 'value': {'name': 'a', 'age': 2}}

Collectives™ on Stack Overflow

How to enumerate columns in Spark's DataFrame? What if columns are nested?

1 Answer 1

Comments

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

Comments

Related