I'd like to explode an array of structs to columns (as defined by the struct fields). E.g.
root |-- arr: array (nullable = true) | |-- element: struct (containsNull = true) | | |-- id: long (nullable = false) | | |-- name: string (nullable = true) Should be transformed to
root |-- id: long (nullable = true) |-- name: string (nullable = true) I can achieve this with
df .select(explode($"arr").as("tmp")) .select($"tmp.*") How can I do that in a single select statement?
I thought this could work, unfortunately it does not:
df.select(explode($"arr")(".*")) Exception in thread "main" org.apache.spark.sql.AnalysisException: No such struct field .* in col;