I'm trying to transpose an object type data frame and want to get back a data frame object (after it's been transposed):
and transpose into this:
I need the transposed object to remain a spark data frame.
Thank you!
1 Answer
check this out. you can use groupby and pivot. Please note i renamed name column because it was ambiguous to the dataframe once the name values are pivoted
df.show() # +------------+-----+ # | name|value| # +------------+-----+ # | Name| str| # |lastActivity| date| # | id| str| # +------------+-----+ df1 = df.withColumnRenamed("name", "name_val").groupBy("name_val").pivot("name_val").agg(F.first("value")) df1.show() # +------------+----+----+------------+ # | name_val|Name| id|lastActivity| # +------------+----+----+------------+ # | Name| str|null| null| # | id|null| str| null| # |lastActivity|null|null| date| # +------------+----+----+------------+ df1.select(*[F.first(column,ignorenulls=True).alias(column) for column in df1.columns if column not in 'name_val']).show() # # +----+---+------------+ # |Name| id|lastActivity| # +----+---+------------+ # | str|str| date| # +----+---+------------+ 