I have a data frame:
+---------+---------------------+ | id| Name| +---------+---------------------+ | 1| 'Gary'| | 1| 'Danny'| | 2| 'Christopher'| | 2| 'Kevin'| +---------+---------------------+ I need to combine all the Name values in the id column. Please tell me how to get from it:
+---------+------------------------+ | id| Name| +---------+------------------------+ | 1| ['Gary', 'Danny']| | 2| ['Kevin','Christopher']| +---------+------------------------+ You can use groupBy and collect functions. Based on your need you can use list or set etc.
df.groupBy(col("id")).agg(collect_list(col("Name")) in case you want duplicate values
df.groupBy(col("id")).agg(collect_set(col("Name")) if you want unique values
Use groupBy and collect_list functions for this case.
from pyspark.sql.functions import * df.groupBy(col("id")).agg(collect_list(col("Name")).alias("Name")).show(10,False) #+---+------------------------+ #|id |Name | #+---+------------------------+ #|1 |['Gary', 'Danny'] | #|2 |['Kevin', 'Christopher']| #+---+------------------------+ 
df.groupby('id')['Name'].apply(list) 