Pyspark - Union two data frames with same column based n same id

Question

DF1

Id	Name	Desc	etc
A	Name1	desc1	etc1
B	name2	desc2	etc2

DF2

Id	Name	Desc	etc
A	Name2	desc2	etc2
C	name2	desc2	etc2

I want to union records from DF2 into DF1 where the ID is equal and include all records from DF1.

Result DF

Id	Name	Desc	etc
A	Name1	desc1	etc1
B	name2	desc2	etc2
A	name2	desc2	etc2

What is the best way to do it? Any help Is appreciated.

just filter df2 for ids that are present in df1 and then append the dataframes. — samkart
– samkart, Commented May 27, 2022 at 14:17

ScootCork · Accepted Answer · 2022-05-28 19:12:39Z

You can do so by doing a semi join keeping only the ids in df2 which exists in df1, followed by a union with df1.

d1 = [ ('A', 'Name1', 'desc1', 'etc1'), ('B', 'name2', 'desc2', 'etc2'), ] d2 = [ ('A', 'Name2', 'desc2', 'etc2'), ('C', 'name2', 'desc2', 'etc2'), ] df1 = spark.createDataFrame(d1, ['Id', 'Name', 'Desc', 'etc']) df2 = spark.createDataFrame(d2, ['Id', 'Name', 'Desc', 'etc']) df2.join(df1, on='Id', how='semi').union(df1).show() +---+-----+-----+----+ | Id| Name| Desc| etc| +---+-----+-----+----+ | A|Name2|desc2|etc2| | A|Name1|desc1|etc1| | B|name2|desc2|etc2| +---+-----+-----+----+

Collectives™ on Stack Overflow

Pyspark - Union two data frames with same column based n same id

1 Answer 1

Comments

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

Comments

Related