How to join on multiple columns in Pyspark?

In PySpark, you can join DataFrames on multiple columns by passing a list of column names to the on parameter in the join method. Here's how you can do it:

1. Setup and Initialize PySpark:

from pyspark.sql import SparkSession spark = SparkSession.builder \ .appName("Multiple Columns Join Example") \ .getOrCreate() 

2. Create Sample DataFrames:

For demonstration purposes, let's create two sample DataFrames:

from pyspark.sql import Row # Sample data for DataFrame1 data1 = [ Row(id=1, name="Alice", timestamp="2022-01-01"), Row(id=2, name="Bob", timestamp="2022-01-02"), ] df1 = spark.createDataFrame(data1) # Sample data for DataFrame2 data2 = [ Row(id=1, name="Alice", timestamp="2022-01-01", value=100), Row(id=2, name="Bob", timestamp="2022-01-02", value=200), ] df2 = spark.createDataFrame(data2) 

3. Join on Multiple Columns:

To join df1 and df2 on both the id and name columns, you can use the following:

joined_df = df1.join(df2, on=["id", "name"], how="inner") joined_df.show() 

The how parameter specifies the type of join to be performed. In the above example, we used an "inner" join. You can replace "inner" with other join types like "left", "right", "outer", etc., based on your requirement.

If you also wanted to join on the timestamp column, you'd simply add it to the list:

joined_df = df1.join(df2, on=["id", "name", "timestamp"], how="inner") joined_df.show() 

And that's how you join on multiple columns in PySpark!

---

More Tags

jenkins-groovy dialog mariasql executequery mat-pagination autotools zero celerybeat store uikeyinput

More Programming Guides

• Loan calculator using Tkinter
• String Operators in Perl
• Python Collections.UserDict
• Convert elements of a Vector to Strings - toString() Function in R
• PHP Mask Error
• Binary, Octal And Hexadecimal
• Sequence addition
• PHP String Case Conversion

Other Guides

• What does the delayed() function do (when used with joblib in Python)
• MVC 4 @HTML.HiddenFor how to update on a postback
• Inserting an item in a python Tuple
• Maven JAR Plugin Error: You have to use a classifier to attach supplemental artifacts to the project instead of replacing them
• WPF Web Browser Control and DPI Scaling

More Programming Examples

• javascript - Array.push() and unique items
• service - Android: Alarm to be play every 30 minutes and it start from 12:30
• Removing duplicates from a list in Haskell without elem
• postgresql - Number of times a value appears (sql)
• sql - Execute select statement IF table exists
• ASP.NET MVC data annotation for currency format