Get value of a particular cell in PySpark Dataframe

Get value of a particular cell in PySpark Dataframe

In PySpark, to get the value of a particular cell in a DataFrame, you typically need to filter rows based on a condition and then select the desired column. After that, you can collect the data back to the driver node.

Here's a step-by-step method to get the value of a cell:

  1. Filter the rows based on a condition.
  2. Select the desired column.
  3. Use the collect or first method to retrieve the value.

For illustration, let's say you have a DataFrame df with columns "id", "name", and "age", and you want to get the age of the person with id = 3.

from pyspark.sql import SparkSession # Create a Spark session spark = SparkSession.builder.appName("exampleApp").getOrCreate() # Sample DataFrame data = [(1, 'Alice', 25), (2, 'Bob', 30), (3, 'Charlie', 35), (4, 'David', 40)] df = spark.createDataFrame(data, ['id', 'name', 'age']) # Get the age of the person with id = 3 age_value = df.filter(df.id == 3).select('age').first()[0] print(age_value) # Outputs: 35

In the above code:

  • filter(df.id == 3) filters rows to get the one with id = 3.
  • select('age') selects the "age" column.
  • first() retrieves the first row (since id should be unique in this example, this gives us the desired row).
  • [0] extracts the age value from the resulting Row object.

More Tags

asp.net-identity-3 spring-boot-actuator single-sign-on mysql-python spaces mod-rewrite moving-average query-string xslt-1.0 presentviewcontroller

More Programming Guides

Other Guides

More Programming Examples