Pyspark rdd : 'RDD' object has no attribute 'flatmap'

Question

I am new to Pyspark and I am actually trying to build a flatmap out of a Pyspark RDD object. However, even if this function clearly exists for pyspark RDD class, according to the documentation, I can't manage to use it and get the following error :

AttributeError: 'RDD' object has no attribute 'flatmap'

I am calling the latter function in the following line :

my_rdd = my_rdd.flatmap(lambda r: (r[5].split('|')))

The imports are the followings :

from pyspark.sql import * from pyspark.sql.functions import * from pyspark.sql import SparkSession from pyspark import SparkContext as sc from pyspark import SparkFiles spark = SparkSession.builder.getOrCreate()

Additionaly, some other functions, as my_rdd.count are working, which let me think that the SparkContext is correctly implemented.

Do you have any ideas about the reason why it could fail ?

Ged · Accepted Answer · 2018-10-28 12:50:38Z

11

my_rdd = my_rdd.flatMap(lambda r: (r[5].split('|')))

uppercase !!!

answered Oct 28, 2018 at 12:50

Ged

18.5k8 gold badges53 silver badges108 bronze badges

Sign up to request clarification or add additional context in comments.

5 Comments

Ged Over a year ago

These things happen, the forest for the trees

Jérémy Over a year ago

I'm still experiencing AttributeError: 'DataFrame' object has no attribute 'flatMap'

Ged Over a year ago

Not how it works. @jeremy

Jérémy Over a year ago

This post, call this function on DataFrame: stackoverflow.com/a/37955947/3710514

Ged Over a year ago

This question was about rdd's not dataframes. Pls post a new question.

Collectives™ on Stack Overflow

Pyspark rdd : 'RDD' object has no attribute 'flatmap'

1 Answer 1

5 Comments

Linked

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

5 Comments

Linked

Related