Create a mapped RDD and save it in text

Question

I runned a K-means example and I have an RDD with my data named parsedData and my model named clusters. I want to create a mapped Rdd with datapoint and prediction cluster from the model. So I tried

val predictions = parsedData.map( point => { val pointPred = clusters.predict(point) Array(point,pointPred) })

when I try

 predictions.first()

I take

Array[Any] = Array([0.8898668778942382,0.89533945283595], 0)

which is the result I want. So then I tried

predictions.saveAsTextFile ("/../ClusterResults");

to save The Arrays from each datapoint in a local file but the file created was

[Ljava.lang.Object;@3b43c55c

[Ljava.lang.Object;@5e523969

[Ljava.lang.Object;@68374cdf ....

had the objects and not the data. I also tried to print from the RDD like

predictions.take(10).map(println)

and took the objects as a result again. How can I take the data and not the objects and save them to a local file?

Soory I edited my code I did us saveAsTextFile and took Ljava.lang.Object;@3b43c55c .... — Michail N
– Michail N, Commented Sep 8, 2017 at 8:30
In my map? Like .map( point =>(point, cluster.predict(point))) — Michail N
– Michail N, Commented Sep 8, 2017 at 8:32

gsamaras · Accepted Answer · 2017-09-08 08:37:32Z

1

The problem lies in the way you map your data. Try using a Tuple, instead of an Array.

Example:

val predictions = parsedData.map( point => { (point, clusters.predict(point)) })

answered Sep 8, 2017 at 8:37

gsamaras

73.7k50 gold badges210 silver badges330 bronze badges

Sign up to request clarification or add additional context in comments.

3 Comments

Michail N Over a year ago

Thank you I tried this and worked. But my question still lies why first() prints the Array and not the object? And how can I do it with Array, can I parse the Array from the object as a String?

gsamaras Over a year ago

I think because it has an array there @MichailN..Maybe, but I am not sure! Glad that I helped.

Greg Over a year ago

The output in the text file from saveAsTextFile is simply calling the Object's toString method. In this case Array.toString outputs like you had seen: [Ljava.lang.Object;@3b43c55

Collectives™ on Stack Overflow

Create a mapped RDD and save it in text

1 Answer 1

3 Comments

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

3 Comments

Related