How to optimize this Scala code?

Question

I'm learning Scala, curious how to optimize this code. What I have is an RDD loaded from Spark. It's a tab delimited dataset. I want to combine the first column with the second column, and append it as a new column to the end of the dataset, with a "-" separating the two.

For example: column1\tcolumn2\tcolumn3

becomes

column1\tcolumn2\tcolumn3\tcolumn1-column2

val f = sc.textFile("path/to/dataset") f.map(line => if (line.split("\t").length > 1) line.split("\t") :+ line.split("\t")(0)+"-"+line.split("\t")(1) else Array[String]()).map(a => a.mkString("\t") ) .saveAsTextFile("output/path")

I'd start by doing that string split only once.

maasg
– maasg

2015-05-18 21:39:56 +00:00
Commented May 18, 2015 at 21:39 — maasg
– maasg, Commented May 18, 2015 at 21:39

David Griffin · Accepted Answer · 2015-05-18 23:36:16Z

1

Try:

f.map{ line => val cols = line.split("\t") if (cols.length > 1) line + "\t" + cols(0) + "-" + cols(1) else line }

edited May 18, 2015 at 23:36

answered May 18, 2015 at 21:57

David Griffin

14k5 gold badges49 silver badges66 bronze badges

Sign up to request clarification or add additional context in comments.

1 Comment

Sean Bollin Over a year ago

Thanks, yeah change it from map{ to map( allowed me to set a val inside.

Collectives™ on Stack Overflow

How to optimize this Scala code?

1 Answer 1

1 Comment

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

1 Comment

Related