Spark - UnsupportedOperationException when calling Java method from Scala code

Question

I've implemented code in Scala that is using a method written in Java. In the code below processSale() is a Java method that takes util.List<Sale> as a parameter.

I've converted Scala Iterable[Sale] to Seq[Sale] and then to util.List<Sale> with the help of scala.collection.JavaConverters._

val parseSales: RDD[(String, Sale)] = rawSales .map(sale => sale.Id -> sale) .groupByKey() .mapValues(a => SaleParser.processSale(a.toSeq.asJava))

However when the code gets executed as part of a Spark driver the job fails due to the task failure with UnsupportedOperationException. I've looked through the logs and it appears that the reason is within the Java processSale method on the call of Collections.sort

 Collections.sort(sales, new Comparator<InvocaCall>() { @Override public int compare(Sale sale1, Sale sale2) { return Long.compare(sale1.timestamp, sale2.timestamp); } });

I'm stuck at this point because I'm passing the required util.List<Sale>. Why could Collections.sort be an unsupported operation in this case?

Duelist · Accepted Answer · 2018-10-11 11:47:40Z

From this documentation:

Because Java does not distinguish between mutable and immutable collections in their type, a conversion from, say, scala.immutable.List will yield a java.util.List, where all mutation operations throw an UnsupportedOperationException

toSeq from your code returns immutable.Seq, that's why you get the exception.

So you can convert your list to mutable data structure like ListBuffer:

list.to[scala.collection.mutable.ListBuffer].asJava

Priyanka W · Accepted Answer · 2018-10-11 12:14:44Z

1

Add null check for rawSales util.List<Sale>.

 val parseSales: RDD[(String, Sale)] = if (rawSales.nonEmpty) //rawSales specific stream operations else //None or any code as per requirement

edited Oct 11, 2018 at 12:14

answered Oct 11, 2018 at 11:28

Priyanka W

1,0201 gold badge15 silver badges30 bronze badges

2 Comments

Duelist Over a year ago

Assigning None to RDD?

Priyanka W Over a year ago

@Duelist programmer would decide if they want something at else; otherwise None.

Collectives™ on Stack Overflow

Spark - UnsupportedOperationException when calling Java method from Scala code

2 Answers 2

Comments

2 Comments

Hot Network Questions

Collectives™ on Stack Overflow

2 Answers 2

Comments

2 Comments

Related