Building jars properly with sbt

Question

I have a map reduce .scala file like this:

import org.apache.spark._ object WordCount { def main(args: Array[String]){ val inputDir = args(0) //val inputDir = "/Users/eksi/Desktop/sherlock.txt" val outputDir = args(1) //val outputDir = "/Users/eksi/Desktop/out.txt" val cnf = new SparkConf().setAppName("Example MapReduce Spark Job") val sc = new SparkContext(cnf) val textFile = sc.textFile(inputDir) val counts = textFile.flatMap(line => line.split(" ")) .map(word => (word, 1)) .reduceByKey(_ + _) counts.saveAsTextFile(outputDir) sc.stop() } }

When I run my code, with setMaster("local[1]") parameters it works fine.

I want to put this code in a .jar and throw it to S3 to work with AWS EMR. Therefore, I use the following build.sbt to do so.

name := "word-count" version := "0.0.1" scalaVersion := "2.11.7" // additional libraries libraryDependencies ++= Seq( "org.apache.spark" % "spark-core_2.10" % "1.0.2" )

It generates a jar file, however none of my scala code is in there. What I see is just a manifest file when I extract the .jar

When I run sbt package this is what I get:

[myMacBook-Pro] > sbt package [info] Loading project definition from /Users/lele/bigdata/wordcount/project [info] Set current project to word-count (in build file:/Users/lele/bigdata/wordcount/) [info] Packaging /Users/lele/bigdata/wordcount/target/scala-2.11/word-count_2.11-0.0.1.jar ... [info] Done packaging. [success] Total time: 0 s, completed Jul 27, 2016 10:33:26 PM

What should I do to create a proper jar file that works like

WordCount.jar WordCount

By calling sbt clean compile package from the terminal where build.sbt lives — Bedi Egilmez
– Bedi Egilmez, Commented Jul 28, 2016 at 4:54
And you don't see the your object WordCount? Weird, I'd expect you'd only see that without the spark-core dependency. Have you looked at the sbt package build log? Anything special? — Yuval Itzchakov
– Yuval Itzchakov, Commented Jul 28, 2016 at 5:28
@YuvalItzchakov I've updated my question, added build log. I noticed something funny, it took 0 seconds this time. I'm sure that it was not 0 before (sbt clean compile package) — Bedi Egilmez
– Bedi Egilmez, Commented Jul 28, 2016 at 5:36
Unrelated directly to your problem, but why are you mixing different scala versions? scalaVersion is set to 2.11.7 while your dependency is cross versioned to _2.10. You should use "org.apache.spark" %% "spark-core" % "1.0.2" instead. — gilad hoch
– gilad hoch, Commented Jul 28, 2016 at 6:10

Community · Accepted Answer · 2020-06-20 09:12:55Z

Ref: It generates a jar file, however none of my scala code is in there. What I see is just a manifest file when I extract the .jar

Make sure your WordCount.scala is in the root or in src/main/scala

From http://www.scala-sbt.org/1.0/docs/Directories.html

Source code can be placed in the project’s base directory as with hello/hw.scala. However, most people don’t do this for real projects; too much clutter.

sbt uses the same directory structure as Maven for source files by default (all paths are relative to the base directory):

RTFM solution is the best solution. You are right, I messed up with directory structure. Merci!

Collectives™ on Stack Overflow

Building jars properly with sbt

1 Answer 1

1 Comment

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

1 Comment

Related