how to debug scala spark code in intelliJ

Question

I'm new to Scala and Spark, and current working on a scala spark jobs project, one thing frustrated me is I don't know how to debug the code in IntelliJ like I did with java.

after I imported the scala project, one thing i noticed is the spark-jobs folder was not marked as source code folder even though some other sub folders in the same module are market as source code folder.

-- utility (marked as source code folder) -- event-sender (marked as source code folder) -- spark-jobs (not marked as source code folder) -- src --main -- resources -- scala -- com -- example -- spark -- jobs

as I checked the spark job I'm working on , there is no main method.

class DailyExport( env: String, )(implicit sc: SparkContext, sqlContext: SQLContext, logger: SparkJobLogger) extends JobAudit with PartitionedWriter { def run(): Unit = ... object DailyExport extends App with SparkJobParameters { { for { env <- getStringParameter("environment", 0, args) } yield { val jobConfig = SparkJobConfig.fromConfig.exportConfig ... new DailyExport( jobConfig = jobConfig ).run() } }.fold( error => { println(s"Some provided parameters are wrong: $error") sys.exit(-1) }, identity ) }

however there is a main method defined in 'App'

trait App extends DelayedInit { ... @deprecatedOverriding("main should not be overridden", "2.11.0") def main(args: Array[String]) = { this._args = args for (proc <- initCode) proc() if (util.Properties.propIsSet("scala.time")) { val total = currentTime - executionStart Console.println("[total " + total + "ms]") } }

then I right clicked on the job that I'm working to choose 'Run..', it complained

'Error: Could not find or load main class com.exmaple.spark.jobs.DailyExport'

this is so different from Java, can anyone tell me how to debug it?

Justin Kaeser · Accepted Answer · 2019-02-22 11:43:11Z

In Scala there's different ways to define a main class. One is to define a main method in an object, similar to Java. The other is to extend the App trait, and just write the application code right into the class, as was done with DailyExport in your example. So you should be able to run this class normally as main class, and you should also see a "Run" icon next to the class in IntelliJ (not the main method in the super class, which is an implementation detail of the App trait).

If you run it that way, and you still get the error, it's possible you're running into a bug in IntelliJ. Try running it again, possibly after switching windows and checking the compile output is really where it is supposed to be.

Utsav · Accepted Answer · 2019-02-22 12:12:34Z

Please have inspect the target folders that get generated by IntelliJ. Make sure that the package structure is maintained.

If the problem still persists, try doing a clean build.

Ravi · Accepted Answer · 2019-02-23 07:50:57Z

IntelliJ IDEA has its own debugger for debugging scala. You can use INTELLIJ IDEA Debugger or sbt shell to debug the issues.

More information is here.

Collectives™ on Stack Overflow

how to debug scala spark code in intelliJ

3 Answers 3

Comments

Comments

Comments

Hot Network Questions

Collectives™ on Stack Overflow

3 Answers 3

Comments

Comments

Comments

Related