0

Using a Standalone Spark Java to execute the below code snippet, I'm getting the Status is always WAITING with the below error.It doesn't work when I try to add the Print statement. Is there any configuration I might have missed to run multiple jobs?

15/09/18 15:02:56 INFO DAGScheduler: Submitting 2 missing tasks from Stage 0 (MapPartitionsRDD[2] at filter at SparkTest.java:143)

15/09/18 15:02:56 INFO TaskSchedulerImpl: Adding task set 0.0 with 2 tasks

15/09/18 15:03:11 WARN TaskSchedulerImpl: Initial job has not accepted any resources; check your cluster UI to ensure that workers are registered and have sufficient resources

15/09/18 15:03:26 WARN TaskSchedulerImpl: Initial job has not accepted any resources; check your cluster UI to ensure that workers are registered and have sufficient resources

15/09/18 15:03:41 WARN TaskSchedulerImpl: Initial job has not accepted any resources; check your cluster UI to ensure that workers are registered and have sufficient resources

JavaRDD<String> words = input.flatMap(new FlatMapFunction<String, String>() //Ln:143 { public Iterable<String> call(String x) { return Arrays.asList(x.split(" ")); } }); // Count all the words System.out.println("Total words is" + words.count())
Improve this question
3
  • What's your infrastructure? Are you using a cluster? What submit command are you using? Please provide more information so we can be helpful :). Commented Sep 18, 2015 at 12:08
  • Its a stand alone spark cluster... I used this command to start:./spark-submit ../lib/sparkapp.jar --class com.test.logs.spark.SparkTest Commented Sep 18, 2015 at 12:17
  • Do you specify the master? It would be great if you could edit your question and add all this information in it. Commented Sep 18, 2015 at 13:07

1 Answer 1

Reset to default
1

This error message means that your application is requesting more resources from the cluster than the cluster can currently provide i.e. more cores or more RAM than available in the cluster.

One of the reasons for this could be that you already have a job running which uses up all the available cores.

When this happens, your job is most probably waiting for another job to finish and release resources.

You can check this in the Spark UI.

Improve this answer
Sign up to request clarification or add additional context in comments.

1 Comment

I am facing similar issue with respect to the Submit PySpark application on AWS EC2 - getting Initial job failed error - Application goes in Wait state due to unavailability of resources. Let me know if there is any workaround. Problem is stated here - stackoverflow.com/questions/38359801/… No other job is running at the moment still it goes in Wait state. @benohead

Start asking to get answers

Find the answer to your question by asking.

Ask question

Explore related questions

See similar questions with these tags.