How to set pyspark logging level to debug?

Question

Can anyone help me with the spark configuration needed to set logging level to debug and capture more logs

Does this answer your question? How can set the default spark logging level? — William Prigol Lopes
– William Prigol Lopes, Commented May 6, 2020 at 18:51

notNull · Accepted Answer · 2020-05-06 18:39:23Z

8

Set setLogLevel property to DEBUG in sparksession.

from pyspark.sql import SparkSession spark = SparkSession.builder.master('local').appName('test').getOrCreate() spark.sparkContext.setLogLevel("DEBUG") #or log4j = sc._jvm.org.apache.log4j log4j.LogManager.getRootLogger().setLevel(log4j.Level.DEBUG) #for custom log log4j.LogManager.getRootLogger().info("custom log") #2020-05-06 13:43:42 INFO root:? - custom log

edited May 6, 2020 at 18:39

answered May 6, 2020 at 18:19

notNull

31.8k4 gold badges41 silver badges58 bronze badges

Sign up to request clarification or add additional context in comments.

3 Comments

user13485171 Over a year ago

Thanks. But i think this line is creating end less console output in my case. The error was around "connection error"

notNull Over a year ago

@user13485171, Could you update the question with steps you are performing and error?

user13485171 Over a year ago

I would like to but i can't as that's little confidential My code looks like Setting environment variables Creating spark session similarly Then i tried to change log level So with the new code recreted the issue I think it's more because of my server settings/permission I'll take this up with my IT and update you why it happened

Collectives™ on Stack Overflow

How to set pyspark logging level to debug?

1 Answer 1

3 Comments

Linked

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

3 Comments

Linked

Related