1

I am trying to find out how to setup logging in pyspark for pytest. However both methods below has an error TypeError: 'JavaPackage' object is not callable. I'm using python 3.7 and pyspark <2.4.0 in pycharm. Could you please help? Many thanks.

 logger = logging.getlogger('py4j') logger.setlevel(logger.WARN) log4jLogger = spark.sparkContext._jvm.org.apache.log4j log = log4jLogger.logManager.getLogger(__name__)
Improve this question
2
  • Most probably this error has nothing to do with logging, but rather with the fact that some jars are not found. Commented Apr 16, 2021 at 22:52
  • @hoefling thanks. so how do I let pyspark sees those jars? I'm using this in pytest. Commented Apr 17, 2021 at 9:04

1 Answer 1

Reset to default
4

Looks like the way you're referring to Java object is incorrect, try this instead

log4jLogger = spark.sparkContext._jvm.org.apache.log4j logger = log4jLogger.LogManager.getLogger("MyLogger") logger.setLevel(log4jLogger.Level.DEBUG)
Improve this answer
Sign up to request clarification or add additional context in comments.

Comments

Start asking to get answers

Find the answer to your question by asking.

Ask question

Explore related questions

See similar questions with these tags.