Share Python logger across multiple files

Question

I have a python application with a file structure similar to the following:

/Project file1.py file2.py file3.py ...

The application is using Python 2.6 as it is running on a CentOS 6 environment.

I am interested in setting up a common logging mechanism where log statements from each of these files write to the same log file on disk. These python files are commonly executed from the command line are are somewhat independent of each other. One suggestion from the documentation (linked here) was the following which I have tested:

log = logging.getLogger(__name__)

What I found is that all log lines would be listed as coming from __main__, regardless of which file the log statement was from. In other words, I was not able to tell from which file a given log line came from.

I am hoping to get to a point were I can have a log file with contents similar to the following:

2017-01-17 10:48:47,446 - file1 - DEBUG - this is from file1 2017-01-17 10:48:47,447 - file2 - DEBUG - this is from file2 2017-01-17 10:48:47,447 - fiel3 - DEBUG - this is from file3

In the above, I would have executed:

log.debug("this is from file1")

in file1.py and a similar log line in the other files, replaces the text in the log statement as appropriate.

I am currently using the following to setup logging:

######################################## # Setup Logging ######################################## LOG_FILENAME = 'app.log' # Set up a specific logger with our desired output level log = logging.getLogger('Logger') log.setLevel(logging.DEBUG) formatter = logging.Formatter('%(asctime)s - %(name)s - %(levelname)s - %(message)s') # Add the log message handler to the logger handler = logging.handlers.RotatingFileHandler( LOG_FILENAME, maxBytes=200000000, backupCount=5) handler.setFormatter(formatter) log.addHandler(handler) ########################################

This setup is all good for logging from one python source, but is less useful as I am now trying to log from multiple python sources

I have read Using Python logging in multiple modules; it has good answers, but due to the independent nature of the files in my project the use of __name__ was not working as described.

If you just want to use the filename rather than the module name, logging.getLogger(__file__) would do it. — jonrsharpe
– jonrsharpe, Commented Jan 23, 2017 at 20:33
I will try this. Two follow up questions: 1) While the log statements from each python file be written to the same log file? Is there any changes needed to my posted logging config that would be required to do this? 2) Will I need to do anything extra to just get the just the name of the file and not the whole path of the file? Or will __file__ be just the filename without path? I am a bit rusty on some of my python :( — Andrew
– Andrew, Commented Jan 23, 2017 at 20:40
Ok. It looks like my existing setup configuration with the use of __file__ did the trick. I will need to work a bit on cutting of the path from the file name but otherwise, this is what I was looking for. — Andrew
– Andrew, Commented Jan 23, 2017 at 21:09

xjcl · Accepted Answer · 2020-12-07 01:02:45Z

Best approach

Rather than trying to set logger name to the filename, just specify %(module)s!

formatter = logging.Formatter('%(asctime)s - %(module)s - %(levelname)s - %(message)s') ^^^^^^^^^^

This will avoid printing __main__ and nicely give you:

2020-11-06 23:19:03,827 - file2 - INFO - This is from file2 2020-11-06 23:19:03,827 - file3 - INFO - This is from file3 2020-11-06 23:19:03,827 - file1 - INFO - This is from file1

Enjoy!

PS: I am slightly nervous about you attaching a RotatingFileHandler of the same file to each logger, rather than to the root logger once. In general I wouldn't recommend copying logging-setup code to each source code file and logger, but configuring the root logger once in a separate file.

Collectives™ on Stack Overflow

Share Python logger across multiple files

1 Answer 1

Comments

Linked

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

Comments

Linked

Related