Rotate logfiles each time the application is started (Python)

Simplest way is just to have a date tag in log file name, so when you start app each time you will get a new log file.

e.g.

dateTag = datetime.datetime.now().strftime("%Y-%b-%d_%H-%M-%S") logging.basicConfig(filename="myapp_%s.log" % dateTag, level=logging.DEBUG) 

so each time you will have log like myapp_2011-Jan-11_12-27-29.log

Another benefit is that you can mix this with RotatingFileHandler to have separate log for each app invocation, where each log itself is further divided into multiple fixed size logs.

Log Rotation and RoatatingFileHandler are usually designed and desirable when the application is running for a very long time (days) and you want the log to keep rotation. Under cases where I have to rotate the log upon restart of the application, I had to do that outside of the Logfile handler, which was easier. It was like, before the log writer call for the first time, I would see if the log file already existed, and if yes, rename it and create a new log file. The renaming should be differentiated from the handler's renaming mechanism.

Another solution is to create your own Handler class (especially if you use a conf file).

For example, if you want to create a new file at runtime with current datetime:

import logging from datetime import datetime class PerExecutionFileHandler(logging.FileHandler): def __init__(self, *args): current_time = datetime.now().strftime("%Y-%m-%d_%H-%M-%S") super().__init__(filename=f"{current_time}.log", *args) 

Therefore you can mention this class in you conf file

[handler_file] class=custom_handler.PerExecutionFileHandler 

Note that the custom class cannot be in the same file that the one where you load config (cf I can't get logging.config to load my custom class)

I had a similar requirement to be able to force a log rotation at startup based on a command-line option, but for the logfiles to otherwise rotate on their regular schedule. This was my solution:

import logging from logging.handlers import BaseRotatingHandler from typing import Union def rotate_logs(loggers: Union[str,list]=None, delimiter: str=','): """Rotate logs. Args: loggers: List of logger names as list object or as string, separated by `delimiter`. delimiter: Separator for logger names, if `loggers` is :obj:`str`. Defaults to ``,`` (comma). """ # Convert loggers to list. if isinstance(loggers, str): loggers = [t.strip() for t in loggers.split(delimiter)] handlers = [] root = logging.getLogger() # Include root logger in dict. ld = {'': root, **root.manager.loggerDict} for k, v in ld.items(): if loggers is not None and k not in loggers: continue try: for h in v.handlers: if (isinstance(h, BaseRotatingHandler) and h not in handlers): handlers.append(h) except AttributeError: pass for h in handlers: h.doRollover() if __name__ == '__main__': pass 

Notes:

• This has been validated to work if maxBytes > 0 on a RotatingFileHandler.

• This method hasn't been tested with a TimedRotatingFileHandler, but should work.

• This method eliminates the need to maintain a reference to the RotatingFileHandler to be rotated; as a result, it can easily be used when configuring logging using logging.config.

As @Senthil Kumaran stated, this kind of logic is best put at the application level. Here's a full working example that sets up a logger that prints to stdout and writes to file. At application start up, if an existing log is found with the same name, it will be rotated out of the way in to a directory named 'old'. The RotatingFileHandler then handles rotating live logs when their size exceeds 512 bytes.

import logging import logging.handlers as handlers from pathlib import Path from typing import List def configure_logger(): file_handler = _get_file_handler() log_handlers: List[logging.Handler] = [logging.StreamHandler(), file_handler] logging.basicConfig( level="INFO", format="[%(asctime)s] [%(levelname)s] [%(name)s]: %(message)s", handlers=log_handlers, ) def _get_file_handler() -> handlers.RotatingFileHandler: log_path_str = "logs/my_app.log" log_path = Path(log_path_str) _rotate_existing_log(log_path) log_path.parent.mkdir(parents=True, exist_ok=True) return handlers.RotatingFileHandler( log_path, maxBytes=512, backupCount=5, # don't append to existing file, instead create a new mode="w", ) def _rotate_existing_log(log_path: Path): """ If log file already exists, rotate it out of the way into an 'old' directory :param log_path: :return: """ if log_path.exists(): old_log_paths = log_path.parent / "old" old_log_paths.mkdir(parents=True, exist_ok=True) i = 1 old_log_name = old_log_paths / (log_path.name + f".{i}") while old_log_name.exists(): i += 1 old_log_name = old_log_paths / (log_path.name + f".{i}") log_path.rename(old_log_name) if __name__ == "__main__": configure_logger() logger = logging.getLogger(__name__) logger.info("hello world") 

The effect of running this on my system a few times gave the following directory and file structure:

C:\USERS\PYCHARMPROJECTS\SCRATCH\LOGS | my_app.log | \---old my_app.log.1 my_app.log.2 my_app.log.3 my_app.log.4