How to duplicate sys.stdout to a log file?

I had this same issue before and found this snippet very useful:

class Tee(object): def __init__(self, name, mode): self.file = open(name, mode) self.stdout = sys.stdout sys.stdout = self def __del__(self): sys.stdout = self.stdout self.file.close() def write(self, data): self.file.write(data) self.stdout.write(data) def flush(self): self.file.flush() 

from: http://mail.python.org/pipermail/python-list/2007-May/438106.html

The print statement will call the write() method of any object you assign to sys.stdout.

I would spin up a small class to write to two places at once...

import sys class Logger(object): def __init__(self): self.terminal = sys.stdout self.log = open("log.dat", "a") def write(self, message): self.terminal.write(message) self.log.write(message) sys.stdout = Logger() 

Now the print statement will both echo to the screen and append to your log file:

# prints "1 2" to <stdout> AND log.dat print "%d %d" % (1,2) 

This is obviously quick-and-dirty. Some notes:

• You probably ought to parametize the log filename.
• You should probably revert sys.stdout to <stdout> if you won't be logging for the duration of the program.
• You may want the ability to write to multiple log files at once, or handle different log levels, etc.

These are all straightforward enough that I'm comfortable leaving them as exercises for the reader. The key insight here is that print just calls a "file-like object" that's assigned to sys.stdout.

What you really want is logging module from standard library. Create a logger and attach two handlers, one would be writing to a file and the other to stdout or stderr.

See Logging to multiple destinations for details

Here is another solution, which is more general than the others -- it supports splitting output (written to sys.stdout) to any number of file-like objects. There's no requirement that __stdout__ itself is included.

import sys class multifile(object): def __init__(self, files): self._files = files def __getattr__(self, attr, *args): return self._wrap(attr, *args) def _wrap(self, attr, *args): def g(*a, **kw): for f in self._files: res = getattr(f, attr, *args)(*a, **kw) return res return g # for a tee-like behavior, use like this: sys.stdout = multifile([ sys.stdout, open('myfile.txt', 'w') ]) # all these forms work: print 'abc' print >>sys.stdout, 'line2' sys.stdout.write('line3\n') 

NOTE: This is a proof-of-concept. The implementation here is not complete, as it only wraps methods of the file-like objects (e.g. write), leaving out members/properties/setattr, etc. However, it is probably good enough for most people as it currently stands.

What I like about it, other than its generality, is that it is clean in the sense it doesn't make any direct calls to write, flush, os.dup2, etc.

I know this question has been answered repeatedly, but for this I've taken the main answer from John T's answer and modified it so it contains the suggested flush and followed its linked revised version. I've also added the enter and exit as mentioned in cladmi's answer for use with the with statement. In addition, the documentation mentions to flush files using os.fsync() so I've added that as well. I don't know if you really need that but its there.

import sys, os class Logger(object): "Lumberjack class - duplicates sys.stdout to a log file and it's okay" #source: https://stackoverflow.com/q/616645 def __init__(self, filename="Red.Wood", mode="a", buff=0): self.stdout = sys.stdout self.file = open(filename, mode, buff) sys.stdout = self def __del__(self): self.close() def __enter__(self): pass def __exit__(self, *args): self.close() def write(self, message): self.stdout.write(message) self.file.write(message) def flush(self): self.stdout.flush() self.file.flush() os.fsync(self.file.fileno()) def close(self): if self.stdout != None: sys.stdout = self.stdout self.stdout = None if self.file != None: self.file.close() self.file = None 

You can then use it

with Logger('My_best_girlie_by_my.side'): print("we'd sing sing sing") 

or

Log=Logger('Sleeps_all.night') print('works all day') Log.close() 

As described elsewhere, perhaps the best solution is to use the logging module directly:

import logging logging.basicConfig(level=logging.DEBUG, filename='mylog.log') logging.info('this should to write to the log file') 

However, there are some (rare) occasions where you really want to redirect stdout. I had this situation when I was extending django's runserver command which uses print: I didn't want to hack the django source but needed the print statements to go to a file.

This is a way of redirecting stdout and stderr away from the shell using the logging module:

import logging, sys class LogFile(object): """File-like object to log text using the `logging` module.""" def __init__(self, name=None): self.logger = logging.getLogger(name) def write(self, msg, level=logging.INFO): self.logger.log(level, msg) def flush(self): for handler in self.logger.handlers: handler.flush() logging.basicConfig(level=logging.DEBUG, filename='mylog.log') # Redirect stdout and stderr sys.stdout = LogFile('stdout') sys.stderr = LogFile('stderr') print 'this should to write to the log file' 

You should only use this LogFile implementation if you really cannot use the logging module directly.

To complete John T answer: https://stackoverflow.com/a/616686/395687

I added __enter__ and __exit__ methods to use it as a context manager with the with keyword, which gives this code

class Tee(object): def __init__(self, name, mode): self.file = open(name, mode) self.stdout = sys.stdout sys.stdout = self def __del__(self): sys.stdout = self.stdout self.file.close() def write(self, data): self.file.write(data) self.stdout.write(data) def __enter__(self): pass def __exit__(self, _type, _value, _traceback): pass 

It can then be used as

with Tee('outfile.log', 'w'): print('I am written to both stdout and outfile.log') 

I wrote a tee() implementation in Python that should work for most cases, and it works on Windows also.

https://github.com/pycontribs/tendo

Also, you can use it in combination with logging module from Python if you want.

(Ah, just reread your question and see that this doesn't quite apply.)

Here is a sample program that makes use of the python logging module. This logging module has been in all versions since 2.3. In this sample, the logging is configurable by command line options.

In quiet mode, it will only log to a file. In normal mode, it will log to both a file and the console.

import os import sys import logging from optparse import OptionParser def initialize_logging(options): """ Log information based upon users options""" logger = logging.getLogger('project') formatter = logging.Formatter('%(asctime)s %(levelname)s\t%(message)s') level = logging.__dict__.get(options.loglevel.upper(),logging.DEBUG) logger.setLevel(level) # Output logging information to screen if not options.quiet: hdlr = logging.StreamHandler(sys.stderr) hdlr.setFormatter(formatter) logger.addHandler(hdlr) # Output logging information to file logfile = os.path.join(options.logdir, "project.log") if options.clean and os.path.isfile(logfile): os.remove(logfile) hdlr2 = logging.FileHandler(logfile) hdlr2.setFormatter(formatter) logger.addHandler(hdlr2) return logger def main(argv=None): if argv is None: argv = sys.argv[1:] # Setup command line options parser = OptionParser("usage: %prog [options]") parser.add_option("-l", "--logdir", dest="logdir", default=".", help="log DIRECTORY (default ./)") parser.add_option("-v", "--loglevel", dest="loglevel", default="debug", help="logging level (debug, info, error)") parser.add_option("-q", "--quiet", action="store_true", dest="quiet", help="do not log to console") parser.add_option("-c", "--clean", dest="clean", action="store_true", default=False, help="remove old log file") # Process command line options (options, args) = parser.parse_args(argv) # Setup logger format and output locations logger = initialize_logging(options) # Examples logger.error("This is an error message.") logger.info("This is an info message.") logger.debug("This is a debug message.") if __name__ == "__main__": sys.exit(main()) 

I've been using Jacob Gabrielson's accepted solution for about 1 year, but now the inevitable has happened and one of my users wants this on Windows. Looking at the other proposed answers, I think most of these answers fail at capturing the outputs of spawned processes (as bolded by the original poster); I think the only way to do this is to do os.dup2(). I think I've figured out how to answer the original poster's exact question, and without using the Unix-specific tool tee: I can now capture all outputs of my Python program, including any spawned shell commands. This works on Windows, Mac and Linux.

The code is as follows:
(The latest version of the code with bugfixes can be found here)

import os, sys, threading, platform class StreamCapture: def __init__(self,stream,writer,echo=True,monkeypatch=None): self.active = True self.writer = writer self.stream = stream self.fd = stream.fileno() self.echo = echo (r,w) = os.pipe() self.pipe_read_fd = r self.pipe_write_fd = w self.dup_fd = os.dup(self.fd) os.dup2(w,self.fd) self.monkeypatch = monkeypatch if monkeypatch is not None else platform.system()=='Windows' if self.monkeypatch: self.oldwrite = stream.write stream.write = lambda z: os.write(self.fd,z.encode() if type(z)==str else z) t = threading.Thread(target=self.printer) self.thread = t t.start() def printer(self): while True: data = os.read(self.pipe_read_fd,100000) if(len(data)==0): self.writer.close() os.close(self.dup_fd) os.close(self.pipe_read_fd) return self.writer.write(data) if self.echo: os.write(self.dup_fd,data) def close(self): if not self.active: return self.active = False self.stream.flush() if self.monkeypatch: self.stream.write = self.oldwrite os.dup2(self.dup_fd,self.fd) os.close(self.pipe_write_fd) self.thread.join() def __enter__(self): return self def __exit__(self,a,b,c): self.close() 

You use it like this (notice the hard case that other solutions, apart from Jacob Gabrielson's, fail to capture):

print("This does not get saved to the log file") with StreamCapture(sys.stdout,open('logfile.txt','wb')): os.write(sys.stdout.fileno(),b"Hello, captured world!\n") os.system('echo Hello from the shell') # Hard case print("More capturing") print("This also does not get saved to the log file") 

This is not a short-and-sweet answer, but I tried to keep it succinct, and it's as simple as I could make it. It's complicated for the following reasons:

1. Since I cannot use tee, I have to somehow perform the task of tee from within my Python process. It was not clear to me that there was a portable way of fork()ing and communicating with an os.pipe() (this states it's hard to share filedescriptors with forked processes in Windows) so I decided to use threading.

2. In Windows, sys.stdout and sys.stderr really don't appreciate when their underlying fileno() get rerouted through an os.pipe() via os.dup2(). The Python interpreter crashes immediately after the first print(...) command.

3. On Windows only, to solve the interpreter crashes, I monkeypatch sys.stdout.write = ... by setting it to a new function that simply calls to os.write(...). By default, I only do this when Windows is detected. Because I monkeypatch, I'm hoping this will reach all cached references to sys.stdout. I chose this monkeypatching approach instead of allocating a brand new stream, e.g. sys.stdout=..., because I was concerned that copies of the old sys.stdout would remain cached in various parts of the interpreter, but I guessed that sys.stdout.write was less likely to have been directly cached.

4. If you daemonize the thread that processes the output of the pipe, then that thread gets killed as soon as the main thread completes, but this does not guarantee that all outputs have been written to the log file. It's actually necessary to not daemonize those helper threads and to let them gracefully terminate themselves when the pipes are closed.

I'm actually not entirely sure that I got all the corner cases right -- threaded code that interacts with delicate OS features is scary to write. Nevertheless, it passed my tests thus far. Because it's kind of hairy, I've made a PyPI package:

pip install streamcapture 

The Github is here.

another solution using logging module:

import logging import sys log = logging.getLogger('stdxxx') class StreamLogger(object): def __init__(self, stream, prefix=''): self.stream = stream self.prefix = prefix self.data = '' def write(self, data): self.stream.write(data) self.stream.flush() self.data += data tmp = str(self.data) if '\x0a' in tmp or '\x0d' in tmp: tmp = tmp.rstrip('\x0a\x0d') log.info('%s%s' % (self.prefix, tmp)) self.data = '' logging.basicConfig(level=logging.INFO, filename='text.log', filemode='a') sys.stdout = StreamLogger(sys.stdout, '[stdout] ') print 'test for stdout' 

None of the answers above really seems to answer the problem posed. I know this is an old thread, but I think this problem is a lot simpler than everyone is making it:

class tee_err(object): def __init__(self): self.errout = sys.stderr sys.stderr = self self.log = 'logfile.log' log = open(self.log,'w') log.close() def write(self, line): log = open(self.log,'a') log.write(line) log.close() self.errout.write(line) 

Now this will repeat everything to the normal sys.stderr handler and your file. Create another class tee_out for sys.stdout.

As per a request by @user5359531 in the comments under @John T's answer, here's a copy of the referenced post to the revised version of the linked discussion in that answer:

Issue of redirecting the stdout to both file and screen Gabriel Genellina gagsl-py2 at yahoo.com.ar Mon May 28 12:45:51 CEST 2007 Previous message: Issue of redirecting the stdout to both file and screen Next message: Formal interfaces with Python Messages sorted by: [ date ] [ thread ] [ subject ] [ author ] En Mon, 28 May 2007 06:17:39 -0300, 人言落日是天涯，望极天涯不见家 <kelvin.you at gmail.com> escribió: > I wanna print the log to both the screen and file, so I simulatered a > 'tee' > > class Tee(file): > > def __init__(self, name, mode): > file.__init__(self, name, mode) > self.stdout = sys.stdout > sys.stdout = self > > def __del__(self): > sys.stdout = self.stdout > self.close() > > def write(self, data): > file.write(self, data) > self.stdout.write(data) > > Tee('logfile', 'w') > print >>sys.stdout, 'abcdefg' > > I found that it only output to the file, nothing to screen. Why? > It seems the 'write' function was not called when I *print* something. You create a Tee instance and it is immediately garbage collected. I'd restore sys.stdout on Tee.close, not __del__ (you forgot to call the inherited __del__ method, btw). Mmm, doesn't work. I think there is an optimization somewhere: if it looks like a real file object, it uses the original file write method, not yours. The trick would be to use an object that does NOT inherit from file: import sys class TeeNoFile(object): def __init__(self, name, mode): self.file = open(name, mode) self.stdout = sys.stdout sys.stdout = self def close(self): if self.stdout is not None: sys.stdout = self.stdout self.stdout = None if self.file is not None: self.file.close() self.file = None def write(self, data): self.file.write(data) self.stdout.write(data) def flush(self): self.file.flush() self.stdout.flush() def __del__(self): self.close() tee=TeeNoFile('logfile', 'w') print 'abcdefg' print 'another line' tee.close() print 'screen only' del tee # should do nothing -- Gabriel Genellina 

I'm writing a script to run cmd-line scripts. ( Because in some cases, there just is no viable substitute for a Linux command -- such as the case of rsync. )

What I really wanted was to use the default python logging mechanism in every case where it was possible to do so, but to still capture any error when something went wrong that was unanticipated.

This code seems to do the trick. It may not be particularly elegant or efficient ( although it doesn't use string+=string, so at least it doesn't have that particular potential bottle- neck ). I'm posting it in case it gives someone else any useful ideas.

import logging import os, sys import datetime # Get name of module, use as application name try: ME=os.path.split(__file__)[-1].split('.')[0] except: ME='pyExec_' LOG_IDENTIFIER="uuu___( o O )___uuu " LOG_IDR_LENGTH=len(LOG_IDENTIFIER) class PyExec(object): # Use this to capture all possible error / output to log class SuperTee(object): # Original reference: http://mail.python.org/pipermail/python-list/2007-May/442737.html def __init__(self, name, mode): self.fl = open(name, mode) self.fl.write('\n') self.stdout = sys.stdout self.stdout.write('\n') self.stderr = sys.stderr sys.stdout = self sys.stderr = self def __del__(self): self.fl.write('\n') self.fl.flush() sys.stderr = self.stderr sys.stdout = self.stdout self.fl.close() def write(self, data): # If the data to write includes the log identifier prefix, then it is already formatted if data[0:LOG_IDR_LENGTH]==LOG_IDENTIFIER: self.fl.write("%s\n" % data[LOG_IDR_LENGTH:]) self.stdout.write(data[LOG_IDR_LENGTH:]) # Otherwise, we can give it a timestamp else: timestamp=str(datetime.datetime.now()) if 'Traceback' == data[0:9]: data='%s: %s' % (timestamp, data) self.fl.write(data) else: self.fl.write(data) self.stdout.write(data) def __init__(self, aName, aCmd, logFileName='', outFileName=''): # Using name for 'logger' (context?), which is separate from the module or the function baseFormatter=logging.Formatter("%(asctime)s \t %(levelname)s \t %(name)s:%(module)s:%(lineno)d \t %(message)s") errorFormatter=logging.Formatter(LOG_IDENTIFIER + "%(asctime)s \t %(levelname)s \t %(name)s:%(module)s:%(lineno)d \t %(message)s") if logFileName: # open passed filename as append fl=logging.FileHandler("%s.log" % aName) else: # otherwise, use log filename as a one-time use file fl=logging.FileHandler("%s.log" % aName, 'w') fl.setLevel(logging.DEBUG) fl.setFormatter(baseFormatter) # This will capture stdout and CRITICAL and beyond errors if outFileName: teeFile=PyExec.SuperTee("%s_out.log" % aName) else: teeFile=PyExec.SuperTee("%s_out.log" % aName, 'w') fl_out=logging.StreamHandler( teeFile ) fl_out.setLevel(logging.CRITICAL) fl_out.setFormatter(errorFormatter) # Set up logging self.log=logging.getLogger('pyExec_main') log=self.log log.addHandler(fl) log.addHandler(fl_out) print "Test print statement." log.setLevel(logging.DEBUG) log.info("Starting %s", ME) log.critical("Critical.") # Caught exception try: raise Exception('Exception test.') except Exception,e: log.exception(str(e)) # Uncaught exception a=2/0 PyExec('test_pyExec',None) 

Obviously, if you're not as subject to whimsy as I am, replace LOG_IDENTIFIER with another string that you're not like to ever see someone write to a log.

Here's a context manager that temporarily duplicates stdout to a file. It's an improvement in my view because it resets sys.stdout & closes the file even when exceptions occur, and the syntax is indicative of an invisible change in the background. Expaneded on John T's solution.

class DuplicateStdout: def __init__(self, path): self.stdout = sys.stdout self.path = path self.f = None def write(self, s): self.stdout.write(s) self.f.write(s) def __enter__(self): self.f = open(self.path, "w") sys.stdout = self def __exit__(self, *args): sys.stdout = self.stdout self.f.close() 

Example usage:

with DuplicateStdout("foo.log"): print("Hey") # also in foo.log print("There") # not in foo.log 

You can also add stderr as well, based on shx2's answer above using class multifile :

class Log(object): def __init__(self, path_log, mode="w", encoding="utf-8"): h = open(path_log, mode, encoding=encoding) sys.stdout = multifile([ sys.stdout, h ]) sys.stderr = multifile([ sys.stderr, h ]) def __enter__(self): """ Necessary if called by with (or with... as) """ return self # only necessary if "as" def __exit__(self, type, value, tb): """ Necessary if call by with """ pass def __del__(self): if sys is not None: # restoring sys.stdout = sys.__stdout__ sys.stderr = sys.__stderr__ log = Log("test.txt") print("line 1") print("line 2", file=sys.stderr) del log print("line 3 only on screen") 

Something like

import sys class Tee(object): def __init__(self, name, mode="w"): self.file = open(name, mode) def write(self, data): self.file.write(data) self.stdout.write(data) def __enter__(self): self.stdout = sys.stdout sys.stdout = self def __exit__(self, _type, _value, _traceback): sys.stdout = self.stdout self.file.close() 

Exploits context manager capabilities available in Python.

Example of usage:

with Tee(diffile) as f: result = subprocess.run(['diff', '-w', '-u0', infile, outfile], stdout=f) 

I wrote a full replacement for sys.stderr and just duplicated the code renaming stderr to stdout to make it also available to replace sys.stdout.

To do this I create the same object type as the current stderr and stdout, and forward all methods to the original system stderr and stdout:

import os import sys import logging class StdErrReplament(object): """ How to redirect stdout and stderr to logger in Python https://stackoverflow.com/questions/19425736/how-to-redirect-stdout-and-stderr-to-logger-in-python Set a Read-Only Attribute in Python? https://stackoverflow.com/questions/24497316/set-a-read-only-attribute-in-python """ is_active = False @classmethod def lock(cls, logger): """ Attach this singleton logger to the `sys.stderr` permanently. """ global _stderr_singleton global _stderr_default global _stderr_default_class_type # On Sublime Text, `sys.__stderr__` is set to None, because they already replaced `sys.stderr` # by some `_LogWriter()` class, then just save the current one over there. if not sys.__stderr__: sys.__stderr__ = sys.stderr try: _stderr_default _stderr_default_class_type except NameError: _stderr_default = sys.stderr _stderr_default_class_type = type( _stderr_default ) # Recreate the sys.stderr logger when it was reset by `unlock()` if not cls.is_active: cls.is_active = True _stderr_write = _stderr_default.write logger_call = logger.debug clean_formatter = logger.clean_formatter global _sys_stderr_write global _sys_stderr_write_hidden if sys.version_info <= (3,2): logger.file_handler.terminator = '\n' # Always recreate/override the internal write function used by `_sys_stderr_write` def _sys_stderr_write_hidden(*args, **kwargs): """ Suppress newline in Python logging module https://stackoverflow.com/questions/7168790/suppress-newline-in-python-logging-module """ try: _stderr_write( *args, **kwargs ) file_handler = logger.file_handler formatter = file_handler.formatter terminator = file_handler.terminator file_handler.formatter = clean_formatter file_handler.terminator = "" kwargs['extra'] = {'_duplicated_from_file': True} logger_call( *args, **kwargs ) file_handler.formatter = formatter file_handler.terminator = terminator except Exception: logger.exception( "Could not write to the file_handler: %s(%s)", file_handler, logger ) cls.unlock() # Only create one `_sys_stderr_write` function pointer ever try: _sys_stderr_write except NameError: def _sys_stderr_write(*args, **kwargs): """ Hides the actual function pointer. This allow the external function pointer to be cached while the internal written can be exchanged between the standard `sys.stderr.write` and our custom wrapper around it. """ _sys_stderr_write_hidden( *args, **kwargs ) try: # Only create one singleton instance ever _stderr_singleton except NameError: class StdErrReplamentHidden(_stderr_default_class_type): """ Which special methods bypasses __getattribute__ in Python? https://stackoverflow.com/questions/12872695/which-special-methods-bypasses-getattribute-in-python """ if hasattr( _stderr_default, "__abstractmethods__" ): __abstractmethods__ = _stderr_default.__abstractmethods__ if hasattr( _stderr_default, "__base__" ): __base__ = _stderr_default.__base__ if hasattr( _stderr_default, "__bases__" ): __bases__ = _stderr_default.__bases__ if hasattr( _stderr_default, "__basicsize__" ): __basicsize__ = _stderr_default.__basicsize__ if hasattr( _stderr_default, "__call__" ): __call__ = _stderr_default.__call__ if hasattr( _stderr_default, "__class__" ): __class__ = _stderr_default.__class__ if hasattr( _stderr_default, "__delattr__" ): __delattr__ = _stderr_default.__delattr__ if hasattr( _stderr_default, "__dict__" ): __dict__ = _stderr_default.__dict__ if hasattr( _stderr_default, "__dictoffset__" ): __dictoffset__ = _stderr_default.__dictoffset__ if hasattr( _stderr_default, "__dir__" ): __dir__ = _stderr_default.__dir__ if hasattr( _stderr_default, "__doc__" ): __doc__ = _stderr_default.__doc__ if hasattr( _stderr_default, "__eq__" ): __eq__ = _stderr_default.__eq__ if hasattr( _stderr_default, "__flags__" ): __flags__ = _stderr_default.__flags__ if hasattr( _stderr_default, "__format__" ): __format__ = _stderr_default.__format__ if hasattr( _stderr_default, "__ge__" ): __ge__ = _stderr_default.__ge__ if hasattr( _stderr_default, "__getattribute__" ): __getattribute__ = _stderr_default.__getattribute__ if hasattr( _stderr_default, "__gt__" ): __gt__ = _stderr_default.__gt__ if hasattr( _stderr_default, "__hash__" ): __hash__ = _stderr_default.__hash__ if hasattr( _stderr_default, "__init__" ): __init__ = _stderr_default.__init__ if hasattr( _stderr_default, "__init_subclass__" ): __init_subclass__ = _stderr_default.__init_subclass__ if hasattr( _stderr_default, "__instancecheck__" ): __instancecheck__ = _stderr_default.__instancecheck__ if hasattr( _stderr_default, "__itemsize__" ): __itemsize__ = _stderr_default.__itemsize__ if hasattr( _stderr_default, "__le__" ): __le__ = _stderr_default.__le__ if hasattr( _stderr_default, "__lt__" ): __lt__ = _stderr_default.__lt__ if hasattr( _stderr_default, "__module__" ): __module__ = _stderr_default.__module__ if hasattr( _stderr_default, "__mro__" ): __mro__ = _stderr_default.__mro__ if hasattr( _stderr_default, "__name__" ): __name__ = _stderr_default.__name__ if hasattr( _stderr_default, "__ne__" ): __ne__ = _stderr_default.__ne__ if hasattr( _stderr_default, "__new__" ): __new__ = _stderr_default.__new__ if hasattr( _stderr_default, "__prepare__" ): __prepare__ = _stderr_default.__prepare__ if hasattr( _stderr_default, "__qualname__" ): __qualname__ = _stderr_default.__qualname__ if hasattr( _stderr_default, "__reduce__" ): __reduce__ = _stderr_default.__reduce__ if hasattr( _stderr_default, "__reduce_ex__" ): __reduce_ex__ = _stderr_default.__reduce_ex__ if hasattr( _stderr_default, "__repr__" ): __repr__ = _stderr_default.__repr__ if hasattr( _stderr_default, "__setattr__" ): __setattr__ = _stderr_default.__setattr__ if hasattr( _stderr_default, "__sizeof__" ): __sizeof__ = _stderr_default.__sizeof__ if hasattr( _stderr_default, "__str__" ): __str__ = _stderr_default.__str__ if hasattr( _stderr_default, "__subclasscheck__" ): __subclasscheck__ = _stderr_default.__subclasscheck__ if hasattr( _stderr_default, "__subclasses__" ): __subclasses__ = _stderr_default.__subclasses__ if hasattr( _stderr_default, "__subclasshook__" ): __subclasshook__ = _stderr_default.__subclasshook__ if hasattr( _stderr_default, "__text_signature__" ): __text_signature__ = _stderr_default.__text_signature__ if hasattr( _stderr_default, "__weakrefoffset__" ): __weakrefoffset__ = _stderr_default.__weakrefoffset__ if hasattr( _stderr_default, "mro" ): mro = _stderr_default.mro def __init__(self): """ Override any super class `type( _stderr_default )` constructor, so we can instantiate any kind of `sys.stderr` replacement object, in case it was already replaced by something else like on Sublime Text with `_LogWriter()`. Assures all attributes were statically replaced just above. This should happen in case some new attribute is added to the python language. This also ignores the only two methods which are not equal, `__init__()` and `__getattribute__()`. """ different_methods = ("__init__", "__getattribute__") attributes_to_check = set( dir( object ) + dir( type ) ) for attribute in attributes_to_check: if attribute not in different_methods \ and hasattr( _stderr_default, attribute ): base_class_attribute = super( _stderr_default_class_type, self ).__getattribute__( attribute ) target_class_attribute = _stderr_default.__getattribute__( attribute ) if base_class_attribute != target_class_attribute: sys.stderr.write( " The base class attribute `%s` is different from the target class:\n%s\n%s\n\n" % ( attribute, base_class_attribute, target_class_attribute ) ) def __getattribute__(self, item): if item == 'write': return _sys_stderr_write try: return _stderr_default.__getattribute__( item ) except AttributeError: return super( _stderr_default_class_type, _stderr_default ).__getattribute__( item ) _stderr_singleton = StdErrReplamentHidden() sys.stderr = _stderr_singleton return cls @classmethod def unlock(cls): """ Detach this `stderr` writer from `sys.stderr` and allow the next call to `lock()` create a new writer for the stderr. """ if cls.is_active: global _sys_stderr_write_hidden cls.is_active = False _sys_stderr_write_hidden = _stderr_default.write class StdOutReplament(object): """ How to redirect stdout and stderr to logger in Python https://stackoverflow.com/questions/19425736/how-to-redirect-stdout-and-stderr-to-logger-in-python Set a Read-Only Attribute in Python? https://stackoverflow.com/questions/24497316/set-a-read-only-attribute-in-python """ is_active = False @classmethod def lock(cls, logger): """ Attach this singleton logger to the `sys.stdout` permanently. """ global _stdout_singleton global _stdout_default global _stdout_default_class_type # On Sublime Text, `sys.__stdout__` is set to None, because they already replaced `sys.stdout` # by some `_LogWriter()` class, then just save the current one over there. if not sys.__stdout__: sys.__stdout__ = sys.stdout try: _stdout_default _stdout_default_class_type except NameError: _stdout_default = sys.stdout _stdout_default_class_type = type( _stdout_default ) # Recreate the sys.stdout logger when it was reset by `unlock()` if not cls.is_active: cls.is_active = True _stdout_write = _stdout_default.write logger_call = logger.debug clean_formatter = logger.clean_formatter global _sys_stdout_write global _sys_stdout_write_hidden if sys.version_info <= (3,2): logger.file_handler.terminator = '\n' # Always recreate/override the internal write function used by `_sys_stdout_write` def _sys_stdout_write_hidden(*args, **kwargs): """ Suppress newline in Python logging module https://stackoverflow.com/questions/7168790/suppress-newline-in-python-logging-module """ try: _stdout_write( *args, **kwargs ) file_handler = logger.file_handler formatter = file_handler.formatter terminator = file_handler.terminator file_handler.formatter = clean_formatter file_handler.terminator = "" kwargs['extra'] = {'_duplicated_from_file': True} logger_call( *args, **kwargs ) file_handler.formatter = formatter file_handler.terminator = terminator except Exception: logger.exception( "Could not write to the file_handler: %s(%s)", file_handler, logger ) cls.unlock() # Only create one `_sys_stdout_write` function pointer ever try: _sys_stdout_write except NameError: def _sys_stdout_write(*args, **kwargs): """ Hides the actual function pointer. This allow the external function pointer to be cached while the internal written can be exchanged between the standard `sys.stdout.write` and our custom wrapper around it. """ _sys_stdout_write_hidden( *args, **kwargs ) try: # Only create one singleton instance ever _stdout_singleton except NameError: class StdOutReplamentHidden(_stdout_default_class_type): """ Which special methods bypasses __getattribute__ in Python? https://stackoverflow.com/questions/12872695/which-special-methods-bypasses-getattribute-in-python """ if hasattr( _stdout_default, "__abstractmethods__" ): __abstractmethods__ = _stdout_default.__abstractmethods__ if hasattr( _stdout_default, "__base__" ): __base__ = _stdout_default.__base__ if hasattr( _stdout_default, "__bases__" ): __bases__ = _stdout_default.__bases__ if hasattr( _stdout_default, "__basicsize__" ): __basicsize__ = _stdout_default.__basicsize__ if hasattr( _stdout_default, "__call__" ): __call__ = _stdout_default.__call__ if hasattr( _stdout_default, "__class__" ): __class__ = _stdout_default.__class__ if hasattr( _stdout_default, "__delattr__" ): __delattr__ = _stdout_default.__delattr__ if hasattr( _stdout_default, "__dict__" ): __dict__ = _stdout_default.__dict__ if hasattr( _stdout_default, "__dictoffset__" ): __dictoffset__ = _stdout_default.__dictoffset__ if hasattr( _stdout_default, "__dir__" ): __dir__ = _stdout_default.__dir__ if hasattr( _stdout_default, "__doc__" ): __doc__ = _stdout_default.__doc__ if hasattr( _stdout_default, "__eq__" ): __eq__ = _stdout_default.__eq__ if hasattr( _stdout_default, "__flags__" ): __flags__ = _stdout_default.__flags__ if hasattr( _stdout_default, "__format__" ): __format__ = _stdout_default.__format__ if hasattr( _stdout_default, "__ge__" ): __ge__ = _stdout_default.__ge__ if hasattr( _stdout_default, "__getattribute__" ): __getattribute__ = _stdout_default.__getattribute__ if hasattr( _stdout_default, "__gt__" ): __gt__ = _stdout_default.__gt__ if hasattr( _stdout_default, "__hash__" ): __hash__ = _stdout_default.__hash__ if hasattr( _stdout_default, "__init__" ): __init__ = _stdout_default.__init__ if hasattr( _stdout_default, "__init_subclass__" ): __init_subclass__ = _stdout_default.__init_subclass__ if hasattr( _stdout_default, "__instancecheck__" ): __instancecheck__ = _stdout_default.__instancecheck__ if hasattr( _stdout_default, "__itemsize__" ): __itemsize__ = _stdout_default.__itemsize__ if hasattr( _stdout_default, "__le__" ): __le__ = _stdout_default.__le__ if hasattr( _stdout_default, "__lt__" ): __lt__ = _stdout_default.__lt__ if hasattr( _stdout_default, "__module__" ): __module__ = _stdout_default.__module__ if hasattr( _stdout_default, "__mro__" ): __mro__ = _stdout_default.__mro__ if hasattr( _stdout_default, "__name__" ): __name__ = _stdout_default.__name__ if hasattr( _stdout_default, "__ne__" ): __ne__ = _stdout_default.__ne__ if hasattr( _stdout_default, "__new__" ): __new__ = _stdout_default.__new__ if hasattr( _stdout_default, "__prepare__" ): __prepare__ = _stdout_default.__prepare__ if hasattr( _stdout_default, "__qualname__" ): __qualname__ = _stdout_default.__qualname__ if hasattr( _stdout_default, "__reduce__" ): __reduce__ = _stdout_default.__reduce__ if hasattr( _stdout_default, "__reduce_ex__" ): __reduce_ex__ = _stdout_default.__reduce_ex__ if hasattr( _stdout_default, "__repr__" ): __repr__ = _stdout_default.__repr__ if hasattr( _stdout_default, "__setattr__" ): __setattr__ = _stdout_default.__setattr__ if hasattr( _stdout_default, "__sizeof__" ): __sizeof__ = _stdout_default.__sizeof__ if hasattr( _stdout_default, "__str__" ): __str__ = _stdout_default.__str__ if hasattr( _stdout_default, "__subclasscheck__" ): __subclasscheck__ = _stdout_default.__subclasscheck__ if hasattr( _stdout_default, "__subclasses__" ): __subclasses__ = _stdout_default.__subclasses__ if hasattr( _stdout_default, "__subclasshook__" ): __subclasshook__ = _stdout_default.__subclasshook__ if hasattr( _stdout_default, "__text_signature__" ): __text_signature__ = _stdout_default.__text_signature__ if hasattr( _stdout_default, "__weakrefoffset__" ): __weakrefoffset__ = _stdout_default.__weakrefoffset__ if hasattr( _stdout_default, "mro" ): mro = _stdout_default.mro def __init__(self): """ Override any super class `type( _stdout_default )` constructor, so we can instantiate any kind of `sys.stdout` replacement object, in case it was already replaced by something else like on Sublime Text with `_LogWriter()`. Assures all attributes were statically replaced just above. This should happen in case some new attribute is added to the python language. This also ignores the only two methods which are not equal, `__init__()` and `__getattribute__()`. """ different_methods = ("__init__", "__getattribute__") attributes_to_check = set( dir( object ) + dir( type ) ) for attribute in attributes_to_check: if attribute not in different_methods \ and hasattr( _stdout_default, attribute ): base_class_attribute = super( _stdout_default_class_type, self ).__getattribute__( attribute ) target_class_attribute = _stdout_default.__getattribute__( attribute ) if base_class_attribute != target_class_attribute: sys.stdout.write( " The base class attribute `%s` is different from the target class:\n%s\n%s\n\n" % ( attribute, base_class_attribute, target_class_attribute ) ) def __getattribute__(self, item): if item == 'write': return _sys_stdout_write try: return _stdout_default.__getattribute__( item ) except AttributeError: return super( _stdout_default_class_type, _stdout_default ).__getattribute__( item ) _stdout_singleton = StdOutReplamentHidden() sys.stdout = _stdout_singleton return cls @classmethod def unlock(cls): """ Detach this `stdout` writer from `sys.stdout` and allow the next call to `lock()` create a new writer for the stdout. """ if cls.is_active: global _sys_stdout_write_hidden cls.is_active = False _sys_stdout_write_hidden = _stdout_default.write 

To use this you can just call StdErrReplament::lock(logger) and StdOutReplament::lock(logger) passing the logger you want to use to send the output text. For example:

import os import sys import logging current_folder = os.path.dirname( os.path.realpath( __file__ ) ) log_file_path = os.path.join( current_folder, "my_log_file.txt" ) file_handler = logging.FileHandler( log_file_path, 'a' ) file_handler.formatter = logging.Formatter( "%(asctime)s %(name)s %(levelname)s - %(message)s", "%Y-%m-%d %H:%M:%S" ) log = logging.getLogger( __name__ ) log.setLevel( "DEBUG" ) log.addHandler( file_handler ) log.file_handler = file_handler log.clean_formatter = logging.Formatter( "", "" ) StdOutReplament.lock( log ) StdErrReplament.lock( log ) log.debug( "I am doing usual logging debug..." ) sys.stderr.write( "Tests 1...\n" ) sys.stdout.write( "Tests 2...\n" ) 

Running this code, you will see on the screen:

And on the file contents:

If you would like to also see the contents of the log.debug calls on the screen, you will need to add a stream handler to your logger. On this case it would be like this:

import os import sys import logging class ContextFilter(logging.Filter): """ This filter avoids duplicated information to be displayed to the StreamHandler log. """ def filter(self, record): return not "_duplicated_from_file" in record.__dict__ current_folder = os.path.dirname( os.path.realpath( __file__ ) ) log_file_path = os.path.join( current_folder, "my_log_file.txt" ) stream_handler = logging.StreamHandler() file_handler = logging.FileHandler( log_file_path, 'a' ) formatter = logging.Formatter( "%(asctime)s %(name)s %(levelname)s - %(message)s", "%Y-%m-%d %H:%M:%S" ) file_handler.formatter = formatter stream_handler.formatter = formatter stream_handler.addFilter( ContextFilter() ) log = logging.getLogger( __name__ ) log.setLevel( "DEBUG" ) log.addHandler( file_handler ) log.addHandler( stream_handler ) log.file_handler = file_handler log.stream_handler = stream_handler log.clean_formatter = logging.Formatter( "", "" ) StdOutReplament.lock( log ) StdErrReplament.lock( log ) log.debug( "I am doing usual logging debug..." ) sys.stderr.write( "Tests 1...\n" ) sys.stdout.write( "Tests 2...\n" ) 

Which would output like this when running:

While it would still saving this to the file my_log_file.txt:

When disabling this with StdErrReplament:unlock(), it will only restore the standard behavior of the stderr stream, as the attached logger cannot be never detached because someone else can have a reference to its older version. This is why it is a global singleton which can never dies. Therefore, in case of reloading this module with imp or something else, it will never recapture the current sys.stderr as it was already injected on it and have it saved internally.