Capture real time `stdout` and `stderr` when run a function in a process python

Question

I have a python function and want to run it as a separate process with multiprocessing package.

def run(ctx: Context): print("hello world!") return ctx

afterward running it as a separate process with the following script:

import multiprocessing p = multiprocessing.Process(target=run, args=(ctx, )) p.start() p.join()

Now, I need to capture live stdout and stderr of the above process. Is there any way like as:

import subprocess proc = subprocess.Popen(['python','fake_utility.py'],stdout=subprocess.PIPE) while True: line = proc.stdout.readline() if not line: break

But I need to pass the function not running a command with Popen. Do you know how can I read stdout when I run my function in a separate process?

You could create a wrapper function which redirects the output of the process before it calls the actual function. — Michael Butscher
– Michael Butscher, Commented Jan 30, 2023 at 11:54

Booboo · Accepted Answer · 2023-02-02 13:28:25Z

My approach would be to create a custom context manager that can temporarily replace sys.stdout and sys.stderr with io.String() instances to capture the output and return this. For this you need to make the target of your Process a new function that can setup the context manager and return the results, for which a multiprocessing.Queue is used (this, by the way, would be needed anyway if you expect run to return its result back to the main process):

from multiprocessing import Process, Queue from io import StringIO import sys class CaptureOutput: def __enter__(self): self._stdout_output = '' self._stderr_output = '' self._stdout = sys.stdout sys.stdout = StringIO() self._stderr = sys.stderr sys.stderr = StringIO() return self def __exit__(self, *args): self._stdout_output = sys.stdout.getvalue() sys.stdout = self._stdout self._stderr_output = sys.stderr.getvalue() sys.stderr = self._stderr def get_stdout(self): return self._stdout_output def get_stderr(self): return self._stderr_output def run(ctx): print("hello world!") print("It works!", file=sys.stderr) raise Exception('Oh oh!') # Comment out to have a successful completion return ctx def worker(ctx, queue): import traceback with CaptureOutput() as capturer: try: result = run(ctx) except Exception as e: result = e print(traceback.format_exc(), file=sys.stderr) queue.put((result, capturer.get_stdout(), capturer.get_stderr())) if __name__ == '__main__': queue = Queue() ctx = None # for demo purposes p = Process(target=worker, args=(ctx, queue)) p.start() # Must do this call before call to join: result, stdout_output, stderr_output = queue.get() p.join() print('stdout:', stdout_output) print('stderr:', stderr_output)

Prints:

stdout: hello world! stderr: It works! Traceback (most recent call last): File "C:\Booboo\test\test.py", line 44, in worker result = run(ctx) File "C:\Booboo\test\test.py", line 36, in run raise Exception('Oh oh!') # Comment out to have a successful completion Exception: Oh oh!

what if one is launching the process using multi-process? (and what about if the child process in it's own term calls tools such as c compilers?) I.e. [1 Main] -> MultiProcessCalls("2child") -> [2 Child] -> Calls[nvcc compile] How can we capture all the output of [2 Child]
What do you mean by "multi-process"? The term I know is "multiprocessing" and the posted code is using that. If the child process wanted to invoke a C compiler, it should use the subprocess module. Read the docs to learn how the child process can capture the output from the C compiler and then print it. In general, though, I would say that if you have a question about extending functionality, you should post it as a new question.
Thanks for your response. Yes, sorry for not being clear enough. I will try and resolve or post a question. I meant a nested multiprocessing + subprocess setup. (with a C compiler being invoked). I'll check it out more, thanks again.

Collectives™ on Stack Overflow

Capture real time `stdout` and `stderr` when run a function in a process python

1 Answer 1

3 Comments

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

3 Comments

Related