18

I want to run several python scripts simultaneously in one bash session and to inspect their outputs respectively in real time. To fulfil this task, I wrote a simple bash script which is shown below:

#!/bin/bash python 1.py > 1.output & python 2.py > 2.output & python 3.py > 3.output &

When I use cat 1.output command to check what have been printed in the middle of executing, however, nothing can be seen.

After thinking a while, I realise that the 1.output must be filled when 1.py finishes executing. In other words, the method which I used here is not a real time fashion.

You may propse that to wait for these python scripts' finish. The fact, unfortunately, is that all there python scripts are actually long run programs, they maybe finish after days or months, and that is why I want to inspect their outputs in real time.

Also, you may suggest me to modify the python scripts to print message to file directly instead of stdout. Sorry, the scripts here are too complex to modify, there are chucks of print function in them.

what can I do now?

Improve this question
2
  • 1
    have the python scripts flush their output periodically... Commented Jul 10, 2015 at 16:32
  • Post the code you are using for this logging action please. Commented Jul 10, 2015 at 16:32

3 Answers 3

Reset to default
40

The -u switch and the equivalent PYTHONUNBUFFERED environment variable forces stdout to be unbuffered. Try this:

#!/bin/bash python -u 1.py > 1.output & python -u 2.py > 2.output & python -u 3.py > 3.output &

or

#!/bin/bash export PYTHONUNBUFFERED=yes python 1.py > 1.output & python 2.py > 2.output & python 3.py > 3.output &

Note that -u has side effects: read the doc to learn more.

Reference:

Improve this answer
Sign up to request clarification or add additional context in comments.

Comments

0

If you can find some sort of main loop in the program, that could be a good indicator of progress, it would be good to do write-to-file. If you say that the output does not fill until the output stream is closed by Python, then this is probably the simplest alternative.

Improve this answer

Comments

0

You can try to override stderr/stdout in your scripts by doing:

# in the beginning of your script import os import sys desired_output_file = open('/path/to/file', 'a') os.dup2(desired_output_file.fileno(), sys.stdout) os.dup2(desired_output_file.fileno(), sys.stderr)

Then all print calls will write to this special file, not to stdout. However, main problem could be with buffered IO. You have to somehow flush content of such file to the disk.

Improve this answer

Comments

Start asking to get answers

Find the answer to your question by asking.

Ask question

Explore related questions

See similar questions with these tags.