Replication of the Bash tee command behavior in python

Question

I have a collection of Bash scripts which I want to recreate in python. One of the key feature of these scripts is when i execute them it will save the contents of the terminal into a logfile. In Bash simply i used the tee command.

2>&1 | tee "logfile.txt";

the problem is to find equal solution for python.

I found two half of this "puzzle" so far (solution A and B), one of the expected behaviour works in one of the scripts but not in the other and vice versa.

solution A)

#!/usr/bin/env python3 import sys from subprocess import Popen, PIPE, STDOUT with Popen(['ffmpeg','-i','1.webm','-y','1.mp3'], stdout=PIPE, stderr=STDOUT, bufsize=1) as p, \ open('logfile.txt', 'ab') as file: for line in p.stdout: sys.stdout.buffer.write(line) file.write(line)

solution B)

#!/usr/bin/env python3 import sys from subprocess import Popen, PIPE with Popen(['ffmpeg','-i','1.webm','-y','1.mp3'], stdout=PIPE, bufsize=1, universal_newlines=True) as p: logfile = open('logfile.txt', 'w') for line in p.stdout: print(line, end='')

i tried to "merge" the features of these 2 code snippets, but i cant figure it out, how to put it together.

What i looking for is the EXACT behavior replication of the tee command in a python script file. which means...

the contents of the terminal appear in the terminal window AND saved into a log file (just like solution A)
when i start the python script file, i want to follow the progress of the process in the terminal, to check how far is from completion (just like solution B). I dont want to stare at a blank screen until the process completes (solution A).

I would appreciate the help.

for testing i use a webm format file (downloaded with youtube-dl) and convert it to mp3 with ffmpeg in cygwin. you can download the ffmpeg binary from here if you want experimenting with it https://www.ffmpeg.org/download.html

Thank you!

Could it be just a matter of flushing stdout after each line in solution A? And why use sys.stdout.buffer.write(line) in solution A instead of print like in solution B? — joanis
– joanis, Commented Mar 17, 2020 at 21:51
I think all you need is indeed sys.stdout.flush() inside your inner loop. Relevant (but not duplicate) question: stackoverflow.com/a/10019605/3216427 — joanis
– joanis, Commented Mar 17, 2020 at 22:00

Geoff Gustafson · Accepted Answer · 2020-03-17 22:50:03Z

I did some testing and no, sys.stdout.flush() doesn't solve it. The problem appears to be within the Popen/PIPE implementation itself -- the way it sets up pipes between the subprocess and your process introduces buffering.

What does seem to fix this is to:

$ export PYTHONUNBUFFERED=1

in the environment you run your Python script from. (The variable can be set to anything.)

To solve this within your Python script itself, there may be a more elegant way, but this rather odd approach seemed to work for me. I'm rerunning the script with that environment variable set:

import os from subprocess import run if not "PYTHONUNBUFFERED" in os.environ: os.environ["PYTHONUNBUFFERED"] = "1" completed = run(sys.argv) sys.exit(completed.returncode)

Found some pointers to this solution here in question 230751.

that other guy · Accepted Answer · 2020-03-17 23:08:21Z

You are reading line by line, but ffmpeg does not output distinct lines.

You should do what tee does and read buffer by buffer, ignoring linefeeds:

#!/usr/bin/env python3.8 import sys from subprocess import Popen, PIPE, STDOUT with Popen(['ffmpeg','-i','1.webm','-y','1.mp3'], stdout=PIPE, stderr=STDOUT, bufsize=0) as p, \ open('logfile.txt', 'ab') as file: while buf := p.stdout.read(4096): sys.stdout.buffer.write(buf); sys.stdout.buffer.flush() file.write(buf)

CuriousOne · Accepted Answer · 2022-09-04 07:02:01Z

I decided to re-open the case, and with some tinkering i was able to come up with the solution, but i thank you all for your efforts!

#! /bin/python3 from subprocess import Popen, PIPE, STDOUT with Popen(['ffmpeg','-i','test.wav','-y','1.mp3'], stdout=PIPE, stderr=STDOUT, universal_newlines=True) as process, \ open('logfile.txt', 'w') as logfile: for line in process.stdout: print(line) logfile.write(line) logfile.close()

Collectives™ on Stack Overflow

Replication of the Bash tee command behavior in python

3 Answers 3

Comments

Comments

Comments

Linked

Hot Network Questions

Collectives™ on Stack Overflow

3 Answers 3

Comments

Comments

Comments

Linked

Related