2

In an attempt to create a simple cat clone in python,

sys.stdout.write(sys.stdin.read())

I noticed that this fails horribly for binary files (i.e. python cat.py < binaryfile > supposed_copy) containing the CTRL+Z EOF / substitude character 0x1a, since that seems to cause read() to consider its work done. I cannot simply loop over the code forever to circumvent this, since obviously at some point stdin.read() will wait until new input is provided, which once the true end of input is reached won't ever happen.

So, how can this be fixed, i.e.

  • How to know when the file redirected to stdin is fully read, or
  • how to properly handle this?
Improve this question
10
  • Sounds like you are opening the file as text instead of binary. Can you show the open statement please? Commented Feb 23, 2015 at 11:26
  • @cdarke: Python opens stdin and stdout for you... Commented Feb 23, 2015 at 11:27
  • 1
    This probably gives you the answer: stackoverflow.com/questions/2850893/… Commented Feb 23, 2015 at 11:29
  • OK, I misunderstood. You probably want to reopen stdin as binary then, but really stdin and stdout are not suitable for binary files. Commented Feb 23, 2015 at 11:30
  • 1
    @Tobias No. Windows libraries have _setmode, which you can call on the associated C file handle (and, as an implementation detail, the C++ streams on Windows will use that). However, that’s then of course Windows specific, and relies on non-standardised behaviour of the streams. In practice this means it can be achieved, but requires platform- and compiler specific code. Commented Feb 23, 2015 at 13:02

3 Answers 3

Reset to default
2

You will need to tell Python to open stdin and stdout in binary mode. You can do this with the -u option. Eg

python -u cat.py < binaryfile > supposed_copy

Note that this will make stdin and stdout unbuffered.

Improve this answer
Sign up to request clarification or add additional context in comments.

4 Comments

That's working great, can the effect of -u be achieved from within Python code?
@TobiasKienzler: Probably not, since those files are already opened when they get passed to your script. There might be some Windows magic that can alter the mode of a stream that's already open, but if so, accessing it from Python would require using something like ctypes (which lets you call functions in DLLs). But anyway, stdin and stdout are special, and usually connected to a terminal, so even if your OS lets you change modes of normal files it might be touchy about stdin & stdout. :) (FWIW, I don't know much about Windows).
Some Windows magic indeed: stackoverflow.com/a/28673339/321973 So much for transparent portability :/
Fortunately. This btw also fixes any CRLF <-> LF issues that binary files would encounter. Oh Microsoft, why, why, why...
2

Expanding on this answer:

if sys.platform == "win32": import msvcrt msvcrt.setmode(sys.stdout.fileno(), os.O_BINARY) msvcrt.setmode(sys.stdin.fileno(), os.O_BINARY)
Improve this answer

3 Comments

Well, that's a bit simpler than I was expecting. :) OTOH, I guess there's a big demand for this kind of thing, so it makes sense that Microsoft have made it fairly easy.
@PM2Ring Indeed, though I wonder why Microsoft made this difference to start with. Just that automated CR/LF conversion mades me curse a lot...
Well, Bill Gates wanted to make MS-DOS different to Unix. And I guess he thought the CRLF and Ctrl-Z things were a good idea at the time. Similar remarks apply to using backslash as a path separator. But discussing these topics can lead to Religious Wars, so I guess I better shut up. :)
1

See Reading binary data from stdin for an explanation of how to make sure stdin/stdout are opened as binary.

Improve this answer

1 Comment

Thanks, indeed one answer there solves this, though it wouldn't harm to mention that here ;) Anyway, I found a similar answer

Start asking to get answers

Find the answer to your question by asking.

Ask question

Explore related questions

See similar questions with these tags.