45

Is it possible to read stdin as binary data in Python 2.6? If so, how?

I see in the Python 3.1 documentation that this is fairly simple, but the facilities for doing this in 2.6 don't seem to be there.

If the methods described in 3.1 aren't available, is there a way to close stdin and reopen in in binary mode?

Just to be clear, I am using 'type' in a MS-DOS shell to pipe the contents of a binary file to my python code. This should be the equivalent of a Unix 'cat' command, as far as I understand. But when I test this out, I always get one byte less than the expected file size.

The reason I'm going the Java/JAR/Jython route is because one of my main external libraries is only available as a Java JAR. But unfortunately, I had started my work as Python. It might have been easier to convert my code over to Java a while ago, but since this stuff was all supposed to be compatible, I figured I would try trucking through it and prove it could be done.

In case anyone was wondering, this is also related to this question I asked a few days ago.

Some of was answered in this question.

So I'll try to update my original question with some notes on what I have figured out so far.

Improve this question

7 Answers 7

Reset to default
30

From the docs (see here):

The standard streams are in text mode by default. To write or read binary data to these, use the underlying binary buffer. For example, to write bytes to stdout, use sys.stdout.buffer.write(b'abc').

But, as in the accepted answer, invoking python with a -u is another option which forces stdin, stdout and stderr to be totally unbuffered. See the python(1) manpage for details.

See the documentation on io for more information on text buffering, and use sys.stdin.detach() to disable buffering from within Python.

Improve this answer
Sign up to request clarification or add additional context in comments.

3 Comments

that doc leads to py3k docs, not what the OP wants to deal with.
I've tried -u with Python v3.2.5 but it did nothing useful. But using sys.stdout.buffer works pretty well though on Python 2.7.8 there is no such feature.
The OP's question was about Python 2.6.
25

Here is the final cut for Linux/Windows Python 2/3 compatible code to read data from stdin without corruption:

import sys PY3K = sys.version_info >= (3, 0) if PY3K: source = sys.stdin.buffer else: # Python 2 on Windows opens sys.stdin in text mode, and # binary data that read from it becomes corrupted on \r\n if sys.platform == "win32": # set sys.stdin to binary mode import os, msvcrt msvcrt.setmode(sys.stdin.fileno(), os.O_BINARY) source = sys.stdin b = source.read()
Improve this answer

Comments

15

Use the -u command line switch to force Python 2 to treat stdin, stdout and stderr as binary unbuffered streams.

C:> type mydoc.txt | python.exe -u myscript.py
Improve this answer

6 Comments

I have tested this with 'type' and it appears to work. That is, if I leave out the -u flag, I get one fewer character per line.
According to docs, setting the PYTHONUNBUFFERED environment variable will have the same effect. Not sure if that helps.
Even easier, it appears that all you need to do is: sys.stdin = os.fdopen(sys.stdin.fileno(), 'rb', 0) That will reopen the fd in unbuffered 'binary' mode.
@thebeav: Oddly enough that doesn't work on my system. I don't know if that's because I'm using CPython instead of Jython, or if it's because I'm running Windows XP Pro, and "type" behaves differently, or its because there is a magnetic anomaly in the Manassas area that makes computers do different things. FWIW, I tried a number of ways to get Python to change the file mode after the interpreter had started, including accessing the C runtime's "setmode" function via ctypes. Nothing works for me.
Uh-oh. I smell a portability issue. Thanks for the info. I guess I'm going to have to do some fairly rigorous testing on multiple platforms. I hope this doesn't have to do with the JVM in use.
|
9

If you still need this... This simple test i've used to read binary file that contains 0x1A character in between

import os, sys, msvcrt msvcrt.setmode (sys.stdin.fileno(), os.O_BINARY) s = sys.stdin.read() print len (s)

My test file data was:

0x23, 0x1A, 0x45

Without setting stdin to binary mode this test prints 1 as soon it treats 0x1A as EOF. Of course it works on windows only, because depends on msvcrt module.

Improve this answer

3 Comments

But Windows is the only system where most people will run into a problem, so this should be an acceptable solution.
This is the correct solution for Python 2 to retrieve the raw bytes from stdin on Windows. On Unix, there is no difference between binary and normal mode. See this thread: code.activestate.com/lists/python-list/20426 (re-opening stdin in raw (binary) mode?)
I was getting a ValueError: insecure string pickle exception on Windows when trying to unpickle data that had been written to stdout in one process which was being piped into another. The solution turned out to be adding a msvcrt.setmode(sys.stdout.fileno(), os.O_BINARY) in the process that wrote the data.
4

You can perform an unbuffered read with:

os.read(0, bytes_to_read)

with 0 being the file descriptor for stdin

Improve this answer

Comments

0

To read binary data from stdin (in Python 2.4–2.7, 3.0–, both Unix and Windows), do this:

import os, sys if sys.platform.startswith('win'): try: __import__('msvcrt').setmode(sys.stdout.fileno(), os.O_BINARY) except ImportError: pass sys.stdin = os.fdopen(sys.stdin.fileno(), 'rb') ... print(sys.stdin.read(4096))

To read binary data from stdin, without modifying sys.stdin (in Python 2.4–2.7, 3.0–, both Unix and Windows), do this:

import os, sys f = os.fdopen(os.dup(sys.stdin.fileno()), 'rb') if sys.platform.startswith('win'): try: __import__('msvcrt').setmode(f.fileno(), os.O_BINARY) except ImportError: pass ... print(f.read(4096))

To read binary data unbuffered (i.e. as soon as it is available to the Python process) from a file object, while putting the underlying file descriptor to binary mode, do this (in Python 2.4–2.7, 3.0–, both Unix and Windows):

import os, sys f = sys.stdin # Or anything other file object. if sys.platform.startswith('win'): try: __import__('msvcrt').setmode(f.fileno(), os.O_BINARY) except ImportError: pass ... print(os.read(f.fileno(), 4096))

For unbuffered binary reads, it's tempting to do f = os.fdopen(f.fileno(), 'rb', 0), but in Python 2.x it doesn't make the read (i.e. f.read(4096)) unbuffered, it would still wait indefinitely for more input until the 4096 bytes are filled or EOF is reached.

Improve this answer

Comments

-3 
import sys data = sys.stdin.read(10) # Read 10 bytes from stdin

If you need to interpret binary data, use the struct module.

Improve this answer

5 Comments

If I then call sys.stdin.read() with no parameter, it should read all the binary data that was piped in, correct? How then do I determine the length correctly? len(data) returns the incorrect value if the last byte of the data was a zero. How do you check and correct for this situation?
len counts the \x00 characters in the string. Python does not have null terminated strings. len("Hello\x00") == 6
I wonder then if it might be the 'type' command from the MS-DOS shell that is causing the loss of the final byte? I guess I will have to test the equivalent on Linux. Thanks.
I think this answer misses the point of the question: if the stream is in "text" mode, the results from read() might be different than if the stream is in "binary" mode.
It might corrupt input stream on Windows e.g., '\r\n' -> '\n'. Also, on Python 3 sys.stdin.read() returns Unicode strings e.g., b'\xf0\x9f\x96\x96' -> '\U0001f596' (4 bytes -> 1 chararcter). It is undesirable behaviour if input is not text.

Start asking to get answers

Find the answer to your question by asking.

Ask question

Explore related questions

See similar questions with these tags.