python gzip "unexpected end of file" when writing a data stream

Question

I'm writing (or more precisely, appending) a real-time data stream to an instance of python's gzip module. If the program that's writing the stream crashes, and relaunches, I would like the stream to automatically be appended to the original file. Unfortunately this fails in practice, as I get an "unexpected end of file" error that corresponds to the exact point at which the original program crashed.

What's the underlying approach for handing this situation, as I can't imagine this should be a hard problem. My approach is outlined below:

f = gzip.GzipFile( 'filename_json.txt.gz' ), mode='at', compresslevel=9) while(something_is_true): f.write(stream['message'] + '\n') f.close()

This runs continuously, but if the program crashes (or gets killed), the end-of-file operator never gets appended, and the gzip file becomes corrupt. In which case, any data appended after that point becomes unreadable.

Thanks!

You could use a try: finally: clause to ensure the file is always closed. — AsksAnyway
– AsksAnyway, Commented May 4, 2014 at 0:04

U2EF1 · Accepted Answer · 2014-05-04 01:14:43Z

1

with gzip.open('filename_json.txt.gz', mode='at', compresslevel=9) as f: while something_is_true: f.write(stream['message'] + '\n')

(This works for me on python 2.7.6)

But if that for some reason isn't working, you can do it the old fashioned way:

try: f = gzip.open('filename_json.txt.gz', mode='at', compresslevel=9) while something_is_true: f.write(stream['message'] + '\n') finally: f.close()

Note that the error will still propagate with this code unless you catch the error. But the file will be closed.

edited May 4, 2014 at 1:14

answered May 4, 2014 at 0:10

U2EF1

13.4k3 gold badges38 silver badges38 bronze badges

Sign up to request clarification or add additional context in comments.

7 Comments

AsksAnyway Over a year ago

Does this function if the process is killed?

U2EF1 Over a year ago

gzip.open and gzip.GzipFile work with with for me on python 2.7.6.

Jeff Epler Over a year ago

@AsksAnyway, no, this only "cures" the case where a Python exception occurs, not the case where the program is killed e.g., by a terminating signal or power loss.

vgoklani Over a year ago

How should we handle the case where the process is killed?

U2EF1 Over a year ago

If you're killing it with e.g. ctrl-c that will send a sigint and raise a python exception. If you're killing it -9 or cutting power then of course it's not going to work.

|

Collectives™ on Stack Overflow

python gzip "unexpected end of file" when writing a data stream

1 Answer 1

7 Comments

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

7 Comments

Related