8

I have 6 gzipped text files, each of which is ~17G when compressed. I need to see the last few lines (decompressed) of each file to check whether a particular problem is there. The obvious approach is very slow:

for i in *; do zcat "$i" | tail -n3; done

I was thinking I could do something clever like:

for i in *; do tail -n 30 "$i" | gunzip | tail -n 4 ; done

Or 

for i in *; do tac "$i" | head -100 | gunzip | tac | tail -n3; done

But both complain about:

gzip: stdin: not in gzip format

I thought that was because I was missing the gzip header, but this also fails:

$ aa=$(head -c 300 file.gz) $ bb=$(tail -c 300 file.gz) $ printf '%s%s' "$aa" "$bb" | gunzip gzip: stdin: unexpected end of file

What I am really looking for is a ztail or ztac but I don't think those exist. Can anyone come up with a clever trick that lets me decompress and print the last few lines of a compressed file without decompressing the entire thing?

Improve this question
2
  • 2
    I don't think you can... See also How can I tail a zipped file without reading its entire contents? and the links there... Commented Jun 28, 2016 at 10:23
  • @don_crissti well, dang! Looks like you're right. I'll leave this for a while in case anyone comes up with something really clever, and if not I'll copy one of the SO answers over. Probably the one stating that gzip doesn't support this. Feel free to do it yourself, by the way. Commented Jun 28, 2016 at 10:27

2 Answers 2

Reset to default
7

You can't, as it has been already said, if the files have been compressed with standard gzip. If you have control over the compression, you can use dictzip to compress the files, it compresses the files in separate blocks and you can decompress just the last block (typically 64KB). And it is backward compatible with gzip, meaning the dictzipped file is perfectly legal gzipped file as well.

Other possibility would be if you get the gzipped file as a concatenation of several already gzipped files, you could search for the last gzip signature and decompress everything after that.

Improve this answer
3
  • I don't, sadly. I was hoping I could use the idea here to get the right chunks but I don't know if that's possible or what the chunks might be. Commented Jun 28, 2016 at 15:05
  • @terdon The idea there is to create the compressed file in chunks and then only uncompress the last chunk. If the compressed file has a single chunk, which is the case if you produced it with gzip, then “only uncompress the last chunk” = “uncompress the sole chunk” = “uncompress the whole file”. Commented Jun 28, 2016 at 21:29
  • @Gilles I see. I had understood that the "chunks" were a result of the gzip algorithm and could perhaps be inferred from the compressed file. Commented Jun 29, 2016 at 8:54
7

Well, you can access randomly a gzipped file if you previously creates an index for each file ...

I've developed the command line tool that you were probably looking for: it can access the tail using the same amount of time than a gunzip... BUT because it creates a little (<<1%/gzip) index, next extractions will be really quick:

https://github.com/circulosmeos/gztool

The tool has two options that may be of interest for you:

  • -S option supervise a still-growing file and creates an index for it as it is growing - this can be useful for gzipped rsyslog files as reduces to zero in the practice the time of index creation.
  • -t tails a gzip file: this way you can do: $ gztool -t foo.gz Please, note that if the index doesn't exists, this will consume the same time as a complete decompression: but as the index is reusable, next searches will be greatly reduced in time - and as it is the same time, why not use it and create the index at the same time?

This tool is based on zran.c demonstration code from original zlib, so there's no out-of-the-rules magic!

Improve this answer

You must log in to answer this question.

Start asking to get answers

Find the answer to your question by asking.

Ask question

Explore related questions

See similar questions with these tags.