3

I am sending compressed file with piping either local or from a network location. And on the receiving end, I would like to detect the type of compression and use the appropriate decompression utility (gzip, bzip2, xz..etc) to extract it. Commands looks as follows:

Local:

cat misteryCompressedFile | [compressionUtility] -d -fc > /opt/files/uncompressedfile

Over network:

ssh user@ipaddr "cat misteryCompressedFile" | [compressionUtility] -d -fc > /opt/files/uncompressedfile

One can tell the type of compression used even if there is no extension provided (e.g., .gz or .bz2) by looking at first few hex values of the file. For example, if I use xxd to look at first few hex values of two compressed files, then I will 1f8b 0808 for gzip and 425a 6836 for bzip2.

However, to still use piping, how can I check the first incoming byte to select the proper decompression utility for the first of the file?

So if unknown compressed file is a gzip type, command will be this:

cat misteryCompressedFile | gzip -d -fc > /opt/files/uncompressedfile

and if unknown compressed file is bzip2 type, command will be this:

cat misteryCompressedFile | bzip2 -d -fc > /opt/files/uncompressedfile

Is it possible to make such decision with piping on the fly without having to download entire file and then make decision what to use for decompression?

Improve this question

2 Answers 2

Reset to default
4

Yes, you can do that in the pipeline, without having to read the whole file.

This first script fragment illustrates the mechanism by which we will intercept and inspect the header and pass it on. Notice that we print the header to stderr (>&2), yet it continues to appear in the output:

$ echo 0123456789ABCDEF | ( HEADER=$(dd bs=1 count=4); printf 'HEADER:%s\n' "$HEADER" >&2; printf '%s\n' "$HEADER"; cat ) 4+0 records in 4+0 records out 4 bytes (4 B) copied, 8.4293e-05 s, 47.5 kB/s HEADER:0123 0123456789ABCDEF $

The key is using the dd the file conversion utility with a small block size bs=1.

Expanding on that, this is a working solution. We'll use a temporary file to store the binary header. If it doesn't see one of the two 4-byte headers then it does nothing:

#!/bin/sh trap "rm -f /tmp/$$; exit 1" 1 2 3 15 # grab the 1st 4 bytes off the input stream, # store them in a file, convert to ascii, # and store in variable: HEADER=$( dd bs=1 count=4 2>/dev/null | tee /tmp/$$ | od -t x1 | sed ' s/^00* // s/ //g q ' ) case "$HEADER" in 1f8b0800) UNCOMPRESS='gzip -d -fc' ;; 425a6839) UNCOMPRESS='bzip2 -d -fc' ;; *) echo >&2 "$0: unknown stream type for header '$HEADER'" exit 2 ;; esac echo >&2 "$0: File header is '$HEADER' using '$UNCOMPRESS' on stream." cat /tmp/$$ - | $UNCOMPRESS rm /tmp/$$
Improve this answer
11
  • nice. still have to somehow use that info to decide which decompression tool to use. Commented Nov 3, 2015 at 3:19
  • OK. Let me go make some compressed files and put my binary hat on! :-) Commented Nov 3, 2015 at 3:26
  • Note that this will mangle the header if it contains null bytes, because shell variables can't contain null bytes (except in zsh). Commented Nov 3, 2015 at 21:12
  • @Gilles No, because the file header passed to the selected uncompress command is stored in a temporary file. The shell variable is used only to select the uncompress command. Commented Nov 3, 2015 at 21:15
  • Indeed your second script is fine, I was refering to the first one, sorry I wasn't precise enough. Commented Nov 3, 2015 at 21:17
1

Use file on the sending machine and use that info to decide which decompression command to run on the remote host.

e.g. 

#! /bin/sh filetype=$(file misteryCompressedFile) case "$filetype" in *gzip*) CMD='gzip' ; ARGS='-d -fc' ;; *bzip2*) CMD='bzip2' ; ARGS='-d -fc' ;; *) echo "error: unknown compression type" ; exit 1 ;; esac cat misteryCompressedFile | ssh user@ipaddr "$CMD $ARGS > /opt/files/uncompressedfile"

In the example shown, the ARGS for gzip and bzip2 commands are the same...but they may be different for other decompression tools.

Here's a version that will decompress a file fetched from the remote host:

#! /bin/sh # set up an anonymous fifo on fd 3 so we can pass the # output of `file` to the second subshell without risking # corruption of stdout/stdin FIFO=$(mktemp -u) mkfifo "$FIFO" exec 3<>"$FIFO" rm -f "$FIFO" ssh user@ipaddr 'cat misteryCompressedFile' | ( HEADER=$(dd bs=1 count=20 2> /dev/null | od -A none -t o1 -w512 | sed -e 's: :\\:g') printf "$HEADER" | file --mimetype - | cut -d/ -f2 >&3 printf "$HEADER" cat ) | ( read -u 3 -r filetype case "$filetype" in gzip) CMD='gzip' ; ARGS='-d -fc' ;; x-bzip2) CMD='bzip2' ; ARGS='-d -fc' ;; x-xz) CMD='unxz' ; ARGS='' ;; x-lzma) CMD='lzcat' ; ARGS='' ;; x-compress) CMD='uncompress' ; ARGS='' ;; x-lrzip) CMD='lrzcat' ; ARGS='' ;; *) echo "error: unknown compression type" >&2 ; exit 1 ;; esac $CMD $ARGS > /opt/files/uncompressedfile )
Improve this answer
2
  • Thanks. But I am making request on the client (to where file is downloaded). On remote host, there is only cat utility or other utility/script for sends file into stdout. So in away client has no idea what file is piped to it.... Commented Nov 3, 2015 at 3:27
  • @Armen, I've added another version of the script that does what you want on the local machine. it makes use of RobertL's idea of using dd in a subshell to extract the files header. Commented Nov 3, 2015 at 4:09

You must log in to answer this question.

Start asking to get answers

Find the answer to your question by asking.

Ask question

Explore related questions

See similar questions with these tags.