I need to unzip a log file, and then display some specific columns of that log file using awk. Can it be done using pipe?
unzip log.zip | awk -F "- -" '{pring $1 " " $2} >newfile 
2 Answers
You need to use unzip's -c option to uncompress the .zip file's contents to stdout....otherwise it will uncompress the file(s) in the .zip file onto the disk, and there will be nothing to pipe into awk.
Like many programs that originated from MS-DOS/Windows instead of Unix, unzip also waffles on verbosely about what it's doing. Unix programs generally don't do that until you explicitly tell them to with a -v or --verbose or --debug option or similar. It's a significant cultural difference between dos/windows and unix software. and programmers.
From the POV of any program taking input on stdin, that is almost certainly garbage that is going to confuse it (this is why unix programs aren't verbose unless you tell them to be). So tell unzip to "just shut up and get to work" with the -q (quiet) or -qq (extra quiet) option.
e.g.
unzip -qq -c log.zip | awk -F'- -' '{print $1, $2}' >newfile BTW, I've changed your awk script slightly. I fixed the pring syntax error. Also, the default output field separator (OFS) in awk is a single space...even if you set the input field separator (FS) to something else (e.g. with -F'- -'). So I just printed $1,$2 - no need to use a double-quoted space character.
- Try
-pinstead of-c. From unzip's manual page: "This option (-c) is similar to the-poption except that the name of each file is printed as it is extracted". So if you're using-cmake sure to consider the filename in the output.Andres– Andres2023-08-10 15:42:24 +00:00Commented Aug 10, 2023 at 15:42
You can do this:
awk -F "--" '{print $1 " "$2}' <(gzip -dc log.zip) The gzip command uses the -d and -c options.
-c --stdout --to-stdout Write output on standard output -d --decompress --uncompress Decompress. You can of course use standard redirection to send the output to a file
awk -F "--" '{print $1 " "$2}' <(gzip -dc log.zip) > my_out_file As Cas pointed out in the comments, gzip works for most .zip files but isn't guaranteed to work with all of them, you can use unzip with this command.
awk -F "--" '{print $1 " "$2}' <(unzip -cq log.zip) > my_out_file The options are slightly different but the result is the same.
-
gzipisn'tzip. Info-zip'sunzipdoes have a-coption, though, to uncompress the contents of the .zip file to stdout.cas– cas2018-02-20 08:12:17 +00:00Commented Feb 20, 2018 at 8:12 - gzip works for the
.zipfile here, i just tried the-con theunzip, this also works, although it is slightly messy with the output you get.alpha– alpha2018-02-20 08:17:02 +00:00Commented Feb 20, 2018 at 8:17 - 1gzip works for most .zip files but isn't guaranteed to work with all of them. It really depends on the exact compression method used for the file(s) inside the .zip. BTW, try the
-qor-qqoptions withunzipto shut it up. Or redirect stder from unzip to /dev/nullcas– cas2018-02-20 08:20:12 +00:00Commented Feb 20, 2018 at 8:20 - ah ha the -q did indeed shut it up. Thank you.alpha– alpha2018-02-20 08:23:55 +00:00Commented Feb 20, 2018 at 8:23
- it's worse than i thought. unzip prints stuff to stdout, not just stderr unless you shut it up with
-q.cas– cas2018-02-20 08:38:03 +00:00Commented Feb 20, 2018 at 8:38
log.zipcontain only one log file, or does it contain several? if only one, then use gzip (or bzip2 or xz) to compress the file. BTW, unix practice has always been to separate compression and archiving (e.g. compress or gzip or whatever for compression, tar or cpio or whatever for archiving). ms-dos (and thus windows) practice is to do both compression and archiving at the same time). unix practice is more convenient for working with compressed data in a pipeline.