Let's say, I have a really big text file (about 10.000.000 lines). I need to grep it from the end and save result to a file. What's the most efficient way to accomplish task?
4 Answers
tac/grep Solution
tac file | grep whatever Or a bit more effective:
grep whatever < <(tac file) Time with a 500MB file:
real 0m1.225s user 0m1.164s sys 0m0.516s sed/grep Solution:
sed '1!G;h;$!d' | grep whatever Time with a 500MB file: Aborted after 10+ minutes.
awk/grep Solution:
awk '{x[NR]=$0}END{while (NR) print x[NR--]}' file | grep whatever Time with a 500MB file:
real 0m5.626s user 0m4.964s sys 0m1.420s perl/grep Solution:
perl -e 'print reverse <>' file | grep whatever Time with a 500MB file:
real 0m3.551s user 0m3.104s sys 0m1.036s - @chaos, I think
grep "somepattern" < <(tac filename)will be faster.Valentin Bajrami– Valentin Bajrami2014-07-23 12:43:53 +00:00Commented Jul 23, 2014 at 12:43 - 2@val0x00ff The
< <(tac filename)should be as fast as a pipe: in both cases, the commands run in parallel.vinc17– vinc172014-07-23 12:46:12 +00:00Commented Jul 23, 2014 at 12:46 - 7If you're going for efficiency, it would be better to put the
tacafter the grep. If you've got a 10,000,000 line file, with only 2 matches,tacwill only have to reverse 2 lines, not 10m.grepis still going to have to go through the whole thing either way.phemmer– phemmer2014-07-23 14:10:49 +00:00Commented Jul 23, 2014 at 14:10 - 3If you put
tacafter thegrep, it will be reading from a pipe and so can't seek. That will make it less efficient (or fail completely) if the number of found lines is large.jjanes– jjanes2014-07-23 19:45:30 +00:00Commented Jul 23, 2014 at 19:45 - 1@Bernhard If you tac a real file, it
lseeks backwards through the file to read it backwards in chunks, and then reverses the lines in each chunk, remembering the line broken across chunks to put them back together. If reading from a pipe, it can't do that. It either needs to read the whole thing into memory, or write it to a temp file, or fail.jjanes– jjanes2014-07-24 15:52:16 +00:00Commented Jul 24, 2014 at 15:52
This solution might help:
tac file_name | grep -e expression - 3
tacis the GNU command. On most other systems, the equivalent istail -r.Stéphane Chazelas– Stéphane Chazelas2014-07-23 14:55:47 +00:00Commented Jul 23, 2014 at 14:55 - @Stéphane: On at least some Unix systems,
tail -ris limited to a small number of lines, this might be an issue.RedGrittyBrick– RedGrittyBrick2014-07-23 16:20:29 +00:00Commented Jul 23, 2014 at 16:20 - 1@RedGrittyBrick, do you have any reference for that, or could you please tell which systems have that limitation?Stéphane Chazelas– Stéphane Chazelas2014-07-23 16:50:21 +00:00Commented Jul 23, 2014 at 16:50
- @StéphaneChazelas,
tail -r /etc/passwdfails withtail: invalid option -- 'r'. I'm using coreutils-8.21-21.fc20.x86_64.Cristian Ciupitu– Cristian Ciupitu2014-07-23 20:14:51 +00:00Commented Jul 23, 2014 at 20:14 - @CristianCiupitu, as I said, GNU has
tac(and only GNU has tac) many other Unices havetail -r. GNUtaildoesn't support-rStéphane Chazelas– Stéphane Chazelas2014-07-23 22:41:00 +00:00Commented Jul 23, 2014 at 22:41
This one exits as soon as it finds the first match:
tac hugeproduction.log | grep -m1 WhatImLookingFor The following gives the 5 lines before and after the first two matches:
tac hugeproduction.log | grep -m2 -A 5 -B 5 WhatImLookingFor Remember not to use -i (case insensitive) unless you have to as that will slow down the grep.
If you know the exact string you are looking for then consider fgrep (Fixed String)
tac hugeproduction.log | grep -F -m2 -A 5 -B 5 'ABC1234XYZ' If the file is really big, can not fit in memory, I will use Perl with File::ReadBackwards module from CPAN:
$ cat reverse-grep.pl #!/usr/bin/perl use strict; use warnings; use File::ReadBackwards; my $pattern = shift; my $rev = File::ReadBackwards->new(shift) or die "$!"; while (defined($_ = $rev->readline)) { print if /$pattern/; } $rev->close; Then:
$ ./reverse-grep.pl pattern file - The advantage of this approach is that you can tweak the Perl to do anything you want.zzapper– zzapper2014-07-24 15:52:07 +00:00Commented Jul 24, 2014 at 15:52
- 1@zzapper: It's memory efficient, too, since when it read file line by line instead of slurp file in memory like
tac.cuonglm– cuonglm2014-07-24 15:54:15 +00:00Commented Jul 24, 2014 at 15:54 - can anyone add a -m support for this ? I'd like to test in on real files. See : gist.githubusercontent.com/ychaouche/…ychaouche– ychaouche2018-11-05 14:29:21 +00:00Commented Nov 5, 2018 at 14:29
grephas a--max-count (number)switch that aborts after a certain number of matches, which might be interesting to you.