2

I had an operation that works thru originallist (dowork.sh originallist), and lets you know what it has finished into cleaned1. cleaned1 is sorted differently than originallist. I need to generate a list of what's left for dowork.sh to process. Essentially: list cleanedR - list cleaned1 = list cleaned2. It's a minus operation. I found that I can do that operation with the following grep options:

  • F for full line match instead of regular expression (we don't want grep freaking out at filename characters thinking they are regular expressions),
  • v for exclude (which is the minus operation),
  • f for look thru the file cleaned1 for the expressions instead of a single given expression ("obtain PATTERN from FILE").
# wc -l cleaned* 9157094 cleaned1 14283591 cleanedR # du -sh cleaned* 1.3G cleaned1 2.0G cleanedR # grep -Fvf cleaned1 originallist > cleaned2

runs for 5 minutes, uses up 42G of ram or less but a lot of it, then exits with failure; cleaned2 is 0 bytes long.

cleaned2 at the end should be 14283591 - 9157094 = 5126497 lines long

This is the correct syntax for doing such an operation (I tested it with a 10 line long cleanedR and a 3 line long cleaned1; the resultant cleaned2 was 7 lines), however it uses up a lot of ram. Is there a way to make this work by making grep not use up so much ram? I know it will take a while, but I am okay with it.

I am looking for something like sort's -T option, which allows you to not use up /tmp (ram in my case), and allow you to use another directory

sort -h -T, --temporary-directory=DIR use DIR for temporaries, not $TMPDIR or /tmp; multiple options specify multiple directories
Improve this question
2
  • 1
    Does the order of stuff in cleaned2 matter? If not, then sort cleaned1 and originallist and use comm. Commented Aug 15, 2017 at 23:54
  • Does adding the grep -x option help? Commented Aug 16, 2017 at 7:13

1 Answer 1

Reset to default
4

The sort command has some specific support for working with huge data sets because that's a relatively common use case. Huge grep patterns is an extremely uncommon use case so you can't hope for the developers to have put a lot of effort into it.

If the order of lines doesn't matter, you can sort the two files, after which they can be compared without storing more than a few lines in memory at a time, no matter how long the files are. Since sort can cope with files that don't fit in memory, this is efficient. 

sort originallist >originallist.sorted sort cleaned1 | comm -23 originallist.sorted - >cleaned2.sorted

If the original order of originallist matters, you can add line numbers to it:

nl -w 22 originallist >originallist.numbered # then generate cleaned1 from the numbered list

Since originallist.numbered is sorted, you can run comm on it to detect common lines.

If the order matters and it's too late to number the lines, then you could try splitting cleaned1 into chunks and making one pass into originallist for each chunk. With a recent GNU split:

cp originalfile cleaned2.in split -l 1000000 --filter='grep -Fxv -f - cleaned2.in >cleaned2.out; mv cleaned2.out cleaned2.in' cleaned1 mv cleaned2.in cleaned2

(Note that F does not do “full line match”, it does a substring match. For a full line string match, you need -x as well.)

Improve this answer

You must log in to answer this question.

Start asking to get answers

Find the answer to your question by asking.

Ask question

Explore related questions

See similar questions with these tags.