Skip to main content
Unix & Linux

You are not logged in. Your edit will be placed in a queue until it is peer reviewed.

We welcome edits that make the post easier to understand and more valuable for readers. Because community members review edits, please try to make the post substantially better than how you found it, for example, by fixing grammar or adding additional resources and hyperlinks.

Required fields*

9
  • Have you considered sorting your file contents, and then running uniq? I've no idea if that would be faster or not, but an idea nonetheless. Commented Sep 11, 2012 at 10:41
  • @ire_and_curses: Indeed, this is a new idea. But how to approach that with multiple files? Joining them seems to be a bad idea. I don't see a "multiple file" option in uniq (but I only had a quick lock at the man page) Commented Sep 11, 2012 at 10:45
  • When you deal with large files, you must not use AWK arrays. Running 'cat foo/file bar/file | sort | uniq' looks like a good solution (if that is what you want to achive). Why do you think its a bad idea? Commented Sep 11, 2012 at 11:14
  • 3
    @EranBen-Natan : There is no need (and its a performance drain) to do cat foo bar|sort|uniq when you can do sort -u foo bar . Commented Sep 11, 2012 at 12:08
  • 1
    Just a shot in the dark, but would you be better off loading them into a database and using an SQL query to process this? Commented Sep 11, 2012 at 13:54