Skip to main content
improved formatting
Source Link
chaos
  • 49.4k
  • 11
  • 128
  • 147

i had multiple files each of few MB in size and i have tried this which works for me sort *.csv | uniq -d will:

sort *.csv | uniq -d 

This will give you duplicate records from your file and then you can redirect output to a single file to get duplicate record and removing -d-d will give you all unique record.

i had multiple files each of few MB in size and i have tried this which works for me sort *.csv | uniq -d will give you duplicate records from your file and then you can redirect output to a single file to get duplicate record and removing -d will give you all unique record.

i had multiple files each of few MB in size and i have tried this which works for me:

sort *.csv | uniq -d 

This will give you duplicate records from your file and then you can redirect output to a single file to get duplicate record and removing -d will give you all unique record.

Source Link

i had multiple files each of few MB in size and i have tried this which works for me sort *.csv | uniq -d will give you duplicate records from your file and then you can redirect output to a single file to get duplicate record and removing -d will give you all unique record.