i had multiple files each of few MB in size and i have tried this which works for me sort *.csv | uniq -d will:
sort *.csv | uniq -d This will give you duplicate records from your file and then you can redirect output to a single file to get duplicate record and removing -d-d will give you all unique record.