Does Perl work for you? It can keep the lines in the original order, even if the duplicates are not adjacent. You could also code it in Python, or awk.
while (<>) { print if $lines{$_}++ == 0; } Which can be shortened to just
perl -ne 'print unless $lines{$_}++;' Given input file:
abc def abc ghi abc def abc ghi jkl It yields the output:
abc def ghi jkl 