Does Perl work for you? It can keep the lines in the original order, even if the duplicates are not adjacent. You could also code it in Python, or awk.

while (<>)  { print if $lines{$_}++ = 0; } 

Given input file:

abc def abc ghi abc def abc ghi jkl 

It yields the output:

abc def ghi jkl 

Does Perl work for you? It can keep the lines in the original order, even if the duplicates are not adjacent. You could also code it in Python, or awk.

while (<>) { print if $lines{$_}++ == 0; } 

Which can be shortened to just

perl -ne 'print unless $lines{$_}++;' 

Given input file:

abc def abc ghi abc def abc ghi jkl 

It yields the output:

abc def ghi jkl 

Does Perl work for you? It can keep the lines in the original order, even if the duplicates are not adjacent. You could also code it in Python.

use strict; use warnings; my %lines; while (<>) {   chomp; $lines{$_} = $. unless defined $lines{$_}; } my @lines; # Evil! foreach my $line (keys %lines) # Worse if using $lines (but Perl would be OK) { $lines[$lines{$line}] = $line; # Really evil! } foreach my $line (@lines) { print "$line\n" if defined $line; } 

Given input file:

abc def abc ghi abc def abc ghi jkl 

It yields the output:

abc def ghi jkl 

A non-evil version of the code could be:

use strict; use warnings; my %lines; while (<>) { chomp; $lines{$_} = $. unless defined $lines{$_}; } my @data; foreach my $line (keys %lines) { $data[$lines{$line}] = $line; } foreach my $line (@data) { print "$line\n" if defined $line; } 

Does Perl work for you? It can keep the lines in the original order, even if the duplicates are not adjacent. You could also code it in Python, or awk.

while (<>) { print if $lines{$_}++ = 0; } 

Given input file:

abc def abc ghi abc def abc ghi jkl 

It yields the output:

abc def ghi jkl