Remove duplicates from list, including original matching item

Question

I tried searching and couldn't find this exact situation, so apologies if it exists already.

I'm trying to remove duplicates from a list as well as the original item I'm searching for. If I have this:

ls = [1, 2, 3, 3]

I want to end up with this:

ls = [1, 2]

I know that using set will remove duplicates like this:

print set(ls) # set([1, 2, 3])

But it still retains that 3 element which I want removed. I'm wondering if there's a way to remove the duplicates and original matching items too.

score 14 · Accepted Answer · 2014-01-03 16:44:38Z

Use a list comprehension and list.count:

>>> ls = [1, 2, 3, 3] >>> [x for x in ls if ls.count(x) == 1] [1, 2] >>>

Here is a reference on both of those.

Edit:

@Anonymous made a good point below. The above solution is perfect for small lists but may become slow with larger ones.

For large lists, you can do this instead:

>>> from collections import Counter >>> ls = [1, 2, 3, 3] >>> c = Counter(ls) >>> [x for x in ls if c[x] == 1] [1, 2] >>>

Here is a reference on collections.Counter.

Thank you, this is great! I need to use list comprehension more.
This is O(N^2), which is fine for small lists, but will be slow for larger ones. You can fix it by counting occurrences first: c = collections.Counter(ls); uniqs = [x for x in ls if c[x] == 1]

Jon Clements · Accepted Answer · 2014-01-03 18:28:50Z

If items are contigious, then you can use groupby which saves building an auxillary data structure in memory...:

from itertools import groupby, islice data = [1, 2, 3, 3] # could also use `sorted(data)` if need be... new = [k for k, g in groupby(data) if len(list(islice(g, 2))) == 1] # [1, 2]

Collectives™ on Stack Overflow

Remove duplicates from list, including original matching item

2 Answers 2

2 Comments

Comments

Hot Network Questions

Collectives™ on Stack Overflow

2 Answers 2

2 Comments

Comments

Related