1

I'm very badly stuck, and every pythonista I've asked can't seem to help.

I'm using vstack to create an array of vectors in a loop like this:

Corr = np.vstack((Corr, S))

I need to remove repeating vectors so that it is an array of unique vectors and to compare all of these vectors.

I know that this comparison can be done in lists, but I have not found a way to append full vectors to a list.

This is the result (I've marked unique vectors with unique letters):

Corr = [[ 0. 0. 0. 0. -2. 4. 4. 2. 2.] #a [-4. -4. -4. -4. 2. 4. 4. 2. 2.]#b [-4. 0. 0. 4. -2. 0. 0. -2. 2.]#c [ 0. -4. -4. 0. 2. 0. 0. -2. 2.]#d [ 0. -4. 4. 0. -2. 0. 0. 2. -2.]#e [-4. 0. 0. -4. 2. 0. 0. 2. -2.]#f [-4. -4. 4. 4. -2. 4. -4. -2. -2.]#g [ 0. 0. 0. 0. 2. 4. -4. -2. -2.]#h [ 0. 4. -4. 0. -2. 0. 0. 2. -2.]#i [-4. 0. 0. -4. 2. 0. 0. 2. -2.]#f [-4. 4. -4. 4. -2. -4. 4. -2. -2.]#j [ 0. 0. 0. 0. 2. -4. 4. -2. -2.]#k [ 0. 0. 0. 0. -2. -4. -4. 2. 2.]#l [-4. 4. 4. -4. 2. -4. -4. 2. 2.]#m [-4. 0. 0. 4. -2. 0. 0. -2. 2.]#n [ 0. 4. 4. 0. 2. 0. 0. -2. 2.]#o [ 4. 0. 0. -4. -2. 0. 0. -2. 2.]#c [ 0. -4. -4. 0. 2. 0. 0. -2. 2.]#d [ 0. 0. 0. 0. -2. -4. -4. 2. 2.]#p [ 4. -4. -4. 4. 2. -4. -4. 2. 2.]#q [ 4. -4. 4. -4. -2. -4. 4. -2. -2.]#r [ 0. 0. 0. 0. 2. -4. 4. -2. -2.]#k [ 0. -4. 4. 0. -2. 0. 0. 2. -2.]#e [ 4. 0. 0. 4. 2. 0. 0. 2. -2.]#s [ 4. 4. -4. -4. -2. 4. -4. -2. -2.]#t [ 0. 0. 0. 0. 2. 4. -4. -2. -2.]#h [ 0. 4. -4. 0. -2. 0. 0. 2. -2.]#i [ 4. 0. 0. 4. 2. 0. 0. 2. -2.]#s [ 4. 0. 0. -4. -2. 0. 0. -2. 2.]#u [ 0. 4. 4. 0. 2. 0. 0. -2. 2.]#o [ 0. 0. 0. 0. -2. 4. 4. 2. 2.]]#a

I don't know why vstack is adding a period instead of a comma (in the loops each vector S has a comma when I print it separately!).

I need the end result to be an array of unique vectors, (so in this case it'll be vectors a-u ie, 21 vectors).

Improve this question
3
  • 1
    As a side note, calling vstack in a loop to build up a matrix one-by-one is poor practice. Instead, create a regular old list of all your vectors and then combine them all at once. This saves tons of copying if the number of vectors is large. Commented Nov 14, 2015 at 14:13
  • John, the "Find unique rows" link you mentioned also has a number of vstack solutions. And the vector elements are separated by comments. I'm using vstack and there are periods showing up. Not being able to use any of those methods to sort through the array. Commented Nov 14, 2015 at 14:47
  • The "periods showing up" are not a problem, they're normal. They show you that your numbers are floats not ints. In a Python list it prints by default with commas between elements. The dots-vs-commas has no impact on what sort of data is inside, it's just a display thing. Commented Nov 14, 2015 at 14:58

3 Answers 3

Reset to default
3

If you convert your vectors to tuples, you can put them in a set which will automatically discard duplicates. For example:

unique_vectors = set(map(tuple, Corr)) array_of_unique_vectors = np.array(list(unique_vectors))

Edit: I was curious, so I quickly benchmarked the three proposed solutions here. The results are the same up to the order of the returned elements, and it appears that the Pandas drop_duplicates method outperforms the others.

import numpy as np import pandas as pd def unique_set(a): return np.vstack(set(map(tuple, a))) def unique_numpy(a): a = np.ascontiguousarray(a) view = a.view(np.dtype(('void', a.itemsize * a.shape[1]))) unique = np.unique(view) return unique.view(a.dtype).reshape(-1, a.shape[1]) def unique_pandas(a): return pd.DataFrame(a).drop_duplicates().values a = np.random.randint(0, 5, (100000, 5)) %timeit unique_set(a) 10 loops, best of 3: 183 ms per loop %timeit unique_numpy(a) 10 loops, best of 3: 43.1 ms per loop %timeit unique_pandas(a) 100 loops, best of 3: 10.3 ms per loop
Improve this answer
Sign up to request clarification or add additional context in comments.

8 Comments

This only works if you the data is not very large and you don't care about optimal performance.
Actually, I think it's pretty close to optimal. What you want is a hash table to do comparisons of new vectors in O[1], and set() is the easiest way to build a hash table to drop conflicts. In Python 3, map() is a generator, so you're not going to face huge memory issues either. In Python 2 itertools.imap would be better.
Are you a regular user of NumPy? Are you familiar with how it works and why people use it?
Yes and yes. If there is a vectorized way to compute a hash table within numpy, I'm not aware of it.
My data will be blowing up as I go to higher and higher numbers. Is there another optimal solution for this?
|
2

Here's an answer that avoids data duplication and doesn't require external packages like Pandas:

Corr = np.ascontiguousarray(Corr) view = Corr.view(np.dtype(('void', Corr.itemsize * Corr.shape[1]))) unique_view = np.unique(view) unique = unique_view.view(Corr.dtype).reshape(-1, Corr.shape[1])

I find it to be about 5 times faster than the set-of-tuple solution I previously proposed.

Improve this answer

Comments

0

Pandas has a fairly direct solution to your problem--the drop_duplicates function:

http://pandas.pydata.org/pandas-docs/version/0.17.0/generated/pandas.DataFrame.drop_duplicates.html

Improve this answer

2 Comments

I'm not able to use pandas at this time. Any numpy alternative?
See my numpy-based solution.

Start asking to get answers

Find the answer to your question by asking.

Ask question

Explore related questions

See similar questions with these tags.