(WIP) PERF: improve .str perf for all-string values (about 2x-)

[sinhrks]

Related to #10081. Make a short path using numpy's string funcs when all the target values are strings. Otherwise, use current path.

Followings are current comparison results:

import pandas as pd import numpy as np import string import random np.random.seed(1) s = [''.join([random.choice(string.ascii_letters + string.digits) for i in range(3)]) for i in range(1000000)] # s_str uses short path s_str = pd.Series(s) # set object s[-1] = 1 # s_obj uses current path s_obj = pd.Series(s) 

%timeit s_str.str.lower() #1 loops, best of 3: 696 ms per loop %timeit s_obj.str.lower() #1 loops, best of 3: 1.46 s per loop %timeit s_str.str.split('a') #1 loops, best of 3: 1.55 s per loop %timeit s_obj.str.split('a') #1 loops, best of 3: 3.52 s per loop 

The logic has an overhead to check whether target values are all-string using lib.is_string_array. But this should be speed-up in most cases because it takes relatively shorter time than string ops, and (I believe) values should be all-string in most cases.

%timeit pd.lib.is_string_array(s_str.values) #10 loops, best of 3: 21.9 ms per loop 

If it looks OK, I'll work on all the funcs which is supported by numpy.

• Add all numpy funcs
  • http://docs.scipy.org/doc/numpy/reference/routines.char.html
• Add tests to check numpy results for all-string values are identical as current results.
• Add vbench (both for all-string and object-included)

[sinhrks]

Ah noticed above comparison is not fair, preparing valid ones...

[sinhrks]

I misunderstood the differnce of str and object is caused by numpy logic. As numpy looks to use similar logic as pandas thus cannot expect such a performance gain... Allow me to close this.

[jorisvandenbossche]

Where did the initial speed-up come from then?

[sinhrks]

The above difference exists on current master depending on dtypes handled. I've misunderstand it is caused be the logic i've changed, but not.