PERF: Improved performance for .str.encode/decode

[Winand]

I need such a patch to read huge sas tables encoded in cp1251. I'm not experienced enough to determine if such a patch is really needed here, but well.. it gives some speed in certain situations.

Optimize string encoding-decoding, leave default implementation for CPython optimized encodings,
(see https://docs.python.org/3.4/library/codecs.html#standard-encodings)

string

import pandas as pd s1 = pd.Series(pd.util.testing.makeStringIndex(k=100000)).astype('category') encs = 'utf-8', 'utf-16', 'utf-32', 'latin1', 'iso-8859-1', 'mbcs', 'ascii', 'cp1251', 'cp1252' for enc in encs: s2 = s1.str.encode(enc).astype('category') print(enc) %timeit s1.str.encode(enc) %timeit s2.str.decode(enc) 

unicode

import pandas as pd s1 = pd.Series(pd.util.testing.makeUnicodeIndex(k=100000)).astype('category') encs = 'utf-8', 'utf-16', 'utf-32' for enc in encs: s2 = s1.str.encode(enc).astype('category') print(enc) %timeit s1.str.encode(enc) %timeit s2.str.decode(enc) 

("10 loops, best of 3: xxx ms per loop")

[jreback]

define these at the top of the file:

_cpython_optimized_encoding = .....

[jreback]

@Winand in your example you miss the point of categorical encoding:

In [2]: pd.util.testing.makeUnicodeIndex(k=100000).nunique() Out[2]: 100000 

categorical in general make sense if k << n. slight overhead when k == n.

have you tried read_sas?

[Winand]

import pandas as pd s1 = pd.Series(pd.util.testing.makeStringIndex(k=1000)) s1 = pd.concat([s1]*100).astype('category') encs = 'utf-8', 'utf-16', 'utf-32', 'latin1', 'iso-8859-1', 'mbcs', 'ascii', 'cp1251', 'cp1252' for enc in encs: s2 = s1.str.encode(enc).astype('category') print(enc) %timeit s1.str.encode(enc) %timeit s2.str.decode(enc) 

@jreback i tried read_sas from 0.18.0, it was more than twice slower than (slightly improved) Jared Hobbs' sas7bdat module. A lot of time is spent on decoding cp1251

[jreback]

try with master. its now 4x faster.

[Winand]

@jreback i've tried.) 2x faster on my test table, which is still amazing improvement

[jreback]

cc @kshedden

ok, seems reasonable.

[jreback]

thanks!