PERF: Avoid materializing values in Categorical.set_categories #17515
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters. Learn more about bidirectional Unicode characters
TomAugspurger added Categorical TomAugspurger force-pushed the set_categories-perf branch from d848c95 to 6d72836 Compare 
Contributor Author
| Here's the ASV: |
TomAugspurger force-pushed the set_categories-perf branch 2 times, most recently from 17dd9f9 to f74da7e Compare 
| Hello @TomAugspurger! Thanks for updating the PR. Cheers ! There are no PEP8 issues in this Pull Request. 🍻 Comment last updated on September 14, 2017 at 10:55 Hours UTC |
TomAugspurger force-pushed the set_categories-perf branch from f74da7e to 9b311f4 Compare 
Codecov Report
@@ Coverage Diff @@ ## master #17515 +/- ## ========================================== - Coverage 91.24% 91.2% -0.05% ========================================== Files 163 163 Lines 49582 49586 +4 ========================================== - Hits 45242 45225 -17 - Misses 4340 4361 +21
Continue to review full report at Codecov.
|
jreback reviewed Sep 13, 2017
| Examples | ||
| -------- | ||
| >>> old_cat = pd.Index(['b', 'a', 'c']) |
Contributor
There was a problem hiding this comment.
isn't this just a special case of union_categoricals?
Contributor
There was a problem hiding this comment.
these both remap codes; seems like they've should share
Contributor Author
There was a problem hiding this comment.
I see. There was a little bit I could extract from union_categorical. Simplified things a bit too by using take_1d. See my latest commit.
Contributor Author
| I don't think so. `union_categoricals` takes the union of the two categories. This replaces the old categories with the new. …On Wed, Sep 13, 2017 at 6:24 PM, Jeff Reback ***@***.***> wrote: ***@***.**** commented on this pull request. ------------------------------ In pandas/core/categorical.py <#17515 (comment)>: > +def _recode_for_categories(codes, old_categories, new_categories): + """ + Convert a set of codes for to a new set of categories + + Parameters + ---------- + codes : array + old_categories, new_categories : Index + + Returns + ------- + new_codes : array + + Examples + -------- + >>> old_cat = pd.Index(['b', 'a', 'c']) isn't this just a special case of union_categoricals? — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#17515 (review)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/ABQHIr3MDXGGPEIv_2U2nqZ4qxzcvvvRks5siGQ9gaJpZM4PWjZF> . |
Mater: ```python In [1]: import pandas as pd; import numpy as np In [2]: arr = ['s%04d' % i for i in np.random.randint(0, 500000 // 10, size=500000)]; s = pd.Series(arr).astype('category') In [3]: %timeit s.cat.set_categories(s.cat.categories) 68.5 ms ± 846 µs per loop (mean ± std. dev. of 7 runs, 10 loops each) ``` HEAD: ```python In [1]: import pandas as pd; import numpy as np In [2]: arr = ['s%04d' % i for i in np.random.randint(0, 500000 // 10, size=500000)] s = pd.Series(arr).astype('category') In [3]: %timeit s.cat.set_categories(s.cat.categories) 7.43 ms ± 110 µs per loop (mean ± std. dev. of 7 runs, 100 loops each) ``` Closes pandas-dev#17508 TomAugspurger force-pushed the set_categories-perf branch from 9b311f4 to d238e3e Compare 
Contributor
| thanks @TomAugspurger |
alanbato pushed a commit to alanbato/pandas that referenced this pull request Nov 10, 2017
Mater: ```python In [1]: import pandas as pd; import numpy as np In [2]: arr = ['s%04d' % i for i in np.random.randint(0, 500000 // 10, size=500000)]; s = pd.Series(arr).astype('category') In [3]: %timeit s.cat.set_categories(s.cat.categories) 68.5 ms ± 846 µs per loop (mean ± std. dev. of 7 runs, 10 loops each) ``` HEAD: ```python In [1]: import pandas as pd; import numpy as np In [2]: arr = ['s%04d' % i for i in np.random.randint(0, 500000 // 10, size=500000)] s = pd.Series(arr).astype('category') In [3]: %timeit s.cat.set_categories(s.cat.categories) 7.43 ms ± 110 µs per loop (mean ± std. dev. of 7 runs, 100 loops each) ``` Closes pandas-dev#17508 No-Stream pushed a commit to No-Stream/pandas that referenced this pull request Nov 28, 2017
Mater: ```python In [1]: import pandas as pd; import numpy as np In [2]: arr = ['s%04d' % i for i in np.random.randint(0, 500000 // 10, size=500000)]; s = pd.Series(arr).astype('category') In [3]: %timeit s.cat.set_categories(s.cat.categories) 68.5 ms ± 846 µs per loop (mean ± std. dev. of 7 runs, 10 loops each) ``` HEAD: ```python In [1]: import pandas as pd; import numpy as np In [2]: arr = ['s%04d' % i for i in np.random.randint(0, 500000 // 10, size=500000)] s = pd.Series(arr).astype('category') In [3]: %timeit s.cat.set_categories(s.cat.categories) 7.43 ms ± 110 µs per loop (mean ± std. dev. of 7 runs, 100 loops each) ``` Closes pandas-dev#17508 Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit. This suggestion is invalid because no changes were made to the code. Suggestions cannot be applied while the pull request is closed. Suggestions cannot be applied while viewing a subset of changes. Only one suggestion per line can be applied in a batch. Add this suggestion to a batch that can be applied as a single commit. Applying suggestions on deleted lines is not supported. You must change the existing code in this line in order to create a valid suggestion. Outdated suggestions cannot be applied. This suggestion has been applied or marked resolved. Suggestions cannot be applied from pending reviews. Suggestions cannot be applied on multi-line comments. Suggestions cannot be applied while the pull request is queued to merge. Suggestion cannot be applied right now. Please check back later.
Mater:
HEAD:
Closes #17508
I'll rebase #16015 on top of this. Running an ASV now.