PRF: Optimize 2d take operations. #6759
Closed
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters. Learn more about bidirectional Unicode characters
Contributor Author
|
Contributor
| yep...I tried to do the same but on the cython level (and using memoryviews...but was complicated)... this could be a simpler soln. pls add a couple of vbenches that exercise this if you can by doing a take then other operations. e.g. reindex, then do stuff with that (on both axes). |
Contributor Author
| Is there an optimal way to run Maybe set n to like 100 :/ |
Contributor
|
|
Contributor Author
| @jreback are we able to use memoryviews? Not sure what version of cython they were introduced. |
Contributor
| yes |
Contributor
| FYI pandas already requires cython 0.17.1 or higher. |
Contributor
| can you post another vbench run after your latest changes |
Contributor
Contributor Author
reran the bottom |
Contributor
| ok looks fine |
Contributor
| @dalejung this was ok...pls put in a release note and rebase...good to 2 go otherwise |
Contributor
| @dalejung just needed a release note on this |
Contributor
| merged via 36357a0 (I added a release note as well) thanks! |
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit. This suggestion is invalid because no changes were made to the code. Suggestions cannot be applied while the pull request is closed. Suggestions cannot be applied while viewing a subset of changes. Only one suggestion per line can be applied in a batch. Add this suggestion to a batch that can be applied as a single commit. Applying suggestions on deleted lines is not supported. You must change the existing code in this line in order to create a valid suggestion. Outdated suggestions cannot be applied. This suggestion has been applied or marked resolved. Suggestions cannot be applied from pending reviews. Suggestions cannot be applied on multi-line comments. Suggestions cannot be applied while the pull request is queued to merge. Suggestion cannot be applied right now. Please check back later.
Wanted to get eyes on this. The original axis=1 2d methods were optimized for Fortran. I changed it to be friendly to C order and transposed the array when F ordered.
These timings are for master.
cdfis just the consolidated dataframe to switch from C to F.http://nbviewer.ipython.org/gist/dalejung/9914956/pandas%20perf_baseline.ipynb
The PR timings are:
http://nbviewer.ipython.org/gist/dalejung/9914956/pandas%20perf.ipynb
It's similar behavior to the shift perf update.
For a numpy baseline i wrote:
Which are similar timings.