PERF/COMPAT: define platform int to np.intp

[chris-b1]

AFAIK this only affects 64 bit python on Windows.

numpy wants an np.intp (i8 on Windows) as a indexer for take, but pandas defines a "platform int" as a np.int_ (i4 on Windows). This hits performance twice, because we often start with i8, cast to i4, then numpy will cast back to i8 in its take.

This is an alternative to #13924 - there I explored replacing ndarray.take with our take_nd fully, but this approach solves the perf problem without much pain or risking new segfaults.

I'd still need to adjust a bunch of tests here to pass on Windows. This is an API change for "advanced" methods like get_indexer, but I don't think anything is necessary beyond a doc note?

ASV:

 before after ratio [4a80521 ] [3ced5d5 ] 3.49ms 2.86ms 0.82 indexing.series_take_dtindex.time_series_take_dtindex 3.28ms 3.03ms 0.92 indexing.series_take_intindex.time_series_take_intindex 15.16ms 14.57ms 0.96 join_merge.join_dataframe_index_single_key_bigger.time_join_dataframe_index_single_key_bigger 19.81ms 16.62ms 0.84 join_merge.join_dataframe_index_single_key_bigger_sort.time_join_dataframe_index_single_key_bigger_sort 14.05ms 13.63ms 0.97 join_merge.join_dataframe_index_single_key_small.time_join_dataframe_index_single_key_small 4.83ms 4.64ms 0.96 join_merge.join_dataframe_integer_2key.time_join_dataframe_integer_2key 1.80ms 1.63ms 0.90 join_merge.join_dataframe_integer_key.time_join_dataframe_integer_key 633.31us 570.37us 0.90 join_merge.join_non_unique_equal.time_join_non_unique_equal 2.39s 1.36s 0.57 join_merge.left_outer_join_index.time_left_outer_join_index 

[codecov-io]

Current coverage is 85.29% (diff: 100%)

Merging #13972 into master will increase coverage by 0.02%

@@ master #13972 diff @@ ========================================== Files 139 139 Lines 50219 50245 +26 Methods 0 0 Messages 0 0 Branches 0 0 ========================================== + Hits 42819 42854 +35  + Misses 7400 7391 -9  Partials 0 0 

Powered by Codecov. Last update 4a80521...322b11a

[jreback]

add the issue ref here.

[jreback]

yes this looks reasonable. go ahead an make ready on windows. Also add an asv as above.

[jreback]

pls update the docs as indicated. otherwise lgtm.

[chris-b1]

What else were you looking for in the docs? I did add an issue ref in the first paragraph.

[jreback]

ok this looks fine. can you run this on 32-bit linux as well and see that it comes up clean (IIRC you tests on windows so that should be good).

[chris-b1]

The changes I just pushed were to get this passing on 32 bit windows. I don't have anything handy to test 32 bit Linux, I can probably set up a VM, although I think it should generally act the same 32 bit Windows (np.int_ == np.intp == np.int32)

[jreback]

ok I'll run on 32 bit tomorrow

I just use a macosx / vm

[jreback]

This currently fails on 32-bit linux (before your PR). I recall had to do something on windows to get this to pass. Something wrong somewhere. Your PR fixes this! yeah!

====================================================================== FAIL: test_alignment_non_pandas (pandas.tests.frame.test_operators.TestDataFrameOperators) ---------------------------------------------------------------------- Traceback (most recent call last): File "/home/jreback/pandas/pandas/tests/frame/test_operators.py", line 1210, in test_alignment_non_pandas Series([1, 2, 3], index=df.index)) File "/home/jreback/pandas/pandas/util/testing.py", line 1154, in assert_series_equal assert_attr_equal('dtype', left, right) File "/home/jreback/pandas/pandas/util/testing.py", line 878, in assert_attr_equal left_attr, right_attr) File "/home/jreback/pandas/pandas/util/testing.py", line 1018, in raise_assert_detail raise AssertionError(msg) AssertionError: Attributes are different Attribute "dtype" are different [left]: int32 [right]: int64 

So as-is this passes on linux32 (and fixes the above).

lmk when good to go.

[chris-b1]

Cool - fixed the lint issue so this should be good to go.

[jreback]

thanks!