PERF: DataFrame(ndarray) constructor ensure to copy to column-major layout

[jorisvandenbossche]

Closes #50756

xref #57431

With CoW enabled now, the default behaviour of DataFrame(ndarray) is to copy the numpy array (before 3.0, this would not copy the data). However, if we do a copy, we should also make sure we copy it to the optimal layout (typically column major). Before CoW, this copy would often happen later on anyway, and an explicit copy() actually does ensure column-major layout:

On main:

>>> arr = np.random.randn(10, 3) >>> df = pd.DataFrame(arr) # does actually make a copy of `arr` under the hood >>> df._mgr.blocks[0].values.flags.c_contiguous False >>> df2 = df.copy() # explicit copy method >>> df2._mgr.blocks[0].values.flags.c_contiguous True 

[jorisvandenbossche]

With the change, the result of the arithmetic.FrameWithFrameWide.time_op_different_blocks op=; shape=(1000000, 10)

n_rows, n_cols = 1_000_000, 10 # construct dataframe with 2 blocks arr1 = np.random.randn(n_rows, n_cols // 2).astype("f8") arr2 = np.random.randn(n_rows, n_cols // 2).astype("f4") df = pd.concat([DataFrame(arr1), DataFrame(arr2)], axis=1, ignore_index=True) df._consolidate_inplace() arr1 = np.random.randn(n_rows, max(n_cols // 4, 3)).astype("f8") arr2 = np.random.randn(n_rows, n_cols // 2).astype("i8") arr3 = np.random.randn(n_rows, n_cols // 4).astype("f8") df2 = pd.concat( [DataFrame(arr1), DataFrame(arr2), DataFrame(arr3)], axis=1, ignore_index=True, ) df2._consolidate_inplace() %timeit df > df2 

gives

In [4]: %timeit df > df2 21.3 ms ± 3.97 ms per loop (mean ± std. dev. of 7 runs, 100 loops each) # main 8.18 ms ± 1.52 ms per loop (mean ± std. dev. of 7 runs, 100 loops each) # PR 

[jorisvandenbossche]

We actually discussed before that we should do this for CoW -> #50756, and it was also at some point included in PR #51731 but then removed again before merging.

[jbrockmendel]

Looks reasonable to me. The perf improvement is real nice.

[mroeschke]

Thanks @jorisvandenbossche

[DeaMariaLeon]

Conbench complaining on these benchmarks:

 `io.hdf.HDF.time_write_hdf` (Python) with format='fixed' `stat_ops.FrameOps.time_op` (Python) with axis=None, dtype='int', op='median' `arithmetic.MixedFrameWithSeriesAxis.time_frame_op_with_series_axis0` (Python) with me='pow' `io.hdf.HDFStoreDataFrame.time_write_store_table_wide` (Python) `frame_methods.Where.time_where` (Python) with dtype=True, param2='float64' `stat_ops.FrameOps.time_op` (Python) with axis=None, dtype='float', op='median' `reshape.Unstack.time_without_last_row` (Python) with param1='int' `stat_ops.FrameOps.time_op` (Python) with axis=None, dtype='float', op='kurt' `groupby.GroupManyLabels.time_sum` (Python) with ncols=1000 `frame_methods.Rank.time_rank` (Python) with dtype='uint' `rolling.TableMethod.time_apply` (Python) with method='table' `reshape.Unstack.time_full_product` (Python) with param1='int' `stat_ops.FrameOps.time_op` (Python) with axis=None, dtype='float', op='skew' 

Please ignore if this was expected.

@jorisvandenbossche