PERF: DataFrame.transpose with dt64tz

[jbrockmendel]

Does what it says on the tin: DatetimeBlock.values is always DatetimeArray, and dt64tzblock.shape == dt64tzblock.values in all cases. Similarly TimedeltaBlock.values is always TimedeltaArray.

Notes:

• It is straightforward to extend this to work for PeriodDtype (i have a branch). Haven't tried it, but I expect it would be similarly easy to do the same for CategoricalDtype.

Things that im not yet fully happy with:

• fillna method on 2D (I think @simonjayhawkins commented on this in another branch recently),
• nargminmax with 2D and mask.any()
• pytables kludge

ASVs: run repeatedly (vs master from yesterday) with --record-samples --append-samples so im pretty confident these are stable (but still include some nonsense xref #40066)

 before after ratio [f4b67b5e] [65792836] <master> <ref-hybrid-3> + 10.1±3ms 13.9±3ms 1.38 eval.Eval.time_add('python', 'all') + 2.06±0.02ms 2.40±0.06ms 1.16 hash_functions.NumericSeriesIndexingShuffled.time_loc_slice(<class 'pandas.core.indexes.numeric.Int64Index'>, 1000000) + 227±2μs 263±2μs 1.15 groupby.GroupByMethods.time_dtype_as_field('datetime', 'head', 'transformation') + 228±2μs 261±2μs 1.15 groupby.GroupByMethods.time_dtype_as_field('datetime', 'head', 'direct') + 238±2μs 272±2μs 1.14 groupby.GroupByMethods.time_dtype_as_field('datetime', 'tail', 'transformation') + 248±6μs 282±5μs 1.14 groupby.GroupByMethods.time_dtype_as_field('datetime', 'tail', 'direct') + 3.92±0.03ms 4.37±0.01ms 1.11 rolling.Engine.time_rolling_apply('DataFrame', 'float', <function Engine.<lambda> at 0x7fb1c0b40670>, 'cython', 'median') + 2.83±0.02ms 3.14±0.06ms 1.11 io.hdf.HDFStoreDataFrame.time_store_info - 275±4μs 248±4μs 0.90 groupby.GroupByMethods.time_dtype_as_field('datetime', 'shift', 'direct') - 1.41±0.05ms 1.27±0.01ms 0.90 stat_ops.FrameOps.time_op('sum', 'int', 1) - 1.13±0.06ms 1.02±0.07ms 0.90 arithmetic.IntFrameWithScalar.time_frame_op_with_scalar(<class 'numpy.int64'>, 5.0, <built-in function ne>) - 271±2μs 242±2μs 0.89 groupby.GroupByMethods.time_dtype_as_field('datetime', 'shift', 'transformation') - 188±3μs 167±1μs 0.89 algos.isin.IsIn.time_isin_empty('datetime64[ns]') - 192±2μs 170±2μs 0.89 algos.isin.IsIn.time_isin_mismatched_dtype('datetime64[ns]') - 227±2μs 200±2μs 0.88 groupby.GroupByMethods.time_dtype_as_field('datetime', 'any', 'direct') - 226±2μs 199±1μs 0.88 groupby.GroupByMethods.time_dtype_as_field('datetime', 'all', 'transformation') - 227±2μs 199±1μs 0.88 groupby.GroupByMethods.time_dtype_as_field('datetime', 'any', 'transformation') - 895±60μs 785±80μs 0.88 arithmetic.IntFrameWithScalar.time_frame_op_with_scalar(<class 'numpy.float64'>, 3.0, <built-in function ge>) - 10.2±0.3ms 8.93±0.7ms 0.88 algos.isin.IsinAlmostFullWithRandomInt.time_isin(<class 'numpy.int64'>, 19, 'inside') - 235±4μs 204±4μs 0.87 groupby.GroupByMethods.time_dtype_as_field('datetime', 'all', 'direct') - 3.26±0.03μs 2.83±0.03μs 0.87 frame_methods.ToNumpy.time_to_numpy_tall - 3.28±0.03μs 2.82±0.02μs 0.86 frame_methods.ToNumpy.time_to_numpy_wide - 9.77±0.2ms 8.40±0.2ms 0.86 indexing.NumericSeriesIndexing.time_loc_slice(<class 'pandas.core.indexes.numeric.UInt64Index'>, 'nonunique_monotonic_inc') - 2.94±0.05μs 2.52±0.02μs 0.86 frame_methods.ToNumpy.time_values_tall - 2.95±0.03μs 2.52±0.02μs 0.85 frame_methods.ToNumpy.time_values_wide - 2.09±0.02ms 1.77±0.01ms 0.85 groupby.FillNA.time_df_ffill - 2.09±0.02ms 1.77±0.01ms 0.85 groupby.FillNA.time_df_bfill - 204±3μs 168±2μs 0.82 arithmetic.OffsetArrayArithmetic.time_add_series_offset(<Day>) - 170±2μs 137±3μs 0.81 groupby.GroupByMethods.time_dtype_as_field('datetime', 'count', 'direct') - 170±2μs 137±4μs 0.80 groupby.GroupByMethods.time_dtype_as_field('datetime', 'count', 'transformation') - 29.1±3ms 22.9±0.4ms 0.79 algos.isin.IsinAlmostFullWithRandomInt.time_isin(<class 'numpy.uint64'>, 20, 'outside') - 26.0±1ms 19.2±2ms 0.74 algos.isin.IsinAlmostFullWithRandomInt.time_isin(<class 'numpy.int64'>, 20, 'inside') - 26.2±0.2ms 18.0±0.07ms 0.69 index_object.SetOperations.time_operation('date_string', 'symmetric_difference') - 11.6±0.1ms 7.32±0.08ms 0.63 reshape.ReshapeExtensionDtype.time_stack('datetime64[ns, US/Pacific]') - 40.2±0.5μs 25.0±0.3μs 0.62 ctors.SeriesDtypesConstructors.time_dtindex_from_index_with_series - 3.77±0.03ms 2.08±0.03ms 0.55 reshape.ReshapeExtensionDtype.time_unstack_slow('datetime64[ns, US/Pacific]') - 32.1±0.5μs 17.0±0.2μs 0.53 ctors.SeriesDtypesConstructors.time_dtindex_from_series - 1.11±0.03ms 408±7μs 0.37 categoricals.Constructor.time_datetimes - 14.1±0.1μs 1.26±0.02μs 0.09 attrs_caching.SeriesArrayAttribute.time_extract_array_numpy('datetime64') - 13.7±0.1μs 1.04±0.03μs 0.08 attrs_caching.SeriesArrayAttribute.time_extract_array('datetime64') - 13.0±0.2μs 455±10ns 0.04 attrs_caching.SeriesArrayAttribute.time_array('datetime64') - 73.8±1ms 1.66±0.03ms 0.02 reshape.ReshapeExtensionDtype.time_unstack_fast('datetime64[ns, US/Pacific]') - 64.3±0.9ms 258±2μs 0.00 reshape.ReshapeExtensionDtype.time_transpose('datetime64[ns, US/Pacific]') 

IIRC the groupby.GroupByMethods.time_dtype_as_field were heavily influenced by constructor overhead, which motivated #40054. Still need to try out @jorisvandenbossche's suggestion of non-cython optimization there.

[jbrockmendel]

@jreback would it help to split off the perf-improving part of this for a follow-up to further trim the diff?

[jreback]

@jreback would it help to split off the perf-improving part of this for a follow-up to further trim the diff?

sure

[jbrockmendel]

@jreback would it help to split off the perf-improving part of this for a follow-up to further trim the diff?

sure

hmm this is looking more involved than i expected. Can try if it makes a difference, basically would split off everything in frame.py

[jreback]

let me look again

i think if u can reduce what is added to frame would be good

[jbrockmendel]

broken in as close to half as possible (not that close) in #41082

[jbrockmendel]

fairly small diff now

[jreback]

looks fine. can you rebase. pls add a whatsnew note (as this is a non-trivial perf increse). ping on green.

[jbrockmendel]

ping

[jreback]

thanks!