PERF/DOC: Option to .info() and .memory_usage() to provide for deep introspection of memory consumption #11595

[jreback]

closes #11595

[jreback]

cc @mrocklin

[mrocklin]

Does this descend into categories and the index?

[jreback]

wondering if you were going to ask that....it DOES do the index. not the categories, but I can fix this ......

[jreback]

Now includes embedded usage for Index & Categorical

In [5]: df = DataFrame({'A' : ['foo']*1000}) In [6]: df['B'] = df['A'].astype('category') In [8]: df.info() <class 'pandas.core.frame.DataFrame'> Int64Index: 1000 entries, 0 to 999 Data columns (total 2 columns): A 1000 non-null object B 1000 non-null category dtypes: category(1), object(1) memory usage: 16.6+ KB In [9]: df.info(deep=True) <class 'pandas.core.frame.DataFrame'> Int64Index: 1000 entries, 0 to 999 Data columns (total 2 columns): A 1000 non-null object B 1000 non-null category dtypes: category(1), object(1) memory usage: 55.7 KB In [11]: df.memory_usage() Out[11]: A 8000 B 1008 dtype: int64 In [12]: df.memory_usage(deep=True) Out[12]: A 48000 B 1048 dtype: int64 

[jreback]

And providing on Series as well

In [6]: df['A'].memory_usage() Out[6]: 8000 In [7]: df['A'].memory_usage(index=True) Out[7]: 16000 In [8]: df['A'].memory_usage(index=True,deep=True) Out[8]: 56000 

[mrocklin]

BTW, I'm glad that memory_usage_of_objects is usable on numpy arrays as well. I may end up using that outside of pandas.

[jreback]

right dask.array could certainly introspect here as well

[max-sixty]

I'll wait until this is merged before adding __getsize__.
Is there a reason index is False by default? I'd have thought that would be 'part of the package'.

[jreback]

@MaximilianR I don't recall the discussion, but I think we should change the default. Note that this is just for a direct call to memory_usage and not for .info where it is included.

why don't you post an issue and we'll change in 0.18 (as its a small API change).