BUG: df.apply handles np.timedelta64 as timestamp, should be timedelta #7778
Description
I think there may be a bug with the row-wise handling of numpy.timedelta64 data types when using DataFrame.apply. As a check, the problem does not appear when using DataFrame.applymap. The problem may be related to #4532, but I'm unsure. I've included an example below.
This is only a minor problem for my use-case, which is cross-checking timestamps from a counter/timer card. I can easily work around the issue with DataFrame.itertuples etc.
Thank you for your time and for making such a useful package!
Example
Version
Import and check versions.
$ date Thu Jul 17 16:28:38 CDT 2014 $ conda update pandas Fetching package metadata: .. # All requested packages already installed. # packages in environment at /Users/harrold/anaconda: # pandas 0.14.1 np18py27_0 $ ipython Python 2.7.8 |Anaconda 2.0.1 (x86_64)| (default, Jul 2 2014, 15:36:00) Type "copyright", "credits" or "license" for more information. IPython 2.1.0 -- An enhanced Interactive Python. Anaconda is brought to you by Continuum Analytics. Please check out: http://continuum.io/thanks and https://binstar.org ? -> Introduction and overview of IPython's features. %quickref -> Quick reference. help -> Python's own help system. object? -> Details about 'object', use 'object??' for extra details. In [1]: from __future__ import print_function In [2]: import numpy as np In [3]: import pandas as pd In [4]: pd.util.print_versions.show_versions() INSTALLED VERSIONS ------------------ commit: None python: 2.7.8.final.0 python-bits: 64 OS: Darwin OS-release: 11.4.2 machine: x86_64 processor: i386 byteorder: little LC_ALL: None LANG: en_US.UTF-8 pandas: 0.14.1 nose: 1.3.3 Cython: 0.20.1 numpy: 1.8.1 scipy: 0.14.0 statsmodels: 0.5.0 IPython: 2.1.0 sphinx: 1.2.2 patsy: 0.2.1 scikits.timeseries: None dateutil: 1.5 pytz: 2014.4 bottleneck: None tables: 3.1.1 numexpr: 2.3.1 matplotlib: 1.3.1 openpyxl: 1.8.5 xlrd: 0.9.3 xlwt: 0.7.5 xlsxwriter: 0.5.5 lxml: 3.3.5 bs4: 4.3.1 html5lib: 0.999 httplib2: 0.8 apiclient: 1.2 rpy2: None sqlalchemy: 0.9.4 pymysql: None psycopg2: None Create test data
Using subset of original raw data as example.
In [5]: datetime_start = np.datetime64(u'2014-05-31T01:23:19.9600345Z') In [6]: timedeltas_elapsed = [30053400, 40053249, 50053098] Compute datetimes from elapsed timedeltas, then create differential timedeltas from datetimes. All elements are either type numpy.datetime64 or numpy.timedelta64.
In [7]: df = pd.DataFrame(dict(datetimes = timedeltas_elapsed)) In [8]: df = df.applymap(lambda elt: np.timedelta64(elt, 'us')) In [9]: df = df.applymap(lambda elt: np.datetime64(datetime_start + elt)) In [10]: df['differential_timedeltas'] = df['datetimes'] - df['datetimes'].shift() In [11]: print(df) datetimes differential_timedeltas 0 2014-05-31 01:23:50.013434500 NaT 1 2014-05-31 01:24:00.013283500 00:00:09.999849 2 2014-05-31 01:24:10.013132500 00:00:09.999849 Expected behavior
With element-wise handling using DataFrame.applymap, all elements are correctly identified as datetimes (timestamps) or timedeltas.
In [12]: print(df.applymap(lambda elt: type(elt))) datetimes differential_timedeltas 0 <class 'pandas.tslib.Timestamp'> <type 'numpy.timedelta64'> 1 <class 'pandas.tslib.Timestamp'> <type 'numpy.timedelta64'> 2 <class 'pandas.tslib.Timestamp'> <type 'numpy.timedelta64'> Bug
With row-wise handling using DataFrame.apply, all elements are type pandas.tslib.Timestamp. I expected 'differential_timedeltas' to be type numpy.timedelta64 or another type of timedelta, not a type of datetime (timestamp).
In [13]: # For 'datetimes': In [14]: print(df.apply(lambda row: type(row['datetimes']), axis=1)) 0 <class 'pandas.tslib.Timestamp'> 1 <class 'pandas.tslib.Timestamp'> 2 <class 'pandas.tslib.Timestamp'> dtype: object In [15]: # For 'differential_timedeltas': In [16]: print(df.apply(lambda row: type(row['differential_timedeltas']), axis=1)) 0 <class 'pandas.tslib.NaTType'> 1 <class 'pandas.tslib.Timestamp'> 2 <class 'pandas.tslib.Timestamp'> dtype: object