AmbiguousTimeError when retrieving Series from HDFStore in table format with DST change #11624
Description
In Pandas 0.17, if I put a Series into an HDFStore in table format, and if this Series contains a day light saving transition, then Pandas crashes with an AmbiguousTimeError.
Here's the minimal code example to reproduce the bug:
import pandas as pd index = pd.date_range("2013-10-26 23:00", "2013-10-27 01:00", tz="Europe/London", freq="H") series = pd.Series(0, index=index) with pd.HDFStore("test.hdf", mode="w") as store: store.put("series", series, format='table') store["series"]And here's the traceback:
In [4]: --------------------------------------------------------------------------- AmbiguousTimeError Traceback (most recent call last) <ipython-input-4-5deb3c83fafd> in <module>() ----> 1 __pyfile = open('''/tmp/py7201vZi''');exec(compile(__pyfile.read(), '''/home/jack/temp/pandas_dst_bug.py''', 'exec'));__pyfile.close(); import os; os.remove('''/tmp/py7201vZi''') /home/jack/temp/pandas_dst_bug.py in <module>() 2 index = pd.date_range("2013-10-26 23:00", "2013-10-27 01:00", 3 tz="Europe/London", freq="H") ----> 4 series = pd.Series(0, index=index) 5 with pd.HDFStore("test.hdf", mode="w") as store: 6 store.put("series", series, format='table') /usr/local/lib/python2.7/dist-packages/pandas/tseries/index.pyc in date_range(start, end, periods, freq, tz, normalize, name, closed) 1912 return DatetimeIndex(start=start, end=end, periods=periods, 1913 freq=freq, tz=tz, normalize=normalize, name=name, -> 1914 closed=closed) 1915 1916 /usr/local/lib/python2.7/dist-packages/pandas/util/decorators.pyc in wrapper(*args, **kwargs) 87 else: 88 kwargs[new_arg_name] = new_arg_value ---> 89 return func(*args, **kwargs) 90 return wrapper 91 return _deprecate_kwarg /usr/local/lib/python2.7/dist-packages/pandas/tseries/index.pyc in __new__(cls, data, freq, start, end, periods, copy, name, tz, verify_integrity, normalize, closed, ambiguous, dtype, **kwargs) 234 return cls._generate(start, end, periods, name, freq, 235 tz=tz, normalize=normalize, closed=closed, --> 236 ambiguous=ambiguous) 237 238 if not isinstance(data, (np.ndarray, Index, ABCSeries)): /usr/local/lib/python2.7/dist-packages/pandas/tseries/index.pyc in _generate(cls, start, end, periods, name, offset, tz, normalize, ambiguous, closed) 450 451 if end is not None and end.tz is None: --> 452 end = end.tz_localize(tz) 453 454 if start and end: pandas/tslib.pyx in pandas.tslib.Timestamp.tz_localize (pandas/tslib.c:11965)() pandas/tslib.pyx in pandas.tslib.tz_localize_to_utc (pandas/tslib.c:64516)() AmbiguousTimeError: Cannot infer dst time from Timestamp('2013-10-27 01:00:00'), try using the 'ambiguous' argumentEverything works fine if:
- we use
format=fixedinstead offormat=table - we used
format=tablebut theSeriesdoes not include a DST change
I'm not certain but I expect this bug is related to #11619
I'm 99.999% certain that this bug did not exist in Pandas 0.16.2
In [30]: show_versions() INSTALLED VERSIONS ------------------ commit: None python: 2.7.9.final.0 python-bits: 64 OS: Linux OS-release: 3.19.0-33-generic machine: x86_64 processor: x86_64 byteorder: little LC_ALL: None LANG: en_GB.UTF-8 pandas: 0.17.0 nose: 1.3.7 pip: 1.5.6 setuptools: 15.2 Cython: 0.23.1 numpy: 1.10.1 scipy: 0.16.0 statsmodels: 0.6.1 IPython: 4.0.0 sphinx: 1.2.3 patsy: 0.3.0 dateutil: 2.4.2 pytz: 2015.7 blosc: None bottleneck: None tables: 3.2.2 numexpr: 2.4.6 matplotlib: 1.4.3 openpyxl: None xlrd: 0.9.2 xlwt: None xlsxwriter: None lxml: None bs4: 4.3.2 html5lib: 0.999 httplib2: 0.9 apiclient: None sqlalchemy: None pymysql: None psycopg2: 2.5.3 (dt dec pq3 ext) Finally, just to say another huge thank you to everyone who supports Pandas. It must be a huge amount of work and I am hugely grateful. Pandas is an awesome tool. Thank you.