BUG: tz info lost by set_index and reindex

[sinhrks]

Closes #6631. Closes #5878. Regarding #3950, original problem can be fixed by this, but groupby problem isn't.

Also, this includes the fix for MultiIndex.get_level_values doesn't retain tz and freq, as the method is used in set_index.

# current master >>> import pandas as pd >>> didx = pd.DatetimeIndex(start='2013/01/01', freq='H', periods=4, tz='Asia/Tokyo') >>> pidx = pd.PeriodIndex(start='2013/01/01', freq='H', periods=4) >>> midx = pd.MultiIndex.from_arrays([didx, pidx]) >>> midx.get_level_values(0) <class 'pandas.tseries.index.DatetimeIndex'> [2012-12-31 15:00:00, ..., 2012-12-31 18:00:00] Length: 4, Freq: None, Timezone: None >>> midx.get_level_values(1) Int64Index([376944, 376945, 376946, 376947], dtype='int64') 

# after fix >>> midx.get_level_values(0) <class 'pandas.tseries.index.DatetimeIndex'> [2013-01-01 00:00:00+09:00, ..., 2013-01-01 03:00:00+09:00] Length: 4, Freq: H, Timezone: Asia/Tokyo >>> midx.get_level_values(1) PeriodIndex([u'2013-01-01 00:00', u'2013-01-01 01:00', u'2013-01-01 02:00', u'2013-01-01 03:00'], freq='H') 

[jreback]

I think you can simply do:

values = Index(filled, freq=getattr(unique_vals,'freq',None), tz=getattr(unique_vals,'tz',None, name = self.names[num]) 

instead of the if/elsif (and don't need to import)

Index will infer the type will is more pythonic here (and the additional args will not be passed on)

[sinhrks]

Thanks. PeriodIndex can work but DatetimeIndex doesn't. Because values are once converted to ndarray, Index performs tz conversion once again and results in incorrect output.

[jreback]

hmm
worth creating a _simple_new for Index?

[sinhrks]

Yeah, it will be an option. Otherwise make DatetimeIndex to support np.datetime64 timezone...

[jreback]

no don't use the tz of np.datetime64 totally broken

[sinhrks]

OK, added _simple_new to Index. Also, I noticed #5878 can be solved by the same fix.

[jreback]

gr8! thanks