BUG: Timestamp.replace chaining not compat with datetime.replace

[sscherfke]

• closes BUG: Timestamp.replace chaining not compat with datetime.replace #15683
• tests added / passed
• passes git diff upstream/master -u -- "*.py" | flake8 --diff
• whatsnew entry

This is a clean version of #16110 and the last thing I’m going to do with this issue.

[gfyoung]

@sscherfke : This look pretty good (test-wise) save for a minor linting issue, which you can see here:

https://travis-ci.org/pandas-dev/pandas/jobs/269100110#L1809

[codecov]

Codecov Report

Merging #17356 into master will decrease coverage by 0.01%.
The diff coverage is n/a.

@@ Coverage Diff @@ ## master #17356 +/- ## ========================================== - Coverage 91.03% 91.02% -0.02%  ========================================== Files 162 162 Lines 49567 49567 ========================================== - Hits 45125 45116 -9  - Misses 4442 4451 +9

Flag	Coverage Δ
#multiple	88.8% <ø> (ø)	⬆️
#single	40.24% <ø> (-0.07%)	⬇️

Impacted Files	Coverage Δ
pandas/io/gbq.py	25% <0%> (-58.34%)	⬇️
pandas/core/frame.py	97.72% <0%> (-0.1%)	⬇️

---

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 473a7f3...02297a5. Read the comment docs.

[codecov]

Codecov Report

Merging #17356 into master will decrease coverage by 0.01%.
The diff coverage is n/a.

@@ Coverage Diff @@ ## master #17356 +/- ## ========================================== - Coverage 91.03% 91.02% -0.02%  ========================================== Files 162 162 Lines 49567 49567 ========================================== - Hits 45125 45116 -9  - Misses 4442 4451 +9

Flag	Coverage Δ
#multiple	88.8% <ø> (ø)	⬆️
#single	40.24% <ø> (-0.07%)	⬇️

Impacted Files	Coverage Δ
pandas/io/gbq.py	25% <0%> (-58.34%)	⬇️
pandas/core/frame.py	97.72% <0%> (-0.1%)	⬇️

---

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 473a7f3...02297a5. Read the comment docs.

[sscherfke]

My bad. Fixed it.

[sscherfke]

Can this PR please be merge? This function computes wrong results since more than half a year by now!

[TomAugspurger]

From #16110 (comment)

change the definition of tz_convert_single to cpdef int64_t

Can you try that?

[TomAugspurger]

I gave this branch + changing tz_convert_single to a cpdef int64_t and ran through a small benchmark (with profiling enabled):

from datetime import datetime import pytz from pandas import Timestamp import cProfile import pstats dt = datetime(2016, 3, 27, 1) tzinfo = pytz.timezone('CET').localize(dt, is_dst=False).tzinfo result_dt = dt.replace(tzinfo=tzinfo) ts = Timestamp(dt) code = ''' for _ in range(10000):  ts.replace(tzinfo=tzinfo) ''' cProfile.runctx(code, globals(), locals(), "Profile.prof") s = pstats.Stats("Profile.prof") s.strip_dirs().sort_stats("time").print_stats()

Master:

Mon Sep 4 08:16:09 2017 Profile.prof 150005 function calls in 0.104 seconds Ordered by: internal time ncalls tottime percall cumtime percall filename:lineno(function) 10000 0.039 0.000 0.072 0.000 tslib.pyx:4210(tz_convert_single) 10000 0.015 0.000 0.097 0.000 tslib.pyx:677(replace) 20000 0.015 0.000 0.026 0.000 tslib.pyx:1708(_get_zone) 20000 0.009 0.000 0.009 0.000 tslib.pyx:4280(_treat_tz_as_dateutil) 1 0.008 0.008 0.105 0.105 <string>:2(<module>) 10000 0.006 0.000 0.006 0.000 tslib.pyx:110(create_timestamp_from_ts) 10000 0.004 0.000 0.005 0.000 tslib.pyx:4322(_get_dst_info) 20000 0.003 0.000 0.003 0.000 tslib.pyx:1705(_is_utc) 10000 0.001 0.000 0.001 0.000 tslib.pyx:236(_is_tzlocal) 10000 0.001 0.000 0.001 0.000 tslib.pyx:4289(_tz_cache_key) 10000 0.001 0.000 0.001 0.000 tslib.pyx:1043(__get__) 10000 0.001 0.000 0.001 0.000 tslib.pyx:1770(_check_dts_bounds) 10000 0.001 0.000 0.001 0.000 tslib.pyx:1044(__get__) 1 0.000 0.000 0.105 0.105 {built-in method builtins.exec} 1 0.000 0.000 0.000 0.000 tslib.pyx:4402(_unbox_utcoffsets) 1 0.000 0.000 0.000 0.000 {method 'disable' of '_lsprof.Profiler' objects} 1 0.000 0.000 0.000 0.000 tslib.pyx:4276(_treat_tz_as_pytz) 

PR:

Mon Sep 4 08:19:48 2017 Profile.prof 210003 function calls in 0.149 seconds Ordered by: internal time ncalls tottime percall cumtime percall filename:lineno(function) 30000 0.043 0.000 0.043 0.000 {method 'replace' of 'datetime.datetime' objects} 10000 0.030 0.000 0.143 0.000 tslib.pyx:676(replace) 10000 0.020 0.000 0.104 0.000 tslib.pyx:1439(convert_to_tsobject) 10000 0.010 0.000 0.045 0.000 tzinfo.py:179(fromutc) 10000 0.010 0.000 0.010 0.000 tslib.pyx:3966(_delta_to_nanoseconds) 1 0.007 0.007 0.150 0.150 <string>:2(<module>) 10000 0.007 0.000 0.067 0.000 tzinfo.py:189(normalize) 10000 0.006 0.000 0.006 0.000 tslib.pyx:109(create_timestamp_from_ts) 10000 0.004 0.000 0.004 0.000 {built-in method _bisect.bisect_right} 10000 0.002 0.000 0.002 0.000 {built-in method builtins.max} 10000 0.002 0.000 0.002 0.000 tslib.pyx:1745(maybe_get_tz) 10000 0.001 0.000 0.001 0.000 tslib.pyx:1704(_is_utc) 20000 0.001 0.000 0.001 0.000 tslib.pyx:1769(_check_dts_bounds) 10000 0.001 0.000 0.001 0.000 datetime.pxd:144(_pydatetime_to_dts) 10000 0.001 0.000 0.001 0.000 tslib.pyx:1432(_get_utcoffset) 10000 0.001 0.000 0.001 0.000 tslib.pyx:1429(__get__) 10000 0.001 0.000 0.001 0.000 tslib.pyx:1042(__get__) 10000 0.001 0.000 0.001 0.000 tslib.pyx:1330(is_timestamp) 10000 0.001 0.000 0.001 0.000 tslib.pyx:1043(__get__) 1 0.000 0.000 0.150 0.150 {built-in method builtins.exec} 1 0.000 0.000 0.000 0.000 {method 'disable' of '_lsprof.Profiler' objects} 

I'm out of time now, but some quick thoughts:

• datetime.datetime.replace is now the most expensive function call
• We hit it 3 times per .replace, can we reduce that?
• We don't call tz_convert_single now (should re-run this with a non-naive one, but that's good)

[jreback]

replaced by #17507