BUG: SparseSeries.value_counts ignores fill_value

[sinhrks]

• closes SparseSeries.value_counts doesn't include fill_value counts #6749
• tests added / passed
• passes git diff upstream/master | flake8 --diff
• whatsnew entry

Also fixed an issue which categorical value_counts resets name.

[jreback]

actually with only a small rewriting I think you could do:

if com.is_extension_dtype(values) or com.is_datetimelike(values): result = values.values_counts(dropna) result.name = name else: ..... 

if we define DateTimeIndex.value_counts that properly handles localization before/after (instead of in here)
will be much cleaner.

Further it makes sense to define .value_counts() in tseries/base.py (and move the inference from core/base.py/value_counts.py there. To handle the dispatching of period/DatetimeIndex.

a bit of refactoring but moves functionaility where it should be.

[sinhrks]

Found current master doesn't work for datetimetz. Let me check.

values = [pd.Timestamp('2011-01-01 09:00', tz='US/Eastern'), pd.Timestamp('2011-01-01 10:00', tz='US/Eastern'), pd.Timestamp('2011-01-01 11:00', tz='US/Eastern'), pd.Timestamp('2011-01-01 09:00', tz='US/Eastern'), pd.Timestamp('2011-01-01 09:00', tz='US/Eastern'), pd.Timestamp('2011-01-01 11:00', tz='US/Eastern')] s = pd.Series(values, name='xxx') # NG, index must have tz s.value_counts() # 2011-01-01 14:00:00 3 # 2011-01-01 16:00:00 2 # 2011-01-01 15:00:00 1 # Name: xxx, dtype: int64 

[sinhrks]

Additionally fixing the other Categorical issue.

pd.CategoricalIndex([1, 2, 3, 1, 2, 3]).value_counts(normalize=True) # UnboundLocalError: local variable 'counts' referenced before assignment 

[jreback]

@sinhrks pls rebase when you can

[codecov-io]

Current coverage is 84.04%

Merging #12835 into master will increase coverage by +0.07%

@@ master #12835 diff @@ ========================================== Files 136 136 Lines 49908 49931 +23 Methods 0 0 Messages 0 0 Branches 0 0 ========================================== + Hits 41876 41962 +86  + Misses 8032 7969 -63  Partials 0 0 

1. File ...ndas/core/groupby.py (not in diff) was modified. more
   • Misses -63
   • Partials 0
   • Hits +63

Powered by Codecov. Last updated by 38a7531

[jreback]

this won't happen here as is_extension_type will do this

[sinhrks]

I removed @slow as groupby.value_coutns has separate logic depending on numpy version.

• https://github.com/pydata/pandas/blob/master/pandas/core/groupby.py#L2892

[jreback]

ok, though I still think we should handle in tseries/base, e.g. define DatetimeIndexOpsMixIN and just defer to it like we are doing for sparse/categorical.

[sinhrks]

sparse/categorical value_counts only supports dropna kwds. It may better to be an internal method.

If we do the same thing using DatetimeIndexOpsMixin, the impl should be:

• DatetimeIndexOpsMixin 's public value_counts calls algorithms.py (as it needs to handle kwds like normalize)
• algorithms.py convert datetime-likes to datetime-like Index to handle Series
• Calls DatetimeIndexOpsMixin 's internal _value_counts

Suppose to be little complex, or do you have anything in your mind?

[jreback]

yeah ok, it can ge tricky. ok, we'll keep in mind for later. ideally to put specific logic. merging.