Merge branch 'master' into dict_keys_as_csv_names

Author: abmyii

doc/source/development/contributing.rst

@@ -1365,16 +1365,16 @@ environments. If you want to use virtualenv instead, write::
 The ``-E virtualenv`` option should be added to all ``asv`` commands
 that run benchmarks. The default value is defined in ``asv.conf.json``.
 
-Running the full test suite can take up to one hour and use up to 3GB of RAM.
-Usually it is sufficient to paste only a subset of the results into the pull
-request to show that the committed changes do not cause unexpected performance
-regressions. You can run specific benchmarks using the ``-b`` flag, which
-takes a regular expression.  For example, this will only run tests from a
-``pandas/asv_bench/benchmarks/groupby.py`` file::
+Running the full benchmark suite can be an all-day process, depending on your
+hardware and its resource utilization. However, usually it is sufficient to paste
+only a subset of the results into the pull request to show that the committed changes
+do not cause unexpected performance regressions. You can run specific benchmarks
+using the ``-b`` flag, which takes a regular expression. For example, this will
+only run benchmarks from a ``pandas/asv_bench/benchmarks/groupby.py`` file::
 
  asv continuous -f 1.1 upstream/master HEAD -b ^groupby
 
-If you want to only run a specific group of tests from a file, you can do it
+If you want to only run a specific group of benchmarks from a file, you can do it
 using ``.`` as a separator. For example::
 
  asv continuous -f 1.1 upstream/master HEAD -b groupby.GroupByMethods

doc/source/user_guide/indexing.rst

@@ -933,6 +933,24 @@ and :ref:`Advanced Indexing <advanced>` you may select along more than one axis
 
  df2.loc[criterion & (df2['b'] == 'x'), 'b':'c']
 
+.. warning::
+
+ ``iloc`` supports two kinds of boolean indexing. If the indexer is a boolean ``Series``,
+ an error will be raised. For instance, in the following example, ``df.iloc[s.values, 1]`` is ok.
+ The boolean indexer is an array. But ``df.iloc[s, 1]`` would raise ``ValueError``.
+
+ .. ipython:: python
+
+ df = pd.DataFrame([[1, 2], [3, 4], [5, 6]],
+ index=list('abc'),
+ columns=['A', 'B'])
+ s = (df['A'] > 2)
+ s
+
+ df.loc[s, 'B']
+
+ df.iloc[s.values, 1]
+
 .. _indexing.basics.indexing_isin:
 
 Indexing with isin

doc/source/user_guide/merging.rst

@@ -154,6 +154,14 @@ functionality below.
  frames = [ process_your_file(f) for f in files ]
  result = pd.concat(frames)
 
+.. note::
+
+ When concatenating DataFrames with named axes, pandas will attempt to preserve
+ these index/column names whenever possible. In the case where all inputs share a
+ common name, this name will be assigned to the result. When the input names do
+ not all agree, the result will be unnamed. The same is true for :class:`MultiIndex`,
+ but the logic is applied separately on a level-by-level basis.
+
 
 Set logic on the other axes
 ~~~~~~~~~~~~~~~~~~~~~~~~~~~

doc/source/whatsnew/v1.2.0.rst

@@ -157,6 +157,26 @@ Alternatively, you can also use the dtype object:
  behaviour or API may still change without warning. Expecially the behaviour
  regarding NaN (distinct from NA missing values) is subject to change.
 
+.. _whatsnew_120.index_name_preservation:
+
+Index/column name preservation when aggregating
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+
+When aggregating using :meth:`concat` or the :class:`DataFrame` constructor, Pandas
+will attempt to preserve index (and column) names whenever possible (:issue:`35847`).
+In the case where all inputs share a common name, this name will be assigned to the
+result. When the input names do not all agree, the result will be unnamed. Here is an
+example where the index name is preserved:
+
+.. ipython:: python
+
+ idx = pd.Index(range(5), name='abc')
+ ser = pd.Series(range(5, 10), index=idx)
+ pd.concat({'x': ser[1:], 'y': ser[:-1]}, axis=1)
+
+The same is true for :class:`MultiIndex`, but the logic is applied separately on a
+level-by-level basis.
+
 .. _whatsnew_120.enhancements.other:
 
 Other enhancements
@@ -269,6 +289,7 @@ Deprecations
 - Deprecated :meth:`Index.is_all_dates` (:issue:`27744`)
 - Deprecated automatic alignment on comparison operations between :class:`DataFrame` and :class:`Series`, do ``frame, ser = frame.align(ser, axis=1, copy=False)`` before e.g. ``frame == ser`` (:issue:`28759`)
 - :meth:`Rolling.count` with ``min_periods=None`` will default to the size of the window in a future version (:issue:`31302`)
+- Deprecated slice-indexing on timezone-aware :class:`DatetimeIndex` with naive ``datetime`` objects, to match scalar indexing behavior (:issue:`36148`)
 - :meth:`Index.ravel` returning a ``np.ndarray`` is deprecated, in the future this will return a view on the same index (:issue:`19956`)
 
 .. ---------------------------------------------------------------------------
@@ -337,6 +358,7 @@ Numeric
 - Bug in :class:`Series` where two :class:`Series` each have a :class:`DatetimeIndex` with different timezones having those indexes incorrectly changed when performing arithmetic operations (:issue:`33671`)
 - Bug in :meth:`pd._testing.assert_almost_equal` was incorrect for complex numeric types (:issue:`28235`)
 - Bug in :meth:`DataFrame.__rmatmul__` error handling reporting transposed shapes (:issue:`21581`)
+- Bug in :class:`Series` flex arithmetic methods where the result when operating with a ``list``, ``tuple`` or ``np.ndarray`` would have an incorrect name (:issue:`36760`)
 - Bug in :class:`IntegerArray` multiplication with ``timedelta`` and ``np.timedelta64`` objects (:issue:`36870`)
 
 Conversion
@@ -394,6 +416,7 @@ I/O
 - Bug in :meth:`read_csv` with ``engine='python'`` truncating data if multiple items present in first row and first element started with BOM (:issue:`36343`)
 - Removed ``private_key`` and ``verbose`` from :func:`read_gbq` as they are no longer supported in ``pandas-gbq`` (:issue:`34654`, :issue:`30200`)
 - Bumped minimum pytables version to 3.5.1 to avoid a ``ValueError`` in :meth:`read_hdf` (:issue:`24839`)
+- Bug in :func:`read_table` and :func:`read_csv` when ``delim_whitespace=True`` and ``sep=default`` (:issue:`36583`)
 - Bug in :meth:`read_parquet` with fixed offset timezones. String representation of timezones was not recognized (:issue:`35997`, :issue:`36004`)
 
 Plotting

pandas/conftest.py

@@ -677,6 +677,43 @@ def all_arithmetic_operators(request):
  return request.param
 
 
+@pytest.fixture(
+ params=[
+ operator.add,
+ ops.radd,
+ operator.sub,
+ ops.rsub,
+ operator.mul,
+ ops.rmul,
+ operator.truediv,
+ ops.rtruediv,
+ operator.floordiv,
+ ops.rfloordiv,
+ operator.mod,
+ ops.rmod,
+ operator.pow,
+ ops.rpow,
+ operator.eq,
+ operator.ne,
+ operator.lt,
+ operator.le,
+ operator.gt,
+ operator.ge,
+ operator.and_,
+ ops.rand_,
+ operator.xor,
+ ops.rxor,
+ operator.or_,
+ ops.ror_,
+ ]
+)
+def all_binary_operators(request):
+ """
+ Fixture for operator and roperator arithmetic, comparison, and logical ops.
+ """
+ return request.param
+
+
 @pytest.fixture(
  params=[
  operator.add,

pandas/core/arraylike.py

@@ -0,0 +1,43 @@
+"""
+Methods that can be shared by many array-like classes or subclasses:
+ Series
+ Index
+ ExtensionArray
+"""
+import operator
+
+from pandas.errors import AbstractMethodError
+
+from pandas.core.ops.common import unpack_zerodim_and_defer
+
+
+class OpsMixin:
+ # -------------------------------------------------------------
+ # Comparisons
+
+ def _cmp_method(self, other, op):
+ raise AbstractMethodError(self)
+
+ @unpack_zerodim_and_defer("__eq__")
+ def __eq__(self, other):
+ return self._cmp_method(other, operator.eq)
+
+ @unpack_zerodim_and_defer("__ne__")
+ def __ne__(self, other):
+ return self._cmp_method(other, operator.ne)
+
+ @unpack_zerodim_and_defer("__lt__")
+ def __lt__(self, other):
+ return self._cmp_method(other, operator.lt)
+
+ @unpack_zerodim_and_defer("__le__")
+ def __le__(self, other):
+ return self._cmp_method(other, operator.le)
+
+ @unpack_zerodim_and_defer("__gt__")
+ def __gt__(self, other):
+ return self._cmp_method(other, operator.gt)
+
+ @unpack_zerodim_and_defer("__ge__")
+ def __ge__(self, other):
+ return self._cmp_method(other, operator.ge)

pandas/core/arrays/boolean.py

@@ -23,6 +23,7 @@
 from pandas.core.dtypes.missing import isna
 
 from pandas.core import ops
+from pandas.core.arraylike import OpsMixin
 
 from .masked import BaseMaskedArray, BaseMaskedDtype
 
@@ -202,7 +203,7 @@ def coerce_to_array(
  return values, mask
 
 
-class BooleanArray(BaseMaskedArray):
+class BooleanArray(OpsMixin, BaseMaskedArray):
  """
  Array of boolean (True/False) data with missing values.
 
@@ -603,52 +604,44 @@ def logical_method(self, other):
  name = f"__{op.__name__}__"
  return set_function_name(logical_method, name, cls)
 
- @classmethod
- def _create_comparison_method(cls, op):
- @ops.unpack_zerodim_and_defer(op.__name__)
- def cmp_method(self, other):
- from pandas.arrays import FloatingArray, IntegerArray
+ def _cmp_method(self, other, op):
+ from pandas.arrays import FloatingArray, IntegerArray
 
-  if isinstance(other, (IntegerArray, FloatingArray)):
-  return NotImplemented
+ if isinstance(other, (IntegerArray, FloatingArray)):
+ return NotImplemented
 
-  mask = None
+ mask = None
 
-  if isinstance(other, BooleanArray):
-  other, mask = other._data, other._mask
+ if isinstance(other, BooleanArray):
+ other, mask = other._data, other._mask
 
- elif is_list_like(other):
- other = np.asarray(other)
- if other.ndim > 1:
- raise NotImplementedError(
- "can only perform ops with 1-d structures"
- )
- if len(self) != len(other):
- raise ValueError("Lengths must match to compare")
+ elif is_list_like(other):
+ other = np.asarray(other)
+ if other.ndim > 1:
+ raise NotImplementedError("can only perform ops with 1-d structures")
+ if len(self) != len(other):
+ raise ValueError("Lengths must match to compare")
 
- if other is libmissing.NA:
- # numpy does not handle pd.NA well as "other" scalar (it returns
- # a scalar False instead of an array)
- result = np.zeros_like(self._data)
- mask = np.ones_like(self._data)
- else:
- # numpy will show a DeprecationWarning on invalid elementwise
- # comparisons, this will raise in the future
- with warnings.catch_warnings():
- warnings.filterwarnings("ignore", "elementwise", FutureWarning)
- with np.errstate(all="ignore"):
- result = op(self._data, other)
-
- # nans propagate
- if mask is None:
- mask = self._mask.copy()
- else:
- mask = self._mask | mask
+ if other is libmissing.NA:
+ # numpy does not handle pd.NA well as "other" scalar (it returns
+ # a scalar False instead of an array)
+ result = np.zeros_like(self._data)
+ mask = np.ones_like(self._data)
+ else:
+ # numpy will show a DeprecationWarning on invalid elementwise
+ # comparisons, this will raise in the future
+ with warnings.catch_warnings():
+ warnings.filterwarnings("ignore", "elementwise", FutureWarning)
+ with np.errstate(all="ignore"):
+ result = op(self._data, other)
 
- return BooleanArray(result, mask, copy=False)
+ # nans propagate
+ if mask is None:
+ mask = self._mask.copy()
+ else:
+ mask = self._mask | mask
 
- name = f"__{op.__name__}"
- return set_function_name(cmp_method, name, cls)
+ return BooleanArray(result, mask, copy=False)
 
  def _reduce(self, name: str, skipna: bool = True, **kwargs):
 
@@ -741,5 +734,4 @@ def boolean_arithmetic_method(self, other):
 
 
 BooleanArray._add_logical_ops()
-BooleanArray._add_comparison_ops()
 BooleanArray._add_arithmetic_ops()