Merge branch 'master' into bug-union_many

Author: jbrockmendel

.github/workflows/ci.yml

@@ -55,7 +55,7 @@ jobs:
 
  - name: Install pyright
  # note: keep version in sync with .pre-commit-config.yaml
- run: npm install -g pyright@1.1.171
+ run: npm install -g pyright@1.1.200
 
  - name: Build Pandas
  uses: ./.github/actions/build_pandas

.pre-commit-config.yaml

@@ -78,7 +78,7 @@ repos:
  types: [python]
  stages: [manual]
  # note: keep version in sync with .github/workflows/ci.yml
- additional_dependencies: ['pyright@1.1.171']
+ additional_dependencies: ['pyright@1.1.200']
 - repo: local
  hooks:
  - id: flake8-rst

doc/source/getting_started/comparison/comparison_with_spreadsheets.rst

@@ -435,13 +435,14 @@ The equivalent in pandas:
 Adding a row
 ~~~~~~~~~~~~
 
-Assuming we are using a :class:`~pandas.RangeIndex` (numbered ``0``, ``1``, etc.), we can use :meth:`DataFrame.append` to add a row to the bottom of a ``DataFrame``.
+Assuming we are using a :class:`~pandas.RangeIndex` (numbered ``0``, ``1``, etc.), we can use :func:`concat` to add a row to the bottom of a ``DataFrame``.
 
 .. ipython:: python
 
  df
- new_row = {"class": "E", "student_count": 51, "all_pass": True}
- df.append(new_row, ignore_index=True)
+ new_row = pd.DataFrame([["E", 51, True]],
+ columns=["class", "student_count", "all_pass"])
+ pd.concat([df, new_row])
 
 
 Find and Replace

doc/source/user_guide/10min.rst

@@ -478,7 +478,6 @@ Concatenating pandas objects together with :func:`concat`:
  a row requires a copy, and may be expensive. We recommend passing a
  pre-built list of records to the :class:`DataFrame` constructor instead
  of building a :class:`DataFrame` by iteratively appending records to it.
- See :ref:`Appending to dataframe <merging.concatenation>` for more.
 
 Join
 ~~~~

doc/source/user_guide/cookbook.rst

@@ -929,9 +929,9 @@ Valid frequency arguments to Grouper :ref:`Timeseries <timeseries.offset_aliases
 Merge
 -----
 
-The :ref:`Concat <merging.concatenation>` docs. The :ref:`Join <merging.join>` docs.
+The :ref:`Join <merging.join>` docs.
 
-`Append two dataframes with overlapping index (emulate R rbind)
+`Concatenate two dataframes with overlapping index (emulate R rbind)
 <https://stackoverflow.com/questions/14988480/pandas-version-of-rbind>`__
 
 .. ipython:: python
@@ -944,7 +944,7 @@ Depending on df construction, ``ignore_index`` may be needed
 
 .. ipython:: python
 
- df = df1.append(df2, ignore_index=True)
+ df = pd.concat([df1, df2], ignore_index=True)
  df
 
 `Self Join of a DataFrame

doc/source/user_guide/merging.rst

@@ -237,59 +237,6 @@ Similarly, we could index before the concatenation:
  p.plot([df1, df4], result, labels=["df1", "df4"], vertical=False);
  plt.close("all");
 
-.. _merging.concatenation:
-
-Concatenating using ``append``
-~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
-
-A useful shortcut to :func:`~pandas.concat` are the :meth:`~DataFrame.append`
-instance methods on ``Series`` and ``DataFrame``. These methods actually predated
-``concat``. They concatenate along ``axis=0``, namely the index:
-
-.. ipython:: python
-
- result = df1.append(df2)
-
-.. ipython:: python
- :suppress:
-
- @savefig merging_append1.png
- p.plot([df1, df2], result, labels=["df1", "df2"], vertical=True);
- plt.close("all");
-
-In the case of ``DataFrame``, the indexes must be disjoint but the columns do not
-need to be:
-
-.. ipython:: python
-
- result = df1.append(df4, sort=False)
-
-.. ipython:: python
- :suppress:
-
- @savefig merging_append2.png
- p.plot([df1, df4], result, labels=["df1", "df4"], vertical=True);
- plt.close("all");
-
-``append`` may take multiple objects to concatenate:
-
-.. ipython:: python
-
- result = df1.append([df2, df3])
-
-.. ipython:: python
- :suppress:
-
- @savefig merging_append3.png
- p.plot([df1, df2, df3], result, labels=["df1", "df2", "df3"], vertical=True);
- plt.close("all");
-
-.. note::
-
- Unlike the :py:meth:`~list.append` method, which appends to the original list
- and returns ``None``, :meth:`~DataFrame.append` here **does not** modify
- ``df1`` and returns its copy with ``df2`` appended.
-
 .. _merging.ignore_index:
 
 Ignoring indexes on the concatenation axis
@@ -309,19 +256,6 @@ do this, use the ``ignore_index`` argument:
  p.plot([df1, df4], result, labels=["df1", "df4"], vertical=True);
  plt.close("all");
 
-This is also a valid argument to :meth:`DataFrame.append`:
-
-.. ipython:: python
-
- result = df1.append(df4, ignore_index=True, sort=False)
-
-.. ipython:: python
- :suppress:
-
- @savefig merging_append_ignore_index.png
- p.plot([df1, df4], result, labels=["df1", "df4"], vertical=True);
- plt.close("all");
-
 .. _merging.mixed_ndims:
 
 Concatenating with mixed ndims
@@ -473,14 +407,13 @@ like GroupBy where the order of a categorical variable is meaningful.
 Appending rows to a DataFrame
 ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
 
-While not especially efficient (since a new object must be created), you can
-append a single row to a ``DataFrame`` by passing a ``Series`` or dict to
-``append``, which returns a new ``DataFrame`` as above.
+If you have a series that you want to append as a single row to a ``DataFrame``, you can convert the row into a
+``DataFrame`` and use ``concat``
 
 .. ipython:: python
 
  s2 = pd.Series(["X0", "X1", "X2", "X3"], index=["A", "B", "C", "D"])
- result = df1.append(s2, ignore_index=True)
+ result = pd.concat([df1, s2.to_frame().T], ignore_index=True)
 
 .. ipython:: python
  :suppress:
@@ -493,20 +426,6 @@ You should use ``ignore_index`` with this method to instruct DataFrame to
 discard its index. If you wish to preserve the index, you should construct an
 appropriately-indexed DataFrame and append or concatenate those objects.
 
-You can also pass a list of dicts or Series:
-
-.. ipython:: python
-
- dicts = [{"A": 1, "B": 2, "C": 3, "X": 4}, {"A": 5, "B": 6, "C": 7, "Y": 8}]
- result = df1.append(dicts, ignore_index=True, sort=False)
-
-.. ipython:: python
- :suppress:
-
- @savefig merging_append_dits.png
- p.plot([df1, pd.DataFrame(dicts)], result, labels=["df1", "dicts"], vertical=True);
- plt.close("all");
-
 .. _merging.join:
 
 Database-style DataFrame or named Series joining/merging

doc/source/user_guide/timeseries.rst

@@ -2102,19 +2102,14 @@ The ``period`` dtype can be used in ``.astype(...)``. It allows one to change th
  # change monthly freq to daily freq
  pi.astype("period[D]")
 
+ # convert to DatetimeIndex
+ pi.astype("datetime64[ns]")
+
  # convert to PeriodIndex
  dti = pd.date_range("2011-01-01", freq="M", periods=3)
  dti
  dti.astype("period[M]")
 
-.. deprecated:: 1.4.0
- Converting PeriodIndex to DatetimeIndex with ``.astype(...)`` is deprecated and will raise in a future version. Use ``obj.to_timestamp(how).tz_localize(dtype.tz)`` instead.
-
-.. ipython:: python
-
- # convert to DatetimeIndex
- pi.to_timestamp(how="start")
-
 PeriodIndex partial string indexing
 ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
 

doc/source/whatsnew/v0.6.1.rst

@@ -6,7 +6,7 @@ Version 0.6.1 (December 13, 2011)
 
 New features
 ~~~~~~~~~~~~
-- Can :ref:`append single rows <merging.append.row>` (as Series) to a DataFrame
+- Can append single rows (as Series) to a DataFrame
 - Add Spearman and Kendall rank :ref:`correlation <computation.correlation>`
  options to Series.corr and DataFrame.corr (:issue:`428`)
 - :ref:`Added <indexing.basics.get_value>` ``get_value`` and ``set_value`` methods to

doc/source/whatsnew/v0.7.0.rst

@@ -19,7 +19,7 @@ New features
  intersection of the other axes. Improves performance of ``Series.append`` and
  ``DataFrame.append`` (:issue:`468`, :issue:`479`, :issue:`273`)
 
-- :ref:`Can <merging.concatenation>` pass multiple DataFrames to
+- Can pass multiple DataFrames to
  ``DataFrame.append`` to concatenate (stack) and multiple Series to
  ``Series.append`` too
 

doc/source/whatsnew/v1.4.0.rst

@@ -111,6 +111,7 @@ Additionally there are specific enhancements to the HTML specific rendering:
  - :meth:`.Styler.to_html` introduces keyword arguments ``sparse_index``, ``sparse_columns``, ``bold_headers``, ``caption``, ``max_rows`` and ``max_columns`` (:issue:`41946`, :issue:`43149`, :issue:`42972`).
  - :meth:`.Styler.to_html` omits CSSStyle rules for hidden table elements as a performance enhancement (:issue:`43619`)
  - Custom CSS classes can now be directly specified without string replacement (:issue:`43686`)
+ - Ability to render hyperlinks automatically via a new ``hyperlinks`` formatting keyword argument (:issue:`45058`)
 
 There are also some LaTeX specific enhancements:
 
@@ -275,7 +276,7 @@ ignored when finding the concatenated dtype. These are now consistently *not* i
 
  df1 = pd.DataFrame({"bar": [pd.Timestamp("2013-01-01")]}, index=range(1))
  df2 = pd.DataFrame({"bar": np.nan}, index=range(1, 2))
- res = df1.append(df2)
+ res = pd.concat([df1, df2])
 
 Previously, the float-dtype in ``df2`` would be ignored so the result dtype would be ``datetime64[ns]``. As a result, the ``np.nan`` would be cast to ``NaT``.
 
@@ -364,10 +365,29 @@ second column is instead renamed to ``a.2``.
 
  res
 
-.. _whatsnew_140.notable_bug_fixes.notable_bug_fix3:
+.. _whatsnew_140.notable_bug_fixes.unstack_pivot_int32_limit:
 
-notable_bug_fix3
-^^^^^^^^^^^^^^^^
+unstack and pivot_table no longer raises ValueError for result that would exceed int32 limit
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+
+Previously :meth:`DataFrame.pivot_table` and :meth:`DataFrame.unstack` would raise a ``ValueError`` if the operation
+could produce a result with more than ``2**31 - 1`` elements. This operation now raises a :class:`errors.PerformanceWarning`
+instead (:issue:`26314`).
+
+*Previous behavior*:
+
+.. code-block:: ipython
+
+ In [3]: df = DataFrame({"ind1": np.arange(2 ** 16), "ind2": np.arange(2 ** 16), "count": 0})
+ In [4]: df.pivot_table(index="ind1", columns="ind2", values="count", aggfunc="count")
+ ValueError: Unstacked DataFrame is too big, causing int32 overflow
+
+*New behavior*:
+
+.. code-block:: python
+
+ In [4]: df.pivot_table(index="ind1", columns="ind2", values="count", aggfunc="count")
+ PerformanceWarning: The following operation may generate 4294967296 cells in the resulting pandas object.
 
 .. ---------------------------------------------------------------------------
 
@@ -398,7 +418,7 @@ If installed, we now require:
 +-----------------+-----------------+----------+---------+
 | pytest (dev) | 6.0 | | |
 +-----------------+-----------------+----------+---------+
-| mypy (dev) | 0.920 | | X |
+| mypy (dev) | 0.930 | | X |
 +-----------------+-----------------+----------+---------+
 
 For `optional libraries <https://pandas.pydata.org/docs/getting_started/install.html>`_ the general recommendation is to use the latest version.
@@ -510,6 +530,49 @@ when given numeric data, but in the future, a :class:`NumericIndex` will be retu
  Out [4]: NumericIndex([1, 2, 3], dtype='uint64')
 
 
+.. _whatsnew_140.deprecations.frame_series_append:
+
+Deprecated Frame.append and Series.append
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+
+:meth:`DataFrame.append` and :meth:`Series.append` have been deprecated and will be removed in Pandas 2.0.
+Use :func:`pandas.concat` instead (:issue:`35407`).
+
+*Deprecated syntax*
+
+.. code-block:: ipython
+
+ In [1]: pd.Series([1, 2]).append(pd.Series([3, 4])
+ Out [1]:
+ <stdin>:1: FutureWarning: The series.append method is deprecated and will be removed from pandas in a future version. Use pandas.concat instead.
+ 0 1
+ 1 2
+ 0 3
+ 1 4
+ dtype: int64
+
+ In [2]: df1 = pd.DataFrame([[1, 2], [3, 4]], columns=list('AB'))
+ In [3]: df2 = pd.DataFrame([[5, 6], [7, 8]], columns=list('AB'))
+ In [4]: df1.append(df2)
+ Out [4]:
+ <stdin>:1: FutureWarning: The series.append method is deprecated and will be removed from pandas in a future version. Use pandas.concat instead.
+ A B
+ 0 1 2
+ 1 3 4
+ 0 5 6
+ 1 7 8
+
+*Recommended syntax*
+
+.. ipython:: python
+
+ pd.concat([pd.Series([1, 2]), pd.Series([3, 4])])
+
+ df1 = pd.DataFrame([[1, 2], [3, 4]], columns=list('AB'))
+ df2 = pd.DataFrame([[5, 6], [7, 8]], columns=list('AB'))
+ pd.concat([df1, df2])
+
+
 .. _whatsnew_140.deprecations.other:
 
 Other Deprecations
@@ -537,7 +600,6 @@ Other Deprecations
 - Deprecated casting behavior when passing an item with mismatched-timezone to :meth:`DatetimeIndex.insert`, :meth:`DatetimeIndex.putmask`, :meth:`DatetimeIndex.where` :meth:`DatetimeIndex.fillna`, :meth:`Series.mask`, :meth:`Series.where`, :meth:`Series.fillna`, :meth:`Series.shift`, :meth:`Series.replace`, :meth:`Series.reindex` (and :class:`DataFrame` column analogues). In the past this has cast to object dtype. In a future version, these will cast the passed item to the index or series's timezone (:issue:`37605`,:issue:`44940`)
 - Deprecated the 'errors' keyword argument in :meth:`Series.where`, :meth:`DataFrame.where`, :meth:`Series.mask`, and :meth:`DataFrame.mask`; in a future version the argument will be removed (:issue:`44294`)
 - Deprecated the ``prefix`` keyword argument in :func:`read_csv` and :func:`read_table`, in a future version the argument will be removed (:issue:`43396`)
-- Deprecated :meth:`PeriodIndex.astype` to ``datetime64[ns]`` or ``DatetimeTZDtype``, use ``obj.to_timestamp(how).tz_localize(dtype.tz)`` instead (:issue:`44398`)
 - Deprecated passing non boolean argument to sort in :func:`concat` (:issue:`41518`)
 - Deprecated passing arguments as positional for :func:`read_fwf` other than ``filepath_or_buffer`` (:issue:`41485`):
 - Deprecated passing ``skipna=None`` for :meth:`DataFrame.mad` and :meth:`Series.mad`, pass ``skipna=True`` instead (:issue:`44580`)
@@ -549,6 +611,7 @@ Other Deprecations
 - Deprecated downcasting column-by-column in :meth:`DataFrame.where` with integer-dtypes (:issue:`44597`)
 - Deprecated :meth:`DatetimeIndex.union_many`, use :meth:`DatetimeIndex.union` instead (:issue:`44091`)
 - Deprecated ``numeric_only=None`` in :meth:`DataFrame.rank`; in a future version ``numeric_only`` must be either ``True`` or ``False`` (the default) (:issue:`45036`)
+- Deprecated the behavior of :meth:`Timestamp.utcfromtimestamp`, in the future it will return a timezone-aware UTC :class:`Timestamp` (:issue:`22451`)
 - Deprecated :meth:`NaT.freq` (:issue:`45071`)
 -
 
@@ -641,6 +704,7 @@ Datetimelike
 - Bug in :meth:`Index.insert` for inserting ``np.datetime64``, ``np.timedelta64`` or ``tuple`` into :class:`Index` with ``dtype='object'`` with negative loc adding ``None`` and replacing existing value (:issue:`44509`)
 - Bug in :meth:`Series.mode` with ``DatetimeTZDtype`` incorrectly returning timezone-naive and ``PeriodDtype`` incorrectly raising (:issue:`41927`)
 - Bug in :class:`DateOffset`` addition with :class:`Timestamp` where ``offset.nanoseconds`` would not be included in the result (:issue:`43968`, :issue:`36589`)
+- Bug in :meth:`Timestamp.fromtimestamp` not supporting the ``tz`` argument (:issue:`45083`)
 - Bug in :class:`DataFrame` construction from dict of :class:`Series` with mismatched index dtypes sometimes raising depending on the ordering of the passed dict (:issue:`44091`)
 -
 
@@ -672,12 +736,14 @@ Conversion
 ^^^^^^^^^^
 - Bug in :class:`UInt64Index` constructor when passing a list containing both positive integers small enough to cast to int64 and integers too large too hold in int64 (:issue:`42201`)
 - Bug in :class:`Series` constructor returning 0 for missing values with dtype ``int64`` and ``False`` for dtype ``bool`` (:issue:`43017`, :issue:`43018`)
+- Bug in constructing a :class:`DataFrame` from a :class:`PandasArray` containing :class:`Series` objects behaving differently than an equivalent ``np.ndarray`` (:issue:`43986`)
 - Bug in :class:`IntegerDtype` not allowing coercion from string dtype (:issue:`25472`)
 - Bug in :func:`to_datetime` with ``arg:xr.DataArray`` and ``unit="ns"`` specified raises TypeError (:issue:`44053`)
 - Bug in :meth:`DataFrame.convert_dtypes` not returning the correct type when a subclass does not overload :meth:`_constructor_sliced` (:issue:`43201`)
 - Bug in :meth:`DataFrame.astype` not propagating ``attrs`` from the original :class:`DataFrame` (:issue:`44414`)
 - Bug in :meth:`DataFrame.convert_dtypes` result losing ``columns.names`` (:issue:`41435`)
 - Bug in constructing a ``IntegerArray`` from pyarrow data failing to validate dtypes (:issue:`44891`)
+- Bug in :meth:`Series.astype` not allowing converting from a ``PeriodDtype`` to ``datetime64`` dtype, inconsistent with the :class:`PeriodIndex` behavior (:issue:`45038`)
 -
 
 Strings
@@ -787,6 +853,7 @@ I/O
 - Bug in :func:`read_csv` not setting name of :class:`MultiIndex` columns correctly when ``index_col`` is not the first column (:issue:`38549`)
 - Bug in :func:`read_csv` silently ignoring errors when failing to create a memory-mapped file (:issue:`44766`)
 - Bug in :func:`read_csv` when passing a ``tempfile.SpooledTemporaryFile`` opened in binary mode (:issue:`44748`)
+- Bug in :func:`read_json` raising ``ValueError`` when attempting to parse json strings containing "://" (:issue:`36271`)
 -
 
 Period