Merge branch 'master' of https://github.com/pandas-dev/pandas into typing-missing

Author: arw2019

.github/workflows/ci.yml

@@ -6,7 +6,7 @@ on:
  pull_request:
  branches:
  - master
- - 1.1.x
+ - 1.2.x
 
 env:
  ENV_FILE: environment.yml

azure-pipelines.yml

@@ -1,11 +1,11 @@
 # Adapted from https://github.com/numba/numba/blob/master/azure-pipelines.yml
 trigger:
 - master
-- 1.1.x
+- 1.2.x
 
 pr:
 - master
-- 1.1.x
+- 1.2.x
 
 variables:
  PYTEST_WORKERS: auto

doc/source/user_guide/io.rst

@@ -3200,6 +3200,13 @@ pandas supports writing Excel files to buffer-like objects such as ``StringIO``
 Excel writer engines
 ''''''''''''''''''''
 
+.. deprecated:: 1.2.0
+
+ As the `xlwt <https://pypi.org/project/xlwt/>`__ package is no longer
+ maintained, the ``xlwt`` engine will be removed from a future version
+ of pandas. This is the only engine in pandas that supports writing to
+ ``.xls`` files.
+
 pandas chooses an Excel writer via two methods:
 
 1. the ``engine`` keyword argument

doc/source/user_guide/options.rst

@@ -432,6 +432,16 @@ display.html.use_mathjax True When True, Jupyter notebook
  dollar symbol.
 io.excel.xls.writer xlwt The default Excel writer engine for
  'xls' files.
+
+ .. deprecated:: 1.2.0
+
+ As `xlwt <https://pypi.org/project/xlwt/>`__
+ package is no longer maintained, the ``xlwt``
+ engine will be removed in a future version of
+ pandas. Since this is the only engine in pandas
+ that supports writing to ``.xls`` files,
+ this option will also be removed.
+
 io.excel.xlsm.writer openpyxl The default Excel writer engine for
  'xlsm' files. Available options:
  'openpyxl' (the default).

doc/source/whatsnew/index.rst

@@ -10,6 +10,14 @@ This is the list of changes to pandas between each release. For full details,
 see the `commit logs <https://github.com/pandas-dev/pandas/commits/>`_. For install and
 upgrade instructions, see :ref:`install`.
 
+Version 1.3
+-----------
+
+.. toctree::
+ :maxdepth: 2
+
+ v1.3.0
+
 Version 1.2
 -----------
 

doc/source/whatsnew/v1.2.0.rst

@@ -10,12 +10,27 @@ including other versions of pandas.
 
 .. warning::
 
+ The packages `xlrd <https://xlrd.readthedocs.io/en/latest/>`_ for reading excel
+ files and `xlwt <https://xlwt.readthedocs.io/en/latest/>`_ for
+ writing excel files are no longer maintained. These are the only engines in pandas
+ that support the xls format.
+
  Previously, the default argument ``engine=None`` to ``pd.read_excel``
- would result in using the `xlrd <https://xlrd.readthedocs.io/en/latest/>`_ engine in
- many cases. The engine ``xlrd`` is no longer maintained, and is not supported with
- python >= 3.9. If `openpyxl <https://pypi.org/project/openpyxl/>`_ is installed,
- many of these cases will now default to using the ``openpyxl`` engine. See the
- :func:`read_excel` documentation for more details.
+ would result in using the ``xlrd`` engine in many cases. If
+ `openpyxl <https://openpyxl.readthedocs.io/en/stable/>`_ is installed,
+ many of these cases will now default to using the ``openpyxl`` engine.
+ See the :func:`read_excel` documentation for more details. Attempting to read
+ ``.xls`` files or specifying ``engine="xlrd"`` to ``pd.read_excel`` will not
+ raise a warning. However users should be aware that ``xlrd`` is already
+ broken with certain package configurations, for example with Python 3.9
+ when `defusedxml <https://github.com/tiran/defusedxml/>`_ is installed, and
+ is anticipated to be unusable in the future.
+
+ Attempting to use the the ``xlwt`` engine will raise a ``FutureWarning``
+ unless the option :attr:`io.excel.xls.writer` is set to ``"xlwt"``.
+ While this option is now deprecated and will also raise a ``FutureWarning``,
+ it can be globally set and the warning suppressed. Users are recommended to
+ write ``.xlsx`` files using the ``openpyxl`` engine instead.
 
 .. ---------------------------------------------------------------------------
 
@@ -260,6 +275,7 @@ Other enhancements
 - Added :meth:`~DataFrame.set_flags` for setting table-wide flags on a Series or DataFrame (:issue:`28394`)
 - :meth:`DataFrame.applymap` now supports ``na_action`` (:issue:`23803`)
 - :class:`Index` with object dtype supports division and multiplication (:issue:`34160`)
+- :meth:`io.sql.get_schema` now supports a ``schema`` keyword argument that will add a schema into the create table statement (:issue:`28486`)
 - :meth:`DataFrame.explode` and :meth:`Series.explode` now support exploding of sets (:issue:`35614`)
 - :meth:`DataFrame.hist` now supports time series (datetime) data (:issue:`32590`)
 - :meth:`.Styler.set_table_styles` now allows the direct styling of rows and columns and can be chained (:issue:`35607`)
@@ -479,7 +495,7 @@ Optional libraries below the lowest tested version may still work, but are not c
 
 See :ref:`install.dependencies` and :ref:`install.optional_dependencies` for more.
 
-.. _whatsnew_200.api.other:
+.. _whatsnew_120.api.other:
 
 Other API changes
 ^^^^^^^^^^^^^^^^^
@@ -850,4 +866,4 @@ Other
 Contributors
 ~~~~~~~~~~~~
 
-.. contributors:: v1.1.4..v1.2.0|HEAD
+.. contributors:: v1.1.5..v1.2.0|HEAD

doc/source/whatsnew/v1.3.0.rst

@@ -0,0 +1,199 @@
+.. _whatsnew_130:
+
+What's new in 1.3.0 (??)
+------------------------
+
+These are the changes in pandas 1.3.0. See :ref:`release` for a full changelog
+including other versions of pandas.
+
+{{ header }}
+
+.. ---------------------------------------------------------------------------
+
+Enhancements
+~~~~~~~~~~~~
+
+
+.. _whatsnew_130.enhancements.other:
+
+Other enhancements
+^^^^^^^^^^^^^^^^^^
+
+-
+-
+
+.. ---------------------------------------------------------------------------
+
+.. _whatsnew_130.notable_bug_fixes:
+
+Notable bug fixes
+~~~~~~~~~~~~~~~~~
+
+These are bug fixes that might have notable behavior changes.
+
+
+
+.. _whatsnew_130.api_breaking.deps:
+
+Increased minimum versions for dependencies
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+
+
+.. _whatsnew_130.api.other:
+
+Other API changes
+^^^^^^^^^^^^^^^^^
+
+-
+-
+
+.. ---------------------------------------------------------------------------
+
+.. _whatsnew_130.deprecations:
+
+Deprecations
+~~~~~~~~~~~~
+
+-
+-
+
+.. ---------------------------------------------------------------------------
+
+
+.. _whatsnew_130.performance:
+
+Performance improvements
+~~~~~~~~~~~~~~~~~~~~~~~~
+
+-
+-
+
+.. ---------------------------------------------------------------------------
+
+.. _whatsnew_130.bug_fixes:
+
+Bug fixes
+~~~~~~~~~
+
+-
+-
+
+Categorical
+^^^^^^^^^^^
+
+-
+-
+
+Datetimelike
+^^^^^^^^^^^^
+
+-
+-
+
+Timedelta
+^^^^^^^^^
+
+-
+-
+
+Timezones
+^^^^^^^^^
+
+-
+-
+
+Numeric
+^^^^^^^
+
+-
+-
+
+Conversion
+^^^^^^^^^^
+-
+-
+
+Strings
+^^^^^^^
+
+-
+-
+
+Interval
+^^^^^^^^
+
+-
+-
+
+Indexing
+^^^^^^^^
+
+-
+-
+
+Missing
+^^^^^^^
+
+-
+-
+
+MultiIndex
+^^^^^^^^^^
+
+-
+-
+
+I/O
+^^^
+
+-
+-
+
+Period
+^^^^^^
+
+-
+-
+
+Plotting
+^^^^^^^^
+
+-
+-
+
+Groupby/resample/rolling
+^^^^^^^^^^^^^^^^^^^^^^^^
+
+-
+-
+
+Reshaping
+^^^^^^^^^
+
+-
+-
+
+Sparse
+^^^^^^
+
+-
+-
+
+ExtensionArray
+^^^^^^^^^^^^^^
+
+-
+-
+
+Other
+^^^^^
+
+-
+-
+
+.. ---------------------------------------------------------------------------
+
+.. _whatsnew_130.contributors:
+
+Contributors
+~~~~~~~~~~~~

pandas/core/algorithms.py

@@ -440,8 +440,7 @@ def isin(comps: AnyArrayLike, values: AnyArrayLike) -> np.ndarray:
  if not isinstance(
  values, (ABCIndexClass, ABCSeries, ABCExtensionArray, np.ndarray)
  ):
- values = construct_1d_object_array_from_listlike(list(values))
- # TODO: could use ensure_arraylike here
+ values = _ensure_arraylike(list(values))
  elif isinstance(values, ABCMultiIndex):
  # Avoid raising in extract_array
  values = np.array(values)

pandas/core/config_init.py

@@ -581,6 +581,13 @@ def use_inf_as_na_cb(key):
  writer_engine_doc.format(ext="xls", others=", ".join(_xls_options)),
  validator=str,
  )
+cf.deprecate_option(
+ "io.excel.xls.writer",
+ msg="As the xlwt package is no longer maintained, the xlwt engine will be "
+ "removed in a future version of pandas. This is the only engine in pandas that "
+ "supports writing in the xls format. Install openpyxl and write to an "
+ "xlsx file instead.",
+)
 
 with cf.config_prefix("io.excel.xlsm"):
  cf.register_option(

pandas/core/dtypes/cast.py

@@ -241,10 +241,6 @@ def maybe_downcast_numeric(result, dtype: DtypeObj, do_round: bool = False):
  # e.g. SparseDtype has no itemsize attr
  return result
 
- if isinstance(result, list):
- # reached via groupby.agg._ohlc; really this should be handled earlier
- result = np.array(result)
-
  def trans(x):
  if do_round:
  return x.round()
@@ -357,12 +353,18 @@ def maybe_cast_result_dtype(dtype: DtypeObj, how: str) -> DtypeObj:
  The desired dtype of the result.
  """
  from pandas.core.arrays.boolean import BooleanDtype
- from pandas.core.arrays.integer import Int64Dtype
-
- if how in ["add", "cumsum", "sum"] and (dtype == np.dtype(bool)):
- return np.dtype(np.int64)
- elif how in ["add", "cumsum", "sum"] and isinstance(dtype, BooleanDtype):
- return Int64Dtype()
+ from pandas.core.arrays.floating import Float64Dtype
+ from pandas.core.arrays.integer import Int64Dtype, _IntegerDtype
+
+ if how in ["add", "cumsum", "sum", "prod"]:
+ if dtype == np.dtype(bool):
+ return np.dtype(np.int64)
+ elif isinstance(dtype, (BooleanDtype, _IntegerDtype)):
+ return Int64Dtype()
+ elif how in ["mean", "median", "var"] and isinstance(
+ dtype, (BooleanDtype, _IntegerDtype)
+ ):
+ return Float64Dtype()
  return dtype