Enable Conversions Between Native Python Enum Types and C++ Enums (#5555)

* Apply smart_holder-branch-based PR #5280 on top of master. * Add pytest.skip("GraalPy does not raise UnicodeDecodeError") * Add `parent_scope` as first argument to `py::native_enum` ctor. * Replace `operator+=` API with `.finalize()` API. The error messages still need cleanup. * Resolve clang-tidy performance-unnecessary-value-param errors * Rename (effectively) native_enum_add_to_parent() -> finalize() * Update error message: pybind11::native_enum<...>("Fake", ...): MISSING .finalize() * Pass py::module_ by reference to resolve clang-tidy errors (this is entirely inconsequential otherwise for all practical purposes). * test_native_enum_correct_use_failure -> test_native_enum_missing_finalize_failure * Add test_native_enum_double_finalize(), test_native_enum_value_after_finalize() * Clean up public/protected API. * [ci skip] Update the Enumerations section in classes.rst * Rename `py::native_enum_kind` → `py::enum_kind` as suggested by gh-henryiii: #5555 (comment) * Experiment: StrEnum enum.StrEnum does not map to C++ enum: * https://chatgpt.com/share/67d5e965-ccb0-8008-95b7-0df2502309b3 ``` ============================= test session starts ============================== platform linux -- Python 3.12.3, pytest-8.3.3, pluggy-1.5.0 C++ Info: 13.3.0 C++20 __pybind11_internals_v10000000_system_libstdcpp_gxx_abi_1xxx_use_cxx11_abi_1__ PYBIND11_SIMPLE_GIL_MANAGEMENT=False PYBIND11_NUMPY_1_ONLY=False configfile: pytest.ini plugins: parallel-0.1.1, xdist-3.6.1 collected 40 items / 39 deselected / 1 selected test_native_enum.py F [100%] =================================== FAILURES =================================== ________________________ test_native_enum_StrEnum_greek ________________________ def test_native_enum_StrEnum_greek(): assert not hasattr(m, "greek") > m.native_enum_StrEnum_greek(m) test_native_enum.py:150: _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ /usr/lib/python3.12/enum.py:764: in __call__ return cls._create_( boundary = None cls = <enum 'StrEnum'> module = None names = [('Alpha', 10), ('Omega', 20)] qualname = None start = 1 type = None value = 'greek' values = () /usr/lib/python3.12/enum.py:917: in _create_ return metacls.__new__(metacls, class_name, bases, classdict, boundary=boundary) _ = <class 'str'> bases = (<enum 'StrEnum'>,) boundary = None class_name = 'greek' classdict = {'_generate_next_value_': <function StrEnum._generate_next_value_ at 0x701ec1711e40>, 'Alpha': 10, 'Omega': 20, '__module__': 'test_native_enum'} cls = <enum 'StrEnum'> first_enum = <enum 'StrEnum'> item = ('Omega', 20) member_name = 'Omega' member_value = 20 metacls = <class 'enum.EnumType'> module = 'test_native_enum' names = [('Alpha', 10), ('Omega', 20)] qualname = None start = 1 type = None /usr/lib/python3.12/enum.py:606: in __new__ raise exc.with_traceback(tb) __class__ = <class 'enum.EnumType'> __new__ = <function StrEnum.__new__ at 0x701ec1711da0> _gnv = <staticmethod(<function StrEnum._generate_next_value_ at 0x701ec1711e40>)> _order_ = None _simple = False bases = (<enum 'StrEnum'>,) boundary = None classdict = {'Alpha': <enum._proto_member object at 0x701ebc74f9b0>, 'Omega': <enum._proto_member object at 0x701ebc74cce0>, '__module__': 'test_native_enum', '_all_bits_': 0, ...} cls = 'greek' exc = TypeError('10 is not a string') first_enum = <enum 'StrEnum'> ignore = ['_ignore_'] invalid_names = set() key = '_ignore_' kwds = {} member_names = {'Alpha': None, 'Omega': None} member_type = <class 'str'> metacls = <class 'enum.EnumType'> name = 'Omega' save_new = False tb = <traceback object at 0x701ebc7a6cc0> use_args = True value = 20 /usr/lib/python3.12/enum.py:596: in __new__ enum_class = super().__new__(metacls, cls, bases, classdict, **kwds) __class__ = <class 'enum.EnumType'> __new__ = <function StrEnum.__new__ at 0x701ec1711da0> _gnv = <staticmethod(<function StrEnum._generate_next_value_ at 0x701ec1711e40>)> _order_ = None _simple = False bases = (<enum 'StrEnum'>,) boundary = None classdict = {'Alpha': <enum._proto_member object at 0x701ebc74f9b0>, 'Omega': <enum._proto_member object at 0x701ebc74cce0>, '__module__': 'test_native_enum', '_all_bits_': 0, ...} cls = 'greek' exc = TypeError('10 is not a string') first_enum = <enum 'StrEnum'> ignore = ['_ignore_'] invalid_names = set() key = '_ignore_' kwds = {} member_names = {'Alpha': None, 'Omega': None} member_type = <class 'str'> metacls = <class 'enum.EnumType'> name = 'Omega' save_new = False tb = <traceback object at 0x701ebc7a6cc0> use_args = True value = 20 /usr/lib/python3.12/enum.py:271: in __set_name__ enum_member = enum_class._new_member_(enum_class, *args) args = (10,) enum_class = <enum 'greek'> member_name = 'Alpha' self = <enum._proto_member object at 0x701ebc74f9b0> value = 10 _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ cls = <enum 'greek'>, values = (10,) def __new__(cls, *values): "values must already be of type `str`" if len(values) > 3: raise TypeError('too many arguments for str(): %r' % (values, )) if len(values) == 1: # it must be a string if not isinstance(values[0], str): > raise TypeError('%r is not a string' % (values[0], )) E TypeError: 10 is not a string cls = <enum 'greek'> values = (10,) /usr/lib/python3.12/enum.py:1322: TypeError =========================== short test summary info ============================ FAILED test_native_enum.py::test_native_enum_StrEnum_greek - TypeError: 10 is... ======================= 1 failed, 39 deselected in 0.07s ======================= ERROR: completed_process.returncode=1 ``` * Remove StrEnum code. * Make enum_kind::Enum the default kind. * Catch redundant .export_values() calls. * [ci skip] Add back original documentation for `py::enum_` under new advanced/deprecated.rst * [ci skip] Add documentation for `py::enum_kind` and `py::detail::type_caster_enum_type_enabled` * Rename `Type` to `EnumType` for readability. * Eliminate py::enum_kind, use "enum.Enum", "enum.IntEnum" directly. This is still WIP. * EXPERIMENTAL StrEnum code. To be removed. * Remove experimental StrEnum code: My judgement: Supporting StrEnum is maybe nice, but not very valuable. I don't think it is worth the extra C++ code. A level of indirection would need to be managed, e.g. RED ↔ Python "r" ↔ C++ 0 Green ↔ Python "g" ↔ C++ 1 These mappings would need to be stored and processed. * Add test with enum.IntFlag (no production code changes required). * First import_or_getattr() implementation (dedicated tests are still missing). * Fix import_or_getattr() implementation, add tests, fix clang-tidy errors. * [ci skip] Update classes.rst: replace `py::enum_kind` with `native_type_name` * For "constructor similar to that of enum.Enum" point to https://docs.python.org/3/howto/enum.html#functional-api, as suggested by gh-timohl (#5555 (comment)). * Advertise Enum, IntEnum, Flag, IntFlags are compatible stdlib enum types in the documentation (as suggested by gh-timohl, #5555 (review)); add test for enum.Flag to ensure that is actually true.

Author: rwgk

CMakeLists.txt

@@ -136,6 +136,7 @@ set(PYBIND11_HEADERS
  include/pybind11/detail/dynamic_raw_ptr_cast_if_possible.h
  include/pybind11/detail/init.h
  include/pybind11/detail/internals.h
+ include/pybind11/detail/native_enum_data.h
  include/pybind11/detail/struct_smart_holder.h
  include/pybind11/detail/type_caster_base.h
  include/pybind11/detail/typeid.h
@@ -162,6 +163,7 @@ set(PYBIND11_HEADERS
  include/pybind11/gil_safe_call_once.h
  include/pybind11/iostream.h
  include/pybind11/functional.h
+ include/pybind11/native_enum.h
  include/pybind11/numpy.h
  include/pybind11/operators.h
  include/pybind11/pybind11.h

docs/advanced/cast/custom.rst

@@ -1,3 +1,5 @@
+.. _custom_type_caster:
+
 Custom type casters
 ===================
 

docs/advanced/deprecated.rst

@@ -0,0 +1,113 @@
+Deprecated
+##########
+
+.. _deprecated_enum:
+
+``py::enum_``
+=============
+
+This is the original documentation for ``py::enum_``, which is deprecated
+because it is not `PEP 435 compatible <https://peps.python.org/pep-0435/>`_
+(see also `#2332 <https://github.com/pybind/pybind11/issues/2332>`_).
+Please prefer ``py::native_enum`` (added with pybind11v3) when writing
+new bindings. See :ref:`native_enum` for more information.
+
+Let's suppose that we have an example class that contains internal types
+like enumerations, e.g.:
+
+.. code-block:: cpp
+
+ struct Pet {
+ enum Kind {
+ Dog = 0,
+ Cat
+ };
+
+ struct Attributes {
+ float age = 0;
+ };
+
+ Pet(const std::string &name, Kind type) : name(name), type(type) { }
+
+ std::string name;
+ Kind type;
+ Attributes attr;
+ };
+
+The binding code for this example looks as follows:
+
+.. code-block:: cpp
+
+ py::class_<Pet> pet(m, "Pet");
+
+ pet.def(py::init<const std::string &, Pet::Kind>())
+ .def_readwrite("name", &Pet::name)
+ .def_readwrite("type", &Pet::type)
+ .def_readwrite("attr", &Pet::attr);
+
+ py::enum_<Pet::Kind>(pet, "Kind")
+ .value("Dog", Pet::Kind::Dog)
+ .value("Cat", Pet::Kind::Cat)
+ .export_values();
+
+ py::class_<Pet::Attributes>(pet, "Attributes")
+ .def(py::init<>())
+ .def_readwrite("age", &Pet::Attributes::age);
+
+
+To ensure that the nested types ``Kind`` and ``Attributes`` are created within the scope of ``Pet``, the
+``pet`` ``py::class_`` instance must be supplied to the :class:`enum_` and ``py::class_``
+constructor. The :func:`enum_::export_values` function exports the enum entries
+into the parent scope, which should be skipped for newer C++11-style strongly
+typed enums.
+
+.. code-block:: pycon
+
+ >>> p = Pet("Lucy", Pet.Cat)
+ >>> p.type
+ Kind.Cat
+ >>> int(p.type)
+ 1L
+
+The entries defined by the enumeration type are exposed in the ``__members__`` property:
+
+.. code-block:: pycon
+
+ >>> Pet.Kind.__members__
+ {'Dog': Kind.Dog, 'Cat': Kind.Cat}
+
+The ``name`` property returns the name of the enum value as a unicode string.
+
+.. note::
+
+ It is also possible to use ``str(enum)``, however these accomplish different
+ goals. The following shows how these two approaches differ.
+
+ .. code-block:: pycon
+
+ >>> p = Pet("Lucy", Pet.Cat)
+ >>> pet_type = p.type
+ >>> pet_type
+ Pet.Cat
+ >>> str(pet_type)
+ 'Pet.Cat'
+ >>> pet_type.name
+ 'Cat'
+
+.. note::
+
+ When the special tag ``py::arithmetic()`` is specified to the ``enum_``
+ constructor, pybind11 creates an enumeration that also supports rudimentary
+ arithmetic and bit-level operations like comparisons, and, or, xor, negation,
+ etc.
+
+ .. code-block:: cpp
+
+ py::enum_<Pet::Kind>(pet, "Kind", py::arithmetic())
+ ...
+
+ By default, these are omitted to conserve space.
+
+.. warning::
+
+ Contrary to Python customs, enum values from the wrappers should not be compared using ``is``, but with ``==`` (see `#1177 <https://github.com/pybind/pybind11/issues/1177>`_ for background).

docs/classes.rst

@@ -459,6 +459,8 @@ you can use ``py::detail::overload_cast_impl`` with an additional set of parenth
  other using the ``.def(py::init<...>())`` syntax. The existing machinery
  for specifying keyword and default arguments also works.
 
+.. _native_enum:
+
 Enumerations and internal types
 ===============================
 
@@ -487,76 +489,102 @@ The binding code for this example looks as follows:
 
 .. code-block:: cpp
 
+ #include <pybind11/native_enum.h> // Not already included with pybind11.h
+
  py::class_<Pet> pet(m, "Pet");
 
  pet.def(py::init<const std::string &, Pet::Kind>())
  .def_readwrite("name", &Pet::name)
  .def_readwrite("type", &Pet::type)
  .def_readwrite("attr", &Pet::attr);
 
- py::enum_<Pet::Kind>(pet, "Kind")
+ py::native_enum<Pet::Kind>(pet, "Kind")
  .value("Dog", Pet::Kind::Dog)
  .value("Cat", Pet::Kind::Cat)
- .export_values();
+ .export_values()
+ .finalize();
 
  py::class_<Pet::Attributes>(pet, "Attributes")
  .def(py::init<>())
  .def_readwrite("age", &Pet::Attributes::age);
 
 
-To ensure that the nested types ``Kind`` and ``Attributes`` are created within the scope of ``Pet``, the
-``pet`` ``py::class_`` instance must be supplied to the :class:`enum_` and ``py::class_``
-constructor. The :func:`enum_::export_values` function exports the enum entries
-into the parent scope, which should be skipped for newer C++11-style strongly
-typed enums.
+To ensure that the nested types ``Kind`` and ``Attributes`` are created
+within the scope of ``Pet``, the ``pet`` ``py::class_`` instance must be
+supplied to the ``py::native_enum`` and ``py::class_`` constructors. The
+``.export_values()`` function is available for exporting the enum entries
+into the parent scope, if desired.
 
-.. code-block:: pycon
+``py::native_enum`` was introduced with pybind11v3. It binds C++ enum types
+to native Python enum types, typically types in Python's
+`stdlib enum <https://docs.python.org/3/library/enum.html>`_ module,
+which are `PEP 435 compatible <https://peps.python.org/pep-0435/>`_.
+This is the recommended way to bind C++ enums.
+The older ``py::enum_`` is not PEP 435 compatible
+(see `issue #2332 <https://github.com/pybind/pybind11/issues/2332>`_)
+but remains supported indefinitely for backward compatibility.
+New bindings should prefer ``py::native_enum``.
 
- >>> p = Pet("Lucy", Pet.Cat)
- >>> p.type
- Kind.Cat
- >>> int(p.type)
- 1L
+.. note::
 
-The entries defined by the enumeration type are exposed in the ``__members__`` property:
+ The deprecated ``py::enum_`` is :ref:`documented here <deprecated_enum>`.
 
-.. code-block:: pycon
+The ``.finalize()`` call above is needed because Python's native enums
+cannot be built incrementally — all name/value pairs need to be passed at
+once. To achieve this, ``py::native_enum`` acts as a buffer to collect the
+name/value pairs. The ``.finalize()`` call uses the accumulated name/value
+pairs to build the arguments for constructing a native Python enum type.
 
- >>> Pet.Kind.__members__
- {'Dog': Kind.Dog, 'Cat': Kind.Cat}
+The ``py::native_enum`` constructor supports a third optional
+``native_type_name`` string argument, with default value ``"enum.Enum"``.
+Other types can be specified like this:
 
-The ``name`` property returns the name of the enum value as a unicode string.
+.. code-block:: cpp
 
-.. note::
+ py::native_enum<Pet::Kind>(pet, "Kind", "enum.IntEnum")
 
- It is also possible to use ``str(enum)``, however these accomplish different
- goals. The following shows how these two approaches differ.
+Any fully-qualified Python name can be specified. The only requirement is
+that the named type is similar to
+`enum.Enum <https://docs.python.org/3/library/enum.html#enum.Enum>`_
+in these ways:
 
- .. code-block:: pycon
+* Has a `constructor similar to that of enum.Enum
+ <https://docs.python.org/3/howto/enum.html#functional-api>`_::
 
- >>> p = Pet("Lucy", Pet.Cat)
- >>> pet_type = p.type
- >>> pet_type
- Pet.Cat
- >>> str(pet_type)
- 'Pet.Cat'
- >>> pet_type.name
- 'Cat'
+ Colors = enum.Enum("Colors", (("Red", 0), ("Green", 1)))
+
+* A `C++ underlying <https://en.cppreference.com/w/cpp/types/underlying_type>`_
+ enum value can be passed to the constructor for the Python enum value::
+
+ red = Colors(0)
+
+* The enum values have a ``.value`` property yielding a value that
+ can be cast to the C++ underlying type::
+
+ underlying = red.value
+
+As of Python 3.13, the compatible `types in the stdlib enum module
+<https://docs.python.org/3/library/enum.html#module-contents>`_ are:
+``Enum``, ``IntEnum``, ``Flag``, ``IntFlag``.
 
 .. note::
 
- When the special tag ``py::arithmetic()`` is specified to the ``enum_``
- constructor, pybind11 creates an enumeration that also supports rudimentary
- arithmetic and bit-level operations like comparisons, and, or, xor, negation,
- etc.
+ In rare cases, a C++ enum may be bound to Python via a
+ :ref:`custom type caster <custom_type_caster>`. In such cases, a
+ template specialization like this may be required:
 
  .. code-block:: cpp
 
- py::enum_<Pet::Kind>(pet, "Kind", py::arithmetic())
- ...
-
- By default, these are omitted to conserve space.
+ #if defined(PYBIND11_HAS_NATIVE_ENUM)
+ namespace pybind11::detail {
+ template <typename FancyEnum>
+ struct type_caster_enum_type_enabled<
+ FancyEnum,
+ std::enable_if_t<is_fancy_enum<FancyEnum>::value>> : std::false_type {};
+ }
+ #endif
 
-.. warning::
+ This specialization is needed only if the custom type caster is templated.
 
- Contrary to Python customs, enum values from the wrappers should not be compared using ``is``, but with ``==`` (see `#1177 <https://github.com/pybind/pybind11/issues/1177>`_ for background).
+ The ``PYBIND11_HAS_NATIVE_ENUM`` guard is needed only if backward
+ compatibility with pybind11v2 is required.

docs/index.rst

@@ -36,6 +36,7 @@
  advanced/pycpp/index
  advanced/embedding
  advanced/misc
+ advanced/deprecated
 
 .. toctree::
  :caption: Extra Information