String annotations [PEP 563] (#4390)

* Document `from __future__ import annotations` * Provide plumbing and tests for `from __future__ import annotations` * Implement unparsing the AST back to string form This is required for PEP 563 and as such only implements a part of the unparsing process that covers expressions.

Author: gvanrossum

Doc/glossary.rst

@@ -372,9 +372,11 @@ Glossary
  may be accessed via the :attr:`__annotations__` special attribute of a
  function object.
 
- Python itself does not assign any particular meaning to function
- annotations. They are intended to be interpreted by third-party libraries
- or tools. See :pep:`3107`, which describes some of their potential uses.
+ See also the :term:`variable annotation` glossary entry.
+
+ Annotations are meant to provide a standard way for programmers to
+ document types of functions they design. See :pep:`484`, which
+ describes this functionality.
 
  __future__
  A pseudo-module which programmers can use to enable new language features
@@ -1021,10 +1023,11 @@ Glossary
  attribute of a class or module object and can be accessed using
  :func:`typing.get_type_hints`.
 
- Python itself does not assign any particular meaning to variable
- annotations. They are intended to be interpreted by third-party libraries
- or type checking tools. See :pep:`526`, :pep:`484` which describe
- some of their potential uses.
+ See also the :term:`function annotation` glossary entry.
+
+ Annotations are meant to provide a standard way for programmers to
+ document types of functions they design. See :pep:`484` and :pep:`526`
+ which describe this functionality.
 
  virtual environment
  A cooperatively isolated runtime environment that allows Python users

Doc/library/__future__.rst

@@ -90,6 +90,11 @@ language using this mechanism:
 | generator_stop | 3.5.0b1 | 3.7 | :pep:`479`: |
 | | | | *StopIteration handling inside generators* |
 +------------------+-------------+--------------+---------------------------------------------+
+| annotations | 3.7.0b1 | 4.0 | :pep:`563`: |
+| | | | *Postponed evaluation of annotations* |
++------------------+-------------+--------------+---------------------------------------------+
+
+.. XXX Adding a new entry? Remember to update simple_stmts.rst, too.
 
 
 .. seealso::

Doc/reference/compound_stmts.rst

@@ -559,12 +559,14 @@ Parameters may have annotations of the form "``: expression``" following the
 parameter name. Any parameter may have an annotation even those of the form
 ``*identifier`` or ``**identifier``. Functions may have "return" annotation of
 the form "``-> expression``" after the parameter list. These annotations can be
-any valid Python expression and are evaluated when the function definition is
-executed. Annotations may be evaluated in a different order than they appear in
-the source code. The presence of annotations does not change the semantics of a
-function. The annotation values are available as values of a dictionary keyed
-by the parameters' names in the :attr:`__annotations__` attribute of the
-function object.
+any valid Python expression. The presence of annotations does not change the
+semantics of a function. The annotation values are available as values of
+a dictionary keyed by the parameters' names in the :attr:`__annotations__`
+attribute of the function object. If the ``annotations`` import from
+:mod:`__future__` is used, annotations are preserved as strings at runtime which
+enables postponed evaluation. Otherwise, they are evaluated when the function
+definition is executed. In this case annotations may be evaluated in
+a different order than they appear in the source code.
 
 .. index:: pair: lambda; expression
 
@@ -587,6 +589,17 @@ access the local variables of the function containing the def. See section
  :pep:`3107` - Function Annotations
  The original specification for function annotations.
 
+ :pep:`484` - Type Hints
+ Definition of a standard meaning for annotations: type hints.
+
+ :pep:`526` - Syntax for Variable Annotations
+ Ability to type hint variable declarations, including class
+ variables and instance variables
+
+ :pep:`563` - Postponed Evaluation of Annotations
+ Support for forward references within annotations by preserving
+ annotations in a string form at runtime instead of eager evaluation.
+
 
 .. _class:
 

Doc/reference/simple_stmts.rst

@@ -853,12 +853,15 @@ can appear before a future statement are:
 * blank lines, and
 * other future statements.
 
-.. XXX change this if future is cleaned out
-
-The features recognized by Python 3.0 are ``absolute_import``, ``division``,
-``generators``, ``unicode_literals``, ``print_function``, ``nested_scopes`` and
-``with_statement``. They are all redundant because they are always enabled, and
-only kept for backwards compatibility.
+The only feature in Python 3.7 that requires using the future statement is
+``annotations``.
+
+All historical features enabled by the future statement are still recognized
+by Python 3. The list includes ``absolute_import``, ``division``,
+``generators``, ``generator_stop``, ``unicode_literals``,
+``print_function``, ``nested_scopes`` and ``with_statement``. They are
+all redundant because they are always enabled, and only kept for
+backwards compatibility.
 
 A future statement is recognized and treated specially at compile time: Changes
 to the semantics of core constructs are often implemented by generating

Doc/whatsnew/3.7.rst

@@ -179,6 +179,57 @@ a normal ``__getattr__`` method, except that it will be defined on module
  PEP written and implemented by Ivan Levkivskyi
 
 
+PEP 563: Postponed evaluation of annotations
+--------------------------------------------
+
+The advent of type hints in Python uncovered two glaring usability issues
+with the functionality of annotations added in :pep:`3107` and refined
+further in :pep:`526`:
+
+* annotations could only use names which were already available in the
+ current scope, in other words they didn't support forward references
+ of any kind; and
+
+* annotating source code had adverse effects on startup time of Python
+ programs.
+
+Both of these issues are fixed by postponing the evaluation of
+annotations. Instead of compiling code which executes expressions in
+annotations at their definition time, the compiler stores the annotation
+in a string form equivalent to the AST of the expression in question.
+If needed, annotations can be resolved at runtime using
+``typing.get_type_hints()``. In the common case where this is not
+required, the annotations are cheaper to store (since short strings
+are interned by the interpreter) and make startup time faster.
+
+Usability-wise, annotations now support forward references, making the
+following syntax valid::
+
+ class C:
+ @classmethod
+ def from_string(cls, source: str) -> C:
+ ...
+
+ def validate_b(self, obj: B) -> bool:
+ ...
+
+ class B:
+ ...
+
+Since this change breaks compatibility, the new behavior can be enabled
+on a per-module basis in Python 3.7 using a ``__future__`` import, like
+this::
+
+ from __future__ import annotations
+
+It will become the default in Python 4.0.
+
+.. seealso::
+
+ :pep:`563` -- Postponed evaluation of annotations
+ PEP written and implemented by Łukasz Langa.
+
+
 PEP 564: Add new time functions with nanosecond resolution
 ----------------------------------------------------------
 

Include/ast.h

@@ -16,6 +16,15 @@ PyAPI_FUNC(mod_ty) PyAST_FromNodeObject(
  PyObject *filename,
  PyArena *arena);
 
+#ifndef Py_LIMITED_API
+
+/* _PyAST_ExprAsUnicode is defined in ast_unparse.c */
+PyAPI_FUNC(PyObject *) _PyAST_ExprAsUnicode(
+ expr_ty e,
+ int omit_parens);
+
+#endif /* !Py_LIMITED_API */
+
 #ifdef __cplusplus
 }
 #endif

Include/code.h

@@ -82,6 +82,7 @@ typedef struct {
 
 #define CO_FUTURE_BARRY_AS_BDFL 0x40000
 #define CO_FUTURE_GENERATOR_STOP 0x80000
+#define CO_FUTURE_ANNOTATIONS 0x100000
 
 /* This value is found in the co_cell2arg array when the associated cell
  variable does not correspond to an argument. */

Include/compile.h

@@ -16,7 +16,7 @@ PyAPI_FUNC(PyCodeObject *) PyNode_Compile(struct _node *, const char *);
 #define PyCF_MASK (CO_FUTURE_DIVISION | CO_FUTURE_ABSOLUTE_IMPORT | \
  CO_FUTURE_WITH_STATEMENT | CO_FUTURE_PRINT_FUNCTION | \
  CO_FUTURE_UNICODE_LITERALS | CO_FUTURE_BARRY_AS_BDFL | \
- CO_FUTURE_GENERATOR_STOP)
+ CO_FUTURE_GENERATOR_STOP | CO_FUTURE_ANNOTATIONS)
 #define PyCF_MASK_OBSOLETE (CO_NESTED)
 #define PyCF_SOURCE_IS_UTF8 0x0100
 #define PyCF_DONT_IMPLY_DEDENT 0x0200
@@ -45,6 +45,7 @@ typedef struct {
 #define FUTURE_UNICODE_LITERALS "unicode_literals"
 #define FUTURE_BARRY_AS_BDFL "barry_as_FLUFL"
 #define FUTURE_GENERATOR_STOP "generator_stop"
+#define FUTURE_ANNOTATIONS "annotations"
 
 struct _mod; /* Declare the existence of this type */
 #define PyAST_Compile(mod, s, f, ar) PyAST_CompileEx(mod, s, f, -1, ar)

Lib/__future__.py

@@ -57,13 +57,14 @@
  "unicode_literals",
  "barry_as_FLUFL",
  "generator_stop",
+ "annotations",
 ]
 
 __all__ = ["all_feature_names"] + all_feature_names
 
-# The CO_xxx symbols are defined here under the same names used by
-# compile.h, so that an editor search will find them here. However,
-# they're not exported in __all__, because they don't really belong to
+# The CO_xxx symbols are defined here under the same names defined in
+# code.h and used by compile.h, so that an editor search will find them here.
+# However, they're not exported in __all__, because they don't really belong to
 # this module.
 CO_NESTED = 0x0010 # nested_scopes
 CO_GENERATOR_ALLOWED = 0 # generators (obsolete, was 0x1000)
@@ -74,6 +75,7 @@
 CO_FUTURE_UNICODE_LITERALS = 0x20000 # unicode string literals
 CO_FUTURE_BARRY_AS_BDFL = 0x40000
 CO_FUTURE_GENERATOR_STOP = 0x80000 # StopIteration becomes RuntimeError in generators
+CO_FUTURE_ANNOTATIONS = 0x100000 # annotations become strings at runtime
 
 class _Feature:
  def __init__(self, optionalRelease, mandatoryRelease, compiler_flag):
@@ -132,9 +134,13 @@ def __repr__(self):
  CO_FUTURE_UNICODE_LITERALS)
 
 barry_as_FLUFL = _Feature((3, 1, 0, "alpha", 2),
- (3, 9, 0, "alpha", 0),
- CO_FUTURE_BARRY_AS_BDFL)
+  (3, 9, 0, "alpha", 0),
+  CO_FUTURE_BARRY_AS_BDFL)
 
 generator_stop = _Feature((3, 5, 0, "beta", 1),
- (3, 7, 0, "alpha", 0),
- CO_FUTURE_GENERATOR_STOP)
+ (3, 7, 0, "alpha", 0),
+ CO_FUTURE_GENERATOR_STOP)
+
+annotations = _Feature((3, 7, 0, "beta", 1),
+ (4, 0, 0, "alpha", 0),
+ CO_FUTURE_ANNOTATIONS)