traverseda
/
cpython
1
0
Fork 0
Code Issues Pull Requests Releases Wiki Activity

String annotations [PEP 563] (#4390) 

* Document `from __future__ import annotations`
* Provide plumbing and tests for `from __future__ import annotations`
* Implement unparsing the AST back to string form

This is required for PEP 563 and as such only implements a part of the
unparsing process that covers expressions.
This commit is contained in:
Guido van Rossum 2018-01-26 08:20:18 -08:00 committed by Łukasz Langa
parent d7773d92bd
commit 95e4d58913
16 changed files with 1476 additions and 29 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

17
Doc/glossary.rst
View File

@ -372,9 +372,11 @@ Glossary
may be accessed via the :attr:`__annotations__` special attribute of a
function object.
Python itself does not assign any particular meaning to function
annotations. They are intended to be interpreted by third-party libraries
or tools. See :pep:`3107`, which describes some of their potential uses.
See also the :term:`variable annotation` glossary entry.
Annotations are meant to provide a standard way for programmers to
document types of functions they design. See :pep:`484`, which
describes this functionality.
__future__
A pseudo-module which programmers can use to enable new language features
@ -1021,10 +1023,11 @@ Glossary
attribute of a class or module object and can be accessed using
:func:`typing.get_type_hints`.
Python itself does not assign any particular meaning to variable
annotations. They are intended to be interpreted by third-party libraries
or type checking tools. See :pep:`526`, :pep:`484` which describe
some of their potential uses.
See also the :term:`function annotation` glossary entry.
Annotations are meant to provide a standard way for programmers to
document types of functions they design. See :pep:`484` and :pep:`526`
which describe this functionality.
virtual environment
A cooperatively isolated runtime environment that allows Python users

5
Doc/library/__future__.rst
View File

@ -90,6 +90,11 @@ language using this mechanism:
| generator_stop | 3.5.0b1 | 3.7 | :pep:`479`: |
| | | | *StopIteration handling inside generators* |
+------------------+-------------+--------------+---------------------------------------------+
| annotations | 3.7.0b1 | 4.0 | :pep:`563`: |
| | | | *Postponed evaluation of annotations* |
+------------------+-------------+--------------+---------------------------------------------+
.. XXX Adding a new entry? Remember to update simple_stmts.rst, too.
.. seealso::

25
Doc/reference/compound_stmts.rst
View File

@ -559,12 +559,14 @@ Parameters may have annotations of the form "``: expression``" following the
parameter name. Any parameter may have an annotation even those of the form
``*identifier`` or ``**identifier``. Functions may have "return" annotation of
the form "``-> expression``" after the parameter list. These annotations can be
any valid Python expression and are evaluated when the function definition is
executed. Annotations may be evaluated in a different order than they appear in
the source code. The presence of annotations does not change the semantics of a
function. The annotation values are available as values of a dictionary keyed
by the parameters' names in the :attr:`__annotations__` attribute of the
function object.
any valid Python expression. The presence of annotations does not change the
semantics of a function. The annotation values are available as values of
a dictionary keyed by the parameters' names in the :attr:`__annotations__`
attribute of the function object. If the ``annotations`` import from
:mod:`__future__` is used, annotations are preserved as strings at runtime which
enables postponed evaluation. Otherwise, they are evaluated when the function
definition is executed. In this case annotations may be evaluated in
a different order than they appear in the source code.
.. index:: pair: lambda; expression
@ -587,6 +589,17 @@ access the local variables of the function containing the def. See section
:pep:`3107` - Function Annotations
The original specification for function annotations.
:pep:`484` - Type Hints
Definition of a standard meaning for annotations: type hints.
:pep:`526` - Syntax for Variable Annotations
Ability to type hint variable declarations, including class
variables and instance variables
:pep:`563` - Postponed Evaluation of Annotations
Support for forward references within annotations by preserving
annotations in a string form at runtime instead of eager evaluation.
.. _class:

13
Doc/reference/simple_stmts.rst
View File

@ -853,12 +853,15 @@ can appear before a future statement are:
* blank lines, and
* other future statements.
.. XXX change this if future is cleaned out
The only feature in Python 3.7 that requires using the future statement is
``annotations``.
The features recognized by Python 3.0 are ``absolute_import``, ``division``,
``generators``, ``unicode_literals``, ``print_function``, ``nested_scopes`` and
``with_statement``. They are all redundant because they are always enabled, and
only kept for backwards compatibility.
All historical features enabled by the future statement are still recognized
by Python 3. The list includes ``absolute_import``, ``division``,
``generators``, ``generator_stop``, ``unicode_literals``,
``print_function``, ``nested_scopes`` and ``with_statement``. They are
all redundant because they are always enabled, and only kept for
backwards compatibility.
A future statement is recognized and treated specially at compile time: Changes
to the semantics of core constructs are often implemented by generating

51
Doc/whatsnew/3.7.rst
View File

@ -179,6 +179,57 @@ a normal ``__getattr__`` method, except that it will be defined on module
PEP written and implemented by Ivan Levkivskyi
PEP 563: Postponed evaluation of annotations
--------------------------------------------
The advent of type hints in Python uncovered two glaring usability issues
with the functionality of annotations added in :pep:`3107` and refined
further in :pep:`526`:
* annotations could only use names which were already available in the
current scope, in other words they didn't support forward references
of any kind; and
* annotating source code had adverse effects on startup time of Python
programs.
Both of these issues are fixed by postponing the evaluation of
annotations. Instead of compiling code which executes expressions in
annotations at their definition time, the compiler stores the annotation
in a string form equivalent to the AST of the expression in question.
If needed, annotations can be resolved at runtime using
``typing.get_type_hints()``. In the common case where this is not
required, the annotations are cheaper to store (since short strings
are interned by the interpreter) and make startup time faster.
Usability-wise, annotations now support forward references, making the
following syntax valid::
class C:
@classmethod
def from_string(cls, source: str) -> C:
...
def validate_b(self, obj: B) -> bool:
...
class B:
...
Since this change breaks compatibility, the new behavior can be enabled
on a per-module basis in Python 3.7 using a ``__future__`` import, like
this::
from __future__ import annotations
It will become the default in Python 4.0.
.. seealso::
:pep:`563` -- Postponed evaluation of annotations
PEP written and implemented by Łukasz Langa.
PEP 564: Add new time functions with nanosecond resolution
----------------------------------------------------------

9
Include/ast.h
View File

@ -16,6 +16,15 @@ PyAPI_FUNC(mod_ty) PyAST_FromNodeObject(
PyObject *filename,
PyArena *arena);
#ifndef Py_LIMITED_API
/* _PyAST_ExprAsUnicode is defined in ast_unparse.c */
PyAPI_FUNC(PyObject *) _PyAST_ExprAsUnicode(
expr_ty e,
int omit_parens);
#endif /* !Py_LIMITED_API */
#ifdef __cplusplus
}
#endif

1
Include/code.h
View File

@ -82,6 +82,7 @@ typedef struct {
#define CO_FUTURE_BARRY_AS_BDFL 0x40000
#define CO_FUTURE_GENERATOR_STOP 0x80000
#define CO_FUTURE_ANNOTATIONS 0x100000
/* This value is found in the co_cell2arg array when the associated cell
variable does not correspond to an argument. */

3
Include/compile.h
View File

@ -16,7 +16,7 @@ PyAPI_FUNC(PyCodeObject *) PyNode_Compile(struct _node *, const char *);
#define PyCF_MASK (CO_FUTURE_DIVISION | CO_FUTURE_ABSOLUTE_IMPORT | \
CO_FUTURE_WITH_STATEMENT | CO_FUTURE_PRINT_FUNCTION | \
CO_FUTURE_UNICODE_LITERALS | CO_FUTURE_BARRY_AS_BDFL | \
CO_FUTURE_GENERATOR_STOP)
CO_FUTURE_GENERATOR_STOP | CO_FUTURE_ANNOTATIONS)
#define PyCF_MASK_OBSOLETE (CO_NESTED)
#define PyCF_SOURCE_IS_UTF8 0x0100
#define PyCF_DONT_IMPLY_DEDENT 0x0200
@ -45,6 +45,7 @@ typedef struct {
#define FUTURE_UNICODE_LITERALS "unicode_literals"
#define FUTURE_BARRY_AS_BDFL "barry_as_FLUFL"
#define FUTURE_GENERATOR_STOP "generator_stop"
#define FUTURE_ANNOTATIONS "annotations"
struct _mod; /* Declare the existence of this type */
#define PyAST_Compile(mod, s, f, ar) PyAST_CompileEx(mod, s, f, -1, ar)

20
Lib/__future__.py
View File

@ -57,13 +57,14 @@ all_feature_names = [
"unicode_literals",
"barry_as_FLUFL",
"generator_stop",
"annotations",
]
__all__ = ["all_feature_names"] + all_feature_names
# The CO_xxx symbols are defined here under the same names used by
# compile.h, so that an editor search will find them here. However,
# they're not exported in __all__, because they don't really belong to
# The CO_xxx symbols are defined here under the same names defined in
# code.h and used by compile.h, so that an editor search will find them here.
# However, they're not exported in __all__, because they don't really belong to
# this module.
CO_NESTED = 0x0010 # nested_scopes
CO_GENERATOR_ALLOWED = 0 # generators (obsolete, was 0x1000)
@ -74,6 +75,7 @@ CO_FUTURE_PRINT_FUNCTION = 0x10000 # print function
CO_FUTURE_UNICODE_LITERALS = 0x20000 # unicode string literals
CO_FUTURE_BARRY_AS_BDFL = 0x40000
CO_FUTURE_GENERATOR_STOP = 0x80000 # StopIteration becomes RuntimeError in generators
CO_FUTURE_ANNOTATIONS = 0x100000 # annotations become strings at runtime
class _Feature:
def __init__(self, optionalRelease, mandatoryRelease, compiler_flag):
@ -132,9 +134,13 @@ unicode_literals = _Feature((2, 6, 0, "alpha", 2),
CO_FUTURE_UNICODE_LITERALS)
barry_as_FLUFL = _Feature((3, 1, 0, "alpha", 2),
(3, 9, 0, "alpha", 0),
CO_FUTURE_BARRY_AS_BDFL)
(3, 9, 0, "alpha", 0),
CO_FUTURE_BARRY_AS_BDFL)
generator_stop = _Feature((3, 5, 0, "beta", 1),
(3, 7, 0, "alpha", 0),
CO_FUTURE_GENERATOR_STOP)
(3, 7, 0, "alpha", 0),
CO_FUTURE_GENERATOR_STOP)
annotations = _Feature((3, 7, 0, "beta", 1),
(4, 0, 0, "alpha", 0),
CO_FUTURE_ANNOTATIONS)

163
Lib/test/test_future.py
View File

@ -1,7 +1,9 @@
# Test various flavors of legal and illegal future statements
from functools import partial
import unittest
from test import support
from textwrap import dedent
import os
import re
@ -102,6 +104,167 @@ class FutureTest(unittest.TestCase):
exec("from __future__ import unicode_literals; x = ''", {}, scope)
self.assertIsInstance(scope["x"], str)
class AnnotationsFutureTestCase(unittest.TestCase):
template = dedent(
"""
from __future__ import annotations
def f() -> {ann}:
...
def g(arg: {ann}) -> None:
...
var: {ann}
var2: {ann} = None
"""
)
def getActual(self, annotation):
scope = {}
exec(self.template.format(ann=annotation), {}, scope)
func_ret_ann = scope['f'].__annotations__['return']
func_arg_ann = scope['g'].__annotations__['arg']
var_ann1 = scope['__annotations__']['var']
var_ann2 = scope['__annotations__']['var2']
self.assertEqual(func_ret_ann, func_arg_ann)
self.assertEqual(func_ret_ann, var_ann1)
self.assertEqual(func_ret_ann, var_ann2)
return func_ret_ann
def assertAnnotationEqual(
self, annotation, expected=None, drop_parens=False, is_tuple=False,
):
actual = self.getActual(annotation)
if expected is None:
expected = annotation if not is_tuple else annotation[1:-1]
if drop_parens:
self.assertNotEqual(actual, expected)
actual = actual.replace("(", "").replace(")", "")
self.assertEqual(actual, expected)
def test_annotations(self):
eq = self.assertAnnotationEqual
eq('...')
eq("'some_string'")
eq("b'\\xa3'")
eq('Name')
eq('None')
eq('True')
eq('False')
eq('1')
eq('1.0')
eq('1j')
eq('True or False')
eq('True or False or None')
eq('True and False')
eq('True and False and None')
eq('(Name1 and Name2) or Name3')
eq('Name1 or (Name2 and Name3)')
eq('(Name1 and Name2) or (Name3 and Name4)')
eq('Name1 or (Name2 and Name3) or Name4')
eq('v1 << 2')
eq('1 >> v2')
eq(r'1 % finished')
eq('((1 + v2) - (v3 * 4)) ^ (((5 ** v6) / 7) // 8)')
eq('not great')
eq('~great')
eq('+value')
eq('-1')
eq('(~int) and (not ((v1 ^ (123 + v2)) | True))')
eq('lambda arg: None')
eq('lambda a=True: a')
eq('lambda a, b, c=True: a')
eq("lambda a, b, c=True, *, d=(1 << v2), e='str': a")
eq("lambda a, b, c=True, *vararg, d=(v1 << 2), e='str', **kwargs: a + b")
eq('1 if True else 2')
eq('(str or None) if True else (str or bytes or None)')
eq('(str or None) if (1 if True else 2) else (str or bytes or None)')
eq("{'2.7': dead, '3.7': (long_live or die_hard)}")
eq("{'2.7': dead, '3.7': (long_live or die_hard), **{'3.6': verygood}}")
eq("{**a, **b, **c}")
eq("{'2.7', '3.6', '3.7', '3.8', '3.9', ('4.0' if gilectomy else '3.10')}")
eq("({'a': 'b'}, (True or False), (+value), 'string', b'bytes') or None")
eq("()")
eq("(1,)")
eq("(1, 2)")
eq("(1, 2, 3)")
eq("[]")
eq("[1, 2, 3, 4, 5, 6, 7, 8, 9, (10 or A), (11 or B), (12 or C)]")
eq("{i for i in (1, 2, 3)}")
eq("{(i ** 2) for i in (1, 2, 3)}")
eq("{(i ** 2) for i, _ in ((1, 'a'), (2, 'b'), (3, 'c'))}")
eq("{((i ** 2) + j) for i in (1, 2, 3) for j in (1, 2, 3)}")
eq("[i for i in (1, 2, 3)]")
eq("[(i ** 2) for i in (1, 2, 3)]")
eq("[(i ** 2) for i, _ in ((1, 'a'), (2, 'b'), (3, 'c'))]")
eq("[((i ** 2) + j) for i in (1, 2, 3) for j in (1, 2, 3)]")
eq(r"{i: 0 for i in (1, 2, 3)}")
eq("{i: j for i, j in ((1, 'a'), (2, 'b'), (3, 'c'))}")
eq("Python3 > Python2 > COBOL")
eq("Life is Life")
eq("call()")
eq("call(arg)")
eq("call(kwarg='hey')")
eq("call(arg, kwarg='hey')")
eq("call(arg, another, kwarg='hey', **kwargs)")
eq("lukasz.langa.pl")
eq("call.me(maybe)")
eq("1 .real")
eq("1.0 .real")
eq("....__class__")
eq("list[str]")
eq("dict[str, int]")
eq("tuple[str, ...]")
eq("tuple[str, int, float, dict[str, int]]")
eq("slice[0]")
eq("slice[0:1]")
eq("slice[0:1:2]")
eq("slice[:]")
eq("slice[:-1]")
eq("slice[1:]")
eq("slice[::-1]")
eq('(str or None) if (sys.version_info[0] > (3,)) else (str or bytes or None)')
eq("f'f-string without formatted values is just a string'")
eq("f'{{NOT a formatted value}}'")
eq("f'some f-string with {a} {few():.2f} {formatted.values!r}'")
eq('''f"{f'{nested} inner'} outer"''')
eq("f'space between opening braces: { {a for a in (1, 2, 3)}}'")
def test_annotations_inexact(self):
"""Source formatting is not always preserved
This is due to reconstruction from AST. We *need to* put the parens
in nested expressions because we don't know if the source code
had them in the first place or not.
"""
eq = partial(self.assertAnnotationEqual, drop_parens=True)
eq('Name1 and Name2 or Name3')
eq('Name1 or Name2 and Name3')
eq('Name1 and Name2 or Name3 and Name4')
eq('Name1 or Name2 and Name3 or Name4')
eq('1 + v2 - v3 * 4 ^ v5 ** 6 / 7 // 8')
eq('~int and not v1 ^ 123 + v2 | True')
eq('str or None if True else str or bytes or None')
eq("{'2.7': dead, '3.7': long_live or die_hard}")
eq("{'2.7', '3.6', '3.7', '3.8', '3.9', '4.0' if gilectomy else '3.10'}")
eq("[1, 2, 3, 4, 5, 6, 7, 8, 9, 10 or A, 11 or B, 12 or C]")
# Consequently, we always drop unnecessary parens if they were given in
# the outer scope:
some_name = self.getActual("(SomeName)")
self.assertEqual(some_name, 'SomeName')
# Interestingly, in the case of tuples (and generator expressions) the
# parens are *required* by the Python syntax in the annotation context.
# But there's no point storing that detail in __annotations__ so we're
# fine with the parens-less form.
eq = partial(self.assertAnnotationEqual, is_tuple=True)
eq("(Good, Bad, Ugly)")
eq("(i for i in (1, 2, 3))")
eq("((i ** 2) for i in (1, 2, 3))")
eq("((i ** 2) for i, _ in ((1, 'a'), (2, 'b'), (3, 'c')))")
eq("(((i ** 2) + j) for i in (1, 2, 3) for j in (1, 2, 3))")
eq("(*starred)")
eq('(yield from outside_of_generator)')
eq('(yield)')
eq('(await some.complicated[0].call(with_args=(True or (1 is not 1))))')
if __name__ == "__main__":

3
Makefile.pre.in
View File

@ -325,6 +325,7 @@ PYTHON_OBJS= \
Python/asdl.o \
Python/ast.o \
Python/ast_opt.o \
Python/ast_unparse.o \
Python/bltinmodule.o \
Python/ceval.o \
Python/compile.o \
@ -840,7 +841,7 @@ regen-opcode:
$(srcdir)/Include/opcode.h.new
$(UPDATE_FILE) $(srcdir)/Include/opcode.h $(srcdir)/Include/opcode.h.new
Python/compile.o Python/symtable.o Python/ast.o: $(srcdir)/Include/graminit.h $(srcdir)/Include/Python-ast.h
Python/compile.o Python/symtable.o Python/ast_unparse.o Python/ast.o: $(srcdir)/Include/graminit.h $(srcdir)/Include/Python-ast.h
Python/getplatform.o: $(srcdir)/Python/getplatform.c
$(CC) -c $(PY_CORE_CFLAGS) -DPLATFORM='"$(MACHDEP)"' -o $@ $(srcdir)/Python/getplatform.c

1
PCbuild/pythoncore.vcxproj
View File

@ -358,6 +358,7 @@
<ClCompile Include="..\Python\asdl.c" />
<ClCompile Include="..\Python\ast.c" />
<ClCompile Include="..\Python\ast_opt.c" />
<ClCompile Include="..\Python\ast_unparse.c" />
<ClCompile Include="..\Python\bltinmodule.c" />
<ClCompile Include="..\Python\bootstrap_hash.c" />
<ClCompile Include="..\Python\ceval.c" />

3
PCbuild/pythoncore.vcxproj.filters
View File

@ -839,6 +839,9 @@
<ClCompile Include="..\Python\ast_opt.c">
<Filter>Python</Filter>
</ClCompile>
<ClCompile Include="..\Python\ast_unparse.c">
<Filter>Python</Filter>
</ClCompile>
<ClCompile Include="..\Python\bltinmodule.c">
<Filter>Python</Filter>
</ClCompile>

1163
Python/ast_unparse.c Normal file
View File

File diff suppressed because it is too large Load Diff

26
Python/compile.c
View File

@ -1699,13 +1699,30 @@ error:
return 0;
}
static int
compiler_visit_annexpr(struct compiler *c, expr_ty annotation)
{
PyObject *ann_as_str;
ann_as_str = _PyAST_ExprAsUnicode(annotation, 1);
if (!ann_as_str) {
return 0;
}
ADDOP_N(c, LOAD_CONST, ann_as_str, consts);
return 1;
}
static int
compiler_visit_argannotation(struct compiler *c, identifier id,
expr_ty annotation, PyObject *names)
{
if (annotation) {
PyObject *mangled;
VISIT(c, expr, annotation);
if (c->c_future->ff_features & CO_FUTURE_ANNOTATIONS) {
VISIT(c, annexpr, annotation)
}
else {
VISIT(c, expr, annotation);
}
mangled = _Py_Mangle(c->u->u_private, id);
if (!mangled)
return 0;
@ -4688,7 +4705,12 @@ compiler_annassign(struct compiler *c, stmt_ty s)
if (!mangled) {
return 0;
}
VISIT(c, expr, s->v.AnnAssign.annotation);
if (c->c_future->ff_features & CO_FUTURE_ANNOTATIONS) {
VISIT(c, annexpr, s->v.AnnAssign.annotation)
}
else {
VISIT(c, expr, s->v.AnnAssign.annotation);
}
/* ADDOP_N decrefs its argument */
ADDOP_N(c, STORE_ANNOTATION, mangled, names);
}

2
Python/future.c
View File

@ -42,6 +42,8 @@ future_check_features(PyFutureFeatures *ff, stmt_ty s, PyObject *filename)
ff->ff_features |= CO_FUTURE_BARRY_AS_BDFL;
} else if (strcmp(feature, FUTURE_GENERATOR_STOP) == 0) {
ff->ff_features |= CO_FUTURE_GENERATOR_STOP;
} else if (strcmp(feature, FUTURE_ANNOTATIONS) == 0) {
ff->ff_features |= CO_FUTURE_ANNOTATIONS;
} else if (strcmp(feature, "braces") == 0) {
PyErr_SetString(PyExc_SyntaxError,
"not a chance");