The fix for Issue #21217 introduced a regression that caused
`inspect.getsource` to return incorrect results on nested
functions. The root cause of the regression was due to
switching the implementation to analyze the underlying
bytecode instead of the source code.
This commit switches things back to analyzing the source code
in a more complete way. The original bug and the regression
are both fixed by the new source code analysis.
The fix for Issue #21217 introduced a regression that caused
`inspect.getsource` to return incorrect results on nested
functions. The root cause of the regression was due to
switching the implementation to analyze the underlying
bytecode instead of the source code.
This commit switches things back to analyzing the source code
in a more complete way. The original bug and the regression
are both fixed by the new source code analysis.
collections.abc.Awaitable and collections.abc.Coroutine no longer
use __instancecheck__ hook to detect generator-based coroutines.
inspect.isawaitable() can be used to detect generator-based coroutines
and to distinguish them from regular generator objects.
isawaitable() was added before collections.abc.Awaitable; now,
with Awaitable, it is no longer needed (we don't have ishashable()
or isiterable() methods in the inspect module either).
Summary of changes:
1. Coroutines now have a distinct, separate from generators
type at the C level: PyGen_Type, and a new typedef PyCoroObject.
PyCoroObject shares the initial segment of struct layout with
PyGenObject, making it possible to reuse existing generators
machinery. The new type is exposed as 'types.CoroutineType'.
As a consequence of having a new type, CO_GENERATOR flag is
no longer applied to coroutines.
2. Having a separate type for coroutines made it possible to add
an __await__ method to the type. Although it is not used by the
interpreter (see details on that below), it makes coroutines
naturally (without using __instancecheck__) conform to
collections.abc.Coroutine and collections.abc.Awaitable ABCs.
[The __instancecheck__ is still used for generator-based
coroutines, as we don't want to add __await__ for generators.]
3. Add new opcode: GET_YIELD_FROM_ITER. The opcode is needed to
allow passing native coroutines to the YIELD_FROM opcode.
Before this change, 'yield from o' expression was compiled to:
(o)
GET_ITER
LOAD_CONST
YIELD_FROM
Now, we use GET_YIELD_FROM_ITER instead of GET_ITER.
The reason for adding a new opcode is that GET_ITER is used
in some contexts (such as 'for .. in' loops) where passing
a coroutine object is invalid.
4. Add two new introspection functions to the inspec module:
getcoroutinestate(c) and getcoroutinelocals(c).
5. inspect.iscoroutine(o) is updated to test if 'o' is a native
coroutine object. Before this commit it used abc.Coroutine,
and it was requested to update inspect.isgenerator(o) to use
abc.Generator; it was decided, however, that inspect functions
should really be tailored for checking for native types.
6. sys.set_coroutine_wrapper(w) API is updated to work with only
native coroutines. Since types.coroutine decorator supports
any type of callables now, it would be confusing that it does
not work for all types of coroutines.
7. Exceptions logic in generators C implementation was updated
to raise clearer messages for coroutines:
Before: TypeError("generator raised StopIteration")
After: TypeError("coroutine raised StopIteration")
The new syntax is highly human readable while still preventing false
positives. The syntax also extends Python syntax to denote "self" and
positional-only parameters, allowing inspect.Signature objects to be
totally accurate for all supported builtins in Python 3.4.
annotate text signatures in docstrings, resulting in fewer false
positives. "self" parameters are also explicitly marked, allowing
inspect.Signature() to authoritatively detect (and skip) said parameters.
Issue #20326: Argument Clinic now generates separate checksums for the
input and output sections of the block, allowing external tools to verify
that the input has not changed (and thus the output is not out-of-date).
PyMethodDescr_Type, _PyMethodWrapper_Type, and PyWrapperDescr_Type)
have been modified to provide introspection information for builtins.
Also: many additional Lib, test suite, and Argument Clinic fixes.
* You may now specify an expression as the default value for a
parameter! Example: "sys.maxsize - 1". This support is
intentionally quite limited; you may only use values that
can be represented as static C values.
* Removed "doc_default", simplified support for "c_default"
and "py_default". (I'm not sure we still even need
"py_default", but I'm leaving it in for now in case a
use presents itself.)
* Parameter lines support a trailing '\\' as a line
continuation character, allowing you to break up long lines.
* The argument parsing code generated when supporting optional
groups now uses PyTuple_GET_SIZE instead of PyTuple_GetSize,
leading to a 850% speedup in parsing. (Just kidding, this
is an unmeasurable difference.)
* A bugfix for the recent regression where the generated
prototype from pydoc for builtins would be littered with
unreadable "=<object ...>"" default values for parameters
that had no default value.
* Converted some asserts into proper failure messages.
* Many doc improvements and fixes.
Okay, hopefully the very last patch for this issue. :/
I realized when playing with Enum that the metaclass attributes weren't always displayed properly.
New patch properly locates DynamicClassAttributes, virtual class attributes (returned by __getattr__ and friends), and metaclass class attributes (if they are also in the metaclass __dir__ method).
Also had to change one line in pydoc to get this to work.
Added tests in test_inspect and test_pydoc to cover these situations.
Order of search is now:
1. Try getattr
2. If that throws an exception, check __dict__ directly
3. If still not found, walk the mro looking for the eldest class that has
the attribute (e.g. things returned by __getattr__)
4. If none of that works (e.g. due to a buggy __dir__, __getattr__, etc.
method or missing __slot__ attribute), ignore the attribute entirely.
inspect.getmembers and inspect.classify_class_attrs now search the metaclass
mro for types.DynamicClassAttributes (what use to be called
enum._RouteClassAttributeToGetattr); in part this means that these two
functions no longer rely solely on dir().
Besides now returning more accurate information, these improvements also
allow a more helpful help() on Enum classes.
Serhiy Storchaka
Berker Peksag
Meador Inge
Yury Selivanov
Antoine Pitrou
Guido van Rossum
Larry Hastings
Benjamin Peterson
Ethan Furman