Tested on macOS 10.11 dtrace, Ubuntu 16.04 SystemTap, and libbcc.
Largely based by an initial patch by Jesús Cea Avión, with some
influence from Dave Malcolm's SystemTap patch and Nikhil Benesch's
unification patch.
Things deliberately left out for simplicity:
- ustack helpers, I have no way of testing them at this point since
they are Solaris-specific
- PyFrameObject * in function__entry/function__return, this is
SystemTap-specific
- SPARC support
- dynamic tracing
- sys module dtrace facility introspection
All of those might be added later.
Issue #27830: Add _PyObject_FastCallKeywords(): avoid the creation of a
temporary dictionary for keyword arguments.
Other changes:
* Cleanup call_function() and fast_function() (ex: rename nk to nkwargs)
* Remove now useless do_call(), replaced with _PyObject_FastCallKeywords()
Issue #27213: Rework CALL_FUNCTION* opcodes to produce shorter and more
efficient bytecode:
* CALL_FUNCTION now only accepts position arguments
* CALL_FUNCTION_KW accepts position arguments and keyword arguments, but keys
of keyword arguments are packed into a constant tuple.
* CALL_FUNCTION_EX is the most generic, it expects a tuple and a dict for
positional and keyword arguments.
CALL_FUNCTION_VAR and CALL_FUNCTION_VAR_KW opcodes have been removed.
2 tests of test_traceback are currently broken: skip test, the issue #28050 was
created to track the issue.
Patch by Demur Rumed, design by Serhiy Storchaka, reviewed by Serhiy Storchaka
and Victor Stinner.
Issue #27830: Similar to _PyObject_FastCallDict(), but keyword arguments are
also passed in the same C array than positional arguments, rather than being
passed as a Python dict.
Issue #27809: PyEval_CallObjectWithKeywords() doesn't increment temporary the
reference counter of the args tuple (positional arguments). The caller already
holds a strong reference to it.
Issue #27128. When a Python function is called with no arguments, but all
parameters have a default value: use default values as arguments for the fast
path.
Issue #27128: Modify PyEval_CallObjectWithKeywords() to use
_PyObject_FastCall() when args==NULL and kw==NULL. It avoids the creation of a
temporary empty tuple for positional arguments.
Issue #27128: Add _PyObject_FastCall(), a new calling convention avoiding a
temporary tuple to pass positional parameters in most cases, but create a
temporary tuple if needed (ex: for the tp_call slot).
The API is prepared to support keyword parameters, but the full implementation
will come later (_PyFunction_FastCall() doesn't support keyword parameters
yet).
Add also:
* _PyStack_AsTuple() helper function: convert a "stack" of parameters to
a tuple.
* _PyCFunction_FastCall(): fast call implementation for C functions
* _PyFunction_FastCall(): fast call implementation for Python functions
Issue #27558: Fix a SystemError in the implementation of "raise" statement.
In a brand new thread, raise a RuntimeError since there is no active
exception to reraise.
Patch written by Xiang Zhang.
Issue #27128, #18295: replace int type with Py_ssize_t for index variables used
for positional arguments. It should help to avoid integer overflow and help to
emit better machine code for "i++" (no trap needed for overflow).
Make also the total_args variable constant.
* Add comments
* Add empty lines for readability
* PEP 7 style for if block
* Remove useless assert(globals != NULL); (globals is tested a few lines
before)
Don't fallback to PyDict_GetItemWithError() if the hash is unknown: compute the
hash instead. Add also comments to explain the optimization a little bit.
requested name doesn't exist in globals: clear the KeyError exception before
calling PyObject_GetItem(). Fail also if the raised exception is not a
KeyError.
Summary of changes:
1. Coroutines now have a distinct, separate from generators
type at the C level: PyGen_Type, and a new typedef PyCoroObject.
PyCoroObject shares the initial segment of struct layout with
PyGenObject, making it possible to reuse existing generators
machinery. The new type is exposed as 'types.CoroutineType'.
As a consequence of having a new type, CO_GENERATOR flag is
no longer applied to coroutines.
2. Having a separate type for coroutines made it possible to add
an __await__ method to the type. Although it is not used by the
interpreter (see details on that below), it makes coroutines
naturally (without using __instancecheck__) conform to
collections.abc.Coroutine and collections.abc.Awaitable ABCs.
[The __instancecheck__ is still used for generator-based
coroutines, as we don't want to add __await__ for generators.]
3. Add new opcode: GET_YIELD_FROM_ITER. The opcode is needed to
allow passing native coroutines to the YIELD_FROM opcode.
Before this change, 'yield from o' expression was compiled to:
(o)
GET_ITER
LOAD_CONST
YIELD_FROM
Now, we use GET_YIELD_FROM_ITER instead of GET_ITER.
The reason for adding a new opcode is that GET_ITER is used
in some contexts (such as 'for .. in' loops) where passing
a coroutine object is invalid.
4. Add two new introspection functions to the inspec module:
getcoroutinestate(c) and getcoroutinelocals(c).
5. inspect.iscoroutine(o) is updated to test if 'o' is a native
coroutine object. Before this commit it used abc.Coroutine,
and it was requested to update inspect.isgenerator(o) to use
abc.Generator; it was decided, however, that inspect functions
should really be tailored for checking for native types.
6. sys.set_coroutine_wrapper(w) API is updated to work with only
native coroutines. Since types.coroutine decorator supports
any type of callables now, it would be confusing that it does
not work for all types of coroutines.
7. Exceptions logic in generators C implementation was updated
to raise clearer messages for coroutines:
Before: TypeError("generator raised StopIteration")
After: TypeError("coroutine raised StopIteration")
* adds missing INCREF in WITH_CLEANUP_START
* adds missing DECREF in WITH_CLEANUP_FINISH
* adds several new tests Yury created while investigating this
which returned an invalid result (result+error or no result without error) in
the exception message.
Add also unit test to check that the exception contains the name of the
function.
Special case: the final _PyEval_EvalFrameEx() check doesn't mention the
function since it didn't execute a single function but a whole frame.
raise a SystemError if a function returns a result and raises an exception.
The SystemError is chained to the previous exception.
Refactor also PyObject_Call() and PyCFunction_Call() to make them more readable.
Remove some checks which became useless (duplicate checks).
Change reviewed by Serhiy Storchaka.
At entry, save or swap the exception state even if PyEval_EvalFrameEx() is
called with throwflag=0. At exit, the exception state is now always restored or
swapped, not only if why is WHY_YIELD or WHY_RETURN. Patch co-written with
Antoine Pitrou.
name, and use it in the representation of a generator (``repr(gen)``). The
default name of the generator (``__name__`` attribute) is now get from the
function instead of the code. Use ``gen.gi_code.co_name`` to get the name of
the code.
crash when a generator is created in a C thread that is destroyed while the
generator is still used. The issue was that a generator contains a frame, and
the frame kept a reference to the Python state of the destroyed C thread. The
crash occurs when a trace function is setup.
with an assertion error if they are called with an exception set
(PyErr_Occurred()).
If these functions are called with an exception set, the exception may be
cleared and so the caller looses its exception.
Add also assertions to PyEval_CallObjectWithKeywords() and call_function() to
check if the function succeed with no exception set, or the function failed
with an exception set.
ImportError.
The exception is raised by import when a module could not be found.
Technically this is defined as no viable loader could be found for the
specified module. This includes ``from ... import`` statements so that
the module usage is consistent for all situations where import
couldn't find what was requested.
This should allow for the common idiom of::
try:
import something
except ImportError:
pass
to be updated to using ModuleNotFoundError and not accidentally mask
ImportError messages that should propagate (e.g. issues with a
loader).
This work was driven by the fact that the ``from ... import``
statement needed to be able to tell the difference between an
ImportError that simply couldn't find a module (and thus silence the
exception so that ceval can raise it) and an ImportError that
represented an actual problem.
Note that this is a potentially disruptive change since it may
release some system resources which would otherwise remain
perpetually alive (e.g. database connections kept in thread-local
storage).
- Make many variables local to the opcode; Kill u, v, w, and x.
- Force every opcode to end with DISPATCH or jump to error handling.
- Simplify error handling.
- Check error statuses in more places.
Closes#16191.
It is now possible to use a custom type for the __builtins__ namespace, instead
of a dict. It can be used for sandboxing for example. Raise also a NameError
instead of ImportError if __build_class__ name if not found in __builtins__.
This allows generators that are using yield from to be seen by debuggers. It
also kills the f_yieldfrom field on frame objects.
Patch mostly from Mark Shannon with a few tweaks by me.
These were just an artifact of the old unicode concatenation hack and likely
just penalized other kinds of adding. Also, this fixes __(i)add__ on string
subclasses.
Issue #11168: Remove filename debug variable from PyEval_EvalFrameEx().
It encoded the Unicode filename to UTF-8, but the encoding fails on
undecodable filename (on surrogate characters) which raises an unexpected
UnicodeEncodeError on recursion limit.
svn+ssh://pythondev@svn.python.org/python/branches/py3k
........
r87796 | david.malcolm | 2011-01-06 12:01:36 -0500 (Thu, 06 Jan 2011) | 6 lines
Issue #10655: Fix the build on PowerPC on Linux with GCC when building with
timestamp profiling (--with-tsc): the preprocessor test for the PowerPC
support now looks for "__powerpc__" as well as "__ppc__": the latter seems to
only be present on OS X; the former is the correct one for Linux with GCC.
........
timestamp profiling (--with-tsc): the preprocessor test for the PowerPC
support now looks for "__powerpc__" as well as "__ppc__": the latter seems to
only be present on OS X; the former is the correct one for Linux with GCC.
namespace if it occurs as a free variable in a nested block. This limitation
of the compiler has been lifted, and a new opcode introduced (DELETE_DEREF).
This sample was valid in 2.6, but fails to compile in 3.x without this change::
>>> def f():
... def print_error():
... print(e)
... try:
... something
... except Exception as e:
... print_error()
... # implicit "del e" here
This sample has always been invalid in Python, and now works::
>>> def outer(x):
... def inner():
... return x
... inner()
... del x
There is no need to bump the PYC magic number: the new opcode is used
for code that did not compile before.
http://code.google.com/p/data-race-test/wiki/ThreadSanitizer is a dynamic data
race detector that runs on top of valgrind. With this patch, the binaries at
http://code.google.com/p/data-race-test/wiki/ThreadSanitizer#Binaries pass many
but not all of the Python tests. All of regrtest still passes outside of tsan.
I've implemented part of the C1x atomic types so that we can explicitly mark
variables that are used across threads, and get defined behavior as compilers
advance.
I've added tsan's client header and implementation to the codebase in
dynamic_annotations.{h,c} (docs at
http://code.google.com/p/data-race-test/wiki/DynamicAnnotations).
Unfortunately, I haven't been able to get helgrind and drd to give sensible
error messages, even when I use their client annotations, so I'm not supporting
them.
svn+ssh://pythondev@svn.python.org/python/trunk
........
r79555 | antoine.pitrou | 2010-04-01 18:42:11 +0200 (jeu., 01 avril 2010) | 5 lines
Issue #8276: PyEval_CallObject() is now only available in macro form. The
function declaration, which was kept for backwards compatibility reasons,
is now removed (the macro was introduced in 1997!).
........
svn+ssh://pythondev@svn.python.org/python/branches/py3k
........
r78875 | victor.stinner | 2010-03-12 18:00:41 +0100 (ven., 12 mars 2010) | 5 lines
Issue #6697: use %U format instead of _PyUnicode_AsString(), because
_PyUnicode_AsString() was not checked for error (NULL).
The unicode string is no more truncated to 200 or 400 *bytes*.
........
svn+ssh://pythondev@svn.python.org/python/trunk
........
r79205 | benjamin.peterson | 2010-03-21 12:34:54 -0500 (Sun, 21 Mar 2010) | 1 line
rewrite a bit
........
r79219 | benjamin.peterson | 2010-03-21 14:24:08 -0500 (Sun, 21 Mar 2010) | 1 line
flatten condition
........
r79228 | benjamin.peterson | 2010-03-21 14:35:39 -0500 (Sun, 21 Mar 2010) | 1 line
remove pointless condition
........
r79230 | benjamin.peterson | 2010-03-21 14:39:52 -0500 (Sun, 21 Mar 2010) | 1 line
co_varnames is certainly a tuple, so let's not waste time finding out
........
r79232 | benjamin.peterson | 2010-03-21 14:54:56 -0500 (Sun, 21 Mar 2010) | 1 line
fix import
........
r79233 | benjamin.peterson | 2010-03-21 14:56:37 -0500 (Sun, 21 Mar 2010) | 1 line
don't write duplicate tests
........
r79235 | benjamin.peterson | 2010-03-21 15:21:00 -0500 (Sun, 21 Mar 2010) | 4 lines
improve error message from passing inadequate number of keyword arguments #6474
Note this removes the "non-keyword" or "keyword" phrases from these messages.
........
r79237 | benjamin.peterson | 2010-03-21 15:30:30 -0500 (Sun, 21 Mar 2010) | 1 line
take into account keyword arguments when passing too many args
........
svn+ssh://pythondev@svn.python.org/python/branches/py3k
................
r75984 | mark.dickinson | 2009-10-31 10:18:44 +0000 (Sat, 31 Oct 2009) | 12 lines
Merged revisions 75982 via svnmerge from
svn+ssh://pythondev@svn.python.org/python/trunk
........
r75982 | mark.dickinson | 2009-10-31 10:11:28 +0000 (Sat, 31 Oct 2009) | 5 lines
Issue #6603: Fix --with-tsc build failures on x86-64 that resulted
from a gcc inline assembler peculiarity. (gcc's "A" constraint
apparently means 'rax or rdx' in 64-bit mode, not edx:eax
or rdx:rax as one might expect.)
........
................
svn+ssh://pythondev@svn.python.org/python/trunk
........
r75982 | mark.dickinson | 2009-10-31 10:11:28 +0000 (Sat, 31 Oct 2009) | 5 lines
Issue #6603: Fix --with-tsc build failures on x86-64 that resulted
from a gcc inline assembler peculiarity. (gcc's "A" constraint
apparently means 'rax or rdx' in 64-bit mode, not edx:eax
or rdx:rax as one might expect.)
........
svn+ssh://pythondev@svn.python.org/python/trunk
........
r73750 | benjamin.peterson | 2009-07-01 19:45:19 -0400 (Wed, 01 Jul 2009) | 1 line
small optimization: avoid popping the current block until we have to
........
svn+ssh://pythondev@svn.python.org/python/trunk
........
r72487 | jeffrey.yasskin | 2009-05-08 17:51:06 -0400 (Fri, 08 May 2009) | 7 lines
PyCode_NewEmpty:
Most uses of PyCode_New found by http://www.google.com/codesearch?q=PyCode_New
are trying to build an empty code object, usually to put it in a dummy frame
object. This patch adds a PyCode_NewEmpty wrapper which lets the user specify
just the filename, function name, and first line number, instead of also
requiring lots of code internals.
........
r72488 | jeffrey.yasskin | 2009-05-08 18:23:21 -0400 (Fri, 08 May 2009) | 13 lines
Issue 5954, PyFrame_GetLineNumber:
Most uses of PyCode_Addr2Line
(http://www.google.com/codesearch?q=PyCode_Addr2Line) are just trying to get
the line number of a specified frame, but there's no way to do that directly.
Forcing people to go through the code object makes them know more about the
guts of the interpreter than they should need.
The remaining uses of PyCode_Addr2Line seem to be getting the line from a
traceback (for example,
http://www.google.com/codesearch/p?hl=en#u_9_nDrchrw/pygame-1.7.1release/src/base.c&q=PyCode_Addr2Line),
which is replaced by the tb_lineno field. So we may be able to deprecate
PyCode_Addr2Line entirely for external use.
........
r72879 | jeffrey.yasskin | 2009-05-23 19:23:01 -0400 (Sat, 23 May 2009) | 14 lines
Issue #6042:
lnotab-based tracing is very complicated and isn't documented very well. There
were at least 3 comment blocks purporting to document co_lnotab, and none did a
very good job. This patch unifies them into Objects/lnotab_notes.txt which
tries to completely capture the current state of affairs.
I also discovered that we've attached 2 layers of patches to the basic tracing
scheme. The first layer avoids jumping to instructions that don't start a line,
to avoid problems in if statements and while loops. The second layer
discovered that jumps backward do need to trace at instructions that don't
start a line, so it added extra lnotab entries for 'while' and 'for' loops, and
added a special case for backward jumps within the same line. I replaced these
patches by just treating forward and backward jumps differently.
........
svn+ssh://pythondev@svn.python.org/python/trunk
........
r73614 | benjamin.peterson | 2009-06-28 11:08:02 -0500 (Sun, 28 Jun 2009) | 1 line
add two generic macros for peeking and setting in the stack
........
r73615 | benjamin.peterson | 2009-06-28 11:14:07 -0500 (Sun, 28 Jun 2009) | 1 line
use stack macros
........
svn+ssh://pythondev@svn.python.org/python/trunk
........
r72912 | benjamin.peterson | 2009-05-25 08:13:44 -0500 (Mon, 25 May 2009) | 5 lines
add a SETUP_WITH opcode
It speeds up the with statement and correctly looks up the special
methods involved.
........
r72920 | benjamin.peterson | 2009-05-25 15:12:57 -0500 (Mon, 25 May 2009) | 1 line
take into account the fact that SETUP_WITH pushes a finally block
........
r72940 | benjamin.peterson | 2009-05-26 07:49:59 -0500 (Tue, 26 May 2009) | 1 line
teach the peepholer about SETUP_WITH
........
svn+ssh://pythondev@svn.python.org/python/trunk
........
r73064 | antoine.pitrou | 2009-05-30 23:27:00 +0200 (sam., 30 mai 2009) | 4 lines
Issue #5330: C functions called with keyword arguments were not reported by
the various profiling modules (profile, cProfile). Patch by Hagen Fürstenau.
........
svn+ssh://pythondev@svn.python.org/python/trunk
........
r69811 | collin.winter | 2009-02-20 13:30:41 -0600 (Fri, 20 Feb 2009) | 2 lines
Issue 5176: special-case string formatting in BINARY_MODULO implementation. This shows a modest (1-3%) speed-up in templating systems, for example.
........
r69947 | jeffrey.yasskin | 2009-02-24 16:48:34 -0600 (Tue, 24 Feb 2009) | 3 lines
Tools/scripts/analyze_dxp.py, a module with some helper functions to
analyze the output of sys.getdxp().
........
This patch by Antoine Pitrou optimizes the bytecode for conditional branches by
merging the following "POP_TOP" instruction into the conditional jump. For
example, the list comprehension "[x for x in l if not x]" produced the
following bytecode:
1 0 BUILD_LIST 0
3 LOAD_FAST 0 (.0)
>> 6 FOR_ITER 23 (to 32)
9 STORE_FAST 1 (x)
12 LOAD_FAST 1 (x)
15 JUMP_IF_TRUE 10 (to 28)
18 POP_TOP
19 LOAD_FAST 1 (x)
22 LIST_APPEND 2
25 JUMP_ABSOLUTE 6
>> 28 POP_TOP
29 JUMP_ABSOLUTE 6
>> 32 RETURN_VALUE
but after the patch it produces the following bytecode:
1 0 BUILD_LIST 0
3 LOAD_FAST 0 (.0)
>> 6 FOR_ITER 18 (to 27)
9 STORE_FAST 1 (x)
12 LOAD_FAST 1 (x)
15 POP_JUMP_IF_TRUE 6
18 LOAD_FAST 1 (x)
21 LIST_APPEND 2
24 JUMP_ABSOLUTE 6
>> 27 RETURN_VALUE
Notice that not only the code is shorter, but the conditional jump
(POP_JUMP_IF_TRUE) jumps right to the start of the loop instead of going through
the JUMP_ABSOLUTE at the end. "continue" statements are helped
similarly.
Furthermore, the old jump opcodes (JUMP_IF_FALSE, JUMP_IF_TRUE) have been
replaced by two new opcodes:
- JUMP_IF_TRUE_OR_POP, which jumps if true and pops otherwise
- JUMP_IF_FALSE_OR_POP, which jumps if false and pops otherwise
on compilers that support it (notably: gcc, SunPro, icc), the bytecode
evaluation loop is compiled with a new dispatch mechanism which gives
speedups of up to 20%, depending on the system, on various benchmarks.
Łukasz Langa
Benjamin Peterson
Victor Stinner
Yury Selivanov
Raymond Hettinger
Brett Cannon
Serhiy Storchaka
Martin Panter
Yury Selivanov
Eric V. Smith
Nick Coghlan
R David Murray
Antoine Pitrou
Guido van Rossum
Christian Heimes
Victor Stinner
Guido van Rossum
Ezio Melotti
Amaury Forgeot d'Arc
Martin v. Löwis
Victor Stinner
Charles-François Natali
Brett Cannon
David Malcolm
Georg Brandl
Stefan Krah
Jeffrey Yasskin
Mark Dickinson
Alexandre Vassalotti