In GH-2866, _Py_Bit_Length() was added to pymath.h for lack of a better
location. GH-20518 added a more appropriate header file for bit utilities. It
also shows how to properly use intrinsics. This allows reconsidering bpo-29782.
* Move the function to the new header.
* Changed return type to match __builtin_clzl() and reviewed usage.
* Use intrinsics where available.
* Pick a fallback implementation suitable for inlining.
The PEP 353, written in 2005, introduced PY_FORMAT_SIZE_T. Python no
longer supports macOS 10.4 and Visual Studio 2010, but requires more
recent macOS and Visual Studio versions. In 2020 with Python 3.10, it
is now safe to use directly "%zu" to format size_t and "%zi" to
format Py_ssize_t.
* Rename pycore_byteswap.h to pycore_bitutils.h.
* Move popcount_digit() to pycore_bitutils.h as _Py_popcount32().
* _Py_popcount32() uses GCC and clang builtin function if available.
* Add unit tests to _Py_popcount32().
In debug mode, ensure that free lists are no longer used after being
finalized. Set numfree to -1 in finalization functions
(eg. _PyList_Fini()), and then check that numfree is not equal to -1
before using a free list (e.g list_dealloc()).
Each interpreter now has its own asynchronous generator free lists:
* Move async gen free lists into PyInterpreterState.
* Move _PyAsyncGen_MAXFREELIST define to pycore_interp.h
* Add _Py_async_gen_state structure.
* Add tstate parameter to _PyAsyncGen_ClearFreeLists
and _PyAsyncGen_Fini().
Each interpreter now has its own list free list:
* Move list numfree and free_list into PyInterpreterState.
* Add _Py_list_state structure.
* Add tstate parameter to _PyList_ClearFreeList()
and _PyList_Fini().
* Remove "#ifdef EXPERIMENTAL_ISOLATED_SUBINTERPRETERS".
* _PyGC_Fini() clears gcstate->garbage list which can be stored in
the list free list. Call _PyGC_Fini() before _PyList_Fini() to
prevent leaking this list.
Each interpreter now has its own frame free list:
* Move frame free list into PyInterpreterState.
* Add _Py_frame_state structure.
* Add tstate parameter to _PyFrame_ClearFreeList()
and _PyFrame_Fini().
* Remove "#if PyFrame_MAXFREELIST > 0".
* Remove "#ifdef EXPERIMENTAL_ISOLATED_SUBINTERPRETERS".
Each interpreter now has its own float free list:
* Move tuple numfree and free_list into PyInterpreterState.
* Add _Py_float_state structure.
* Add tstate parameter to _PyFloat_ClearFreeList()
and _PyFloat_Fini().
Each interpreter now has its own tuple free lists:
* Move tuple numfree and free_list arrays into PyInterpreterState.
* Define PyTuple_MAXSAVESIZE and PyTuple_MAXFREELIST macros in
pycore_interp.h.
* Add _Py_tuple_state structure. Pass it explicitly to tuple_alloc().
* Add tstate parameter to _PyTuple_ClearFreeList()
* Each interpreter now has its own empty tuple singleton.
* bpo-29882: Add an efficient popcount method for integers
* Update 'sign bit' and versionadded in docs
* Add entry to whatsnew document
* Doc: use positive example, mention population count
* Minor cleanups of the core code
* Move popcount_digit closer to where it's used
* Use z instead of self after conversion
* Add 'absolute value' and 'population count' to docstring
* Fix clinic error about missing summary line
* Ensure popcount_digit is portable with 64-bit ints
Co-authored-by: Mark Dickinson <dickinsm@gmail.com>
Previously, the result could have been an instance of a subclass of int.
Also revert bpo-26202 and make attributes start, stop and step of the range
object having exact type int.
Add private function _PyNumber_Index() which preserves the old behavior
of PyNumber_Index() for performance to use it in the conversion functions
like PyLong_AsLong().
Convert Py_REFCNT() and Py_SIZE() macros to static inline functions.
They cannot be used as l-value anymore: use Py_SET_REFCNT() and
Py_SET_SIZE() to set an object reference count and size.
Replace &Py_SIZE(self) with &((PyVarObject*)self)->ob_size
in arraymodule.c.
This change is backward incompatible on purpose, to prepare the C API
for an opaque PyObject structure.
* Fix outdated __int__ and nb_int references in comments
* Also update C-API documentation
* Add back missing 'method' word
* Remove .. deprecated notices
This updates _PyErr_ChainStackItem() to use _PyErr_SetObject()
instead of _PyErr_ChainExceptions(). This prevents a hang in
certain circumstances because _PyErr_SetObject() performs checks
to prevent cycles in the exception context chain while
_PyErr_ChainExceptions() doesn't.
When an asyncio.Task is cancelled, the exception traceback now
starts with where the task was first interrupted. Previously,
the traceback only had "depth one."
Move PyInterpreterState.fs_codec into a new
PyInterpreterState.unicode structure.
Give a name to the fs_codec structure and use this structure in
unicodeobject.c.
The previous commits on bpo-29587 got exception chaining working
with gen.throw() in the `yield` case. This patch also gets the
`yield from` case working.
As a consequence, implicit exception chaining now also works in
the asyncio scenario of awaiting on a task when an exception is
already active.
Tests are included for both the asyncio case and the pure
generator-only case.
Avoid unnecessary overhead in _PyDict_GetItemIdWithError() by calling
_PyDict_GetItem_KnownHash() instead of the more generic PyDict_GetItemWithError(),
since we already know the hash of interned strings.
* bpo-37986: Improve perfomance of PyLong_FromDouble()
* Use strict bound check for safety and symmetry
* Remove possibly outdated performance claims
Co-authored-by: Mark Dickinson <dickinsm@gmail.com>
Module C state is now accessible from C-defined heap type methods (PEP 573).
Patch by Marcel Plch and Petr Viktorin.
Co-authored-by: Marcel Plch <mplch@redhat.com>
Co-authored-by: Victor Stinner <vstinner@python.org>
When Python is built with experimental isolated interpreters, disable
the list free list.
Temporary workaround until this cache is made per-interpreter.
When Python is built with experimental isolated interpreters, disable
the type method cache.
Temporary workaround until the cache is made per-interpreter.
When Python is built with experimental isolated interpreters, disable
tuple, dict and free free lists.
Temporary workaround until these caches are made per-interpreter.
Add frame_alloc() and frame_get_builtins() subfunctions to simplify
_PyFrame_New_NoTrack().
When Python is built in the experimental isolated subinterpreters
mode, disable Unicode singletons and Unicode interned strings since
they are shared by all interpreters.
Temporary workaround until these caches are made per-interpreter.
_PyErr_ChainExceptions() now ensures that the first parameter is an
exception type, as done by _PyErr_SetObject().
* The following function now check PyExceptionInstance_Check() in an
assertion using a new _PyBaseExceptionObject_cast() helper
function:
* PyException_GetTraceback(), PyException_SetTraceback()
* PyException_GetCause(), PyException_SetCause()
* PyException_GetContext(), PyException_SetContext()
* PyExceptionClass_Name() now checks PyExceptionClass_Check() with an
assertion.
* Remove XXX comment and add gi_exc_state variable to _gen_throw().
* Remove comment from test_generators
This is a follow-up to GH-19823 that removes the check that the
exception value isn't NULL, prior to calling _PyErr_ChainExceptions().
This enables implicit exception chaining for gen.throw() in more
circumstances.
The commit also adds a test that a particular code snippet involving
gen.throw() doesn't crash. The test shows why the new
`gi_exc_state.exc_type != Py_None` check that was added is necessary.
Without the new check, the code snippet (as well as a number of other
tests) crashes on certain platforms (e.g. Fedora but not Mac).
Before this commit, if an exception was active inside a generator
when calling gen.throw(), that exception was lost (i.e. there was
no implicit exception chaining). This commit fixes that by
setting exc.__context__ when calling gen.throw(exc).
Before this commit, if an exception was active inside a generator
when calling gen.throw(), then that exception was lost (i.e. there
was no implicit exception chaining). This commit fixes that.
New PyFrame_GetBack() function: get the frame next outer frame.
Replace frame->f_back with PyFrame_GetBack(frame) in most code but
frameobject.c, ceval.c and genobject.c.
Remove the following function from the C API:
* PyAsyncGen_ClearFreeLists()
* PyContext_ClearFreeList()
* PyDict_ClearFreeList()
* PyFloat_ClearFreeList()
* PyFrame_ClearFreeList()
* PyList_ClearFreeList()
* PySet_ClearFreeList()
* PyTuple_ClearFreeList()
Make these functions private, move them to the internal C API and
change their return type to void.
Call explicitly PyGC_Collect() to free all free lists.
Note: PySet_ClearFreeList() did nothing.
PyFrame_GetCode(frame): return a borrowed reference to the frame
code.
Replace frame->f_code with PyFrame_GetCode(frame) in most code,
except in frameobject.c, genobject.c and ceval.c.
Also add PyFrame_GetLineNumber() to the limited C API.
Add a new separated pyframe.h header file of the PyFrame public C
API: it is included by Python.h.
Add PyFrame_GetLineNumber() to the limited C API.
Replace "struct _frame" with "PyFrameObject" in header files.
PyFrameObject is now defined as struct _frame by pyframe.h which is
included early enough in Python.h.
Added str.removeprefix and str.removesuffix methods and corresponding
bytes, bytearray, and collections.UserString methods to remove affixes
from a string if present. See PEP 616 for a full description.
* Replace PY_INT64_T with int64_t
* Replace PY_UINT32_T with uint32_t
* Replace PY_UINT64_T with uint64_t
sha3module.c no longer checks if PY_UINT64_T is defined since it's
always defined and uint64_t is always available on platforms
supported by Python.
Add a new internal pycore_byteswap.h header file with the following
functions:
* _Py_bswap16()
* _Py_bswap32()
* _Py_bswap64()
Use these functions in _ctypes, sha256 and sha512 modules,
and also use in the UTF-32 encoder.
sha256, sha512 and _ctypes modules are now built with the internal
C API.
Rename _PyInterpreterState_GET_UNSAFE() to _PyInterpreterState_GET()
for consistency with _PyThreadState_GET() and to have a shorter name
(help to fit into 80 columns).
Add also "assert(tstate != NULL);" to the function.
Don't access PyInterpreterState.config member directly anymore, but
use new functions:
* _PyInterpreterState_GetConfig()
* _PyInterpreterState_SetConfig()
* _Py_GetConfig()
* Keep constness of pointer objects.
Also moved an auto variable that got consted into its innermost necessary scope.
* move def
Co-authored-by: Benjamin Peterson <benjamin@python.org>
Always declare PyIndex_Check() as an opaque function to hide
implementation details: remove PyIndex_Check() macro. The macro
accessed directly the PyTypeObject.tp_as_number member.
Add _PyIndex_Check() function to the internal C API: fast inlined
verson of PyIndex_Check().
Add Include/internal/pycore_abstract.h header file.
Replace PyIndex_Check() with _PyIndex_Check() in C files of Objects
and Python subdirectories.
PyType_HasFeature() now always calls PyType_GetFlags() to hide
implementation details. Previously, it accessed directly the
PyTypeObject.tp_flags member when the limited C API was not used.
Add fast inlined version _PyType_HasFeature() and _PyType_IS_GC()
for object.c and typeobject.c.
The PyObject_NEW() macro becomes an alias to the PyObject_New()
macro, and the PyObject_NEW_VAR() macro becomes an alias to the
PyObject_NewVar() macro, to hide implementation details. They no
longer access directly the PyTypeObject.tp_basicsize member.
Exclude _PyObject_SIZE() and _PyObject_VAR_SIZE() macros from
the limited C API.
Replace PyObject_NEW() with PyObject_New() and replace
PyObject_NEW_VAR() with PyObject_NewVar().
This implements things like `list[int]`,
which returns an object of type `types.GenericAlias`.
This object mostly acts as a proxy for `list`,
but has attributes `__origin__` and `__args__`
that allow recovering the parts (with values `list` and `(int,)`.
There is also an approximate notion of type variables;
e.g. `list[T]` has a `__parameters__` attribute equal to `(T,)`.
Type variables are objects of type `typing.TypeVar`.
str.encode() and str.decode() no longer check the encoding and errors
in development mode or in debug mode during Python finalization. The
codecs machinery can no longer work on very late calls to
str.encode() and str.decode().
This change should help to call _PyObject_Dump() to debug during late
Python finalization.
Convert the PyObject_GET_WEAKREFS_LISTPTR() macro to a function to
hide implementation details: the macro accessed directly to the
PyTypeObject.tp_weaklistoffset member.
Add _PyObject_GET_WEAKREFS_LISTPTR() static inline function to the
internal C API.
Speed up calls to list() by using the PEP 590 vectorcall
calling convention. Patch by Mark Shannon.
Co-authored-by: Mark Shannon <mark@hotpy.org>
Co-authored-by: Dong-hee Na <donghee.na92@gmail.com>
Extension modules: m_traverse, m_clear and m_free functions of
PyModuleDef are no longer called if the module state was requested
but is not allocated yet. This is the case immediately after the
module is created and before the module is executed (Py_mod_exec
function). More precisely, these functions are not called if m_size is
greater than 0 and the module state (as returned by
PyModule_GetState()) is NULL.
Extension modules without module state (m_size <= 0) are not affected.
Co-Authored-By: Petr Viktorin <encukou@gmail.com>
Replace _PyInterpreterState_Get() function call with
_PyInterpreterState_GET_UNSAFE() macro which is more efficient but
don't check if tstate or interp is NULL.
_Py_GetConfigsAsDict() now uses _PyThreadState_GET().
Py_TRASHCAN_BEGIN_CONDITION and Py_TRASHCAN_END macro no longer
access PyThreadState attributes, but call new private
_PyTrash_begin() and _PyTrash_end() functions which hide
implementation details.
Move the static inline function flavor of Py_EnterRecursiveCall() and
Py_LeaveRecursiveCall() to the internal C API: they access
PyThreadState attributes. The limited C API provides regular
functions which hide implementation details.
We make `|=` raise TypeError, since it would be surprising if `C.__dict__ |= {'x': 0}` silently did nothing, while `C.__dict__.update({'x': 0})` is an error.
The Py_FatalError() function is replaced with a macro which logs
automatically the name of the current function, unless the
Py_LIMITED_API macro is defined.
Changes:
* Add _Py_FatalErrorFunc() function.
* Remove the function name from the message of Py_FatalError() calls
which included the function name.
* Update tests.
This continues the `range()` part of #13930. The complete pull request is stalled on discussions around dicts, but `range()` should not be controversial. (And I plan to open PRs for other parts if this is merged.)
On top of Mark's change, I unified `range_new` and `range_vectorcall`, which had a lot of duplicate code.
https://bugs.python.org/issue37207
The constants `RESTRICTED` and `PY_WRITE_RESTRICTED` no longer have a meaning in Python 3. Therefore, CPython should not use them.
CC @matrixise
https://bugs.python.org/issue36347
The fix for [bpo-39386](https://bugs.python.org/issue39386) attempted to make it so you couldn't reuse a
agen.aclose() coroutine object. It accidentally also prevented you
from calling aclose() at all on an async generator that was already
closed or exhausted. This commit fixes it so we're only blocking the
actually illegal cases, while allowing the legal cases.
The new tests failed before this patch. Also confirmed that this fixes
the test failures we were seeing in Trio with Python dev builds:
https://github.com/python-trio/trio/pull/1396https://bugs.python.org/issue39606
Move the dtoa.h header file to the internal C API as pycore_dtoa.h:
it only contains private functions (prefixed by "_Py").
The math and cmath modules must now be compiled with the
Py_BUILD_CORE macro defined.
Move the bytes_methods.h header file to the internal C API as
pycore_bytes_methods.h: it only contains private symbols (prefixed by
"_Py"), except of the PyDoc_STRVAR_shared() macro.
gcc -Wcast-qual turns up a number of instances of casting away constness of pointers. Some of these can be safely modified, by either:
Adding the const to the type cast, as in:
- return _PyUnicode_FromUCS1((unsigned char*)s, size);
+ return _PyUnicode_FromUCS1((const unsigned char*)s, size);
or, Removing the cast entirely, because it's not necessary (but probably was at one time), as in:
- PyDTrace_FUNCTION_ENTRY((char *)filename, (char *)funcname, lineno);
+ PyDTrace_FUNCTION_ENTRY(filename, funcname, lineno);
These changes will not change code, but they will make it much easier to check for errors in consts
The bulk of this patch was generated automatically with:
for name in \
PyObject_Vectorcall \
Py_TPFLAGS_HAVE_VECTORCALL \
PyObject_VectorcallMethod \
PyVectorcall_Function \
PyObject_CallOneArg \
PyObject_CallMethodNoArgs \
PyObject_CallMethodOneArg \
;
do
echo $name
git grep -lwz _$name | xargs -0 sed -i "s/\b_$name\b/$name/g"
done
old=_PyObject_FastCallDict
new=PyObject_VectorcallDict
git grep -lwz $old | xargs -0 sed -i "s/\b$old\b/$new/g"
and then cleaned up:
- Revert changes to in docs & news
- Revert changes to backcompat defines in headers
- Nudge misaligned comments
Add a fast-path for UTF-8 encoding in PyUnicode_EncodeFSDefault()
and PyUnicode_DecodeFSDefaultAndSize().
Add _PyUnicode_FiniEncodings() helper function for _PyUnicode_Fini().
In the limited C API, PyObject_INIT() and PyObject_INIT_VAR() are now
defined as aliases to PyObject_Init() and PyObject_InitVar() to make
their implementation opaque. It avoids to leak implementation details
in the limited C API.
Exclude the following functions from the limited C API, move them
from object.h to cpython/object.h:
* _Py_NewReference()
* _Py_ForgetReference()
* _PyTraceMalloc_NewReference()
* _Py_GetRefTotal()
The macro is defined after Py_DECREF() and so is no longer used by
Py_DECREF().
Moving _Py_Dealloc() macro back from cpython/object.h to object.h
would require to move a lot of definitions as well: PyTypeObject and
many related types used by PyTypeObject.
Keep _Py_Dealloc() as an opaque function call to avoid leaking
implementation details in the limited C API (object.h): remove
_Py_Dealloc() macro from cpython/object.h.
_Py_NewReference() becomes a regular opaque function, rather than a
static inline function in the C API (object.h), to better hide
implementation details.
Move _Py_tracemalloc_config from public pymem.h to internal
pycore_pymem.h header.
Make _Py_AddToAllObjects() private.
Restore PyObject_IsInstance() comment explaining why only tuples of
types are accepted, but not general sequence. Comment written by
Guido van Rossum in commit 03290ecbf1
which implements isinstance(x, (A, B, ...)). The comment was lost in
a PyObject_IsInstance() optimization:
commit ec569b7947.
Cleanup also the code. recursive_isinstance() is no longer recursive,
so rename it to object_isinstance(), whereas object_isinstance() is
recursive and so rename it to object_recursive_isinstance().
* Remove _Py_INC_REFTOTAL and _Py_DEC_REFTOTAL macros: modify
directly _Py_RefTotal.
* _Py_ForgetReference() is no longer defined if the Py_TRACE_REFS
macro is not defined.
* Remove _Py_NewReference() implementation from object.c:
unify the two implementations in object.h inline function.
* Fix Py_TRACE_REFS build: _Py_INC_TPALLOCS() macro has been removed.
Replace Py_FatalError() calls with _PyErr_WriteUnraisableMsg(),
_PyObject_ASSERT_FAILED_MSG() or Py_UNREACHABLE()
in unicode_dealloc() and unicode_release_interned().
Rename init_slotdefs() to _PyTypes_InitSlotDefs() and add a return
value of type PyStatus. The function is now called exactly once from
_PyTypes_Init(). Replace calls to init_slotdefs() with an assertion
checking that slotdefs is initialized.
Replace Py_FatalError() with _PyObject_ASSERT_FAILED_MSG() in
object.c and typeobject.c to also dump the involved Python object on
a fatal error. It should ease debug when such fatal error occurs.
If the double linked list is inconsistent, _Py_ForgetReference() no
longer dumps previous and next objects in the fatal error, it now
only dumps the current object. It ensures that the error message
is displayed even if dumping the object does crash Python.
Enhance _Py_ForgetReference() error messages;
_PyObject_ASSERT_FAILED_MSG() logs the "_Py_ForgetReference" function
name.
Improve multi-threaded performance by dropping the GIL in the fast path
of bytes.join. To avoid increasing overhead for small joins, it is only
done if the output size exceeds a threshold.
intern_strings() now raises a SystemError, rather than calling
Py_FatalError().
intern_string_constants() now reports exceptions to the caller,
rather than ignoring silently exceptions.
If the export count is negative, _memory_release() now raises a
SystemError and returns -1, rather than calling Py_FatalError()
which aborts the process.
If PyModule_Create2() is called when the Python import machinery is
not initialized, it now raises a SystemError and returns NULL,
instead of calling Py_FatalError() which aborts the process.
The caller must be prepared to handle NULL anyway.
Replace int with intptr_t to fix the warning:
objects\frameobject.c(341): warning C4244: 'initializing':
conversion from '__int64' to 'int', possible loss of data
The public API symbols being removed are:
_PyBytes_InsertThousandsGroupingLocale, _PyBytes_InsertThousandsGrouping, _Py_InitializeFromArgs, _Py_InitializeFromWideArgs, _PyFloat_Repr, _PyFloat_Digits,
_PyFloat_DigitsInit, PyFrame_ExtendStack, _PyAIterWrapper_Type, PyNullImporter_Type, PyCmpWrapper_Type, PySortWrapper_Type, PyNoArgsFunction.
Niklas Fiekas
Pablo Galindo
Dennis Sweeney
Ammar Askar
Victor Stinner
scoder
Mark Shannon
Serhiy Storchaka
Mark Dickinson
Dong-hee Na
Sergey Fedoseev
Zackery Spytz
Hai Shi
Chris Jerdonek
Raymond Hettinger
Petr Viktorin
Ethan Smith
Andy Lester
Guido van Rossum
Inada Naoki
Brandt Bucher
Benjamin Peterson
MojoVampire
Stefan Krah
Yonatan Goldschmidt
Jeroen Demeyer
Nathaniel J. Smith
Stefan Pochmann
Bruce Merry
Andrew Svetlov
Anthony Sottile