We only statically initialize for core code and builtin modules. Extension modules still create
the tuple at runtime. We'll solve that part of interpreter isolation separately.
This change includes generated code. The non-generated changes are in:
* Tools/clinic/clinic.py
* Python/getargs.c
* Include/cpython/modsupport.h
* Makefile.pre.in (re-generate global strings after running clinic)
* very minor tweaks to Modules/_codecsmodule.c and Python/Python-tokenize.c
All other changes are generated code (clinic, global strings).
* Store tp_weaklist on the interpreter state for static builtin types.
* Factor out _PyStaticType_GET_WEAKREFS_LISTPTR().
* Add _PyStaticType_ClearWeakRefs().
* Add a comment about how _PyStaticType_ClearWeakRefs() loops.
* Document the change.
* Update Doc/whatsnew/3.12.rst
* Fix a typo.
This is the last precursor to storing tp_subclasses (and tp_weaklist) on the interpreter state for static builtin types.
Here we add per-type storage on PyInterpreterState, but only for the static builtin types. This involves the following:
* add PyInterpreterState.types
* move PyInterpreterState.type_cache to it
* add a "num_builtins_initialized" field
* add a "builtins" field (a static array big enough for all the static builtin types)
* add _PyStaticType_GetState() to look up a static builtin type's state
* (temporarily) add PyTypeObject.tp_static_builtin_index (to hold the type's index into PyInterpreterState.types.builtins)
We will be eliminating tp_static_builtin_index in a later change.
* Add _Py_memory_repeat function to pycore_list
* Add _Py_RefcntAdd function to pycore_object
* Use the new functions in tuplerepeat, list_repeat, and list_inplace_repeat
This is the first of several precursors to storing tp_subclasses (and tp_weaklist) on the interpreter state for static builtin types.
We do the following:
* add `_PyStaticType_InitBuiltin()`
* add `_Py_TPFLAGS_STATIC_BUILTIN`
* set it on all static builtin types in `_PyStaticType_InitBuiltin()`
* shuffle some code around to be able to use _PyStaticType_InitBuiltin()
* rename `_PyStructSequence_InitType()` to `_PyStructSequence_InitBuiltinWithFlags()`
* add `_PyStructSequence_InitBuiltin()`.
Move the follow functions and type from frameobject.h to pyframe.h,
so the standard <Python.h> provide frame getter functions:
* PyFrame_Check()
* PyFrame_GetBack()
* PyFrame_GetBuiltins()
* PyFrame_GetGenerator()
* PyFrame_GetGlobals()
* PyFrame_GetLasti()
* PyFrame_GetLocals()
* PyFrame_Type
Remove #include "frameobject.h" from many C files. It's no longer
needed.
Deprecate global configuration variable like
Py_IgnoreEnvironmentFlag: the Py_InitializeFromConfig() API should be
instead.
Fix declaration of Py_GETENV(): use PyAPI_FUNC(), not PyAPI_DATA().
It combines PyImport_ImportModule() and PyObject_GetAttrString()
and saves 4-6 lines of code on every use.
Add also _PyImport_GetModuleAttr() which takes Python strings as arguments.
Add C++ overloads for _Py_CAST_impl() to handle 0/NULL. This will allow
C++ extensions that pass 0 or NULL to macros using _Py_CAST() to
continue to compile. Without this, you get an error like:
invalid ‘static_cast’ from type ‘int’ to type ‘_object*’
The modern way to use a NULL value in C++ is to use nullptr. However,
we want to not break extensions that do things the old way.
Co-authored-by: serge-sans-paille
This was added for bpo-40514 (gh-84694) to test out a per-interpreter GIL. However, it has since proven unnecessary to keep the experiment in the repo. (It can be done as a branch in a fork like normal.) So here we are removing:
* the configure option
* the macro
* the code enabled by the macro
Added a new stable API function ``PyType_FromMetaclass``, which mirrors
the behavior of ``PyType_FromModuleAndSpec`` except that it takes an
additional metaclass argument. This is, e.g., useful for language
binding tools that need to store additional information in the type
object.
Python now always use the ``%zu`` and ``%zd`` printf formats to
format a size_t or Py_ssize_t number. Building Python 3.12 requires a
C11 compiler, so these printf formats are now always supported.
* PyObject_Print() and _PyObject_Dump() now use the printf %zd format
to display an object reference count.
* Update PY_FORMAT_SIZE_T comment.
* Remove outdated notes about the %zd format in PyBytes_FromFormat()
and PyUnicode_FromFormat() documentations.
* configure no longer checks for the %zd format and no longer defines
PY_FORMAT_SIZE_T macro in pyconfig.h.
* pymacconfig.h no longer undefines PY_FORMAT_SIZE_T: macOS 10.4 is
no longer supported. Python 3.12 now requires macOS 10.6 (Snow
Leopard) or newer.
Also while there, clarify a few things about why we reduce the hash to 32 bits.
Co-authored-by: Eli Libman <eli@hyro.ai>
Co-authored-by: Yury Selivanov <yury@edgedb.com>
Co-authored-by: Łukasz Langa <lukasz@langa.pl>
Update documentation of PyUnicode_DecodeFSDefault(),
PyUnicode_DecodeFSDefaultAndSize() and PyUnicode_EncodeFSDefault():
they now use the filesystem encoding and error handler of PyConfig,
Py_FileSystemDefaultEncoding and Py_FileSystemDefaultEncodeErrors
variables are no longer used.
Avoid mixing declarations and code in the C API to fix the compiler
warning: "ISO C90 forbids mixed declarations and code"
[-Werror=declaration-after-statement].
Remove the PyUnicode_InternImmortal() function and the
SSTATE_INTERNED_IMMORTAL macro.
The PyUnicode_InternImmortal() function is still exported in the
stable ABI. The function is removed from the API.
PyASCIIObject.state.interned size is now a single bit, rather than 2
bits.
Keep SSTATE_NOT_INTERNED and SSTATE_INTERNED_MORTAL macros for
backward compatibility, but no longer use them internally since the
interned member is now a single bit and so can only have two values
(interned or not interned).
Update stats of _PyUnicode_ClearInterned().
In the limited C API version 3.12, PyUnicode_KIND() is now
implemented as a static inline function. Keep the macro for the
regular C API and for the limited C API version 3.11 and older to
prevent introducing new compiler warnings.
Update _decimal.c and stringlib/eq.h for PyUnicode_KIND().
Use _Py_CAST() and _Py_STATIC_CAST() in macros wrapping static inline
functions of unicodeobject.h.
Change also the kind type from unsigned int to int: same parameter
type than PyUnicode_FromKindAndData().
The limited API version 3.11 no longer casts arguments to expected
types.
Use the PyObject* type for parameters of static inline functions:
* Py_SIZE(): same parameter type than PyObject_Size()
* PyList_GET_SIZE(), PyList_SET_ITEM(): same parameter type than
PyList_Size() and PyList_SetItem()
* PyTuple_GET_SIZE(), PyTuple_SET_ITEM(): same parameter type than
PyTuple_Size() and PyTuple_SetItem().
The limited API version 3.11 no longer casts arguments to expected
types of functions of functions:
* PyList_GET_SIZE(), PyList_SET_ITEM()
* PyTuple_GET_SIZE(), PyTuple_SET_ITEM()
* PyWeakref_GET_OBJECT()
Convert the following Unicode macros to static inline functions.
Surrogate functions:
* Py_UNICODE_IS_SURROGATE()
* Py_UNICODE_IS_HIGH_SURROGATE()
* Py_UNICODE_IS_LOW_SURROGATE()
* Py_UNICODE_HIGH_SURROGATE()
* Py_UNICODE_LOW_SURROGATE()
* Py_UNICODE_JOIN_SURROGATES()
"Is" functions:
* Py_UNICODE_ISALNUM()
* Py_UNICODE_ISSPACE()
In the implementation of these functions, the character type is now
well defined to Py_UCS4.
Convert the following macros to static inline functions:
* PyCell_GET()
* PyCell_SET()
Limited C API version 3.12 no longer casts arguments.
Fix also usage of PyCell_SET(): only delete the old value after
setting the new value.
Remove the token.h header file. There was never any public tokenizer
C API. The token.h header file was only designed to be used by Python
internals.
Move Include/token.h to Include/internal/pycore_token.h. Including
this header file now requires that the Py_BUILD_CORE macro is
defined. It no longer checks for the Py_LIMITED_API macro.
Rename functions:
* PyToken_OneChar() => _PyToken_OneChar()
* PyToken_TwoChars() => _PyToken_TwoChars()
* PyToken_ThreeChars() => _PyToken_ThreeChars()
Convert the following macros to static inline functions:
* _Py_AS_GC()
* _PyGCHead_FINALIZED(), _PyGCHead_SET_FINALIZED()
* _PyGCHead_NEXT(), _PyGCHead_SET_NEXT()
* _PyGCHead_PREV(), _PyGCHead_SET_PREV()
* _PyGC_FINALIZED(), _PyGC_SET_FINALIZED()
* _PyObject_GC_IS_TRACKED()
* _PyObject_GC_MAY_BE_TRACKED()
Add a macro wrapping the _PyObject_GC_IS_TRACKED() function to cast
the argument to PyObject*.
In C++, the _PyObject_EXTRA_INIT macro now uses nullptr, rather than
0, to initialize the _ob_next and _ob_prev members of the PyObject
structure.
Fix test_cppext failure when Python is built with
./configure --with-trace-refs.
Add the -P command line option and the PYTHONSAFEPATH environment
variable to not prepend a potentially unsafe path to sys.path.
* Add sys.flags.safe_path flag.
* Add PyConfig.safe_path member.
* Programs/_bootstrap_python.c uses config.safe_path=0.
* Update subprocess._optim_args_from_interpreter_flags() to handle
the -P command line option.
* Modules/getpath.py sets safe_path to 1 if a "._pth" file is
present.
Convert the following macros to static inline functions:
* PyCFunction_GET_CLASS()
* PyCFunction_GET_FLAGS()
* PyCFunction_GET_FUNCTION()
* PyCFunction_GET_SELF()
Limited C API version 3.11 no longer casts arguments.
Currently, calling Py_EnterRecursiveCall() and
Py_LeaveRecursiveCall() may use a function call or a static inline
function call, depending if the internal pycore_ceval.h header file
is included or not. Use a different name for the static inline
function to ensure that the static inline function is always used in
Python internals for best performance. Similar approach than
PyThreadState_GET() (function call) and _PyThreadState_GET() (static
inline function).
* Rename _Py_EnterRecursiveCall() to _Py_EnterRecursiveCallTstate()
* Rename _Py_LeaveRecursiveCall() to _Py_LeaveRecursiveCallTstate()
* pycore_ceval.h: Rename Py_EnterRecursiveCall() to
_Py_EnterRecursiveCall() and Py_LeaveRecursiveCall() and
_Py_LeaveRecursiveCall()
Fix C++ compiler warnings: "zero as null pointer constant"
(clang -Wzero-as-null-pointer-constant).
* Add the _Py_NULL macro used by static inline functions to use
nullptr in C++.
* Replace NULL with nullptr in _testcppext.cpp.
Use _Py_CAST(), _Py_STATIC_CAST() and _PyASCIIObject_CAST() in
static inline functions to fix C++ compiler warnings:
"use of old-style cast" (clang -Wold-style-cast).
test_cppext now builds the C++ test extension with -Wold-style-cast.
Convert the following macros to static inline functions:
* PyByteArray_AS_STRING()
* PyByteArray_GET_SIZE()
* PyBytes_AS_STRING()
* PyBytes_GET_SIZE()
Limited C API version 3.11 no longer casts arguments.
Add _PyBytes_CAST() and _PyByteArray_CAST() macros.
Fix C++ compiler warnings on cast macros, like _PyObject_CAST(), when
casting a constant expression to a non constant type: use
const_cast<> in C++.
* In C++, Py_SAFE_DOWNCAST() now uses static_cast<> rather than
reinterpret_cast<>.
* Add tests to the _testcppext C++ extension.
* test_cppext no longer captures stdout in verbose mode.
When Python is built with "./configure --enable-pystats" (if the
Py_STATS macro is defined), the _Py_GetSpecializationStats() function
must be exported, since it's used by the _opcode extension which is
built as a shared library.
Fix C++ compiler warnings about "old-style cast"
(g++ -Wold-style-cast) in the Python C API. Use C++
reinterpret_cast<> and static_cast<> casts when the Python C API is
used in C++.
Example of fixed warning:
Include/object.h:107:43: error: use of old-style cast to
‘PyObject*’ {aka ‘struct _object*’} [-Werror=old-style-cast]
#define _PyObject_CAST(op) ((PyObject*)(op))
Add _Py_reinterpret_cast() and _Py_static_cast() macros.
Move the following API from Include/opcode.h (public C API) to a new
Include/internal/pycore_opcode.h header file (internal C API):
* EXTRA_CASES
* _PyOpcode_Caches
* _PyOpcode_Deopt
* _PyOpcode_Jump
* _PyOpcode_OpName
* _PyOpcode_RelativeJump
Fix signal.NSIG value on FreeBSD to accept signal numbers greater
than 32, like signal.SIGRTMIN and signal.SIGRTMAX.
* Add Py_NSIG constant.
* Add pycore_signal.h internal header file.
* _Py_Sigset_Converter() now includes the range of valid signals in
the error message.
In the limited C API version 3.11 and newer, the following functions
no longer cast their object pointer argument with _PyObject_CAST() or
_PyObject_CAST_CONST():
* Py_REFCNT(), Py_TYPE(), Py_SIZE()
* Py_SET_REFCNT(), Py_SET_TYPE(), Py_SET_SIZE()
* Py_IS_TYPE()
* Py_INCREF(), Py_DECREF()
* Py_XINCREF(), Py_XDECREF()
* Py_NewRef(), Py_XNewRef()
* PyObject_TypeCheck()
* PyType_Check()
* PyType_CheckExact()
Split Py_DECREF() implementation in 3 versions to make the code more
readable.
Update the xxlimited.c extension, which uses the limited C API
version 3.11, to pass PyObject* to these functions.
Deprecate functions:
* PyUnicode_AS_DATA()
* PyUnicode_AS_UNICODE()
* PyUnicode_GET_DATA_SIZE()
* PyUnicode_GET_SIZE()
Previously, these functions were macros and so it wasn't possible to
decorate them with Py_DEPRECATED().
Convert unicodeobject.h macros to static inline functions:
* PyUnicode_AS_DATA()
* PyUnicode_AS_UNICODE()
* PyUnicode_GET_DATA_SIZE()
* PyUnicode_GET_SIZE()
Static inline functions are wrapped by macros which casts arguments
with _PyObject_CAST() to prevent introducing new compiler warnings
when passing "const PyObject*".
Py_REFCNT(), Py_TYPE(), Py_SIZE() and Py_IS_TYPE() functions argument
type is now "PyObject*", rather than "const PyObject*".
* Replace also "const PyObject*" with "PyObject*" in functions:
* _Py_strhex_impl()
* _Py_strhex_with_sep()
* _Py_strhex_bytes_with_sep()
* Remove _PyObject_CAST_CONST() and _PyVarObject_CAST_CONST() macros.
* Py_IS_TYPE() can now use Py_TYPE() in its implementation.
Convert unicodeobject.h macros to static inline functions:
* PyUnicode_MAX_CHAR_VALUE()
* PyUnicode_READ()
* PyUnicode_READY()
* PyUnicode_READ_CHAR()
* PyUnicode_WRITE()
Move PyUnicode_READY() after _PyUnicode_Ready(), since it uses
_PyUnicode_Ready().
Static inline functions are wrapped by macros which casts arguments
with _PyObject_CAST() and casts 'kind' arguments to "unsigned int" to
prevent introducing new compiler warnings when passing "const
PyObject*".
Apparently a switch on an 8-bit quantity where all cases are
present generates a more efficient jump (doing only one indexed
memory load instead of two).
So we make opcode and use_tracing uint8_t, and generate a macro
full of extra `case NNN:` lines for all unused opcodes.
See https://github.com/faster-cpython/ideas/issues/321#issuecomment-1103263673
* Stores all location info in linetable to conform to PEP 626.
* Remove column table from code objects.
* Remove end-line table from code objects.
* Document new location table format
Convert macros to static inline functions:
* PyTuple_GET_SIZE()
* PyTuple_SET_ITEM()
* PyList_GET_SIZE()
* PyList_SET_ITEM()
Add a macro converting arguments to PyTupleObject*, PyListObject* or
PyObject* to prevent emitting new compiler warnings.
According to PEP 670, PyTuple_GET_ITEM() and PyList_GET_ITEM() are
left as macros.
Convert the PyWeakref_GET_OBJECT() macro to a static inline function.
Add an assertion to check the argument with PyWeakref_Check(). Add a
macro converting the argument to PyObject* to prevent emitting new
compiler warning.
Convert unicodeobject.h macros to static inline functions:
* PyUnicode_CHECK_INTERNED()
* PyUnicode_DATA(), _PyUnicode_COMPACT_DATA(),
_PyUnicode_NONCOMPACT_DATA()
* PyUnicode_GET_LENGTH()
* PyUnicode_IS_ASCII()
* PyUnicode_IS_COMPACT()
* PyUnicode_IS_COMPACT_ASCII()
* PyUnicode_IS_READY()
Reorder functions to declare functions before their first usage.
Static inline functions are wrapped by macros which casts
arguments with _PyObject_CAST() to prevent introducing
new compiler warnings when passing "const PyObject*".
Remove the Include/code.h header file. C extensions should only
include the main <Python.h> header file.
Python.h includes directly Include/cpython/code.h instead.
Copying and pickling instances of subclasses of builtin types
bytearray, set, frozenset, collections.OrderedDict, collections.deque,
weakref.WeakSet, and datetime.tzinfo now copies and pickles instance attributes
implemented as slots.
* Revert "bpo-46850: Move _PyInterpreterState_SetEvalFrameFunc() to internal C API (GH-32054)"
This reverts commit f877b40e3f.
* Revert "bpo-46850: Move _PyEval_EvalFrameDefault() to internal C API (GH-32052)"
This reverts commit b9a5522dd9.
The fact interpreter frames were split out from full frame objects
rather than always being part of the eval loop implementation means
that it's tricky to infer the expected naming conventions simply
from looking at the code.
Documenting the de facto conventions in pycore_frame.h means future
readers of the code will have a clear explanation of the rationale
for those conventions (i.e. minimising non-functional code churn).
Move the private _PyFrameEvalFunction type, and private
_PyInterpreterState_GetEvalFrameFunc() and
_PyInterpreterState_SetEvalFrameFunc() functions to the internal C
API. The _PyFrameEvalFunction callback function type now uses the
_PyInterpreterFrame type which is part of the internal C API.
Update the _PyFrameEvalFunction documentation.
Move the private undocumented _PyEval_EvalFrameDefault() function to
the internal C API. The function now uses the _PyInterpreterFrame
type which is part of the internal C API.
Add _PyCFunctionObject_CAST() and _PyCMethodObject_CAST() macros to
make macros casting their argument easier to read, but also to check
the type of their input in debug mode: assert(PyCFunction_Check(func)
and assert(PyCMethod_Check(func).
Reformat also PyCFunction_XXX() macros for readability.
Add macros to cast objects to PyASCIIObject*, PyCompactUnicodeObject*
and PyUnicodeObject*: _PyASCIIObject_CAST(),
_PyCompactUnicodeObject_CAST() and _PyUnicodeObject_CAST(). Using
these new macros make the code more readable and check their argument
with: assert(PyUnicode_Check(op)).
Remove redundant assert(PyUnicode_Check(op)) in macros using directly
or indirectly these new CAST macros.
Replacing existing casts with these macros.
* `PyFrame_FastToLocalsWithError` and `PyFrame_LocalsToFast` are no longer called during profile and tracing.
(Contributed by Fabio Zadrozny)
* Make accesses to a frame's `f_locals` safe from C code, not relying on calls to `PyFrame_FastToLocals` or `PyFrame_LocalsToFast`.
* Document new `PyFrame_GetLocals` C-API function.
In a gh-32003 comment, I realized it wasn't very clear how _Py_DECLARE_STR() should be used. This changes adds a comment to clarify.
https://bugs.python.org/issue46541
* Moves the bytecode to the end of the corresponding PyCodeObject, and quickens it in-place.
* Removes the almost-always-unused co_varnames, co_freevars, and co_cellvars member caches
* _PyOpcode_Deopt is a new mapping from all opcodes to their un-quickened forms.
* _PyOpcode_InlineCacheEntries is renamed to _PyOpcode_Caches
* _Py_IncrementCountAndMaybeQuicken is renamed to _PyCode_Warmup
* _Py_Quicken is renamed to _PyCode_Quicken
* _co_quickened is renamed to _co_code_adaptive (and is now a read-only memoryview).
* Do not emit unused nonzero opargs anymore in the compiler.
Remove the private undocumented function
_PyEval_GetCoroutineOriginTrackingDepth() from the C API. Call the
public sys.get_coroutine_origin_tracking_depth() function instead.
Change the internal function
_PyEval_SetCoroutineOriginTrackingDepth():
* Remove the 'tstate' parameter;
* Add return value and raises an exception if depth is negative;
* No longer export the function: call the public
sys.set_coroutine_origin_tracking_depth() function instead.
Uniformize also function declarations in pycore_ceval.h.
Remove the following private undocumented functions from the C API:
* _PyEval_GetAsyncGenFirstiter()
* _PyEval_GetAsyncGenFinalizer()
* _PyEval_SetAsyncGenFirstiter()
* _PyEval_SetAsyncGenFinalizer()
Call the public sys.get_asyncgen_hooks() and sys.set_asyncgen_hooks()
functions instead.
Add new functions to pack and unpack C double (serialize and
deserialize):
* PyFloat_Pack2(), PyFloat_Pack4(), PyFloat_Pack8()
* PyFloat_Unpack2(), PyFloat_Unpack4(), PyFloat_Unpack8()
Document these functions and add unit tests.
Rename private functions and move them from the internal C API
to the public C API:
* _PyFloat_Pack2() => PyFloat_Pack2()
* _PyFloat_Pack4() => PyFloat_Pack4()
* _PyFloat_Pack8() => PyFloat_Pack8()
* _PyFloat_Unpack2() => PyFloat_Unpack2()
* _PyFloat_Unpack4() => PyFloat_Unpack4()
* _PyFloat_Unpack8() => PyFloat_Unpack8()
Replace the "unsigned char*" type with "char*" which is more common
and easy to use.
Add methods __typing_subst__() in TypeVar and ParamSpec.
Simplify code by using more object-oriented approach, especially
the C code for types.GenericAlias and the Python code for
collections.abc.Callable.
Rename also struct _cframe to struct _PyCFrame.
Add a comment suggesting using public functions rather than using
directly the private _PyCFrame structure.
Rename private functions (no exported), add an underscore prefix:
* PyLineTable_InitAddressRange() => _PyLineTable_InitAddressRange()
* PyLineTable_NextAddressRange() => _PyLineTable_NextAddressRange()
* PyLineTable_PreviousAddressRange() => _PyLineTable_PreviousAddressRange()
Move private functions to the internal C API:
* _PyCode_Addr2EndLine()
* _PyCode_Addr2EndOffset()
* _PyCode_Addr2Offset()
* _PyCode_InitAddressRange()
* _PyCode_InitEndAddressRange(
* _PyLineTable_InitAddressRange()
* _PyLineTable_NextAddressRange()
* _PyLineTable_PreviousAddressRange()
No longer export the following internal functions:
* _PyCode_GetVarnames()
* _PyCode_GetCellvars()
* _PyCode_GetFreevars()
* _Py_GetSpecializationStats()
Add "extern" to pycore_code.h functions to identify them more easiliy
(they are still not exported).
<stdbool.h> is the standard/modern way to define embedd/extends Python free to define bool, true and false, but there are existing applications that use slightly different redefinitions, which fail if the header is included.
It's OK to use stdbool outside the public headers, though.
https://bugs.python.org/issue46748
Move forward declarations of Python C API types to a new pytypedefs.h
header file to solve interdependency issues between header files.
pytypedefs.h contains forward declarations of the following types:
* PyCodeObject
* PyFrameObject
* PyGetSetDef
* PyInterpreterState
* PyLongObject
* PyMemberDef
* PyMethodDef
* PyModuleDef
* PyObject
* PyThreadState
* PyTypeObject
Remove the HAVE_PY_SET_53BIT_PRECISION macro (moved to the internal
C API).
* Move HAVE_PY_SET_53BIT_PRECISION macro to pycore_pymath.h.
* Replace PY_NO_SHORT_FLOAT_REPR macro with _PY_SHORT_FLOAT_REPR
macro which is always defined. gcc -Wundef emits a warning when
using _PY_SHORT_FLOAT_REPR but the macro is not defined, if
pycore_pymath.h include was forgotten.
Rename Include/buffer.h header file to Include/pybuffer.h to avoid
conflicts with projects having an existing "buffer.h" header file.
* Incude pybuffer.h before object.h in Python.h.
* Remove #include "buffer.h" from Include/cpython/object.h.
* Add a forward declaration of the PyObject type in pybuffer.h to fix
an inter-dependency issue.
Instead of manually enumerating the global strings in generate_global_objects.py, we extrapolate the list from usage of _Py_ID() and _Py_STR() in the source files.
This is partly inspired by gh-31261.
https://bugs.python.org/issue46541
When a static inline function is wrapped by a macro which casts its
arguments to the expected type, there is no need that the function
has a different name than the macro. Use the same name for the macro
and the function to avoid confusion.
Rename _PyUnicode_get_wstr_length() to PyUnicode_WSTR_LENGTH().
Don't rename static inline _Py_NewRef() and _Py_XNewRef() functions,
since the C API exports Py_NewRef() and Py_XNewRef() functions as
regular functions. The name cannot be reused in this case.
We're no longer using _Py_IDENTIFIER() (or _Py_static_string()) in any core CPython code. It is still used in a number of non-builtin stdlib modules.
The replacement is: PyUnicodeObject (not pointer) fields under _PyRuntimeState, statically initialized as part of _PyRuntime. A new _Py_GET_GLOBAL_IDENTIFIER() macro facilitates lookup of the fields (along with _Py_GET_GLOBAL_STRING() for non-identifier strings).
https://bugs.python.org/issue46541#msg411799 explains the rationale for this change.
The core of the change is in:
* (new) Include/internal/pycore_global_strings.h - the declarations for the global strings, along with the macros
* Include/internal/pycore_runtime_init.h - added the static initializers for the global strings
* Include/internal/pycore_global_objects.h - where the struct in pycore_global_strings.h is hooked into _PyRuntimeState
* Tools/scripts/generate_global_objects.py - added generation of the global string declarations and static initializers
I've also added a --check flag to generate_global_objects.py (along with make check-global-objects) to check for unused global strings. That check is added to the PR CI config.
The remainder of this change updates the core code to use _Py_GET_GLOBAL_IDENTIFIER() instead of _Py_IDENTIFIER() and the related _Py*Id functions (likewise for _Py_GET_GLOBAL_STRING() instead of _Py_static_string()). This includes adding a few functions where there wasn't already an alternative to _Py*Id(), replacing the _Py_Identifier * parameter with PyObject *.
The following are not changed (yet):
* stop using _Py_IDENTIFIER() in the stdlib modules
* (maybe) get rid of _Py_IDENTIFIER(), etc. entirely -- this may not be doable as at least one package on PyPI using this (private) API
* (maybe) intern the strings during runtime init
https://bugs.python.org/issue46541
Building Python now requires a C99 <math.h> header file providing a
NAN constant, or the __builtin_nan() built-in function. If a platform
does not support Not-a-Number (NaN), the Py_NO_NAN macro can be
defined in the pyconfig.h file.
This reduces the size of the data segment by **300 KB** of the executable because if the modules are deep-frozen then the marshalled frozen data just wastes space. This was inspired by comment by @gvanrossum in https://github.com/python/cpython/pull/29118#issuecomment-958521863. Note: There is a new option `--deepfreeze-only` in `freeze_modules.py` to change this behavior, it is on be default to save disk space.
```console
# du -s ./python before
27892 ./python
# du -s ./python after
27524 ./python
```
Automerge-Triggered-By: GH:ericsnowcurrently
* Add PRECALL_FUNCTION opcode.
* Move 'call shape' varaibles into struct.
* Replace CALL_NO_KW and CALL_KW with KW_NAMES and CALL instructions.
* Specialize for builtin methods taking using the METH_FASTCALL | METH_KEYWORDS protocol.
* Allow kwnames for specialized calls to builtin types.
* Specialize calls to tuple(arg) and str(arg).
Remove the PyHeapType_GET_MEMBERS() macro. It was exposed in the
public C API by mistake, it must only be used by Python internally.
Use the PyTypeObject.tp_members member instead.
Rename PyHeapType_GET_MEMBERS() to _PyHeapType_GET_MEMBERS() and move
it to the internal C API.
Move _Py_GetAllocatedBlocks() and _PyObject_DebugMallocStats()
declarations to pycore_pymem.h. These functions are related to memory
allocators, not to the PyObject structure.
This change is a prerequisite for generating code for other global objects (like strings in gh-30928).
(We borrowed some code from Tools/scripts/deepfreeze.py.)
https://bugs.python.org/issue46541
Convert the PyType_SUPPORTS_WEAKREFS() macro to a regular function.
It no longer access the PyTypeObject.tp_weaklistoffset member
directly.
Add _PyType_SUPPORTS_WEAKREFS() static inline functions, used
internally by Python for best performance.
Add _PyUnicode_FiniTypes() function, called by
finalize_interp_types(). It clears these static types:
* EncodingMapType
* PyFieldNameIter_Type
* PyFormatterIter_Type
_PyStaticType_Dealloc() now does nothing if tp_subclasses
is not NULL.
Add 'static_exceptions' list to factorize code between
_PyExc_InitTypes() and _PyBuiltins_AddExceptions().
_PyExc_InitTypes() does nothing if it's not the main interpreter.
Sort exceptions in Lib/test/exception_hierarchy.txt.
* Move PyContext static types into object.c static_types list.
* Rename PyContextTokenMissing_Type to _PyContextTokenMissing_Type
and declare it in pycore_context.h.
* _PyHamtItems types are no long exported: replace PyAPI_DATA() with
extern.
Add a new _PyType_GetSubclasses() function to get type's subclasses.
_PyType_GetSubclasses(type) returns a list which holds strong
refererences to subclasses. It is safer than iterating on
type->tp_subclasses which yields weak references and can be modified
in the loop.
_PyType_GetSubclasses(type) now holds a reference to the tp_subclasses
dict while creating the list of subclasses.
set_collection_flag_recursive() of _abc.c now uses
_PyType_GetSubclasses().
Add _PyTypes_FiniTypes() best-effort function to clear static types:
don't deallocate a type if it still has subclasses.
remove_subclass() now sets tp_subclasses to NULL when removing the
last subclass.
The _curses module now creates its ncurses_version type as a heap
type using PyStructSequence_NewType(), rather than using a static
type.
* Move _PyStructSequence_FiniType() definition to pycore_structseq.h.
* test.pythoninfo: log curses.ncurses_version.
Add _PyStructSequence_FiniType() and _PyStaticType_Dealloc()
functions to finalize a structseq static type in Py_Finalize().
Currrently, these functions do nothing if Python is built in release
mode.
Clear static types:
* AsyncGenHooksType: sys.set_asyncgen_hooks()
* FlagsType: sys.flags
* FloatInfoType: sys.float_info
* Hash_InfoType: sys.hash_info
* Int_InfoType: sys.int_info
* ThreadInfoType: sys.thread_info
* UnraisableHookArgsType: sys.unraisablehook
* VersionInfoType: sys.version
* WindowsVersionType: sys.getwindowsversion()
* Add RETURN_GENERATOR and JUMP_NO_INTERRUPT opcodes.
* Trim frame and generator by word each.
* Minor refactor of frame.c
* Update test.test_sys to account for smaller frames.
* Treat generator functions as normal functions when evaluating and specializing.
Previously, the main interpreter was allocated on the heap during runtime initialization. Here we instead embed it into _PyRuntimeState, which means it is statically allocated as part of the _PyRuntime global. The same goes for the initial thread state (of each interpreter, including the main one). Consequently there are fewer allocations during runtime/interpreter init, fewer possible failures, and better memory locality.
FYI, this also helps efforts to consolidate globals, which in turns helps work on subinterpreter isolation.
https://bugs.python.org/issue45953
The empty bytes object (b'') and the 256 one-character bytes objects were allocated at runtime init. Now we statically allocate and initialize them.
https://bugs.python.org/issue45953
Move almost all private functions of Include/cpython/fileutils.h to
the internal C API Include/internal/pycore_fileutils.h.
Only keep _Py_fopen_obj() in Include/cpython/fileutils.h, since it's
used by _testcapi which must not use the internal C API.
Move EncodeLocaleEx() and DecodeLocaleEx() functions from _testcapi
to _testinternalcapi, since the C API moved to the internal C API.
This reverts commit ea251806b8.
Keep "assert(interned == NULL);" in _PyUnicode_Fini(), but only for
the main interpreter.
Keep _PyUnicode_ClearInterned() changes avoiding the creation of a
temporary Python list object.
* bpo-46218: Change long_pow() to sliding window algorithm
The primary motivation is to eliminate long_pow's reliance on that the number of bits in a long "digit" is a multiple of 5. Now it no longer cares how many bits are in a digit.
But the sliding window approach also allows cutting the precomputed table of small powers in half, which reduces initialization overhead enough that the approach pays off for smaller exponents too. Depending on exponent bit patterns, a sliding window may also be able to save some bigint multiplies (sometimes when at least 5 consecutive exponent bits are 0, regardless of their starting bit position modulo 5).
Note: boosting the window width to 6 didn't work well overall. It give marginal speed improvements for huge exponents, but the increased overhead (the small-power table needs twice as many entries) made it a loss for smaller exponents.
Co-authored-by: Oleg Iarygin <dralife@yandex.ru>
* Do not PUSH/POP traceback or type to the stack as part of exc_info
* Remove exc_traceback and exc_type from _PyErr_StackItem
* Add to what's new, because this change breaks things like Cython
The array of small PyLong objects has been statically declared. Here I also statically initialize them. Consequently they are no longer initialized dynamically during runtime init.
I've also moved them under a new sub-struct in _PyRuntimeState, in preparation for static allocation and initialization of other global objects.
https://bugs.python.org/issue45953
When Python is embedded in other applications, it is not easy to determine which version of Python is being used. This change exposes the Python version as part of the API data. Tools like Austin (https://github.com/P403n1x87/austin) can benefit from this data when targeting applications like uWSGI, as the Python version can then be inferred systematically by looking at the exported symbols rather than relying on unreliable pattern matching or other hacks (like remote code execution etc...).
Automerge-Triggered-By: GH:pablogsal
This change is strictly renames and moving code around. It helps in the following ways:
* ensures type-related init functions focus strictly on one of the three aspects (state, objects, types)
* passes in PyInterpreterState * to all those functions, simplifying work on moving types/objects/state to the interpreter
* consistent naming conventions help make what's going on more clear
* keeping API related to a type in the corresponding header file makes it more obvious where to look for it
https://bugs.python.org/issue46008
Exclude the PyUnicode_CHECK_INTERNED() macro from the limited C API,
because it uses the PyASCIIObject structure which is excluded from
the limited C API.
Automerge-Triggered-By: GH:encukou
Previously, basic initialization of PyInterprterState happened in PyInterpreterState_New() (along with allocation and adding the new interpreter to the runtime state). This prevented us from initializing interpreter states that were allocated separately (e.g. statically or in a free list). We've addressed that here by factoring out a separate function just for initialization. We've done the same for PyThreadState. _PyRuntimeState was sorted out when we added it since _PyRuntime is statically allocated. However, here we update the existing init code to line up with the functions for PyInterpreterState and PyThreadState.
https://bugs.python.org/issue46008
PyInterpreterState_Main() is a plain function exposed in the public C-API. For internal usage we can take the more efficient approach in this PR.
https://bugs.python.org/issue46008
This simplifies new_threadstate(). We also rename _PyThreadState_Init() to _PyThreadState_SetCurrent() to reflect what it actually does.
https://bugs.python.org/issue46008
This parallels _PyRuntimeState.interpreters. Doing this helps make it more clear what part of PyInterpreterState relates to its threads.
https://bugs.python.org/issue46008
This falls into the category of keep-allocation-and-initialization separate. It also allows us to use _PyEval_InitState() safely in functions that return void.
https://bugs.python.org/issue46008
* Make generator, coroutine and async gen structs all the same size.
* Store interpreter frame in generator (and coroutine). Reduces the number of allocations neeeded for a generator from two to one.
The getpath.py file is frozen at build time and executed as code over a namespace. It is never imported, nor is it meant to be importable or reusable. However, it should be easier to read, modify, and patch than the previous code.
This commit attempts to preserve every previously tested quirk, but these may be changed in the future to better align platforms.
Rename PyConfig.no_debug_ranges to PyConfig.code_debug_ranges and
invert the value.
Document -X no_debug_ranges and PYTHONNODEBUGRANGES env var in
PyConfig.code_debug_ranges documentation.
* Split exit paths into exceptional and non-exceptional.
* Move exit tracing code to individual bytecodes.
* Wrap all trace entry and exit events in macros to make them clearer and easier to enhance.
* Move return sequence into RETURN_VALUE, YIELD_VALUE and YIELD_FROM. Distinguish between normal trace events and dtrace events.
The following internal macros can not longer be used as l-value:
* asdl_seq_GET()
* asdl_seq_GET_UNTYPED()
* asdl_seq_LEN()
They are modified to use the _Py_RVALUE() macro.
Add a new _Py_RVALUE() macro to prevent using an expression as an
l-value.
Replace a "(void)" cast with the _Py_RVALUE() macro in the following
macros:
* PyCell_SET()
* PyList_SET_ITEM()
* PyTuple_SET_ITEM()
* _PyGCHead_SET_FINALIZED()
* _PyGCHead_SET_NEXT()
* asdl_seq_SET()
* asdl_seq_SET_UNTYPED()
Add also parentheses around macro arguments in PyCell_SET() and
PyTuple_SET_ITEM() macros.
* Make internal APIs that take PyFrameConstructor take a PyFunctionObject instead.
* Add reference to function to frame, borrow references to builtins and globals.
* Add COPY_FREE_VARS instruction to allow specialization of calls to inner functions.
Mark Shannon
Irit Katriel
Kumar Aditya
Brandt Bucher
Matthias Görgens
Ken Jin
fluesvamp
Eric Snow
Petr Viktorin
Christian Heimes
Aivars Kalvāns
Pieter Eendebak
Pablo Galindo Salgado
Itamar Ostricher
Victor Stinner
Dennis Sweeney
Ken Jin
Ken Jin
Serhiy Storchaka
Neil Schemenauer
Will Hawkins
Wenzel Jakob
Yury Selivanov
serge-sans-paille
David CARLIER
Inada Naoki
Ganesh Kathiresan
Yurii Karabas
Guido van Rossum
John Belmonte
Hood Chatham
Nick Coghlan
neonene
Jeremy Kloth
jonasdlindner
penguin_wwy
Dong-hee Na
Oleg Iarygin
Brandt Bucher
Mark Dickinson
Tim Peters
David Hewitt
Raymond Hettinger
Matti Picus
Gabriele N. Tornetta
Benjamin Peterson
Steve Dower