The remove private _PyGILState_GetInterpreterStateUnsafe() function
from the public C API: move it the internal C API (pycore_pystate.h).
No longer export the function.
The remove private _Py_UniversalNewlineFgetsWithSize() function from
the public C API: move it the internal C API (pycore_fileutils.h).
No longer export the function.
Remove these private functions from the public C API:
* _PyRun_AnyFileObject()
* _PyRun_InteractiveLoopObject()
* _PyRun_SimpleFileObject()
* _Py_SourceAsString()
Move them to the internal C API: add a new pycore_pythonrun.h header
file. No longer export these functions.
Remove the private _Py_Identifier type and related private functions
from the public C API:
* _PyObject_GetAttrId()
* _PyObject_LookupSpecialId()
* _PyObject_SetAttrId()
* _PyType_LookupId()
* _Py_IDENTIFIER()
* _Py_static_string()
* _Py_static_string_init()
Move them to the internal C API: add a new pycore_identifier.h header
file. No longer export these functions.
* Rename _PyUnstable_GetUnaryIntrinsicName() to
PyUnstable_GetUnaryIntrinsicName()
* Rename _PyUnstable_GetBinaryIntrinsicName()
to PyUnstable_GetBinaryIntrinsicName().
Move these private functions to the internal C API
(pycore_abstract.h):
* _Py_convert_optional_to_ssize_t()
* _PyNumber_Index()
Argument Clinic now emits #include "pycore_abstract.h" when these
functions are used.
The parser of the c-analyzer tool now uses a list of files which use
the limited C API, rather than a list of files using the internal C
API.
Move the private _PyLong converter functions to the internal C API
* _PyLong_FileDescriptor_Converter(): moved to pycore_fileutils.h
* _PyLong_Size_t_Converter(): moved to pycore_long.h
Argument Clinic now emits includes for pycore_fileutils.h and
pycore_long.h when these functions are used.
Move these private functions to the internal C API (pycore_long.h):
* _PyLong_UnsignedInt_Converter()
* _PyLong_UnsignedLongLong_Converter()
* _PyLong_UnsignedLong_Converter()
* _PyLong_UnsignedShort_Converter()
Argument Clinic now emits #include "pycore_long.h" when these
functions are used.
Instead of using `GO_TO_INSTRUCTION(CALL_PY_EXACT_ARGS)` we just add the macro elements of the latter to the macro for the former. This requires lengthening the uops array in struct opcode_macro_expansion. (It also required changes to stacking.py that were merged already.)
Move private functions to the internal C API (pycore_sysmodule.h):
* _PySys_GetAttr()
* _PySys_GetSizeOf()
No longer export most of these functions.
Fix also a typo in Include/cpython/optimizer.h: add a missing space.
Move private functions to the internal C API (pycore_dict.h):
* _PyDictView_Intersect()
* _PyDictView_New()
* _PyDict_ContainsId()
* _PyDict_DelItemId()
* _PyDict_DelItem_KnownHash()
* _PyDict_GetItemIdWithError()
* _PyDict_GetItem_KnownHash()
* _PyDict_HasSplitTable()
* _PyDict_NewPresized()
* _PyDict_Next()
* _PyDict_Pop()
* _PyDict_SetItemId()
* _PyDict_SetItem_KnownHash()
* _PyDict_SizeOf()
No longer export most of these functions.
Move also the _PyDictViewObject structure to the internal C API.
Move dict_getitem_knownhash() function from _testcapi to the
_testinternalcapi extension. Update test_capi.test_dict for this
change.
Move private _PyEval functions to the internal C API
(pycore_ceval.h):
* _PyEval_GetBuiltin()
* _PyEval_GetBuiltinId()
* _PyEval_GetSwitchInterval()
* _PyEval_MakePendingCalls()
* _PyEval_SetProfile()
* _PyEval_SetSwitchInterval()
* _PyEval_SetTrace()
No longer export most of these functions.
106320: Remove private float C API functions
Remove private C API functions:
* _Py_parse_inf_or_nan()
* _Py_string_to_number_with_underscores()
Move these functions to the internal C API and no longer export them.
No longer export _PyCompile_AstOptimize() internal C API function.
Change comment style to "// comment" and add comment explaining why
other functions have to be exported.
No longer export _PyUnicode_FromId() internal C API function.
Change comment style to "// comment" and add comment explaining why
other functions have to be exported.
Update Tools/build/generate_token.py to update Include/internal/pycore_token.h
comments.
Remove the internal _PyDict_GetItemStringWithError() function. It can
now be replaced with the new public PyDict_ContainsString() and
PyDict_GetItemStringRef() functions.
getargs.c now now uses a strong reference for current_arg.
find_keyword() returns a strong reference.
No longer export these 5 internal C API functions:
* _PyArena_AddPyObject()
* _PyArena_Free()
* _PyArena_Malloc()
* _PyArena_New()
* _Py_FatalRefcountErrorFunc()
Change comment style to "// comment" and add comment explaining why
other functions have to be exported.
No longer export these 2 internal C API functions:
* _PyTime_AsNanoseconds()
* _PyTime_GetSystemClockWithInfo()
Change comment style to "// comment" and add comment explaining why
other functions have to be exported.
The _socket extension uses _PyCapsule_SetTraverse() to visit and clear
the socket type in the garbage collector. So the _socket.socket type
can be cleared in some corner cases when it wasn't possible before.
Remove private _PyDict_GetItemStringWithError() function of the
public C API: the new PyDict_GetItemStringRef() can be used instead.
* Move private _PyDict_GetItemStringWithError() to the internal C API.
* _testcapi get_code_extra_index() uses PyDict_GetItemStringRef().
Avoid using private functions in _testcapi which tests the public C
API.
* Add missing includes.
* Remove unused includes.
* Update old include/symbol names to newer names.
* Mention at least one included symbol.
* Sort includes.
* Update Tools/cases_generator/generate_cases.py used to generated
pycore_opcode_metadata.h.
* Update Parser/asdl_c.py used to generate pycore_ast.h.
* Cleanup also includes in _testcapimodule.c and _testinternalcapi.c.
* pycore_intrinsics.h does nothing if included twice
(add #ifndef and #define).
* Update Tools/cases_generator/generate_cases.py to generate the
Py_BUILD_CORE test.
* _bz2, _lzma, _opcode and zlib extensions now define the
Py_BUILD_CORE_MODULE macro to use internal headers
(pycore_code.h, pycore_intrinsics.h and pycore_blocks_output_buffer.h).
This finishes the work begun in gh-107760. When, while projecting a superblock, we encounter a call to a short, simple function, the superblock will now enter the function using `_PUSH_FRAME`, continue through it, and leave it using `_POP_FRAME`, and then continue through the original code. Multiple frame pushes and pops are even possible. It is also possible to stop appending to the superblock in the middle of a called function, when running out of space or encountering an unsupported bytecode.
* Split `CALL_PY_EXACT_ARGS` into uops
This is only the first step for doing `CALL` in Tier 2.
The next step involves tracing into the called code object and back.
After that we'll have to do the remaining `CALL` specialization.
Finally we'll have to deal with `KW_NAMES`.
Note: this moves setting `frame->return_offset` directly in front of
`DISPATCH_INLINED()`, to make it easier to move it into `_PUSH_FRAME`.
Introducing a new file, stacking.py, that takes over several responsibilities related to symbolic evaluation of push/pop operations, with more generality.
The linked list of objects was a global variable, which broke isolation between interpreters, causing crashes. To solve this, we've moved the linked list to each interpreter.
There's no need to use a dummy uop to skip unused cache entries. The macro syntax lets you write `unused/1` instead.
Similarly, move `unused/5` from op `_LOAD_ATTR_INSTANCE_VALUE` to macro `LOAD_ATTR_INSTANCE_VALUE`.
This fixes a crasher due to a race condition, triggered infrequently when two isolated (own GIL) subinterpreters simultaneously initialize their sys or builtins modules. The crash happened due the combination of the "detached" thread state we were using and the "last holder" logic we use for the GIL. It turns out it's tricky to use the same thread state for different threads. Who could have guessed?
We solve the problem by eliminating the one object we were still sharing between interpreters. We replace it with a low-level hashtable, using the "raw" allocator to avoid tying it to the main interpreter.
We also remove the accommodations for "detached" thread states, which were a dubious idea to start with.
The _xxsubinterpreters module should not rely on internal API. Some of the functions it uses were recently moved there however. Here we move them back (and expose them properly).
We tried this before with a dict and for all interned strings. That ran into problems due to interpreter isolation. However, exclusively using a per-interpreter cache caused some inconsistency that can eliminate the benefit of interning. Here we circle back to using a global cache, but only for statically allocated strings. We also use a more-basic _Py_hashtable_t for that global cache instead of a dict.
Ideally we would only have the global cache, but the optional isolation of each interpreter's allocator means that a non-static string object must not outlive its interpreter. Thus we would have to store a copy of each such interned string in the global cache, tied to the main interpreter.
Fix warnings C4100 in Py_UNUSED() when Python is built with "cl /W4".
Example with this function included by Python.h:
static inline unsigned int
PyUnicode_IS_READY(PyObject* Py_UNUSED(op))
{ return 1; }
Without this change, building a C program with "cl /W4" which just
includes Python.h emits the warning:
Include\cpython/unicodeobject.h(199):
warning C4100: '_unused_op': unreferenced formal parameter
This change fix this warning.
No longer export these 5 internal C API variables:
* _PyBufferWrapper_Type
* _PyImport_FrozenBootstrap
* _PyImport_FrozenStdlib
* _PyImport_FrozenTest
* _Py_SwappedOp
Fix the definition of these internal functions, replace PyAPI_DATA()
with PyAPI_FUNC():
* _PyImport_ClearExtension()
* _PyObject_IsFreed()
* _PyThreadState_GetCurrent()
No longer export these 2 internal C API functions:
* _PyEval_SignalAsyncExc()
* _PyEval_SignalReceived()
Add also comments explaining why some internal functions have to be
exported, and update existing comments.
Move PyInterpreterConfig structure and associated macros from
initconfig.h to pylifecycle.h: it's not related to the Python
Initialization Configuration.
Declare the following 3 PyUnstable functions in
Include/cpython/pyframe.h rather than Include/cpython/frameobject.h,
so they are now provided by the standard "#include <Python.h>".
Move private _PyMem functions to the internal C API (pycore_pymem.h):
* _PyMem_GetCurrentAllocatorName()
* _PyMem_RawStrdup()
* _PyMem_RawWcsdup()
* _PyMem_Strdup()
No longer export these functions.
Move pymem_getallocatorsname() function from _testcapi to _testinternalcapi,
since the API moved to the internal C API.
Move private _PyDict functions to the internal C API (pycore_dict.h):
* _PyDict_Contains_KnownHash()
* _PyDict_DebugMallocStats()
* _PyDict_DelItemIf()
* _PyDict_GetItemWithError()
* _PyDict_HasOnlyStringKeys()
* _PyDict_MaybeUntrack()
* _PyDict_MergeEx()
No longer export these functions.
Move private _PyObject and private _PyType functions to the internal
C API (pycore_object.h):
* _PyObject_GetMethod()
* _PyObject_IsAbstract()
* _PyObject_NextNotImplemented()
* _PyType_CalculateMetaclass()
* _PyType_GetDocFromInternalDoc()
* _PyType_GetTextSignatureFromInternalDoc()
No longer export these functions.
Move private _PyBytes functions to the internal C API
(pycore_bytesobject.h):
* _PyBytes_DecodeEscape()
* _PyBytes_FormatEx()
* _PyBytes_FromHex()
* _PyBytes_Join()
No longer export these functions.
Move private debug _PyObject functions to the internal C API
(pycore_object.h):
* _PyDebugAllocatorStats()
* _PyObject_CheckConsistency()
* _PyObject_DebugTypeStats()
* _PyObject_IsFreed()
No longer export most of these functions, except of
_PyObject_IsFreed().
Move test functions using _PyObject_IsFreed() from _testcapi to
_testinternalcapi. check_pyobject_is_freed() test no longer catch
_testcapi.error: the tested function cannot raise _testcapi.error.
Move private _PyModule API to the internal C API
(pycore_moduleobject.h):
* _PyModule_Clear()
* _PyModule_ClearDict()
* _PyModuleSpec_IsInitializing()
* _PyModule_IsExtension()
No longer export these functions.
Rename private C API constants:
* Rename PY_MONITORING_UNGROUPED_EVENTS to _PY_MONITORING_UNGROUPED_EVENTS
* Rename PY_MONITORING_EVENTS to _PY_MONITORING_EVENTS
Move the private _PyInterpreterID C API to the internal C API: add a
new pycore_interp_id.h header file.
Remove Include/interpreteridobject.h and
Include/cpython/interpreteridobject.h header files.
Move private _PyGen API to internal C API:
* _PyAsyncGenAThrow_Type
* _PyAsyncGenWrappedValue_Type
* _PyCoroWrapper_Type
* _PyGen_FetchStopIterationValue()
* _PyGen_Finalize()
* _PyGen_SetStopIterationValue()
No longer these symbols, except of the ones by the _asyncio shared
extensions.
* No longer export most private _PyHash symbols, only export the ones
which are needed by shared extensions.
* Modules/_xxtestfuzz/fuzzer.c now uses the internal C API.
Remove private _PyUnicode_AsString() alias to PyUnicode_AsUTF8(). It
was kept for backward compatibility with Python 3.0 - 3.2.
The PyUnicode_AsUTF8() is available since Python
3.3. The PyUnicode_AsUTF8String() function can be used to keep
compatibility with Python 3.2 and older.
* Add PyDict_GetItemRef() and PyDict_GetItemStringRef() functions.
Add these functions to the stable ABI version 3.13.
* Add unit tests on the PyDict C API in test_capi.
By turning `assert(kwnames == NULL)` into a macro that is not in the "forbidden" list, many instructions that formerly were skipped because they contained such an assert (but no other mention of `kwnames`) are now supported in Tier 2. This covers 10 instructions in total (all specializations of `CALL` that invoke some C code):
- `CALL_NO_KW_TYPE_1`
- `CALL_NO_KW_STR_1`
- `CALL_NO_KW_TUPLE_1`
- `CALL_NO_KW_BUILTIN_O`
- `CALL_NO_KW_BUILTIN_FAST`
- `CALL_NO_KW_LEN`
- `CALL_NO_KW_ISINSTANCE`
- `CALL_NO_KW_METHOD_DESCRIPTOR_O`
- `CALL_NO_KW_METHOD_DESCRIPTOR_NOARGS`
- `CALL_NO_KW_METHOD_DESCRIPTOR_FAST`
This moves EXIT_TRACE, SAVE_IP, JUMP_TO_TOP, and
_POP_JUMP_IF_{FALSE,TRUE} from ceval.c to bytecodes.c.
They are no less special than before, but this way
they are discoverable o the copy-and-patch tooling.
* Convert PyObject_DelAttr() and PyObject_DelAttrString() macros to
functions.
* Add PyObject_DelAttr() and PyObject_DelAttrString() functions to
the stable ABI.
* Replace PyObject_SetAttr(obj, name, NULL) with
PyObject_DelAttr(obj, name).
Remove private pylifecycle.h functions: move them to the internal C
API ( pycore_atexit.h, pycore_pylifecycle.h and pycore_signal.h). No
longer export most of these functions.
Move _testcapi.test_atexit() to _testinternalcapi.
Remove private _PyUnicode_TransformDecimalAndSpaceToASCII() and other
private _PyUnicode C API functions: move them to the internal C API
(pycore_unicodeobject.h). No longer most of these functions.
Replace _testcapi.unicode_transformdecimalandspacetoascii() with
_testinternal._PyUnicode_TransformDecimalAndSpaceToASCII().
* Remove private _PyImport C API functions: move them to the internal
C API (pycore_import.h).
* No longer export most of these private functions.
* _testcapi avoids private _PyImport_GetModuleAttrString().
Remove more private _PyUnicode C API functions:
move them to the internal C API (pycore_unicodeobject.h).
No longer export most pycore_unicodeobject.h functions.
- Tweak uops debugging output
- Fix the bug from gh-106290
- Rename `SET_IP` to `SAVE_IP` (per https://github.com/faster-cpython/ideas/issues/558)
- Add a `SAVE_IP` uop at the start of the trace (ditto)
- Allow `unbound_local_error`; this gives us uops for `LOAD_FAST_CHECK`, `LOAD_CLOSURE`, and `DELETE_FAST`
- Longer traces
- Support `STORE_FAST_LOAD_FAST`, `STORE_FAST_STORE_FAST`
- Add deps on pycore_uops.h to Makefile(.pre.in)
Remove the following functions from the C API, move them to the internal C
API: add a new pycore_modsupport.h internal header file:
* PyModule_CreateInitialized()
* _PyArg_NoKwnames()
* _Py_VaBuildStack()
No longer export these functions.
Remove private _PyThreadState and _PyInterpreterState C API
functions: move them to the internal C API (pycore_pystate.h and
pycore_interp.h). Don't export most of these functions anymore, but
still export functions used by tests.
Remove _PyThreadState_Prealloc() and _PyThreadState_Init() from the C
API, but keep it in the stable API.
* Remove private _PyTraceMalloc C API functions: move them to the
internal C API.
* Don't export most of these functions anymore, but still export
_PyTraceMalloc_GetTraceback() used by tests.
* Rename Include/tracemalloc.h to Include/cpython/tracemalloc.h
Remove the "cpython/pytime.h" header file: it only contained private
functions. Move functions to the internal pycore_time.h header file.
Move tests from _testcapi to _testinternalcapi. Rename also test
methods to have the same name than tested C functions.
No longer export these functions:
* _PyTime_Add()
* _PyTime_As100Nanoseconds()
* _PyTime_FromMicrosecondsClamp()
* _PyTime_FromTimespec()
* _PyTime_FromTimeval()
* _PyTime_GetPerfCounterWithInfo()
* _PyTime_MulDiv()
Remove the following private functions of the C API:
* _PyCodecInfo_GetIncrementalDecoder()
* _PyCodecInfo_GetIncrementalEncoder()
* _PyCodec_DecodeText()
* _PyCodec_EncodeText()
* _PyCodec_Forget()
* _PyCodec_Lookup()
* _PyCodec_LookupTextEncoding()
Move these functions to a new pycore_codecs.h internal header file.
These functions are no longer exported.
PyTuple_SET_ITEM() and PyList_SET_ITEM() now check the index argument
with an assertion if Python is built in debug mode or is built with
assertions.
* list_extend() and _PyList_AppendTakeRef() now set the list size
before calling PyList_SET_ITEM().
* PyStructSequence_GetItem() and PyStructSequence_SetItem() now check
the index argument: must be lesser than REAL_SIZE(op).
* PyStructSequence_GET_ITEM() and PyStructSequence_SET_ITEM() are now
aliases to PyStructSequence_GetItem() and
PyStructSequence_SetItem().
Remove the following private functions from the public C API:
* _Py_CheckFunctionResult()
* _PyObject_CallMethod()
* _PyObject_CallMethodId()
* _PyObject_CallMethodIdNoArgs()
* _PyObject_CallMethodIdObjArgs()
* _PyObject_CallMethodIdOneArg()
* _PyObject_MakeTpCall()
* _PyObject_VectorcallMethodId()
* _PyStack_AsDict()
Move these functions to the internal C API (pycore_call.h).
No longer export the following functions:
* _PyObject_Call()
* _PyObject_CallMethod()
* _PyObject_CallMethodId()
* _PyObject_CallMethodIdObjArgs()
* _PyObject_Call_Prepend()
* _PyObject_FastCallDictTstate()
* _PyStack_AsDict()
The following functions are still exported for stdlib shared
extensions:
* _Py_CheckFunctionResult()
* _PyObject_MakeTpCall()
Mark the following internal functions as extern:
* _PyStack_UnpackDict()
* _PyStack_UnpackDict_Free()
* _PyStack_UnpackDict_FreeNoDecRef()
Added a new, experimental, tracing optimizer and interpreter (a.k.a. "tier 2"). This currently pessimizes, so don't use yet -- this is infrastructure so we can experiment with optimizing passes. To enable it, pass ``-Xuops`` or set ``PYTHONUOPS=1``. To get debug output, set ``PYTHONUOPSDEBUG=N`` where ``N`` is a debug level (0-4, where 0 is no debug output and 4 is excessively verbose).
All of this code is likely to change dramatically before the 3.13 feature freeze. But this is a first step.
Remove the following functions from the public C API:
* _PyObject_RealIsInstance()
* _PyObject_RealIsSubclass()
* _Py_add_one_to_index_F()
* _Py_add_one_to_index_C()
Move _PyObject_RealIsInstance() and _PyObject_RealIsSubclass() to the
internal C API (pycore_abstract.h) and no longer export their symbols
(in libpython).
Make _Py_add_one_to_index_F() and _Py_add_one_to_index_C() functions
static: no longer export them.
Remove _PyObject_HasLen() and _PySequence_IterSearch() functions from
the public C API: move them to the internal C API
(pycore_abstract.h).
No longer export these symbols (in libpython).
Remove also unused pycore_initconfig.h include in typeobject.c.
Remove private _PySequence_BytesToCharpArray() and
_Py_FreeCharPArray() functions from the public C API: move these
functions from Objects/abstract.c to Modules/_posixsubprocess.c.
Remove old aliases which were kept backwards compatibility with
Python 3.8:
* _PyObject_CallMethodNoArgs()
* _PyObject_CallMethodOneArg()
* _PyObject_CallOneArg()
* _PyObject_FastCallDict()
* _PyObject_Vectorcall()
* _PyObject_VectorcallMethod()
* _PyVectorcall_Function()
Update code which used these aliases to use new names.
Remove _PyWeakref_GetWeakrefCount() and _PyWeakref_ClearRef() from
the public C API: move them to the internal C API.
Refactor also _weakref_getweakrefs() code to make it more readable.
* Add _PyWeakref_IS_DEAD() internal function.
* Modify is_dead_weakref() of Modules/_weakref.c and
_pysqlite_drop_unused_cursor_references() to replace
PyWeakref_GET_OBJECT() with _PyWeakref_IS_DEAD().
* Replace "int i" with "Py_ssize_t i" to iterate on cursors
in _pysqlite_drop_unused_cursor_references().
finalize_modules_clear_weaklist() now holds a strong reference to the
module longer than before: replace PyWeakref_GET_OBJECT() with
_PyWeakref_GET_REF().
* Add tests on PyImport_AddModuleRef(), PyImport_AddModule() and
PyImport_AddModuleObject().
* pythonrun.c: Replace Py_XNewRef(PyImport_AddModule(name)) with
PyImport_AddModuleRef(name).
* Add table describing possible executable classes for out-of-process debuggers.
* Remove shim code object creation code as it is no longer needed.
* Make lltrace a bit more robust w.r.t. non-standard frames.
Victor Stinner
Irit Katriel
Guido van Rossum
Serhiy Storchaka
Pablo Galindo Salgado
Mark Shannon
brandonardenwalli
Ken Jin
Finn Womack
Carl Meyer
Brandt Bucher
Eric Snow
Ivin Lee
Dong-hee Na
Inada Naoki
Md Sadman Chowdhury
hms
Eddie Elizondo