Move the static inline function flavor of Py_EnterRecursiveCall() and
Py_LeaveRecursiveCall() to the internal C API: they access
PyThreadState attributes. The limited C API provides regular
functions which hide implementation details.
PyInterpreterState.eval_frame function now requires a tstate (Python
thread state) parameter.
Add private functions to the C API to get and set the frame
evaluation function:
* Add tstate parameter to _PyFrameEvalFunction function type.
* Add _PyInterpreterState_GetEvalFrameFunc() and
_PyInterpreterState_SetEvalFrameFunc() functions.
* Add tstate parameter to _PyEval_EvalFrameDefault().
PyGILState_Ensure() doesn't call PyEval_InitThreads() anymore when a
new Python thread state is created. The GIL is created by
Py_Initialize() since Python 3.7, it's not needed to call
PyEval_InitThreads() explicitly.
Add an assertion to ensure that the GIL is already created.
Convert _PyRuntimeState.finalizing field to an atomic variable:
* Rename it to _finalizing
* Change its type to _Py_atomic_address
* Add _PyRuntimeState_GetFinalizing() and _PyRuntimeState_SetFinalizing()
functions
* Remove _Py_CURRENTLY_FINALIZING() function: replace it with testing
directly _PyRuntimeState_GetFinalizing() value
Convert _PyRuntimeState_GetThreadState() to static inline function.
* Add _PyWarnings_InitState() which only initializes the _warnings
module state (tstate->interp->warnings) without creating a module
object
* Py_InitializeFromConfig() now calls _PyWarnings_InitState() instead
of _PyWarnings_Init()
* Rename also private functions of _warnings.c to avoid confusion
between the public C API and the private C API.
Move the dtoa.h header file to the internal C API as pycore_dtoa.h:
it only contains private functions (prefixed by "_Py").
The math and cmath modules must now be compiled with the
Py_BUILD_CORE macro defined.
Move the bytes_methods.h header file to the internal C API as
pycore_bytes_methods.h: it only contains private symbols (prefixed by
"_Py"), except of the PyDoc_STRVAR_shared() macro.
Add a fast-path for UTF-8 encoding in PyUnicode_EncodeFSDefault()
and PyUnicode_DecodeFSDefaultAndSize().
Add _PyUnicode_FiniEncodings() helper function for _PyUnicode_Fini().
_Py_NewReference() becomes a regular opaque function, rather than a
static inline function in the C API (object.h), to better hide
implementation details.
Move _Py_tracemalloc_config from public pymem.h to internal
pycore_pymem.h header.
Make _Py_AddToAllObjects() private.
Move the following functions from the public C API to the internal C
API:
* _PyDebug_PrintTotalRefs(),
* _Py_PrintReferenceAddresses()
* _Py_PrintReferences()
Rename init_slotdefs() to _PyTypes_InitSlotDefs() and add a return
value of type PyStatus. The function is now called exactly once from
_PyTypes_Init(). Replace calls to init_slotdefs() with an assertion
checking that slotdefs is initialized.
The public API symbols being removed are:
_PyBytes_InsertThousandsGroupingLocale, _PyBytes_InsertThousandsGrouping, _Py_InitializeFromArgs, _Py_InitializeFromWideArgs, _PyFloat_Repr, _PyFloat_Digits,
_PyFloat_DigitsInit, PyFrame_ExtendStack, _PyAIterWrapper_Type, PyNullImporter_Type, PyCmpWrapper_Type, PySortWrapper_Type, PyNoArgsFunction.
The comment about the collection rules for the permanent generation was
incorrectly referenced by a comment in gcmodule.c (the comment has been
moved long ago into a header file). Moving the comment into the relevant
code helps with readability and avoids broken references.
Each Python subinterpreter now has its own "small integer
singletons": numbers in [-5; 257] range.
It is no longer possible to change the number of small integers at
build time by overriding NSMALLNEGINTS and NSMALLPOSINTS macros:
macros should now be modified manually in pycore_pystate.h header
file.
For now, continue to share _PyLong_Zero and _PyLong_One singletons
between all subinterpreters.
new_interpreter() now calls _PySys_Create() to create a new sys
module isolated from the main interpreter. It now calls
_PySys_InitCore() and _PyImport_FixupBuiltin().
init_interp_main() now calls _PySys_InitMain().
new_interpreter() now calls _PyBuiltin_Init() to create the builtins
module and calls _PyImport_FixupBuiltin(), rather than using
_PyImport_FindBuiltin(tstate, "builtins").
pycore_init_builtins() is now responsible to initialize
intepr->builtins_copy: inline _PyImport_Init() and remove this
function.
If _PyImport_FixupExtensionObject() is called from a subinterpreter,
leave extensions unchanged and don't copy the module dictionary
into def->m_base.m_copy.
Remove PyMethod_ClearFreeList() and PyCFunction_ClearFreeList()
functions: the free lists of bound method objects have been removed.
Remove also _PyMethod_Fini() and _PyCFunction_Fini() functions.
* Add GCState type for readability
* gcmodule.c now gets its gcstate from tstate
* _PyGC_DumpShutdownStats() now expects tstate rather than runtime
* Rename "state" to "gcstate" for readability: to avoid confusion
between "state" and "tstate" for example.
* collect() now only expects tstate: it gets gcstate from tstate.
* Pass tstate to _PyErr_xxx() functions
* Factorize code in common between Py_FinalizeEx() and
Py_EndInterpreter().
* Py_EndInterpreter() now also calls _PyWarnings_Fini().
* Call _PyExc_Fini() and _PyGC_Fini() later in the finalization.
Add PyInterpreterState.runtime field: reference to the _PyRuntime
global variable. This field exists to not have to pass runtime in
addition to tstate to a function. Get runtime from tstate:
tstate->interp->runtime.
Remove "_PyRuntimeState *runtime" parameter from functions already
taking a "PyThreadState *tstate" parameter.
_PyGC_Init() first parameter becomes "PyThreadState *tstate".
bpo-3605, bpo-38733: Optimize _PyErr_Occurred(): remove "tstate ==
NULL" test.
Py_FatalError() no longer calls PyErr_Occurred() if called without
holding the GIL. So PyErr_Occurred() no longer has to support
tstate==NULL case.
_Py_CheckFunctionResult(): use directly _PyErr_Occurred() to avoid
explicit "!= NULL" test.
bpo-36389, bpo-38376: The _PyObject_CheckConsistency() function is
now also available in release mode. For example, it can be used to
debug a crash in the visit_decref() function of the GC.
Modify the following functions to also work in release mode:
* _PyDict_CheckConsistency()
* _PyObject_CheckConsistency()
* _PyType_CheckConsistency()
* _PyUnicode_CheckConsistency()
Other changes:
* _PyMem_IsPtrFreed(ptr) now also returns 1 if ptr is NULL
(equals to 0).
* _PyBytesWriter_CheckConsistency() now returns 1 and is only used
with assert().
* Reorder _PyObject_Dump() to write safe fields first, and only
attempt to render repr() at the end.
* _Py_FindEnvConfigValue() now returns a string allocated
by PyMem_RawMalloc().
* calculate_init() now decodes VPATH macro.
* Add calculate_open_pyenv() function.
* Add substring() and joinpath2() functions.
* Fix add_exe_suffix()
And a few cleanup changes.
For now, we'll rely on the fact that the config structures aren't covered by the stable ABI.
We may revisit this in the future if we further explore the idea of offering a stable embedding API.
(cherry picked from commit bdace21b76)
Fix warnings options priority: PyConfig.warnoptions has the highest
priority, as stated in the PEP 587.
* Document options order in PyConfig.warnoptions documentation.
* Make PyWideStringList_INIT macro private: replace "Py" prefix
with "_Py".
* test_embed: add test_init_warnoptions().
Add a new struct_size field to PyPreConfig and PyConfig structures to
allow to modify these structures in the future without breaking the
backward compatibility.
* Replace private _config_version field with public struct_size field
in PyPreConfig and PyConfig.
* Public PyPreConfig_InitIsolatedConfig() and
PyPreConfig_InitPythonConfig()
return type becomes PyStatus, instead of void.
* Internal _PyConfig_InitCompatConfig(),
_PyPreConfig_InitCompatConfig(), _PyPreConfig_InitFromConfig(),
_PyPreConfig_InitFromPreConfig() return type becomes PyStatus,
instead of void.
* Remove _Py_CONFIG_VERSION
* Update the Initialization Configuration documentation.
* Py_InitializeFromConfig() now writes PyConfig path configuration to
the global path configuration (_Py_path_config).
* Add test_embed.test_get_pathconfig().
* Fix typo in _PyWideStringList_Join().
* Add test_embed.test_init_setpath_config(): test Py_SetPath()
with PyConfig.
* test_init_setpath() and test_init_setpythonhome() no longer call
Py_SetProgramName(), but use the default program name.
* _PyPathConfig: isolated, site_import and base_executable
fields are now only available on Windows.
* If executable is set explicitly in the configuration, ignore
calculated base_executable: _PyConfig_InitPathConfig() copies
executable to base_executable.
* Complete path config documentation.
The DLL path is not computed from any user configuration and cannot
be configured by PyConfig. Instead, add a new _Py_dll_path global variable.
Remove _PyConfig_SetPathConfig(): replaced with _PyPathConfig_Init().
Py_Initialize() now longer sets the "global path configuration",
but only initialize _Py_dll_path.
Add _PyRuntimeState.preinitializing field: set to 1 while
Py_PreInitialize() is running.
_PyRuntimeState: rename also pre_initialized field to preinitialized.
* Rename PyThreadState_DeleteCurrent()
to _PyThreadState_DeleteCurrent()
* Move it to the internal C API
Co-Authored-By: Carol Willing <carolcode@willingconsulting.com>
PyConfig_Read() is now responsible to handle early calls to
PySys_AddXOption() and PySys_AddWarnOption().
Options added by PySys_AddXOption() are now handled the same way than
PyConfig.xoptions and command line -X options.
For example, PySys_AddXOption(L"faulthandler") enables faulthandler
as expected.
sys._base_executable is now always defined on all platforms, and can be overridden through configuration.
Also adds test.support.PythonSymlink to encapsulate platform-specific logic for symlinking sys.executable
Remove sys.getcheckinterval() and sys.setcheckinterval() functions.
They were deprecated since Python 3.2. Use sys.getswitchinterval()
and sys.setswitchinterval() instead.
Remove also check_interval field of the PyInterpreterState structure.
* Rename PyImport_Cleanup() to _PyImport_Cleanup() and move it to the
internal C API. Add 'tstate' parameters.
* Remove documentation of _PyImport_Init(), PyImport_Cleanup(),
_PyImport_Fini(). All three were documented as "For internal use
only.".
* Add 'tstate' parameter to many internal import.c functions.
* _PyImportZip_Init() now gets 'tstate' parameter rather than
'interp'.
* Add 'interp' parameter to _PyState_ClearModules() and rename it
to _PyInterpreterState_ClearModules().
* Move private _PyImport_FindBuiltin() to the internal C API; add
'tstate' parameter to it.
* Remove private _PyImport_AddModuleObject() from the C API:
use public PyImport_AddModuleObject() instead.
* Remove private _PyImport_FindExtensionObjectEx() from the C API:
use private _PyImport_FindExtensionObject() instead.
* Add Include/cpython/import.h and Include/internal/pycore_import.h
header files.
* Move _PyImport_ReInitLock() to the internal C API. Don't export the
symbol anymore.
* Replace global var Py_VerboseFlag with interp->config.verbose.
* Add _PyErr_NoMemory(tstate) function.
* Add tstate parameter to _PyEval_SetCoroutineOriginTrackingDepth()
and move the function to the internal API.
* Replace _PySys_InitMain(runtime, interp)
with _PySys_InitMain(runtime, tstate).
Add a new threading.excepthook() function which handles uncaught
Thread.run() exception. It can be overridden to control how uncaught
exceptions are handled.
threading.ExceptHookArgs is not documented on purpose: it should not
be used directly.
* threading.excepthook() and threading.ExceptHookArgs.
* Add _PyErr_Display(): similar to PyErr_Display(), but accept a
'file' parameter.
* Add _thread._excepthook(): C implementation of the exception hook
calling _PyErr_Display().
* Add _thread._ExceptHookArgs: structseq type.
* Add threading._invoke_excepthook_wrapper() which handles the gory
details to ensure that everything remains alive during Python
shutdown.
* Add unit tests.
PyErr_WriteUnraisable() now creates a traceback object if there is no
current traceback. Moreover, call PyErr_NormalizeException() and
PyException_SetTraceback() to normalize the exception value. Ignore
silently any error.
* _PyPreConfig_GetGlobalConfig() and _PyCoreConfig_GetGlobalConfig()
now do nothing if the configuration was not initialized with
_PyPreConfig_InitCompatConfig() and _PyCoreConfig_InitCompatConfig()
* Remove utf8_mode=-2 special case: use utf8_mode=-1 instead.
* Fix _PyPreConfig_InitPythonConfig():
* isolated = 0 instead of -1
* use_environment = 1 instead of -1
* Rename _PyConfig_INIT to _PyConfig_INIT_COMPAT
* Rename _PyPreConfig_Init() to _PyPreConfig_InitCompatConfig()
* Rename _PyCoreConfig_Init() to _PyCoreConfig_InitCompatConfig()
* PyInterpreterState_New() now uses _PyCoreConfig_InitPythonConfig()
as default configuration, but it's very quickly overriden anyway.
* _freeze_importlib.c uses _PyCoreConfig_SetString() to set
program_name.
* Cleanup preconfig_init_utf8_mode(): cmdline is always non-NULL.
Add new sys.unraisablehook() function which can be overridden to
control how "unraisable exceptions" are handled. It is called when an
exception has occurred but there is no way for Python to handle it.
For example, when a destructor raises an exception or during garbage
collection (gc.collect()).
Changes:
* Add an internal UnraisableHookArgs type used to pass arguments to
sys.unraisablehook.
* Add _PyErr_WriteUnraisableDefaultHook().
* The default hook now ignores exception on writing the traceback.
* test_sys now uses unittest.main() to automatically discover tests:
remove test_main().
* Add _PyErr_Init().
* Fix PyErr_WriteUnraisable(): hold a strong reference to sys.stderr
while using it
* Add _PyPreConfig.parse_argv
* Add _PyCoreConfig._config_init field and _PyCoreConfigInitEnum enum
type
* Initialization functions: reject preconfig=NULL and config=NULL
* Add config parameter to _PyCoreConfig_DecodeLocaleErr(): pass
config->argv to _Py_PreInitializeFromPyArgv(), to parse config
command line arguments in preinitialization.
* Add config parameter to _PyCoreConfig_SetString(). It now
preinitializes Python.
* _PyCoreConfig_SetPyArgv() now also preinitializes Python for wide
argv
* Fix _Py_PreInitializeFromCoreConfig(): don't pass args to
_Py_PreInitializeFromPyArgv() if config.parse_argv=0.
* Use "char * const *" and "wchar_t * const *" types for 'argv'
parameters and _PyArgv.argv.
* Add unit test on preinitialization from argv.
* _PyPreConfig.allocator type becomes int
* Add _PyPreConfig_InitFromPreConfig() and
_PyPreConfig_InitFromCoreConfig() helper functions
_PyPreConfig_InitPythonConfig() and _PyCoreConfig_InitPythonConfig()
no longer inherit their values from global configuration variables.
Changes:
* _PyPreCmdline_Read() now ignores -X dev and PYTHONDEVMODE
if dev_mode is already set.
* Inline _PyPreConfig_INIT macro into _PyPreConfig_Init() function.
* Inline _PyCoreConfig_INIT macro into _PyCoreConfig_Init() function.
* Replace _PyCoreConfig_Init() with _PyCoreConfig_InitPythonConfig()
in most tests of _testembed.c.
* Replace _PyCoreConfig_Init() with _PyCoreConfig_InitIsolatedConfig()
in _freeze_importlib.c.
* Move some initialization functions from the internal
to the private API.
* The type of initlization function 'argc' parameters becomes
Py_ssize_t, instead of int.
* Change _PyPreConfig_Copy() return type to void, instead of int.
The function cannot fail anymore.
* Fix compilation warnings on Windows.
* Add _PyInitError functions:
* _PyInitError_Ok()
* _PyInitError_Error()
* _PyInitError_NoMemory()
* _PyInitError_Exit()
* _PyInitError_IsError()
* _PyInitError_IsExit()
* _PyInitError_Failed()
* frozenmain.c and _testembed.c now use functions rather than macros.
* Move _Py_INIT_xxx() macros to the internal API.
* Move _PyWstrList_INIT macro to the internal API.
Py_Main() and _Py_RunMain() now return the exitcode rather than
calling Py_Exit(exitcode) when calling PyErr_Print() if the current
exception type is SystemExit.
* Add _Py_HandleSystemExit().
* Add pymain_exit_err_print().
* Add pymain_exit_print().
Add new functions to get the Python interpreter behavior:
* _PyPreConfig_InitPythonConfig()
* _PyCoreConfig_InitPythonConfig()
Add new functions to get an isolated configuration:
* _PyPreConfig_InitIsolatedConfig()
* _PyCoreConfig_InitIsolatedConfig()
Replace _PyPreConfig_INIT and _PyCoreConfig_INIT with new functions
_PyPreConfig_Init() and _PyCoreConfig_Init().
_PyCoreConfig: set configure_c_stdio and parse_argv to 0 by default
to behave as Python 3.6 in the default configuration.
_PyCoreConfig_Read() no longer sets coerce_c_locale_warn to 1 if it's
equal to 0. coerce_c_locale_warn must now be set to -1 (ex: using
_PyCoreConfig_InitPythonConfig()) to enable C locale coercion
warning.
Add unit tests for _PyCoreConfig_InitPythonConfig()
and _PyCoreConfig_InitIsolatedConfig().
Changes:
* Rename _PyCoreConfig_GetCoreConfig() to _PyPreConfig_GetCoreConfig()
* Fix core_read_precmdline(): handle parse_argv=0
* Fix _Py_PreInitializeFromCoreConfig(): pass coreconfig.argv
to _Py_PreInitializeFromPyArgv(), except if parse_argv=0
* Add PyMemAllocatorName enum
* _PyPreConfig.allocator type becomes PyMemAllocatorName, instead of
char*
* Remove _PyPreConfig_Clear()
* Add _PyMem_GetAllocatorName()
* Rename _PyMem_GetAllocatorsName() to
_PyMem_GetCurrentAllocatorName()
* Remove _PyPreConfig_SetAllocator(): just call
_PyMem_SetupAllocators() directly, we don't have do reallocate the
configuration with the new allocator anymore!
* _PyPreConfig_Write() parameter becomes const, as it should be in
the first place!
Remove the PyEval_ReInitThreads() function from the Python C API.
It should not be called explicitly: use PyOS_AfterFork_Child()
instead.
Rename PyEval_ReInitThreads() to _PyEval_ReInitThreads() and add a
'runtime' parameter.
Add "struct _ceval_runtime_state *ceval = &_PyRuntime.ceval;" local
variables to function to better highlight the dependency on the
global variable _PyRuntime and to point directly to _PyRuntime.ceval
field rather than on the larger _PyRuntime.
Changes:
* Add _PyRuntimeState_GetThreadState(runtime) macro.
* Add _PyEval_AddPendingCall(ceval, ...) and
_PyThreadState_Swap(gilstate, ...) functions.
* _PyThreadState_GET() macro now calls
_PyRuntimeState_GetThreadState() using &_PyRuntime.
* Add 'ceval' parameter to COMPUTE_EVAL_BREAKER(),
SIGNAL_PENDING_SIGNALS(), _PyEval_SignalAsyncExc(),
_PyEval_SignalReceived() and _PyEval_FiniThreads() macros and
functions.
* Add 'tstate' parameter to call_function(), do_call_core() and
do_raise().
* Add 'runtime' parameter to _Py_CURRENTLY_FINALIZING(),
_Py_FinishPendingCalls() and _PyThreadState_DeleteExcept()
macros and functions.
* Declare 'runtime', 'tstate', 'ceval' and 'eval_breaker' variables
as constant.
_PyCoreConfig_SetPyArgv() and _PyCoreConfig_SetWideString() now
pre-initialize Python if needed to ensure that the locale encoding is
properly configured.
* Add _Py_PreInitializeFromPyArgv() internal function.
* Add 'args' parameter to _Py_PreInitializeFromCoreConfig()
_PyCoreConfig: Change filesystem_encoding, filesystem_errors,
stdio_encoding and stdio_errors fields type from char* to wchar_t*.
Changes:
* PyInterpreterState: replace fscodec_initialized (int) with fs_codec
structure.
* Add get_error_handler_wide() and unicode_encode_utf8() helper
functions.
* Add error_handler parameter to unicode_encode_locale()
and unicode_decode_locale().
* Remove _PyCoreConfig_SetString().
* Rename _PyCoreConfig_SetWideString() to _PyCoreConfig_SetString().
* Rename _PyCoreConfig_SetWideStringFromString()
to _PyCoreConfig_DecodeLocale().
* Add _PyEval_FiniThreads2(). _PyEval_FiniThreads() now only clears
the pending lock, whereas _PyEval_FiniThreads2() destroys the GIL.
* pymain_free() now calls _PyEval_FiniThreads2().
* Py_FinalizeEx() now calls _PyEval_FiniThreads().
Py_FinalizeEx() now clears _PyRuntime.warnings variables and
_PyRuntime.exitfuncs.
Changes:
* Add _PyWarnings_Fini(): called by Py_FinalizeEx()
* call_ll_exitfuncs() now clears _PyRuntime.exitfuncs while iterating
on it (on backward order).
Py_InitializeEx() now uses a runtime variable passed to subfunctions,
rather than working directly on the global variable _PyRuntime.
Add 'runtime' parameter to _PyCoreConfig_Write(), _PySys_Create(),
_PySys_InitMain(), _PyGILState_Init(),
emit_stderr_warning_for_legacy_locale() and other subfunctions.
Victor Stinner
Dong-hee Na
Pablo Galindo
Vinay Sajip
Victor Stinner
Joannah Nanjekye
Steve Dower
Inada Naoki
Eric Snow
Steve Dower