Python built with "configure --with-trace-refs" (tracing references)
is now ABI compatible with Python release build and debug build.
Moreover, it now also supports the Limited API.
Change Py_TRACE_REFS build:
* Remove _PyObject_EXTRA_INIT macro.
* The PyObject structure no longer has two extra members (_ob_prev
and _ob_next).
* Use a hash table (_Py_hashtable_t) to trace references (all
objects): PyInterpreterState.object_state.refchain.
* Py_TRACE_REFS build is now ABI compatible with release build and
debug build.
* Limited C API extensions can now be built with Py_TRACE_REFS:
xxlimited, xxlimited_35, _testclinic_limited.
* No longer rename PyModule_Create2() and PyModule_FromDefAndSpec2()
functions to PyModule_Create2TraceRefs() and
PyModule_FromDefAndSpec2TraceRefs().
* _Py_PrintReferenceAddresses() is now called before
finalize_interp_delete() which deletes the refchain hash table.
* test_tracemalloc find_trace() now also filters by size to ignore
the memory allocated by _PyRefchain_Trace().
Test changes for Py_TRACE_REFS:
* Add test.support.Py_TRACE_REFS constant.
* Add test_sys.test_getobjects() to test sys.getobjects() function.
* test_exceptions skips test_recursion_normalizing_with_no_memory()
and test_memory_error_in_PyErr_PrintEx() if Python is built with
Py_TRACE_REFS.
* test_repl skips test_no_memory().
* test_capi skisp test_set_nomemory().
Remove _PyErr_ChainExceptions(), _PyErr_ChainExceptions1() and
_PyErr_SetFromPyStatus() functions from the public C API.
* Move the private _PyErr_ChainExceptions() and
_PyErr_ChainExceptions1() function to the internal C API
(pycore_pyerrors.h).
* Move the private _PyErr_SetFromPyStatus() to the internal C API
(pycore_initconfig.h).
* No longer export the _PyErr_ChainExceptions() function.
* Move run_in_subinterp_with_config() from _testcapi to
_testinternalcapi.
This is a follow-up to gh-107567 and gh-107733.
We skip test_basic_multiple_interpreters_deleted_no_reset on tracerefs builds. The test breaks interpreter isolation a little, which doesn't work well with Py_TRACE_REFS builds, so I feel fine about skipping the test.
gh-107184 introduced a refleak in test_import.SubinterpImportTests (specifically test_singlephase_check_with_setting_and_override and test_single_init_extension_compat). We fix it here by making sure _testsinglephase is removed from sys.modules whenever we clear the runtime's internal state for the module.
The underlying problem is strictly contained in the internal function _PyImport_ClearExtension() (AKA _testinternalcapi.clear_extension()), which is only used in tests.
(This also fixes an intermittent segfault introduced in the same place, in test_disallowed_reimport.)
* Add tests on PyImport_AddModuleRef(), PyImport_AddModule() and
PyImport_AddModuleObject().
* pythonrun.c: Replace Py_XNewRef(PyImport_AddModule(name)) with
PyImport_AddModuleRef(name).
We also add PyInterpreterState.ceval.own_gil to record if the interpreter actually has its own GIL.
Note that for now we don't actually respect own_gil; all interpreters still share the one GIL. However, PyInterpreterState.ceval.own_gil does reflect PyInterpreterConfig.own_gil. That lie is a temporary one that we will fix when the GIL really becomes per-interpreter.
This PR makes some minor linting adjustments to the Lib/test module
caught by [ruff](https://github.com/charliermarsh/ruff). The adjustments
are all related to the `F541 f-string without any placeholders` issue.
Issue: https://github.com/python/cpython/issues/103805
<!-- gh-issue-number: gh-103805 -->
* Issue: gh-103805
<!-- /gh-issue-number -->
---------
Co-authored-by: blurb-it[bot] <43283697+blurb-it[bot]@users.noreply.github.com>
Co-authored-by: Gregory P. Smith <greg@krypto.org>
This is strictly about moving the "obmalloc" runtime state from
`_PyRuntimeState` to `PyInterpreterState`. Doing so improves isolation
between interpreters, specifically most of the memory (incl. objects)
allocated for each interpreter's use. This is important for a
per-interpreter GIL, but such isolation is valuable even without it.
FWIW, a per-interpreter obmalloc is the proverbial
canary-in-the-coalmine when it comes to the isolation of objects between
interpreters. Any object that leaks (unintentionally) to another
interpreter is highly likely to cause a crash (on debug builds at
least). That's a useful thing to know, relative to interpreter
isolation.
I recently added some tests to test_imp, but @warsaw is removing that file in gh-98573. The tests are worth keeping so here I'm moving them to test_import.
This involves 3 changes: some general cleanup, checks to match the kind of module, and switch from testing against sys to _imp.
This is a precursor to gh-103150, though the changes are meant to stand on their own.
Enforcing (optionally) the restriction set by PEP 489 makes sense. Furthermore, this sets the stage for a potential restriction related to a per-interpreter GIL.
This change includes the following:
* add tests for extension module subinterpreter compatibility
* add _PyInterpreterConfig.check_multi_interp_extensions
* add Py_RTFLAGS_MULTI_INTERP_EXTENSIONS
* add _PyImport_CheckSubinterpIncompatibleExtensionAllowed()
* fail iff the module does not implement multi-phase init and the current interpreter is configured to check
https://github.com/python/cpython/issues/98627
``os.geteuid() == 0`` is not a reliable check whether the current user
has the capability to bypass permission checks. Tests now probe for DAC
override.
- Mark more ``umask()`` cases
- ``dup()`` is not supported
- ``/dev/null`` is not available
- document missing features
- mark more modules as not available
- Add requires_fork and requires_subprocess to more tests
- Skip extension import tests if dlopen is not available
- Don't assume that _testcapi is a shared extension
- Skip a lot of socket tests that don't work on Emscripten
- Skip mmap tests, mmap emulation is incomplete
- venv does not work yet
- Cannot get libc from executable
The "entire" test suite is now passing on Emscripten with EMSDK from git head (91 suites are skipped).
Doing this provides significant performance gains for runtime startup (~15% with all the imported modules frozen). We don't yet freeze all the imported modules because there are a few hiccups in the build systems we need to sort out first. (See bpo-45186 and bpo-45188.)
Note that in PR GH-28320 we added a command-line flag (-X frozen_modules=[on|off]) that allows users to opt out of (or into) using frozen modules. The default is still "off" but we will change it to "on" as soon as we can do it in a way that does not cause contributors pain.
https://bugs.python.org/issue45020
* bpo-39336: Allow setattr to fail on modules which aren't assignable
When attaching a child module to a package if the object in sys.modules raises an AttributeError (e.g. because it is immutable) it causes the whole import to fail. This now allows immutable packages to exist and an ImportWarning is reported and the AttributeError exception is ignored.
Relative imports use resolve_name to get the absolute target name,
which first seeks the current module's absolute package name from the globals:
If __package__ (and __spec__.parent) are missing then
import uses __name__, truncating the last segment if
the module is a submodule rather than a package __init__.py
(which it guesses from whether __path__ is defined).
The __name__ attempt should fail if there is no parent package (top level modules),
if __name__ is '__main__' (-m entry points), or both (scripts).
That is, if both __name__ has no subcomponents and the module does not seem
to be a package __init__ module then import should fail.
Shantanu
Eric Snow
Nikita Sobolev
Victor Stinner
Carl Meyer
Filipe Laíns
Erlend E. Aasland
chgnrdv
sunmy2019
Rodolfo M. Pereira
Christian Heimes
Pablo Galindo Salgado
Dong-hee Na
Steve Dower
Inada Naoki
Hai Shi
Serhiy Storchaka
Armin Rigo
Dino Viehland
Ben Lewis