We're no longer using _Py_IDENTIFIER() (or _Py_static_string()) in any core CPython code. It is still used in a number of non-builtin stdlib modules.
The replacement is: PyUnicodeObject (not pointer) fields under _PyRuntimeState, statically initialized as part of _PyRuntime. A new _Py_GET_GLOBAL_IDENTIFIER() macro facilitates lookup of the fields (along with _Py_GET_GLOBAL_STRING() for non-identifier strings).
https://bugs.python.org/issue46541#msg411799 explains the rationale for this change.
The core of the change is in:
* (new) Include/internal/pycore_global_strings.h - the declarations for the global strings, along with the macros
* Include/internal/pycore_runtime_init.h - added the static initializers for the global strings
* Include/internal/pycore_global_objects.h - where the struct in pycore_global_strings.h is hooked into _PyRuntimeState
* Tools/scripts/generate_global_objects.py - added generation of the global string declarations and static initializers
I've also added a --check flag to generate_global_objects.py (along with make check-global-objects) to check for unused global strings. That check is added to the PR CI config.
The remainder of this change updates the core code to use _Py_GET_GLOBAL_IDENTIFIER() instead of _Py_IDENTIFIER() and the related _Py*Id functions (likewise for _Py_GET_GLOBAL_STRING() instead of _Py_static_string()). This includes adding a few functions where there wasn't already an alternative to _Py*Id(), replacing the _Py_Identifier * parameter with PyObject *.
The following are not changed (yet):
* stop using _Py_IDENTIFIER() in the stdlib modules
* (maybe) get rid of _Py_IDENTIFIER(), etc. entirely -- this may not be doable as at least one package on PyPI using this (private) API
* (maybe) intern the strings during runtime init
https://bugs.python.org/issue46541
Replace PyModule_AddObject() with PyModule_AddObjectRef() in the
_warnings module to fix a reference leak on error.
Use also PyModule_AddObjectRef() in importdl.c.
A bunch of code currently uses PyInterpreterState.modules directly instead of PyImport_GetModuleDict(). This complicates efforts to make changes relative to sys.modules. This patch switches to using PyImport_GetModuleDict() uniformly. Also, a number of related uses of sys.modules are updated for uniformity for the same reason.
Note that this code was already reviewed and merged as part of #1638. I reverted that and am now splitting it up into more focused parts.
PR #1638, for bpo-28411, causes problems in some (very) edge cases. Until that gets sorted out, we're reverting the merge. PR #3506, a fix on top of #1638, is also getting reverted.
Known limitations of the current implementation:
- documentation changes are incomplete
- there's a reference leak I haven't tracked down yet
The leak is most visible by running:
./python -m test -R3:3 test_importlib
However, you can also see it by running:
./python -X showrefcount
Importing the array or _testmultiphase modules, and
then deleting them from both sys.modules and the local
namespace shows significant increases in the total
number of active references each cycle. By contrast,
with _testcapi (which continues to use single-phase
initialisation) the global refcounts stabilise after
a couple of cycles.
importlib._bootstrap is now frozen into Python/importlib.h and stored
as _frozen_importlib in sys.modules. Py_Initialize() loads the frozen
code along with sys and imp and then uses _frozen_importlib._install()
to set builtins.__import__() w/ _frozen_importlib.__import__().
The name must be encodable to ASCII because dynamic module must have a function
called "PyInit_NAME", they are written in C, and the C language doesn't accept
non-ASCII identifiers.
- is_builtin(), init_builtin(), load_builtin() and other builtin related
functions use Unicode strings, instead of byte strings
- Rename _PyImport_FixupExtensionUnicode() to _PyImport_FixupExtensionObject()
- Rename _PyImport_FindExtensionUnicode() to _PyImport_FindExtensionObject()
* Rename _PyImport_FindExtension() to _PyImport_FindExtensionUnicode():
the filename becomes a Unicode object instead of byte string
* Rename _PyImport_FixupExtension() to _PyImport_FixupExtensionUnicode():
the filename becomes a Unicode object instead of byte string
Changes to make __file__ a proper Unicode object, using the default
filesystem encoding.
This is a bit tricky because the default filesystem encoding isn't
set by the time we import the first modules; at that point we fudge
things a bit. This is okay since __file__ isn't really used much
except for error reporting.
Tested on OSX and Linux only so far.
When an extension imports another extension in its
initXXX() function, the variable _Py_PackageContext is
prematurely reset to NULL. If the outer extension then
calls Py_InitModule(), the extension is installed in
sys.modules without its package name. The
manifestation of this bug is a "SystemError:
_PyImport_FixupExtension: module <package>.<extension>
not loaded".
To fix this, importdl.c just needs to retain the old
value of _Py_PackageContext and restore it after the
initXXX() method is called. The attached patch does this.
This patch applies to Python 2.1.1 and the current CVS.
* in import.c, #ifdef out references to dynamic loading based on
HAVE_DYNAMIC_LOADING
* clean out the platform-specific crud from importdl.c.
[ maybe fold this function into import.c and drop the importdl.c file? Greg.]
* change GetDynLoadFunc's "funcname" parameter to "shortname". change
"name" to "fqname" for clarification.
* each GetDynLoadFunc now creates its own funcname value.
WARNING: as I mentioned previously, we may run into an issue with a
missing "_" on some platforms. Testing will show this pretty quickly,
however.
* move pathname munging into dynload_shlib.c
Duzan, for AIX, to support C++ objects with static initializers, when
using the genuine IBM C++ compiler (namely xlC/xlC_r).
See accompanying patches to configure.in and acconfig.h.
"""
Following up Robin Dunn's troubles with freeze, here's a patch that
fixes an oddity regarding the import logic of shared modules on AIX.
Symbol resolution of shared modules is now handled properly for the cases
when the python library is linked to a binary with an arbitrary name.
This includes the standard python[version] executable, but also applications
that are embedding the python core (i.e. linked with libpython[version].a,
the latter being static or shared).
"""
The following patches (relative to 1.5.2b1) enable Python dynamic
loading to work on NetBSD platforms that use ELF (presnetly mips and
alpha systems). They automaticly determine wether the system is ELF or
a.out rather than using astatic list of platforms so that when other
NetBSD platforms move to ELF, python will continue to work without
change.
Christian Heimes
Eric Snow
Serhiy Storchaka
Dong-hee Na
Victor Stinner
Andy Lester
Steve Dower
Oren Milman
Benjamin Peterson
Nick Coghlan
Victor Stinner
Christian Heimes
Brett Cannon
Victor Stinner
Antoine Pitrou
Martin v. Löwis
Neal Norwitz
Guido van Rossum
Fred Drake
Thomas Wouters