These functions are considered not safe because they suppress all internal errors
and can return wrong result. PyDict_GetItemString and _PyDict_GetItemId can
also silence current exception in rare cases.
Remove no longer used _PyDict_GetItemId.
Add _PyDict_ContainsId and rename _PyDict_Contains into
_PyDict_Contains_KnownHash.
Since c19c5a6, AIX builds have defaulted to using dynload_shlib over
dynload_aix when dlopen is available. This function has been available
since AIX 4.3, which went out of support in 2003, the same year the
previously referenced commit was made. It has been nearly 20 years
since a version of AIX has been supported which has not used
dynload_shlib so there's no reason to keep this legacy code around.
When running in a non-UTF-8 locale, if an error occurs while importing a
native Python module (say because a dependent share library is missing),
the error message string returned may contain non-ASCII code points
causing a UnicodeDecodeError.
PyUnicode_DecodeFSDefault is used for buffers which may contain
filesystem paths. For consistency with os.strerror(),
PyUnicode_DecodeLocale is used for buffers which contain system error
messages. While the shortname parameter is always encoded in ASCII
according to PEP 489, it is left decoded using PyUnicode_FromString to
minimize the changes and since it should not affect the decoding (albeit
_potentially_ slower).
In dynload_hpux, since the error buffer contains a message generated
from a static ASCII string and the module filesystem path,
PyUnicode_DecodeFSDefault is used instead of PyUnicode_DecodeLocale as
is used elsewhere.
* bpo-41894: Fix bugs in dynload error msg handling
For both dynload_aix and dynload_hpux, properly handle the possibility
that decoding strings may return NULL and when such an error happens,
properly decrement any previously decoded strings and return early.
In addition, in dynload_aix, ensure that we pass the decoded string
*object* pathname_ob to PyErr_SetImportError instead of the original
pathname buffer.
Co-authored-by: Serhiy Storchaka <storchaka@gmail.com>
All references to this dynamic loading method were removed in b9949db,
when support for this method was dropped, but the implementation code
was not dropped (seemingly in oversight).
This API is relatively lightweight and organizationally, given that it's
used by multiple modules, it makes sense to move it to fileutils.
Requires making sure that _posixsubprocess is compiled with the appropriate
Py_BUIILD_CORE_BUILTIN macro.
* PyMapping_HasKey() is not safe because it silences all exceptions and can return incorrect result.
* Informative exceptions from PyMapping_DelItem() are overridden with RuntimeError and
the original exception raised before calling remove_module() is lost.
* There is a race condition between PyMapping_HasKey() and PyMapping_DelItem().
- Use the proper asdl sequence when creating empty arguments
- Remove reduntant casts (thanks to new typed asdl_sequences)
- Remove MarshalPrototypeVisitor and some utilities from asdl generator
- Fix the header of `Python/ast.c` (kept from pgen times)
Automerge-Triggered-By: @pablogsal
The hard part was making all the tests pass; there are some subtle issues here, because apparently the future import wasn't tested very thoroughly in previous Python versions.
For example, `inspect.signature()` returned type objects normally (except for forward references), but strings with the future import. We changed it to try and return type objects by calling `typing.get_type_hints()`, but fall back on returning strings if that function fails (which it may do if there are future references in the annotations that require passing in a specific namespace to resolve).
Remove the global _Py_CheckRecursionLimit variable: it has been
replaced by ceval.recursion_limit of the PyInterpreterState
structure.
There is no need to keep the variable for the stable ABI, since
Py_EnterRecursiveCall() and Py_LeaveRecursiveCall() were not usable
in Python 3.8 and older: these macros accessed PyThreadState members,
whereas the PyThreadState structure is opaque in the limited C API.
The new API allows to efficiently send values into native generators
and coroutines avoiding use of StopIteration exceptions to signal
returns.
ceval loop now uses this method instead of the old "private"
_PyGen_Send C API. This translates to 1.6x increased performance
of 'await' calls in micro-benchmarks.
Aside from CPython core improvements, this new API will also allow
Cython to generate more efficient code, benefiting high-performance
IO libraries like uvloop.
* Add new capability to the PEG parser to type variable assignments. For instance:
```
| a[asdl_stmt_seq*]=';'.small_stmt+ [';'] NEWLINE { a }
```
* Add new sequence types from the asdl definition (automatically generated)
* Make `asdl_seq` type a generic aliasing pointer type.
* Create a new `asdl_generic_seq` for the generic case using `void*`.
* The old `asdl_seq_GET`/`ast_seq_SET` macros now are typed.
* New `asdl_seq_GET_UNTYPED`/`ast_seq_SET_UNTYPED` macros for dealing with generic sequences.
* Changes all possible `asdl_seq` types to use specific versions everywhere.
Partially revert commit ac46eb4ad6662cf6d771b20d8963658b2186c48c:
"bpo-38113: Update the Python-ast.c generator to PEP384 (gh-15957)".
Using a module state per module instance is causing subtle practical
problems.
For example, the Mercurial project replaces the __import__() function
to implement lazy import, whereas Python expected that "import _ast"
always return a fully initialized _ast module.
Add _PyAST_Fini() to clear the state at exit.
The _ast module has no state (set _astmodule.m_size to 0). Remove
astmodule_traverse(), astmodule_clear() and astmodule_free()
functions.
Fix GCC 9.3 (using -O3) warnings on x86:
initconfig.c: In function ‘init_dump_ascii_wstr’:
initconfig.c:2679:34: warning: format ‘%lc’ expects argument of type
‘wint_t’, but argument 2 has type ‘wchar_t’ {aka ‘long int’}
2679 | PySys_WriteStderr("%lc", ch);
initconfig.c:2682:38: warning: format ‘%x’ expects argument of type
‘unsigned int’, but argument 2 has type ‘wchar_t’ {aka ‘long int’}
2682 | PySys_WriteStderr("\\x%02x", ch);
initconfig.c:2686:38: warning: format ‘%x’ expects argument of type
‘unsigned int’, but argument 2 has type ‘wchar_t’ {aka ‘long int’}
2686 | PySys_WriteStderr("\\U%08x", ch);
initconfig.c:2690:38: warning: format ‘%x’ expects argument of type
‘unsigned int’, but argument 2 has type ‘wchar_t’ {aka ‘long int’}
2690 | PySys_WriteStderr("\\u%04x", ch);
* bpo-41524: fix pointer bug in PyOS_mystr{n}icmp
The existing implementations of PyOS_mystrnicmp and PyOS_mystricmp
can increment pointers beyond the end of a string.
This commit fixes those cases by moving the mutation out of the condition.
* 📜🤖 Added by blurb_it.
* Address comments
Co-authored-by: blurb-it[bot] <43283697+blurb-it[bot]@users.noreply.github.com>
* Merge gen and frame state variables into one.
* Replace stack pointer with depth in PyFrameObject. Makes code easier to read and saves a word of memory.
Fix a crash in the _ast module: it can no longer be loaded more than
once. It now uses a global state rather than a module state.
* Move _ast module state: use a global state instead.
* Set _astmodule.m_size to -1, so the extension cannot be loaded more
than once.
Rework asdl_c.py to pass the module state to functions in
Python-ast.c, instead of using astmodulestate_global.
Handle also PyState_AddModule() failure in init_types().
Add sys.orig_argv attribute: the list of the original command line
arguments passed to the Python executable.
Rename also PyConfig._orig_argv to PyConfig.orig_argv and
document it.
This commit changes the parsing of f-string expressions with the new parser. The parser gets pre-fed with the location of the expression itself (not the f-string, which was what we were doing before). This allows us to completely skip the shifting of the AST nodes after the parsing is completed.
Always create the empty bytes string singleton.
Optimize PyBytes_FromStringAndSize(str, 0): it no longer has to check
if the empty string singleton was created or not, it is always
available.
Add functions:
* _PyBytes_Init()
* bytes_get_empty(), bytes_new_empty()
* bytes_create_empty_string_singleton()
* unicode_create_empty_string_singleton()
_Py_unicode_state: rename empty structure member to empty_string.
Py_InitializeFromConfig() now always creates the empty tuple
singleton as soon as possible.
Optimize PyTuple_New(0): it no longer has to check if the empty tuple
was created or not, it is always creatd.
* Add tuple_create_empty_tuple_singleton() function.
* Add tuple_get_empty() function.
* Remove state parameter of tuple_alloc().
Each interpreter now has its own MemoryError free list: it is not
longer shared by all interpreters.
Add _Py_exc_state structure and PyInterpreterState.exc_state member.
Move also errnomap into _Py_exc_state.
* Revert "bpo-40521: Make the empty frozenset per interpreter (GH-21068)"
This reverts commit 261cfedf76.
* bpo-40521: Empty frozensets are no longer singletons
* Complete the removal of the frozenset singleton
Each interpreter now has its own empty bytes string and single byte
character singletons.
Replace STRINGLIB_EMPTY macro with STRINGLIB_GET_EMPTY() macro.
Each interpreter now has its own dict free list:
* Move dict free lists into PyInterpreterState.
* Move PyDict_MAXFREELIST define to pycore_interp.h
* Add _Py_dict_state structure.
* Add tstate parameter to _PyDict_ClearFreeList() and _PyDict_Fini().
* In debug mode, ensure that the dict free lists are not used after
_PyDict_Fini() is called.
* Remove "#ifdef EXPERIMENTAL_ISOLATED_SUBINTERPRETERS".
zip() now supports PEP 618's strict parameter, which raises a
ValueError if the arguments are exhausted at different lengths.
Patch by Brandt Bucher.
Co-authored-by: Brandt Bucher <brandtbucher@gmail.com>
Co-authored-by: Ram Rachum <ram@rachum.com>
The PY_SSIZE_T_CLEAN macro must now be defined to use
PyArg_ParseTuple() and Py_BuildValue() "#" formats: "es#", "et#",
"s#", "u#", "y#", "z#", "U#" and "Z#". See the PEP 353.
Update _testcapi.test_buildvalue_issue38913().
When a file ends with a line that contains a line continuation character
the text of the emitted SyntaxError is empty, contrary to the old
parser, where the error text contained the text of the last line.
The C99 functions snprintf() and vsnprintf() are now required
to build Python.
PyOS_snprintf() and PyOS_vsnprintf() no longer call Py_FatalError().
Previously, they called Py_FatalError() on a buffer overflow on platforms
which don't provide vsnprintf().
On Windows, #include "pyerrors.h" no longer defines "snprintf" and
"vsnprintf" macros.
PyOS_snprintf() and PyOS_vsnprintf() should be used to get portable
behavior.
Replace snprintf() calls with PyOS_snprintf() and replace vsnprintf()
calls with PyOS_vsnprintf().
In GH-2866, _Py_Bit_Length() was added to pymath.h for lack of a better
location. GH-20518 added a more appropriate header file for bit utilities. It
also shows how to properly use intrinsics. This allows reconsidering bpo-29782.
* Move the function to the new header.
* Changed return type to match __builtin_clzl() and reviewed usage.
* Use intrinsics where available.
* Pick a fallback implementation suitable for inlining.
This commit removes the old parser, the deprecated parser module, the old parser compatibility flags and environment variables and all associated support code and documentation.
The PEP 353, written in 2005, introduced PY_FORMAT_SIZE_T. Python no
longer supports macOS 10.4 and Visual Studio 2010, but requires more
recent macOS and Visual Studio versions. In 2020 with Python 3.10, it
is now safe to use directly "%zu" to format size_t and "%zi" to
format Py_ssize_t.
Export explicitly the Py_GetArgcArgv() function to the C API and
document the function. Previously, it was exported implicitly which
no longer works since Python is built with -fvisibility=hidden.
* Add PyConfig._orig_argv member.
* Py_InitializeFromConfig() no longer calls _PyConfig_Write() twice.
* PyConfig_Read() no longer initializes Py_GetArgcArgv(): it is now
_PyConfig_Write() responsibility.
* _PyConfig_Write() result type becomes PyStatus instead of void.
* Write an unit test on Py_GetArgcArgv().
* Rename pycore_byteswap.h to pycore_bitutils.h.
* Move popcount_digit() to pycore_bitutils.h as _Py_popcount32().
* _Py_popcount32() uses GCC and clang builtin function if available.
* Add unit tests to _Py_popcount32().
* Provide native .files support on SourceFileLoader.
* Add native importlib.resources.files() support to zipimporter. Remove fallback support.
* make regen-all
* 📜🤖 Added by blurb_it.
* Move 'files' into the ResourceReader so it can carry the relevant module name context.
* Create 'importlib.readers' module and add FileReader to it.
* Add zip reader and rely on it for a TraversableResources object on zipimporter.
* Remove TraversableAdapter, no longer needed.
* Update blurb.
* Replace backslashes with forward slashes.
* Incorporate changes from importlib_metadata 2.0, finalizing the interface for extension via get_resource_reader.
Co-authored-by: blurb-it[bot] <43283697+blurb-it[bot]@users.noreply.github.com>
In debug mode, ensure that free lists are no longer used after being
finalized. Set numfree to -1 in finalization functions
(eg. _PyList_Fini()), and then check that numfree is not equal to -1
before using a free list (e.g list_dealloc()).
Each interpreter now has its own context free list:
* Move context free list into PyInterpreterState.
* Add _Py_context_state structure.
* Add tstate parameter to _PyContext_ClearFreeList()
and _PyContext_Fini().
* Pass tstate to clear_freelists().
Each interpreter now has its own asynchronous generator free lists:
* Move async gen free lists into PyInterpreterState.
* Move _PyAsyncGen_MAXFREELIST define to pycore_interp.h
* Add _Py_async_gen_state structure.
* Add tstate parameter to _PyAsyncGen_ClearFreeLists
and _PyAsyncGen_Fini().
Serhiy Storchaka
TIGirardi
Pablo Galindo
Kevin Adler
Hai Shi
Kyle Evans
Batuhan Taskaya
Vladimir Matveev
Victor Stinner
Mark Shannon
Samuel Marks
han-solo
Tony Solomonik
wmeehan
wasiher
Benjamin Peterson
Matthias Bussonnier
Steve Dower
Konge
tkmikan
Lysandros Nikolaou
Raymond Hettinger
Christian Heimes
Gregory P. Smith
Guido van Rossum
Inada Naoki
Niklas Fiekas
An Long
Dong-hee Na
Sandro Mani
Jason R. Coombs
Batuhan Taskaya