* Refactor zipfile._strip_extra to use higher level abstractions for extras instead of a heavy-state loop.
* Add blurb
* Remove _strip_extra and use _Extra.strip directly.
* Use memoryview to avoid unnecessary copies while splitting Extras.
Fix test_zippath_from_non_installed_posix() of test_venv: don't copy
__pycache__/ sub-directories, because they can be modified by other
Python tests running in parallel.
PyImport_GetImporter() now sets RuntimeError if it fails to get sys.path_hooks
or sys.path_importer_cache or they are not list and dict correspondingly.
Previously it could return NULL without setting error in obscure cases,
crash or raise SystemError if these attributes have wrong type.
This fixes issue #105829, https://github.com/python/cpython/issues/105829
Co-authored-by: blurb-it[bot] <43283697+blurb-it[bot]@users.noreply.github.com>
Co-authored-by: Antoine Pitrou <antoine@python.org>
Co-authored-by: Chris Withers <chris@withers.org>
Co-authored-by: Thomas Moreau <thomas.moreau.2010@gmail.com>
Fix os.stat() and os.DirEntry.stat(): check for exceptions.
Previously, on Python built in debug mode, these functions could
trigger a fatal Python error (and abort the process) when a function
succeeded with an exception set.
_pystat_fromstructstat() now exits immediately if an exception is
raised, rather only checking for exceptions at the end. It fix
following fatal error in fill_time():
Fatal Python error: _Py_CheckSlotResult:
Slot * of type int succeeded with an exception set
Skip test_freeze_simple_script() of test_tools.test_freeze if Python
is built with "./configure --enable-optimizations", which means with
Profile Guided Optimization (PGO): it just makes the test too slow.
The freeze tool is tested by many other CIs with other (faster)
compiler flags.
test.pythoninfo now gets also get_build_info() of
test.libregrtests.utils.
Adds APIs to get the TLS certificate chains, verified or full unverified, from SSLSocket and SSLObject.
Co-authored-by: Gregory P. Smith [Google LLC] <greg@krypto.org>
The filename was previously intentionally omitted from exception because
"it might confuse the user". Uncaught exceptions are not generally a
replacement for user-facing error messages, so obscuring this
information only has the effect of making the programmer's life more
difficult.
PyMutex is a one byte lock with fast, inlineable lock and unlock functions for the common uncontended case. The design is based on WebKit's WTF::Lock.
PyMutex is built using the _PyParkingLot APIs, which provides a cross-platform futex-like API (based on WebKit's WTF::ParkingLot). This internal API will be used for building other synchronization primitives used to implement PEP 703, such as one-time initialization and events.
This also includes tests and a mini benchmark in Tools/lockbench/lockbench.py to compare with the existing PyThread_type_lock.
Uncontended acquisition + release:
* Linux (x86-64): PyMutex: 11 ns, PyThread_type_lock: 44 ns
* macOS (arm64): PyMutex: 13 ns, PyThread_type_lock: 18 ns
* Windows (x86-64): PyMutex: 13 ns, PyThread_type_lock: 38 ns
PR Overview:
The primary purpose of this PR is to implement PyMutex, but there are a number of support pieces (described below).
* PyMutex: A 1-byte lock that doesn't require memory allocation to initialize and is generally faster than the existing PyThread_type_lock. The API is internal only for now.
* _PyParking_Lot: A futex-like API based on the API of the same name in WebKit. Used to implement PyMutex.
* _PyRawMutex: A word sized lock used to implement _PyParking_Lot.
* PyEvent: A one time event. This was used a bunch in the "nogil" fork and is useful for testing the PyMutex implementation, so I've included it as part of the PR.
* pycore_llist.h: Defines common operations on doubly-linked list. Not strictly necessary (could do the list operations manually), but they come up frequently in the "nogil" fork. ( Similar to https://man.freebsd.org/cgi/man.cgi?queue)
---------
Co-authored-by: Eric Snow <ericsnowcurrently@gmail.com>
On a Python built in debug mode, Py_DECREF() now calls
_Py_NegativeRefcount() if the object is a dangling pointer to
deallocated memory: memory filled with 0xDD "dead byte" by the debug
hook on memory allocators. The fix is to check the reference count
*before* checking for _Py_IsImmortal().
Add test_decref_freed_object() to test_capi.test_misc.
This feature is off by default via code but on by default via the CLI. The `.gitignore` file contains `*` which causes the entire directory to be ignored.
Co-authored-by: Adam Turner <9087854+AA-Turner@users.noreply.github.com>
Co-authored-by: Hugo van Kemenade <hugovk@users.noreply.github.com>
There is a WIP proposal to enable webassembly stack switching which have been
implemented in v8:
https://github.com/WebAssembly/js-promise-integration
It is not possible to switch stacks that contain JS frames so the Emscripten JS
trampolines that allow calling functions with the wrong number of arguments
don't work in this case. However, the js-promise-integration proposal requires
the [type reflection for Wasm/JS API](https://github.com/WebAssembly/js-types)
proposal, which allows us to actually count the number of arguments a function
expects.
For better compatibility with stack switching, this PR checks if type reflection
is available, and if so we use a switch block to decide the appropriate
signature. If type reflection is unavailable, we should use the current EMJS
trampoline.
We cache the function argument counts since when I didn't cache them performance
was negatively affected.
Co-authored-by: T. Wouters <thomas@python.org>
Co-authored-by: Brett Cannon <brett@python.org>
libregrtest now decodes stdout of test worker processes with the
"backslashreplace" error handler to log corrupted stdout, instead of
failing with an error and not logging the stdout.
Fix test_gdb on Python built with LLVM clang 16 on Linux ppc64le (ex:
Fedora 38). Search patterns in gdb "bt" command output to detect
when gdb fails to retrieve the traceback. For example, skip a test if
"Backtrace stopped: frame did not save the PC" is found.
libregrtest now calls random.seed() before running each test file
when -r/--randomize command line option is used. Moreover, it's also
called in worker processes. It should help to make tests more
deterministic. Previously, it was only called once in the main
process before running all test files and it was not called in worker
processes.
* Convert some f-strings to regular strings in test_regrtest when
f-string is not needed.
* Remove unused all_methods variable from test_regrtest.
* Add RunTests members are now mandatory.
libregrtest now uses a separated file descriptor to write test result
as JSON. Previously, if a test wrote debug messages late around the
JSON, the main test process failed to parse JSON.
Rename TestResult.write_json() to TestResult.write_json_into().
worker_process() no longer writes an empty line at the end. There is
no need to separate test process output from the JSON output anymore,
since JSON is now written into a separated file descriptor.
create_worker_process() now always spawn the process with
close_fds=True.
Fix _thread.start_new_thread() race condition. If a thread is created
during Python finalization, the newly spawned thread now exits
immediately instead of trying to access freed memory and lead to a
crash.
thread_run() calls PyEval_AcquireThread() which checks if the thread
must exit. The problem was that tstate was dereferenced earlier in
_PyThreadState_Bind() which leads to a crash most of the time.
Move _PyThreadState_CheckConsistency() from thread_run() to
_PyThreadState_Bind().
Fix a race condition in concurrent.futures. When a process in the
process pool was terminated abruptly (while the future was running or
pending), close the connection write end. If the call queue is
blocked on sending bytes to a worker process, closing the connection
write end interrupts the send, so the queue can be closed.
Changes:
* _ExecutorManagerThread.terminate_broken() now closes
call_queue._writer.
* multiprocessing PipeConnection.close() now interrupts
WaitForMultipleObjects() in _send_bytes() by cancelling the
overlapped operation.
Fix test_pyexpat.test_exception(): it can now be run from a directory
different than Python source code directory. Before, the test failed
in this case.
Skip the test if Modules/pyexpat.c source is not available. Skip also
the test on Python implementations other than CPython.
Fix test_site.test_underpth_basic() when the working directory
contains at least one non-ASCII character: encode the "._pth" file to
UTF-8 and enable the UTF-8 Mode to use UTF-8 for the child process
stdout.
Attempts to pickle or create a shallow or deep copy of codecs streams
now raise a TypeError.
Previously, copying failed with a RecursionError, while pickling
produced wrong results that eventually caused unpickling to fail with
a RecursionError.
Fix building the _testcapi extension on Linux AArch64 which requires
linking to libatomic when <cpython/pyatomic.h> is used: the
_Py_atomic_or_uint64() function requires libatomic
__atomic_fetch_or_8() on this platform.
The configure script now checks if linking to libatomic is needed and
generates a new LIBATOMIC variable used to build the _testcapi
extension.
Building the _testcapi extension now uses the LIBATOMIC variable in
its LDFLAGS, since Modules/_testcapi/pyatomic.c uses
<cpython/pyatomic.h>.
Co-authored-by: Erlend E. Aasland <erlend.aasland@protonmail.com>
add guard so that ``Enum('bar')`` raises a TypeError instead of
creating a new enum class called `bar`. To create the new but
empty class, use:
huh = Enum('bar', names=())
It creates a modified copy of an object by calling the object's
__replace__() method.
It is a generalization of dataclasses.replace(), named tuple's _replace()
method and replace() methods in various classes, and supports all these
stdlib classes.
Fix a race condition in "make regen-all". The deepfreeze.c source and
files generated by Argument Clinic are now generated or updated
before generating "global objects". Previously, some identifiers may
miss depending on the order in which these files were generated.
* "make regen-global-objects": Make sure that deepfreeze.c is
generated and up to date, and always run "make clinic".
* "make clinic" no longer runs generate_global_objects.py script.
* "make regen-deepfreeze" now only updates deepfreeze.c (C file).
It doesn't build deepfreeze.o (object) anymore.
* Remove misleading messages in "make regen-global-objects" and
"make clinic". They are now outdated, these commands are now
safe to use.
* Document generates files in Doc/using/configure.rst.
Co-authored-by: Erlend E. Aasland <erlend@python.org>
* Add get_recursion_available() and get_recursion_depth() functions
to the test.support module.
* Change infinite_recursion() default max_depth from 75 to 100.
* Fix test_tomllib recursion tests for WASI buildbots: reduce the
recursion limit and compute the maximum nested array/dict depending
on the current available recursion limit.
* test.pythoninfo logs sys.getrecursionlimit().
* Enhance test_sys tests on sys.getrecursionlimit()
and sys.setrecursionlimit().
Cirrus-CI is a hosted CI service that supports FreeBSD, Linux, macOS,
and Windows. Add a .cirrus.yml configuration file to provide CI coverage
on pull requests for FreeBSD 13.2.
Co-authored-by: Victor Stinner <vstinner@python.org>
Skip test_gdb if gdb is unable to retrieve Python frame objects: if a
frame is "<optimized out>". When Python is built with "clang -Og",
gdb can fail to retrive the 'frame' parameter of
_PyEval_EvalFrameDefault(). In this case, tests like py_bt() are
likely to fail. Without getting access to Python frames,
python-gdb.py is mostly clueless on retrieving the Python traceback.
Moreover, test_gdb is no longer skipped on macOS if Python is built
with Clang.
When using --rerun option, regrtest now re-runs failed tests
in verbose mode in fresh worker processes to have more
deterministic behavior. So it can write its final report even
if a test killed a worker progress.
Add --fail-rerun option to regrtest: exit with non-zero exit code
if a test failed pass passed when re-run in verbose mode (in a
fresh process). That's now more useful since tests can pass
when re-run in a fresh worker progress, whereas they failed
when run after other tests when tests are run sequentially.
Rename --verbose2 option (-w) to --rerun. Keep --verbose2 as a
deprecated alias.
Changes:
* Fix and enhance statistics in regrtest summary. Add "(filtered)"
when --match and/or --ignore options are used.
* Add RunTests class.
* Add TestResult.get_rerun_match_tests() method
* Rewrite code to serialize/deserialize worker arguments as JSON
using a new WorkerJob class.
* Fix stats when a test is run with --forever --rerun.
* If failed test names cannot be parsed, log a warning and don't
filter tests.
* test_regrtest.test_rerun_success() now uses a marker file, since
the test is re-run in a separated process.
* Add tests on normalize_test_name() function.
* Add test_success() and test_skip() tests to test_regrtest.
Remove <ctype.h> in C files which don't use it; only sre.c and
_decimal.c still use it.
Remove _PY_PORT_CTYPE_UTF8_ISSUE code from pyport.h:
* Code added by commit b5047fd019
in 2004 for MacOSX and FreeBSD.
* Test removed by commit 52ddaefb6b
in 2007, since Python str type now uses locale independent
functions like Py_ISALPHA() and Py_TOLOWER() and the Unicode
database.
Modules/_sre/sre.c replaces _PY_PORT_CTYPE_UTF8_ISSUE with new
functions: sre_isalnum(), sre_tolower(), sre_toupper().
Remove unused includes:
* _localemodule.c: remove <stdio.h>.
* getargs.c: remove <float.h>.
* dynload_win.c: remove <direct.h>, it no longer calls _getcwd()
since commit fb1f68ed7c (in 2001).
Python.h no longer includes <time.h>, <sys/select.h> and <sys/time.h>
standard header files.
* Add <time.h> include to xxsubtype.c.
* Add <sys/time.h> include to posixmodule.c and semaphore.c.
* readline.c includes <sys/select.h> instead of <sys/time.h>.
* resource.c no longer includes <time.h> and <sys/time.h>.
* Add 'skipped' attribute to TestResults.
* Add 'skips' attribute to DocTestRunner.
* Rename private DocTestRunner._name2ft attribute
to DocTestRunner._stats.
* Use f-string for string formatting.
* Add some tests.
* Document DocTestRunner attributes and its API for statistics.
* Document TestResults class.
Co-authored-by: Alex Waygood <Alex.Waygood@Gmail.com>
This adds a new header that provides atomic operations on common data
types. The intention is that this will be exposed through Python.h,
although that is not the case yet. The only immediate use is in
the test file.
Co-authored-by: Sam Gross <colesbury@gmail.com>
When overriding the `__new__` method of an enum, the underlying data type should be created directly; i.e. .
member = object.__new__(cls)
member = int.__new__(cls, value)
member = str.__new__(cls, value)
Calling `super().__new__()` finds the lookup version of `Enum.__new__`, and will now raise an exception when detected.
Python built with "configure --with-trace-refs" (tracing references)
is now ABI compatible with Python release build and debug build.
Moreover, it now also supports the Limited API.
Change Py_TRACE_REFS build:
* Remove _PyObject_EXTRA_INIT macro.
* The PyObject structure no longer has two extra members (_ob_prev
and _ob_next).
* Use a hash table (_Py_hashtable_t) to trace references (all
objects): PyInterpreterState.object_state.refchain.
* Py_TRACE_REFS build is now ABI compatible with release build and
debug build.
* Limited C API extensions can now be built with Py_TRACE_REFS:
xxlimited, xxlimited_35, _testclinic_limited.
* No longer rename PyModule_Create2() and PyModule_FromDefAndSpec2()
functions to PyModule_Create2TraceRefs() and
PyModule_FromDefAndSpec2TraceRefs().
* _Py_PrintReferenceAddresses() is now called before
finalize_interp_delete() which deletes the refchain hash table.
* test_tracemalloc find_trace() now also filters by size to ignore
the memory allocated by _PyRefchain_Trace().
Test changes for Py_TRACE_REFS:
* Add test.support.Py_TRACE_REFS constant.
* Add test_sys.test_getobjects() to test sys.getobjects() function.
* test_exceptions skips test_recursion_normalizing_with_no_memory()
and test_memory_error_in_PyErr_PrintEx() if Python is built with
Py_TRACE_REFS.
* test_repl skips test_no_memory().
* test_capi skisp test_set_nomemory().
gh-107275 introduced a regression where a SemLock would fail being passed along nested child processes, as the `is_fork_ctx` attribute would be left missing after the first deserialization.
---------
Co-authored-by: blurb-it[bot] <43283697+blurb-it[bot]@users.noreply.github.com>
Co-authored-by: Antoine Pitrou <pitrou@free.fr>
Deprecate passing the callback callable by keyword for the following
sqlite3.Connection APIs:
- set_authorizer(authorizer_callback)
- set_progress_handler(progress_handler, ...)
- set_trace_callback(trace_callback)
The affected parameters will become positional-only in Python 3.15.
This mis-initialization caused the executor optimization to kick in sooner than intended. It also set the lower 4 bits of the counter to `1` -- those bits are supposed to be reserved (the actual counter is in the upper 12 bits).
Deprecate passing name, number of arguments, and the callable as keyword
arguments, for the following sqlite3.Connection APIs:
- create_function(name, nargs, callable, ...)
- create_aggregate(name, nargs, callable)
The affected parameters will become positional-only in Python 3.15.
Functions like PyErr_SetFromErrno() and SetFromWindowsErr() should be
called immediately after using the C API which sets errno or the Windows
error code.
Argument Clinic now has a partial support of the
Limited API:
* Add --limited option to clinic.c.
* Add '_testclinic_limited' extension which is built with
the limited C API version 3.13.
* For now, hardcode in clinic.py that "_testclinic_limited.c" targets
the limited C API.
- Move platform triplet detection code into Misc/platform_triplet.c
- Refactor MIPS detection, use defined(__mips64) to detect MIPS64
- Compute libc values in separate section
- Add detection for MIPS soft float
- Add detection for musl
musl supports SPE with its soft-float ABI:
https://git.musl-libc.org/cgit/musl/commit/?id=7be59733d71ada3a32a98622507399253f1d5e48
Original patch by Christian Heimes.
Co-authored-by: Christian Heimes <christian@python.org>
Co-authored-by: Erlend E. Aasland <erlend@python.org>
Split test_multiprocessing_fork, test_multiprocessing_forkserver and
test_multiprocessing_spawn into test packages. Each package is made
of 4 sub-tests: processes, threads, manager and misc. It allows
running more tests in parallel and so reduce the total test duration.
Ensure multiprocessing SemLock is valid for spawn Process before serializing it.
Creating a multiprocessing SemLock with a fork context, and then trying to pass it to a spawn-created Process, would segfault if not detected early.
---------
Co-authored-by: blurb-it[bot] <43283697+blurb-it[bot]@users.noreply.github.com>
Co-authored-by: Antoine Pitrou <pitrou@free.fr>
Antoine Pitrou
Serhiy Storchaka
Victor Stinner
Jason R. Coombs
Irit Katriel
Alex Waygood
Tian Gao
Hugo van Kemenade
Pablo Galindo Salgado
Adam Turner
elfstrom
EliseevEgor
AN Long
Heinz-Alexander Fuetterer
James Gerity
Jelle Zijlstra
Mateusz Nowak
Ronan Pigott
Sam Gross
Xuehai Pan
Shantanu
Brett Cannon
Hood Chatham
Gregory P. Smith
buermarc
Carl Meyer
Brandt Bucher
Nikita Sobolev
Mark Shannon
云line
Pieter Eendebak
Ethan Furman
Mark Dickinson
Ijtaba Hussain
Ed Maste
Anthony Shaw
Zachary Ware
Виталий Дмитриев
Barney Gale
albanD
Erlend E. Aasland
Corvin
Guido van Rossum
Matthias Bussonnier
Ori Avtalion
Raymond Hettinger
R
denballakh
Peeyush Aggarwal
Samuel Thibault
Jeffery To
Chris Markiewicz
Petr Viktorin