Fix os.stat() and os.DirEntry.stat(): check for exceptions.
Previously, on Python built in debug mode, these functions could
trigger a fatal Python error (and abort the process) when a function
succeeded with an exception set.
_pystat_fromstructstat() now exits immediately if an exception is
raised, rather only checking for exceptions at the end. It fix
following fatal error in fill_time():
Fatal Python error: _Py_CheckSlotResult:
Slot * of type int succeeded with an exception set
Skip test_freeze_simple_script() of test_tools.test_freeze if Python
is built with "./configure --enable-optimizations", which means with
Profile Guided Optimization (PGO): it just makes the test too slow.
The freeze tool is tested by many other CIs with other (faster)
compiler flags.
test.pythoninfo now gets also get_build_info() of
test.libregrtests.utils.
Adds APIs to get the TLS certificate chains, verified or full unverified, from SSLSocket and SSLObject.
Co-authored-by: Gregory P. Smith [Google LLC] <greg@krypto.org>
The filename was previously intentionally omitted from exception because
"it might confuse the user". Uncaught exceptions are not generally a
replacement for user-facing error messages, so obscuring this
information only has the effect of making the programmer's life more
difficult.
PyMutex is a one byte lock with fast, inlineable lock and unlock functions for the common uncontended case. The design is based on WebKit's WTF::Lock.
PyMutex is built using the _PyParkingLot APIs, which provides a cross-platform futex-like API (based on WebKit's WTF::ParkingLot). This internal API will be used for building other synchronization primitives used to implement PEP 703, such as one-time initialization and events.
This also includes tests and a mini benchmark in Tools/lockbench/lockbench.py to compare with the existing PyThread_type_lock.
Uncontended acquisition + release:
* Linux (x86-64): PyMutex: 11 ns, PyThread_type_lock: 44 ns
* macOS (arm64): PyMutex: 13 ns, PyThread_type_lock: 18 ns
* Windows (x86-64): PyMutex: 13 ns, PyThread_type_lock: 38 ns
PR Overview:
The primary purpose of this PR is to implement PyMutex, but there are a number of support pieces (described below).
* PyMutex: A 1-byte lock that doesn't require memory allocation to initialize and is generally faster than the existing PyThread_type_lock. The API is internal only for now.
* _PyParking_Lot: A futex-like API based on the API of the same name in WebKit. Used to implement PyMutex.
* _PyRawMutex: A word sized lock used to implement _PyParking_Lot.
* PyEvent: A one time event. This was used a bunch in the "nogil" fork and is useful for testing the PyMutex implementation, so I've included it as part of the PR.
* pycore_llist.h: Defines common operations on doubly-linked list. Not strictly necessary (could do the list operations manually), but they come up frequently in the "nogil" fork. ( Similar to https://man.freebsd.org/cgi/man.cgi?queue)
---------
Co-authored-by: Eric Snow <ericsnowcurrently@gmail.com>
On a Python built in debug mode, Py_DECREF() now calls
_Py_NegativeRefcount() if the object is a dangling pointer to
deallocated memory: memory filled with 0xDD "dead byte" by the debug
hook on memory allocators. The fix is to check the reference count
*before* checking for _Py_IsImmortal().
Add test_decref_freed_object() to test_capi.test_misc.
This feature is off by default via code but on by default via the CLI. The `.gitignore` file contains `*` which causes the entire directory to be ignored.
Co-authored-by: Adam Turner <9087854+AA-Turner@users.noreply.github.com>
Co-authored-by: Hugo van Kemenade <hugovk@users.noreply.github.com>
There is a WIP proposal to enable webassembly stack switching which have been
implemented in v8:
https://github.com/WebAssembly/js-promise-integration
It is not possible to switch stacks that contain JS frames so the Emscripten JS
trampolines that allow calling functions with the wrong number of arguments
don't work in this case. However, the js-promise-integration proposal requires
the [type reflection for Wasm/JS API](https://github.com/WebAssembly/js-types)
proposal, which allows us to actually count the number of arguments a function
expects.
For better compatibility with stack switching, this PR checks if type reflection
is available, and if so we use a switch block to decide the appropriate
signature. If type reflection is unavailable, we should use the current EMJS
trampoline.
We cache the function argument counts since when I didn't cache them performance
was negatively affected.
Co-authored-by: T. Wouters <thomas@python.org>
Co-authored-by: Brett Cannon <brett@python.org>
libregrtest now decodes stdout of test worker processes with the
"backslashreplace" error handler to log corrupted stdout, instead of
failing with an error and not logging the stdout.
Fix test_gdb on Python built with LLVM clang 16 on Linux ppc64le (ex:
Fedora 38). Search patterns in gdb "bt" command output to detect
when gdb fails to retrieve the traceback. For example, skip a test if
"Backtrace stopped: frame did not save the PC" is found.
libregrtest now calls random.seed() before running each test file
when -r/--randomize command line option is used. Moreover, it's also
called in worker processes. It should help to make tests more
deterministic. Previously, it was only called once in the main
process before running all test files and it was not called in worker
processes.
* Convert some f-strings to regular strings in test_regrtest when
f-string is not needed.
* Remove unused all_methods variable from test_regrtest.
* Add RunTests members are now mandatory.
libregrtest now uses a separated file descriptor to write test result
as JSON. Previously, if a test wrote debug messages late around the
JSON, the main test process failed to parse JSON.
Rename TestResult.write_json() to TestResult.write_json_into().
worker_process() no longer writes an empty line at the end. There is
no need to separate test process output from the JSON output anymore,
since JSON is now written into a separated file descriptor.
create_worker_process() now always spawn the process with
close_fds=True.
Fix _thread.start_new_thread() race condition. If a thread is created
during Python finalization, the newly spawned thread now exits
immediately instead of trying to access freed memory and lead to a
crash.
thread_run() calls PyEval_AcquireThread() which checks if the thread
must exit. The problem was that tstate was dereferenced earlier in
_PyThreadState_Bind() which leads to a crash most of the time.
Move _PyThreadState_CheckConsistency() from thread_run() to
_PyThreadState_Bind().
Fix a race condition in concurrent.futures. When a process in the
process pool was terminated abruptly (while the future was running or
pending), close the connection write end. If the call queue is
blocked on sending bytes to a worker process, closing the connection
write end interrupts the send, so the queue can be closed.
Changes:
* _ExecutorManagerThread.terminate_broken() now closes
call_queue._writer.
* multiprocessing PipeConnection.close() now interrupts
WaitForMultipleObjects() in _send_bytes() by cancelling the
overlapped operation.
Fix test_pyexpat.test_exception(): it can now be run from a directory
different than Python source code directory. Before, the test failed
in this case.
Skip the test if Modules/pyexpat.c source is not available. Skip also
the test on Python implementations other than CPython.
Fix test_site.test_underpth_basic() when the working directory
contains at least one non-ASCII character: encode the "._pth" file to
UTF-8 and enable the UTF-8 Mode to use UTF-8 for the child process
stdout.
Attempts to pickle or create a shallow or deep copy of codecs streams
now raise a TypeError.
Previously, copying failed with a RecursionError, while pickling
produced wrong results that eventually caused unpickling to fail with
a RecursionError.
AN Long
Victor Stinner
Adam Turner
Irit Katriel
Heinz-Alexander Fuetterer
James Gerity
Jelle Zijlstra
Mateusz Nowak
Ronan Pigott
Sam Gross
Xuehai Pan
Tian Gao
Shantanu
Serhiy Storchaka
Brett Cannon
Hood Chatham
Gregory P. Smith
buermarc
Carl Meyer
Brandt Bucher
Nikita Sobolev
Mark Shannon
云line
Pieter Eendebak