Starts adding thread safety to dict objects.
Use @critical_section for APIs which are exposed via argument clinic and don't directly correlate with a public C API which needs to acquire the lock
Use a _lock_held suffix for keeping changes to complicated functions simple and just wrapping them with a critical section
Acquire and release the lock in an existing function where it won't be overly disruptive to the existing logic
This marks dead ThreadHandles as non-joinable earlier in
`PyOS_AfterFork_Child()` before we execute any Python code. The handles
are stored in a global linked list in `_PyRuntimeState` because `fork()`
affects the entire process.
We do not want to add locking in `tp_traverse` slot implementations.
Instead, stop the world when calling `gc.get_referents`. Note that the the
stop the world call is a no-op in the default build.
Co-authored-by: Pablo Galindo Salgado <Pablogsal@gmail.com>
The `PyDict_SetDefaultRef` function is similar to `PyDict_SetDefault`,
but returns a strong reference through the optional `**result` pointer
instead of a borrowed reference.
Co-authored-by: Petr Viktorin <encukou@gmail.com>
Add optional 'filter' parameter to iterdump() that allows a "LIKE"
pattern for filtering database objects to dump.
Co-authored-by: Erlend E. Aasland <erlend@python.org>
The new `PyList_GetItemRef` is similar to `PyList_GetItem`, but returns
a strong reference instead of a borrowed reference. Additionally, if the
passed "list" object is not a list, the function sets a `TypeError`
instead of calling `PyErr_BadInternalCall()`.
* gh-112529: Remove PyGC_Head from object pre-header in free-threaded build
This avoids allocating space for PyGC_Head in the free-threaded build.
The GC implementation for free-threaded CPython does not use the
PyGC_Head structure.
* The trashcan mechanism uses the `ob_tid` field instead of `_gc_prev`
in the free-threaded build.
* The GDB libpython.py file now determines the offset of the managed
dict field based on whether the running process is a free-threaded
build. Those are identified by the `ob_ref_local` field in PyObject.
* Fixes `_PySys_GetSizeOf()` which incorrectly incorrectly included the
size of `PyGC_Head` in the size of static `PyTypeObject`.
PyObject_GetBuffer() now raises a SystemError if called with
PyBUF_READ or PyBUF_WRITE as flags. These flags should
only be used with the PyMemoryView_* C API.
Signed-off-by: Soumendra Ganguly <soumendraganguly@gmail.com>
Co-authored-by: Gregory P. Smith <greg@krypto.org>
Co-authored-by: Petr Viktorin <encukou@gmail.com>
`threading.Lock` is now the underlying class and is constructable rather than the old
factory function. This allows for type annotations to refer to it which had no non-ugly
way to be expressed prior to this.
---------
Co-authored-by: Alex Waygood <Alex.Waygood@Gmail.com>
Co-authored-by: Gregory P. Smith <greg@krypto.org>
* use the ParkingLot API to manage waiting threads
* use Argument Clinic's critical section directive to protect queue methods
* remove unnecessary overflow check
Co-authored-by: Erlend E. Aasland <erlend.aasland@protonmail.com>
Use a ring buffer instead of a Python list in order to simplify the
process of making queue.SimpleQueue thread-safe in free-threaded
builds. The ring buffer implementation has no places where critical
sections may be released.
Only set filename to cwd if it was caused by failed chdir(cwd).
_fork_exec() now returns "noexec:chdir" for failed chdir(cwd).
Co-authored-by: Robert O'Shea <PurityLake@users.noreply.github.com>
This comment appears to have been mistakenly copied from what is now
called iobase_check_closed() in commit 4d9aec0220.
Also unite the iobase_check_closed() code with the relevant comment.
Co-authored-by: Serhiy Storchaka <storchaka@gmail.com>
If *trackfd* is False, the file descriptor specified by *fileno*
will not be duplicated.
Co-authored-by: Erlend E. Aasland <erlend@python.org>
Co-authored-by: Petr Viktorin <encukou@gmail.com>
Co-authored-by: Serhiy Storchaka <storchaka@gmail.com>
On Windows, set _O_NOINHERIT flag on file descriptors
created by os.pipe() and io.WindowsConsoleIO.
Add test_pipe_spawnl() to test_os.
Co-authored-by: Zackery Spytz <zspytz@gmail.com>
Previously the C implementation of pickle.Pickler and pickle.Unpickler
classes did not have such methods and they could only be used if
they were overloaded in subclasses or set as instance attributes.
Fixed calling super().persistent_id() and super().persistent_load() in
subclasses of the C implementation of pickle.Pickler and pickle.Unpickler
classes. It no longer causes an infinite recursion.
When an `StopIteration` raises into `asyncio.Future`, this will cause
a thread to hang. This commit address this by not raising an exception
and silently transforming the `StopIteration` with a `RuntimeError`,
which the caller can reconstruct from `fut.exception().__cause__`
Remove LibreSSL specific workaround ifdefs from `_ssl.c` and delete the non-version-specific `_ssl_data.h` file (relevant for OpenSSL < 1.1.1, which we no longer support per PEP 644).
Co-authored-by: Christian Heimes <christian@python.org>
Co-authored-by: Gregory P. Smith <greg@krypto.org>
gh-113750: Fix object resurrection on free-threaded builds
This avoids the undesired re-initializing of fields like `ob_gc_bits`,
`ob_mutex`, and `ob_tid` when an object is resurrected due to its
finalizer being called.
This change has no effect on the default (with GIL) build.
This splits part of Modules/gcmodule.c of into Python/gc.c, which
now contains the core garbage collection implementation. The Python
module remain in the Modules/gcmodule.c file.
* gh-113536: Expose `os.waitid` on macOS
This API has been available on macOS for a long time, but was
explicitly excluded due to unspecified problems with the API
in ancient versions of macOS.
* Document that the API is available on macOS starting in Python 3.13
The length field of StgDictObject for Structure class contains now
the total number of items in ffi_type_pointer.elements (excluding
the trailing null).
The old behavior of using the number of elements in the parent class can
cause the array to be truncated when it is copied, especially when there
are multiple layers of subclassing.
Co-authored-by: Serhiy Storchaka <storchaka@gmail.com>
First fix resolve situation when pyexpat module (which contains expat_CAPI
capsule) deallocates before _elementtree, so we need to hold a strong
reference to pyexpat module to.
Second fix resolve situation when module state is deallocated before
deallocation of XMLParser instances, which uses module state to clear
some stuff.
Add support for `os.POSIX_SPAWN_CLOSEFROM` and
`posix_spawn_file_actions_addclosefrom_np` and have the `subprocess` module use
them when available. This means `posix_spawn` can now be used in the default
`close_fds=True` situation on many platforms.
Co-authored-by: Gregory P. Smith [Google LLC] <greg@krypto.org>
* Allow posix_spawn to inherit environment form parent environ variable.
With this change, posix_spawn call can behave similarly to execv with regards to environments when used in subprocess functions.
The `PyThreadState_Clear()` function must only be called with the GIL
held and must be called from the same interpreter as the passed in
thread state. Otherwise, any Python objects on the thread state may be
destroyed using the wrong interpreter, leading to memory corruption.
This is also important for `Py_GIL_DISABLED` builds because free lists
will be associated with PyThreadStates and cleared in
`PyThreadState_Clear()`.
This fixes two places that called `PyThreadState_Clear()` from the wrong
interpreter and adds an assertion to `PyThreadState_Clear()`.
* bpo-43120: Add a number of LOG_* constants to syslog
This adds a number of syslog facilities to the syslogmodule.c.
These values are available on macOS.
* Switch contant documentation to the data directive
This fixes a CI warning and matches the pattern
used in the documentation for ``os``.
* Update Doc/library/syslog.rst
Co-authored-by: Hugo van Kemenade <hugovk@users.noreply.github.com>
Co-authored-by: Alex Waygood <Alex.Waygood@Gmail.com>
* gh-112529: Use atomic operations for `gcstate->collecting`
The `collecting` field in `GCState` is used to prevent overlapping garbage
collections within the same interpreter. This is updated to use atomic
operations in order to be thread-safe in `--disable-gil` builds.
The GC code is refactored a bit to support this. More of the logic is pushed
down to `gc_collect_main()` so that we can safely order the logic setting
`collecting`, the selection of the generation, and the invocation of callbacks
with respect to the atomic operations and the (future) stop-the-world pauses.
The change uses atomic operations for both `--disable-gil` and the default
build (with the GIL) to avoid extra `#ifdef` guards and ease the maintenance
burden.
* gh-51944: Add some macOS constants to termios
This changeset adds all public constants in <termio.h>
and <sys/termios.h> on macOS that weren't present
already.
Based on the macOS 14.2 SDK
Co-authored-by: Serhiy Storchaka <storchaka@gmail.com>
* Implement _Py_HashPointerRaw() as a static inline function.
* Add Py_HashPointer() tests to test_capi.test_hash.
* Keep _Py_HashPointer() function as an alias to Py_HashPointer().
Set MAX_STRUCT_SIZE to 32 in stgdict.c when on Arm platforms.
This because on Arm platforms structs with at most 4 elements of any
floating point type values can be passed through registers. If the type
is double the maximum size of the struct is 32 bytes.
On x86-64 Linux, it's maximum 16 bytes hence we need to differentiate.
Restore `subprocess`'s intended use of `vfork()` by default for performance on Linux;
also fixes the behavior of `extra_groups=[]` which was unintentionally broken in 3.12.0:
Fixed a performance regression in 3.12's :mod:`subprocess` on Linux where it
would no longer use the fast-path ``vfork()`` system call when it could have
due to a logic bug, instead falling back to the safe but slower ``fork()``.
Also fixed a security bug introduced in 3.12.0. If a value of ``extra_groups=[]``
was passed to :mod:`subprocess.Popen` or related APIs, the underlying
``setgroups(0, NULL)`` system call to clear the groups list would not be made
in the child process prior to ``exec()``.
The security issue was identified via code inspection in the process of
fixing the first bug. Thanks to @vain for the detailed report and
analysis in the initial bug on Github.
Co-authored-by: Serhiy Storchaka <storchaka@gmail.com>
Work around a macOS bug, limit zlib crc32 calls to 1GiB.
Without this, `zlib.crc32` and `binascii.crc32` could produce incorrect
results on multi-gigabyte inputs depending on the macOS version's Apple
supplied zlib implementation.
Use a fraction internally in the _PyTime API to reduce the risk of
integer overflow: simplify the fraction using Greatest Common
Divisor (GCD). The fraction API is used by time functions:
perf_counter(), monotonic() and process_time().
For example, QueryPerformanceFrequency() usually returns 10 MHz on
Windows 10 and newer. The fraction SEC_TO_NS / frequency =
1_000_000_000 / 10_000_000 can be simplified to 100 / 1.
* Add _PyTimeFraction type.
* Add functions:
* _PyTimeFraction_Set()
* _PyTimeFraction_Mul()
* _PyTimeFraction_Resolution()
* No longer check "numer * denom <= _PyTime_MAX" in
_PyTimeFraction_Set(). _PyTimeFraction_Mul() uses _PyTime_Mul()
which handles integer overflow.
* Move _PyRuntimeState.time to _posixstate.ticks_per_second and
time_module_state.ticks_per_second.
* Add time_module_state.clocks_per_second.
* Rename _PyTime_GetClockWithInfo() to py_clock().
* Rename _PyTime_GetProcessTimeWithInfo() to py_process_time().
* Add process_time_times() helper function, called by
py_process_time().
* os.times() is now always built: no longer rely on HAVE_TIMES.
If OpenSSL was built without PSK support, the python TLS-PSK
methods will raise "NotImplementedError" if called.
Add a constant "ssl.HAS_PSK" to check if TLS-PSK is supported
Prevents a segmentation fault in registered hooks for the readline library, but only when the readline module is loaded inside an isolated sub interpreter. The module is single-phase init so loading it fails, but not until the module init function has already run, where the readline hooks get registered.
The readlinestate_global macro was error-prone to PyImport_FindModule returning NULL and crashing in about 18 places. I could reproduce 1 easily, but this PR replaces the macro with a function and adds error conditions to the other functions.
Add support for TLS-PSK (pre-shared key) to the ssl module.
---------
Co-authored-by: Oleg Iarygin <oleg@arhadthedev.net>
Co-authored-by: Gregory P. Smith <greg@krypto.org>
Revert commit c8c0afc713 (PR #94532),
which moved `struct.Struct` initialisation from `Struct.__init__` to `Struct.__new__`.
This caused issues with code in the wild that subclasses `struct.Struct`.
* Run again test_ast_recursion_limit() on WASI platform.
* Add _testinternalcapi.get_c_recursion_remaining().
* Fix test_ast and test_sys_settrace: test_ast_recursion_limit() and
test_trace_unpack_long_sequence() now adjust the maximum recursion
depth depending on the the remaining C recursion.
Mark Shannon
Dino Viehland
Sam Gross
Mariusz Felisiak
Serhiy Storchaka
Erlend E. Aasland
Nikita Sobolev
Jason Zhang
Diego Russo
Eugene Toder
Soumendra Ganguly
Petr Viktorin
mpage
Steve Dower
Michael Droettboom
neonene
Skip Montanaro
AN Long
Radislav Chugunov
Jonathon Reinhart
Zackery Spytz
Ronald Oussoren
Peter Lazorchak
Victor Stinner
Jamie Phan
Pablo Galindo Salgado
Rami
Christopher Chavez
Neil Schemenauer
Jeffrey Kintscher
Kirill Podoprigora
David Benjamin
Donghee Na
Gregory P. Smith
Itamar Oren
Jakub Kulík
Carson Radtke
Raymond Hettinger
Eric Snow
Brad Larsen
Tian Gao
Grant Ramsay
Sergey B Kirpichev
Anthony Shaw
kale-smoothie
Mark Dickinson
Mayuresh Kedari
Ned Deily
Hugo van Kemenade
Wanderxjtu
Tomas R