On macOS the statvfs interface returns block counts as
32-bit integers, and that results in bad reporting for
larger disks.
Therefore reimplement statvfs in terms of statfs, which
does use 64-bit integers for block counts.
Tested using a sparse filesystem image of 100TB.
Biased reference counting maintains two refcount fields in each object:
`ob_ref_local` and `ob_ref_shared`. The true refcount is the sum of these two
fields. In some cases, when refcounting operations are split across threads,
the ob_ref_shared field can be negative (although the total refcount must be
at least zero). In this case, the thread that decremented the refcount
requests that the owning thread give up ownership and merge the refcount
fields.
This changes a number of internal usages of `PyDict_SetDefault` to use `PyDict_SetDefaultRef`.
Co-authored-by: Erlend E. Aasland <erlend.aasland@protonmail.com>
Starts adding thread safety to dict objects.
Use @critical_section for APIs which are exposed via argument clinic and don't directly correlate with a public C API which needs to acquire the lock
Use a _lock_held suffix for keeping changes to complicated functions simple and just wrapping them with a critical section
Acquire and release the lock in an existing function where it won't be overly disruptive to the existing logic
This marks dead ThreadHandles as non-joinable earlier in
`PyOS_AfterFork_Child()` before we execute any Python code. The handles
are stored in a global linked list in `_PyRuntimeState` because `fork()`
affects the entire process.
We do not want to add locking in `tp_traverse` slot implementations.
Instead, stop the world when calling `gc.get_referents`. Note that the the
stop the world call is a no-op in the default build.
Co-authored-by: Pablo Galindo Salgado <Pablogsal@gmail.com>
The `PyDict_SetDefaultRef` function is similar to `PyDict_SetDefault`,
but returns a strong reference through the optional `**result` pointer
instead of a borrowed reference.
Co-authored-by: Petr Viktorin <encukou@gmail.com>
Add optional 'filter' parameter to iterdump() that allows a "LIKE"
pattern for filtering database objects to dump.
Co-authored-by: Erlend E. Aasland <erlend@python.org>
The new `PyList_GetItemRef` is similar to `PyList_GetItem`, but returns
a strong reference instead of a borrowed reference. Additionally, if the
passed "list" object is not a list, the function sets a `TypeError`
instead of calling `PyErr_BadInternalCall()`.
* gh-112529: Remove PyGC_Head from object pre-header in free-threaded build
This avoids allocating space for PyGC_Head in the free-threaded build.
The GC implementation for free-threaded CPython does not use the
PyGC_Head structure.
* The trashcan mechanism uses the `ob_tid` field instead of `_gc_prev`
in the free-threaded build.
* The GDB libpython.py file now determines the offset of the managed
dict field based on whether the running process is a free-threaded
build. Those are identified by the `ob_ref_local` field in PyObject.
* Fixes `_PySys_GetSizeOf()` which incorrectly incorrectly included the
size of `PyGC_Head` in the size of static `PyTypeObject`.
PyObject_GetBuffer() now raises a SystemError if called with
PyBUF_READ or PyBUF_WRITE as flags. These flags should
only be used with the PyMemoryView_* C API.
Signed-off-by: Soumendra Ganguly <soumendraganguly@gmail.com>
Co-authored-by: Gregory P. Smith <greg@krypto.org>
Co-authored-by: Petr Viktorin <encukou@gmail.com>
`threading.Lock` is now the underlying class and is constructable rather than the old
factory function. This allows for type annotations to refer to it which had no non-ugly
way to be expressed prior to this.
---------
Co-authored-by: Alex Waygood <Alex.Waygood@Gmail.com>
Co-authored-by: Gregory P. Smith <greg@krypto.org>
* use the ParkingLot API to manage waiting threads
* use Argument Clinic's critical section directive to protect queue methods
* remove unnecessary overflow check
Co-authored-by: Erlend E. Aasland <erlend.aasland@protonmail.com>
Use a ring buffer instead of a Python list in order to simplify the
process of making queue.SimpleQueue thread-safe in free-threaded
builds. The ring buffer implementation has no places where critical
sections may be released.
Only set filename to cwd if it was caused by failed chdir(cwd).
_fork_exec() now returns "noexec:chdir" for failed chdir(cwd).
Co-authored-by: Robert O'Shea <PurityLake@users.noreply.github.com>
This comment appears to have been mistakenly copied from what is now
called iobase_check_closed() in commit 4d9aec0220.
Also unite the iobase_check_closed() code with the relevant comment.
Co-authored-by: Serhiy Storchaka <storchaka@gmail.com>
If *trackfd* is False, the file descriptor specified by *fileno*
will not be duplicated.
Co-authored-by: Erlend E. Aasland <erlend@python.org>
Co-authored-by: Petr Viktorin <encukou@gmail.com>
Co-authored-by: Serhiy Storchaka <storchaka@gmail.com>
On Windows, set _O_NOINHERIT flag on file descriptors
created by os.pipe() and io.WindowsConsoleIO.
Add test_pipe_spawnl() to test_os.
Co-authored-by: Zackery Spytz <zspytz@gmail.com>
Previously the C implementation of pickle.Pickler and pickle.Unpickler
classes did not have such methods and they could only be used if
they were overloaded in subclasses or set as instance attributes.
Fixed calling super().persistent_id() and super().persistent_load() in
subclasses of the C implementation of pickle.Pickler and pickle.Unpickler
classes. It no longer causes an infinite recursion.
When an `StopIteration` raises into `asyncio.Future`, this will cause
a thread to hang. This commit address this by not raising an exception
and silently transforming the `StopIteration` with a `RuntimeError`,
which the caller can reconstruct from `fut.exception().__cause__`
Remove LibreSSL specific workaround ifdefs from `_ssl.c` and delete the non-version-specific `_ssl_data.h` file (relevant for OpenSSL < 1.1.1, which we no longer support per PEP 644).
Co-authored-by: Christian Heimes <christian@python.org>
Co-authored-by: Gregory P. Smith <greg@krypto.org>
gh-113750: Fix object resurrection on free-threaded builds
This avoids the undesired re-initializing of fields like `ob_gc_bits`,
`ob_mutex`, and `ob_tid` when an object is resurrected due to its
finalizer being called.
This change has no effect on the default (with GIL) build.
This splits part of Modules/gcmodule.c of into Python/gc.c, which
now contains the core garbage collection implementation. The Python
module remain in the Modules/gcmodule.c file.
* gh-113536: Expose `os.waitid` on macOS
This API has been available on macOS for a long time, but was
explicitly excluded due to unspecified problems with the API
in ancient versions of macOS.
* Document that the API is available on macOS starting in Python 3.13
The length field of StgDictObject for Structure class contains now
the total number of items in ffi_type_pointer.elements (excluding
the trailing null).
The old behavior of using the number of elements in the parent class can
cause the array to be truncated when it is copied, especially when there
are multiple layers of subclassing.
Co-authored-by: Serhiy Storchaka <storchaka@gmail.com>
First fix resolve situation when pyexpat module (which contains expat_CAPI
capsule) deallocates before _elementtree, so we need to hold a strong
reference to pyexpat module to.
Second fix resolve situation when module state is deallocated before
deallocation of XMLParser instances, which uses module state to clear
some stuff.
Add support for `os.POSIX_SPAWN_CLOSEFROM` and
`posix_spawn_file_actions_addclosefrom_np` and have the `subprocess` module use
them when available. This means `posix_spawn` can now be used in the default
`close_fds=True` situation on many platforms.
Co-authored-by: Gregory P. Smith [Google LLC] <greg@krypto.org>
* Allow posix_spawn to inherit environment form parent environ variable.
With this change, posix_spawn call can behave similarly to execv with regards to environments when used in subprocess functions.
Ronald Oussoren
Sam Gross
Serhiy Storchaka
Artem Chernyshev
Alex Gaynor
Mark Shannon
Dino Viehland
Mariusz Felisiak
Erlend E. Aasland
Nikita Sobolev
Jason Zhang
Diego Russo
Eugene Toder
Soumendra Ganguly
Petr Viktorin
mpage
Steve Dower
Michael Droettboom
neonene
Skip Montanaro
AN Long
Radislav Chugunov
Jonathon Reinhart
Zackery Spytz
Peter Lazorchak
Victor Stinner
Jamie Phan
Pablo Galindo Salgado
Rami
Christopher Chavez
Neil Schemenauer
Jeffrey Kintscher
Kirill Podoprigora
David Benjamin
Donghee Na
Gregory P. Smith
Itamar Oren
Jakub Kulík
Carson Radtke
Raymond Hettinger