Commit Graph

14239 Commits

Author SHA1 Message Date
Steve Dower 687616877b
gh-111140: PyLong_From/AsNativeBytes: Take *flags* rather than just *endianness* (GH-116053) 2024-04-05 16:21:16 +02:00
Sam Gross de5ca0bf71
gh-117435: Make `SemLock` thread-safe in free-threaded build (#117436)
Use critical sections to make acquire, release, and _count thread-safe
without the GIL.
2024-04-04 14:09:38 -04:00
Guido van Rossum 060a96f1a9
gh-116968: Reimplement Tier 2 counters (#117144)
Introduce a unified 16-bit backoff counter type (``_Py_BackoffCounter``),
shared between the Tier 1 adaptive specializer and the Tier 2 optimizer. The
API used for adaptive specialization counters is changed but the behavior is
(supposed to be) identical.

The behavior of the Tier 2 counters is changed:
- There are no longer dynamic thresholds (we never varied these).
- All counters now use the same exponential backoff.
- The counter for ``JUMP_BACKWARD`` starts counting down from 16.
- The ``temperature`` in side exits starts counting down from 64.
2024-04-04 15:03:27 +00:00
Steve Dower 985917dc8d
gh-117267: Ensure DirEntry.stat().st_ctime still contains creation time during deprecation period (GH-117354) 2024-04-03 23:14:55 +01:00
Victor Stinner 2057c92125
gh-114329: Fix PyList_GetItemRef() limited C API definition (#117520) 2024-04-03 21:02:42 +00:00
Peter Lazorchak 1c43468886
gh-116168: Remove extra `_CHECK_STACK_SPACE` uops (#117242)
This merges all `_CHECK_STACK_SPACE` uops in a trace into a single `_CHECK_STACK_SPACE_OPERAND` uop that checks whether there is enough stack space for all calls included in the entire trace.
2024-04-03 17:14:18 +00:00
Eric Snow 976bcb2379
gh-76785: Raise InterpreterError, Not RuntimeError (gh-117489)
I had meant to switch everything to InterpreterError when I added it a while back.  At the time I missed a few key spots.

As part of this, I've added print-the-exception to _PyXI_InitTypes() and fixed an error case in `_PyStaticType_InitBuiltin().
2024-04-03 10:58:39 -06:00
Barney Gale 345194de8c
GH-114847: Raise FileNotFoundError when getcwd() returns '(unreachable)' (#117481)
On Linux >= 2.6.36 with glibc < 2.27, `getcwd()` can return a relative
pathname starting with '(unreachable)'. We detect this and fail with
ENOENT, matching new glibc behaviour.

Co-authored-by: Petr Viktorin <encukou@gmail.com>
2024-04-03 16:39:40 +01:00
Ezio Melotti 8987a5c809
gh-91565: Update issue tracker URL in error message. (#117450)
* Update issue tracker URL in commit message.

* Also update issue tracker URL in comment.
2024-04-03 10:43:52 +02:00
Eric Snow 857d3151c9
gh-76785: Consolidate Some Interpreter-related Testing Helpers (gh-117485)
This eliminates the duplication of functionally identical helpers in the _testinternalcapi and _xxsubinterpreters modules.
2024-04-02 23:16:50 +00:00
Eric Snow f341d6017d
gh-76785: Add PyInterpreterConfig Helpers (gh-117170)
These helpers make it easier to customize and inspect the config used to initialize interpreters.  This is especially valuable in our tests.  I found inspiration from the PyConfig API for the PyInterpreterConfig dict conversion stuff.  As part of this PR I've also added a bunch of tests.
2024-04-02 20:35:52 +00:00
Sam Gross 954d616b4c
gh-117440: Make `syslog` thread-safe in free-threaded builds (#117441)
Use critical sections to protect access to the syslog module.
2024-04-02 10:44:26 -04:00
Mark Shannon c32dc47aca
GH-115776: Embed the values array into the object, for "normal" Python objects. (GH-116115) 2024-04-02 11:59:21 +01:00
Petr Viktorin 179869af92
gh-94808: Fix refcounting in PyObject_Print tests (GH-117421) 2024-04-01 17:01:22 +02:00
neonene dd44ab994b
gh-117142: ctypes: Unify meta tp slot functions (GH-117143)
Integrates the following ctypes meta tp slot functions:
* `CDataType_traverse()` into `CType_Type_traverse()`.
* `CDataType_clear()` into `CType_Type_clear()`.
* `CDataType_dealloc()` into `CType_Type_dealloc()`.
* `CDataType_repeat()` into `CType_Type_repeat()`.
2024-04-01 15:28:14 +02:00
MonadChains 90c3c68a65
gh-94808:Improve coverage of PyObject_Print (GH-98749) 2024-04-01 12:52:25 +00:00
neonene 7e2fef8658
gh-117142: ctypes: Migrate global vars to module state (GH-117189) 2024-03-29 10:40:48 +01:00
Gregory P. Smith 8cb7d7ff86
gh-117310: Remove extra DECREF on "no ciphers" error path in `_ssl._SSLContext` constructor (#117309)
Remove extra self DECREF on ssl "no ciphers" error path.

This doesn't come up in practice because nobody links against a broken
OpenSSL library that provides nothing.
2024-03-28 11:11:58 +01:00
yevgeny hong ea9a296fce
gh-115627: Fix PySSL_SetError handling SSL_ERROR_SYSCALL (GH-115628)
Python 3.10 changed from using SSL_write() and SSL_read() to SSL_write_ex() and
SSL_read_ex(), but did not update handling of the return value.

Change error handling so that the return value is not examined.
OSError (not EOF) is now returned when retval is 0.

According to *recent* man pages of all functions for which we call
PySSL_SetError, (in OpenSSL 3.0 and 1.1.1), their return value should
be used to determine whether an error happened (i.e. if PySSL_SetError
should be called), but not what kind of error happened (so,
PySSL_SetError shouldn't need retval). To get the error,
we need to use SSL_get_error.

Co-authored-by: Serhiy Storchaka <storchaka@gmail.com>
Co-authored-by: Petr Viktorin <encukou@gmail.com>
2024-03-26 08:45:43 +01:00
Jonathan Protzenko 872e212378
gh-99108: Refresh HACL*; update modules accordingly; fix namespacing (GH-117237)
Pulls in a new update from https://github.com/hacl-star/hacl-star and fixes our C "namespacing" done by `Modules/_hacl/refresh.sh`.
2024-03-26 00:35:26 +00:00
Serhiy Storchaka 0c1a42cf9c
gh-87193: Support bytes objects with refcount > 1 in _PyBytes_Resize() (GH-117160)
Create a new bytes object and destroy the old one if it has refcount > 1.
2024-03-25 16:32:11 +01:00
Irit Katriel d610d821fd
gh-112383: teach dis how to interpret ENTER_EXECUTOR (#117171) 2024-03-23 22:32:33 +00:00
Erik Soma f11d0d8be8
gh-91227: Ignore ERROR_PORT_UNREACHABLE in proactor recvfrom() (#32011) 2024-03-23 08:39:35 -07:00
Mark Shannon e28477f214
GH-117108: Change the size of the GC increment to about 1% of the total heap size. (GH-117120) 2024-03-22 18:43:25 +00:00
Serhiy Storchaka e2e0b4b4b9
gh-113024: C API: Add PyObject_GenericHash() function (GH-113025) 2024-03-22 20:19:10 +02:00
NGRsoftlab 63d6f2623e
gh-117068: Remove useless code in bytesio.c:resize_buffer() (GH-117069)
Co-authored-by: i.khabibulin <i.khabibulin@ngrsoftlab.ru>
2024-03-22 11:25:38 +00:00
Eric Snow b3d25df8d3
gh-105716: Fix _PyInterpreterState_IsRunningMain() For Embedders (gh-117140)
When I added _PyInterpreterState_IsRunningMain() and friends last year, I tried to accommodate applications that embed Python but don't call _PyInterpreterState_SetRunningMain() (not that they're expected to).  That mostly worked fine until my recent changes in gh-117049, where the subtleties with the fallback code led to failures; the change ended up breaking test_tools.test_freeze, which exercises a basic embedding situation.

The simplest fix is to drop the fallback code I originally added to _PyInterpreterState_IsRunningMain() (and later to _PyThreadState_IsRunningMain()).  I've kept the fallback in the _xxsubinterpreters module though.  I've also updated Py_FrozenMain() to call _PyInterpreterState_SetRunningMain().
2024-03-21 18:20:20 -06:00
Sam Gross 1f72fb5447
gh-116522: Refactor `_PyThreadState_DeleteExcept` (#117131)
Split `_PyThreadState_DeleteExcept` into two functions:

- `_PyThreadState_RemoveExcept` removes all thread states other than one
  passed as an argument. It returns the removed thread states as a
  linked list.

- `_PyThreadState_DeleteList` deletes those dead thread states. It may
  call destructors, so we want to "start the world" before calling
  `_PyThreadState_DeleteList` to avoid potential deadlocks.
2024-03-21 11:21:02 -07:00
Eric Snow 617158e078
gh-76785: Drop PyInterpreterID_Type (gh-117101)
I added it quite a while ago as a strategy for managing interpreter lifetimes relative to the PEP 554 (now 734) implementation.  Relatively recently I refactored that implementation to no longer rely on InterpreterID objects.  Thus now I'm removing it.
2024-03-21 17:15:02 +00:00
Victor Stinner 8bea6c411d
gh-115754: Add Py_GetConstant() function (#116883)
Add Py_GetConstant() and Py_GetConstantBorrowed() functions.

In the limited C API version 3.13, getting Py_None, Py_False,
Py_True, Py_Ellipsis and Py_NotImplemented singletons is now
implemented as function calls at the stable ABI level to hide
implementation details. Getting these constants still return borrowed
references.

Add _testlimitedcapi/object.c and test_capi/test_object.py to test
Py_GetConstant() and Py_GetConstantBorrowed() functions.
2024-03-21 16:07:00 +00:00
Eric Snow 5a76d1be8e
gh-105716: Update interp->threads.main After Fork (gh-117049)
I missed this in gh-109921.

We also update Py_Exit() to call _PyInterpreterState_SetNotRunningMain(), if necessary.
2024-03-21 10:06:35 -06:00
Eric Snow bbee57fa8c
gh-76785: Clean Up Interpreter ID Conversions (gh-117048)
Mostly we unify the two different implementations of the conversion code (from PyObject * to int64_t.  We also drop the PyArg_ParseTuple()-style converter function, as well as rename and move PyInterpreterID_LookUp().
2024-03-21 09:56:12 -06:00
Sam Gross e728303532
gh-116522: Stop the world before fork() and during shutdown (#116607)
This changes the free-threaded build to perform a stop-the-world pause
before deleting other thread states when forking and during shutdown.
This fixes some crashes when using multiprocessing and during shutdown
when running with `PYTHON_GIL=0`.

This also changes `PyOS_BeforeFork` to acquire the runtime lock
(i.e., `HEAD_LOCK(&_PyRuntime)`) before forking to ensure that data
protected by the runtime lock (and not just the GIL or stop-the-world)
is in a consistent state before forking.
2024-03-21 10:01:16 -04:00
Petr Viktorin dcaf33a41d
gh-114314: ctypes: remove stgdict and switch to heap types (GH-116458)
Before this change, ctypes classes used a custom dict subclass, `StgDict`,
as their `tp_dict`. This acts like a regular dict but also includes extra information
about the type.

This replaces stgdict by `StgInfo`, a C struct on the type, accessed by
`PyObject_GetTypeData()` (PEP-697).
All usage of `StgDict` (mainly variables named `stgdict`, `dict`, `edict` etc.) is
converted to `StgInfo` (named `stginfo`, `info`, `einfo`, etc.).
Where the dict is actually used for class attributes (as a regular PyDict), it's now
called `attrdict`.

This change -- not overriding `tp_dict` -- is made to make me comfortable with
the next part of this PR: moving the initialization logic from `tp_new` to `tp_init`.

The `StgInfo` is set up in `__init__` of each class, with a guard that prevents
calling `__init__` more than once. Note that abstract classes (like `Array` or
`Structure`) are created using `PyType_FromMetaclass` and do not have
`__init__` called.
Previously, this was done in `__new__`, which also wasn't called for abstract
classes.
Since `__init__` can be called from Python code or skipped, there is a tested
guard to ensure `StgInfo` is initialized exactly once before it's used.

Co-authored-by: neonene <53406459+neonene@users.noreply.github.com>
Co-authored-by: Erlend E. Aasland <erlend.aasland@protonmail.com>
2024-03-20 17:33:08 +01:00
jkriegshauser fc45998007
gh-116773: Ensure overlapped objects on Windows are not deallocated too early by asyncio (GH-116774) 2024-03-20 14:33:28 +00:00
Serhiy Storchaka 519b2ae22b
gh-117021: Fix integer overflow in PyLong_AsPid() on non-Windows 64-bit platforms (GH-117064) 2024-03-20 15:39:53 +02:00
Nikita Sobolev 8182319de3
gh-94808: add tests covering `PyFunction_{Get,Set}Closure` (GH-99429) 2024-03-20 11:43:20 +01:00
Mark Shannon 15309329b6
GH-108362: Incremental Cycle GC (GH-116206) 2024-03-20 08:54:42 +00:00
Sam Gross 60e105c1c1
gh-113964: Don't prevent new threads until all non-daemon threads exit (#116677)
Starting in Python 3.12, we prevented calling fork() and starting new threads
during interpreter finalization (shutdown). This has led to a number of
regressions and flaky tests. We should not prevent starting new threads
(or `fork()`) until all non-daemon threads exit and finalization starts in
earnest.

This changes the checks to use `_PyInterpreterState_GetFinalizing(interp)`,
which is set immediately before terminating non-daemon threads.
2024-03-19 14:40:20 -04:00
Victor Stinner 61c659e2dc
gh-116417: Move limited C API complex.c tests to _testlimitedcapi (#117014)
Split complex.c tests of _testcapi into two parts: limited C API
tests in _testlimitedcapi and non-limited C API tests in _testcapi.
2024-03-19 17:23:12 +01:00
Victor Stinner f55e1880c1
gh-116417: Move limited C API dict.c tests to _testlimitedcapi (#117006)
Split dict.c tests of _testcapi into two parts: limited C API tests
in _testlimitedcapi and non-limited C API tests in _testcapi.
2024-03-19 15:06:20 +00:00
Victor Stinner 3cac2af5ec
gh-116417: Move limited C API long.c tests to _testlimitedcapi (#117001)
* Split long.c tests of _testcapi into two parts: limited C API tests
  in _testlimitedcapi and non-limited C API tests in _testcapi.
* Move testcapi_long.h from Modules/_testcapi/ to
  Modules/_testlimitedcapi/.
* Add MODULE__TESTLIMITEDCAPI_DEPS to Makefile.pre.in.
2024-03-19 14:04:23 +00:00
Victor Stinner a557478987
gh-116417: Move limited C API unicode.c tests to _testlimitedcapi (#116993)
Split unicode.c tests of _testcapi into two parts: limited C API
tests in _testlimitedcapi and non-limited C API tests in _testcapi.

Update test_codecs.
2024-03-19 12:30:39 +00:00
Victor Stinner 039d20ae54
gh-116417: Move limited C API abstract.c tests to _testlimitedcapi (#116986)
Split abstract.c and float.c tests of _testcapi into two parts:
limited C API tests in _testlimitedcapi and non-limited C API tests
in _testcapi.

Update test_bytes and test_class.
2024-03-19 10:44:13 +00:00
Victor Stinner dc2d0f4654
gh-116417: Fix WASI build of _testlimitedcapi (#116974)
Use different function names between _testcapi and _testlimitedcapi
to not confuse the WASI linker.
2024-03-18 23:06:52 +01:00
Victor Stinner ecb4a2b711
gh-116417: Move limited C API list.c tests to _testlimitedcapi (#116602)
Split list.c and set.c tests of _testcapi into two parts: limited C
API tests in _testlimitedcapi and non-limited C API tests in
_testcapi.
2024-03-18 22:03:55 +01:00
Zachary Ware 849e0716d3
gh-115119: Switch Windows build to mpdecimal external (GH-115182)
This includes adding what should be a relatively temporary
`Modules/_decimal/windows/mpdecimal.h` shim to choose between `mpdecimal32vc.h`
or `mpdecimal64vc.h` based on which of `CONFIG_64` or `CONFIG_32` is defined.
2024-03-18 12:07:25 -05:00
Erlend E. Aasland e2fcaf19d3
gh-115874: Don't use module state in teedataobject tp_dealloc (#116204)
Co-authored-by: Brandt Bucher <brandtbucher@microsoft.com>
2024-03-18 13:24:24 +01:00
AN Long cd2ed91780
gh-115538: Emit warning when use bool as fd in _io.WindowsConsoleIO (GH-116925) 2024-03-18 11:48:50 +00:00
mpage b3f0c1591a
gh-116915: Make `_thread._ThreadHandle` support GC (#116934)
Even though it has no internal references to Python objects it still
has a reference to its type by virtue of being a heap type. We need
to provide a traverse function that visits the type, but we do not
need to provide a clear function.
2024-03-18 09:40:16 +01:00
Victor Stinner 2982bdb936
gh-85283: Build _statistics extension with the limited C API (#116927)
Argument Clinic now inlines _PyArg_CheckPositional() for the limited
C API. The generated code should be as fast or even a little bit
faster.
2024-03-17 18:59:02 +01:00
Victor Stinner 1cf0301086
gh-85283: Build termios extension with the limited C API (#116928) 2024-03-17 15:12:29 +00:00
Victor Stinner 8e3c953b3a
gh-73468: Add math.fma() function (#116667)
Added new math.fma() function, wrapping C99's ``fma()`` operation:
fused multiply-add function.

Co-authored-by: Mark Dickinson <mdickinson@enthought.com>
2024-03-17 13:58:26 +00:00
Antoine Pitrou b8d808ddd7
GH-112536: Add more TSan tests (#116911)
These may all exercise some non-trivial aspects of thread synchronization.
2024-03-17 09:47:14 +01:00
John Sloboda 649857a157
gh-85287: Change codecs to raise precise UnicodeEncodeError and UnicodeDecodeError (#113674)
Co-authored-by: Inada Naoki <songofacandy@gmail.com>
2024-03-17 04:58:42 +00:00
mpage 33da0e844c
gh-114271: Fix race in `Thread.join()` (#114839)
There is a race between when `Thread._tstate_lock` is released[^1] in `Thread._wait_for_tstate_lock()`
and when `Thread._stop()` asserts[^2] that it is unlocked. Consider the following execution
involving threads A, B, and C:

1. A starts.
2. B joins A, blocking on its `_tstate_lock`.
3. C joins A, blocking on its `_tstate_lock`.
4. A finishes and releases its `_tstate_lock`.
5. B acquires A's `_tstate_lock` in `_wait_for_tstate_lock()`, releases it, but is swapped
   out before calling `_stop()`.
6. C is scheduled, acquires A's `_tstate_lock` in `_wait_for_tstate_lock()` but is swapped
   out before releasing it.
7. B is scheduled, calls `_stop()`, which asserts that A's `_tstate_lock` is not held.
   However, C holds it, so the assertion fails.

The race can be reproduced[^3] by inserting sleeps at the appropriate points in
the threading code. To do so, run the `repro_join_race.py` from the linked repo.

There are two main parts to this PR:

1. `_tstate_lock` is replaced with an event that is attached to `PyThreadState`.
   The event is set by the runtime prior to the thread being cleared (in the same
   place that `_tstate_lock` was released). `Thread.join()` blocks waiting for the
   event to be set.
2. `_PyInterpreterState_WaitForThreads()` provides the ability to wait for all
   non-daemon threads to exit. To do so, an `is_daemon` predicate was added to
   `PyThreadState`. This field is set each time a thread is created. `threading._shutdown()`
   now calls into `_PyInterpreterState_WaitForThreads()` instead of waiting on
   `_tstate_lock`s.

[^1]: 441affc9e7/Lib/threading.py (L1201)
[^2]: 441affc9e7/Lib/threading.py (L1115)
[^3]: 8194653279

---------

Co-authored-by: blurb-it[bot] <43283697+blurb-it[bot]@users.noreply.github.com>
Co-authored-by: Antoine Pitrou <antoine@python.org>
2024-03-16 13:56:30 +01:00
Victor Stinner 8fc8fbb43a
gh-85283: Build pwd extension with the limited C API (#116841)
Argument Clinic now uses the PEP 737 "%T" format to format type name
for the limited C API.
2024-03-15 08:49:58 +01:00
vxiiduu be1c808fca
gh-116195: Implements a fast path for nt.getppid (GH-116205)
Use the NtQueryInformationProcess system call to efficiently retrieve the parent process ID in a single step, rather than using the process snapshots API which retrieves large amounts of unnecessary information and is more prone to failure (since it makes heap allocations).

Includes a fallback to the original win32_getppid implementation in case the unstable API appears to return strange results.
2024-03-14 23:09:36 +00:00
Victor Stinner c432df6d56
gh-111696, PEP 737: Add PyType_GetModuleName() function (#116824)
Co-authored-by: Eric Snow <ericsnowcurrently@gmail.com>
2024-03-14 18:17:43 +00:00
Victor Stinner 19c3a2ff91
gh-111696, PEP 737: Add PyType_GetFullyQualifiedName() function (#116815)
Rewrite tests on type names in Python, they were written in C.
2024-03-14 16:19:36 +00:00
Victor Stinner a76288ad9b
gh-116646, AC: Always use PyObject_AsFileDescriptor() in fildes (#116806)
The fildes converter of Argument Clinic now always call
PyObject_AsFileDescriptor(), not only for the limited C API.

The _PyLong_FileDescriptor_Converter() converter stays as a fallback
when PyObject_AsFileDescriptor() cannot be used.
2024-03-14 14:58:07 +01:00
Victor Stinner 2a54c4b25e
gh-116646, AC: Add CConverter.use_converter() method (#116793)
Only add includes when the converter is effectively used.
2024-03-14 13:57:02 +01:00
Victor Stinner 97b80af897
gh-85283: Build fcntl extension with the limited C API (#116791) 2024-03-14 12:01:13 +00:00
Victor Stinner d4028724f2
gh-116646: Add limited C API support to AC fildes converter (#116769)
Add tests on the "fildes" converter to _testclinic_limited.
2024-03-14 10:28:58 +01:00
Erlend E. Aasland 3b7fe117fa
gh-116616: Use relaxed atomic ops to access socket module defaulttimeout (#116623)
Co-authored-by: Sam Gross <colesbury@gmail.com>
2024-03-12 14:44:39 +01:00
Nikita Sobolev f8147d01da
gh-116541: Handle errors correctly in `_pystatvfs_fromstructstatvfs` (#116542) 2024-03-12 13:10:00 +03:00
Victor Stinner 3cc5ae5c2c
gh-85283: Convert grp extension to the limited C API (#116611)
posixmodule.h: remove check on the limited C API, since these helpers
are not part of the public C API.
2024-03-12 00:46:53 +00:00
Victor Stinner 2b67fc57f6
gh-108494: Fix Argument Clinic LIMITED_CAPI_REGEX (#116610)
Accept spaces in "#  define Py_LIMITED_API 0x030d0000".
2024-03-11 22:42:18 +00:00
Victor Stinner 113053a070
gh-110850: Fix _PyTime_FromSecondsDouble() API (#116606)
Return 0 on success. Set an exception and return -1 on error.

Fix os.timerfd_settime(): properly report exceptions on
_PyTime_FromSecondsDouble() failure.

No longer export _PyTime_FromSecondsDouble().
2024-03-11 16:35:29 +00:00
Victor Stinner 546eb7a3be
gh-116417: Build _testinternalcapi with limited C API version 3.5 (#116598) 2024-03-11 15:20:04 +01:00
Nikita Sobolev ffd79bea0f
gh-116545: Fix error handling in `mkpwent` in `pwdmodule` (#116548) 2024-03-11 13:58:24 +03:00
Victor Stinner 1cc02ca063
gh-116417: Move 4 limited C API test files to _testlimitedcapi (#116571)
Move the following files from Modules/_testcapi/ to
Modules/_testlimitedcapi/:

* bytearray.c
* bytes.c
* pyos.c
* sys.c

Changes:

* Replace PyBytes_AS_STRING() with PyBytes_AsString().
* Replace PyBytes_GET_SIZE() with PyBytes_Size().
* Update related test_capi tests.
* Copy Modules/_testcapi/util.h to Modules/_testlimitedcapi/util.h.
2024-03-11 10:28:16 +00:00
Victor Stinner c5fa796619
gh-116417: Fix make check-c-globals for _testlimitedcapi (#116570)
* Remove unused '_testcapimodule' global in Modules/_testcapi/unicode.c.
* Update c-analyzer to not use the internal C API in
  _testlimitedcapi.c.
2024-03-10 20:19:47 +00:00
Victor Stinner 729bfb3105
gh-116417: Avoid PyFloat_AS_DOUBLE() in AC limited C API (#116568)
Argument Clinic no longer calls PyFloat_AS_DOUBLE() when the usage of
the limited C API is requested.
2024-03-10 20:42:40 +01:00
Nikita Sobolev b4b4e764a7
gh-116520: Fix error handling in `os_get_terminal_size_impl` in `posixmodule` (#116521) 2024-03-09 14:18:38 +03:00
Nikita Sobolev fdb2d90a27
gh-116447: Fix possible UB in `arraymodule` and `getargs` (#116459) 2024-03-08 13:49:52 +03:00
Erlend E. Aasland d864b0094f
gh-116303: Explicitly check for the _testsinglephase module in configure.ac (#116479) 2024-03-07 23:42:43 +00:00
Victor Stinner d9bcdda39c
gh-116417: Add _testlimitedcapi C extension (#116419)
Add a new C extension "_testlimitedcapi" which is only built with the
limited C API.

Move heaptype_relative.c and vectorcall_limited.c from
Modules/_testcapi/ to Modules/_testlimitedcapi/.

* configure: add _testlimitedcapi test extension.
* Update generate_stdlib_module_names.py.
* Update make check-c-globals.

Co-authored-by: Erlend E. Aasland <erlend.aasland@protonmail.com>
2024-03-07 18:31:12 +00:00
Serhiy Storchaka 72d3cc94cd
gh-116437: Use new C API PyDict_Pop() to simplify the code (GH-116438) 2024-03-07 11:21:08 +02:00
Nikita Sobolev 882fcede83
gh-116448: Handle errors correctly in `os_waitid_impl` in `posixmodule` (#116449) 2024-03-07 08:28:48 +00:00
Nikita Sobolev 40b79efae7
gh-116386: Fix format string "%ld" warning in `_xxinterpqueuesmodule` (#116387) 2024-03-07 09:44:26 +03:00
Russell Keith-Magee b33980a2e3
gh-114099 - Add iOS testbed, plus Makefile target to invoke it. (gh-115930) 2024-03-06 23:24:52 -05:00
mpage c62144a02c
gh-114271: Make `_thread.lock` thread-safe in free-threaded builds (#116433)
Previously, the `locked` field was set after releasing the lock. This reverses
the order so that the `locked` field is set while the lock is still held.

There is still one thread-safety issue where `locked` is checked prior to
releasing the lock, however, in practice that will only be an issue when
unlocking the lock is contended, which should be rare.
2024-03-06 15:46:36 -05:00
Nikita Sobolev 22ccf13b33
gh-116404: Handle errors correctly in `wait_helper` in `posixmodule` (#116405) 2024-03-06 08:46:47 +00:00
Eric Snow dab85e0189
gh-76785: Use PRId64 to Fix a Compiler Warning on Windows (gh-116369)
I accidentally introduced the warning in gh-116328.
2024-03-05 18:51:04 +00:00
Eric Snow 4402b3cbcf
gh-76785: Minor Improvements to "interpreters" Module (gh-116328)
This includes adding pickle support to various classes, and small changes to improve the maintainability of the low-level _xxinterpqueues module.
2024-03-05 08:54:46 -07:00
Eric Snow eb22e2b251
gh-115490: Make the interpreter.channels and interpreter.queues Modules Handle Reloading Properly (gh-115493)
The problem manifested when the .py module got reloaded and the corresponding extension module didn't. The .py module registers types with the extension and the extension was not allowing that to happen more than once. The solution: let it happen more than once.
2024-03-04 20:59:30 +00:00
Eric Snow 01440d3a39
gh-76785: Simplify Channels XID Types (gh-116318)
I had added an extra cleanup abstraction a while back that has turned out to be unnecessary.
2024-03-04 19:32:39 +00:00
Yuriy Chernyshov 9b9e819b51
gh-116116: Backport blake2 change to fix building with clang-cl on windows-i686 (GH-116117) 2024-03-04 16:59:57 +00:00
Raymond Hettinger 15dc2979bc
Consistently spell out *predicate* instead of *pred*. (gh-116308) 2024-03-04 15:51:29 +00:00
Sergey B Kirpichev 002a5948fc
gh-108562: Fix compiler warnings for libmpdec (#114751)
If awailable, enable -fstrict-overflow for libmpdec. Also
shut off false positive warnings (-Warray-bounds).

The later was backported from mpdecimal-4.0.0.
2024-03-03 08:25:39 +01:00
Eric Snow cad3745b87
gh-116102: Silence a Compiler Warning in _xxinterpqueues (gh-116230) 2024-03-02 00:28:05 +00:00
mpage 9e88173d36
gh-114271: Make `_thread.ThreadHandle` thread-safe in free-threaded builds (GH-115190)
Make `_thread.ThreadHandle` thread-safe in free-threaded builds

We protect the mutable state of `ThreadHandle` using a `_PyOnceFlag`.
Concurrent operations (i.e. `join` or `detach`) on `ThreadHandle` block
until it is their turn to execute or an earlier operation succeeds.
Once an operation has been applied successfully all future operations
complete immediately.

The `join()` method is now idempotent. It may be called multiple times
but the underlying OS thread will only be joined once. After `join()`
succeeds, any future calls to `join()` will succeed immediately.

The internal thread handle `detach()` method has been removed.
2024-03-01 13:43:12 -08:00
Malcolm Smith e6e35327d8
gh-115773: Add missing preprocessor guard in _testexternalinspection (#116212)
Add missing preprocessor guard in _testexternalinspection
2024-03-01 17:50:48 +00:00
Brett Simmers 2e94a6687c
gh-116099: Fix refcounting bug in `_queueobj_shared()` (gh-116164)
This code decrefs `qidobj` twice in some paths. Since `qidobj` isn't used after
the first `Py_DECREF()`, remove the others, and replace the `Py_DECREF()` with
`Py_CLEAR()` to make it clear that the variable is dead.

With this fix, `python -mtest test_interpreters -R 3:3 -mtest_queues` no longer
fails with `_Py_NegativeRefcount: Assertion failed: object has negative ref
count`.
2024-03-01 01:02:12 +00:00
AN Long ca56c3a172
gh-103092: Add a mutex to make the PRNG state of rotatingtree concurrent-safe (#115301) 2024-03-01 00:04:16 +01:00
Sebastian Pipping 6a95676bb5
gh-115398: Expose Expat >=2.6.0 reparse deferral API (CVE-2023-52425) (GH-115623)
Allow controlling Expat >=2.6.0 reparse deferral (CVE-2023-52425) by adding five new methods:

- `xml.etree.ElementTree.XMLParser.flush`
- `xml.etree.ElementTree.XMLPullParser.flush`
- `xml.parsers.expat.xmlparser.GetReparseDeferralEnabled`
- `xml.parsers.expat.xmlparser.SetReparseDeferralEnabled`
- `xml.sax.expatreader.ExpatParser.flush`

Based on the "flush" idea from https://github.com/python/cpython/pull/115138#issuecomment-1932444270 .

### Notes

- Please treat as a security fix related to CVE-2023-52425.

Includes code suggested-by: Snild Dolkow <snild@sony.com>
and by core dev Serhiy Storchaka.
2024-02-29 14:52:50 -08:00
Malcolm Smith fa1d675309
gh-71052: Fix several Android build issues (#115955)
This change is part of the work on PEP-738: Adding Android as a 
supported platform.

* Remove the "1.0" suffix from libpython's filename on Android, which 
  would prevent Gradle from packaging it into an app. 
* Simplify the build command in the Makefile so that libpython always 
  gets given an SONAME with the `-Wl-h` argument, even if the SONAME is
  identical to the actual filename.
* Disable a number of functions on Android which can be compiled and 
  linked against, but always fail at runtime. As a result, the native
  _multiprocessing module is no longer built for Android.
* gh-115390 (bee7bb331) added some pre-determined results to the 
  configure script for things that can't be autodetected when
  cross-compiling; this change adds Android to these where appropriate.
* Add a couple more pre-determined results for Android, and making them 
  cover iOS as well. This means the --enable-ipv6 configure option will 
  no longer be required on either platform.
2024-02-29 22:58:20 +01:00
Eric Snow e80abd57a8
gh-76785: Update test.support.interpreters to Align With PEP 734 (gh-115566)
This brings the code under test.support.interpreters, and the corresponding extension modules, in line with recent updates to PEP 734.

(Note: PEP 734 has not been accepted at this time.  However, we are using an internal copy of the implementation in the test suite to exercise the existing subinterpreters feature.)
2024-02-28 16:08:08 -07:00
Sam Gross df5212df6c
gh-112529: Simplify PyObject_GC_IsTracked and PyObject_GC_IsFinalized (#114732) 2024-02-28 15:37:59 -05:00
Pablo Galindo Salgado 1752b51012
gh-115773: Add tests to exercise the _Py_DebugOffsets structure (#115774) 2024-02-28 10:17:34 +00:00
Mark Shannon 6ecfcfe894
GH-115816: Assorted naming and formatting changes to improve maintainability. (GH-115987)
* Rename _Py_UOpsAbstractInterpContext to _Py_UOpsContext and _Py_UOpsSymType to _Py_UopsSymbol.

* #define shortened form of _Py_uop_... names for improved readability.
2024-02-27 13:25:02 +00:00
Mark Shannon 10fbcd6c5d
GH-115816: Make tier2 optimizer symbols testable, and add a few tests. (GH-115953) 2024-02-27 10:51:26 +00:00
Michael Droettboom b05afdd5ec
gh-115168: Add pystats counter for invalidated executors (GH-115169) 2024-02-26 17:51:47 +00:00
Yuriy Chernyshov 96c10c6485
gh-115882: Reference Unknwn.h for ctypes on Windows (GH-115350)
This allows the module to be compiled with WIN32_LEAN_AND_MEAN enabled
2024-02-26 17:21:55 +00:00
Furkan Onder 8f5be78bce
gh-72249: Include the module name in the repr of partial object (GH-101910)
Co-authored-by: Anilyka Barry <vgr255@live.ca>
2024-02-25 22:55:19 +02:00
Serhiy Storchaka 79811ededd
gh-115886: Handle embedded null characters in shared memory name (GH-115887)
shm_open() and shm_unlink() now check for embedded null characters in
the name and raise an error instead of silently truncating it.
2024-02-25 11:31:03 +02:00
Serhiy Storchaka c688c0f130
gh-67044: Always quote or escape \r and \n in csv.writer() (GH-115741) 2024-02-23 22:25:09 +02:00
Petr Viktorin 8aa372edcd
gh-115714: Don't use CLOCK_PROCESS_CPUTIME_ID and times() on WASI (GH-115757)
* gh-115714: Don't use CLOCK_PROCESS_CPUTIME_ID and times() on WASI

* Add blurb
2024-02-22 12:39:45 +01:00
Malcolm Smith 7f5e3f04f8
gh-111225: Link extension modules against libpython on Android (#115780)
Part of the work on PEP 738: Adding Android as a supported platform.

* Rename the LIBPYTHON variable to MODULE_LDFLAGS, to more accurately 
  reflect its purpose.
* Edit makesetup to use MODULE_LDFLAGS when linking extension modules.
* Edit the Makefile so that extension modules depend on libpython on 
  Android and Cygwin.
* Restore `-fPIC` on Android. It was removed several years ago with a 
  note that the toolchain used it automatically, but this is no longer
  the case. Omitting it causes all linker commands to fail with an error
  like `relocation R_AARCH64_ADR_PREL_PG_HI21 cannot be used against
  symbol '_Py_FalseStruct'; recompile with -fPIC`.
2024-02-21 23:18:57 +00:00
Victor Stinner e4c34f04a1
gh-110850: Cleanup PyTime API: PyTime_t are nanoseconds (#115753)
PyTime_t no longer uses an arbitrary unit, it's always a number of
nanoseconds (64-bit signed integer).

* Rename _PyTime_FromNanosecondsObject() to _PyTime_FromLong().
* Rename _PyTime_AsNanosecondsObject() to _PyTime_AsLong().
* Remove pytime_from_nanoseconds().
* Remove pytime_as_nanoseconds().
* Remove _PyTime_FromNanoseconds().
2024-02-21 11:46:00 +01:00
Victor Stinner 77430b6a32
gh-110850: Replace private _PyTime_MAX with public PyTime_MAX (#115751)
Remove references to the old names _PyTime_MIN
and _PyTime_MAX, now that PyTime_MIN and
PyTime_MAX are public.

Replace also _PyTime_MIN with PyTime_MIN.
2024-02-21 08:11:40 +00:00
Victor Stinner 145bc2d638
gh-110850: Use public PyTime functions (#115746)
Replace private _PyTime functions with public PyTime functions.

random_seed_time_pid() now reports errors to its caller.
2024-02-20 23:31:30 +00:00
Victor Stinner 52d1477566
gh-110850: Rename internal PyTime C API functions (#115734)
Rename functions:

* _PyTime_GetSystemClock() => _PyTime_TimeUnchecked()
* _PyTime_GetPerfCounter() => _PyTime_PerfCounterUnchecked()
* _PyTime_GetMonotonicClock() => _PyTime_MonotonicUnchecked()
* _PyTime_GetSystemClockWithInfo() => _PyTime_TimeWithInfo()
* _PyTime_GetMonotonicClockWithInfo() => _PyTime_MonotonicWithInfo()
* _PyTime_GetMonotonicClockWithInfo() => _PyTime_MonotonicWithInfo()

Changes:

* Remove "typedef PyTime_t PyTime_t;" which was
  "typedef PyTime_t _PyTime_t;" before a previous rename.
* Update comments of "Unchecked" functions.
* Remove invalid PyTime_Time() comment.
2024-02-20 22:16:37 +00:00
Victor Westerhuis e1fdc3c323
gh-104061: Add socket.SO_BINDTOIFINDEX constant (GH-104062)
Add socket.SO_BINDTOIFINDEX constant

This socket option avoids a race condition between SO_BINDTODEVICE and network interface renaming.
2024-02-20 23:08:15 +02:00
Guido van Rossum 142502ea8d
Tier 2 cleanups and tweaks (#115534)
* Rename `_testinternalcapi.get_{uop,counter}_optimizer` to `new_*_optimizer`
* Use `_PyUOpName()` instead of` _PyOpcode_uop_name[]`
* Add `target` to executor iterator items -- `list(ex)` now returns `(opcode, oparg, target, operand)` quadruples
* Add executor methods `get_opcode()` and `get_oparg()` to get `vmdata.opcode`, `vmdata.oparg`
* Define a helper for printing uops, and unify various places where they are printed
* Add a hack to summarize_stats.py to fix legacy uop names (e.g. `POP_TOP` -> `_POP_TOP`)
* Define helpers in `test_opt.py` for accessing the set or list of opnames of an executor
2024-02-20 20:24:35 +00:00
Victor Stinner d207c7cd5a
gh-110850: Cleanup pycore_time.h includes (#115724)
<pycore_time.h> include is no longer needed to get the PyTime_t type
in internal header files. This type is now provided by <Python.h>
include. Add <pycore_time.h> includes to C files instead.
2024-02-20 16:50:43 +00:00
Serhiy Storchaka 937d282150
gh-115712: Support CSV dialects with delimiter=' ' and skipinitialspace=True (GH-115721)
Restore support of such combination, disabled in gh-113796.

csv.writer() now quotes empty fields if delimiter is a space and
skipinitialspace is true and raises exception if quoting is not possible.
2024-02-20 18:09:50 +02:00
Victor Stinner 9af80ec83d
gh-110850: Replace _PyTime_t with PyTime_t (#115719)
Run command:

sed -i -e 's!\<_PyTime_t\>!PyTime_t!g' $(find -name "*.c" -o -name "*.h")
2024-02-20 15:02:27 +00:00
Brett Simmers 0749244d13
gh-112175: Add `eval_breaker` to `PyThreadState` (#115194)
This change adds an `eval_breaker` field to `PyThreadState`. The primary
motivation is for performance in free-threaded builds: with thread-local eval
breakers, we can stop a specific thread (e.g., for an async exception) without
interrupting other threads.

The source of truth for the global instrumentation version is stored in the
`instrumentation_version` field in PyInterpreterState. Threads usually read the
version from their local `eval_breaker`, where it continues to be colocated
with the eval breaker bits.
2024-02-20 09:57:48 -05:00
Victor Stinner e00960a74d
gh-110850: Enhance PyTime C API tests (#115715) 2024-02-20 15:53:40 +01:00
Mark Shannon 7b21403ccd
GH-112354: Initial implementation of warm up on exits and trace-stitching (GH-114142) 2024-02-20 09:39:55 +00:00
6t8k 26800cf25a
gh-95782: Fix io.BufferedReader.tell() etc. being able to return offsets < 0 (GH-99709)
lseek() always returns 0 for character pseudo-devices like
`/dev/urandom` (for other non-regular files, e.g. `/dev/stdin`, it
always returns -1, to which CPython reacts by raising appropriate
exceptions). They are thus technically seekable despite not having seek
semantics.

When calling read() on e.g. an instance of `io.BufferedReader` that
wraps such a file, `BufferedReader` reads ahead, filling its buffer,
creating a discrepancy between the number of bytes read and the internal
`tell()` always returning 0, which previously resulted in e.g.
`BufferedReader.tell()` or `BufferedReader.seek()` being able to return
positions < 0 even though these are supposed to be always >= 0.

Invariably keep the return value non-negative by returning
max(former_return_value, 0) instead, and add some corresponding tests.
2024-02-17 11:16:06 +00:00
Sam Gross 5903190727
gh-115103: Implement delayed memory reclamation (QSBR) (#115180)
This adds a safe memory reclamation scheme based on FreeBSD's "GUS" and
quiescent state based reclamation (QSBR). The API provides a mechanism
for callers to detect when it is safe to free memory that may be
concurrently accessed by readers.
2024-02-16 15:25:19 -05:00
mpage f366e21504
gh-114271: Make `thread._rlock` thread-safe in free-threaded builds (#115102)
The ID of the owning thread (`rlock_owner`) may be accessed by
multiple threads without holding the underlying lock; relaxed
atomics are used in place of the previous loads/stores.

The number of times that the lock has been acquired (`rlock_count`)
is only ever accessed by the thread that holds the lock; we do not
need to use atomics to access it.
2024-02-16 13:29:25 -05:00
Sam Gross b24c9161a6
gh-112529: Make the GC scheduling thread-safe (#114880)
The GC keeps track of the number of allocations (less deallocations)
since the last GC. This buffers the count in thread-local state and uses
atomic operations to modify the per-interpreter count. The thread-local
buffering avoids contention on shared state.

A consequence is that the GC scheduling is not as precise, so
"test_sneaky_frame_object" is skipped because it requires that the GC be
run exactly after allocating a frame object.
2024-02-16 11:22:27 -05:00
David Benjamin bce693111b
gh-114572: Fix locking in cert_store_stats and get_ca_certs (#114573)
* gh-114572: Fix locking in cert_store_stats and get_ca_certs

cert_store_stats and get_ca_certs query the SSLContext's X509_STORE with
X509_STORE_get0_objects, but reading the result requires a lock. See
https://github.com/openssl/openssl/pull/23224 for details.

Instead, use X509_STORE_get1_objects, newly added in that PR.
X509_STORE_get1_objects does not exist in current OpenSSLs, but we can
polyfill it with X509_STORE_lock and X509_STORE_unlock.

* Work around const-correctness problem

* Add missing X509_STORE_get1_objects failure check

* Add blurb
2024-02-15 19:24:51 -05:00
Nikita Sobolev fd2bb4be3d
gh-115498: Fix `SET_COUNT` error handling in `_xxinterpchannelsmodule` (#115499) 2024-02-16 00:31:23 +03:00
Dino Viehland ae460d450a
gh-113743: Make the MRO cache thread-safe in free-threaded builds (#113930)
Makes _PyType_Lookup thread safe, including:
    Thread safety of the underlying cache.
    Make mutation of mro and type members thread safe
    Also _PyType_GetMRO and _PyType_GetBases are currently returning borrowed references which aren't safe.
2024-02-15 10:54:57 -08:00
monkeyman192 298bcdc185
gh-112433: Add optional _align_ attribute to ctypes.Structure (GH-113790) 2024-02-15 16:40:20 +02:00
Sam Gross ad4f909e0e
gh-115432: Add critical section variant that handles a NULL object (#115433)
This adds `Py_XBEGIN_CRITICAL_SECTION` and
`Py_XEND_CRITICAL_SECTION`, which accept a possibly NULL object as an
argument. If the argument is NULL, then nothing is locked or unlocked.
Otherwise, they behave like `Py_BEGIN/END_CRITICAL_SECTION`.
2024-02-15 08:37:54 -05:00
mpage dc978f6ab6
gh-112050: Make collections.deque thread-safe in free-threaded builds (#113830)
Use critical sections to make deque methods that operate on mutable 
state thread-safe when the GIL is disabled. This is mostly accomplished
by using the @critical_section Argument Clinic directive, though there
are a few places where this was not possible and critical sections had
to be manually acquired/released.
2024-02-15 09:22:47 +01:00
Victor Stinner 3e7b7df5cb
gh-114570: Add PythonFinalizationError exception (#115352)
Add PythonFinalizationError exception. This exception derived from
RuntimeError is raised when an operation is blocked during the Python
finalization.

The following functions now raise PythonFinalizationError, instead of
RuntimeError:

* _thread.start_new_thread()
* subprocess.Popen
* os.fork()
* os.fork1()
* os.forkpty()

Morever, _winapi.Overlapped finalizer now logs an unraisable
PythonFinalizationError, instead of an unraisable RuntimeError.
2024-02-14 23:35:06 +01:00
Seth Michael Larson 4b2d1786cc
gh-115399: Upgrade bundled libexpat to 2.6.0 (#115431) 2024-02-14 16:29:06 +00:00
kcatss 671360161f
gh-115243: Fix crash in deque.index() when the deque is concurrently modified (GH-115247) 2024-02-14 16:08:26 +00:00
Eric Snow 514b1c91b8
gh-76785: Improved Subinterpreters Compatibility with 3.12 (gh-115424)
For the most part, these changes make is substantially easier to backport subinterpreter-related code to 3.12, especially the related modules (e.g. _xxsubinterpreters). The main motivation is to support releasing a PyPI package with the 3.13 capabilities compiled for 3.12.

A lot of the changes here involve either hiding details behind macros/functions or splitting up some files.
2024-02-13 14:56:49 -07:00
Kirill Podoprigora 225cd55fe6
gh-115417: Remove accidentally left debugging print (#115418)
gh-115417: Remove debugging print
2024-02-13 18:45:37 +01:00
Steve Dower ea25f32d5f
gh-89240: Enable multiprocessing on Windows to use large process pools (GH-107873)
We add _winapi.BatchedWaitForMultipleObjects to wait for larger numbers of handles.
This is an internal module, hence undocumented, and should be used with caution.
Check the docstring for info before using BatchedWaitForMultipleObjects.
2024-02-13 00:28:35 +00:00
Steve Dower 7861dfd26a
gh-111140: Adds PyLong_AsNativeBytes and PyLong_FromNative[Unsigned]Bytes functions (GH-114886) 2024-02-12 20:13:13 +00:00
mpage de7d67b19b
gh-114271: Make `PyInterpreterState.threads.count` thread-safe in free-threaded builds (gh-115093)
Use atomics to mutate PyInterpreterState.threads.count.
2024-02-12 10:44:00 -07:00
Petr Viktorin 879f4546bf
gh-110850: Add PyTime_t C API (GH-115215)
* gh-110850: Add PyTime_t C API

Add PyTime_t API:

* PyTime_t type.
* PyTime_MIN and PyTime_MAX constants.
* PyTime_AsSecondsDouble(), PyTime_Monotonic(),
  PyTime_PerfCounter() and PyTime_GetSystemClock() functions.

Co-authored-by: Victor Stinner <vstinner@python.org>
2024-02-12 18:13:10 +01:00
Nikita Sobolev 91bf01d4b1
gh-87804: Fix the refleak in error handling of `_pystatvfs_fromstructstatfs` (#115335)
It was the macro expansion! Sorry!
2024-02-12 19:27:27 +03:00
Kirill Podoprigora 93ac78ac3e
gh-115058: Add ``reset_rare_event_counters`` function in `_testinternalcapi` (GH-115128) 2024-02-12 16:05:30 +00:00
Eugene Toder 46190d9ea8
gh-89039: Call subclass constructors in datetime.*.replace (GH-114780)
When replace() method is called on a subclass of datetime, date or time,
properly call derived constructor. Previously, only the base class's
constructor was called.

Also, make sure to pass non-zero fold values when creating subclasses in
various methods. Previously, fold was silently ignored.
2024-02-12 14:44:56 +02:00
John Belmonte 72340d15cd
gh-114563: C decimal falls back to pydecimal for unsupported format strings (GH-114879)
Immediate merits:
* eliminate complex workarounds for 'z' format support
  (NOTE: mpdecimal recently added 'z' support, so this becomes
  efficient in the long term.)
* fix 'z' format memory leak
* fix 'z' format applied to 'F'
* fix missing '#' format support

Suggested and prototyped by Stefan Krah.

Fixes gh-114563, gh-91060

Co-authored-by: Stefan Krah <skrah@bytereef.org>
2024-02-12 13:17:51 +02:00
Nikita Sobolev 54bde5dcc3
gh-87804: Fix error handling and style in `_pystatvfs_fromstructstatfs` (#115236) 2024-02-12 10:27:12 +03:00
Serhiy Storchaka b104360788
gh-49766: Make date-datetime comparison more symmetric and flexible (GH-114760)
Now the special comparison methods like `__eq__` and `__lt__` return
NotImplemented if one of comparands is date and other is datetime
instead of ignoring the time part and the time zone or forcefully
return "not equal" or raise TypeError.

It makes comparison of date and datetime subclasses more symmetric
and allows to change the default behavior by overriding
the special comparison methods in subclasses.

It is now the same as if date and datetime was independent classes.
2024-02-11 13:06:43 +02:00
Soumendra Ganguly bf75f1b147
gh-85984: Add _POSIX_VDISABLE from unistd.h to termios module. (#114985)
Signed-off-by: Soumendra Ganguly <soumendraganguly@gmail.com>
Co-authored-by: Gregory P. Smith <greg@krypto.org>
2024-02-11 10:29:44 +00:00
Nikita Sobolev 3a5b38e3b4
gh-114670: Fix `_testbuffer` module initialization (#114672) 2024-02-11 00:48:28 +03:00
Mike Zimin 9d1a353230
gh-114894: add array.array.clear() method (#114919)
Co-authored-by: blurb-it[bot] <43283697+blurb-it[bot]@users.noreply.github.com>
Co-authored-by: AN Long <aisk@users.noreply.github.com>
Co-authored-by: Jelle Zijlstra <jelle.zijlstra@gmail.com>
2024-02-10 07:59:46 -08:00
Ronald Oussoren 6e222a55b1
GH-87804: Fix counter overflow in statvfs on macOS (#99570)
On macOS the statvfs interface returns block counts as
32-bit integers, and that results in bad reporting for
larger disks.

Therefore reimplement statvfs in terms of statfs, which
does use 64-bit integers for block counts.

Tested using a sparse filesystem image of 100TB.
2024-02-10 11:16:45 +01:00
Sam Gross a3af3cb4f4
gh-110481: Implement inter-thread queue for biased reference counting (#114824)
Biased reference counting maintains two refcount fields in each object:
`ob_ref_local` and `ob_ref_shared`. The true refcount is the sum of these two
fields. In some cases, when refcounting operations are split across threads,
the ob_ref_shared field can be negative (although the total refcount must be
at least zero). In this case, the thread that decremented the refcount
requests that the owning thread give up ownership and merge the refcount
fields.
2024-02-09 17:08:32 -05:00
Serhiy Storchaka 846fd721d5
gh-115059: Flush the underlying write buffer in io.BufferedRandom.read1() (GH-115163) 2024-02-09 12:36:12 +02:00
Artem Chernyshev 9e90313320
gh-115136: Fix possible NULL deref in getpath_joinpath() (GH-115137)
Signed-off-by: Artem Chernyshev <artem.chernyshev@red-soft.ru>
2024-02-08 08:40:38 +00:00
Alex Gaynor 38b970dfcc
When the Py_CompileStringExFlags fuzzer encounters a SystemError, abort (#115147)
This allows us to catch bugs beyond memory corruption and assertions.
2024-02-07 17:21:33 -05:00
Sam Gross ef3ceab09d
gh-112066: Use `PyDict_SetDefaultRef` in place of `PyDict_SetDefault`. (#112211)
This changes a number of internal usages of `PyDict_SetDefault` to use `PyDict_SetDefaultRef`.

Co-authored-by: Erlend E. Aasland <erlend.aasland@protonmail.com>
2024-02-07 13:43:18 -05:00
Mark Shannon 8a3c499ffe
GH-108362: Revert "GH-108362: Incremental GC implementation (GH-108038)" (#115132)
Revert "GH-108362: Incremental GC implementation (GH-108038)"

This reverts commit 36518e69d7.
2024-02-07 12:38:34 +00:00
Dino Viehland 92abb01240
gh-112075: Add critical sections for most dict APIs (#114508)
Starts adding thread safety to dict objects.


Use @critical_section for APIs which are exposed via argument clinic and don't directly correlate with a public C API which needs to acquire the lock
Use a _lock_held suffix for keeping changes to complicated functions simple and just wrapping them with a critical section
Acquire and release the lock in an existing function where it won't be overly disruptive to the existing logic
2024-02-06 14:03:43 -08:00
Sam Gross b6228b521b
gh-115035: Mark ThreadHandles as non-joinable earlier after forking (#115042)
This marks dead ThreadHandles as non-joinable earlier in
`PyOS_AfterFork_Child()` before we execute any Python code. The handles
are stored in a global linked list in `_PyRuntimeState` because `fork()`
affects the entire process.
2024-02-06 14:45:04 -05:00
Sam Gross 7fdd4235d7
gh-112529: Stop the world around gc.get_referents (#114823)
We do not want to add locking in `tp_traverse` slot implementations.
Instead, stop the world when calling `gc.get_referents`. Note that the the
stop the world call is a no-op in the default build.

Co-authored-by: Pablo Galindo Salgado <Pablogsal@gmail.com>
2024-02-06 11:45:42 -05:00
Sam Gross de61d4bd4d
gh-112066: Add `PyDict_SetDefaultRef` function. (#112123)
The `PyDict_SetDefaultRef` function is similar to `PyDict_SetDefault`,
but returns a strong reference through the optional `**result` pointer
instead of a borrowed reference.

Co-authored-by: Petr Viktorin <encukou@gmail.com>
2024-02-06 11:36:23 -05:00
Mariusz Felisiak 1a10437a14
gh-91602: Add iterdump() support for filtering database objects (#114501)
Add optional 'filter' parameter to iterdump() that allows a "LIKE"
pattern for filtering database objects to dump.

Co-authored-by: Erlend E. Aasland <erlend@python.org>
2024-02-06 12:34:56 +01:00
Serhiy Storchaka 652fbf88c4
gh-82626: Emit a warning when bool is used as a file descriptor (GH-111275) 2024-02-05 22:51:11 +02:00
Erlend E. Aasland 09096a1647
gh-115015: Argument Clinic: fix generated code for METH_METHOD methods without params (#115016) 2024-02-05 21:49:17 +01:00
Mark Shannon 36518e69d7
GH-108362: Incremental GC implementation (GH-108038) 2024-02-05 18:28:51 +00:00
Nikita Sobolev 87cd20a567
gh-115026: Argument Clinic: handle PyBuffer_FillInfo errors in generated code (#115027) 2024-02-05 11:45:09 +01:00
Nikita Sobolev 929d44e15a
gh-114685: PyBuffer_FillInfo() now raises on PyBUF_{READ,WRITE} (GH-114802) 2024-02-04 19:16:43 +00:00
Jason Zhang efc489021c
gh-111417: Remove unused code block in math.trunc() and round() (GH-111454)
_PyObject_LookupSpecial() now ensures that the type is ready.
2024-02-03 17:11:10 +02:00
Sam Gross d0f1307580
gh-114329: Add `PyList_GetItemRef` function (GH-114504)
The new `PyList_GetItemRef` is similar to `PyList_GetItem`, but returns
a strong reference instead of a borrowed reference. Additionally, if the
passed "list" object is not a list, the function sets a `TypeError`
instead of calling `PyErr_BadInternalCall()`.
2024-02-02 14:03:15 +01:00
Mark Shannon 0e71a295e9
GH-113710: Add a "globals to constants" pass (GH-114592)
Converts specializations of `LOAD_GLOBAL` into constants during tier 2 optimization.
2024-02-02 12:14:34 +00:00
Sam Gross 587d480203
gh-112529: Remove PyGC_Head from object pre-header in free-threaded build (#114564)
* gh-112529: Remove PyGC_Head from object pre-header in free-threaded build

This avoids allocating space for PyGC_Head in the free-threaded build.
The GC implementation for free-threaded CPython does not use the
PyGC_Head structure.

 * The trashcan mechanism uses the `ob_tid` field instead of `_gc_prev`
   in the free-threaded build.
 * The GDB libpython.py file now determines the offset of the managed
   dict field based on whether the running process is a free-threaded
   build. Those are identified by the `ob_ref_local` field in PyObject.
 * Fixes `_PySys_GetSizeOf()` which incorrectly incorrectly included the
   size of `PyGC_Head` in the size of static `PyTypeObject`.
2024-02-01 12:29:19 -08:00
Serhiy Storchaka b7688ef71e
gh-114685: Check flags in PyObject_GetBuffer() (GH-114707)
PyObject_GetBuffer() now raises a SystemError if called with
PyBUF_READ or PyBUF_WRITE as flags. These flags should
only be used with the PyMemoryView_* C API.
2024-01-31 13:11:35 +02:00
Diego Russo a06b606462
gh-110190: Fix ctypes structs with array on Windows ARM64 (GH-114753) 2024-01-30 23:53:04 +00:00
Eugene Toder 1f515e8a10
gh-112919: Speed-up datetime, date and time.replace() (GH-112921)
Use argument clinic and call new_* functions directly. This speeds up
these functions 6x to 7.5x when calling with keyword arguments.
2024-01-30 15:19:46 +00:00
Serhiy Storchaka ea30a28c3e
gh-113732: Fix support of QUOTE_NOTNULL and QUOTE_STRINGS in csv.reader (GH-113738) 2024-01-30 14:21:12 +02:00
Serhiy Storchaka aa3402ad45
gh-114678: Fix incorrect deprecation warning for 'N' specifier in Decimal format (GH-114683)
Co-authored-by: Stefan Krah <skrah@bytereef.org>
2024-01-29 19:58:31 +02:00
Soumendra Ganguly e351ca3c20
gh-85984: Add POSIX pseudo-terminal functions. (GH-102413)
Signed-off-by: Soumendra Ganguly <soumendraganguly@gmail.com>
Co-authored-by: Gregory P. Smith <greg@krypto.org>
Co-authored-by: Petr Viktorin <encukou@gmail.com>
2024-01-29 16:10:28 +00:00
Petr Viktorin 15fe8cea17
gh-91325: Skip Stable ABI checks with Py_TRACE_REFS special build (GH-92046)
Skip Stable ABI checks with Py_TRACE_REFS special build

This build is not compatible with Py_LIMITED_API nor with
the stable ABI.
2024-01-29 16:45:31 +01:00
mpage c87233fd3f
gh-112050: Adapt collections.deque to Argument Clinic (#113963) 2024-01-29 15:08:23 +00:00
Steve Dower 102569d150
Use Unicode unconditionally for _winapi.CreateFile (GH-114611)
Currently it switches based on build settings, but argument clinic does not handle it correctly.
2024-01-26 17:27:29 +00:00
Erlend E. Aasland dcd28b5c35
gh-114569: Use PyMem_* APIs for most non-PyObject uses (#114574)
Fix usage in Modules, Objects, and Parser subdirectories.
2024-01-26 10:11:35 +00:00
Michael Droettboom d0f7f5c41d
gh-114312: Fix rare event counter tests on aarch64 (GH-114554) 2024-01-26 10:10:03 +00:00
Nikita Sobolev d96358ff9d
gh-114315: Make `threading.Lock` a real class, not a factory function (#114479)
`threading.Lock` is now the underlying class and is constructable rather than the old
factory function. This allows for type annotations to refer to it which had no non-ugly
way to be expressed prior to this.

---------

Co-authored-by: Alex Waygood <Alex.Waygood@Gmail.com>
Co-authored-by: Gregory P. Smith <greg@krypto.org>
2024-01-25 19:46:32 +00:00
Michael Droettboom ea3cd0498c
gh-114312: Collect stats for unlikely events (GH-114493) 2024-01-25 11:10:51 +00:00
Serhiy Storchaka d22c066b80
gh-114492: Initialize struct termios before calling tcgetattr() (GH-114495)
On Alpine Linux it could leave some field non-initialized.
2024-01-23 23:27:04 +02:00
mpage 925907ea36
gh-113884: Make queue.SimpleQueue thread-safe when the GIL is disabled (#114161)
* use the ParkingLot API to manage waiting threads
* use Argument Clinic's critical section directive to protect queue methods
* remove unnecessary overflow check

Co-authored-by: Erlend E. Aasland <erlend.aasland@protonmail.com>
2024-01-23 20:25:41 +01:00
Serhiy Storchaka 1b719b39b9
gh-114321: Expose more constants in the fcntl module (GH-114322) 2024-01-22 18:09:22 +02:00
neonene 9f7176d360
gh-103092: Ensure `_ctypes.c` static types are accessed via global state (#113857) 2024-01-22 14:40:36 +01:00
Serhiy Storchaka c8351a617b
gh-113796: Add more validation checks in the csv.Dialect constructor (GH-113797)
ValueError is now raised if the same character is used in different roles.
2024-01-22 15:34:16 +02:00
Nikita Sobolev d1b031cc58
gh-114414: Assert PyType_GetModuleByDef result in _threadmodule (#114415) 2024-01-22 10:19:25 +01:00
mpage 28eacf27ef
gh-113884: Refactor `queue.SimpleQueue` to use a ring buffer to store items (#114259)
Use a ring buffer instead of a Python list in order to simplify the
process of making queue.SimpleQueue thread-safe in free-threaded
builds. The ring buffer implementation has no places where critical
sections may be released.
2024-01-19 12:17:51 +00:00
Nikita Sobolev 05e47202a3
gh-114286: Fix `maybe-uninitialized` warning in `Modules/_io/fileio.c` (GH-114287) 2024-01-19 10:25:05 +00:00
Skip Montanaro 72abb8c5d4
gh-114123: Migrate docstring from _csv to csv (#114124)
Co-authored-by: Adam Turner <9087854+AA-Turner@users.noreply.github.com>
Co-authored-by: Éric <merwok@netwok.org>
2024-01-18 22:18:42 +00:00
AN Long 8e31cdc945
gh-103092: Convert some `_ctypes` metatypes to heap types (GH-113620)
Co-authored-by: Erlend E. Aasland <erlend@python.org>
2024-01-18 16:30:27 +01:00
Serhiy Storchaka e2c097ebde
gh-104522: Fix OSError raised when run a subprocess (#114195)
Only set filename to cwd if it was caused by failed chdir(cwd).

_fork_exec() now returns "noexec:chdir" for failed chdir(cwd).

Co-authored-by: Robert O'Shea <PurityLake@users.noreply.github.com>
2024-01-17 16:52:42 -08:00
Radislav Chugunov 0154405350
gh-104282: Fix null pointer dereference in `lzma._decode_filter_properties` (GH-104283) 2024-01-17 13:15:44 +00:00
Steve Dower de4ced54eb
gh-114096: Restore privileges in _winapi.CreateJunction after creating the junction (GH-114089)
This avoids impact on later parts of the application which may be able to do things they otherwise shouldn't.
2024-01-16 16:40:02 +00:00
Jonathon Reinhart e454f9383c
Fix an incorrect comment in iobase_is_closed (GH-102952)
This comment appears to have been mistakenly copied from what is now
called iobase_check_closed() in commit 4d9aec0220.

Also unite the iobase_check_closed() code with the relevant comment.

Co-authored-by: Serhiy Storchaka <storchaka@gmail.com>
2024-01-16 18:27:17 +02:00
AN Long 6c502ba809
gh-114101: Correct PyErr_Format arguments in _testcapi module (#114102)
- use PyErr_SetString() iso. PyErr_Format() in parse_tuple_and_keywords()
- fix misspelled format specifier in CHECK_SIGNNESS() macro
2024-01-16 09:32:39 +01:00
Zackery Spytz 8fd287b18f
gh-78502: Add a trackfd parameter to mmap.mmap() (GH-25425)
If *trackfd* is False, the file descriptor specified by *fileno*
will not be duplicated.

Co-authored-by: Erlend E. Aasland <erlend@python.org>
Co-authored-by: Petr Viktorin <encukou@gmail.com>
Co-authored-by: Serhiy Storchaka <storchaka@gmail.com>
2024-01-16 08:51:46 +01:00