Commit Graph

8316 Commits

Author SHA1 Message Date
Petr Viktorin d655c65561
[3.13] gh-123091: Use more _Py_IsImmortalLoose() (GH-123602) (GH-123622)
Switch more _Py_IsImmortal(...) assertions to _Py_IsImmortalLoose(...)

The remaining calls to _Py_IsImmortal are in free-threaded-only code,
initialization of core objects, tests, and guards that fall back to
code that works with mortal objects.

(cherry picked from commit 57c471a688)
2024-09-03 12:36:42 +02:00
Victor Stinner 10735bff9e
[3.13] gh-123091: Use _Py_IsImmortalLoose() (#123511) (#123600)
gh-123091: Use _Py_IsImmortalLoose() (#123511)

Use _Py_IsImmortalLoose() in bytesobject.c, typeobject.c
and ceval.c.

(cherry picked from commit f1a0d96f41)
2024-09-02 15:23:29 +02:00
Serhiy Storchaka 8b6dd92db7
[3.13] gh-122688: Fix support of var-positional parameter in Argument Clinic (GH-122689) (#122852)
* Parameters after the var-positional parameter are now keyword-only
  instead of positional-or-keyword.
* Correctly calculate min_kw_only.
* Raise errors for invalid combinations of the var-positional parameter
  with "*", "/" and deprecation markers.
(cherry picked from commit 8393608dd9)
2024-09-02 13:03:04 +02:00
Miss Islington (bot) 57ba3b0c6e
[3.13] gh-122527: Fix a crash on deallocation of `PyStructSequence` (GH-122577) (#122625)
gh-122527: Fix a crash on deallocation of `PyStructSequence` (GH-122577)

The `PyStructSequence` destructor would crash if it was deallocated after
its type's dictionary was cleared by the GC, because it couldn't compute
the "real size" of the instance. This could occur with relatively
straightforward code in the free-threaded build or with a reference
cycle involving the type in the default build, due to differing orders
in which `tp_clear()` was called.

Account for the non-sequence fields in `tp_basicsize` and use that,
along with `Py_SIZE()`, to compute the "real" size of a
`PyStructSequence` in the dealloc function. This avoids the accesses to
the type's dictionary during dealloc, which were unsafe.
(cherry picked from commit 4b63cd170e)

Co-authored-by: Sam Gross <colesbury@gmail.com>
2024-09-02 12:47:18 +02:00
Miss Islington (bot) bd29ce8509
[3.13] gh-123448: Move `_PyNoDefault_Type` to the static types array (GH-123449) (#123450)
(cherry picked from commit c9930f5022)

Co-authored-by: Peter Bierma <zintensitydev@gmail.com>
2024-08-28 18:57:28 -07:00
Miss Islington (bot) abaaaff328
[3.13] gh-122982: Extend the deprecation period for bool inversion by two years (GH-123306) (#123316)
gh-122982: Extend the deprecation period for bool inversion by two years (GH-123306)
(cherry picked from commit 249b083ed8)

Co-authored-by: Kirill Podoprigora <kirill.bast9@mail.ru>
2024-08-25 19:48:34 +00:00
Miss Islington (bot) 95b4f9c9ad
[3.13] GH-120097: Make FrameLocalsProxy a mapping (GH-120101) (GH-120749)
Co-authored-by: Alyssa Coghlan <ncoghlan@gmail.com>
2024-08-23 10:26:03 +01:00
Donghee Na 6cd67e413b
[3.13] gh-123083: Fix a potential use-after-free in ``STORE_ATTR_WITH… (#123235)
[3.13] gh-123083: Fix a potential use-after-free in ``STORE_ATTR_WITH_HINT`` (gh-123092)
(cherry picked from commit 297f2e093e)
2024-08-23 01:37:26 +09:00
Miss Islington (bot) ff58c3cb48
[3.13] gh-123022: Fix crash with `Py_Initialize` in background thread (GH-123052) (#123114)
Check that the current default heap is initialized in
`_mi_os_get_aligned_hint` and `mi_os_claim_huge_pages`.

The mimalloc function `_mi_os_get_aligned_hint` assumes that there is an
initialized default heap. This is true for our main thread, but not for
background threads. The problematic code path is usually called during
initialization (i.e., `Py_Initialize`), but it may also be called if the
program allocates large amounts of memory in total.

The crash only affected the free-threaded build.
(cherry picked from commit d061ffea7b)

Co-authored-by: Sam Gross <colesbury@gmail.com>
2024-08-17 20:38:02 +00:00
Miss Islington (bot) 55aede7342
[3.13] gh-122888: Fix crash on certain calls to str() (GH-122889) (#122947)
Fixes GH-122888
(cherry picked from commit 53ebb6232a)

Co-authored-by: Jelle Zijlstra <jelle.zijlstra@gmail.com>
2024-08-12 16:53:05 +00:00
Miss Islington (bot) c1efeb3234
[3.13] gh-120974: Make asyncio `swap_current_task` safe in free-threaded build (GH-122317) (#122612)
gh-120974: Make asyncio `swap_current_task` safe in free-threaded build (GH-122317)

* gh-120974: Make asyncio `swap_current_task` safe in free-threaded build
(cherry picked from commit b5e6fb39a2)

Co-authored-by: Sam Gross <colesbury@gmail.com>
2024-08-02 13:58:24 +00:00
Petr Viktorin 8c8c43e1dc
[3.13] gh-120906: Support arbitrary hashable keys in FrameLocalsProxy (GH-122309) (#122488)
[3.13] gh-120906: Support arbitrary hashable keys in FrameLocalsProxy  (GH-122309)

Co-authored-by: Alyssa Coghlan <ncoghlan@gmail.com>
(cherry picked from commit 5912487938)
2024-07-31 09:31:44 +00:00
Miss Islington (bot) 5212624c42
[3.13] gh-116622: Fix testPyObjectPrintOSError on Android (GH-122487) (#122490)
gh-116622: Fix testPyObjectPrintOSError on Android (GH-122487)

Adds extra handling for way BSD/Android return errors from calls to fwrite.
(cherry picked from commit 82db572813)

Co-authored-by: Malcolm Smith <smith@chaquo.com>
2024-07-31 00:49:14 +00:00
Dino Viehland 11292ab935
[3.13] gh-122208: Don't delivery PyDict_EVENT_ADDED until it can't fail (#122326)
Don't delivery PyDict_EVENT_ADDED until it can't fail
2024-07-30 09:13:31 -07:00
Eric Snow 10cf7d6d00
[3.13] gh-117482: Fix the Slot Wrapper Inheritance Tests (gh-122249)
The tests were only checking cases where the slot wrapper was present in the initial case.  They were missing when the slot wrapper was added in the additional initializations.  This fixes that.

(cherry-picked from commit 490e0ad83a, AKA gh-122248)
2024-07-29 10:25:02 -06:00
Miss Islington (bot) 40925103fc
[3.13] gh-122291: Intern latin-1 one-byte strings at startup (GH-122303) (GH-122347)
(cherry picked from commit bb09ba6792)

Co-authored-by: Petr Viktorin <encukou@gmail.com>
2024-07-27 08:51:02 +00:00
Miss Islington (bot) aca41cfe99
[3.13] GH-121832: Assert that the version number of static builtin types is not changed by PyType_Modified (gh-122290)
Update datetime module and test_type_cache.py to not call PyType_Modified.

(cherry picked from commit e55b05f29e, AKA gh--122182)

Co-authored-by: Mark Shannon <mark@hotpy.org>
2024-07-25 17:34:01 +00:00
Miss Islington (bot) 88a96fc391
[3.13] gh-116322: Fix typo in the GH-ifdef check (GH-122268) (#122284)
gh-116322: Fix typo in the GH-ifdef check (GH-122268)
(cherry picked from commit 9bb2e4623f)

Co-authored-by: Serhiy Storchaka <storchaka@gmail.com>
2024-07-25 14:58:34 +00:00
Eric Snow 716c6771fc
[3.13] gh-117482: Simplify the Fix For Builtin Types Slot Wrappers (gh-121932)
In gh-121602, I applied a fix to a builtin types initialization bug.
That fix made sense in the context of some broader future changes,
but introduced a little bit of extra complexity.  For earlier versions
those future changes are not relevant; we can avoid the extra complexity.
Thus we can revert that earlier change and replace it with this one,
which is more focused and conceptually simpler.  This is essentially
the implementation of an idea that @markshannon pointed out to me.

Note that this change would be much smaller if we didn't have to deal
with repr compatibility for builtin types that explicitly inherit tp slots
(see expect_manually_inherited()).  The alternative is to stop
*explicitly* inheriting tp slots in static PyTypeObject values,
which is churn that we can do separately.
2024-07-24 12:02:36 -06:00
Miss Islington (bot) 77ab53a5f3
[3.13] gh-120974: Make _asyncio._leave_task atomic in the free-threaded build (GH-122139) (#122186)
gh-120974: Make _asyncio._leave_task atomic in the free-threaded build (GH-122139)

* gh-120974: Make _asyncio._leave_task atomic in the free-threaded build

Update `_PyDict_DelItemIf` to allow for an argument to be passed to the
predicate.
(cherry picked from commit a15feded71)

Co-authored-by: Sam Gross <colesbury@gmail.com>
2024-07-23 17:30:58 +00:00
Miss Islington (bot) e458268143
[3.13] gh-121266: Remove Py_ALWAYS_INLINE in dictobject.c (GH-121493) (#122095)
gh-121266: Remove Py_ALWAYS_INLINE in dictobject.c (GH-121493)

compare_unicode_generic(), compare_unicode_unicode() and
compare_generic() are callbacks used by do_lookup(). When enabling
assertions, it's not possible to inline these functions.
(cherry picked from commit c5a6b9afd8)

Co-authored-by: Victor Stinner <vstinner@python.org>
2024-07-21 17:11:12 +00:00
Miss Islington (bot) 2fff3c44b3
[3.13] gh-121266: Change dict check_lookup() return type to int (GH-121581) (#121949)
gh-121266: Change dict check_lookup() return type to int (GH-121581)
(cherry picked from commit 51da3dfbf3)

Co-authored-by: Victor Stinner <vstinner@python.org>
2024-07-20 18:57:27 +00:00
Serhiy Storchaka a45d9051ed
[3.13] gh-121905: Consistently use "floating-point" instead of "floating point" (GH-121907) (GH-122012)
(cherry picked from commit 1a0c7b9ba4)
2024-07-19 09:13:08 +00:00
Miss Islington (bot) 233ed46e6d
[3.13] gh-118934: Make PyEval_GetLocals return borrowed reference (GH-119769) (#121869)
gh-118934: Make PyEval_GetLocals return borrowed reference (GH-119769)
(cherry picked from commit e65cb4c6f0)

Co-authored-by: Tian Gao <gaogaotiantian@hotmail.com>
Co-authored-by: Alyssa Coghlan <ncoghlan@gmail.com>
2024-07-19 01:38:28 +10:00
Petr Viktorin 4395d68c70
[3.13] gh-113993: Don't immortalize in PyUnicode_InternInPlace; keep immortalizing in other API (GH-121364) (GH-121854)
* Switch PyUnicode_InternInPlace to _PyUnicode_InternMortal, clarify docs

* Document immortality in some functions that take `const char *`

This is PyUnicode_InternFromString;
PyDict_SetItemString, PyObject_SetAttrString;
PyObject_DelAttrString; PyUnicode_InternFromString;
and the PyModule_Add convenience functions.

Always point out a non-immortalizing alternative.

* Don't immortalize user-provided attr names in _ctypes
(cherry picked from commit b4aedb23ae)
2024-07-17 14:51:42 +02:00
Miss Islington (bot) 72cd53ea15
[3.13] gh-121863: Immortalize names in code objects to avoid crash (GH-121903) (GH-121904)
(cherry picked from commit cffad5c6ef)

Co-authored-by: Petr Viktorin <encukou@gmail.com>
2024-07-17 09:55:22 +00:00
Serhiy Storchaka 09ff4ec14f
[3.13] gh-121153: Fix some errors with use of _PyLong_CompactValue() (GH-121154) (GH-121900)
* The result has type Py_ssize_t, not intptr_t.
* Type cast between unsigned and signed integer types should be explicit.
* Downcasting should be explicit.
* Fix integer overflow check in sum().
(cherry picked from commit 1801545)
2024-07-17 08:04:45 +00:00
Miss Islington (bot) 6613029084
[3.13] gh-121860: Fix crash when materializing managed dict (GH-121866) (#121867)
The object's inline values may be marked invalid if the materialized
dict was already initialized and then deleted.
(cherry picked from commit 162b41f577)

Co-authored-by: Sam Gross <colesbury@gmail.com>
2024-07-16 19:26:01 +00:00
Miss Islington (bot) 281ffb60cc
[3.13] gh-113993: For string interning, do not rely on (or assert) _Py_IsImmortal (GH-121358) (GH-121851)
gh-113993: For string interning, do not rely on (or assert) _Py_IsImmortal (GH-121358)

Older stable ABI extensions are allowed to make immortal objects mortal.
Instead, use `_PyUnicode_STATE` (`interned` and `statically_allocated`).
(cherry picked from commit 956270d08d)

Co-authored-by: Petr Viktorin <encukou@gmail.com>
2024-07-16 13:42:49 +00:00
Miss Islington (bot) 0794220a69
[3.13] gh-121794: Don't set `ob_tid` to zero in fast-path dealloc (GH-121799) (#121821)
We should maintain the invariant that a zero `ob_tid` implies the
refcount fields are merged.

* Move the assignment in `_Py_MergeZeroLocalRefcount` to immediately
  before the refcount merge.
* Update `_PyTrash_thread_destroy_chain` to set `ob_ref_shared` to
  `_Py_REF_MERGED` when setting `ob_tid` to zero.

Also check this invariant with assertions in the GC in debug builds.
That uncovered a bug when running out of memory during GC.
(cherry picked from commit d23be3947c)

Co-authored-by: Sam Gross <colesbury@gmail.com>
2024-07-15 22:15:23 +00:00
Miss Islington (bot) 767f71466d
[3.13] gh-121731: Fix mimalloc compile error on GNU/Hurd (GH-121732) (#121773)
(cherry picked from commit d005f2c186)

Co-authored-by: Samuel Thibault <samuel.thibault@ens-lyon.org>
2024-07-14 17:14:30 +00:00
Miss Islington (bot) 97b06d543f
[3.13] gh-121660: Fix `ga_getitem` by explicitly checking for `NULL` result (GH-121661) (#121761)
gh-121660: Fix `ga_getitem` by explicitly checking for `NULL` result (GH-121661)
(cherry picked from commit bb802db8cf)

Co-authored-by: sobolevn <mail@sobolevn.me>
2024-07-14 11:48:33 +00:00
Miss Islington (bot) f78e1aa1f0
[3.13] gh-121652: Handle `allocate_weakref` returning NULL (GH-121653) (#121721)
The `allocate_weakref` may return NULL when out of memory. We need to
handle that case and propagate the error.
(cherry picked from commit a640a605a8)

Co-authored-by: Sam Gross <colesbury@gmail.com>
2024-07-13 16:31:50 +00:00
Ken Jin cd74ed0a71
[3.13] gh-120198: Stop the world when setting __class__ on free-threaded build (#121591)
(cherry-picked from commit 3bfc9c8)
2024-07-12 19:35:53 +08:00
Miss Islington (bot) c6dbfbbe3c
[3.13] gh-117482: Fix Builtin Types Slot Wrappers (gh-121630)
When builtin static types are initialized for a subinterpreter, various "tp" slots have already been inherited (for the main interpreter).  This was interfering with the logic in add_operators() (in Objects/typeobject.c), causing a wrapper to get created when it shouldn't.  This change fixes that by preserving the original data from the static type struct and checking that.

(cherry picked from commit 5250a03133, AKA gh-121602)

Co-authored-by: Eric Snow <ericsnowcurrently@gmail.com>
2024-07-11 20:47:38 +00:00
Miss Islington (bot) 927f882d76
[3.13] gh-117657: Fix TSan race in _PyDict_CheckConsistency (GH-121551) (#121590)
The only remaining race in dictobject.c was in _PyDict_CheckConsistency
when the dictionary has shared keys.
(cherry picked from commit 3ec719fabf)

Co-authored-by: Sam Gross <colesbury@gmail.com>
2024-07-10 18:28:44 +00:00
Miss Islington (bot) 867cf40279
[3.13] gh-121460: Skip freeing unallocated arenas (gh-121589)
`munmap(NULL)` is not noop, like `free(NULL)` is.

Fixes an observed testsuite hang on 32-bit ARM systems.

(cherry picked from commit a802277914, AKA gh-121491)

Co-authored-by: Stefano Rivera <stefano@rivera.za.net>
2024-07-10 17:05:11 +00:00
Miss Islington (bot) a1fc8d31a7
[3.13] GH-121439: Allow PyTupleObjects with an ob_size of 20 in the free_list to be reused (gh-121428) (gh-121565)
GH-121439: Allow PyTupleObjects with an ob_size of 20 in the free_list to be reused (gh-121428)
(cherry picked from commit 9585a1a2a2)

Co-authored-by: satori1995 <132636720+satori1995@users.noreply.github.com>
2024-07-10 08:12:54 +00:00
Miss Islington (bot) 2f15383161
[3.13] gh-112075: Fix dict thread safety issues (GH-119288) (#121545)
(cherry picked from commit f0ed1863bd)

Co-authored-by: Germán Méndez Bravo <kronuz@fb.com>
2024-07-09 18:13:23 +00:00
Miss Islington (bot) f0d16f7e82
[3.13] gh-117657: Fix TSAN races in setobject.c (GH-121511) (#121541)
The `used` field must be written using atomic stores because `set_len`
and iterators may access the field concurrently without holding the
per-object lock.
(cherry picked from commit 9c08f40a61)

Co-authored-by: Sam Gross <colesbury@gmail.com>
2024-07-09 16:36:36 +00:00
Miss Islington (bot) c128718f30
[3.13] gh-121368: Fix seq lock memory ordering in _PyType_Lookup (GH-121388) (#121505)
The `_PySeqLock_EndRead` function needs an acquire fence to ensure that
the load of the sequence happens after any loads within the read side
critical section. The missing fence can trigger bugs on macOS arm64.

Additionally, we need a release fence in `_PySeqLock_LockWrite` to
ensure that the sequence update is visible before any modifications to
the cache entry.
(cherry picked from commit 1d3cf79a50)

Co-authored-by: Sam Gross <colesbury@gmail.com>
2024-07-08 19:15:58 +00:00
Miss Islington (bot) 06fd745dd9
[3.13] gh-117657: Fix data races reported by TSAN in some set methods (GH-120914) (#121240)
Refactor the fast Unicode hash check into `_PyObject_HashFast` and use relaxed
atomic loads in the free-threaded build.

After this change, the TSAN doesn't report data races for this method.
(cherry picked from commit 294e724964)

Co-authored-by: AN Long <aisk@users.noreply.github.com>
2024-07-01 19:40:28 +00:00
Miss Islington (bot) 58a3c3c0ad
gh-121115: Skip __index__ in PyLong_AsNativeBytes by default (GH-121118)
(cherry picked from commit 2894aa14f2)

Co-authored-by: Steve Dower <steve.dower@python.org>
2024-06-28 15:52:26 +00:00
Miss Islington (bot) 0a77058b79
[3.13] gh-120858: PyDict_Next should not lock the dict (GH-120859) (#120964)
PyDict_Next no longer locks the dictionary in the free-threaded build. Locking
around individual PyDict_Next calls is not sufficient because the function
returns borrowed references and because it allows concurrent modifications
during the iteraiton loop.

The internal locking also interferes with correct external synchronization
because it may suspend outer critical sections created by the caller.
(cherry picked from commit 375b723d58)

Co-authored-by: Sam Gross <colesbury@gmail.com>
2024-06-24 18:41:19 +00:00
Miss Islington (bot) 6aee5edb84
[3.13] gh-120860: Fix a few bugs in `type_setattro` error paths. (GH-120861) (#120963)
Moves the logic to update the type's dictionary to its own function in order
to make the lock scoping more clear.

Also, ensure that `name` is decref'd on the error path.
(cherry picked from commit dee63cb359)

Co-authored-by: Sam Gross <colesbury@gmail.com>
2024-06-24 18:33:39 +00:00
Petr Viktorin 9769b7ae06
[3.13] gh-113993: Allow interned strings to be mortal, and fix related issues (GH-120520) (GH-120945)
* Add an InternalDocs file describing how interning should work and how to use it.

* Add internal functions to *explicitly* request what kind of interning is done:
  - `_PyUnicode_InternMortal`
  - `_PyUnicode_InternImmortal`
  - `_PyUnicode_InternStatic`

* Switch uses of `PyUnicode_InternInPlace` to those.

* Disallow using `_Py_SetImmortal` on strings directly.
  You should use `_PyUnicode_InternImmortal` instead:
  - Strings should be interned before immortalization, otherwise you're possibly
    interning a immortalizing copy.
  - `_Py_SetImmortal` doesn't handle the `SSTATE_INTERNED_MORTAL` to
    `SSTATE_INTERNED_IMMORTAL` update, and those flags can't be changed in
    backports, as they are now part of public API and version-specific ABI.

* Add private `_only_immortal` argument for `sys.getunicodeinternedsize`, used in refleak test machinery.

* Make sure the statically allocated string singletons are unique. This means these sets are now disjoint:
  - `_Py_ID`
  - `_Py_STR` (including the empty string)
  - one-character latin-1 singletons

  Now, when you intern a singleton, that exact singleton will be interned.

* Add a `_Py_LATIN1_CHR` macro, use it instead of `_Py_ID`/`_Py_STR` for one-character latin-1 singletons everywhere (including Clinic).

* Intern `_Py_STR` singletons at startup.

* For free-threaded builds, intern `_Py_LATIN1_CHR` singletons at startup.

* Beef up the tests. Cover internal details (marked with `@cpython_only`).

* Add lots of assertions

Co-authored-by: Eric Snow <ericsnowcurrently@gmail.com>
2024-06-24 20:24:19 +02:00
Miss Islington (bot) 447e07ab3d
[3.13] gh-119521: Rename IncompleteInputError to _IncompleteInputError and remove from public API/ABI (GH-119680, GH-120955) (GH-120944)
- gh-119521: Rename IncompleteInputError to _IncompleteInputError and remove from public API/ABI (GH-119680)
  (cherry picked from commit ce1064e4c9)

- gh-119521: Use `PyAPI_DATA`, not `extern`, for `_PyExc_IncompleteInputError` (GH-120955)
  (cherry picked from commit ac61d58db0)

Co-authored-by: Pablo Galindo Salgado <Pablogsal@gmail.com>
Co-authored-by: Petr Viktorin <encukou@gmail.com>
2024-06-24 20:23:30 +02:00
Sam Gross 4dc27bc0b7
[3.13] gh-119344: Make critical section API public (GH-119353) (#120856)
This makes the following macros public as part of the non-limited C-API for
locking a single object or two objects at once.

* `Py_BEGIN_CRITICAL_SECTION(op)` / `Py_END_CRITICAL_SECTION()`
* `Py_BEGIN_CRITICAL_SECTION2(a, b)` / `Py_END_CRITICAL_SECTION2()`

The supporting functions and structs used by the macros are also exposed for
cases where C macros are not available.
(cherry picked from commit 8f17d69b7b)
2024-06-21 20:20:41 +00:00
Miss Islington (bot) f3d7823ede
[3.13] gh-120384: Fix array-out-of-bounds crash in `list_ass_subscript` (GH-120442) (#120826)
gh-120384: Fix array-out-of-bounds crash in `list_ass_subscript` (GH-120442)
(cherry picked from commit 8334a1b55c)

Co-authored-by: Nikita Sobolev <mail@sobolevn.me>
2024-06-21 11:16:24 +00:00
Sam Gross 3cb6c4cd60
[3.13] gh-117511: Make PyMutex public in the non-limited API (GH-117731) (#120800)
(cherry picked from commit 3af7263037)
2024-06-20 16:00:25 +00:00