Commit Graph

5436 Commits

Author SHA1 Message Date
Sam Gross 1d3cf79a50
gh-121368: Fix seq lock memory ordering in _PyType_Lookup (#121388)
The `_PySeqLock_EndRead` function needs an acquire fence to ensure that
the load of the sequence happens after any loads within the read side
critical section. The missing fence can trigger bugs on macOS arm64.

Additionally, we need a release fence in `_PySeqLock_LockWrite` to
ensure that the sequence update is visible before any modifications to
the cache entry.
2024-07-08 14:52:07 -04:00
Marc Mueller 31873bea47
gh-121487: Fix deprecation warning for ATOMIC_VAR_INIT in mimalloc (gh-121488) 2024-07-08 14:32:30 -04:00
Gabriele N. Tornetta f5c8d67de6
gh-106597: Remove unnecessary CFrame offsets (#121369) 2024-07-04 17:28:23 +00:00
Irit Katriel 19d1e43e43
gh-121352: use _Py_SourceLocation in symtable (#121353) 2024-07-04 11:28:44 +01:00
Victor Stinner f8373db153
gh-112136: Restore removed _PyArg_Parser (#121262)
Restore the private _PyArg_Parser structure and the private
_PyArg_ParseTupleAndKeywordsFast() function, previously removed
in Python 3.13 alpha 1.

Recreate Include/cpython/modsupport.h header file.
2024-07-03 18:36:57 +02:00
Ken Jin 722229e5dc
gh-121263: Macro-ify most stackref functions for MSVC (GH-121270)
Macro-ify most stackref functions for MSVC
2024-07-03 17:49:31 +08:00
Gabriele N. Tornetta c9bdfbe868
gh-106597: Add more offsets to _Py_DebugOffsets (#121311)
Add more offsets to _Py_DebugOffsets

We add a few more offsets that are required by some out-of-process
tools, such as [Austin](https://github.com/p403n1x87/austin).
2024-07-03 08:53:44 +00:00
Pablo Galindo Salgado b180788d4a
gh-115773: Add sizes to debug offset structure (#120112) 2024-07-02 17:54:33 +00:00
Sam Gross 8e8d202f55
gh-117139: Add _PyTuple_FromStackRefSteal and use it (#121244)
Avoids the extra conversion from stack refs to PyObjects.
2024-07-02 12:30:14 -04:00
Brandt Bucher 33903c53db
GH-116017: Get rid of _COLD_EXITs (GH-120960) 2024-07-01 13:17:40 -07:00
AN Long 294e724964
gh-117657: Fix data races reported by TSAN in some set methods (#120914)
Refactor the fast Unicode hash check into `_PyObject_HashFast` and use relaxed
atomic loads in the free-threaded build.

After this change, the TSAN doesn't report data races for this method.
2024-07-01 15:11:39 -04:00
Erlend E. Aasland f80376b129
gh-113565: Improve and harden detection of curses dependencies (#119816)
1. Use pkg-config to check for ncursesw/panelw. If that fails, use 
   pkg-config to check for ncurses/panel.
2. Regardless of pkg-config output, search for curses/panel headers, so
   we're sure we have all defines in pyconfig.h.
3. Regardless of pkg-config output, check if libncurses or libncursesw
   contains the 'initscr' symbol; if it does _and_ pkg-config failed
   earlier, add the resulting -llib linker option to CURSES_LIBS.
   Ditto for 'update_panels' and PANEL_LIBS.
4. Wrap the rest of the checks with WITH_SAVE_ENV and make sure we're 
   using updated LIBS and CPPFLAGS for those.

Add the PY_CHECK_CURSES convenience macro.
2024-07-01 08:10:03 +00:00
Yureka c3677befbe
gh-119447: Fix build with _PY_SHORT_FLOAT_REPR == 0 (#121178) 2024-06-30 11:40:40 +02:00
Steve Dower 2894aa14f2
gh-121115: Skip __index__ in PyLong_AsNativeBytes by default (GH-121118) 2024-06-28 16:26:21 +01:00
Victor Stinner 43709d5d54
Check for compiler warnings in test_cext on Windows (#121088)
On Windows, test_cext and test_cppext now pass /WX flag to the MSC
compiler to treat all compiler warnings as errors. In verbose mode,
these tests now log the compiler commands to help debugging.

Change Py_BUILD_ASSERT_EXPR implementation on Windows to avoid a
compiler warning about an unnamed structure.
2024-06-28 14:41:37 +02:00
Irit Katriel 6f7acaab50
gh-120686: remove unused internal c api functions (#120687) 2024-06-27 11:09:30 +01:00
Victor Stinner 12af8ec864
gh-121040: Use __attribute__((fallthrough)) (#121044)
Fix warnings when using -Wimplicit-fallthrough compiler flag.

Annotate explicitly "fall through" switch cases with a new
_Py_FALLTHROUGH macro which uses __attribute__((fallthrough)) if
available. Replace "fall through" comments with _Py_FALLTHROUGH.

Add _Py__has_attribute() macro. No longer define __has_attribute()
macro if it's not defined. Move also _Py__has_builtin() at the top
of pyport.h.

Co-Authored-By: Nikita Sobolev <mail@sobolevn.me>
2024-06-27 09:58:44 +00:00
Ken Jin 22b0de2755
gh-117139: Convert the evaluation stack to stack refs (#118450)
This PR sets up tagged pointers for CPython.

The general idea is to create a separate struct _PyStackRef for everything on the evaluation stack to store the bits. This forces the C compiler to warn us if we try to cast things or pull things out of the struct directly.

Only for free threading: We tag the low bit if something is deferred - that means we skip incref and decref operations on it. This behavior may change in the future if Mark's plans to defer all objects in the interpreter loop pans out.

This implies a strict stack reference discipline is required. ALL incref and decref operations on stackrefs must use the stackref variants. It is unsafe to untag something then do normal incref/decref ops on it.

The new incref and decref variants are called dup and close. They mimic a "handle" API operating on these stackrefs.

Please read Include/internal/pycore_stackref.h for more information!

---------

Co-authored-by: Mark Shannon <9448417+markshannon@users.noreply.github.com>
2024-06-27 03:10:43 +08:00
Victor Stinner e51e880e75
gh-120593: Fix const qualifier in _PyLong_CompactValue() (#121053)
Remove the const qualifier of the argument of functions:

* _PyLong_IsCompact()
* _PyLong_CompactValue()

Py_TYPE() argument is not const.

Fix the compiler warning:

  Include/cpython/longintrepr.h: In function ‘_PyLong_CompactValue’:
  Include/pyport.h:19:31: error: cast discards ‘const’ qualifier from
  pointer target type [-Werror=cast-qual]
    (...)
  Include/cpython/longintrepr.h:133:30: note: in expansion of macro
  ‘Py_TYPE’
    assert(PyType_HasFeature(Py_TYPE(op), Py_TPFLAGS_LONG_SUBCLASS));
2024-06-26 20:11:21 +02:00
Victor Stinner 9cd2dcbb87
gh-120593: Fix const qualifier in pyatomic.h (#121055) 2024-06-26 20:10:47 +02:00
Victor Stinner 9e4a81f00f
gh-120642: Move private PyCode APIs to the internal C API (#120643)
* Move _Py_CODEUNIT and related functions to pycore_code.h.
* Move _Py_BackoffCounter to pycore_backoff.h.
* Move Include/cpython/optimizer.h content to pycore_optimizer.h.
* Remove Include/cpython/optimizer.h.
* Remove PyUnstable_Replace_Executor().

Rename functions:

* PyUnstable_GetExecutor() => _Py_GetExecutor()
* PyUnstable_GetOptimizer() => _Py_GetOptimizer()
* PyUnstable_SetOptimizer() => _Py_SetTier2Optimizer()
* PyUnstable_Optimizer_NewCounter() => _PyOptimizer_NewCounter()
* PyUnstable_Optimizer_NewUOpOptimizer() => _PyOptimizer_NewUOpOptimizer()
2024-06-26 13:54:03 +02:00
Kumar Aditya 82235449b8
gh-107803: fix thread safety issue in double linked list implementation (#121007) 2024-06-26 05:11:32 +00:00
Eric Snow a905721b9c
gh-120838: Add _PyThreadState_WHENCE_FINI (gh-121010)
We also add _PyThreadState_NewBound() and drop _PyThreadState_SetWhence().

This change only affects internal API.
2024-06-25 14:35:12 -06:00
Victor Stinner 2e157851e3
gh-119182: Add PyUnicodeWriter_WriteUCS4() function (#120849) 2024-06-24 17:40:39 +02:00
Petr Viktorin ce1064e4c9
gh-119521: Use `PyAPI_DATA`, not `extern`, for `_PyExc_IncompleteInputError` (GH-120955) 2024-06-24 17:30:29 +02:00
Pablo Galindo Salgado ac61d58db0
gh-119521: Rename IncompleteInputError to _IncompleteInputError and remove from public API/ABI (GH-119680)
Signed-off-by: Pablo Galindo <pablogsal@gmail.com>
Co-authored-by: Petr Viktorin <encukou@gmail.com>
2024-06-24 14:08:12 +02:00
Irit Katriel 65a12c559c
gh-120834: fix type of *_iframe field in _PyGenObject_HEAD declaration (#120835) 2024-06-24 10:23:38 +01:00
Kumar Aditya 4717aaa1a7
GH-107803: double linked list implementation for asyncio tasks (GH-107804)
* linked list

* add tail optmiization to linked list

* wip

* wip

* wip

* more fixes

* finally it works

* add tests

* remove weakreflist

* add some comments

* reduce code duplication in _asynciomodule.c

* address some review comments

* add invariants about the state of the linked list

* add better explanation

* clinic regen

* reorder branches for better branch prediction

* Update Modules/_asynciomodule.c

* Apply suggestions from code review

Co-authored-by: Itamar Oren <itamarost@gmail.com>

* fix capturing of eager tasks

* add comment to task finalization

* fix tests and couple c implmentation to c task

improved linked-list logic and more comments

* fix test

---------

Co-authored-by: Itamar Oren <itamarost@gmail.com>
2024-06-22 10:58:35 -07:00
Sam Gross 8f17d69b7b
gh-119344: Make critical section API public (#119353)
This makes the following macros public as part of the non-limited C-API for
locking a single object or two objects at once.

* `Py_BEGIN_CRITICAL_SECTION(op)` / `Py_END_CRITICAL_SECTION()`
* `Py_BEGIN_CRITICAL_SECTION2(a, b)` / `Py_END_CRITICAL_SECTION2()`

The supporting functions and structs used by the macros are also exposed for
cases where C macros are not available.
2024-06-21 15:50:18 -04:00
Victor Stinner 4123226bbd
gh-119182: Add PyUnicodeWriter_DecodeUTF8Stateful() (#120639)
Add PyUnicodeWriter_WriteWideChar() and
PyUnicodeWriter_DecodeUTF8Stateful() functions.

Co-authored-by: Serhiy Storchaka <storchaka@gmail.com>
2024-06-21 19:33:15 +02:00
Petr Viktorin 6f1d448bc1
gh-113993: Allow interned strings to be mortal, and fix related issues (GH-120520)
* Add an InternalDocs file describing how interning should work and how to use it.

* Add internal functions to *explicitly* request what kind of interning is done:
  - `_PyUnicode_InternMortal`
  - `_PyUnicode_InternImmortal`
  - `_PyUnicode_InternStatic`

* Switch uses of `PyUnicode_InternInPlace` to those.

* Disallow using `_Py_SetImmortal` on strings directly.
  You should use `_PyUnicode_InternImmortal` instead:
  - Strings should be interned before immortalization, otherwise you're possibly
    interning a immortalizing copy.
  - `_Py_SetImmortal` doesn't handle the `SSTATE_INTERNED_MORTAL` to
    `SSTATE_INTERNED_IMMORTAL` update, and those flags can't be changed in
    backports, as they are now part of public API and version-specific ABI.

* Add private `_only_immortal` argument for `sys.getunicodeinternedsize`, used in refleak test machinery.

* Make sure the statically allocated string singletons are unique. This means these sets are now disjoint:
  - `_Py_ID`
  - `_Py_STR` (including the empty string)
  - one-character latin-1 singletons

  Now, when you intern a singleton, that exact singleton will be interned.

* Add a `_Py_LATIN1_CHR` macro, use it instead of `_Py_ID`/`_Py_STR` for one-character latin-1 singletons everywhere (including Clinic).

* Intern `_Py_STR` singletons at startup.

* For free-threaded builds, intern `_Py_LATIN1_CHR` singletons at startup.

* Beef up the tests. Cover internal details (marked with `@cpython_only`).

* Add lots of assertions

Co-Authored-By: Eric Snow <ericsnowcurrently@gmail.com>
2024-06-21 17:19:31 +02:00
Sam Gross 3af7263037
gh-117511: Make PyMutex public in the non-limited API (#117731) 2024-06-20 11:29:08 -04:00
Xie Yanbo 656a1c8108
Fix typos in comments (#120481) 2024-06-19 23:16:14 -04:00
Mark Shannon 00257c746c
GH-119462: Enforce invariants of type versioning (GH-120731)
* Remove uses of Py_TPFLAGS_VALID_VERSION_TAG
2024-06-19 17:38:45 +01:00
Victor Stinner 16f8e22e7c
gh-120600: Make Py_TYPE() opaque in limited C API 3.14 (#120601)
In the limited C API 3.14 and newer, Py_TYPE() is now implemented as
an opaque function call to hide implementation details.
2024-06-18 14:28:48 +00:00
Sam Gross e8752d7b80
gh-118789: Add `PyUnstable_Object_ClearWeakRefsNoCallbacks` (#118807)
This exposes `PyUnstable_Object_ClearWeakRefsNoCallbacks` as an unstable
C-API function to provide a thread-safe mechanism for clearing weakrefs
without executing callbacks.

Some C-API extensions need to clear weakrefs without calling callbacks,
such as after running finalizers like we do in subtype_dealloc.
Previously they could use `_PyWeakref_ClearRef` on each weakref, but
that's not thread-safe in the free-threaded build.

Co-authored-by: Petr Viktorin <encukou@gmail.com>
2024-06-18 09:57:23 -04:00
Mark Shannon 9cefcc0ee7
GH-120507: Lower the `BEFORE_WITH` and `BEFORE_ASYNC_WITH` instructions. (#120640)
* Remove BEFORE_WITH and BEFORE_ASYNC_WITH instructions.

* Add LOAD_SPECIAL instruction

* Reimplement `with` and `async with` statements using LOAD_SPECIAL
2024-06-18 12:17:46 +01:00
Victor Stinner 5c4235cd8c
gh-119182: Add PyUnicodeWriter C API (#119184) 2024-06-17 17:10:52 +02:00
Bénédikt Tran 4bf17c381f
gh-119933: Improve ``SyntaxError`` message for invalid type parameters expressions (#119976)
Co-authored-by: Jelle Zijlstra <jelle.zijlstra@gmail.com>
2024-06-17 06:51:03 -07:00
Mark Shannon 274f844830
GH-120619: Clean up `RETURN_VALUE` instruction (GH-120624)
* Rename _POP_FRAME to _RETURN_VALUE as it returns a value as well as popping a frame.

* Remove remaining _POP_FRAMEs
2024-06-17 14:40:11 +01:00
Victor Stinner c2d5df5787
gh-83754: Use the Py_TYPE() macro (#120599)
Don't access directly PyObject.ob_type, but use the Py_TYPE() macro
instead.
2024-06-17 10:34:29 +02:00
Ken Jin 6f63dfff6f
gh-117657: Make PyType_HasFeature (exported version) atomic (#120484)
Make PyType_HasFeature (exported version) atomic
2024-06-15 22:39:22 +08:00
Eric Snow b2e71ff4f8
gh-120161: Fix a Crash in the _datetime Module (gh-120182)
In gh-120009 I used an atexit hook to finalize the _datetime module's static types at interpreter shutdown.  However, atexit hooks are executed very early in finalization, which is a problem in the few cases where a subclass of one of those static types is still alive until the final GC collection.  The static builtin types don't have this probably because they are finalized toward the end, after the final GC collection.  To avoid the problem for _datetime, I have applied a similar approach here.

Also, credit goes to @mgorny and @neonene for the new tests.

FYI, I would have liked to take a slightly cleaner approach with managed static types, but wanted to get a smaller fix in first for the sake of backporting.  I'll circle back to the cleaner approach with a future change on the main branch.
2024-06-14 13:29:09 -06:00
Ken Jin eebae2c460
gh-117657: Make PyType_HasFeature atomic (GH-120210)
Make PyType_HasFeature atomic
2024-06-13 17:29:19 +08:00
neonene 127c1d2771
gh-71587: Drop local reference cache to `_strptime` module in `_datetime` (gh-120224)
The _strptime module object was cached in a static local variable (in the datetime.strptime() implementation).  That's a problem when it crosses isolation boundaries, such as reinitializing the runtme or between interpreters.  This change fixes the problem by dropping the static variable, instead always relying on the normal sys.modules cache (via PyImport_Import()).
2024-06-12 10:46:39 -06:00
Bénédikt Tran 32d3e05fe6
gh-120029: remove unused macros in ``symtable.c`` (#120222)
Co-authored-by: Carl Meyer <carl@oddbird.net>
Co-authored-by: Jelle Zijlstra <jelle.zijlstra@gmail.com>
2024-06-12 09:23:45 -04:00
Ken Jin e16aed63f6
gh-117657: Make Py_TYPE and Py_SET_TYPE thread safe (GH-120165)
Co-authored-by: Bénédikt Tran <10796600+picnixz@users.noreply.github.com>
Co-authored-by: Nadeshiko Manju <me@manjusaka.me>
2024-06-12 20:41:07 +08:00
Bénédikt Tran 755dab719d
gh-120029: make `symtable.Symbol.__repr__` correctly reflect the compiler's flags, add methods (#120099)
Expose :class:`symtable.Symbol` methods :meth:`~symtable.Symbol.is_free_class`,
:meth:`~symtable.Symbol.is_comp_iter` and :meth:`~symtable.Symbol.is_comp_cell`.

---------

Co-authored-by: Carl Meyer <carl@oddbird.net>
2024-06-12 05:14:50 -06:00
Kirill Podoprigora 939c201e00
gh-120326: Include <intrin.h> on Windows with Free Threading (#120329) 2024-06-11 19:50:21 +02:00
Jelle Zijlstra 9b8611eeea
gh-119180: PEP 649 compiler changes (#119361) 2024-06-11 13:06:49 +00:00