Commit Graph

1468 Commits

Author SHA1 Message Date
Brett Simmers 2731913dd5
gh-116167: Allow disabling the GIL with `PYTHON_GIL=0` or `-X gil=0` (#116338)
In free-threaded builds, running with `PYTHON_GIL=0` will now disable the
GIL. Follow-up issues track work to re-enable the GIL when loading an
incompatible extension, and to disable the GIL by default.

In order to support re-enabling the GIL at runtime, all GIL-related data
structures are initialized as usual, and disabling the GIL simply sets a flag
that causes `take_gil()` and `drop_gil()` to return early.
2024-03-11 11:02:58 -04:00
Mark Shannon b6ae6da1bd
GH-116596: Better determination of escaping uops. (GH-116597) 2024-03-11 13:37:48 +00:00
Donghee Na 6c4fc209e1
gh-112536: Define MI_TSAN to 1 for --with-mimalloc and --with-thread-sanitizer (gh-116558) 2024-03-11 22:25:55 +09:00
Mark Shannon 4e5df2013f
GH-116468: Use constants instead of `oparg` in stack effects when `oparg` is known to be a constant. (GH-116469) 2024-03-11 09:30:15 +00:00
Dino Viehland 7db871e4fa
gh-112075: Support freeing object memory via QSBR (#116344)
Free objects with qsbr if shared
2024-03-08 09:56:36 -08:00
Ken Jin 41457c7fdb
gh-116381: Remove bad specializations, add fail stats (GH-116464)
* Remove bad specializations, add fail stats
2024-03-08 00:21:21 +08:00
Ken Jin 7114cf20c0
gh-116381: Specialize CONTAINS_OP (GH-116385)
* Specialize CONTAINS_OP

* 📜🤖 Added by blurb_it.

* Add PyAPI_FUNC for JIT

---------

Co-authored-by: blurb-it[bot] <43283697+blurb-it[bot]@users.noreply.github.com>
2024-03-07 03:30:11 +08:00
Sam Gross c012c8ab7b
gh-115103: Delay reuse of mimalloc pages that store PyObjects (#115435)
This implements the delayed reuse of mimalloc pages that contain Python
objects in the free-threaded build.

Allocations of the same size class are grouped in data structures called
pages. These are different from operating system pages. For thread-safety, we
want to ensure that memory used to store PyObjects remains valid as long as
there may be concurrent lock-free readers; we want to delay using it for
other size classes, in other heaps, or returning it to the operating system.

When a mimalloc page becomes empty, instead of immediately freeing it, we tag
it with a QSBR goal and insert it into a per-thread state linked list of
pages to be freed. When mimalloc needs a fresh page, we process the queue and
free any still empty pages that are now deemed safe to be freed. Pages
waiting to be freed are still available for allocations of the same size
class and allocating from a page prevent it from being freed. There is
additional logic to handle abandoned pages when threads exit.
2024-03-06 09:42:11 -05:00
Mark Shannon 27858e2a17
GH-113710: Tier 2 optimizer: check the function instead of checking globals. (GH-116410) 2024-03-06 13:12:23 +00:00
Sam Gross 72714c0266
gh-115103: Enable internal mimalloc assertions in debug builds (#116343)
This sets `MI_DEBUG` to `2` in debug builds to enable `mi_assert_internal()`
calls. Expensive internal assertions are not enabled.

This also disables an assertion in free-threaded builds that would be
triggered by the free-threaded GC because we traverse heaps that are not
owned by the current thread.
2024-03-05 13:54:20 -05:00
cui fliter e7ba6e9dbe
chore: fix typos (#116345)
Signed-off-by: cui fliter <imcusg@gmail.com>
2024-03-05 09:05:52 -07:00
Mark Shannon 23db9c6227
GH-115685: Split `_TO_BOOL_ALWAYS_TRUE` into micro-ops (GH-116352) 2024-03-05 15:23:08 +00:00
Mark Shannon 0c81ce1360
GH-115819: Eliminate Boolean guards when value is known (GH-116355) 2024-03-05 15:06:00 +00:00
Mark Shannon cbf3d38cbe
GH-115685: Optimize `TO_BOOL` and variants based on truthiness of input. (GH-116311) 2024-03-05 11:23:46 +00:00
Brett Cannon 90a1e9880f
GH-116226: include `pthread_stubs.h` in `pycore_pythreads.h` (#116227) 2024-03-01 15:22:31 -08:00
mpage 9e88173d36
gh-114271: Make `_thread.ThreadHandle` thread-safe in free-threaded builds (GH-115190)
Make `_thread.ThreadHandle` thread-safe in free-threaded builds

We protect the mutable state of `ThreadHandle` using a `_PyOnceFlag`.
Concurrent operations (i.e. `join` or `detach`) on `ThreadHandle` block
until it is their turn to execute or an earlier operation succeeds.
Once an operation has been applied successfully all future operations
complete immediately.

The `join()` method is now idempotent. It may be called multiple times
but the underlying OS thread will only be joined once. After `join()`
succeeds, any future calls to `join()` will succeed immediately.

The internal thread handle `detach()` method has been removed.
2024-03-01 13:43:12 -08:00
Brett Simmers 339c8e1c13
gh-115999: Disable the specializing adaptive interpreter in free-threaded builds (#116013)
For now, disable all specialization when the GIL might be disabled.
2024-02-29 21:53:32 -05:00
Ken Jin d01886c5c9
gh-115685: Type/values propagate for TO_BOOL in tier 2 (GH-115686) 2024-03-01 06:13:38 +08:00
Guido van Rossum 0656509033
gh-116088: Insert bottom checks after all sym_set_...() calls (#116089)
This changes the `sym_set_...()` functions to return a `bool` which is `false`
when the symbol is `bottom` after the operation.

All calls to such functions now check this result and go to `hit_bottom`,
a special error label that prints a different message and then reports
that it wasn't able to optimize the trace. No executor will be produced
in this case.
2024-02-29 18:55:29 +00:00
Brandt Bucher f0df35eeca
GH-115802: JIT "small" code for Windows (GH-115964) 2024-02-29 08:11:28 -08:00
Guido van Rossum 3409bc29c9
gh-115859: Re-enable T2 optimizer pass by default (#116062)
This undoes the *temporary* default disabling of the T2 optimizer pass in gh-115860.

- Add a new test that reproduces Brandt's example from gh-115859; it indeed crashes before gh-116028 with PYTHONUOPSOPTIMIZE=1
- Re-enable the optimizer pass in T2, stop checking PYTHONUOPSOPTIMIZE
- Rename the env var to disable T2 entirely to PYTHON_UOPS_OPTIMIZE (must be explicitly set to 0 to disable)
- Fix skipIf conditions on tests in test_opt.py accordingly
- Export sym_is_bottom() (for debugging)
- Fix various things in the `_BINARY_OP_` specializations in the abstract interpreter:
  - DECREF(temp)
  - out-of-space check after sym_new_const()
  - add sym_matches_type() checks, so even if we somehow reach a binary op with symbolic constants of the wrong type on the stack we won't trigger the type assert
2024-02-28 22:38:01 +00:00
Guido van Rossum e2a3e4b748
gh-115816: Improve internal symbols API in optimizer (#116028)
- Any `sym_set_...` call that attempts to set conflicting information
  cause the symbol to become `bottom` (contradiction).
- All `sym_is...` and similar calls return false or NULL for `bottom`.
- Everything's tested.
- The tests still pass with `PYTHONUOPSOPTIMIZE=1`.
2024-02-28 17:55:56 +00:00
Pablo Galindo Salgado 1752b51012
gh-115773: Add tests to exercise the _Py_DebugOffsets structure (#115774) 2024-02-28 10:17:34 +00:00
Jelle Zijlstra d53560deb2
gh-105858: Expose some union-related objects as internal APIs (GH-116025)
We now use these in the AST parsing code after gh-105880. A few comparable types (e.g.,
NoneType) are already exposed as internal APIs.
2024-02-28 09:56:40 +00:00
Jelle Zijlstra ed4dfd8825
gh-105858: Improve AST node constructors (#105880)
Demonstration:

>>> ast.FunctionDef.__annotations__
{'name': <class 'str'>, 'args': <class 'ast.arguments'>, 'body': list[ast.stmt], 'decorator_list': list[ast.expr], 'returns': ast.expr | None, 'type_comment': str | None, 'type_params': list[ast.type_param]}
>>> ast.FunctionDef()
<stdin>:1: DeprecationWarning: FunctionDef.__init__ missing 1 required positional argument: 'name'. This will become an error in Python 3.15.
<stdin>:1: DeprecationWarning: FunctionDef.__init__ missing 1 required positional argument: 'args'. This will become an error in Python 3.15.
<ast.FunctionDef object at 0x101959460>
>>> node = ast.FunctionDef(name="foo", args=ast.arguments())
>>> node.decorator_list
[]
>>> ast.FunctionDef(whatever="you want", name="x", args=ast.arguments())
<stdin>:1: DeprecationWarning: FunctionDef.__init__ got an unexpected keyword argument 'whatever'. Support for arbitrary keyword arguments is deprecated and will be removed in Python 3.15.
<ast.FunctionDef object at 0x1019581f0>
2024-02-27 18:13:03 -08:00
Mark Shannon 6ecfcfe894
GH-115816: Assorted naming and formatting changes to improve maintainability. (GH-115987)
* Rename _Py_UOpsAbstractInterpContext to _Py_UOpsContext and _Py_UOpsSymType to _Py_UopsSymbol.

* #define shortened form of _Py_uop_... names for improved readability.
2024-02-27 13:25:02 +00:00
Mark Shannon 10fbcd6c5d
GH-115816: Make tier2 optimizer symbols testable, and add a few tests. (GH-115953) 2024-02-27 10:51:26 +00:00
Dino Viehland 1002fbe12e
gh-112075: Iterating a dict shouldn't require locks (#115108)
Makes iteration of a dict be lock free for the forward iteration case.
2024-02-22 12:02:39 -08:00
AN Long 87a65a5bd4
gh-115304: Add doc for initializing PyMutex as a global variable (#115305) 2024-02-21 12:35:53 -05:00
Victor Stinner e4c34f04a1
gh-110850: Cleanup PyTime API: PyTime_t are nanoseconds (#115753)
PyTime_t no longer uses an arbitrary unit, it's always a number of
nanoseconds (64-bit signed integer).

* Rename _PyTime_FromNanosecondsObject() to _PyTime_FromLong().
* Rename _PyTime_AsNanosecondsObject() to _PyTime_AsLong().
* Remove pytime_from_nanoseconds().
* Remove pytime_as_nanoseconds().
* Remove _PyTime_FromNanoseconds().
2024-02-21 11:46:00 +01:00
Victor Stinner 77430b6a32
gh-110850: Replace private _PyTime_MAX with public PyTime_MAX (#115751)
Remove references to the old names _PyTime_MIN
and _PyTime_MAX, now that PyTime_MIN and
PyTime_MAX are public.

Replace also _PyTime_MIN with PyTime_MIN.
2024-02-21 08:11:40 +00:00
Donghee Na 259730bbb5
gh-112087: Make list_{concat, repeat, inplace_repeat, ass_item) to be thread-safe (gh-115605) 2024-02-21 01:38:09 +00:00
Dino Viehland 54071460d7
gh-112075: Accessing a single element should optimistically avoid locking (#115109)
Makes accessing a single element thread safe and typically lock free
2024-02-20 17:08:14 -08:00
Dino Viehland 176df09adb
gh-112075: Make PyDictKeysObject thread-safe (#114741)
Adds locking for shared PyDictKeysObject's for dictionaries
2024-02-20 16:40:37 -08:00
Victor Stinner 145bc2d638
gh-110850: Use public PyTime functions (#115746)
Replace private _PyTime functions with public PyTime functions.

random_seed_time_pid() now reports errors to its caller.
2024-02-20 23:31:30 +00:00
Victor Stinner 52d1477566
gh-110850: Rename internal PyTime C API functions (#115734)
Rename functions:

* _PyTime_GetSystemClock() => _PyTime_TimeUnchecked()
* _PyTime_GetPerfCounter() => _PyTime_PerfCounterUnchecked()
* _PyTime_GetMonotonicClock() => _PyTime_MonotonicUnchecked()
* _PyTime_GetSystemClockWithInfo() => _PyTime_TimeWithInfo()
* _PyTime_GetMonotonicClockWithInfo() => _PyTime_MonotonicWithInfo()
* _PyTime_GetMonotonicClockWithInfo() => _PyTime_MonotonicWithInfo()

Changes:

* Remove "typedef PyTime_t PyTime_t;" which was
  "typedef PyTime_t _PyTime_t;" before a previous rename.
* Update comments of "Unchecked" functions.
* Remove invalid PyTime_Time() comment.
2024-02-20 22:16:37 +00:00
Sam Gross e3ad6ca56f
gh-115103: Implement delayed free mechanism for free-threaded builds (#115367)
This adds `_PyMem_FreeDelayed()` and supporting functions. The
`_PyMem_FreeDelayed()` function frees memory with the same allocator as
`PyMem_Free()`, but after some delay to ensure that concurrent lock-free
readers have finished.
2024-02-20 13:04:37 -05:00
Victor Stinner d207c7cd5a
gh-110850: Cleanup pycore_time.h includes (#115724)
<pycore_time.h> include is no longer needed to get the PyTime_t type
in internal header files. This type is now provided by <Python.h>
include. Add <pycore_time.h> includes to C files instead.
2024-02-20 16:50:43 +00:00
Sam Gross cc82e33af9
gh-115491: Keep some fields valid across allocations (free-threading) (#115573)
This avoids filling the memory occupied by ob_tid, ob_ref_local, and
ob_ref_shared with debug bytes (e.g., 0xDD) in mimalloc in the
free-threaded build.
2024-02-20 10:36:40 -05:00
Victor Stinner 9af80ec83d
gh-110850: Replace _PyTime_t with PyTime_t (#115719)
Run command:

sed -i -e 's!\<_PyTime_t\>!PyTime_t!g' $(find -name "*.c" -o -name "*.h")
2024-02-20 15:02:27 +00:00
Brett Simmers 0749244d13
gh-112175: Add `eval_breaker` to `PyThreadState` (#115194)
This change adds an `eval_breaker` field to `PyThreadState`. The primary
motivation is for performance in free-threaded builds: with thread-local eval
breakers, we can stop a specific thread (e.g., for an async exception) without
interrupting other threads.

The source of truth for the global instrumentation version is stored in the
`instrumentation_version` field in PyInterpreterState. Threads usually read the
version from their local `eval_breaker`, where it continues to be colocated
with the eval breaker bits.
2024-02-20 09:57:48 -05:00
Ken Jin dcba21f905
gh-115687: Split up guards from COMPARE_OP (GH-115688) 2024-02-20 11:30:49 +00:00
Mark Shannon 626c414995
GH-115457: Support splitting and replication of micro ops. (GH-115558) 2024-02-20 10:50:59 +00:00
Mark Shannon 7b21403ccd
GH-112354: Initial implementation of warm up on exits and trace-stitching (GH-114142) 2024-02-20 09:39:55 +00:00
Donghee Na 8db8d7118e
gh-111968: Split _Py_async_gen_asend_freelist out of _Py_async_gen_fr… (gh-115546) 2024-02-17 10:03:10 +09:00
Sam Gross 5903190727
gh-115103: Implement delayed memory reclamation (QSBR) (#115180)
This adds a safe memory reclamation scheme based on FreeBSD's "GUS" and
quiescent state based reclamation (QSBR). The API provides a mechanism
for callers to detect when it is safe to free memory that may be
concurrently accessed by readers.
2024-02-16 15:25:19 -05:00
Sam Gross b24c9161a6
gh-112529: Make the GC scheduling thread-safe (#114880)
The GC keeps track of the number of allocations (less deallocations)
since the last GC. This buffers the count in thread-local state and uses
atomic operations to modify the per-interpreter count. The thread-local
buffering avoids contention on shared state.

A consequence is that the GC scheduling is not as precise, so
"test_sneaky_frame_object" is skipped because it requires that the GC be
run exactly after allocating a frame object.
2024-02-16 11:22:27 -05:00
Donghee Na 321d13fd2b
gh-111968: Split _Py_dictkeys_freelist out of _Py_dict_freelist (gh-115505) 2024-02-16 01:01:36 +00:00
Dino Viehland 454d7963e3
gh-113743: Use per-interpreter locks for types (#115541)
Move type-lock to per-interpreter lock to avoid heavy contention in interpreters test
2024-02-15 16:28:31 -08:00
Dino Viehland ae460d450a
gh-113743: Make the MRO cache thread-safe in free-threaded builds (#113930)
Makes _PyType_Lookup thread safe, including:
    Thread safety of the underlying cache.
    Make mutation of mro and type members thread safe
    Also _PyType_GetMRO and _PyType_GetBases are currently returning borrowed references which aren't safe.
2024-02-15 10:54:57 -08:00
monkeyman192 298bcdc185
gh-112433: Add optional _align_ attribute to ctypes.Structure (GH-113790) 2024-02-15 16:40:20 +02:00
Sam Gross ad4f909e0e
gh-115432: Add critical section variant that handles a NULL object (#115433)
This adds `Py_XBEGIN_CRITICAL_SECTION` and
`Py_XEND_CRITICAL_SECTION`, which accept a possibly NULL object as an
argument. If the argument is NULL, then nothing is locked or unlocked.
Otherwise, they behave like `Py_BEGIN/END_CRITICAL_SECTION`.
2024-02-15 08:37:54 -05:00
mpage dc978f6ab6
gh-112050: Make collections.deque thread-safe in free-threaded builds (#113830)
Use critical sections to make deque methods that operate on mutable 
state thread-safe when the GIL is disabled. This is mostly accomplished
by using the @critical_section Argument Clinic directive, though there
are a few places where this was not possible and critical sections had
to be manually acquired/released.
2024-02-15 09:22:47 +01:00
Sam Gross 326119d373
gh-112529: Use _PyThread_Id() in mimalloc in free-threaded build (#115488)
The free-threaded GC uses mimallocs segment thread IDs to restore
the overwritten `ob_tid` thread ids in PyObjects. For that reason, it's
important that PyObjects and mimalloc use the same identifiers.
2024-02-14 16:41:29 -05:00
mpage a95b1a56bb
gh-115041: Add wrappers that are atomic only in free-threaded builds (#115046)
These are intended to be used in places where atomics are required in
free-threaded builds but not in the default build. We don't want to
introduce the potential performance overhead of an atomic operation in the
default build.
2024-02-14 15:15:05 -05:00
Sam Gross 17773fcb86
gh-115441: Fix missing braces warning (#115460)
Removes `_py_object_state_INIT`. We want to initialize the `object_state` field to zero.
2024-02-14 12:27:39 -05:00
Donghee Na f15795c9a0
gh-111968: Rename freelist related struct names to Eric's suggestion (gh-115329) 2024-02-14 00:32:51 +00:00
Eric Snow 514b1c91b8
gh-76785: Improved Subinterpreters Compatibility with 3.12 (gh-115424)
For the most part, these changes make is substantially easier to backport subinterpreter-related code to 3.12, especially the related modules (e.g. _xxsubinterpreters). The main motivation is to support releasing a PyPI package with the 3.13 capabilities compiled for 3.12.

A lot of the changes here involve either hiding details behind macros/functions or splitting up some files.
2024-02-13 14:56:49 -07:00
Mark Shannon 681778c56a
GH-113710: Improve `_SET_IP` and `_CHECK_VALIDITY` (GH-115248) 2024-02-13 16:28:19 +00:00
Mark Shannon f9f6156c5a
GH-113710: Backedge counter improvements. (GH-115166) 2024-02-13 14:16:37 +00:00
Ken Jin 7cce857622
gh-114058: Foundations of the Tier2 redundancy eliminator (GH-115085)
---------

Co-authored-by: Mark Shannon <9448417+markshannon@users.noreply.github.com>
Co-authored-by: Jules <57632293+JuliaPoo@users.noreply.github.com>
Co-authored-by: Guido van Rossum <gvanrossum@users.noreply.github.com>
2024-02-13 21:24:48 +08:00
Steve Dower ea25f32d5f
gh-89240: Enable multiprocessing on Windows to use large process pools (GH-107873)
We add _winapi.BatchedWaitForMultipleObjects to wait for larger numbers of handles.
This is an internal module, hence undocumented, and should be used with caution.
Check the docstring for info before using BatchedWaitForMultipleObjects.
2024-02-13 00:28:35 +00:00
mpage de7d67b19b
gh-114271: Make `PyInterpreterState.threads.count` thread-safe in free-threaded builds (gh-115093)
Use atomics to mutate PyInterpreterState.threads.count.
2024-02-12 10:44:00 -07:00
Petr Viktorin 879f4546bf
gh-110850: Add PyTime_t C API (GH-115215)
* gh-110850: Add PyTime_t C API

Add PyTime_t API:

* PyTime_t type.
* PyTime_MIN and PyTime_MAX constants.
* PyTime_AsSecondsDouble(), PyTime_Monotonic(),
  PyTime_PerfCounter() and PyTime_GetSystemClock() functions.

Co-authored-by: Victor Stinner <vstinner@python.org>
2024-02-12 18:13:10 +01:00
Mark Shannon 8144661017
GH-113710: Fix updating of dict version tag and add watched dict stats (GH-115221) 2024-02-12 16:07:38 +00:00
Donghee Na d4d5bae147
gh-111968: Refactor _PyXXX_Fini to integrate with _PyObject_ClearFreeLists (gh-114899) 2024-02-10 00:57:04 +00:00
Sam Gross a3af3cb4f4
gh-110481: Implement inter-thread queue for biased reference counting (#114824)
Biased reference counting maintains two refcount fields in each object:
`ob_ref_local` and `ob_ref_shared`. The true refcount is the sum of these two
fields. In some cases, when refcounting operations are split across threads,
the ob_ref_shared field can be negative (although the total refcount must be
at least zero). In this case, the thread that decremented the refcount
requests that the owning thread give up ownership and merge the refcount
fields.
2024-02-09 17:08:32 -05:00
Carl Meyer 8f0998e844
gh-114828: parenthesize non-atomic macro definitions in pycore_symtable.h (#115143) 2024-02-07 13:19:47 -07:00
Mark Shannon 8a3c499ffe
GH-108362: Revert "GH-108362: Incremental GC implementation (GH-108038)" (#115132)
Revert "GH-108362: Incremental GC implementation (GH-108038)"

This reverts commit 36518e69d7.
2024-02-07 12:38:34 +00:00
Dino Viehland 92abb01240
gh-112075: Add critical sections for most dict APIs (#114508)
Starts adding thread safety to dict objects.


Use @critical_section for APIs which are exposed via argument clinic and don't directly correlate with a public C API which needs to acquire the lock
Use a _lock_held suffix for keeping changes to complicated functions simple and just wrapping them with a critical section
Acquire and release the lock in an existing function where it won't be overly disruptive to the existing logic
2024-02-06 14:03:43 -08:00
Sam Gross b6228b521b
gh-115035: Mark ThreadHandles as non-joinable earlier after forking (#115042)
This marks dead ThreadHandles as non-joinable earlier in
`PyOS_AfterFork_Child()` before we execute any Python code. The handles
are stored in a global linked list in `_PyRuntimeState` because `fork()`
affects the entire process.
2024-02-06 14:45:04 -05:00
Mariusz Felisiak 1a10437a14
gh-91602: Add iterdump() support for filtering database objects (#114501)
Add optional 'filter' parameter to iterdump() that allows a "LIKE"
pattern for filtering database objects to dump.

Co-authored-by: Erlend E. Aasland <erlend@python.org>
2024-02-06 12:34:56 +01:00
Dino Viehland bcccf1fb63
gh-112075: Add gc shared bits (#114931)
Add GC shared flags for objects to the GC bit states in free-threaded builds
2024-02-05 10:35:59 -08:00
Mark Shannon 36518e69d7
GH-108362: Incremental GC implementation (GH-108038) 2024-02-05 18:28:51 +00:00
Andrew Rogers b3f0b698da
gh-104530: Enable native Win32 condition variables by default (GH-104531) 2024-02-02 13:50:51 +00:00
Mark Shannon 0e71a295e9
GH-113710: Add a "globals to constants" pass (GH-114592)
Converts specializations of `LOAD_GLOBAL` into constants during tier 2 optimization.
2024-02-02 12:14:34 +00:00
Donghee Na 13907968d7
gh-111968: Use per-thread freelists for dict in free-threading (gh-114323) 2024-02-01 20:53:53 +00:00
Sam Gross 587d480203
gh-112529: Remove PyGC_Head from object pre-header in free-threaded build (#114564)
* gh-112529: Remove PyGC_Head from object pre-header in free-threaded build

This avoids allocating space for PyGC_Head in the free-threaded build.
The GC implementation for free-threaded CPython does not use the
PyGC_Head structure.

 * The trashcan mechanism uses the `ob_tid` field instead of `_gc_prev`
   in the free-threaded build.
 * The GDB libpython.py file now determines the offset of the managed
   dict field based on whether the running process is a free-threaded
   build. Those are identified by the `ob_ref_local` field in PyObject.
 * Fixes `_PySys_GetSizeOf()` which incorrectly incorrectly included the
   size of `PyGC_Head` in the size of static `PyTypeObject`.
2024-02-01 12:29:19 -08:00
He Weidong 97cc58f977
Fix comment in pycore_runtime.h (GH-110540) 2024-02-01 19:27:53 +00:00
Donghee Na 7b9d406729
gh-112087: Make PyList_{Append,Size,GetSlice} to be thread-safe (gh-114651) 2024-02-01 08:58:08 +09:00
Eugene Toder 1f515e8a10
gh-112919: Speed-up datetime, date and time.replace() (GH-112921)
Use argument clinic and call new_* functions directly. This speeds up
these functions 6x to 7.5x when calling with keyword arguments.
2024-01-30 15:19:46 +00:00
Dino Viehland 0cd9bacb8a
gh-112075: Dictionary global version counter should use atomic increments (#114568)
Dictionary global version counter should use atomic increments
2024-01-29 09:47:54 -08:00
mpage c87233fd3f
gh-112050: Adapt collections.deque to Argument Clinic (#113963) 2024-01-29 15:08:23 +00:00
Brandt Bucher f6d9e5926b
GH-113464: Add a JIT backend for tier 2 (GH-113465)
Add an option (--enable-experimental-jit for configure-based builds
or --experimental-jit for PCbuild-based ones) to build an
*experimental* just-in-time compiler, based on copy-and-patch (https://fredrikbk.com/publications/copy-and-patch.pdf).

See Tools/jit/README.md for more information on how to install the required build-time tooling.
2024-01-28 18:48:48 -08:00
Neil Schemenauer 7a7bce5a0a
gh-113055: Use pointer for interp->obmalloc state (gh-113412)
For interpreters that share state with the main interpreter, this points
to the same static memory structure.  For interpreters with their own
obmalloc state, it is heap allocated.  Add free_obmalloc_arenas() which
will free the obmalloc arenas and radix tree structures for interpreters
with their own obmalloc state.

Co-authored-by: Eric Snow <ericsnowcurrently@gmail.com>
2024-01-26 19:38:14 -08:00
Donghee Na 699779256e
gh-111968: Unify freelist naming schema to Eric's suggestion (gh-114581) 2024-01-27 00:25:16 +09:00
Sam Gross b52fc70d1a
gh-112529: Implement GC for free-threaded builds (#114262)
* gh-112529: Implement GC for free-threaded builds

This implements a mark and sweep GC for the free-threaded builds of
CPython. The implementation relies on mimalloc to find GC tracked
objects (i.e., "containers").
2024-01-25 10:27:36 -08:00
Dino Viehland 4850410b60
gh-112075: Add try-incref functions from nogil branch for use in dict thread safety (#114512)
* Bring in a subset of biased reference counting:
https://github.com/colesbury/nogil/commit/b6b12a9a94e

The NoGIL branch has functions for attempting to do an incref on an object which may or may not be in flight. This just brings those functions over so that they will be usable from in the dict implementation to get items w/o holding a lock.

There's a handful of small simple modifications:

    Adding inline to the force inline functions to avoid a warning, and switching from _Py_ALWAYS_INLINE to Py_ALWAYS_INLINE as that's available
    Remove _Py_REF_LOCAL_SHIFT as it doesn't exist yet (and is currently 0 in the 3.12 nogil branch anyway)
    ob_ref_shared is currently Py_ssize_t and not uint32_t, so use that
    _PY_LIKELY doesn't exist, so drop it
    _Py_ThreadLocal becomes _Py_IsOwnedByCurrentThread
    Add '_PyInterpreterState_GET()' to _Py_IncRefTotal calls.


Co-Authored-By: Sam Gross <colesbury@gmail.com>
2024-01-25 09:34:03 -08:00
Michael Droettboom ea3cd0498c
gh-114312: Collect stats for unlikely events (GH-114493) 2024-01-25 11:10:51 +00:00
Mark Shannon 981d172f7f
GH-112354: `END_FOR` instruction to only pop one value. (GH-114247)
* Compiler emits END_FOR; POP_TOP instead of END_FOR. To support tier 2 side exits in loops.
2024-01-24 15:10:17 +00:00
Mark Shannon 384429d1c0
GH-113710: Add a tier 2 peephole optimization pass. (GH-114487)
* Convert _LOAD_CONST to inline versions

* Remove PEP 523 checks
2024-01-24 12:08:31 +00:00
Sam Gross 441affc9e7
gh-111964: Implement stop-the-world pauses (gh-112471)
The `--disable-gil` builds occasionally need to pause all but one thread.  Some
examples include:

* Cyclic garbage collection, where this is often called a "stop the world event"
* Before calling `fork()`, to ensure a consistent state for internal data structures
* During interpreter shutdown, to ensure that daemon threads aren't accessing Python objects

This adds the following functions to implement global and per-interpreter pauses:

* `_PyEval_StopTheWorldAll()` and `_PyEval_StartTheWorldAll()` (for the global runtime)
* `_PyEval_StopTheWorld()` and `_PyEval_StartTheWorld()` (per-interpreter)

(The function names may change.)

These functions are no-ops outside of the `--disable-gil` build.
2024-01-23 11:08:23 -07:00
Sam Gross 412920a41e
gh-112532: Improve mimalloc page visiting (#114133)
This adds support for visiting abandoned pages in mimalloc and improves
the performance of the page visiting code. Abandoned pages contain
memory blocks from threads that have exited. At some point, they may be
later reclaimed by other threads. We still need to visit those pages in
the free-threaded GC because they contain live objects.

This also reduces the overhead of visiting mimalloc pages:

 * Special cases for full, empty, and pages containing only a single
   block.
 * Fix free_map to use one bit instead of one byte per block.
 * Use fast integer division by a constant algorithm when computing
   block offset from block size and index.
2024-01-22 13:10:21 -08:00
Sam Gross 1d6d5e854c
gh-112529: Use GC heaps for GC allocations in free-threaded builds (gh-114157)
* gh-112529: Use GC heaps for GC allocations in free-threaded builds

The free-threaded build's garbage collector implementation will need to
find GC objects by traversing mimalloc heaps. This hooks up the
allocation calls with the correct heaps by using a thread-local
"current_obj_heap" variable.

* Refactor out setting heap based on type
2024-01-21 01:14:45 +09:00
Donghee Na 7fa511ba57
gh-111968: Use per-thread freelists for generator in free-threading (gh-114189) 2024-01-18 18:15:00 +00:00
Sam Gross b331381485
gh-112529: Track if debug allocator is used as underlying allocator (#113747)
* gh-112529: Track if debug allocator is used as underlying allocator

The GC implementation for free-threaded builds will need to accurately
detect if the debug allocator is used because it affects the offset of
the Python object from the beginning of the memory allocation. The
current implementation of `_PyMem_DebugEnabled` only considers if the
debug allocator is the outer-most allocator; it doesn't handle the case
of "hooks" like tracemalloc being used on top of the debug allocator.

This change enables more accurate detection of the debug allocator by
tracking when debug hooks are enabled.

* Simplify _PyMem_DebugEnabled
2024-01-16 13:42:15 -08:00
Donghee Na 867f59f234
gh-111968: Use per-thread freelists for PyContext in free-threading (gh-114122) 2024-01-16 16:14:56 +00:00
Serhiy Storchaka d2d8332f71
gh-113626: Add allow_code parameter in marshal functions (GH-113648)
Passing allow_code=False prevents serialization and de-serialization of
code objects which is incompatible between Python versions.
2024-01-16 18:05:15 +02:00
Donghee Na 3eae76554b
gh-111968: Use per-thread slice_cache in free-threading (gh-113972) 2024-01-16 00:38:57 +09:00
Mark Shannon ac10947ba7
GH-112354: `_GUARD_IS_TRUE_POP` side-exits to target the next instruction, not themselves. (GH-114078) 2024-01-15 11:41:06 +00:00
Joseph Pearson 9a71750a29
Fix a grammatical error in `pycore_pymem.h` (#112993) 2024-01-12 22:25:52 +00:00
Ken Jin ac92527c08
gh-113710: Add types to the interpreter DSL (#113711)
Co-authored-by: Jules <57632293+JuliaPoo@users.noreply.github.com>
Co-authored-by: blurb-it[bot] <43283697+blurb-it[bot]@users.noreply.github.com>
2024-01-13 01:30:27 +08:00
Brandt Bucher 30e6cbdba2
GH-113860: Get rid of `_PyUOpExecutorObject` (GH-113954) 2024-01-12 11:58:23 +00:00
Donghee Na 2e7577b622
gh-111968: Use per-thread freelists for tuple in free-threading (gh-113921) 2024-01-12 03:46:28 +09:00
Nikita Sobolev 2ac4cf4743
gh-112640: Add `kwdefaults` parameter to `types.FunctionType.__new__` (#112641) 2024-01-11 00:42:30 -08:00
Donghee Na c65ae26f2b
gh-111968: Unify naming scheme for freelist (gh-113919) 2024-01-11 08:51:51 +09:00
Sam Gross 73ae2023a7
gh-113753: Clear finalized bit when putting PyAsyncGenASend back into free list (#113754) 2024-01-10 10:18:38 -08:00
Donghee Na f728f7242c
gh-111968: Use per-thread freelists for float in free-threading (gh-113886) 2024-01-10 15:47:13 +00:00
Mark Shannon a0c9cf9456
GH-113860: All executors are now defined in terms of micro ops. Convert counter executor to use uops. (GH-113864) 2024-01-10 15:44:34 +00:00
Donghee Na 57bdc6c30d
gh-111968: Introduce _PyFreeListState and _PyFreeListState_GET API (gh-113584) 2024-01-10 08:04:41 +09:00
Pablo Galindo Salgado a03ec20bcd
gh-110721: Remove unused code from suggestions.c after moving PyErr_Display to use the traceback module (#113712) 2024-01-08 15:10:45 +00:00
Sam Gross 99854ce170
gh-113688: Split up gcmodule.c (gh-113715)
This splits part of Modules/gcmodule.c of into Python/gc.c, which
now contains the core garbage collection implementation. The Python
module remain in the Modules/gcmodule.c file.
2024-01-05 12:17:16 -08:00
Sam Gross 0b7476080b
gh-112532: Tag mimalloc heaps and pages (#113742)
* gh-112532: Tag mimalloc heaps and pages

Mimalloc pages are data structures that contain contiguous allocations
of the same block size. Note that they are distinct from operating
system pages. Mimalloc pages are contained in segments.

When a thread exits, it abandons any segments and contained pages that
have live allocations. These segments and pages may be later reclaimed
by another thread. To support GC and certain thread-safety guarantees in
free-threaded builds, we want pages to only be reclaimed by the
corresponding heap in the claimant thread. For example, we want pages
containing GC objects to only be claimed by GC heaps.

This allows heaps and pages to be tagged with an integer tag that is
used to ensure that abandoned pages are only claimed by heaps with the
same tag. Heaps can be initialized with a tag (0-15); any page allocated
by that heap copies the corresponding tag.

* Fix conversion warning
2024-01-05 12:08:50 -08:00
Sam Gross fcb3c2a444
gh-112532: Isolate abandoned segments by interpreter (#113717)
* gh-112532: Isolate abandoned segments by interpreter

Mimalloc segments are data structures that contain memory allocations along
with metadata. Each segment is "owned" by a thread. When a thread exits,
it abandons its segments to a global pool to be later reclaimed by other
threads. This changes the pool to be per-interpreter instead of process-wide.

This will be important for when we use mimalloc to find GC objects in the
`--disable-gil` builds. We want heaps to only store Python objects from a
single interpreter. Absent this change, the abandoning and reclaiming process
could break this isolation.

* Add missing '&_mi_abandoned_default' to 'tld_empty'
2024-01-04 22:21:40 +00:00
Donghee Na 0c3455a969
gh-111926: Set up basic sementics of weakref API for freethreading (gh-113621)
---------

Co-authored-by: Sam Gross <colesbury@gmail.com>
2024-01-03 13:25:27 +00:00
Sam Gross acf3bcc886
gh-112532: Use separate mimalloc heaps for GC objects (gh-113263)
* gh-112532: Use separate mimalloc heaps for GC objects

In `--disable-gil` builds, we now use four separate heaps in
anticipation of using mimalloc to find GC objects when the GIL is
disabled. To support this, we also make a few changes to mimalloc:

* `mi_heap_t` and `mi_tld_t` initialization is split from allocation.
  This allows us to have a `mi_tld_t` per-`PyThreadState`, which is
  important to keep interpreter isolation, since the same OS thread may
  run in multiple interpreters (using different PyThreadStates.)

* Heap abandoning (mi_heap_collect_ex) can now be called from a
  different thread than the one that created the heap. This is necessary
  because we may clear and delete the containing PyThreadStates from a
  different thread during finalization and after fork().

* Use enum instead of defines and guard mimalloc includes.

* The enum typedef will be convenient for future PRs that use the type.
* Guarding the mimalloc includes allows us to unconditionally include
  pycore_mimalloc.h from other header files that rely on things like
  `struct _mimalloc_thread_state`.

* Only define _mimalloc_thread_state in Py_GIL_DISABLED builds
2023-12-27 01:53:20 +09:00
Yilei Yang 48c49739f5
gh-106905: Use separate structs to track recursion depth in each PyAST_mod2obj call. (GH-113035)
Co-authored-by: Gregory P. Smith [Google LLC] <greg@krypto.org>
2023-12-25 19:36:59 +02:00
Mark Shannon 723f4d6698
GH-111485: Delete the old generator code. (GH-113321) 2023-12-21 12:46:28 +00:00
Mark Shannon e96f26083b
GH-111485: Generate instruction and uop metadata (GH-113287) 2023-12-20 14:27:25 +00:00
Christopher Chavez a545a86ec6
gh-111178: Make slot functions in typeobject.c have compatible types (GH-112752) 2023-12-20 15:13:44 +01:00
Sam Gross 5ae75e1be2
gh-111964: Add _PyRWMutex a "readers-writer" lock (gh-112859)
This adds `_PyRWMutex`, a "readers-writer" lock, which wil be used to
serialize global stop-the-world pauses with per-interpreter pauses.
2023-12-15 18:56:55 -07:00
Mark Shannon e24eccbc1c
GH-111485: Sort metadata tables for easier checking of future diffs (GH-113101) 2023-12-14 16:41:52 +00:00
Mark Shannon 6873555955
GH-112354: Treat _EXIT_TRACE like an unconditional side exit (GH-113104) 2023-12-14 14:26:44 +00:00
Eric Snow c6e614fd81
gh-76785: Avoid Pickled TracebackException for Propagated Subinterpreter Exceptions (gh-113036)
We need the TracebackException of uncaught exceptions for a single purpose: the error display.  Thus we only need to pass the formatted error display between interpreters.  Passing a pickled TracebackException is overkill.
2023-12-13 00:31:30 +00:00
Sam Gross a3c031884d
gh-112723: Call `PyThreadState_Clear()` from the correct interpreter (#112776)
The `PyThreadState_Clear()` function must only be called with the GIL
held and must be called from the same interpreter as the passed in
thread state. Otherwise, any Python objects on the thread state may be
destroyed using the wrong interpreter, leading to memory corruption.

This is also important for `Py_GIL_DISABLED` builds because free lists
will be associated with PyThreadStates and cleared in
`PyThreadState_Clear()`.

This fixes two places that called `PyThreadState_Clear()` from the wrong
interpreter and adds an assertion to `PyThreadState_Clear()`.
2023-12-12 17:20:21 -07:00
Eric Snow 8a4c1f3ff1
gh-76785: Show the Traceback for Uncaught Subinterpreter Exceptions (gh-113034)
When an exception is uncaught in Interpreter.exec_sync(), it helps to show that exception's error display if uncaught in the calling interpreter.  We do so here by generating a TracebackException in the subinterpreter and passing it between interpreters using pickle.
2023-12-13 00:00:54 +00:00
Mark Shannon 956023826a
GH-108866: Guarantee forward progress in executors. (GH-113006) 2023-12-12 19:02:24 +00:00
Eric Snow 86a77f4e1a
gh-76785: Fixes for test.support.interpreters (gh-112982)
This involves a number of changes for PEP 734.
2023-12-12 08:24:31 -07:00
Mark Shannon 0c55f27060
GH-111485: Factor out tier 2 code generation from the rest of the interpreter code generator (GH-112968) 2023-12-12 12:12:17 +00:00
Sam Gross fdee7b7b3e
gh-112532: Require mimalloc in `--disable-gil` builds (gh-112883) 2023-12-12 09:04:48 +09:00
Mark Shannon c27e9d5d17
GH-111485: Factor out generation of uop IDs from cases generator. (GH-112877) 2023-12-11 14:14:36 +00:00
Neil Schemenauer 890ce430d9
gh-112867: fix for WITH_PYMALLOC_RADIX_TREE=0 (GH-112885)
The _obmalloc_usage structure is only defined if the obmalloc radix tree
is enabled.
2023-12-09 13:50:48 -08:00
Sam Gross cf6110ba13
gh-111924: Use PyMutex for Runtime-global Locks. (gh-112207)
This replaces some usages of PyThread_type_lock with PyMutex, which does not require memory allocation to initialize.

This simplifies some of the runtime initialization and is also one step towards avoiding changing the default raw memory allocator during initialize/finalization, which can be non-thread-safe in some circumstances.
2023-12-07 12:33:40 -07:00
Sam Gross db460735af
gh-112538: Add internal-only _PyThreadStateImpl "wrapper" for PyThreadState (gh-112560)
Every PyThreadState instance is now actually a _PyThreadStateImpl.
It is safe to cast from `PyThreadState*` to `_PyThreadStateImpl*` and back.
The _PyThreadStateImpl will contain fields that we do not want to expose
in the public C API.
2023-12-07 12:11:45 -07:00
Sam Gross 2d76be251d
gh-111962: Make dtoa thread-safe in `--disable-gil` builds. (#112049)
This updates `dtoa.c` to avoid using the Bigint free-list in --disable-gil builds and
to pre-computes the needed powers of 5 during interpreter initialization.

* gh-111962: Make dtoa thread-safe in `--disable-gil` builds.

This avoids using the Bigint free-list in `--disable-gil` builds
and pre-computes the needed powers of 5 during interpreter initialization.

* Fix size of cached powers of 5 array.

We need the powers of 5 up to 5**512 because we only jump straight to
underflow when the exponent is less than -512 (or larger than 308).

* Rename Py_NOGIL to Py_GIL_DISABLED

* Changes from review

* Fix assertion placement
2023-12-07 13:47:55 +00:00
andrewluotechnologies 9c3458e058
gh-112125: Fix None.__ne__(None) returning NotImplemented instead of False (#112504) 2023-12-07 13:56:01 +01:00
Mark Shannon b449415b2f
GH-111485: Separate out parsing, analysis and code-gen phases of tier 1 code generator (GH-112299) 2023-12-07 12:49:40 +00:00
Victor Stinner 828451dfde
gh-111545: Add Py_HashPointer() function (#112096)
* Implement _Py_HashPointerRaw() as a static inline function.
* Add Py_HashPointer() tests to test_capi.test_hash.
* Keep _Py_HashPointer() function as an alias to Py_HashPointer().
2023-12-06 15:09:22 +01:00
Victor Stinner a74902a14c
gh-106550: Fix sign conversion in pycore_code.h (#112613)
Fix sign conversion in pycore_code.h: use unsigned integers and cast
explicitly when needed.
2023-12-04 11:42:58 +01:00
Victor Stinner 5c5022b862
gh-112567: Add _PyTimeFraction C API (#112568)
Use a fraction internally in the _PyTime API to reduce the risk of
integer overflow: simplify the fraction using Greatest Common
Divisor (GCD). The fraction API is used by time functions:
perf_counter(), monotonic() and process_time().

For example, QueryPerformanceFrequency() usually returns 10 MHz on
Windows 10 and newer. The fraction SEC_TO_NS / frequency =
1_000_000_000 / 10_000_000 can be simplified to 100 / 1.

* Add _PyTimeFraction type.
* Add functions:

  * _PyTimeFraction_Set()
  * _PyTimeFraction_Mul()
  * _PyTimeFraction_Resolution()

* No longer check "numer * denom <= _PyTime_MAX" in
  _PyTimeFraction_Set(). _PyTimeFraction_Mul() uses _PyTime_Mul()
  which handles integer overflow.
2023-12-01 19:50:10 +01:00
Victor Stinner 05a370abd6
gh-112567: Add _Py_GetTicksPerSecond() function (#112587)
* Move _PyRuntimeState.time to _posixstate.ticks_per_second and
  time_module_state.ticks_per_second.
* Add time_module_state.clocks_per_second.
* Rename _PyTime_GetClockWithInfo() to py_clock().
* Rename _PyTime_GetProcessTimeWithInfo() to py_process_time().
* Add process_time_times() helper function, called by
  py_process_time().
* os.times() is now always built: no longer rely on HAVE_TIMES.
2023-12-01 17:05:56 +01:00
Pablo Galindo Salgado a73aa48e6b
gh-112367: Only free perf trampoline arenas at shutdown (#112368)
Signed-off-by: Pablo Galindo <pablogsal@gmail.com>
2023-12-01 13:20:51 +00:00
Irit Katriel 07ebd46f9e
gh-112519: Make it possible to specify instruction flags for pseudo instructions in bytecodes.c (#112520) 2023-11-30 11:03:30 +00:00
Kirill Podoprigora 0785c68559
gh-111972: Make Unicode name C APIcapsule initialization thread-safe (#112249) 2023-11-30 11:12:49 +01:00
Guido van Rossum e723700190
Rename ...Uop... to ...UOp... (uppercase O) for consistency (#112327)
* Rename _PyUopExecute to _PyUOpExecute (uppercase O) for consistency
* Also rename _PyUopName and _PyUOp_Replacements, and some output strings
2023-11-28 17:10:11 -08:00
Grant Ramsay e954ac7205
gh-63284: Add support for TLS-PSK (pre-shared key) to the ssl module (#103181)
Add support for TLS-PSK (pre-shared key) to the ssl module.

---------

Co-authored-by: Oleg Iarygin <oleg@arhadthedev.net>
Co-authored-by: Gregory P. Smith <greg@krypto.org>
2023-11-27 04:01:44 +00:00
Eric Snow 9e56eedd01
gh-76785: Return an "excinfo" Object From Interpreter.run() (gh-111573) 2023-11-23 00:55:00 +00:00
Eric Snow 790db85c77
gh-76785: Add _PyType_GetModuleName() to the Internal C-API (gh-112323)
The new function corresponds to the existing (public) PyType_GetName() and PyType_GetQualName().
2023-11-22 15:03:33 -07:00
Guido van Rossum 8deb8bc2e5
gh-112287: Speed up Tier 2 (uop) interpreter a little (#112286)
This makes the Tier 2 interpreter a little faster.
I calculated by about 3%,
though I hesitate to claim an exact number.

This starts by doubling the trace size limit (to 512),
making it more likely that loops fit in a trace.

The rest of the approach is to only load
`oparg` and `operand` in cases that use them.
The code generator know when these are used.

For `oparg`, it will conditionally emit
```
oparg = CURRENT_OPARG();
```
at the top of the case block.
(The `oparg` variable may be referenced multiple times
by the instructions code block, so it must be in a variable.)

For `operand`, it will use `CURRENT_OPERAND()` directly
instead of referencing the `operand` variable,
which no longer exists.
(There is only one place where this will be used.)
2023-11-20 11:25:32 -08:00
Guido van Rossum 1995955173
gh-106529: Make FOR_ITER a viable uop (#112134)
This uses the new mechanism whereby certain uops
are replaced by others during translation,
using the `_PyUop_Replacements` table.
We further special-case the `_FOR_ITER_TIER_TWO` uop
to update the deoptimization target to point
just past the corresponding `END_FOR` opcode.

Two tiny code cleanups are also part of this PR.
2023-11-20 10:08:53 -08:00
Hugo van Kemenade 3b3ec0d77f
gh-111863: Rename `Py_NOGIL` to `Py_GIL_DISABLED` (#111864)
Rename Py_NOGIL to Py_GIL_DISABLED
2023-11-20 15:52:00 +02:00
Donghee Na 7c9f2677fb
gh-111926: Update _PyWeakref_IS_DEAD to be thread-safe (gh-112267) 2023-11-20 07:36:45 +09:00
Guido van Rossum be0bd54c6b
gh-106529: Cleanups split off gh-112134 (#112214)
- Double max trace size to 256
- Add a dependency on executor_cases.c.h for ceval.o
- Mark `_SPECIALIZE_UNPACK_SEQUENCE` as `TIER_ONE_ONLY`
- Add debug output back showing the optimized trace
- Bunch of cleanups to Tools/cases_generator/
2023-11-17 11:49:42 -08:00
Sam Gross 446f18a911
gh-111956: Add thread-safe one-time initialization. (gh-111960) 2023-11-16 12:19:54 -07:00
Victor Stinner 58469244be
gh-112026: Restore removed private C API (#112115)
Restore removed private C API functions, macros and structures which
have no simple replacement for now:

* _PyDict_GetItem_KnownHash()
* _PyDict_NewPresized()
* _PyHASH_BITS
* _PyHASH_IMAG
* _PyHASH_INF
* _PyHASH_MODULUS
* _PyHASH_MULTIPLIER
* _PyLong_Copy()
* _PyLong_FromDigits()
* _PyLong_New()
* _PyLong_Sign()
* _PyObject_CallMethodId()
* _PyObject_CallMethodNoArgs()
* _PyObject_CallMethodOneArg()
* _PyObject_CallOneArg()
* _PyObject_EXTRA_INIT
* _PyObject_FastCallDict()
* _PyObject_GetAttrId()
* _PyObject_Vectorcall()
* _PyObject_VectorcallMethod()
* _PyStack_AsDict()
* _PyThread_CurrentFrames()
* _PyUnicodeWriter structure
* _PyUnicodeWriter_Dealloc()
* _PyUnicodeWriter_Finish()
* _PyUnicodeWriter_Init()
* _PyUnicodeWriter_Prepare()
* _PyUnicodeWriter_PrepareKind()
* _PyUnicodeWriter_WriteASCIIString()
* _PyUnicodeWriter_WriteChar()
* _PyUnicodeWriter_WriteLatin1String()
* _PyUnicodeWriter_WriteStr()
* _PyUnicodeWriter_WriteSubstring()
* _PyUnicode_AsString()
* _PyUnicode_FromId()
* _PyVectorcall_Function()
* _Py_HashDouble()
* _Py_HashPointer()
* _Py_IDENTIFIER()
* _Py_c_abs()
* _Py_c_diff()
* _Py_c_neg()
* _Py_c_pow()
* _Py_c_prod()
* _Py_c_quot()
* _Py_c_sum()
* _Py_static_string()
* _Py_static_string_init()
2023-11-15 16:38:31 +00:00
Mark Shannon 4bbb367ba6
GH-111848: Set the IP when de-optimizing (GH-112065)
* Replace jumps with deopts in tier 2

* Fewer special cases of uop names

* Add target field to uop IR

* Remove more redundant SET_IP and _CHECK_VALIDITY micro-ops

* Extend whitelist of non-escaping API functions.
2023-11-15 15:48:58 +00:00
Mark Shannon a519b87958
GH-111848: Convert remaining jumps to deopts into tier 2 code. (GH-112045) 2023-11-14 15:30:33 +00:00
Victor Stinner 4f04172c92
gh-111262: Add PyDict_Pop() function (#112028)
_PyDict_Pop_KnownHash(): remove the default value and the return type
becomes an int.

Co-authored-by: Stefan Behnel <stefan_ml@behnel.de>
Co-authored-by: Antoine Pitrou <pitrou@free.fr>
2023-11-14 12:51:00 +00:00
Irit Katriel 36aab34fab
gh-107149: make new opcode util functions private rather than public and unstable (#112042) 2023-11-14 00:31:02 +00:00
Serhiy Storchaka 771bd3c94a
Add private _PyUnicode_AsUTF8NoNUL() function (GH-111957)
Like PyUnicode_AsUTF8(), but check for embedded null characters.
2023-11-10 21:31:36 +02:00
Mark Shannon 25c4956488
GH-109369: Exit tier 2 if executor is invalid (GH-111657) 2023-11-09 11:19:51 +00:00
Irit Katriel 30ec968bef
gh-111354: remove comparisons with enum values, variable reuse, unused imports in genobject.c (#111708) 2023-11-09 10:27:20 +00:00
Sam Gross 31c90d5838
gh-111569: Implement Python critical section API (gh-111571)
Critical sections are helpers to replace the global interpreter lock
with finer grained locking.  They provide similar guarantees to the GIL
and avoid the deadlock risk that plain locking involves.  Critical
sections are implicitly ended whenever the GIL would be released.  They
are resumed when the GIL would be acquired.  Nested critical sections
behave as if the sections were interleaved.
2023-11-08 15:39:29 -07:00
Mark Shannon 06efb60264
GH-111848: Tidy up tier 2 handling of FOR_ITER specialization by using DEOPT_IF instead of jumps. (GH-111849) 2023-11-08 13:31:55 +00:00
Mark Shannon 931f4438c9
GH-111485: Allow arbitrary annotations on instructions and micro-ops. (GH-111697) 2023-11-07 09:42:39 +00:00
Brandt Bucher 3e99c9cbf6
GH-111485: Make BEFORE_WITH a uop (GH-111812) 2023-11-06 16:42:49 -08:00
Eric Snow d4426e8d00
gh-76785: Move _Py_excinfo Functions Out of the Internal C-API (gh-111715)
I added _Py_excinfo to the internal API (and added its functions in Python/errors.c) in gh-111530 (9322ce9).  Since then I've had a nagging sense that I should have added the type and functions in its own PR.  While I do plan on using _Py_excinfo outside crossinterp.c very soon (see gh-111572/gh-111573), I'd still feel more comfortable if the _Py_excinfo stuff went in as its own PR.  Hence, here we are.

(FWIW, I may combine that with gh-111572, which I may, in turn, combine with gh-111573.  We'll see.)
2023-11-06 11:09:22 -07:00
Antoine Pitrou 0e9c364f4a
GH-110829: Ensure Thread.join() joins the OS thread (#110848)
Joining a thread now ensures the underlying OS thread has exited. This is required for safer fork() in multi-threaded processes.

---------

Co-authored-by: blurb-it[bot] <43283697+blurb-it[bot]@users.noreply.github.com>
2023-11-04 13:59:24 +00:00
Tian Gao e0afed7e27
gh-103615: Use local events for opcode tracing (GH-109472)
* Use local monitoring for opcode trace

* Remove f_opcode_trace_set

* Add test for setting f_trace_opcodes after settrace
2023-11-03 16:39:50 +00:00
Michael Droettboom 2bc01cc0c7
gh-111652: Fix --enable-pystats build (GH-111653) 2023-11-03 15:21:16 +00:00
scoder 24ddaee5ca
gh-106168: Revert the "size before item" setting (#111683)
gh-106168: Update the size only after setting the item, to avoid temporary inconsistencies.
Also remove the "what's new" sentence regarding the size setting since tuples cannot grow after allocation.
2023-11-03 11:02:39 +00:00
Irit Katriel d49aba5a7a
gh-111354: Simplify _PyGen_yf by moving some of its work to the compiler and frame state (#111648) 2023-11-03 10:01:36 +00:00
Serhiy Storchaka 26c0e5e03a
gh-108082: Remove _PyErr_WriteUnraisableMsg() (GH-111643)
Replace the remaining calls with PyErr_FormatUnraisable().
2023-11-03 09:45:53 +02:00
Irit Katriel 52cc4af6ae
gh-111354: simplify detection of RESUME after YIELD_VALUE at except-depth 1 (#111459) 2023-11-02 10:18:43 +00:00
Eric Snow 9322ce90ac
gh-76785: Crossinterp utils additions (gh-111530)
This moves several general internal APIs out of _xxsubinterpretersmodule.c and into the new Python/crossinterp.c (and the corresponding internal headers).

Specifically:

* _Py_excinfo, etc.:  the initial implementation for non-object exception snapshots (in pycore_pyerrors.h and Python/errors.c)
* _PyXI_exception_info, etc.:  helpers for passing an exception beween interpreters (wraps _Py_excinfo)
* _PyXI_namespace, etc.:  helpers for copying a dict of attrs between interpreters
* _PyXI_Enter(), _PyXI_Exit():  functions that abstract out the transitions between one interpreter and a second that will do some work temporarily

Again, these were all abstracted out of _xxsubinterpretersmodule.c as generalizations.  I plan on proposing these as public API at some point.
2023-11-01 17:36:40 -06:00
Mark Shannon b14e882428
GH-111485: Use micro-ops to split specialization code from base action (GH-111561) 2023-11-01 10:53:27 +00:00
Mark Shannon 2904d99839
GH-111485: Remove some special cases from the code generator and bytecodes.c (GH-111540) 2023-10-31 13:21:07 +00:00
Mark Shannon d27acd4461
GH-111485: Increment `next_instr` consistently at the start of the instruction. (GH-111486) 2023-10-31 10:09:54 +00:00
Michael Droettboom 84b4533e84
gh-109329: Count tier2 opcode misses (#110561)
This keeps a separate 'miss' counter for each micro-opcode, incremented whenever a guard uop takes a deoptimization side exit.
2023-10-30 17:02:45 -07:00
Eric Snow c6fe0869ab
gh-76785: Move the Cross-Interpreter Code to Its Own File (gh-111502)
This is partly to clear this stuff out of pystate.c, but also in preparation for moving some code out of _xxsubinterpretersmodule.c.  This change also moves this stuff to the internal API (new: Include/internal/pycore_crossinterp.h).  @vstinner did this previously and I undid it.  Now I'm re-doing it. :/
2023-10-30 16:53:10 -06:00
Victor Stinner 801741ff81
gh-90815: Fix mimalloc atomic.h on Windows arm64 (#111527)
mi_atomic_load_explicit() casts 'p' argument to drop the 'const'
qualifier on Windows arm64 platform. Fix the compiler warning:

    'function': different 'const' qualifiers
    (compiling source file ..\Objects\mimalloc\options.c)
2023-10-30 22:33:49 +00:00
Sam Gross 6dfb8fe023
gh-110481: Implement biased reference counting (gh-110764) 2023-10-30 16:06:09 +00:00
Dino Viehland 05f2f0ac92
gh-90815: Add mimalloc memory allocator (#109914)
* Add mimalloc v2.12

Modified src/alloc.c to remove include of alloc-override.c and not
compile new handler.

Did not include the following files:

 - include/mimalloc-new-delete.h
 - include/mimalloc-override.h
 - src/alloc-override-osx.c
 - src/alloc-override.c
 - src/static.c
 - src/region.c

mimalloc is thread safe and shares a single heap across all runtimes,
therefore finalization and getting global allocated blocks across all
runtimes is different.

* mimalloc: minimal changes for use in Python:

 - remove debug spam for freeing large allocations
 - use same bytes (0xDD) for freed allocations in CPython and mimalloc
   This is important for the test_capi debug memory tests

* Don't export mimalloc symbol in libpython.
* Enable mimalloc as Python allocator option.
* Add mimalloc MIT license.
* Log mimalloc in Lib/test/pythoninfo.py.
* Document new mimalloc support.
* Use macro defs for exports as done in:
  https://github.com/python/cpython/pull/31164/

Co-authored-by: Sam Gross <colesbury@gmail.com>
Co-authored-by: Christian Heimes <christian@python.org>
Co-authored-by: Victor Stinner <vstinner@python.org>
2023-10-30 15:43:11 +00:00
Savannah Ostrowski 4a929d432b
GH-111339: Fix initialization and finalization of static optimizer types (GH-111430) 2023-10-29 13:53:25 -07:00
gsallam 21f068d80c
gh-109587: Allow "precompiled" perf-trampolines to largely mitigate the cost of enabling perf-trampolines (#109666) 2023-10-27 03:57:29 +00:00
Irit Katriel a0c414c35d
gh-111354: define names for RESUME oparg values (#111365) 2023-10-26 16:30:18 +01:00
Irit Katriel 67a91f78e4
gh-109094: replace frame->prev_instr by frame->instr_ptr (#109095) 2023-10-26 13:43:10 +00:00
Pablo Galindo Salgado 90a1b2859f
gh-67224: Show source lines in tracebacks when using the -c option when running Python (#111200) 2023-10-26 15:17:28 +09:00
scoder a8a89fcd1f
gh-106320: Re-add some PyLong/PyDict C-API functions (GH-#111162)
* gh-106320: Re-add _PyLong_FromByteArray(), _PyLong_AsByteArray() and _PyLong_GCD() to the public header files since they are used by third-party packages and there is no efficient replacement.

See https://github.com/python/cpython/issues/111140
See https://github.com/python/cpython/issues/111139

* gh-111262: Re-add _PyDict_Pop() to have a C-API until a new public one is designed.
2023-10-25 11:33:48 +02:00
Brandt Bucher e5168ff3f8
GH-109214: _SET_IP before _PUSH_FRAME (but not _POP_FRAME) (GH-111001) 2023-10-24 13:27:42 -07:00
Radislav Chugunov 47d3e2ed93
gh-109894: Fix initialization of static `MemoryError` in subinterpreter (gh-110911)
Fixes #109894

* set `interp.static_objects.last_resort_memory_error.args` to empty tuple to avoid crash on `PyErr_Display()` call
* allow `_PyExc_InitGlobalObjects()` to be called on subinterpreter init

---------

Co-authored-by: blurb-it[bot] <43283697+blurb-it[bot]@users.noreply.github.com>
2023-10-23 17:06:59 -06:00
Mark Shannon 52e902ccf0
GH-109369: Add machinery for deoptimizing tier2 executors, both individually and globally. (GH-110384) 2023-10-23 14:49:09 +01:00
Eric Snow c58c63fdf6
gh-84570: Add Timeouts to SendChannel.send() and RecvChannel.recv() (gh-110567) 2023-10-17 23:05:49 +00:00
Eric Snow a53d7cb672
gh-84570: Send-Wait Fixes for _xxinterpchannels (gh-111006)
There were a few things I did in gh-110565 that need to be fixed. I also forgot to add tests in that PR.

(Note that this PR exposes a refleak introduced by gh-110246. I'll take care of that separately.)
2023-10-17 16:32:00 -06:00
Donghee Na 2dcc57008b
gh-109693: Remove pycore_atomic.h (gh-110992) 2023-10-18 00:33:50 +09:00
Victor Stinner 6db6b30ac2
gh-85283: Build winsound extension with limited C API (#110978)
Replace type->tp_name with PyType_GetQualName().
2023-10-17 15:57:10 +02:00
Victor Stinner be5e8a0103
gh-110964: Remove private _PyArg functions (#110966)
Move the following private functions and structures to
pycore_modsupport.h internal C API:

* _PyArg_BadArgument()
* _PyArg_CheckPositional()
* _PyArg_NoKeywords()
* _PyArg_NoPositional()
* _PyArg_ParseStack()
* _PyArg_ParseStackAndKeywords()
* _PyArg_Parser structure
* _PyArg_UnpackKeywords()
* _PyArg_UnpackKeywordsWithVararg()
* _PyArg_UnpackStack()
* _Py_ANY_VARARGS()

Changes:

* Python/getargs.h now includes pycore_modsupport.h to export
  functions.
* clinic.py now adds pycore_modsupport.h when one of these functions
  is used.
* Add pycore_modsupport.h includes when a C extension uses one of
  these functions.
* Define Py_BUILD_CORE_MODULE in C extensions which now include
  directly or indirectly (via code generated by Argument Clinic)
  pycore_modsupport.h:

  * _csv
  * _curses_panel
  * _dbm
  * _gdbm
  * _multiprocessing.posixshmem
  * _sqlite.row
  * _statistics
  * grp
  * resource
  * syslog

* _testcapi: bad_get() no longer uses METH_FASTCALL calling
  convention but METH_VARARGS. Replace _PyArg_UnpackStack() with
  PyArg_ParseTuple().
* _testcapi: add PYTESTCAPI_NEED_INTERNAL_API macro which is defined
  by _testcapi sub-modules which need the internal C API
  (pycore_modsupport.h): exceptions.c, float.c, vectorcall.c,
  watchers.c.
* Remove Include/cpython/modsupport.h header file.
  Include/modsupport.h no longer includes the removed header file.
* Fix mypy clinic.py
2023-10-17 14:30:31 +02:00
Donghee Na 86559ddfec
gh-109693: Update _gil_runtime_state.locked to use pyatomic.h (gh-110836) 2023-10-17 07:32:50 +09:00
Donghee Na b2ab210aae
gh-109693: Update pyruntimestate._finalizing to use pyatomic.h (gh-110837) 2023-10-13 16:40:15 +00:00
Pablo Galindo Salgado e1d8c65e1d
gh-110805: Allow the repl to show source code and complete tracebacks (#110775) 2023-10-13 09:25:37 +00:00