Commit Graph

441 Commits

Author SHA1 Message Date
gsallam be0c106789
gh-103295: expose API for writing perf map files (#103546)
Co-authored-by: Aniket Panse <aniketpanse@fb.com>
Co-authored-by: Gregory P. Smith <greg@krypto.org>
Co-authored-by: Carl Meyer <carl@oddbird.net>
2023-05-21 11:12:24 +01:00
Mark Shannon f7df173949
GH-101520: Move tracemalloc functionality into core, leaving interface in Modules. (#104508) 2023-05-17 14:17:16 +01:00
Jelle Zijlstra 24d8b88420
gh-103763: Implement PEP 695 (#103764)
This implements PEP 695, Type Parameter Syntax. It adds support for:

- Generic functions (def func[T](): ...)
- Generic classes (class X[T](): ...)
- Type aliases (type X = ...)
- New scoping when the new syntax is used within a class body
- Compiler and interpreter changes to support the new syntax and scoping rules 

Co-authored-by: Marc Mueller <30130371+cdce8p@users.noreply.github.com>
Co-authored-by: Eric Traut <eric@traut.com>
Co-authored-by: Larry Hastings <larry@hastings.org>
Co-authored-by: Alex Waygood <Alex.Waygood@Gmail.com>
2023-05-15 20:36:23 -07:00
Eric Snow 26baa747c2
gh-104341: Adjust tstate_must_exit() to Respect Interpreter Finalization (gh-104437)
With the move to a per-interpreter GIL, this check slipped through the cracks.
2023-05-15 13:59:26 -06:00
Erlend E. Aasland 186bf39f5c
gh-101819: Isolate `_io` (#101948)
Co-authored-by: Kumar Aditya <59607654+kumaraditya303@users.noreply.github.com>
Co-authored-by: Victor Stinner <vstinner@python.org>
2023-05-15 09:26:27 +00:00
Eric Snow 92d8bfffbf
gh-99113: Make Sure the GIL is Acquired at the Right Places (gh-104208)
This is a pre-requisite for a per-interpreter GIL.  Without it this change isn't strictly necessary.  However, there is no real downside otherwise.
2023-05-06 15:59:30 -06:00
Eric Snow f3e7eb48f8
gh-99113: Add PyInterpreterConfig.own_gil (gh-104204)
We also add PyInterpreterState.ceval.own_gil to record if the interpreter actually has its own GIL.

Note that for now we don't actually respect own_gil; all interpreters still share the one GIL.  However, PyInterpreterState.ceval.own_gil does reflect PyInterpreterConfig.own_gil.  That lie is a temporary one that we will fix when the GIL really becomes per-interpreter.
2023-05-05 15:59:20 -06:00
Victor Stinner 45398ad512
gh-103323: Remove PyRuntimeState_GetThreadState() (#104171)
This function no longer makes sense, since its runtime parameter is
no longer used. Use directly _PyThreadState_GET() and
_PyInterpreterState_GET() instead.
2023-05-04 16:21:01 +02:00
Eric Snow 292076a9aa
gh-104109: Expose Py_NewInterpreterFromConfig() in the Public C-API (gh-104110)
We also expose PyInterpreterConfig. This is part of the PEP 684 (per-interpreter GIL) implementation.  We will add docs as soon as we can.

FYI, I'm adding the new config field for per-interpreter GIL in gh-99114.
2023-05-02 21:40:00 -06:00
Eric Snow fdd878650d
gh-94673: Properly Initialize and Finalize Static Builtin Types for Each Interpreter (gh-104072)
Until now, we haven't been initializing nor finalizing the per-interpreter state properly.
2023-05-01 19:36:00 -06:00
Barry Warsaw e1f14643dc
gh-98040: Remove just the `imp` module (#98573) 2023-04-28 16:17:58 -07:00
Eric Snow d2e2e53f73
gh-94673: Ensure Builtin Static Types are Readied Properly (gh-103940)
There were cases where we do unnecessary work for builtin static types. This also simplifies some work necessary for a per-interpreter GIL.
2023-04-27 16:19:43 -06:00
Eric Snow df3173d28e
gh-101659: Isolate "obmalloc" State to Each Interpreter (gh-101660)
This is strictly about moving the "obmalloc" runtime state from
`_PyRuntimeState` to `PyInterpreterState`.  Doing so improves isolation
between interpreters, specifically most of the memory (incl. objects)
allocated for each interpreter's use.  This is important for a
per-interpreter GIL, but such isolation is valuable even without it.

FWIW, a per-interpreter obmalloc is the proverbial
canary-in-the-coalmine when it comes to the isolation of objects between
interpreters.  Any object that leaks (unintentionally) to another
interpreter is highly likely to cause a crash (on debug builds at
least).  That's a useful thing to know, relative to interpreter
isolation.
2023-04-24 17:23:57 -06:00
Eddie Elizondo ea2c001650
gh-84436: Implement Immortal Objects (gh-19474)
This is the implementation of PEP683

Motivation:

The PR introduces the ability to immortalize instances in CPython which bypasses reference counting. Tagging objects as immortal allows up to skip certain operations when we know that the object will be around for the entire execution of the runtime.

Note that this by itself will bring a performance regression to the runtime due to the extra reference count checks. However, this brings the ability of having truly immutable objects that are useful in other contexts such as immutable data sharing between sub-interpreters.
2023-04-22 13:39:37 -06:00
Eric Snow 03089fdccc
gh-101659: Add _Py_AtExit() (gh-103298)
The function is like Py_AtExit() but for a single interpreter.  This is a companion to the atexit module's register() function, taking a C callback instead of a Python one.

We also update the _xxinterpchannels module to use _Py_AtExit(), which is the motivating case.  (This is inspired by pain points felt while working on gh-101660.)
2023-04-05 18:42:02 -06:00
Eric Snow 89e67ada69
gh-100227: Revert gh-102925 "gh-100227: Make the Global Interned Dict Safe for Isolated Interpreters" (gh-103063)
This reverts commit 87be8d9.

This approach to keeping the interned strings safe is turning out to be too complex for my taste (due to obmalloc isolation). For now I'm going with the simpler solution, making the dict per-interpreter. We can revisit that later if we want a sharing solution.
2023-03-27 16:53:05 -06:00
Eric Snow 87be8d9522
gh-100227: Make the Global Interned Dict Safe for Isolated Interpreters (gh-102925)
This is effectively two changes.  The first (the bulk of the change) is where we add _Py_AddToGlobalDict() (and _PyRuntime.cached_objects.main_tstate, etc.).  The second (much smaller) change is where we update PyUnicode_InternInPlace() to use _Py_AddToGlobalDict() instead of calling PyDict_SetDefault() directly.

Basically, _Py_AddToGlobalDict() is a wrapper around PyDict_SetDefault() that should be used whenever we need to add a value to a runtime-global dict object (in the few cases where we are leaving the container global rather than moving it to PyInterpreterState, e.g. the interned strings dict).  _Py_AddToGlobalDict() does all the necessary work to make sure the target global dict is shared safely between isolated interpreters.  This is especially important as we move the obmalloc state to each interpreter (gh-101660), as well as, potentially, the GIL (PEP 684).

https://github.com/python/cpython/issues/100227
2023-03-22 18:30:04 -06:00
Eric Snow e6ecd3e6b4
gh-94673: Isolate the _io module to Each Interpreter (gh-102663)
Aside from sys and builtins, _io is the only core builtin module that hasn't been ported to multi-phase init.  We may do so later (e.g. gh-101948), but in the meantime we must at least take care of the module's static types properly.  (This came up while working on gh-101660.)

https://github.com/python/cpython/issues/94673
2023-03-21 14:01:38 -06:00
Eric Snow d1b883b52a
gh-98608: Fix Failure-handling in new_interpreter() (gh-102658)
The error-handling code in new_interpreter() has been broken for a while.  We hadn't noticed because those code mostly doesn't fail.  (I noticed while working on gh-101660.)  The problem is that we try to clear/delete the newly-created thread/interpreter using itself, which just failed.  The solution is to switch back to the calling thread state first.

https://github.com/python/cpython/issues/98608
2023-03-21 12:47:55 -06:00
Eric Snow 743687434c
gh-102304: Move the Total Refcount to PyInterpreterState (gh-102545)
Moving it valuable with a per-interpreter GIL.  However, it is also useful without one, since it allows us to identify refleaks within a single interpreter or where references are escaping an interpreter.  This becomes more important as we move the obmalloc state to PyInterpreterState.

https://github.com/python/cpython/issues/102304
2023-03-21 11:46:09 -06:00
Eric Snow 3bb475662b
gh-98608: Stop Treating All Errors from _Py_NewInterpreterFromConfig() as Fatal (gh-102657)
Prior to this change, errors in _Py_NewInterpreterFromConfig() were always fatal.  Instead, callers should be able to handle such errors and keep going.  That's what this change supports.  (This was an oversight in the original implementation of _Py_NewInterpreterFromConfig().)  Note that the existing [fatal] behavior of the public Py_NewInterpreter() is preserved.

https://github.com/python/cpython/issues/98608
2023-03-21 10:49:12 -06:00
Eric Snow ad77d16a62
gh-102304: Move _Py_RefTotal to _PyRuntimeState (gh-102543)
The essentially eliminates the global variable, with the associated benefits. This is also a precursor to isolating this bit of state to PyInterpreterState.

Folks that currently read _Py_RefTotal directly would have to start using _Py_GetGlobalRefTotal() instead.

https://github.com/python/cpython/issues/102304
2023-03-20 10:03:04 -06:00
Irit Katriel ccb5af7bfe
gh-102755: fix refleak (#102826) 2023-03-19 15:18:24 +00:00
Irit Katriel e1e9bab006
gh-102778: Add sys.last_exc, deprecate sys.last_type, sys.last_value,sys.last_traceback (#102779) 2023-03-18 11:47:11 +00:00
Irit Katriel 3f9285a8c5
gh-102755: Add PyErr_DisplayException(exc) (#102756) 2023-03-16 22:18:04 +00:00
Max Bachmann c6858d1e7f
gh-102255: Improve build support for Windows API partitions (GH-102256)
Add `MS_WINDOWS_DESKTOP`, `MS_WINDOWS_APPS`, `MS_WINDOWS_SYSTEM` and `MS_WINDOWS_GAMES` preprocessor definitions to allow switching off functionality missing from particular API partitions ("partitions" are used in Windows to identify overlapping subsets of APIs).
CPython only officially supports `MS_WINDOWS_DESKTOP` and `MS_WINDOWS_SYSTEM` (APPS is included by normal desktop builds, but APPS without DESKTOP is not covered). Other configurations are a convenience for people building their own runtimes.
`MS_WINDOWS_GAMES` is for the Xbox subset of the Windows API, which is also available on client OS, but is restricted compared to `MS_WINDOWS_DESKTOP`. These restrictions may change over time, as they relate to the build headers rather than the OS support, and so we assume that Xbox builds will use the latest available version of the GDK.
2023-03-09 21:09:12 +00:00
Eric Snow 89ac665891
gh-98627: Add an Optional Check for Extension Module Subinterpreter Compatibility (gh-99040)
Enforcing (optionally) the restriction set by PEP 489 makes sense. Furthermore, this sets the stage for a potential restriction related to a per-interpreter GIL.

This change includes the following:

* add tests for extension module subinterpreter compatibility
* add _PyInterpreterConfig.check_multi_interp_extensions
* add Py_RTFLAGS_MULTI_INTERP_EXTENSIONS
* add _PyImport_CheckSubinterpIncompatibleExtensionAllowed()
* fail iff the module does not implement multi-phase init and the current interpreter is configured to check

https://github.com/python/cpython/issues/98627
2023-02-15 18:16:00 -07:00
Eric Snow b2fc549278
gh-101758: Clean Up Uses of Import State (gh-101919)
This change is almost entirely moving code around and hiding import state behind internal API.  We introduce no changes to behavior, nor to non-internal API.  (Since there was already going to be a lot of churn, I took this as an opportunity to re-organize import.c into topically-grouped sections of code.)  The motivation is to simplify a number of upcoming changes.

Specific changes:

* move existing import-related code to import.c, wherever possible
* add internal API for interacting with import state (both global and per-interpreter)
* use only API outside of import.c (to limit churn there when changing the location, etc.)
* consolidate the import-related state of PyInterpreterState into a single struct field (this changes layout slightly)
* add macros for import state in import.c (to simplify changing the location)
* group code in import.c into sections
*remove _PyState_AddModule()

https://github.com/python/cpython/issues/101758
2023-02-15 15:32:31 -07:00
Erlend E. Aasland eb0c485b6c
gh-101819: Remove _PyWindowsConsoleIO_Type from the Windows DLL (GH-101904)
Automerge-Triggered-By: GH:erlend-aasland
2023-02-15 05:07:59 -08:00
Eric Snow e11fc032a7
gh-59956: Clarify Runtime State Status Expectations (gh-101308)
A PyThreadState can be in one of many states in its lifecycle, represented by some status value.  Those statuses haven't been particularly clear, so we're addressing that here.  Specifically:

* made the distinct lifecycle statuses clear on PyThreadState
* identified expectations of how various lifecycle-related functions relate to status
* noted the various places where those expectations don't match the actual behavior

At some point we'll need to address the mismatches.

(This change also includes some cleanup.)

https://github.com/python/cpython/issues/59956
2023-01-30 12:07:48 -07:00
Eric Snow 6036c3e856
gh-59956: Clarify GILState-related Code (gh-101161)
The objective of this change is to help make the GILState-related code easier to understand.  This mostly involves moving code around and some semantically equivalent refactors.  However, there are a also a small number of slight changes in structure and behavior:

* tstate_current is moved out of _PyRuntimeState.gilstate
* autoTSSkey is moved out of _PyRuntimeState.gilstate
* autoTSSkey is initialized earlier
* autoTSSkey is re-initialized (after fork) earlier

https://github.com/python/cpython/issues/59956
2023-01-19 16:04:14 -07:00
Eric Snow 5eb28bca9f
gh-81057: Move Signal-Related Globals to _PyRuntimeState (gh-100085)
https://github.com/python/cpython/issues/81057
2022-12-12 16:50:19 -07:00
Eric Snow 8a3f06c54b
gh-81057: Move time Globals to _PyRuntimeState (gh-100122)
https://github.com/python/cpython/issues/81057
2022-12-08 16:46:09 -07:00
Ronald Oussoren cdde29dde9
gh-99337: Fix compile errors with gcc 12 on macOS (#99470)
Fix a number of compile errors with GCC-12 on macOS:

1. In pylifecycle.c the compile rejects _Pragma within a declaration
2. posixmodule.c was missing a number of ..._RUNTIME macros for non-clang on macOS
3. _ctypes assumed that __builtin_available is always present on macOS
2022-11-21 10:50:20 +01:00
Eric Snow 5f55067e23
gh-81057: Move More Globals in Core Code to _PyRuntimeState (gh-99516)
https://github.com/python/cpython/issues/81057
2022-11-16 09:37:14 -07:00
Eric Snow 3c57971a2d
gh-81057: Move Globals in Core Code to _PyRuntimeState (gh-99496)
This is the first of several changes to consolidate non-object globals in core code.

https://github.com/python/cpython/issues/81057
2022-11-15 09:45:11 -07:00
Eric Snow 7f3a4b967c
gh-81057: Move PyImport_Inittab to _PyRuntimeState (gh-99402)
We actually don't move PyImport_Inittab.  Instead, we make a copy that we keep on _PyRuntimeState and use only that after Py_Initialize().  We also prevent folks from modifying PyImport_Inittab (the best we can) after that point.

https://github.com/python/cpython/issues/81057
2022-11-11 17:06:05 -07:00
Eric Snow 67807cfc87
gh-81057: Move the Allocators to _PyRuntimeState (gh-99217)
The global allocators were stored in 3 static global variables: _PyMem_Raw, _PyMem, and _PyObject.  State for the "small block" allocator was stored in another 13.  That makes a total of 16 global variables. We are moving all 16 to the _PyRuntimeState struct as part of the work for gh-81057.  (If PEP 684 is accepted then we will follow up by moving them all to PyInterpreterState.)

https://github.com/python/cpython/issues/81057
2022-11-11 16:30:46 -07:00
Eric Snow f531b6879b
gh-81057: Add PyInterpreterState.static_objects (gh-99397)
As we consolidate global variables, we find some objects that are almost suitable to add to _PyRuntimeState.global_objects, but have some small/sneaky bit of per-interpreter state (e.g. a weakref list). We're adding PyInterpreterState.static_objects so we can move such objects there. (We'll removed the _not_used field once we've added others.)

https://github.com/python/cpython/issues/81057
2022-11-11 14:24:18 -07:00
Mark Shannon 1e197e63e2
GH-96421: Insert shim frame on entry to interpreter (GH-96319)
* Adds EXIT_INTERPRETER instruction to exit PyEval_EvalDefault()

* Simplifies RETURN_VALUE, YIELD_VALUE and RETURN_GENERATOR instructions as they no longer need to check for entry frames.
2022-11-10 12:34:57 +00:00
Victor Stinner d8f239d86e
gh-99300: Use Py_NewRef() in Python/ directory (#99302)
Replace Py_INCREF() and Py_XINCREF() with Py_NewRef() and
Py_XNewRef() in C files of the Python/ directory.
2022-11-10 09:03:39 +01:00
Eric Snow 52f91c642b
gh-90868: Adjust the Generated Objects (gh-99223)
We do the following:

* move the generated _PyUnicode_InitStaticStrings() to its own file
* move the generated _PyStaticObjects_CheckRefcnt() to its own file
* include pycore_global_objects.h in extension modules instead of pycore_runtime_init.h

These changes help us avoid including things that aren't needed.

https://github.com/python/cpython/issues/90868
2022-11-08 10:03:03 -07:00
Eric Snow 4702552885
gh-98610: Adjust the Optional Restrictions on Subinterpreters (GH-98618)
Previously, the optional restrictions on subinterpreters were: disallow fork, subprocess, and threads.  By default, we were disallowing all three for "isolated" interpreters.  We always allowed all three for the main interpreter and those created through the legacy `Py_NewInterpreter()` API.

Those settings were a bit conservative, so here we've adjusted the optional restrictions to: fork, exec, threads, and daemon threads.  The default for "isolated" interpreters disables fork, exec, and daemon threads.  Regular threads are allowed by default.  We continue always allowing everything For the main interpreter and the legacy API.

In the code, we add `_PyInterpreterConfig.allow_exec` and  `_PyInterpreterConfig.allow_daemon_threads`.  We also add `Py_RTFLAGS_DAEMON_THREADS` and `Py_RTFLAGS_EXEC`.
2022-10-31 12:35:54 -07:00
Nick Coghlan 05e48865be
gh-96853: Restore test coverage for Py_Initialize(Ex) (GH-98212)
* As most of `test_embed` now uses `Py_InitializeFromConfig`, add
  a specific test case to cover `Py_Initialize` (and `Py_InitializeEx`)
* Rename `_testembed` init helper to clarify the API used
* Add a `PyConfig_Clear` call in `Py_InitializeEx` to make
  the code more obviously correct (it already didn't leak as
  none of the dynamically allocated config fields were being
  populated, but it's clearer if the wrappers follow the
  documented API usage guidelines)
2022-10-30 22:01:30 +10:00
Eric Snow f32369480d
gh-98608: Change _Py_NewInterpreter() to _Py_NewInterpreterFromConfig() (gh-98609)
(see https://github.com/python/cpython/issues/98608)

This change does the following:

1. change the argument to a new `_PyInterpreterConfig` struct
2. rename the function to `_Py_NewInterpreterFromConfig()`, inspired by `Py_InitializeFromConfig()` (takes a `_PyInterpreterConfig`  instead of `isolated_subinterpreter`)
3. split up the boolean `isolated_subinterpreter` into the corresponding multiple granular settings
   * allow_fork
   * allow_subprocess
   * allow_threads
4. add `PyInterpreterState.feature_flags` to store those settings
5. add a function for checking if a feature is enabled on an opaque `PyInterpreterState *`
6. drop `PyConfig._isolated_interpreter`

The existing default (see `Py_NewInterpeter()` and `Py_Initialize*()`) allows fork, subprocess, and threads and the optional "isolated" interpreter (see the `_xxsubinterpreters` module) disables all three.  None of that changes here; the defaults are preserved.

Note that the given `_PyInterpreterConfig` will not be used outside `_Py_NewInterpreterFromConfig()`, nor preserved.  This contrasts with how `PyConfig` is currently preserved, used, and even modified outside `Py_InitializeFromConfig()`.  I'd rather just avoid that mess from the start for `_PyInterpreterConfig`.  We can preserve it later if we find an actual need.

This change allows us to follow up with a number of improvements (e.g. stop disallowing subprocess and support disallowing exec instead).

(Note that this PR adds "private" symbols.  We'll probably make them public, and add docs, in a separate change.)
2022-10-26 11:16:30 -06:00
Eric Snow 9c8dde0fa5
gh-98417: Store int_max_str_digits on the Interpreter State (GH-98418) 2022-10-19 13:27:46 -07:00
Kumar Aditya 4e4bfffe2d
GH-90699: use statically allocated interned strings in typeobject's slotdefs (GH-94706) 2022-09-07 15:02:08 -07:00
Pablo Galindo Salgado 6d791a9736
gh-96143: Allow Linux perf profiler to see Python calls (GH-96123)
⚠️  ⚠️ Note for reviewers, hackers and fellow systems/low-level/compiler engineers ⚠️ ⚠️ 

If you have a lot of experience with this kind of shenanigans and want to improve the **first** version, **please make a PR against my branch** or **reach out by email** or **suggest code changes directly on GitHub**. 

If you have any **refinements or optimizations** please, wait until the first version is merged before starting hacking or proposing those so we can keep this PR productive.
2022-08-30 10:11:18 -07:00
Ken Jin 7276ca25f5
GH-93911: Specialize `LOAD_ATTR` for custom `__getattribute__` (GH-93988) 2022-08-17 12:37:07 +01:00
Eric Snow 2d26449b06
gh-94673: Always Finalize Static Builtin Types (#95153)
Static builtin types are finalized by calling _PyStaticType_Dealloc().  Before this change, we were skipping finalizing such a type if it still had subtypes (i.e. its tp_subclasses hadn't been cleared yet).  The problem is that types hold several heap objects, which leak if we skip the type's finalization.  This change addresses that.

For context, there's an old comment (from e9e3eab0b8) that says the following:

   // If a type still has subtypes, it cannot be deallocated.
   // A subtype can inherit attributes and methods of its parent type,
   // and a type must no longer be used once it's deallocated.

However, it isn't clear that is actually still true.  Clearing tp_dict should mean it isn't a problem.

Furthermore, the only subtypes that might still be around come from extension modules that didn't clean them up when unloaded (i.e. extensions that do not implement multi-phase initialization, AKA PEP 489).  Those objects are already leaking, so this change doesn't change anything in that regard.  Instead, this change means more objects gets cleaned up that before.
2022-07-25 14:23:41 -06:00