Commit Graph

8160 Commits

Author SHA1 Message Date
Pablo Galindo Salgado 1ef61cf71a
gh-102856: Initial implementation of PEP 701 (#102855)
Co-authored-by: Lysandros Nikolaou <lisandrosnik@gmail.com>
Co-authored-by: Batuhan Taskaya <isidentical@gmail.com>
Co-authored-by: Marta Gómez Macías <mgmacias@google.com>
Co-authored-by: sunmy2019 <59365878+sunmy2019@users.noreply.github.com>
2023-04-19 11:18:16 -05:00
Nikita Sobolev 07804ce24c
GH-100530: Change the error message for non-class class patterns (GH-103576) 2023-04-18 10:41:14 -07:00
Benjamin Peterson 7b95d23591
Fix unused functions warnings in instrumentation.c (GH-103515) 2023-04-13 18:45:03 +01:00
Mark Shannon efb8a2553c
GH-103488: Use return-offset, not yield-offset. (GH-103502)
* Use return-offset, not yield-offset, so that instruction pointer is correct when sending to a generator or coroutine.
2023-04-13 16:19:07 +01:00
Mark Shannon 70e0a28bed
GH-103082: Turn on branch events for FOR_ITER instructions. (#103507)
Turn on branch events for FOR_ITER instructions.
2023-04-13 13:56:09 +01:00
Nikita Sobolev 7f3c106503
gh-103326: Remove `Python/importlib.h` (GH-103331)
Co-authored-by: Brett Cannon <brett@python.org>
Co-authored-by: Hugo van Kemenade <hugovk@users.noreply.github.com>
2023-04-12 11:27:14 -07:00
Mark Shannon 411b169281
GH-103082: Implementation of PEP 669: Low Impact Monitoring for CPython (GH-103083)
* The majority of the monitoring code is in instrumentation.c

* The new instrumentation bytecodes are in bytecodes.c

* legacy_tracing.c adapts the new API to the old sys.setrace and sys.setprofile APIs
2023-04-12 12:04:55 +01:00
Irit Katriel 9db2db4fa4
gh-87092: fix refleak in peepholer test harness (#103448) 2023-04-11 21:08:29 +01:00
Irit Katriel 55c99d97e1
gh-77757: replace exception wrapping by PEP-678 notes in typeobject's __set_name__ (#103402) 2023-04-11 11:53:06 +01:00
Irit Katriel 21bea68e2e
gh-91276: remove unused _PyOpcode_RelativeJump (#103156) 2023-04-11 11:20:39 +01:00
Irit Katriel 33822d037a
gh-87092: move assembler related code from compile.c to assemble.c (#103277) 2023-04-11 11:15:09 +01:00
Irit Katriel 78b763f630
gh-103176: sys._current_exceptions() returns mapping to exception instances instead of exc_info tuples (#103177) 2023-04-11 09:38:37 +01:00
Eric Snow 52e9b389a8
gh-100227: Use an Array for _PyRuntime's Set of Locks During Init (gh-103315)
This cleans things up a bit and simplifies adding new granular global locks.
2023-04-06 12:00:49 -06:00
Eric Snow 03089fdccc
gh-101659: Add _Py_AtExit() (gh-103298)
The function is like Py_AtExit() but for a single interpreter.  This is a companion to the atexit module's register() function, taking a C callback instead of a Python one.

We also update the _xxinterpchannels module to use _Py_AtExit(), which is the motivating case.  (This is inspired by pain points felt while working on gh-101660.)
2023-04-05 18:42:02 -06:00
Brandt Bucher b4978ff872
GH-88691: Shrink the CALL caches (GH-103230) 2023-04-05 14:15:49 -07:00
Nikita Sobolev 119f67de08
gh-103167: Fix `-Wstrict-prototypes` warnings by using `(void)` for functions with no args (GH-103168) 2023-04-05 09:22:33 +02:00
Eric Snow f513d5c806
gh-102660: Fix is_core_module() (gh-103257)
In gh-102744 we added is_core_module() (in Python/import.c), which relies on get_core_module_dict() (also added in that PR).  The problem is that_PyImport_FixupBuiltin(), which ultimately calls is_core_module(), is called on the builtins module before interp->builtins_copyis set.  Consequently, the builtins module isn't considered a "core" module while it is getting "fixed up" and its module def m_copy erroneously gets set.  Under isolated interpreters this causes problems since sys and builtins are allowed even though they are still single-phase init modules.  (This was discovered while working on gh-101660.)

The solution is to stop relying on get_core_module_dict() in is_core_module().
2023-04-04 17:03:40 -06:00
Irit Katriel 848bdbe166
gh-102192: use PyErr_SetHandledException instead of the legacy PyErr_SetExcInfo (#103157) 2023-04-01 10:31:48 +05:30
Brett Cannon d97aef8ebf
Add missing variables to `bytecodes.c` (GH-103153)
The code works without this change, but it does cause C tooling to complain less about undeclared variables.
2023-03-31 14:23:55 -07:00
Eric Snow dde028480e
gh-100227: Fix Cleanup of the Extensions Cache (gh-103150)
Decref the key in the right interpreter in _extensions_cache_set().

This is a follow-up to gh-103084. I found the bug while working on gh-101660.
2023-03-31 12:09:10 -06:00
Irit Katriel 80163e17d3
gh-87092: move CFG related code from compile.c to flowgraph.c (#103021) 2023-03-31 18:17:59 +01:00
Eric Snow dcd6f226d6
gh-100227: Make the Global PyModuleDef Cache Safe for Isolated Interpreters (gh-103084)
Sharing mutable (or non-immortal) objects between interpreters is generally not safe.  We can work around that but not easily. 
 There are two restrictions that are critical for objects that break interpreter isolation.

The first is that the object's state be guarded by a global lock.  For now the GIL meets this requirement, but a granular global lock is needed once we have a per-interpreter GIL.

The second restriction is that the object (and, for a container, its items) be deallocated/resized only when the interpreter in which it was allocated is the current one.  This is because every interpreter has (or will have, see gh-101660) its own object allocator.  Deallocating an object with a different allocator can cause crashes.

The dict for the cache of module defs is completely internal, which simplifies what we have to do to meet those requirements.  To do so, we do the following:

* add a mechanism for re-using a temporary thread state tied to the main interpreter in an arbitrary thread
   * add _PyRuntime.imports.extensions.main_tstate` 
   * add _PyThreadState_InitDetached() and _PyThreadState_ClearDetached() (pystate.c)
   * add _PyThreadState_BindDetached() and _PyThreadState_UnbindDetached() (pystate.c)
* make sure the cache dict (_PyRuntime.imports.extensions.dict) and its items are all owned by the main interpreter)
* add a placeholder using for a granular global lock

Note that the cache is only used for legacy extension modules and not for multi-phase init modules.

https://github.com/python/cpython/issues/100227
2023-03-29 17:15:43 -06:00
Brandt Bucher 121057aa36
GH-89987: Shrink the BINARY_SUBSCR caches (GH-103022) 2023-03-29 15:53:30 -07:00
Eric Snow 89e67ada69
gh-100227: Revert gh-102925 "gh-100227: Make the Global Interned Dict Safe for Isolated Interpreters" (gh-103063)
This reverts commit 87be8d9.

This approach to keeping the interned strings safe is turning out to be too complex for my taste (due to obmalloc isolation). For now I'm going with the simpler solution, making the dict per-interpreter. We can revisit that later if we want a sharing solution.
2023-03-27 16:53:05 -06:00
Brandt Bucher 0444ae2487
GH-100982: Break up COMPARE_AND_BRANCH (GH-102801) 2023-03-23 15:25:09 -07:00
Nikita Sobolev 0f2ba65805
gh-102939: Fix "conversion from Py_ssize_t to long" warning in builtins (GH-102940) 2023-03-23 10:37:04 +00:00
Eric Snow 87be8d9522
gh-100227: Make the Global Interned Dict Safe for Isolated Interpreters (gh-102925)
This is effectively two changes.  The first (the bulk of the change) is where we add _Py_AddToGlobalDict() (and _PyRuntime.cached_objects.main_tstate, etc.).  The second (much smaller) change is where we update PyUnicode_InternInPlace() to use _Py_AddToGlobalDict() instead of calling PyDict_SetDefault() directly.

Basically, _Py_AddToGlobalDict() is a wrapper around PyDict_SetDefault() that should be used whenever we need to add a value to a runtime-global dict object (in the few cases where we are leaving the container global rather than moving it to PyInterpreterState, e.g. the interned strings dict).  _Py_AddToGlobalDict() does all the necessary work to make sure the target global dict is shared safely between isolated interpreters.  This is especially important as we move the obmalloc state to each interpreter (gh-101660), as well as, potentially, the GIL (PEP 684).

https://github.com/python/cpython/issues/100227
2023-03-22 18:30:04 -06:00
Irit Katriel 3468c768ce
gh-102859: Remove JUMP_IF_FALSE_OR_POP and JUMP_IF_TRUE_OR_POP (#102870) 2023-03-22 18:10:48 +00:00
Mark Shannon 7559f5fda9
GH-101291: Rearrange the size bits in PyLongObject (GH-102464)
* Eliminate all remaining uses of Py_SIZE and Py_SET_SIZE on PyLongObject, adding asserts.

* Change layout of size/sign bits in longobject to support future addition of immortal ints and tagged medium ints.

* Add functions to hide some internals of long object, and for setting sign and digit count.

* Replace uses of IS_MEDIUM_VALUE macro with _PyLong_IsCompact().
2023-03-22 14:49:51 +00:00
Irit Katriel 76350e85eb
gh-102406: replace exception chaining by PEP-678 notes in codecs (#102407) 2023-03-21 21:36:31 +00:00
Eric Snow e6ecd3e6b4
gh-94673: Isolate the _io module to Each Interpreter (gh-102663)
Aside from sys and builtins, _io is the only core builtin module that hasn't been ported to multi-phase init.  We may do so later (e.g. gh-101948), but in the meantime we must at least take care of the module's static types properly.  (This came up while working on gh-101660.)

https://github.com/python/cpython/issues/94673
2023-03-21 14:01:38 -06:00
Irit Katriel 8d015fa000
gh-102860: improve performance of compiler's instr_sequence_to_cfg (#102861) 2023-03-21 19:13:49 +00:00
Eric Snow d1b883b52a
gh-98608: Fix Failure-handling in new_interpreter() (gh-102658)
The error-handling code in new_interpreter() has been broken for a while.  We hadn't noticed because those code mostly doesn't fail.  (I noticed while working on gh-101660.)  The problem is that we try to clear/delete the newly-created thread/interpreter using itself, which just failed.  The solution is to switch back to the calling thread state first.

https://github.com/python/cpython/issues/98608
2023-03-21 12:47:55 -06:00
Eric Snow 743687434c
gh-102304: Move the Total Refcount to PyInterpreterState (gh-102545)
Moving it valuable with a per-interpreter GIL.  However, it is also useful without one, since it allows us to identify refleaks within a single interpreter or where references are escaping an interpreter.  This becomes more important as we move the obmalloc state to PyInterpreterState.

https://github.com/python/cpython/issues/102304
2023-03-21 11:46:09 -06:00
Eric Snow 3bb475662b
gh-98608: Stop Treating All Errors from _Py_NewInterpreterFromConfig() as Fatal (gh-102657)
Prior to this change, errors in _Py_NewInterpreterFromConfig() were always fatal.  Instead, callers should be able to handle such errors and keep going.  That's what this change supports.  (This was an oversight in the original implementation of _Py_NewInterpreterFromConfig().)  Note that the existing [fatal] behavior of the public Py_NewInterpreter() is preserved.

https://github.com/python/cpython/issues/98608
2023-03-21 10:49:12 -06:00
Nikita Sobolev 82eb9469e7
gh-102598: Remove obsolete optimization from `FORMAT_VALUE` opcode (#102599) 2023-03-21 00:47:15 -07:00
Eric Snow 5c75b7a91c
gh-102304: Fix Non-Debug Builds (gh-102846)
Some debug-only code slipped in with gh-102543.

https://github.com/python/cpython/issues/102304
2023-03-20 11:28:13 -06:00
Eric Snow ad77d16a62
gh-102304: Move _Py_RefTotal to _PyRuntimeState (gh-102543)
The essentially eliminates the global variable, with the associated benefits. This is also a precursor to isolating this bit of state to PyInterpreterState.

Folks that currently read _Py_RefTotal directly would have to start using _Py_GetGlobalRefTotal() instead.

https://github.com/python/cpython/issues/102304
2023-03-20 10:03:04 -06:00
Irit Katriel ccb5af7bfe
gh-102755: fix refleak (#102826) 2023-03-19 15:18:24 +00:00
Irit Katriel ad77b80b05
gh-102192: Replace PyErr_Fetch/Restore etc by more efficient alternatives (#102816) 2023-03-19 15:17:59 +00:00
Irit Katriel 1cb75a9ce0
gh-102192: Replace PyErr_Fetch/Restore etc by more efficient alternatives (#102769) 2023-03-18 13:44:47 +00:00
Irit Katriel e1e9bab006
gh-102778: Add sys.last_exc, deprecate sys.last_type, sys.last_value,sys.last_traceback (#102779) 2023-03-18 11:47:11 +00:00
gaogaotiantian 039714d00f
gh-101975: Fixed a potential SegFault on garbage collection (GH-102803) 2023-03-18 10:59:21 +00:00
Irit Katriel 3f9285a8c5
gh-102755: Add PyErr_DisplayException(exc) (#102756) 2023-03-16 22:18:04 +00:00
Irit Katriel 6372e290c0
gh-102192: Replace PyErr_Fetch/Restore etc by more efficient alternatives (#102760) 2023-03-16 19:03:52 +00:00
Steve Dower 0f175766e2
gh-99726: Improves correctness of stat results for Windows, and uses faster API when available (GH-102149)
This deprecates `st_ctime` fields on Windows, with the intent to change them to contain the correct value in 3.14. For now, they should keep returning the creation time as they always have.
2023-03-16 17:27:21 +00:00
Irit Katriel 61d6c110d6
gh-102192: Replace PyErr_Fetch/Restore etc by more efficient alternatives (#102743) 2023-03-16 16:21:49 +00:00
Irit Katriel 51d693c584
gh-102594: PyErr_SetObject adds note to exception raised on normalization error (#102675) 2023-03-16 10:16:01 +00:00
Eric Snow 2a03ed034e
gh-102660: Fix Refleaks in import.c (#102744)
gh-102661 introduced some leaks. This fixes them.

https://github.com/python/cpython/issues/102660
2023-03-15 18:43:54 -06:00
Irit Katriel 675b97a6ab
gh-102738: remove from cases generator the code related to register instructions (#102739) 2023-03-15 21:25:31 +00:00
Guido van Rossum 70185de1ab
gh-102654: Insert #line directives in generated_cases.c.h (#102669)
This behavior is optional, because in some extreme cases it
may just make debugging harder. The tool defaults it to off,
but it is on in Makefile.pre.in.

Also note that this makes diffs to generated_cases.c.h noisier,
since whenever you insert or delete a line in bytecodes.c,
all subsequent #line directives will change.
2023-03-15 08:37:36 -07:00
Max Bachmann afa6092ee4
gh-102281: Fix potential nullptr dereference + use of uninitialized memory (gh-102282) 2023-03-15 21:58:43 +09:00
Eric Snow cdb21ba74d
gh-102660: Handle m_copy Specially for the sys and builtins Modules (gh-102661)
It doesn't make sense to use multi-phase init for these modules. Using a per-interpreter "m_copy" (instead of PyModuleDef.m_base.m_copy) makes this work okay. (This came up while working on gh-101660.)

Note that we might instead end up disallowing re-load for sys/builtins since they are so special.

https://github.com/python/cpython/issues/102660
2023-03-14 14:01:35 -06:00
Guido van Rossum 392f2ad3cb
gh-98831: Use DECREF_INPUTS() more (#102409) 2023-03-13 15:08:45 -07:00
Mark Shannon 2d370da570
GH-100987: Don't cache references to the names and consts array in `_PyEval_EvalFrameDefault`. (#102640)
* Rename local variables, names and consts, from the interpeter loop. Will allow non-code objects in frames for better introspection of C builtins and extensions.

* Remove unused dummy variables.
2023-03-13 18:35:37 +00:00
Irit Katriel 634cb61909
gh-87092: refactor assemble() to a number of separate functions, which do not need the compiler struct (#102562) 2023-03-13 15:59:20 +00:00
Irit Katriel ca01cae1e9
gh-102192: Replace PyErr_Fetch/Restore etc by more efficient alternatives (#102631) 2023-03-13 15:56:24 +00:00
Mark Shannon 233e32f936
GH-102300: Reuse objects with refcount == 1 in float specialized binary ops. (GH-102301) 2023-03-13 10:34:54 +00:00
Brandt Bucher 08b67fb34f
GH-90997: Shrink the LOAD_GLOBAL caches (#102569) 2023-03-10 17:01:16 -08:00
Max Bachmann c6858d1e7f
gh-102255: Improve build support for Windows API partitions (GH-102256)
Add `MS_WINDOWS_DESKTOP`, `MS_WINDOWS_APPS`, `MS_WINDOWS_SYSTEM` and `MS_WINDOWS_GAMES` preprocessor definitions to allow switching off functionality missing from particular API partitions ("partitions" are used in Windows to identify overlapping subsets of APIs).
CPython only officially supports `MS_WINDOWS_DESKTOP` and `MS_WINDOWS_SYSTEM` (APPS is included by normal desktop builds, but APPS without DESKTOP is not covered). Other configurations are a convenience for people building their own runtimes.
`MS_WINDOWS_GAMES` is for the Xbox subset of the Windows API, which is also available on client OS, but is restricted compared to `MS_WINDOWS_DESKTOP`. These restrictions may change over time, as they relate to the build headers rather than the OS support, and so we assume that Xbox builds will use the latest available version of the GDK.
2023-03-09 21:09:12 +00:00
Eric Snow cf6e7c5e55
gh-100227: Isolate the Import State to Each Interpreter (gh-101941)
Specific changes:

* move the import lock to PyInterpreterState
* move the "find_and_load" diagnostic state to PyInterpreterState

Note that the import lock exists to keep multiple imports of the same module in the same interpreter (but in different threads) from stomping on each other.  Independently, we use a distinct global lock to protect globally shared import state, especially related to loaded extension modules.  For now we can rely on the GIL as that lock but with a per-interpreter GIL we'll need a new global lock.

The remaining state in _PyRuntimeState.imports will (probably) continue being global.

https://github.com/python/cpython/issues/100227
2023-03-09 09:46:21 -07:00
Eric Snow b45d14b886
gh-100227: Move dict_state.global_version to PyInterpreterState (gh-102338)
https://github.com/python/cpython/issues/100227
2023-03-09 08:16:30 -07:00
Eric Snow 5e5acd291f
gh-100227: Move next_keys_version to PyInterpreterState (gh-102335)
https://github.com/python/cpython/issues/100227
2023-03-08 18:04:16 -07:00
Irit Katriel a33ca2ad1f
gh-102493: fix normalization in PyErr_SetObject (#102502)
Co-authored-by: Jelle Zijlstra <jelle.zijlstra@gmail.com>
2023-03-07 13:27:46 -08:00
Irit Katriel 54060ae91d
gh-87092: compiler's CFG construction moved to after codegen stage (#102320) 2023-03-07 18:16:32 +00:00
Eric Snow 8606697f49
gh-90110: Fix the c-analyzer Tool (#102483)
Some incompatible changes had gone in, and the "ignore" lists weren't properly undated. This change fixes that. It's necessary prior to enabling test_check_c_globals, which I hope to do soon.

Note that this does include moving last_resort_memory_error to PyInterpreterState.

https://github.com/python/cpython/issues/90110
2023-03-06 19:40:09 -07:00
Marta Gómez Macías 66aa78cbe6
gh-102356: Add thrashcan macros to filter object dealloc (#102426)
Add thrashcan macros to the deallocator of the filter objects to protect against deeply nested destruction of chains of nested filters.
2023-03-05 12:00:41 +01:00
Jacob Bower 8de59c1bb9
gh-102021 : Allow multiple input files for interpreter loop generator (#102022)
The input files no longer use `-i`.
2023-03-03 20:59:21 -08:00
Irit Katriel 71db5dbcd7
gh-102371: move _Py_Mangle from compile.c to symtable.c (#102372) 2023-03-02 18:38:22 +00:00
Max Bachmann 938e36f824
gh-102336: Remove code specifically for handling Windows 7 (GH-102337) 2023-03-01 00:31:21 +00:00
Eric Snow 880437d4ec
gh-100227: Move _str_replace_inf to PyInterpreterState (gh-102333)
https://github.com/python/cpython/issues/100227
2023-02-28 14:16:39 -07:00
Eric Snow f300a1fa4c
gh-100227: Move the dtoa State to PyInterpreterState (gh-102331)
https://github.com/python/cpython/issues/100227
2023-02-28 13:14:40 -07:00
Guido van Rossum b5ff382433
GH-102305: Expand some macros in generated_cases.c.h (#102309)
* Emit straight stack_pointer[-i] instead of PEEK(i), POKE(i, ...)
* Expand JUMPBY() and NEXTOPARG(), and fix a perf bug
2023-02-28 08:49:35 -08:00
Irit Katriel 4c87537efb
gh-102192: Replace PyErr_Fetch/Restore etc by more efficient alternatives (in Python/) (#102193) 2023-02-28 11:50:52 +00:00
Irit Katriel 9f799ab020
gh-87092: Make jump target label equal to the offset of the target in the instructions sequence (#102093) 2023-02-28 11:29:32 +00:00
Petr Viktorin 6b2d7c0ddb
gh-101101: Unstable C API tier (PEP 689) (GH-101102) 2023-02-28 09:31:01 +01:00
Eric Snow bb0cf8fd60
gh-102251: Updates to test_imp Toward Fixing Some Refleaks (gh-102254)
This is related to fixing the refleaks introduced by commit 096d009.  I haven't been able to find the leak yet, but these changes are a consequence of that effort.  This includes some cleanup, some tweaks to the existing tests, and a bunch of new test cases.  The only change here that might have impact outside the tests in question is in imp.py, where I update imp.load_dynamic() to use spec_from_file_location() instead of creating a ModuleSpec directly.

Also note that I've updated the tests to only skip if we're checking for refleaks (regrtest's --huntrleaks), whereas in gh-101969 I had skipped the tests entirely.  The tests will be useful for some upcoming work and I'd rather the refleaks not hold that up.  (It isn't clear how quickly we'll be able to fix the leaking code, though it will certainly be done in the short term.)

https://github.com/python/cpython/issues/102251
2023-02-27 09:21:18 -07:00
Dennis Sweeney e3c3f9fec0
gh-102250: Fix double-decref in COMPARE_AND_BRANCH error case (GH-102287) 2023-02-27 10:46:40 +00:00
Kumar Aditya 5f11478ce7
GH-102126: fix deadlock at shutdown when clearing thread states (#102222) 2023-02-25 12:21:36 +05:30
Mark Shannon 22b8d77b98
GH-100719: Remove redundant `gi_code` field from generator object. (GH-100749) 2023-02-23 10:19:01 +00:00
Mark Shannon 7c106a443f
GH-100982: Restrict `FOR_ITER_RANGE` to a single instruction to allow instrumentation. (GH-101985) 2023-02-22 11:11:57 +00:00
Owain Davies 7346a381be
gh-101903: Remove obsolete undefs for previously removed macros Py_EnterRecursiveCall and Py_LeaveRecursiveCall (#101923) 2023-02-21 11:58:47 +00:00
Irit Katriel 022b44f254
gh-102056: Fix a few bugs in error handling of exception printing code (#102078) 2023-02-20 22:16:09 +00:00
Steve Dower a99eb5cd99
gh-101907: Stop using `_Py_OPCODE` and `_Py_OPARG` macros (GH-101912)
* gh-101907: Removes use of non-standard C++ extension from Include/cpython/code.h

* Make cases_generator correct on Windows
2023-02-20 14:56:48 +00:00
Mark Dickinson b1b375e267
gh-97786: Fix compiler warnings in pytime.c (#101826)
Fixes compiler warnings in pytime.c.
2023-02-19 17:16:11 -08:00
Eclips4 89413bbccb
gh-101967: add a missing error check (#101968) 2023-02-18 00:52:23 +00:00
Eric Snow 4d8959b73a
gh-101758: Add _PyState_AddModule() Back for the Stable ABI (gh-101956)
We're adding the function back, only for the stable ABI symbol and not as any form of API. I had removed it yesterday.

This undocumented "private" function was added with the implementation for PEP 3121 (3.0, 2007) for internal use and later moved out of the limited API (3.6, 2016) and then into the internal API (3.9, 2019). I removed it completely yesterday, including from the stable ABI manifest (where it was added because the symbol happened to be exported). It's unlikely that anyone is using _PyState_AddModule(), especially any stable ABI extensions built against 3.2-3.5, but we're playing it safe.

https://github.com/python/cpython/issues/101758
2023-02-16 14:05:31 -07:00
Eclips4 68bd8c5e2e
gh-101952: Fix possible segfault in `BUILD_SET` opcode (#101958) 2023-02-16 09:46:43 -08:00
Rayyan Ansari 739c026f44
gh-101881: Support (non-)blocking read/write functions on Windows pipes (GH-101882)
* fileutils: handle non-blocking pipe IO on Windows

Handle erroring operations on non-blocking pipes by reading the _doserrno code.
Limit writes on non-blocking pipes that are too large.

* Support blocking functions on Windows

Use the GetNamedPipeHandleState and SetNamedPipeHandleState Win32 API functions to add support for os.get_blocking and os.set_blocking.
2023-02-16 14:52:24 +00:00
penguin_wwy df7ccf6138
gh-101928: fix crash in compiler on multi-line lambda in function call (#101933) 2023-02-16 11:31:41 +00:00
Gregory P. Smith 0b13575e74
gh-99108: Refactor _sha256 & _sha512 into _sha2. (#101924)
This merges their code. They're backed by the same single HACL* static library, having them be a single module simplifies maintenance.

This should unbreak the wasm enscripten builds that currently fail due to linking in --whole-archive mode and the HACL* library appearing twice.

Long unnoticed error fixed: _sha512.SHA384Type was doubly assigned and was actually SHA512Type. Nobody depends on those internal names.

Also rename LIBHACL_ make vars to LIBHACL_SHA2_ in preperation for other future HACL things.
2023-02-15 22:08:20 -08:00
Eric Snow 89ac665891
gh-98627: Add an Optional Check for Extension Module Subinterpreter Compatibility (gh-99040)
Enforcing (optionally) the restriction set by PEP 489 makes sense. Furthermore, this sets the stage for a potential restriction related to a per-interpreter GIL.

This change includes the following:

* add tests for extension module subinterpreter compatibility
* add _PyInterpreterConfig.check_multi_interp_extensions
* add Py_RTFLAGS_MULTI_INTERP_EXTENSIONS
* add _PyImport_CheckSubinterpIncompatibleExtensionAllowed()
* fail iff the module does not implement multi-phase init and the current interpreter is configured to check

https://github.com/python/cpython/issues/98627
2023-02-15 18:16:00 -07:00
Eric Snow 3dea4ba6c1
gh-101758: Fix the wasm Buildbots (gh-101943)
They were broken by gh-101920.

https://github.com/python/cpython/issues/101758
2023-02-15 17:54:05 -07:00
Eric Snow b365d88465
gh-101758: Add a Test For Single-Phase Init Modules in Multiple Interpreters (gh-101920)
The test verifies the behavior of single-phase init modules when loaded in multiple interpreters.

https://github.com/python/cpython/issues/101758
2023-02-15 16:05:07 -07:00
Eric Snow b2fc549278
gh-101758: Clean Up Uses of Import State (gh-101919)
This change is almost entirely moving code around and hiding import state behind internal API.  We introduce no changes to behavior, nor to non-internal API.  (Since there was already going to be a lot of churn, I took this as an opportunity to re-organize import.c into topically-grouped sections of code.)  The motivation is to simplify a number of upcoming changes.

Specific changes:

* move existing import-related code to import.c, wherever possible
* add internal API for interacting with import state (both global and per-interpreter)
* use only API outside of import.c (to limit churn there when changing the location, etc.)
* consolidate the import-related state of PyInterpreterState into a single struct field (this changes layout slightly)
* add macros for import state in import.c (to simplify changing the location)
* group code in import.c into sections
*remove _PyState_AddModule()

https://github.com/python/cpython/issues/101758
2023-02-15 15:32:31 -07:00
Erlend E. Aasland eb0c485b6c
gh-101819: Remove _PyWindowsConsoleIO_Type from the Windows DLL (GH-101904)
Automerge-Triggered-By: GH:erlend-aasland
2023-02-15 05:07:59 -08:00
Mark Shannon c7766245c1
GH-87849: Fix refleak in SEND instruction. (GH-101908)
Fix refleak in SEND instruction.
2023-02-15 12:21:40 +00:00
Eric Snow 096d0097a0
gh-101758: Add a Test For Single-Phase Init Module Variants (gh-101891)
The new test exercises the most important variants for single-phase init extension modules. We also add some explanation about those variants to import.c.

https://github.com/python/cpython/issues/101758
2023-02-14 14:26:03 -07:00
Irit Katriel 81e3aa835c
gh-101799: implement PREP_RERAISE_STAR as an intrinsic function (#101800) 2023-02-14 11:54:13 +00:00
James Lee 95cbb3d908
gh-101810: Remove duplicated st_ino calculation (GH-101811) 2023-02-13 13:49:44 +00:00
Mark Shannon 160f2fe2b9
GH-87849: Simplify stack effect of SEND and specialize it for generators and coroutines. (GH-101788) 2023-02-13 11:24:55 +00:00
Sergey B Kirpichev cb2411886a
gh-101670: typo fix in PyImport_ExtendInittab() (#101723)
Co-authored-by: Eric Snow <ericsnowcurrently@gmail.com>
2023-02-09 18:49:02 +02:00
Guido van Rossum 65b7b6bd23
gh-98831: Use opcode metadata for stack_effect() (#101704)
* Write output and metadata in a single run
  This halves the time to run the cases generator
  (most of the time goes into parsing the input).
* Declare or define opcode metadata based on NEED_OPCODE_TABLES
* Use generated metadata for stack_effect()
* compile.o depends on opcode_metadata.h
* Return -1 from _PyOpcode_num_popped/pushed for unknown opcode
2023-02-08 16:23:19 -08:00
Guido van Rossum 616aec1ff1
gh-98831: Modernize CALL and family (#101508)
Includes a slight improvement to `DECREF_INPUTS()`.
2023-02-08 11:40:10 -08:00
Sergey B Kirpichev 35dd55005e
gh-101670: typo fix in PyImport_AppendInittab() (GH-101672) 2023-02-08 07:49:04 -08:00
David Hewitt 3a88de7a0a
gh-101614: Don't treat python3_d.dll as a Python DLL when checking extension modules for incompatibility (GH-101615) 2023-02-08 14:23:57 +00:00
Mark Shannon feec49c407
GH-101578: Normalize the current exception (GH-101607)
* Make sure that the current exception is always normalized.

* Remove redundant type and traceback fields for the current exception.

* Add new API functions: PyErr_GetRaisedException, PyErr_SetRaisedException

* Add new API functions: PyException_GetArgs, PyException_SetArgs
2023-02-08 09:31:12 +00:00
Guido van Rossum a9f01448a9
gh-98831: Modernize CALL_FUNCTION_EX (#101627)
New generator feature: Move CHECK_EVAL_BREAKER() call to just before DISPATCH().
2023-02-07 20:03:22 -08:00
Guido van Rossum b2b85b5db9
gh-98831: Modernize FORMAT_VALUE (#101628)
Generator update: support balanced parentheses and brackets in conditions and size expressions.
2023-02-07 17:35:55 -08:00
Guido van Rossum aacbdb0c65
gh-98831: Finish the UNPACK_SEQUENCE family (#101666)
New generator feature: Generate useful glue for output arrays, so you can just write values to the output array (no bounds checking). Rewrote UNPACK_SEQUENCE_TWO_TUPLE to use this, and also UNPACK_SEQUENCE_{TUPLE,LIST}.
2023-02-07 15:44:37 -08:00
penguin_wwy 753fc8a5d6
gh-101632: Add the new RETURN_CONST opcode (#101633) 2023-02-07 22:32:21 +00:00
Irit Katriel dec1ab0387
gh-98831: rewrite UNPACK_EX, UNPACK_SEQUENCE, UNPACK_SEQUENCE_TWO_TUPLE in the instruction definition DSL (#101641) 2023-02-07 20:37:43 +00:00
Guido van Rossum d54b8d8fbd
gh-98831: Modernize the FOR_ITER family of instructions (#101626)
Co-authored-by: Irit Katriel <1055913+iritkatriel@users.noreply.github.com>
2023-02-07 08:28:28 -08:00
Matthieu Dartiailh ae62bddaf8
gh-101072: support default and kw default in PyEval_EvalCodeEx for 3.11+ (#101127)
Co-authored-by: Łukasz Langa <lukasz@langa.pl>
2023-02-07 10:34:21 +01:00
Irit Katriel 38752760c9
gh-98831: rewrite COPY and SWAP in the instruction definition DSL (#101620) 2023-02-06 22:45:18 +00:00
Eric Snow 132b3f8302
gh-59956: Partial Fix for GILState API Compatibility with Subinterpreters (gh-101431)
The GILState API (PEP 311) implementation from 2003 made the assumption that only one thread state would ever be used for any given OS thread, explicitly disregarding the case of subinterpreters.  However, PyThreadState_Swap() still facilitated switching between subinterpreters, meaning the "current" thread state (holding the GIL), and the GILState thread state could end up out of sync, causing problems (including crashes).

This change addresses the issue by keeping the two in sync in PyThreadState_Swap().  I verified the fix against gh-99040.

Note that the other GILState-subinterpreter incompatibility (with autoInterpreterState) is not resolved here.

https://github.com/python/cpython/issues/59956
2023-02-06 14:39:25 -07:00
Irit Katriel 433fb3ef08
gh-98831: rewrite MAKE_FUNCTION and BUILD_SLICE in the instruction definition DSL (#101529) 2023-02-03 14:40:45 +00:00
Irit Katriel 04e06e20ee
gh-98831: rewrite SEND, GET_YIELD_FROM_ITER, RETURN_GENERATOR in the instruction definition DSL (#101516) 2023-02-03 11:30:21 +00:00
Irit Katriel 0675b8f032
gh-98831: rewrite RERAISE and CLEANUP_THROW in the instruction definition DSL (#101511) 2023-02-02 10:02:57 +00:00
Guido van Rossum ae9b38f424
gh-98831: Modernize the LOAD_GLOBAL family (#101502) 2023-02-01 13:12:49 -08:00
Irit Katriel b91b42d236
gh-98831: rewrite PUSH_EXC_INFO and conditional jumps in the instruction definition DSL (#101481) 2023-02-01 19:38:06 +00:00
Guido van Rossum 7840ff3cdb
gh-98831: Modernize the LOAD_ATTR family (#101488) 2023-02-01 10:56:52 -08:00
Brandt Bucher 76efcb4093
GH-100288: Skip extra work when failing to specialize LOAD_ATTR (GH-101354) 2023-01-31 13:28:32 -08:00
Irit Katriel 0062f538d9
gh-98831: rewrite BEFORE_ASYNC_WITH and END_ASYNC_FOR in the instruction definition DSL (#101458) 2023-01-31 18:47:50 +00:00
Irit Katriel 29a858b85f
gh-98831: rewrite GET_LEN, GET_ITER, BEFORE_WITH and a few simple opcodes in the instruction definition DSL (#101443) 2023-01-31 10:23:15 +00:00
Irit Katriel af7b2db384
gh-99955: use SUCCESS/ERROR return values in optimizer and assembler. Use RETURN_IF_ERROR where appropriate. Fix a couple of bugs. (#101412) 2023-01-31 00:06:19 +00:00
Dong-hee Na e867c1b753
gh-101400: Fix incorrect lineno in exception message on continue/break which are not in a loop (#101413) 2023-01-30 23:33:54 +00:00
Eric Snow e11fc032a7
gh-59956: Clarify Runtime State Status Expectations (gh-101308)
A PyThreadState can be in one of many states in its lifecycle, represented by some status value.  Those statuses haven't been particularly clear, so we're addressing that here.  Specifically:

* made the distinct lifecycle statuses clear on PyThreadState
* identified expectations of how various lifecycle-related functions relate to status
* noted the various places where those expectations don't match the actual behavior

At some point we'll need to address the mismatches.

(This change also includes some cleanup.)

https://github.com/python/cpython/issues/59956
2023-01-30 12:07:48 -07:00
Mark Shannon c1b1f51cd1
GH-101291: Refactor the `PyLongObject` struct into object header and PyLongValue struct. (GH-101292) 2023-01-30 10:03:04 +00:00
Guido van Rossum f5a3d91b6c
gh-98831: Support conditional effects; use for LOAD_ATTR (#101333) 2023-01-29 17:28:39 -08:00
Irit Katriel c4170c36b0
gh-39615: fix warning on return type mismatch (#101407) 2023-01-29 16:41:27 -08:00
Gregory P. Smith 052f53d65d
gh-39615: Add warnings.warn() skip_file_prefixes support (#100840)
`warnings.warn()` gains the ability to skip stack frames based on code
filename prefix rather than only a numeric `stacklevel=` via a new
`skip_file_prefixes=` keyword argument.
2023-01-27 18:35:14 -08:00
Виталий Дмитриев 37f15a5efa
Fix typos in pystate.c file (#101348) 2023-01-26 15:04:11 -08:00
Irit Katriel b400219df5
gh-98831: rewrite RAISE_VARARGS in the instruction definition DSL (#101306) 2023-01-25 22:29:56 +00:00
Irit Katriel 19f90d6b97
gh-98831: add variable stack effect support to cases generator (#101309) 2023-01-25 20:41:03 +00:00
Guido van Rossum 498598e8c2
Fix some comments in ceval.c and fix lltrace output (#101297)
The comment at the top was rather outdated. :-)

Also added a note about the dangers of dump_stack().
2023-01-24 14:58:18 -08:00
Irit Katriel 1a9d8c750b
gh-98831: rewrite pattern matching opcodes in the instruction definition DSL (#101287) 2023-01-24 22:39:13 +00:00
Mark Shannon f02fa64bf2
GH-100762: Don't call `gen.throw()` in `gen.close()`, unless necessary. (GH-101013)
* Store exception stack depth in YIELD_VALUE's oparg and use it avoid expensive gen.throw() in gen.close() where possible.
2023-01-24 17:25:37 +00:00
Irit Katriel 8c183cddd3
gh-98831: rewrite CHECK_EG_MATCH opcode in the instruction definition DSL (#101269) 2023-01-24 09:43:16 +00:00
Eric Snow 7b20a0f55a
gh-59956: Allow the "Trashcan" Mechanism to Work Without a Thread State (gh-101209)
We've factored out a struct from the two PyThreadState fields. This accomplishes two things:

* make it clear that the trashcan-related code doesn't need any other parts of PyThreadState
* allows us to use the trashcan mechanism even when there isn't a "current" thread state

We still expect the caller to hold the GIL.

https://github.com/python/cpython/issues/59956
2023-01-23 08:30:20 -07:00
Ikko Eltociear Ashimine 4510fbb216
Fix typo in comment in compile.c (#101194) 2023-01-20 18:34:36 +00:00
Nikita Sobolev 8be6992620
gh-101181: Fix `unused-variable` warning in `pystate.c` (#101188)
Co-authored-by: Kumar Aditya <59607654+kumaraditya303@users.noreply.github.com>
2023-01-20 23:31:30 +05:30
Eric Snow f30c94024f
gh-59956: Fix Function Groupings in pystate.c (gh-101172)
This is a follow-up to gh-101161.  The objective is to make it easier to read Python/pystate.c by grouping the functions there in a consistent way.  This exclusively involves moving code around and adding various kinds of comments.

https://github.com/python/cpython/issues/59956
2023-01-19 17:23:53 -07:00
Irit Katriel 9ec9b203ea
gh-101169: reduce the implementation of except* by one bytecode instruction (#101170) 2023-01-19 23:30:49 +00:00
Eric Snow 6036c3e856
gh-59956: Clarify GILState-related Code (gh-101161)
The objective of this change is to help make the GILState-related code easier to understand.  This mostly involves moving code around and some semantically equivalent refactors.  However, there are a also a small number of slight changes in structure and behavior:

* tstate_current is moved out of _PyRuntimeState.gilstate
* autoTSSkey is moved out of _PyRuntimeState.gilstate
* autoTSSkey is initialized earlier
* autoTSSkey is re-initialized (after fork) earlier

https://github.com/python/cpython/issues/59956
2023-01-19 16:04:14 -07:00
Irit Katriel e9ccfe4a63
gh-100712: make it possible to disable specialization (for debugging) (#100713) 2023-01-19 18:14:55 +00:00
Guido van Rossum 1f0d0a432c
GH-98831: Move assorted macros from ceval.h to a new header (#101116) 2023-01-18 10:41:07 -08:00
Guido van Rossum 80e3e3423c
GH-98831: Implement array support in cases generator (#100912)
You can now write things like this:
```
inst(BUILD_STRING, (pieces[oparg] -- str)) { ... }
inst(LIST_APPEND, (list, unused[oparg-1], v -- list, unused[oparg-1])) { ... }
```
Note that array output effects are only partially supported (they must be named `unused` or correspond to an input effect).
2023-01-17 15:59:19 -08:00
Mark Shannon 7b14c2ef19
GH-100982: Add `COMPARE_AND_BRANCH` instruction (GH-100983) 2023-01-16 12:35:21 +00:00
Kumar Aditya f6307d4416
GH-100892: consolidate `HEAD_LOCK/HEAD_UNLOCK` macros (#100953) 2023-01-15 20:39:26 +05:30
Guido van Rossum 5134ef4878
GH-98831: Identify instructions that don't use oparg (#100957)
For these the instr_format field uses IX instead of IB.
Register instructions use IX, IB, IBBX, IBBB, etc.

Also: Include the closing '}' in Block.tokens, for completeness
2023-01-13 17:06:45 -08:00
Steve Dower b5d4347950
gh-86682: Adds sys._getframemodulename as an alternative to using _getframe (GH-99520)
Also updates calls in collections, doctest, enum, and typing modules to use _getframemodulename first when available.
2023-01-13 11:31:06 +00:00
Mark Shannon 6e4e14d98f
GH-100923: Embed jump mask in `COMPARE_OP` oparg (GH-100924) 2023-01-11 20:40:43 +00:00
Guido van Rossum 3f3c78e32f
GH-98831: Refactor instr format code and change to enum (#100895) 2023-01-09 15:53:01 -08:00
Brandt Bucher 61762b9387
GH-100126: Skip incomplete frames in more places (GH-100613) 2023-01-09 12:20:04 -08:00
Nikita Sobolev bc0a686f82
gh-87447: Fix walrus comprehension rebind checking (#100581)
Co-authored-by: Pablo Galindo Salgado <Pablogsal@gmail.com>
Co-authored-by: Shantanu <12621235+hauntsaninja@users.noreply.github.com>
2023-01-08 15:51:29 -07:00
Benjamin Peterson 11f99323c2
Update copyright years to 2023. (gh-100848) 2023-01-08 09:13:25 -06:00
Nnarol 0741da8d28
GH-90829: Fix empty iterable error message in min/max (#31181) 2023-01-08 19:21:20 +05:30
Shantanu a2141882f2
gh-100776: Fix misleading default value in help(input) (#100788) 2023-01-08 13:27:41 +05:30
Carl Meyer 7116030a25
gh-88696: clean up dead argument to compiler_make_closure (GH-100806) 2023-01-07 12:20:48 +00:00
Irit Katriel 15c44789bb
gh-100758: Refactor initialisation of frame headers into a single function (_PyFrame_Initialize) (GH-100759) 2023-01-06 14:55:56 +00:00
Mark Shannon 78068126a1
GH-99005: More intrinsics (GH-100774)
* Remove UNARY_POSITIVE, LIST_TO_TUPLE and ASYNC_GEN_WRAP, replacing them with intrinsics.
2023-01-06 14:47:57 +00:00
Zachary Ware af5149f30b
gh-98831: Regenerate Python/opcode_metadata.h (GH-100778) 2023-01-05 15:47:18 -06:00
Guido van Rossum 14b7f00fdf
GH-98831: Update generate_cases.py: register inst, opcode_metadata.h (#100735)
(These aren't used yet, but may be coming soon,
and it's easier to keep this tool the same between branches.)

Added a sanity check for all this to compile.c.

Co-authored-by: Irit Katriel <iritkatriel@yahoo.com>
2023-01-05 13:01:07 -08:00
Mark Shannon 28187141cc
GH-99005: Add `CALL_INTRINSIC_1` instruction (GH-100771)
* Remove PRINT_EXPR instruction

* Remove STOPITERATION_ERROR instruction

* Remove IMPORT_STAR instruction
2023-01-05 16:05:51 +00:00
Mark Shannon f20c553a45
GH-100288: Remove LOAD_ATTR_METHOD_WITH_DICT instruction. (GH-100753) 2023-01-05 12:20:09 +00:00
Irit Katriel 52017dbe16
gh-100747: some compiler macros use c instead of C to access the compiler (#100748) 2023-01-04 20:19:20 +00:00
Mark Shannon 15aecf8dd7
GH-100719: Remove the `co_nplaincellvars` field from code objects. (GH-100721) 2023-01-04 15:41:39 +00:00
L. A. F. Pereira e6d4440782
gh-100146: Steal references from stack when building a list (#100147)
When executing the BUILD_LIST opcode, steal the references from the stack,
in a manner similar to the BUILD_TUPLE opcode.  Implement this by offloading
the logic to a new private API, _PyList_FromArraySteal(), that works similarly
to _PyTuple_FromArraySteal().

This way, instead of performing multiple stack pointer adjustments while the
list is being initialized, the stack is adjusted only once and a fast memory
copy operation is performed in one fell swoop.
2023-01-03 10:49:49 -08:00
Éric 46521826cb
gh-95778: add doc missing in some places (GH-100627) 2022-12-30 13:21:15 -08:00
Nikita Sobolev 5e1adb4f88
gh-100577: Replace `assert(0)` with `Py_UNREACHABLE` in `symtable.c` (#100579) 2022-12-28 21:01:53 +05:30
Guido van Rossum 08e5594cf3
GH-98831: Modernize a ton of simpler instructions (#100545)
* load_const and load_fast aren't families for now
* Don't decref unmoved names
* Modernize GET_ANEXT
* Modernize GET_AWAITABLE
* Modernize ASYNC_GEN_WRAP
* Modernize YIELD_VALUE
* Modernize POP_EXCEPT (in more than one way)
* Modernize PREP_RERAISE_STAR
* Modernize LOAD_ASSERTION_ERROR
* Modernize LOAD_BUILD_CLASS
* Modernize STORE_NAME
* Modernize LOAD_NAME
* Modernize LOAD_CLASSDEREF
* Modernize LOAD_DEREF
* Modernize STORE_DEREF
* Modernize COPY_FREE_VARS (mark it as done)
* Modernize LIST_TO_TUPLE
* Modernize LIST_EXTEND
* Modernize SET_UPDATE
* Modernize SETUP_ANNOTATIONS
* Modernize DICT_UPDATE
* Modernize DICT_MERGE
* Modernize MAP_ADD
* Modernize IS_OP
* Modernize CONTAINS_OP
* Modernize CHECK_EXC_MATCH
* Modernize IMPORT_NAME
* Modernize IMPORT_STAR
* Modernize IMPORT_FROM
* Modernize JUMP_FORWARD (mark it as done)
* Modernize JUMP_BACKWARD (mark it as done)
2022-12-27 17:11:03 -08:00
Nikita Sobolev bdfb694386
gh-100357: Convert several functions in `bltinsmodule` to AC (#100358) 2022-12-24 06:45:47 -08:00
Sebastian Berg 474220e3a5
gh-99947: Ensure unreported errors are chained for SystemError during import (GH-99946) 2022-12-23 15:43:19 -08:00
Raymond Hettinger 5d84966cce
GH-100425: Improve accuracy of builtin sum() for float inputs (GH-100426) 2022-12-23 14:35:58 -08:00
Ken Jin 36d358348d
Revert "gh-100288: Specialise LOAD_ATTR_METHOD for managed dictionaries (GH-100289)" (#100468)
This reverts commit c3c7848a48.
2022-12-24 01:48:43 +08:00
Ken Jin c3c7848a48
gh-100288: Specialise LOAD_ATTR_METHOD for managed dictionaries (GH-100289) 2022-12-24 00:26:42 +08:00
Irit Katriel 2659036c75
GH-100459: fix copy-paste errors in specialization stats (GH-100460) 2022-12-23 14:42:24 +00:00
Brandt Bucher 09edde95f4
GH-99554: Trim trailing whitespace (GH-100435)
Automerge-Triggered-By: GH:brandtbucher
2022-12-22 12:13:51 -08:00
penguin_wwy a02161286a
GH-99770: Make the correct call specialization fail kind show up in the stats (GH-99771) 2022-12-22 12:44:59 +00:00
Pieter Eendebak 2b82c36f17
gh-99761: Add `_PyLong_IsPositiveSingleDigit` function to check for single digit integers (#100064) 2022-12-22 17:00:18 +05:30
Brandt Bucher 3c033a2e6f
GH-99554: Pack location tables more effectively (GH-99556) 2022-12-21 16:41:18 -08:00
Dennis Sweeney c18d831188
gh-100188: Reduce misses in BINARY_SUBSCR_(LIST/TUPLE)_INT (#100189)
Don't specialize if the index is negative.
2022-12-20 15:46:16 -05:00
Jeremy Paige 1cf3d78c92
Correct CVE-2020-10735 documentation (#100306) 2022-12-18 12:41:21 +05:30
colorfulappl efbb1eb9f5
gh-99240: Reset pointer to NULL when the pointed memory is freed in argument parsing (#99890)
Co-authored-by: Kumar Aditya <59607654+kumaraditya303@users.noreply.github.com>
Co-authored-by: Erlend E. Aasland <erlend.aasland@protonmail.com>
2022-12-17 12:07:44 +05:30
Guido van Rossum 9cdd2fa63b
GH-98831: Add DECREF_INPUTS(), expanding to DECREF() each stack input (#100205)
The presence of this macro indicates that a particular instruction
may be considered for conversion to a register-based format
(see https://github.com/faster-cpython/ideas/issues/485).

An invariant (currently unchecked) is that `DEOPT_IF()` may only
occur *before* `DECREF_INPUTS()`, and `ERROR_IF()` may only occur
*after* it. One reason not to check this is that there are a few
places where we insert *two* `DECREF_INPUTS()` calls, in different
branches of the code. The invariant checking would have to be able
to do some flow control analysis to understand this.

Note that many instructions, especially specialized ones,
can't be converted to use this macro straightforwardly.
This is because the generator currently only generates plain
`Py_DECREF(variable)` statements, and cannot generate
things like `_Py_DECREF_SPECIALIZED()` let alone deal with
`_PyList_AppendTakeRef()`.
2022-12-16 20:45:55 -08:00
Brandt Bucher 9076455d1b
GH-90043: Handle NaNs in COMPARE_OP_FLOAT_JUMP (GH-100278) 2022-12-16 10:18:31 -08:00
Mark Shannon d4052d835b
Improve stats presentation for calls. (GH-100274) 2022-12-16 15:43:04 +00:00
Mark Shannon 289c1126dd
Better stats for `LOAD_ATTR` and `STORE_ATTR` (GH-100295)
* Don't attempt to specialize for LOAD_ATTR on instance if class has attribute

* Improvement to LOAD_ATTR and STORE_ATTR specialization stats.
2022-12-16 15:41:23 +00:00
Mark Shannon 48e352a241
Move stats for the method cache into the `Py_STAT` machinery (GH-100255) 2022-12-15 09:45:03 +00:00
Carl Meyer bdd86741be
GH-100222: fix typo _py_set_opocde -> _py_set_opcode (GH-100259)
Typo introduced in #100223.

Automerge-Triggered-By: GH:brandtbucher
2022-12-14 16:39:00 -08:00
Itamar Ostricher ae83c78215
GH-100000: Cleanup and polish various watchers code (GH-99998)
* Initialize `type_watchers` array to `NULL`s
* Optimize code watchers notification
* Optimize func watchers notification
2022-12-14 19:14:16 +00:00
Mark Shannon 5693f45b19
Assorted minor fixes for specialization stats. (GH-100219) 2022-12-14 15:50:02 +00:00
Mark Shannon 6997e77bdf
GH-100222: Redefine _Py_CODEUNIT as a union to clarify structure of code unit. (GH-100223) 2022-12-14 11:12:53 +00:00
Irit Katriel 985a71032b
gh-99955: undef ERROR and SUCCESS before redefining (fixes sanitizer warning) (#100215) 2022-12-13 13:55:10 +00:00
Eric Snow 5eb28bca9f
gh-81057: Move Signal-Related Globals to _PyRuntimeState (gh-100085)
https://github.com/python/cpython/issues/81057
2022-12-12 16:50:19 -07:00
Michael Droettboom 1583c6e326
GH-100143: Improve collecting pystats for parts of runs (GH-100144)
* pystats off by default

* Add -Xpystats flag

* Always dump pystats, even if turned off
2022-12-12 14:50:43 +00:00
Irit Katriel e4ea33b178
gh-99955: standardize return values of functions in compiler's code-gen (#100010) 2022-12-12 14:22:15 +00:00
chgnrdv 3221b0de67
gh-96715 Remove redundant NULL check in `profile_trampoline` function (#96716)
Closes #96715
2022-12-12 17:29:27 +05:30
Eric Snow 8d0bd93ae2
gh-81057: Fix the wasm32-wasi Buildbot (gh-100139)
The build was broken by gh-100084.

https://github.com/python/cpython/issues/81057
2022-12-09 10:17:54 -07:00
Mark Shannon fb713b2183
GH-98522: Add version number to code objects. (GH-98525)
* Add version number to code object for better versioning of functions.

* Improves specialization for closures and list comprehensions.
2022-12-09 12:18:45 +00:00
Ken Jin 748c6c0921
GH-100110: Specialize FOR_ITER for tuples (GH-100109)
* Specialize FOR_ITER for tuples
2022-12-09 10:27:01 +00:00
Eric Snow 1160001b34
gh-81057: Move Threading-Related Globals to _PyRuntimeState (#100084)
https://github.com/python/cpython/issues/81057
2022-12-08 17:50:58 -07:00
Eric Snow bc8cdf8c3d
gh-81057: Move Ceval Trampoline Globals to _PyRuntimeState (gh-100083)
https://github.com/python/cpython/issues/81057
2022-12-08 17:17:20 -07:00
Guido van Rossum 1cfa704f64
GH-98831: Generate things in the input order (#100123)
This makes it easier to see what changed in the generated code
when converting an instruction to super or macro.
2022-12-08 15:54:07 -08:00
Eric Snow 8a3f06c54b
gh-81057: Move time Globals to _PyRuntimeState (gh-100122)
https://github.com/python/cpython/issues/81057
2022-12-08 16:46:09 -07:00
Guido van Rossum c85be734d1
GH-98831: Typed stack effects, and more instructions converted (#99764)
Stack effects can now have a type, e.g. `inst(X, (left, right -- jump/uint64_t)) { ... }`.

Instructions converted to the non-legacy format:

* COMPARE_OP
* COMPARE_OP_FLOAT_JUMP
* COMPARE_OP_INT_JUMP
* COMPARE_OP_STR_JUMP
* STORE_ATTR
* DELETE_ATTR
* STORE_GLOBAL
* STORE_ATTR_INSTANCE_VALUE
* STORE_ATTR_WITH_HINT
* STORE_ATTR_SLOT, and complete the store_attr family
* Complete the store_subscr family: STORE_SUBSCR{,DICT,LIST_INT}
  (STORE_SUBSCR was alread half converted,
  but wasn't using cache effects yet.)
* DELETE_SUBSCR
* PRINT_EXPR
* INTERPRETER_EXIT (a bit weird, ends in return)
* RETURN_VALUE
* GET_AITER (had to restructure it some)
  The original had mysterious `SET_TOP(NULL)` before `goto error`.
  I assume those just account for `obj` having been decref'ed,
  so I got rid of them in favor of the cleanup implied by `ERROR_IF()`.
* LIST_APPEND (a bit unhappy with it)
* SET_ADD (also a bit unhappy with it)

Various other improvements/refactorings as well.
2022-12-08 13:31:27 -08:00
Eric Snow 91a8e002c2
gh-81057: Move More Globals to _PyRuntimeState (gh-100092)
https://github.com/python/cpython/issues/81057
2022-12-07 15:56:31 -07:00
Brandt Bucher b72014c783
GH-99729: Unlink frames before clearing them (GH-100030) 2022-12-06 14:01:38 +00:00
Eric Snow 530cc9dbb6
gh-99741: Implement Multi-Phase Init for the _xxsubinterpreters Module (gh-99742)
_xxsubinterpreters is an internal module used for testing.

https://github.com/python/cpython/issues/99741
2022-12-05 13:40:20 -07:00
Serhiy Storchaka a87c46eab3
bpo-15999: Accept arbitrary values for boolean parameters. (#15609)
builtins and extension module functions and methods that expect boolean values for parameters now accept any Python object rather than just a bool or int type. This is more consistent with how native Python code itself behaves.
2022-12-03 11:52:21 -08:00
Guido van Rossum acf9184e6b
GH-98831: Support cache effects in super- and macro instructions (#99601) 2022-12-02 19:57:30 -08:00
Eric Snow 0547a981ae
gh-99741: Clean Up the _xxsubinterpreters Module (gh-99940)
This cleanup up resolves a few subtle bugs and makes the implementation for multi-phase init much cleaner.

https://github.com/python/cpython/issues/99741
2022-12-02 11:36:57 -07:00
Irit Katriel ab02262cd0
gh-99955: standardize return values of functions in assembler and optimizer. (#99956) 2022-12-02 17:43:10 +00:00
Eric Snow b4f3505549
gh-99741: Fix the Cross-Interpreter Data API (gh-99939)
There were some minor issues that showed up while I was working on porting _xxsubinterpreters to multi-phase init. This fixes them.

https://github.com/python/cpython/issues/99741
2022-12-02 10:39:17 -07:00
Itamar Ostricher 3c137dc613
GH-91054: Add code object watchers API (GH-99859)
* Add API to allow extensions to set callback function on creation and destruction of PyCodeObject

Co-authored-by: Ye11ow-Flash <janshah@cs.stonybrook.edu>
2022-12-02 17:28:27 +00:00
Serhiy Storchaka 787764219f
gh-89189: More compact range iterator (GH-27986) 2022-11-30 23:04:30 +02:00
Irit Katriel 18a6967544
GH-99877) 2022-11-30 16:16:54 +00:00
Irit Katriel ac12e3941f
gh-87092: move all localsplus preparation into separate function called from assembler stage (GH-99869) 2022-11-30 12:37:30 +00:00
Kumar Aditya fe17d35313
GH-81057: remove static state from suggestions.c (#99411) 2022-11-30 16:55:16 +05:30
colorfulappl 0da728387c
gh-64490: Fix bugs in argument clinic varargs processing (#32092) 2022-11-24 20:56:50 +01:00
Irit Katriel ae185fdcca
gh-99708: fix bug where compiler crashes on if expression with an empty body block (GH-99732) 2022-11-24 10:59:07 +00:00
Victor Stinner 81f7359f67
gh-99537: Use Py_SETREF(var, NULL) in C code (#99687)
Replace "Py_DECREF(var); var = NULL;" with "Py_SETREF(var, NULL);".
2022-11-23 14:57:50 +01:00
Guido van Rossum 8f18ac04d3
GH-98831: Add `macro` and `op` and their implementation to DSL (#99495)
Newly supported interpreter definition syntax:
- `op(NAME, (input_stack_effects -- output_stack_effects)) { ... }`
- `macro(NAME) = OP1 + OP2;`

Also some other random improvements:
- Convert `WITH_EXCEPT_START` to use stack effects
- Fix lexer to balk at unrecognized characters, e.g. `@`
- Fix moved output names; support object pointers in cache
- Introduce `error()` method to print errors
- Introduce read_uint16(p) as equivalent to `*p`

Co-authored-by: Brandt Bucher <brandtbucher@gmail.com>
2022-11-22 16:04:57 -08:00
Skip Montanaro d4cf192826
gh-88226: Emit TARGET labels in Python/ceval.c when debugging, even if computed gotos aren't enabled (GH-98265)
Keep target labels when debugging, but don't warn about lack of use.

Co-authored-by: Eryk Sun <eryksun@gmail.com>
2022-11-22 22:13:54 +02:00
Victor Stinner 135ec7cefb
gh-99537: Use Py_SETREF() function in C code (#99657)
Fix potential race condition in code patterns:

* Replace "Py_DECREF(var); var = new;" with "Py_SETREF(var, new);"
* Replace "Py_XDECREF(var); var = new;" with "Py_XSETREF(var, new);"
* Replace "Py_CLEAR(var); var = new;" with "Py_XSETREF(var, new);"

Other changes:

* Replace "old = var; var = new; Py_DECREF(var)"
  with "Py_SETREF(var, new);"
* Replace "old = var; var = new; Py_XDECREF(var)"
  with "Py_XSETREF(var, new);"
* And remove the "old" variable.
2022-11-22 13:39:11 +01:00
mpage 3db0a21f73
gh-91053: Add an optional callback that is invoked whenever a function is modified (#98175) 2022-11-22 13:06:44 +01:00
Victor Stinner cb2ef8b2ac
gh-99578: Fix refleak in _imp.create_builtin() (#99642)
Fix a reference bug in _imp.create_builtin() after the creation of
the first sub-interpreter for modules "builtins" and "sys".
2022-11-21 12:14:54 +01:00
Ronald Oussoren cdde29dde9
gh-99337: Fix compile errors with gcc 12 on macOS (#99470)
Fix a number of compile errors with GCC-12 on macOS:

1. In pylifecycle.c the compile rejects _Pragma within a declaration
2. posixmodule.c was missing a number of ..._RUNTIME macros for non-clang on macOS
3. _ctypes assumed that __builtin_available is always present on macOS
2022-11-21 10:50:20 +01:00
Guido van Rossum 4f5e1cb00a
GH-98831: Refactor and fix cases generator (#99526)
Also complete cache effects for BINARY_SUBSCR family.
2022-11-17 17:06:07 -08:00
Brandt Bucher b629fdd88a
GH-99298: Clean up attribute specializations (GH-99398) 2022-11-17 15:09:18 -08:00
Brandt Bucher 8555dee5ae
GH-98686: Get rid of BINARY_OP_GENERIC and COMPARE_OP_GENERIC (GH-99399) 2022-11-17 11:36:57 -08:00
Brandt Bucher 6f8b0e781c
Add a macro for "inlining" new frames (GH-99490) 2022-11-17 11:36:03 -08:00
Steve Dower 5fdd49dc65
gh-99377: Revert audit events for thread state creation and free, because the GIL is not properly held at these times (GH-99543) 2022-11-17 00:24:16 +00:00
Eric Snow 9db1e17c80
gh-81057: Move the global Dict-Related Versions to _PyRuntimeState (gh-99497)
We also move the global func version.

https://github.com/python/cpython/issues/81057
2022-11-16 10:37:29 -07:00
Victor Stinner 8211cf5d28
gh-99300: Replace Py_INCREF() with Py_NewRef() (#99530)
Replace Py_INCREF() and Py_XINCREF() using a cast with Py_NewRef()
and Py_XNewRef().
2022-11-16 18:34:24 +01:00
Steve Dower 19c1462e8d
gh-99377: Add audit events for thread creation and clear (GH-99378) 2022-11-16 17:15:52 +00:00
Eric Snow 01fa907aa8
gh-81057: Move contextvars-related Globals to _PyRuntimeState (gh-99400)
This is part of the effort to consolidate global variables, to make them easier to manage (and make it easier to later move some of them to PyInterpreterState).

https://github.com/python/cpython/issues/81057
2022-11-16 09:54:28 -07:00
Eric Snow 5f55067e23
gh-81057: Move More Globals in Core Code to _PyRuntimeState (gh-99516)
https://github.com/python/cpython/issues/81057
2022-11-16 09:37:14 -07:00
Guido van Rossum e37744f289
GH-98831: Implement basic cache effects (#99313) 2022-11-15 19:59:19 -08:00
Hood Chatham 86a49e0673
gh-99460 Emscripten trampolines on optimized METH_O and METH_NOARGS code paths (#99461) 2022-11-15 09:53:39 -08:00
Eric Snow 3c57971a2d
gh-81057: Move Globals in Core Code to _PyRuntimeState (gh-99496)
This is the first of several changes to consolidate non-object globals in core code.

https://github.com/python/cpython/issues/81057
2022-11-15 09:45:11 -07:00
Victor Stinner f13f466474
gh-99300: Use Py_NewRef() in Python/Python-ast.c (#99499)
Replace Py_INCREF() and Py_XINCREF() with Py_NewRef() and
Py_XNewRef() in Python/Python-ast.c.

Update Parser/asdl_c.py to regenerate code.
2022-11-15 10:29:56 +01:00
Kumar Aditya dc3e4350a5
GH-99205: remove `_static` field from `PyThreadState` and `PyInterpreterState` (GH-99385) 2022-11-14 16:35:37 -08:00
Eric Snow e874c2f198
gh-81057: Move the Remaining Import State Globals to _PyRuntimeState (gh-99488)
https://github.com/python/cpython/issues/81057
2022-11-14 15:56:16 -07:00
Steve Dower 4e4b13e8f6
gh-87604: Avoid publishing list of active per-interpreter audit hooks via the gc module (GH-99373) 2022-11-14 21:39:18 +00:00
Eric Snow a088290f9d
gh-81057: Move Global Variables Holding Objects to _PyRuntimeState. (gh-99487)
This moves nearly all remaining object-holding globals in core code (other than static types).

https://github.com/python/cpython/issues/81057
2022-11-14 13:50:56 -07:00
Irit Katriel a3ac9232f8
gh-87092: expose the compiler's codegen to python for unit tests (GH-99111) 2022-11-14 13:56:40 +00:00
Batuhan Taskaya 57be545959
gh-99103: Normalize specialized traceback anchors against the current line (GH-99145)
Automerge-Triggered-By: GH:isidentical
2022-11-12 15:37:25 -08:00
Irit Katriel fb844e1931
gh-98762: Fix locations of match sub-patterns (GH-98775) 2022-11-12 18:42:17 +00:00
Eric Snow 7f3a4b967c
gh-81057: Move PyImport_Inittab to _PyRuntimeState (gh-99402)
We actually don't move PyImport_Inittab.  Instead, we make a copy that we keep on _PyRuntimeState and use only that after Py_Initialize().  We also prevent folks from modifying PyImport_Inittab (the best we can) after that point.

https://github.com/python/cpython/issues/81057
2022-11-11 17:06:05 -07:00
Eric Snow 67807cfc87
gh-81057: Move the Allocators to _PyRuntimeState (gh-99217)
The global allocators were stored in 3 static global variables: _PyMem_Raw, _PyMem, and _PyObject.  State for the "small block" allocator was stored in another 13.  That makes a total of 16 global variables. We are moving all 16 to the _PyRuntimeState struct as part of the work for gh-81057.  (If PEP 684 is accepted then we will follow up by moving them all to PyInterpreterState.)

https://github.com/python/cpython/issues/81057
2022-11-11 16:30:46 -07:00
Eric Snow f531b6879b
gh-81057: Add PyInterpreterState.static_objects (gh-99397)
As we consolidate global variables, we find some objects that are almost suitable to add to _PyRuntimeState.global_objects, but have some small/sneaky bit of per-interpreter state (e.g. a weakref list). We're adding PyInterpreterState.static_objects so we can move such objects there. (We'll removed the _not_used field once we've added others.)

https://github.com/python/cpython/issues/81057
2022-11-11 14:24:18 -07:00
Eric Snow dd36b71fa6
gh-81057: Move the Extension Modules Cache to _PyRuntimeState (gh-99355)
We also move the closely related max_module_number and add comments documenting the group of struct members.

https://github.com/python/cpython/issues/81057
2022-11-11 14:16:28 -07:00
Irit Katriel 3dd6ee2c00
gh-99254: remove all unused consts from code objects (GH-99255) 2022-11-11 10:53:43 +00:00
Guido van Rossum 694cdb24a6
GH-98831: Remove all remaining DISPATCH() calls from bytecodes.c (#99271)
Also mark those opcodes that have no stack effect as such.

Co-authored-by: Brandt Bucher <brandtbucher@gmail.com>
2022-11-10 10:50:57 -08:00
Brandt Bucher 00ee6d506e
GH-99298: Don't perform jumps before error handling (GH-99299) 2022-11-10 08:46:56 -08:00
Mark Shannon 1e197e63e2
GH-96421: Insert shim frame on entry to interpreter (GH-96319)
* Adds EXIT_INTERPRETER instruction to exit PyEval_EvalDefault()

* Simplifies RETURN_VALUE, YIELD_VALUE and RETURN_GENERATOR instructions as they no longer need to check for entry frames.
2022-11-10 12:34:57 +00:00
Brandt Bucher 9d69284169
GH-99257: Check the owner's type when specializing slots (GH-99258) 2022-11-10 11:50:34 +00:00
Victor Stinner 2e343fc465
gh-99300: Use Py_NewRef() in Python/ceval.c (#99318)
Replace Py_INCREF() and Py_XINCREF() with Py_NewRef() and
Py_XNewRef() in Python/ceval.c and related files.
2022-11-10 11:25:33 +01:00
Victor Stinner 231d83b724
gh-99300: Use Py_NewRef() in Python/ directory (#99317)
Replace Py_INCREF() and Py_XINCREF() with Py_NewRef() and
Py_XNewRef() in C files of the Python/ directory.

Update Parser/asdl_c.py to regenerate Python/Python-ast.c.
2022-11-10 11:23:36 +01:00
Victor Stinner d8f239d86e
gh-99300: Use Py_NewRef() in Python/ directory (#99302)
Replace Py_INCREF() and Py_XINCREF() with Py_NewRef() and
Py_XNewRef() in C files of the Python/ directory.
2022-11-10 09:03:39 +01:00
Brandt Bucher 283ab0e1c0
GH-99205: Mark new interpreters and threads as non-static (GH-99268) 2022-11-09 13:55:20 -08:00
Brandt Bucher c7f5708714
GH-98686: Get rid of "adaptive" and "quick" instructions (GH-99182) 2022-11-09 10:50:09 -08:00
Eric Snow 52f91c642b
gh-90868: Adjust the Generated Objects (gh-99223)
We do the following:

* move the generated _PyUnicode_InitStaticStrings() to its own file
* move the generated _PyStaticObjects_CheckRefcnt() to its own file
* include pycore_global_objects.h in extension modules instead of pycore_runtime_init.h

These changes help us avoid including things that aren't needed.

https://github.com/python/cpython/issues/90868
2022-11-08 10:03:03 -07:00
Guido van Rossum f1a654648b
GH-98831: Simple input-output stack effects for bytecodes.c (#99120) 2022-11-08 08:22:56 -08:00
Nikita Sobolev e56e33d271
gh-72719: Remove asyncore and asynchat modules (#96580)
Remove modules asyncore and asynchat, which were deprecated by PEP 594.

Co-authored-by: Jelle Zijlstra <jelle.zijlstra@gmail.com>
2022-11-08 16:48:58 +01:00
Mark Shannon 4a1c58d504
GH-96793: Specialize FOR_ITER for generators. (GH-98772) 2022-11-07 14:49:51 +00:00
Guido van Rossum 7dcd28eb41
GH-98831: Implement super-instruction generation (#99084)
Co-authored-by: C.A.M. Gerlach <CAM.Gerlach@Gerlach.CAM>
2022-11-06 09:40:47 -08:00
Pablo Galindo Salgado 99e2e60cb2
gh-99139: Improve NameError error suggestion for instances (#99140) 2022-11-06 13:52:06 +00:00
Guido van Rossum d04899abb0
GH-99104: Update headers for bytecodes.c and generate_cases.py (#99112)
Also tweak the labels near the end of bytecodes.c.
2022-11-04 17:40:43 -07:00
Brandt Bucher 6a8d3c57af
GH-98686: Fix compiler warning for HAS_ARG (GH-99106) 2022-11-04 16:54:32 -07:00
Mark Shannon 7a020b8e5b
GH-98831: Add some macros definitions to bytecodes.c to reduce IDE warnings. (#99093) 2022-11-04 16:37:53 -07:00
Guido van Rossum c885623e9f
GH-98831: Auto-generate PREDICTED() macro calls (#99102)
Co-authored-by: Brandt Bucher <brandtbucher@gmail.com>
2022-11-04 15:30:17 -07:00
Victor Stinner 0faa0ba240
gh-92584: Remove the distutils package (#99061)
Remove the distutils package. It was deprecated in Python 3.10 by PEP
632 "Deprecate distutils module". For projects still using distutils
and cannot be updated to something else, the setuptools project can
be installed: it still provides distutils.

* Remove Lib/distutils/ directory
* Remove test_distutils
* Remove references to distutils
* Skip test_check_c_globals and test_peg_generator since they use
  distutils
2022-11-03 19:27:27 +01:00
Victor Stinner b07f546ea3
gh-98978: Fix Py_SetPythonHome(NULL) (#99066)
Fix use-after-free in Py_SetPythonHome(NULL), Py_SetProgramName(NULL)
and _Py_SetProgramFullPath(NULL) function calls.

Issue reported by Benedikt Reinartz.
2022-11-03 18:34:32 +01:00
Mark Shannon f4adb97506
GH-96793: Implement PEP 479 in bytecode. (GH-99006)
* Handle converting StopIteration to RuntimeError in bytecode.

* Add custom instruction for converting StopIteration into RuntimeError.
2022-11-03 11:38:51 +00:00
Guido van Rossum 41bc101dd6
GH-98831: "Generate" the interpreter (#98830)
The switch cases (really TARGET(opcode) macros) have been moved from ceval.c to generated_cases.c.h. That file is generated from instruction definitions in bytecodes.c (which impersonates a C file so the C code it contains can be edited without custom support in e.g. VS Code).

The code generator lives in Tools/cases_generator (it has a README.md explaining how it works). The DSL used to describe the instructions is a work in progress, described in https://github.com/faster-cpython/ideas/blob/main/3.12/interpreter_definition.md.

This is surely a work-in-progress. An easy next step could be auto-generating super-instructions.

**IMPORTANT: Merge Conflicts**

If you get a merge conflict for instruction implementations in ceval.c, your best bet is to port your changes to bytecodes.c. That file looks almost the same as the original cases, except instead of `TARGET(NAME)` it uses `inst(NAME)`, and the trailing `DISPATCH()` call is omitted (the code generator adds it automatically).
2022-11-02 21:31:26 -07:00
Brandt Bucher 276d77724f
GH-98686: Quicken everything (GH-98687) 2022-11-02 10:42:57 -07:00
Irit Katriel df84b7b0bc
gh-87092: remove unused SET_LOC/UNSET_LOC macros (GH-98914) 2022-11-02 15:52:11 +00:00
Irit Katriel 6d683d8525
gh-87092: do not allocate PyFutureFeatures dynamically (GH-98913) 2022-11-02 15:13:07 +00:00
Brett Cannon 9711265182
gh-98925: Lower marshal recursion depth for WASI (GH-98938)
For wasmtime 2.0, the stack depth cost is 6% higher. This causes the default max `marshal` recursion depth to blow the stack.

As the default marshal depth is 2000 and Windows is set to 1000, split the difference and choose 1500 for WASI to be safe.
2022-11-01 15:51:05 -07:00
Guido van Rossum 7640ede177
Rename JUMP_TO_INSTRUCTION to GO_TO_INSTRUCTION (#98934)
This reduces confusion between jumps at the bytecode level
(e.g. JUMPTO(), JUMPBY(), and various JUMP_*() opcodes)
and jumps in the C code (which are 'goto' statements).
2022-10-31 19:53:32 -07:00
Eric Snow 4702552885
gh-98610: Adjust the Optional Restrictions on Subinterpreters (GH-98618)
Previously, the optional restrictions on subinterpreters were: disallow fork, subprocess, and threads.  By default, we were disallowing all three for "isolated" interpreters.  We always allowed all three for the main interpreter and those created through the legacy `Py_NewInterpreter()` API.

Those settings were a bit conservative, so here we've adjusted the optional restrictions to: fork, exec, threads, and daemon threads.  The default for "isolated" interpreters disables fork, exec, and daemon threads.  Regular threads are allowed by default.  We continue always allowing everything For the main interpreter and the legacy API.

In the code, we add `_PyInterpreterConfig.allow_exec` and  `_PyInterpreterConfig.allow_daemon_threads`.  We also add `Py_RTFLAGS_DAEMON_THREADS` and `Py_RTFLAGS_EXEC`.
2022-10-31 12:35:54 -07:00
Irit Katriel 39448adc9d
gh-98811: use full source location to simplify __future__ imports error checking. This also fixes an incorrect error offset. (GH-98812) 2022-10-31 13:08:03 +00:00
Dennis Sweeney 87b5fd9d9d
gh-98879: Remove unreachable error case from COMPARE_OP_STR_JUMP (GH-98882)
Thanks to PEP 623 changes, the comparison cannot fail.
2022-10-30 19:07:11 -04:00
Nick Coghlan 05e48865be
gh-96853: Restore test coverage for Py_Initialize(Ex) (GH-98212)
* As most of `test_embed` now uses `Py_InitializeFromConfig`, add
  a specific test case to cover `Py_Initialize` (and `Py_InitializeEx`)
* Rename `_testembed` init helper to clarify the API used
* Add a `PyConfig_Clear` call in `Py_InitializeEx` to make
  the code more obviously correct (it already didn't leak as
  none of the dynamically allocated config fields were being
  populated, but it's clearer if the wrappers follow the
  documented API usage guidelines)
2022-10-30 22:01:30 +10:00
Dennis Sweeney fbcafa6eee
gh-98789: Fix FOR_ITER assert on big-endian (GH-98792)
Fix FOR_ITER assertion syntax
2022-10-28 14:42:39 +08:00
Erlend E. Aasland 723ebe76e7
gh-96143: Improve perf profiler docs (#96445) 2022-10-27 14:06:48 +01:00
Mark Shannon 22863df7ca
GH-96793: Change `FOR_ITER` to not pop the iterator on exhaustion. (GH-96801)
Change FOR_ITER to have the same stack effect regardless of whether it branches or not.
Performance is unchanged as FOR_ITER (and specialized forms jump over the cleanup code).
2022-10-27 11:55:03 +01:00
Eric Snow f32369480d
gh-98608: Change _Py_NewInterpreter() to _Py_NewInterpreterFromConfig() (gh-98609)
(see https://github.com/python/cpython/issues/98608)

This change does the following:

1. change the argument to a new `_PyInterpreterConfig` struct
2. rename the function to `_Py_NewInterpreterFromConfig()`, inspired by `Py_InitializeFromConfig()` (takes a `_PyInterpreterConfig`  instead of `isolated_subinterpreter`)
3. split up the boolean `isolated_subinterpreter` into the corresponding multiple granular settings
   * allow_fork
   * allow_subprocess
   * allow_threads
4. add `PyInterpreterState.feature_flags` to store those settings
5. add a function for checking if a feature is enabled on an opaque `PyInterpreterState *`
6. drop `PyConfig._isolated_interpreter`

The existing default (see `Py_NewInterpeter()` and `Py_Initialize*()`) allows fork, subprocess, and threads and the optional "isolated" interpreter (see the `_xxsubinterpreters` module) disables all three.  None of that changes here; the defaults are preserved.

Note that the given `_PyInterpreterConfig` will not be used outside `_Py_NewInterpreterFromConfig()`, nor preserved.  This contrasts with how `PyConfig` is currently preserved, used, and even modified outside `Py_InitializeFromConfig()`.  I'd rather just avoid that mess from the start for `_PyInterpreterConfig`.  We can preserve it later if we find an actual need.

This change allows us to follow up with a number of improvements (e.g. stop disallowing subprocess and support disallowing exec instead).

(Note that this PR adds "private" symbols.  We'll probably make them public, and add docs, in a separate change.)
2022-10-26 11:16:30 -06:00
Neil Schemenauer de6981680b
gh-90716: add _pylong.py module (#96673)
Add Python implementations of certain longobject.c functions. These use
asymptotically faster algorithms that can be used for operations on
integers with many digits. In those cases, the performance overhead of
the Python implementation is not significant since the asymptotic
behavior is what dominates runtime. Functions provided by this module
should be considered private and not part of any public API.

Co-author: Tim Peters <tim.peters@gmail.com>
Co-author: Mark Dickinson <dickinsm@gmail.com>
Co-author: Bjorn Martinsson
2022-10-25 22:00:50 -07:00
Pablo Galindo Salgado 7cfbb49fcd
gh-91058: Add error suggestions to 'import from' import errors (#98305) 2022-10-25 23:56:59 +01:00
Pablo Galindo Salgado 1f737edb67
gh-96143: Move the perf trampoline files to the Python directory (#98675) 2022-10-25 23:34:22 +01:00
Irit Katriel a1a8828e42
gh-98461: Fix location of RETURN_VALUE in async generator bytecode. compiler_jump_if no longer needs a pointer to the loc. (GH-98494) 2022-10-25 12:26:26 +01:00
Dennis Sweeney 39bc70e267
gh-97912: Avoid quadratic behavior when adding LOAD_FAST_CHECK (GH-97952)
* The compiler analyzes the usage of the first 64 local variables all at once using bit masks.

* Local variables beyond the first 64 are only partially analyzed, achieving linear time.
2022-10-20 18:27:41 -04:00
Irit Katriel 4ec9ed8fde
gh-98461: Fix source location in comprehensions bytecode (GH-98464) 2022-10-20 16:58:37 +01:00
chgnrdv 1f369ad07f
gh-98354: Add unicode check for 'name' attribute in _imp_create_builtin (GH-98412)
Fixes #98354
2022-10-19 17:25:10 -07:00
Victor Stinner a8fe4bbd6b
gh-98257: Make _PyEval_SetTrace() reentrant (#98258)
Make sys.setprofile() and sys.settrace() functions reentrant.  They
can no long fail with: RuntimeError("Cannot install a trace function
while another trace function is being installed").

Make _PyEval_SetTrace() and _PyEval_SetProfile() functions reentrant,
rather than detecting and rejecting reentrant calls. Only delete the
reference to function arguments once the new function is fully set,
when a reentrant call is safe. Call also _PySys_Audit() earlier.
2022-10-20 00:31:47 +02:00
Eric Snow 9c8dde0fa5
gh-98417: Store int_max_str_digits on the Interpreter State (GH-98418) 2022-10-19 13:27:46 -07:00