Commit Graph

4195 Commits

Author SHA1 Message Date
Alex Waygood ee78d01a61
gh-104146: Argument clinic: remove unused methods and variables (#107608) 2023-08-04 00:17:17 +00:00
Erlend E. Aasland 1cd479c6d3
gh-104683: Rework Argument Clinic error handling (#107551)
Introduce ClinicError, and use it in fail(). The CLI runs main(),
catches ClinicError, formats the error message, prints to stderr
and exits with an error.

As a side effect, this refactor greatly improves the accuracy of
reported line numbers in case of error.

Also, adapt the test suite to work with ClinicError.

Co-authored-by: Alex Waygood <Alex.Waygood@Gmail.com>
2023-08-03 00:00:06 +00:00
Erlend E. Aasland bcdd307231
gh-104683: Make Argument Clinic template strings class level members (#107556)
The motivation for this change is to clean up the output_templates()
method a little bit, as it accounts for ~10% of the lines of code in
clinic.py; removing some clutter helps readability.
2023-08-02 21:37:36 +02:00
Irit Katriel dd693d6320
gh-105481: simplify definition of pseudo ops in Lib/opcode.py (#107561) 2023-08-02 18:16:57 +01:00
Erlend E. Aasland 9ff7b4af13
gh-107559: Argument Clinic: complain about non-ASCII chars in param docstrings (#107560)
Previously, only function docstrings were checked for non-ASCII characters.
Also, improve the warn() message.

Co-authored-by: Alex Waygood <Alex.Waygood@Gmail.com>
2023-08-02 12:40:23 +00:00
Erlend E. Aasland 439466a02b
gh-104146: Remove dead code from Argument Clinic (#107555) 2023-08-02 09:22:17 +00:00
Erlend E. Aasland b4d8897781
gh-104683: Argument Clinic: Refactor and simplify 'add docstring' states (#107550)
Introduce docstring_append() helper, and use it for both parameter and
function docstrings. Remove docstring fixup from
do_post_block_processing_cleanup(); instead, make sure no fixup is needed.

Co-authored-by: Alex Waygood <Alex.Waygood@Gmail.com>
2023-08-01 23:32:27 +00:00
Alex Waygood 818c83cf81
gh-104683: Argument clinic: cleanup `DLSParser` `state_foo` methods (#107543) 2023-08-01 21:42:39 +01:00
Alex Waygood 030f6b1e84
gh-104683: Argument clinic: remove the `LandMine` class (#107541) 2023-08-01 21:10:54 +01:00
Irit Katriel 2bd04d4234
gh-105481: combine regen-opcode-targets with regen-opcode to avoid calculating the specialized opcodes in two places (#107540) 2023-08-01 21:05:48 +01:00
Irit Katriel 6ef8f8ca88
gh-105481: the ENABLE_SPECIALIZATION flag does not need to be generated by the build script, or exposed in opcode.py (#107534) 2023-08-01 17:05:00 +00:00
Erlend E. Aasland 49f238e78c
gh-107467: Restructure Argument Clinic command-line interface (#107469)
- Use ArgumentParser.error() to handle CLI errors
- Put the entire CLI in main()
- Rework ClinicExternalTest to call main() instead of using subprocesses

Co-authored-by: AlexWaygood <alex.waygood@gmail.com>
2023-08-01 18:24:23 +02:00
Charlie Zhao 3979150a0d
gh-106263: Fix segfault in `signaldict_repr` in `_decimal` module (#106270)
Co-authored-by: sunmy2019 <59365878+sunmy2019@users.noreply.github.com>
2023-07-30 13:58:54 +05:30
Alex Waygood 5113ed7a2b
gh-104683: Argument clinic: pass `clinic` as a parameter where possible (#107435) 2023-07-29 19:47:42 +01:00
Alex Waygood 6c74b2f669
gh-104683: Argument clinic: Make the `filename` parameter to `Clinic` required (#107439) 2023-07-29 19:46:52 +01:00
Alex Waygood 87de2fbb1e
gh-104050: Argument clinic: enable mypy's `--warn-return-any` setting (#107405) 2023-07-29 13:39:21 +01:00
Alex Waygood 55ed85e49c
gh-104050: Argument clinic: complete type annotations (#107399) 2023-07-28 21:27:21 +01:00
Alex Waygood ecc05e23a1
gh-106368: Argument clinic: Fix minor bug in `state_modulename_name` (#107387) 2023-07-28 19:10:45 +01:00
Erlend E. Aasland 3a1d819ebc
gh-104629: Build _testclinic extension module on Windows (#104723) 2023-07-28 18:31:30 +02:00
Charlie Zhao a43cc3fa1f
gh-106078: Isolate `decimal` module (#107287) 2023-07-28 11:28:14 +05:30
Alex Waygood c2b1689abc
gh-104683: Argument clinic: cleanup `state_modulename_name()` (#107340) 2023-07-27 22:51:18 +01:00
Eric Snow 8bdae1424b
gh-101524: Only Use Public C-API in the _xxsubinterpreters Module (gh-107359)
The _xxsubinterpreters module should not rely on internal API.  Some of the functions it uses were recently moved there however.  Here we move them back (and expose them properly).
2023-07-27 15:30:16 -06:00
Eric Snow b72947a8d2
gh-106931: Intern Statically Allocated Strings Globally (gh-107272)
We tried this before with a dict and for all interned strings.  That ran into problems due to interpreter isolation.  However, exclusively using a per-interpreter cache caused some inconsistency that can eliminate the benefit of interning.  Here we circle back to using a global cache, but only for statically allocated strings.  We also use a more-basic _Py_hashtable_t for that global cache instead of a dict.

Ideally we would only have the global cache, but the optional isolation of each interpreter's allocator means that a non-static string object must not outlive its interpreter.  Thus we would have to store a copy of each such interned string in the global cache, tied to the main interpreter.
2023-07-27 13:56:59 -06:00
Irit Katriel d77d973335
gh-105481: remove dependency of _inline_cache_entries on opname (#107339) 2023-07-27 14:15:25 +01:00
Kumar Aditya 71b3eda02c
remove outdated `_asyncio` globals from globals-to-fix.tsv (#107334) 2023-07-27 07:24:44 +00:00
Erlend E. Aasland 2ad699002e
Docs: Argument Clinic: Restructure "Basic concepts and usage" (#106981)
Split "Basic concepts and usage" into:

- Reference
  - Terminology
  - CLI reference

- Background
  - Basic concepts

Co-authored-by: Adam Turner <9087854+AA-Turner@users.noreply.github.com>
Co-authored-by: Alex Waygood <Alex.Waygood@Gmail.com>
Co-authored-by: Ezio Melotti <ezio.melotti@gmail.com>
2023-07-26 22:08:43 +00:00
Alex Waygood 5aa6964a5c
gh-104050: Argument clinic: Annotate `str_converter_key()` (#107294) 2023-07-26 21:12:18 +01:00
Alex Waygood f9bcdf2368
gh-104050: Argument clinic: Complete `get_destination_buffer` annotations (#107293) 2023-07-26 21:11:15 +01:00
Pablo Galindo Salgado da8f87b7ea
gh-107015: Remove async_hacks from the tokenizer (#107018) 2023-07-26 16:34:15 +01:00
Alex Waygood 14d074e1fb
gh-104683: Argument Clinic: Make most arguments to `Class` and `Function` required (#107289) 2023-07-26 11:54:03 +01:00
Alex Waygood c362678dd2
gh-104050: Argument clinic: improve typing around `parse_arg()` methods (#107288) 2023-07-26 11:51:24 +01:00
Alex Waygood 33838fedf7
gh-104050: Argument clinic: annotate `post_parsing()` and `cleanup()` (#107225) 2023-07-25 23:33:03 +01:00
Alex Waygood ee9010784c
gh-104050: Argument clinic: more misc typing improvements (#107264) 2023-07-25 22:08:52 +01:00
Alex Waygood d9e34db993
gh-104050: Argument clinic: more misc typing coverage improvements (#107210) 2023-07-25 08:49:07 +00:00
Alex Waygood dbfe73837d
gh-104050: Argument clinic: improve typing around adding C converters (#107209) 2023-07-25 09:18:19 +01:00
Alex Waygood 4bbf071635
gh-104050: Argument Clinic: Annotate `CRenderData.__init__` (#107207)
Argument Clinic: Annotate `CRenderData.__init__`
2023-07-24 21:38:50 +00:00
Alex Waygood d93fe5afe2
gh-104050: Argument clinic: misc improvements to type annotation coverage (#107206) 2023-07-24 21:23:58 +00:00
Alex Waygood ac2d85a174
gh-104050: Argument clinic: annotate `format_docstring()` (#107200) 2023-07-24 20:29:50 +00:00
Alex Waygood db17529f80
gh-104050: Argument Clinic: annotate `main()` (#107192) 2023-07-24 19:32:38 +00:00
Guido van Rossum 032f480909
Thoroughly refactor the cases generator (#107151)
This mostly extracts a whole bunch of stuff out of generate_cases.py into separate files, but there are a few other things going on here.

- analysis.py: `Analyzer` etc.
- instructions.py: `Instruction` etc.
- flags.py: `InstructionFlags`, `variable_used`, `variable_used_unspecialized`
- formatting.py: `Formatter` etc.
- Rename parser.py to parsing.py, to avoid conflict with stdlib parser.py
- Blackify most things
- Fix most mypy errors
- Remove output filenames from Generator state, add them to `write_instructions()` etc.
- Fix unit tests
2023-07-24 09:38:23 -07:00
Erlend E. Aasland 3aeffc0d8f
gh-104050: Argument Clinic: Increase typing coverage (#107074)
Co-authored-by: Alex Waygood <Alex.Waygood@Gmail.com>
2023-07-22 22:30:42 +00:00
Victor Stinner 22422e9d1a
gh-106320: Remove private _PyInterpreterID C API (#107053)
Move the private _PyInterpreterID C API to the internal C API: add a
new pycore_interp_id.h header file.

Remove Include/interpreteridobject.h and
Include/cpython/interpreteridobject.h header files.
2023-07-22 19:31:55 +00:00
Erlend E. Aasland c5adf26b18
gh-104050: Argument Clinic: Annotate the BufferSeries class (#106935)
Co-authored-by: Alex Waygood <Alex.Waygood@Gmail.com>
2023-07-22 12:46:42 +02:00
Erlend E. Aasland 3372bcba98
gh-106970: Fix Argument Clinic 'destination <name> clear' command (#106972)
Add test for the 'destination <name> clear' command,
and the 'destination' directive in general.

Fix two bugs in 'destination <name> clear' command:

1. The text attribute of the allocator is called 'text', not '_text'
2. Return after processing the 'clear' command,
   instead of proceeding directly to the fail().
2023-07-22 09:43:13 +02:00
Erlend E. Aasland d81b4f8ff8
gh-104050: Argument Clinic: Annotate the IndentStack class (#106934) 2023-07-20 23:19:11 +00:00
Guido van Rossum 1218910860
gh-105540: Show source files relative to root (#106927)
This restores a corner case: when the generator is run with working directory set to Tools/cases_generator, the source filenames listed in the generated provenance header should be relative to the repo root directory.
2023-07-20 23:08:52 +00:00
Erlend E. Aasland 1d5a625eb0
gh-104050: Argument Clinic: Increase CConverter typing coverage (#106932)
Co-authored-by: Alex Waygood <Alex.Waygood@Gmail.com>
2023-07-20 22:45:02 +00:00
Erlend E. Aasland 42c6300d4f
gh-104050: Argument Clinic: Annotate CLanguage.render_option_group_parsing() (#106929)
Co-authored-by: Alex Waygood <Alex.Waygood@Gmail.com>
2023-07-20 21:33:33 +00:00
Brandt Bucher 8f4de57699
GH-106701: Move _PyUopExecute to Python/executor.c (GH-106924) 2023-07-20 20:37:19 +00:00
Irit Katriel 9c81fc2dbe
gh-105481: do not auto-generate pycore_intrinsics.h (#106913) 2023-07-20 17:46:04 +01:00
Brandt Bucher 214a25dd81
GH-104584: Miscellaneous fixes for -Xuops (GH-106908) 2023-07-20 16:35:39 +00:00
Charlie Zhao a1620dd7b7
gh-106078: Prepare to isolate decimal module (#106880)
* move signal_map to global_state

* move cond_map to global_state
2023-07-20 09:24:35 +00:00
Irit Katriel 40f3f11a77
gh-105481: Generate the opcode lists in dis from data extracted from bytecodes.c (#106758) 2023-07-18 19:42:44 +01:00
Guido van Rossum 1e36ca63f9
Small fixes to code generator (#106845)
These repair nits I found in PR gh-106798 (issue gh-106797) and in PR gh-106716 (issue gh-106706).
2023-07-18 01:30:41 +00:00
Erlend E. Aasland 00e52acebd
gh-104683: Argument Clinic: Modernise parse_special_symbol() (#106837)
Co-authored-by: Alex Waygood <Alex.Waygood@Gmail.com>
2023-07-17 22:37:11 +00:00
Guido van Rossum 8e9a1a0322
gh-106603: Make uop struct a triple (opcode, oparg, operand) (#106794) 2023-07-17 12:12:33 -07:00
Guido van Rossum 2b94a05a0e
gh-106581: Add 10 new opcodes by allowing `assert(kwnames == NULL)` (#106707)
By turning `assert(kwnames == NULL)` into a macro that is not in the "forbidden" list, many instructions that formerly were skipped because they contained such an assert (but no other mention of `kwnames`) are now supported in Tier 2. This covers 10 instructions in total (all specializations of `CALL` that invoke some C code):
- `CALL_NO_KW_TYPE_1`
- `CALL_NO_KW_STR_1`
- `CALL_NO_KW_TUPLE_1`
- `CALL_NO_KW_BUILTIN_O`
- `CALL_NO_KW_BUILTIN_FAST`
- `CALL_NO_KW_LEN`
- `CALL_NO_KW_ISINSTANCE`
- `CALL_NO_KW_METHOD_DESCRIPTOR_O`
- `CALL_NO_KW_METHOD_DESCRIPTOR_NOARGS`
- `CALL_NO_KW_METHOD_DESCRIPTOR_FAST`
2023-07-17 11:02:58 -07:00
Erlend E. Aasland 036bb73656
gh-104050: Improve Argument Clinic type annotation coverage (#106810)
Add various missing annotations in the following classes:
- BlockPrinter
- CConverter
- CLanguage
- FormatCounterFormatter
- Language
- _TextAccumulator

Co-authored-by: Alex Waygood <Alex.Waygood@Gmail.com>
2023-07-17 13:47:08 +02:00
Dong-hee Na 48956cc60e
gh-106797: Remove warning logs from Python/generated_cases.c.h (gh-106798) 2023-07-17 09:09:11 +09:00
Erlend E. Aasland 383dcbebcd
gh-104050: Argument Clinic: Annotate Clinic.parse() (#106760)
Co-authored-by: Alex Waygood <Alex.Waygood@Gmail.com>
2023-07-17 00:04:10 +00:00
Nikita Sobolev c41320701b
gh-105540: Convert `pytest` tests of `cases_generator` to regular tests (#106713) 2023-07-16 16:05:24 -07:00
Kevin Diem cc25ca16ee
gh-106706: Streamline family syntax in cases generator DSL (#106716)
From `family(opname, STRUCTSIZE) = OPNAME + SPEC1 + ... +  SPECn;`
to `family(OPNAME, STRUCTSIZE) = SPEC1 + ... + SPECn;`
2023-07-16 08:16:34 -07:00
Erlend E. Aasland bbf6297985
gh-104050: Argument Clinic: Annotate BlockParser (#106750) 2023-07-15 10:11:32 +00:00
Irit Katriel 6a70edf24c
gh-105481: expose opcode metadata via the _opcode module (#106688) 2023-07-14 18:41:52 +01:00
Erlend E. Aasland 7c95345e4f
gh-104050: Argument Clinic: Annotate `output_templates()` (#106732)
Co-authored-by: AlexWaygood <alex.waygood@gmail.com>
2023-07-14 12:20:58 +00:00
Alex Waygood 128a6c1d88
gh-104683: Argument clinic: use an enum to describe the different kinds of functions (#106721)
Argument clinic: use an enum to describe the different kinds of functions
2023-07-13 22:54:05 +00:00
Guido van Rossum e6e0ea0113
gh-106701: Move the hand-written Tier 2 uops to bytecodes.c (#106702)
This moves EXIT_TRACE, SAVE_IP, JUMP_TO_TOP, and
_POP_JUMP_IF_{FALSE,TRUE} from ceval.c to bytecodes.c.

They are no less special than before, but this way
they are discoverable o the copy-and-patch tooling.
2023-07-13 12:14:51 -07:00
Mark Shannon 487861c6ae
GH-104909: Split `LOAD_ATTR_INSTANCE_VALUE` into micro-ops (GH-106678) 2023-07-13 16:36:19 +01:00
Erlend E. Aasland 2d43beec22
gh-104050: Argument Clinic: Annotate nested function parser_body() in the CLanguage class (#106699) 2023-07-12 22:49:30 +00:00
Alex Waygood 8aa4beaad0
gh-104683: Argument clinic: modernise `cpp.Monitor` (#106698) 2023-07-12 22:48:36 +00:00
Alex Waygood a180e7a0df
gh-104050: Argument clinic: Annotate the `Destination` class (#106655)
Co-authored-by: Nikita Sobolev <mail@sobolevn.me>
2023-07-12 22:33:47 +01:00
Irit Katriel 2ca008e2b7
gh-105481: move Python/opcode_metadata.h to Include/internal/pycore_opcode_metadata.h (#106673) 2023-07-12 11:30:25 +01:00
Alex Waygood 7ce3ea4906
gh-104683: Argument clinic: Minor readability improvements for `Destination.__init__` (#106652) 2023-07-11 23:08:28 +00:00
Ijtaba Hussain de827322ca
gh-103186: In test_tools.freeze, fetch CONFIG_ARGS from original source directory (#103213)
Fetch CONFIG_ARGS from the original source directory, instead of from
the copied source tree. When "make clean" is executed in the copied
source tree, the build directory is cleared and the configure argument
lookup fails. However, the original source directory still contains this
information.
2023-07-11 22:22:18 +00:00
Alex Waygood d0972c77aa
gh-104050: Argument Clinic: Annotate the `Block` class (#106519) 2023-07-11 21:21:14 +00:00
Guido van Rossum cabd6e8a10
gh-106529: Support JUMP_BACKWARD in Tier 2 (uops) (#106543)
During superblock generation, a JUMP_BACKWARD instruction is translated to either a JUMP_TO_TOP micro-op (when the target of the jump is exactly the beginning of the superblock, closing the loop), or a SAVE_IP + EXIT_TRACE pair, when the jump goes elsewhere.

The new JUMP_TO_TOP instruction includes a CHECK_EVAL_BREAKER() call, so a closed loop can still be interrupted.
2023-07-11 18:08:10 +00:00
Eisuke Kawashima 64c0890b69
wasm: do not use inline comment in .editorconfig (#106610)
It is no longer valid since 0.15.0
https://github.com/editorconfig/specification/blob/v0.15/index.rst#no-inline-comments
2023-07-11 23:55:37 +09:00
Irit Katriel 388b5daa52
gh-106360: remove redundant #ifdef (#106622) 2023-07-11 13:41:50 +01:00
Charlie Zhao e5c32a811c
gh-106078: Move external C-API functions to decimal module global state (#106616) 2023-07-11 16:36:42 +05:30
Guido van Rossum 22988c323a
gh-106529: Implement POP_JUMP_IF_XXX uops (#106551)
- Hand-written uops JUMP_IF_{TRUE,FALSE}.
  These peek at the top of the stack.
  The jump target (in superblock space) is absolute.

- Hand-written translation for POP_JUMP_IF_{TRUE,FALSE},
  assuming the jump is unlikely.
  Once we implement jump-likelihood profiling,
  we can implement the jump-unlikely case (in another PR).

- Tests (including some test cleanup).

- Improvements to len(ex) and ex[i] to expose the whole trace.
2023-07-10 16:04:26 -07:00
Charlie Zhao 93846657a3
gh-106078: Move static variables initialized once to decimal module global state (#106475) 2023-07-10 17:27:25 +05:30
Charlie Zhao 1c9e493462
gh-106078: Move static objects related to `CONTEXTVAR` to the decimal module global state (#106395)
Co-authored-by: Erlend E. Aasland <erlend@python.org>
2023-07-08 13:14:24 +05:30
Guido van Rossum b3648f036e
gh-104584: Allow unspecialized instructions in superblocks (#106497)
This adds several of unspecialized opcodes to superblocks:

TO_BOOL, BINARY_SUBSCR, STORE_SUBSCR,
UNPACK_SEQUENCE, LOAD_GLOBAL, LOAD_ATTR,
COMPARE_OP, BINARY_OP.

While we may not want that eventually, for now this helps finding bugs.

There is a rudimentary test checking for UNPACK_SEQUENCE.

Once we're ready to undo this, that would be simple:
just replace the call to variable_used_unspecialized
with a call to variable_used (as shown in a comment).
Or add individual opcdes to FORBIDDEN_NAMES_IN_UOPS.
2023-07-07 18:03:27 +00:00
Guido van Rossum 11038c56ad
gh-104584: Move super-instruction special-casing to generator (#106500)
Instead of special-casing specific instructions,
we add a few more special values to the 'size' field of expansions,
so in the future we can automatically handle
additional super-instructions in the generator.
2023-07-07 17:42:10 +00:00
Alex Waygood 363f4f99c5
gh-104683: clinic.py: refactor `Parameter` and `Function` as dataclasses (#106477) 2023-07-07 13:10:07 +00:00
Alex Waygood 5548097925
gh-104683: clinic.py: refactor four simple classes as dataclasses (#106476) 2023-07-06 22:17:12 +01:00
Alex Waygood d0c6ba956f
gh-104683: clinic.py: Don't needlessly reimplement `contextlib.redirect_stdout` (#106478)
clinic.py: Don't needlessly reimplement `contextlib.redirect_stdout`
2023-07-06 14:23:02 +01:00
Erlend E. Aasland 9d1d4f9c73
gh-64595: Fix regression in file write logic in Argument Clinic (#106449)
Revert the two commits that introduced the regressions:

 - gh-104152
 - gh-104507
2023-07-05 13:23:22 +02:00
Erlend E. Aasland 7bb9fa5ae4
gh-104050: Partially annotate Argument Clinic CLanguage class (#106437)
Co-authored-by: Alex Waygood <Alex.Waygood@Gmail.com>
2023-07-04 23:07:57 +00:00
Erlend E. Aasland 22087516bc
gh-104050: Annotate Argument Clinic parameter permutation helpers (#106431)
Co-authored-by: Alex Waygood <Alex.Waygood@Gmail.com>
2023-07-04 22:15:10 +00:00
Erlend E. Aasland 110b97c94c
gh-104050: Annotate toplevel functions in clinic.py (#106435)
Co-authored-by: Alex Waygood <Alex.Waygood@Gmail.com>
2023-07-04 22:13:30 +00:00
Mark Shannon 318ea2c72e
GH-106360: Support very basic superblock introspection (#106422)
* Add len() and indexing support to uop superblocks.
2023-07-04 17:23:00 +01:00
Erlend E. Aasland 8f6df5e9cb
gh-106368: Add tests for permutation helpers in Argument Clinic (#106407)
Added new test class PermutationTests()
2023-07-04 11:36:40 +00:00
Erlend E. Aasland b24479dcba
gh-104050: Annotate more Argument Clinic DSLParser state methods (#106376)
Annotate the following methods:

- state_parameter()
- state_parameter_docstring_start()

Co-authored-by: Alex Waygood <Alex.Waygood@Gmail.com>
2023-07-03 22:10:46 +00:00
Erlend E. Aasland 71b40443fe
gh-104683: Modernise Argument Clinic parameter state machine (#106362)
Use enums and pattern matching to make the code more readable.

Co-authored-by: Alex Waygood <Alex.Waygood@Gmail.com>
2023-07-03 21:16:21 +00:00
Charlie Zhao bf06c6825c
gh-106078: Move `context template` to decimal module global state (#106346) 2023-07-04 01:40:01 +05:30
Guido van Rossum 2028a4f6d9
gh-106290: Fix edge cases around uops (#106319)
- Tweak uops debugging output
- Fix the bug from gh-106290
- Rename `SET_IP` to `SAVE_IP` (per https://github.com/faster-cpython/ideas/issues/558)
- Add a `SAVE_IP` uop at the start of the trace (ditto)
- Allow `unbound_local_error`; this gives us uops for `LOAD_FAST_CHECK`, `LOAD_CLOSURE`, and `DELETE_FAST`
- Longer traces
- Support `STORE_FAST_LOAD_FAST`, `STORE_FAST_STORE_FAST`
- Add deps on pycore_uops.h to Makefile(.pre.in)
2023-07-03 20:05:11 +00:00
Erlend E. Aasland 2e2daac2f1
gh-104050: Add more type hints to Argument Clinic DSLParser() (#106354)
Co-authored-by: Alex Waygood <Alex.Waygood@Gmail.com>
2023-07-03 14:03:31 +00:00
Erlend E. Aasland 0da4c883cf
gh-106359: Fix corner case bugs in Argument Clinic converter parser (#106361)
DSLParser.parse_converter() could return unusable kwdicts in some rare cases

Co-authored-by: Alex Waygood <Alex.Waygood@Gmail.com>
2023-07-03 13:14:59 +00:00
Erlend E. Aasland 60e41a0cbb
gh-104146: Remove unused attr 'parameter_indent' from clinic.DLParser (#106358) 2023-07-03 11:00:35 +00:00
Erlend E. Aasland 18fedd04a7
gh-104050: Annotate Argument Clinic DSLParser attributes (#106357) 2023-07-03 10:06:54 +00:00
Victor Stinner 5ccbbe5bb9
gh-106320: Move _PyUnicodeWriter to the internal C API (#106342)
Move also _PyUnicode_FormatAdvancedWriter().

CJK codecs and multibytecodec.c now define the Py_BUILD_CORE_MODULE
macro.
2023-07-03 10:23:43 +02:00
Erlend E. Aasland 569887aa9d
gh-104050: Add more type hints to Argument Clinic DSLParser() (#106343)
Annotate the following method signatures:

- state_dsl_start()
- state_parameter_docstring_start()
- state_parameters_start()

Inverting ignore_line() logic, add type hints (including type guard) to
it, and rename to valid_line().
2023-07-02 23:42:38 +00:00
Charlie Zhao dbefa88b27
gh-106078: Move DecimalException to _decimal state (#106301) 2023-07-02 16:50:40 +00:00
Victor Stinner 46d77610fc
gh-106316: Remove pytime.h header file (#106317)
Remove the "cpython/pytime.h" header file: it only contained private
functions. Move functions to the internal pycore_time.h header file.

Move tests from _testcapi to _testinternalcapi. Rename also test
methods to have the same name than tested C functions.

No longer export these functions:

* _PyTime_Add()
* _PyTime_As100Nanoseconds()
* _PyTime_FromMicrosecondsClamp()
* _PyTime_FromTimespec()
* _PyTime_FromTimeval()
* _PyTime_GetPerfCounterWithInfo()
* _PyTime_MulDiv()
2023-07-01 22:27:18 +00:00
Irit Katriel 200f255411
gh-106149: move unconditional jump direction resolution from optimizer to assembler (#106291) 2023-07-01 11:28:07 +01:00
dependabot[bot] c336b4c14f
build(deps-dev): bump mypy from 1.3.0 to 1.4.1 in /Tools/clinic (#106305)
Bumps [mypy](https://github.com/python/mypy) from 1.3.0 to 1.4.1.
- [Commits](https://github.com/python/mypy/compare/v1.3.0...v1.4.1)

---
updated-dependencies:
- dependency-name: mypy
  dependency-type: direct:development
  update-type: version-update:semver-minor
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2023-07-01 09:54:32 +00:00
Kirill Podoprigora 2062e11501
gh-106267: Add type cast to generated code (#106289) 2023-06-30 12:11:10 -07:00
Guido van Rossum 6e9f83d9ae
GH-106250: Support insts using one cache entry and no oparg (GH-106252) 2023-06-29 13:02:25 -07:00
Charlie Zhao fb0d9b9ac1
gh-106078: Convert `_decimal` types to heap types (#106079)
- Establish global state struct
- Convert static types to heap types and add them to global state:
    * PyDecContextManager_Type
    * PyDecContext_Type
    * PyDecSignalDictMixin_Type
    * PyDec_Type
- Add to global state:
    * PyDecSignalDict_Type
    * DecimalTuple

Co-authored-by: Kumar Aditya <59607654+kumaraditya303@users.noreply.github.com>
Co-authored-by: Erlend E. Aasland <erlend@python.org>
2023-06-29 10:27:20 +00:00
Guido van Rossum 11731434df
gh-104584: Emit macro expansions to opcode_metadata.h (#106163)
This produces longer traces (superblocks?).

Also improved debug output (uop names are now printed instead of numeric opcodes). This would be simpler if the numeric opcode values were generated by generate_cases.py, but that's another project.

Refactored some code in generate_cases.py so the essential algorithm for cache effects is only run once. (Deciding which effects are used and what the total cache size is, regardless of what's used.)
2023-06-28 18:28:07 +00:00
Nikita Sobolev d830c4a944
gh-106200: Remove unused imports (#106201) 2023-06-28 11:55:41 +00:00
Victor Stinner 1f74b9e933
Fix c-analyzer for GCC: ignore LANG env var (#106173)
The c-analyzer doesn't support GCC localized messages, so just unset
the LANG environment variable.
2023-06-28 02:50:51 +00:00
Guido van Rossum 0762775a15
gh-104584: Add #line directives to executor_cases.c.h (#106126) 2023-06-27 06:56:39 -07:00
Guido van Rossum 51fc725117
gh-104584: Baby steps towards generating and executing traces (#105924)
Added a new, experimental, tracing optimizer and interpreter (a.k.a. "tier 2"). This currently pessimizes, so don't use yet -- this is infrastructure so we can experiment with optimizing passes. To enable it, pass ``-Xuops`` or set ``PYTHONUOPS=1``. To get debug output, set ``PYTHONUOPSDEBUG=N`` where ``N`` is a debug level (0-4, where 0 is no debug output and 4 is excessively verbose).

All of this code is likely to change dramatically before the 3.13 feature freeze. But this is a first step.
2023-06-26 19:02:57 -07:00
YAMAMOTO Takashi d8f87cdf94
gh-101538: Add experimental wasi-threads build (#101537)
Co-authored-by: Brett Cannon <brett@python.org>
Co-authored-by: Erlend E. Aasland <erlend.aasland@protonmail.com>
2023-06-22 11:26:10 -07:00
Irit Katriel c01da2896a
gh-105481: refactor instr flag related code into a new InstructionFlags class (#105950) 2023-06-21 23:14:43 +00:00
Victor Stinner 155577de1b
make regen-stdlib-module-names rejects test modules (#105921)
Make sure that sys.stdlib_module_names doesn't contain test modules.
2023-06-20 10:12:44 +02:00
Irit Katriel 33f0a8578b
gh-105481: generate _specializations and _specialized_instructions from bytecodes.c (#105913) 2023-06-19 23:47:04 +01:00
Irit Katriel 14d01262da
gh-105481: remove HAS_ARG, HAS_CONST, IS_JUMP_OPCODE, IS_PSEUDO_OPCODE and replace by their new versions (#105865) 2023-06-17 17:00:16 +01:00
Irit Katriel d1b0297d3e
gh-105481: add HAS_JUMP flag to opcode metadata (#105791) 2023-06-14 23:14:22 +00:00
Guido van Rossum 4caa728b2c
gh-104909: Implement conditional stack effects for macros (#105748) 2023-06-14 20:50:48 +00:00
Guido van Rossum d50930a6a8
gh-105481: Fix types and a bug for pseudos (#105788) 2023-06-14 10:06:50 -07:00
Guido van Rossum 74c2422fa2
Update DSL docs for cases generator (#105753)
* Clarify things around goto error/ERROR_IF a bit
* Remove docs for super-instructions
* Add pseudo; fix heading markup
2023-06-14 08:19:24 -07:00
Mark Shannon 7199584ac8
GH-100987: Allow objects other than code objects as the "executable" of an internal frame. (GH-105727)
* Add table describing possible executable classes for out-of-process debuggers.

* Remove shim code object creation code as it is no longer needed.

* Make lltrace a bit more robust w.r.t. non-standard frames.
2023-06-14 13:46:37 +01:00
Victor Stinner 0316063031
gdb libpython.py: Remove compatibility code (#105739)
Remove compatibility code for Python 2 and early Python 3 versions.

* Remove os_fsencode() reimplementation: use os.fsencode() directly.
  os.fsencode() was added to Python 3.2.
* Remove references to Python 2 and "Python 3": just say "Python".
* Remove outdated u'' string format: use '' instead.
2023-06-14 12:30:22 +02:00
Eric Snow b87d288275
gh-105699: Use a Thread-Local Variable for PKGCONTEXT (gh-105740)
This fixes a race during import. The existing _PyRuntimeState.imports.pkgcontext is shared between interpreters, and occasionally this would cause a crash when multiple interpreters were importing extensions modules at the same time.  To solve this we add a thread-local variable for the value.  We also leave the existing state (and infrequent race) in place for platforms that do not support thread-local variables.
2023-06-13 18:58:23 -06:00
Victor Stinner 457a459c78
gh-98040: Fix importbench: use types.ModuleType() (#105743)
Replace removed imp.new_module(name) with types.ModuleType(name).
2023-06-14 00:32:12 +02:00
Eric Snow 757b402ea1
gh-104812: Run Pending Calls in any Thread (gh-104813)
For a while now, pending calls only run in the main thread (in the main interpreter).  This PR changes things to allow any thread run a pending call, unless the pending call was explicitly added for the main thread to run.
2023-06-13 15:02:19 -06:00
Irit Katriel be2779c0cb
gh-105481: add flags to each instr in the opcode metadata table, to replace opcode.hasarg/hasname/hasconst (#105482) 2023-06-13 21:42:03 +01:00
Guido van Rossum 8da9d1b163
gh-105540: Fix code generator tests (#105707)
This involves expanding PEEK, POKE and JUMPBY macros,
and removing super and register tests (those features no longer exist).
2023-06-12 21:55:15 +00:00
Guido van Rossum 9544948e7e
Remove support for legacy bytecode instructions (#105705)
(A legacy instruction is of the form `instr(FOOBAR)`,
i.e. missing the `(... -- ...)` stack/cache effect annotation.)
2023-06-12 18:19:04 +00:00
Guido van Rossum b9e7dc797d
gh-105229: Remove syntactic support for super-instructions (#105703)
It will not be used again.
2023-06-12 17:47:08 +00:00
Irit Katriel 58f5227d7c
gh-105481: add pseudo-instructions to the bytecodes DSL (#105506) 2023-06-11 22:31:59 +01:00
Guido van Rossum c85be33acc
gh-104610: Stop looking for PREDICT() in the cases generator (#105459)
We no longer use `PREDICT()`, it doesn't have any benefits.
2023-06-07 17:00:10 +00:00
Victor Stinner d1b0015ea8
gh-105407: Remove unused imports in Tools/c-analyzer/ (#105410) 2023-06-06 21:08:48 +00:00
Victor Stinner b8c79b2b98
gh-105407: Remove unused imports in tools (#105409) 2023-06-06 22:51:26 +02:00
Victor Stinner bae415ad02
gh-102304: doc: Add links to Stable ABI and Limited C API (#105345)
* Add "limited-c-api" and "stable-api" references.
* Rename "stable-abi-list" reference to "limited-api-list".
* Makefile: Document files regenerated by "make regen-limited-abi"
* Remove first empty line in generated files:

  - Lib/test/test_stable_abi_ctypes.py
  - PC/python3dll.c
2023-06-06 08:40:32 +00:00
Eric Snow e6373c0d8b
gh-101524: Only Use Public C-API in the _xxsubinterpreters Module (gh-105258)
The _xxsubinterpreters module was meant to only use public API.  Some internal C-API usage snuck in over the last few years (e.g. gh-28969).  This fixes that.
2023-06-02 22:52:33 +00:00
Brett Cannon 70dc2fb973
GH-102404, GH-100956: Document how to do a WASI build (GH-105251)
Also includes a reference shell script to implements what is documented.
2023-06-02 15:15:41 -07:00
Mark Shannon 4bfa01b9d9
GH-104584: Plugin optimizer API (GH-105100) 2023-06-02 11:46:18 +01:00
Victor Stinner cbb9ba844f
gh-92536: Argument Clinic no longer emits PyUnicode_READY() (#105208)
Since Python 3.12, PyUnicode_READY() does nothing and always
returns 0.

Argument Clinic now also checks for .cpp files (PC/_wmimodule.cpp).
2023-06-02 01:31:58 +02:00
Eric Snow 146939306a
gh-104614: Make Sure ob_type is Always Set Correctly by PyType_Ready() (gh-105122)
When I added the relevant condition to type_ready_set_bases() in gh-103912, I had missed that the function also sets tp_base and ob_type (if necessary).  That led to problems for third-party static types.

We fix that here, by making those extra operations distinct and by adjusting the condition to be more specific.
2023-06-01 22:28:31 +00:00
Gregory P. Smith ede89af605
gh-103142: Upgrade binary builds and CI to OpenSSL 1.1.1u (#105174)
Upgrade builds to OpenSSL 1.1.1u.

This OpenSSL version addresses a pile if less-urgent CVEs since 1.1.1t.

The Mac/BuildScript/build-installer.py was already updated.

Also updates _ssl_data_111.h from OpenSSL 1.1.1u, _ssl_data_300.h from 3.0.9, and adds a new _ssl_data_31.h file from 3.1.1 along with the ssl.c code to use it.

Manual edits to the _ssl_data_300.h file prevent it from removing any existing definitions in case those exist in some peoples builds and were important (avoiding regressions during backporting).

backports of this prior to 3.12 will not include the openssl 3.1 header.
2023-06-01 09:42:18 -07:00
Lysandros Nikolaou a241003d04
gh-105191: Cleanup peg generator; keep only necessary files (#105197) 2023-06-01 17:24:15 +02:00
Victor Stinner 7d07e5891d
gh-105156: Cleanup usage of old Py_UNICODE type (#105158)
* refcounts.dat:

  * Remove Py_UNICODE functions.
  * Replace Py_UNICODE argument type with wchar_t.

* _PyUnicode_ToLowercase(), _PyUnicode_ToUppercase(),
  _PyUnicode_ToTitlecase() are no longer deprecated in comments.
  It's no longer needed since they now use Py_UCS4 type, rather than
  the deprecated Py_UNICODE type.
* gdb: Remove unused char_width() method.
2023-06-01 07:18:09 +00:00
Steve Dower ed86e14b16
gh-105146: Update links at end of Windows installer (uninstall/repair) (GH-105147) 2023-05-31 21:42:54 +01:00
Victor Stinner dd29ae26f8
gh-105156: Argument Clinic avoids Py_UNICODE type (#105161)
Argument Clinic now uses "const wchar_t*" type instead of
"const Py_UNICODE*" type for the "Py_UNICODE" format.
2023-05-31 17:52:33 +00:00
Guido van Rossum df396b59af
gh-104909: Split BINARY_OP into micro-ops (#104910)
Co-authored-by: Brandt Bucher <brandtbucher@gmail.com>
2023-05-31 08:09:23 -07:00
Petr Viktorin 0656d23d82
gh-105091: stable_abi.py: Remove "Unixy" check from --all on other platforms (GH-105092) 2023-05-30 17:45:56 +02:00
Zachary Ware a989b73e8e
gh-75552: Remove deprecated tkinter.tix module (GH-104902) 2023-05-27 12:34:19 -05:00
Lysandros Nikolaou afa759fb80
gh-100238: Use setuptools in peg-generator and reenable tests (#104798) 2023-05-26 15:32:03 +02:00
Victor Stinner f66be6b11a
gh-104773: PEP 594: Remove the audioop module (#104937) 2023-05-25 17:59:00 +02:00
Victor Stinner 036da3bd43
gh-104773: PEP 594: Remove the aifc module (#104933)
* Remove .aifc and .aiff test files of Lib/test/audiodata/
* Remove Lib/test/Sine-1000Hz-300ms.aif test file
2023-05-25 17:20:48 +02:00
Steve Dower 569b2b8d61
Improves the Windows MSI test run on PR (GH-104929)
Correctly set the exit code when builds fail
Also build docs as part of the test
2023-05-25 16:17:12 +01:00
Victor Stinner dbc8216f4c
gh-104773: PEP 594: Remove the uu module (#104932)
Doc/license.rst: Keep the UUencode and UUdecode license since it's
also used by the uu codec.
2023-05-25 16:25:27 +02:00
Victor Stinner e4127eaa1e
gh-104773: PEP 594: Remove the crypt module (#104908)
Remove the crypt module and its private _crypt extension, deprecated
in Python 3.11.
2023-05-25 15:45:46 +02:00
Zachary Ware 98c4333e88
gh-104773: Remove the msilib package (GH-104911) 2023-05-24 20:06:00 -05:00
Victor Stinner 087c1a6539
gh-104773: PEP 594: Remove the xdrlib module (#104900)
pickle documentation no longer mentions the XDR format.
2023-05-25 00:40:30 +02:00
Victor Stinner 17e1fe0f9b
gh-104773: PEP 594: Remove the nis module (#104897) 2023-05-25 00:08:36 +02:00
Victor Stinner ded5f1f287
gh-104773: PEP 594: Remove the nntplib module (#104894)
* socket_helper.transient_internet() no longer imports nntplib to
  catch nntplib.NNTPTemporaryError.
* ssltests.py no longer runs test_nntplib.
* "make quicktest" no longer runs test_nntplib.
* WASM: remove nntplib from OMIT_NETWORKING_FILES.
* Remove mentions to nntplib in the email documentation.
2023-05-24 21:15:43 +00:00
Victor Stinner 9d457e1154
gh-104773: PEP 594: Remove the mailcap module (#104867)
Remove Lib/test/mailcap.txt test file.
2023-05-24 14:16:55 +00:00
Victor Stinner a5e0f5b230
gh-104773: PEP 594: Remove the sunau module (#104863)
* Remove Lib/test/audiodata/pluck-*.au files.
* Remove Lib/test/audiotest.au file.
2023-05-24 13:51:59 +00:00
Victor Stinner fc07fe4e37
gh-104773: PEP 594: Remove the ossaudiodev module (#104862)
* Remove ossaudiodev extension in configure.ac and regenerate
  the configure script.
* Remove ossaudiodev in Modules/Setup and Modules/Setup.stdlib.in.
2023-05-24 15:00:51 +02:00
Victor Stinner 08d5923896
gh-104773: PEP 594: Remove cgi and cgitb modules (#104775)
* Replace "cgi" with "!cgi" in the Sphinx documentation to avoid
  warnings on broken references.
* test_pyclbr no longer tests the cgi module.
2023-05-24 09:04:53 +00:00
Victor Stinner ae00b810d1
gh-104780: Remove 2to3 program and lib2to3 module (#104781)
* Remove the Tools/scripts/2to3 script.
* Remove the Lib/test/test_lib2to3/ directory.
* Doc/tools/extensions/pyspecific.py: remove the "2to3fixer" object
  type.
* Makefile and PC/layout/main.py no longer compile lib2to3 grammar
  files.
* Update Makefile for 2to3 removal.
2023-05-23 19:40:02 +02:00
Victor Stinner 9dc476be2d
gh-104773: PEP 594: Remove the telnetlib module (#104778) 2023-05-23 07:09:02 +00:00
Alex Waygood 4b107d86f3
gh-104683: clinic.py: Modernise `parse_converter()` using pattern-matching (#104696) 2023-05-21 23:29:43 +00:00
Erlend E. Aasland 6ba8406cb6
gh-104050: Add more type annotations to Argument Clinic (#104631)
Annotate methods of the following classes:

- class Function
- class Parameter
- class LandMine
2023-05-21 21:24:26 +00:00
Erlend E. Aasland bf6dd8f5fd
gh-104050: Add basic type hints to Argument Clinic clinic class (#104705)
Co-authored-by: Alex Waygood <Alex.Waygood@Gmail.com>
2023-05-21 20:49:34 +00:00
Erlend E. Aasland c482e9a4e6
gh-104050: Annotate Argument Clinic return converters (#104706)
Co-authored-by: Alex Waygood <Alex.Waygood@Gmail.com>
2023-05-21 20:23:14 +00:00
gsallam be0c106789
gh-103295: expose API for writing perf map files (#103546)
Co-authored-by: Aniket Panse <aniketpanse@fb.com>
Co-authored-by: Gregory P. Smith <greg@krypto.org>
Co-authored-by: Carl Meyer <carl@oddbird.net>
2023-05-21 11:12:24 +01:00
Erlend E. Aasland 27a68be77f
gh-104050: Add basic type hints to Argument Clinic DSL parser (#104704)
Co-authored-by: Alex Waygood <Alex.Waygood@Gmail.com>
2023-05-20 21:55:02 +00:00
Alex Waygood 19dd5aa89a
gh-104683: Modernise Tools/clinic/ (#104684)
- Make some string interpolations more readable using f-strings or
  explicit parametrisation
- Remove unneeded open() mode specifiers

Co-authored-by: Erlend E. Aasland <erlend.aasland@protonmail.com>
2023-05-20 20:16:49 +00:00
Alex Waygood 663c049ff7
gh-104683: Modernise `clinic.py` using `str.removeprefix` and `str.removesuffix` (#104685)
Both methods were new in Python 3.9.
2023-05-20 11:08:28 +00:00
Erlend E. Aasland ae147d01a0
gh-104146: Purge dead code from Argument Clinic (#104680)
The following local variables were assigned but never used:

- line 551:  result
- line 1341: groups
- line 1431: default_return_converter
- line 1529: ignore_self
- line 1809: input_checksum
- line 4224: new'

---

Co-authored-by: Alex Waygood <Alex.Waygood@Gmail.com>
2023-05-20 10:52:32 +00:00
Alex Waygood 02b60035ce
gh-104683: Argument clinic: use `dict` over `OrderedDict` (#104647)
For code readability. Instances of `builtins.dict` have been ordered since 3.6, and have been guaranteed by the language to be ordered since Python 3.7. Argument Clinic now requires Python 3.10+.
2023-05-20 11:24:00 +01:00
Erlend E. Aasland 94c8edabd3
gh-104050: Add more type annotations to Argument Clinic (#104628)
Annotate the following:

- methods of class Class
- methods of class Module
- methods of class PythonParser
- function compute_checksum()
- function parse_file()
- global variable unsupported_special_methods
2023-05-18 22:57:26 +00:00
Jelle Zijlstra b9dce3aec4
gh-104549: Set __module__ on TypeAliasType (#104550) 2023-05-18 15:56:15 -07:00
Alex Waygood 1c55e8d007
gh-104050: Improve some typing around `default`s and sentinel values (#104626)
- Convert `unspecified` and `unknown` to be members of a `Sentinels` enum, rather than instances of bespoke classes.
  - An enum feels more idiomatic here, and works better with type checkers.
  - Convert some `==` and `!=` checks for these values to identity checks, which are more idiomatic with sentinels.
  - _Don't_ do the same for `Null`, as this needs to be a distinct type due to its usage in `clinic.py`.
- Use `object` as the annotation for `default` across `clinic.py`. `default` can be literally any object, so `object` is the correct annotation here.

---

Co-authored-by: Erlend E. Aasland <erlend.aasland@protonmail.com>
2023-05-18 21:58:42 +00:00
Erlend E. Aasland 61027c0211
gh-104146: Remove unused vars from Argument Clinic (#104627)
Remove 'in_classes' and 'so_far' from DSLParser.directive_module()
2023-05-18 21:52:59 +00:00
Erlend E. Aasland 2f369cafee
gh-104050: Add type hints to Argument Clinic converter keywords (#104588)
Introduce TypeSet, and use it to annotate the 'accept' keyword of
various C converters. Also add some missing return annotations for
converter init functions.
2023-05-17 21:05:22 +00:00
Erlend E. Aasland 0afc473f50
gh-104050: Add typing to Argument Clinic converters (#104547) 2023-05-16 22:32:11 +00:00
Kumar Aditya 20e994c535
GH-103092: isolate `pyexpat` (#104506) 2023-05-16 20:03:01 +00:00
Nikita Sobolev a454a6651b
gh-104050: Add more annotations to `Tools/clinic.py` (#104544) 2023-05-16 17:18:28 +00:00
Erlend E. Aasland c22fced96c
gh-104050: Don't star-import 'types' in Argument Clinic (#104543) 2023-05-16 12:02:18 +00:00
Erlend E. Aasland 505e2954a9
gh-104050: Add basic typing to CConverter in clinic.py (#104538) 2023-05-16 11:47:35 +00:00
Erlend E. Aasland cca90b6906
gh-64595: Fix write file logic in Argument Clinic (#104507)
Check if any clinic output actually changes any of the output files
before deciding if we should touch the source file.
2023-05-16 12:25:43 +02:00
Erlend E. Aasland 186bf39f5c
gh-101819: Isolate `_io` (#101948)
Co-authored-by: Kumar Aditya <59607654+kumaraditya303@users.noreply.github.com>
Co-authored-by: Victor Stinner <vstinner@python.org>
2023-05-15 09:26:27 +00:00
dependabot[bot] 35bf0916d9
Bump mypy from 1.2.0 to 1.3.0 in /Tools/clinic (#104501)
build(deps-dev): bump mypy from 1.2.0 to 1.3.0 in /Tools/clinic

Bumps [mypy](https://github.com/python/mypy) from 1.2.0 to 1.3.0.
- [Commits](https://github.com/python/mypy/compare/v1.2.0...v1.3.0)

---
updated-dependencies:
- dependency-name: mypy
  dependency-type: direct:development
  update-type: version-update:semver-minor
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2023-05-15 09:17:48 +00:00
Alex Waygood 9d41f83c58
gh-104050: Run mypy on `clinic.py` in CI (#104421)
* Add basic mypy workflow to CI
* Make the type check pass

---------

Co-authored-by: Erlend E. Aasland <erlend.aasland@protonmail.com>
Co-authored-by: Nikita Sobolev <mail@sobolevn.me>
Co-authored-by: Hugo van Kemenade <hugovk@users.noreply.github.com>
2023-05-15 08:49:28 +00:00
Mark Shannon 45f5aa8fc7
GH-103082: Filter LINE events in VM, to simplify tool implementation. (GH-104387)
When monitoring LINE events, instrument all instructions that can have a predecessor on a different line.
Then check that the a new line has been hit in the instrumentation code.
This brings the behavior closer to that of 3.11, simplifying implementation and porting of tools.
2023-05-12 12:21:20 +01:00
Erlend E. Aasland b2c1b4da19
gh-104389: Add 'unused' keyword to Argument Clinic C converters (#104390)
Use the unused keyword param in the converter to explicitly
mark an argument as unused:

    /*[clinic input]
    SomeBaseClass.stubmethod
        flag: bool(unused=True)
    [clinic start generated code]*/
2023-05-12 10:34:00 +02:00
Carl Meyer 77262458fe
gh-87729: improve hit rate of LOAD_SUPER_ATTR specialization (#104270) 2023-05-11 08:08:13 -06:00
penguin_wwy 373bca0cc5
GH-102181: Improve specialization stats for SEND (GH-102182) 2023-05-10 22:40:59 +00:00
Alex Waygood fe694a6db6
gh-90953: Don't use deprecated AST nodes in clinic.py (#104322) 2023-05-09 22:16:22 +01:00
Hugo van Kemenade d513ddee94
Trim trailing whitespace and test on CI (#104275)
Co-authored-by: Alex Waygood <Alex.Waygood@Gmail.com>
2023-05-08 17:03:52 +03:00
Jonathan Protzenko 15665d896b
gh-99108: Replace SHA3 implementation HACL* version (#103597)
Replaces our built-in SHA3 implementation with a verified one from the HACL* project.

This implementation is used when OpenSSL does not provide SHA3 or is not present.

3.11 shiped with a very slow tiny sha3 implementation to get off of the <=3.10 reference implementation that wound up having serious bugs. This brings us back to a reasonably performing built-in implementation consistent with what we've just replaced our other guaranteed available standard hash algorithms with: code from the HACL* project.

---------

Co-authored-by: Gregory P. Smith <greg@krypto.org>
2023-05-07 20:50:04 -07:00
Erlend E. Aasland ac020624b3
gh-64660: Don't hardcode Argument Clinic return converter result variable name (#104200) 2023-05-07 21:55:37 +00:00
Erlend E. Aasland 7a7eaff95c
gh-101819: Port _io.PyBytesIOBuffer_Type to heap type (#104264) 2023-05-07 14:01:27 +00:00
Erlend E. Aasland cab1298a60
gh-101819: Adapt _io.PyWindowsConsoleIO_Type to heap type (#104197) 2023-05-07 11:23:11 +02:00
Erlend E. Aasland 3952379655
gh-101819: Port _io.PyIncrementalNewlineDecoder_Type to heap type (#104249) 2023-05-07 11:20:34 +02:00
Erlend E. Aasland 66558d2a16
gh-104146: Remove unused var 'parser_body_declarations' from clinic.py (#104214) 2023-05-05 21:35:24 +00:00
Tian Gao b979741731
gh-103533: Use PEP 669 APIs for cprofile (GH-103534) 2023-05-05 18:38:47 +01:00
Erlend E. Aasland 45a9e3834a
gh-64595: Argument Clinic: Touch source file if any output file changed (#104152) 2023-05-05 12:38:38 +00:00
Erlend E. Aasland 5245cb64d9
gh-68395: Avoid naming conflicts by mangling variable names in Argument Clinic (#104065)
Add all internally used variable names to CLINIC_PREFIXED_ARGS.
2023-05-05 13:40:18 +02:00
Carl Meyer f5c38382f9
gh-103963: fix 'make regen-opcode' in out-of-tree builds (#104177) 2023-05-04 17:45:56 +00:00
Jelle Zijlstra 04f6733275
gh-102500: Implement PEP 688 (#102521)
Co-authored-by: Kumar Aditya <59607654+kumaraditya303@users.noreply.github.com>
2023-05-04 07:59:46 -07:00
Erlend E. Aasland 9885677b04
gh-104146: Remove unused 'second_pass_replacements' from clinic.py (#104147)
The code that manipulated 'second_pass_replacements' was removed in 2015
with commit 0759f84 (gh-67688, bpo-23500).
2023-05-04 00:11:29 +00:00
Juhi Chandalia 872cbc6132
GH-103963: Make dis display names of args for intrinsics opcodes (#104029) 2023-05-02 19:00:17 -07:00
Barry Warsaw e1f14643dc
gh-98040: Remove just the `imp` module (#98573) 2023-04-28 16:17:58 -07:00
Erlend E. Aasland 8a0c7f1e40
gh-103583: Isolate CJK codec modules (#103869) 2023-04-27 15:02:43 +02:00
Erlend E. Aasland e9c777208f
gh-103092: Isolate _ctypes, part 1 (#103893)
Establish global state and port the following types to heap types:

- DictRemover_Type
- PyCArg_Type
- PyCThunk_Type
- PyCField_Type
- StructParam_Type
2023-04-27 14:57:54 +02:00
Yichen Yan 214e5684c2
gh-103875: Use ascii and latin1 singletons in deepfreeze (#103876) 2023-04-26 11:45:58 +00:00
Eric Snow df3173d28e
gh-101659: Isolate "obmalloc" State to Each Interpreter (gh-101660)
This is strictly about moving the "obmalloc" runtime state from
`_PyRuntimeState` to `PyInterpreterState`.  Doing so improves isolation
between interpreters, specifically most of the memory (incl. objects)
allocated for each interpreter's use.  This is important for a
per-interpreter GIL, but such isolation is valuable even without it.

FWIW, a per-interpreter obmalloc is the proverbial
canary-in-the-coalmine when it comes to the isolation of objects between
interpreters.  Any object that leaks (unintentionally) to another
interpreter is highly likely to cause a crash (on debug builds at
least).  That's a useful thing to know, relative to interpreter
isolation.
2023-04-24 17:23:57 -06:00
Daniel Versoza 01be52e42e
gh-103801: Tools/wasm linting and formatting (#103796)
This PR makes three minor linting adjustments to the `wasm` module
caught by [ruff](https://github.com/charliermarsh/ruff).

<!-- gh-issue-number: gh-103801 -->
* Issue: gh-103801
<!-- /gh-issue-number -->

---------

Co-authored-by: blurb-it[bot] <43283697+blurb-it[bot]@users.noreply.github.com>
2023-04-24 22:42:02 +00:00
Eddie Elizondo ea2c001650
gh-84436: Implement Immortal Objects (gh-19474)
This is the implementation of PEP683

Motivation:

The PR introduces the ability to immortalize instances in CPython which bypasses reference counting. Tagging objects as immortal allows up to skip certain operations when we know that the object will be around for the entire execution of the runtime.

Note that this by itself will bring a performance regression to the runtime due to the extra reference count checks. However, this brings the ability of having truly immutable objects that are useful in other contexts such as immutable data sharing between sub-interpreters.
2023-04-22 13:39:37 -06:00
Pablo Galindo Salgado 1ef61cf71a
gh-102856: Initial implementation of PEP 701 (#102855)
Co-authored-by: Lysandros Nikolaou <lisandrosnik@gmail.com>
Co-authored-by: Batuhan Taskaya <isidentical@gmail.com>
Co-authored-by: Marta Gómez Macías <mgmacias@google.com>
Co-authored-by: sunmy2019 <59365878+sunmy2019@users.noreply.github.com>
2023-04-19 11:18:16 -05:00
Pradyun Gedam ece20dba12
gh-95299: Stop installing setuptools as a part of ensurepip and venv (#101039)
Remove the bundled setuptools wheel from ensurepip, and stop installing setuptools in environments created by venv.

Co-Authored-by: Hugo van Kemenade <hugovk@users.noreply.github.com>
Co-authored-by: C.A.M. Gerlach <CAM.Gerlach@Gerlach.CAM>
Co-authored-by: Oleg Iarygin <oleg@arhadthedev.net>
2023-04-17 23:43:34 -05:00
Erlend E. Aasland 217911ede5
gh-103583: Add codecs and maps to _codecs_* module state (#103540) 2023-04-17 02:41:25 +02:00
Nikita Sobolev 7f3c106503
gh-103326: Remove `Python/importlib.h` (GH-103331)
Co-authored-by: Brett Cannon <brett@python.org>
Co-authored-by: Hugo van Kemenade <hugovk@users.noreply.github.com>
2023-04-12 11:27:14 -07:00
Erlend E. Aasland 52f96d3ea3
gh-103092: Isolate `_collections` (#103093)
Co-authored-by: Kumar Aditya <59607654+kumaraditya303@users.noreply.github.com>
2023-04-12 18:21:28 +05:30
Mark Shannon 411b169281
GH-103082: Implementation of PEP 669: Low Impact Monitoring for CPython (GH-103083)
* The majority of the monitoring code is in instrumentation.c

* The new instrumentation bytecodes are in bytecodes.c

* legacy_tracing.c adapts the new API to the old sys.setrace and sys.setprofile APIs
2023-04-12 12:04:55 +01:00
Benjamin Peterson 96663875b2
Remove redundant words from interpreter_definition.md. (GH-103455) 2023-04-11 15:30:05 -05:00
Irit Katriel 21bea68e2e
gh-91276: remove unused _PyOpcode_RelativeJump (#103156) 2023-04-11 11:20:39 +01:00
Nikita Sobolev 40db5c65b7
gh-103334: Ignore `Tools/c-analyzer/cpython/_parser.py` from `patchcheck` (GH-103335)
I've also added a small comment to `Tools/c-analyzer/cpython/_parser.py` to trigger the `patchcheck` CI. It must pass now.

Automerge-Triggered-By: GH:ericsnowcurrently
2023-04-10 07:58:48 -07:00
Oleg Iarygin 86d2044155
gh-103300: Fix `Popen.wait()` deadlock in patchcheck.py (#103301) 2023-04-09 13:48:53 +05:30
Erlend E. Aasland f329a8bc1e
gh-103092: Isolate `socket` module (#103094) 2023-04-09 06:33:52 +05:30
Hugo van Kemenade b22d021ee6
gh-100176: Tools/iobench: Remove redundant compat code for Python <= 3.2 (#100197)
Co-authored-by: Alex Waygood <Alex.Waygood@Gmail.com>
2023-04-08 12:04:47 +03:00
Erlend E. Aasland c00dcf0e38
gh-103092: Isolate `_pickle` module (#102982)
Co-authored-by: Mohamed Koubaa <koubaa.m@gmail.com>
Co-authored-by: Kumar Aditya <59607654+kumaraditya303@users.noreply.github.com>
2023-04-04 15:38:54 +05:30
Irit Katriel 80163e17d3
gh-87092: move CFG related code from compile.c to flowgraph.c (#103021) 2023-03-31 18:17:59 +01:00
Missoupro 01a49d1745
gh-102110: Add all tools description missed (GH-102625)
Added missing descriptions of some tools.

Automerge-Triggered-By: GH:hugovk
2023-03-30 13:49:07 -07:00
Eric Snow e647dbaded
gh-90110: Bring the whitelists up to date. (gh-103114)
https://github.com/python/cpython/issues/90110
2023-03-29 16:08:40 -06:00
Brett Cannon 0b1d9c44f1
GH-102973: add a dev container (GH-102975)
On content update, builds `python` and the docs. Also adds a Dockerfile that should include everything but autoconf 2.69 that's necessary to build CPython and the entire stdlib on Fedora.

Co-authored-by: Ronald Oussoren <ronaldoussoren@mac.com>
Co-authored-by: Dusty Phillips <dusty@phillips.codes>
2023-03-29 13:28:08 -07:00
Eric Snow ba65a065cf
gh-100227: Move the Dict of Interned Strings to PyInterpreterState (gh-102339)
We can revisit the options for keeping it global later, if desired.  For now the approach seems quite complex, so we've gone with the simpler isolation solution in the meantime.

https://github.com/python/cpython/issues/100227
2023-03-28 12:52:28 -06:00
Brandt Bucher 0444ae2487
GH-100982: Break up COMPARE_AND_BRANCH (GH-102801) 2023-03-23 15:25:09 -07:00
Nikita Sobolev 1ca315538f
gh-102033: Fix syntax error in `Tools/c-analyzer` (GH-102066)
The easiest way to format strings with `{}` meaningful chars is via `%`.
2023-03-22 07:59:32 -07:00
Mark Shannon 7559f5fda9
GH-101291: Rearrange the size bits in PyLongObject (GH-102464)
* Eliminate all remaining uses of Py_SIZE and Py_SET_SIZE on PyLongObject, adding asserts.

* Change layout of size/sign bits in longobject to support future addition of immortal ints and tagged medium ints.

* Add functions to hide some internals of long object, and for setting sign and digit count.

* Replace uses of IS_MEDIUM_VALUE macro with _PyLong_IsCompact().
2023-03-22 14:49:51 +00:00
Eric Snow ad77d16a62
gh-102304: Move _Py_RefTotal to _PyRuntimeState (gh-102543)
The essentially eliminates the global variable, with the associated benefits. This is also a precursor to isolating this bit of state to PyInterpreterState.

Folks that currently read _Py_RefTotal directly would have to start using _Py_GetGlobalRefTotal() instead.

https://github.com/python/cpython/issues/102304
2023-03-20 10:03:04 -06:00
Carl Meyer 174c4bfd0f
gh-102781: fix cwd dependence in cases generator (#102782) 2023-03-17 11:01:10 -06:00
Eric Snow 84e20c689a
gh-102737: Un-ignore ceval.c in the CI globals check (gh-102745)
The tool now allows user-added #LINE preprocessor directives.

https://github.com/python/cpython/issues/102737
2023-03-16 09:26:42 -06:00
Irit Katriel 675b97a6ab
gh-102738: remove from cases generator the code related to register instructions (#102739) 2023-03-15 21:25:31 +00:00
Alex Waygood 215007b576
Exclude `ceval.c` from the C-analyzer tool (#102735)
The "check if generated files are up to date" CI check appears to be currently failing on all PRs (but not on pushes to main) 

See, for example:
- https://github.com/python/cpython/pull/94468
- https://github.com/python/cpython/pull/94468
- https://github.com/python/cpython/pull/102731

This appears to be because the C-analyzer tool doesn't like the `#line` directives introduced in 70185de1ab. I'm advised by the message printed to the terminal in https://github.com/python/cpython/actions/runs/4428706945/jobs/7768216988#step:14:84 that this is the appropriate short-term fix!
2023-03-15 18:19:07 +00:00
Guido van Rossum 70185de1ab
gh-102654: Insert #line directives in generated_cases.c.h (#102669)
This behavior is optional, because in some extreme cases it
may just make debugging harder. The tool defaults it to off,
but it is on in Makefile.pre.in.

Also note that this makes diffs to generated_cases.c.h noisier,
since whenever you insert or delete a line in bytecodes.c,
all subsequent #line directives will change.
2023-03-15 08:37:36 -07:00
Eric Snow 1ff81c0cb6
gh-81057: Add a CI Check for New Unsupported C Global Variables (gh-102506)
This will keep us from adding new unsupported (i.e. non-const) C global variables, which would break interpreter isolation.

FYI, historically it is very uncommon for new global variables to get added. Furthermore, it is rare for new code to break the c-analyzer. So the check should almost always pass unnoticed.

Note that I've removed test_check_c_globals. A test wasn't a great fit conceptually and was super slow on debug builds. A CI check is a better fit.

This also resolves gh-100237.

https://github.com/python/cpython/issues/81057
2023-03-14 10:05:54 -06:00
Kumar Aditya a703f743db
GH-94851: check unicode consistency of static strings in debug mode (#102684) 2023-03-14 15:21:38 +05:30
Eric Snow ca066bdbed
gh-81057: Vendor a Subset of distutils for the c-analyzer Tool (gh-102505)
distutils was removed in November. However, the c-analyzer relies on it. To solve that here, we vendor the parts the tool needs so it can be run against 3.12+. (Also see gh-92584.)

Note that we may end up removing this code later in favor of a solution in common with the peg_generator tool (which also relies on distutils).  At the least, the copy here makes sure the c-analyzer tool works on 3.12+ in the meantime.
2023-03-09 09:50:33 -07:00
JosephSBoyle b097925858
gh-102507 Remove invisible pagebreak characters (#102531)
Co-authored-by: AlexWaygood <alex.waygood@gmail.com>
2023-03-08 13:58:14 +00:00
Eric Snow 8606697f49
gh-90110: Fix the c-analyzer Tool (#102483)
Some incompatible changes had gone in, and the "ignore" lists weren't properly undated. This change fixes that. It's necessary prior to enabling test_check_c_globals, which I hope to do soon.

Note that this does include moving last_resort_memory_error to PyInterpreterState.

https://github.com/python/cpython/issues/90110
2023-03-06 19:40:09 -07:00
Pablo Galindo Salgado f533f216e6
gh-102416: Do not memoize incorrectly loop rules in the parser (#102467) 2023-03-06 14:41:53 +01:00
Jacob Bower 8de59c1bb9
gh-102021 : Allow multiple input files for interpreter loop generator (#102022)
The input files no longer use `-i`.
2023-03-03 20:59:21 -08:00
Guido van Rossum b5ff382433
GH-102305: Expand some macros in generated_cases.c.h (#102309)
* Emit straight stack_pointer[-i] instead of PEEK(i), POKE(i, ...)
* Expand JUMPBY() and NEXTOPARG(), and fix a perf bug
2023-02-28 08:49:35 -08:00
Erlend E. Aasland c3a178398c
gh-102151: Correctly fetch CONFIG_ARGS in Tools/freeze/test/freeze.py (#102152) 2023-02-23 15:09:51 +01:00
Steve Dower a99eb5cd99
gh-101907: Stop using `_Py_OPCODE` and `_Py_OPARG` macros (GH-101912)
* gh-101907: Removes use of non-standard C++ extension from Include/cpython/code.h

* Make cases_generator correct on Windows
2023-02-20 14:56:48 +00:00
Erlend E. Aasland c00faf7943
gh-101819: Adapt _io types to heap types, batch 1 (GH-101949)
Adapt StringIO, TextIOWrapper, FileIO, Buffered*, and BytesIO types.

Automerge-Triggered-By: GH:erlend-aasland
2023-02-20 05:46:20 -08:00
Erlend E. Aasland c1ce0d178f
gh-99138: Isolate _zoneinfo (#99218)
* Convert zone info type to heap type and add it to module state
* Add global variables to module state
2023-02-15 22:58:48 +01:00
Steve Dower 0c6fe81dce
gh-101849: Add upgrade codes for old versions of launcher that ended up with later version numbers (GH-101877) 2023-02-13 20:33:48 +00:00
Gregory P. Smith dfc2e065a2
gh-89792: Limit test_tools freeze test build parallelism based on the number of cores (#101841)
unhardcode freeze test build parallelism. base it on the number of cpus, don't use more than max(2, os.cpu_count()/3).
2023-02-11 22:07:52 -08:00
Gregory P. Smith 1d194235e4
gh-89792: Prevent test_tools from copying 1000M of "source" in freeze test (#101837)
Prevent test_tools from copying 1000M of "source"
    
It doesn't need a git repo, just the checkout.  We skip .git metadata, Doc/build, Doc/venv, and `__pycache__` subdirs, that developers often have in their clients to reduce the size of the source tree copy ten-fold.

This should significantly reduce IO and presumably time on buildbots during this long test.
2023-02-11 20:54:28 -08:00
Erlend E. Aasland 826bf0e695
gh-101277: Finalise isolating itertools (GH-101305)
Add repeat, islice, chain, tee, teedataobject, and batched types to module state.

Automerge-Triggered-By: GH:erlend-aasland
2023-02-10 03:58:14 -08:00
Gregory P. Smith b41c47cd06
gh-101726: Update the OpenSSL version to 1.1.1t (GH-101727)
Fixes CVE-2023-0286 (High) and a couple of Medium security issues.
https://www.openssl.org/news/secadv/20230207.txt
2023-02-09 17:40:51 +00:00
Guido van Rossum 65b7b6bd23
gh-98831: Use opcode metadata for stack_effect() (#101704)
* Write output and metadata in a single run
  This halves the time to run the cases generator
  (most of the time goes into parsing the input).
* Declare or define opcode metadata based on NEED_OPCODE_TABLES
* Use generated metadata for stack_effect()
* compile.o depends on opcode_metadata.h
* Return -1 from _PyOpcode_num_popped/pushed for unknown opcode
2023-02-08 16:23:19 -08:00
Guido van Rossum 616aec1ff1
gh-98831: Modernize CALL and family (#101508)
Includes a slight improvement to `DECREF_INPUTS()`.
2023-02-08 11:40:10 -08:00
Guido van Rossum a9f01448a9
gh-98831: Modernize CALL_FUNCTION_EX (#101627)
New generator feature: Move CHECK_EVAL_BREAKER() call to just before DISPATCH().
2023-02-07 20:03:22 -08:00
Guido van Rossum b2b85b5db9
gh-98831: Modernize FORMAT_VALUE (#101628)
Generator update: support balanced parentheses and brackets in conditions and size expressions.
2023-02-07 17:35:55 -08:00
Guido van Rossum aacbdb0c65
gh-98831: Finish the UNPACK_SEQUENCE family (#101666)
New generator feature: Generate useful glue for output arrays, so you can just write values to the output array (no bounds checking). Rewrote UNPACK_SEQUENCE_TWO_TUPLE to use this, and also UNPACK_SEQUENCE_{TUPLE,LIST}.
2023-02-07 15:44:37 -08:00
Guido van Rossum 694e346a01
gh-98831: Move DSL documentation here from ideas repo (#101629) 2023-02-06 21:03:58 -08:00
Eric Snow c67b00534a
gh-101524: Split Up the _xxsubinterpreters Module (gh-101526)
This is step 1 in potentially dropping all the "channel"-related code. Channels have already been removed from PEP 554.

https://github.com/python/cpython/issues/101524
2023-02-03 18:14:43 -07:00
Irit Katriel b91b42d236
gh-98831: rewrite PUSH_EXC_INFO and conditional jumps in the instruction definition DSL (#101481) 2023-02-01 19:38:06 +00:00
Erlend E. Aasland 2753cf2ed6
gh-101409: Improve generated clinic code for self type checks (#101411) 2023-01-31 21:42:03 +01:00
Guido van Rossum 04ab767d28
gh-98831: Clean up and add cache size static_assert to macro (#101442) 2023-01-30 17:27:51 -08:00
Guido van Rossum 7a3752338a
GH-101369: Allow macros as family members (#101399)
Also check for instructions straddling families (this includes macro parts).
2023-01-30 11:23:57 -08:00
Mark Shannon c1b1f51cd1
GH-101291: Refactor the `PyLongObject` struct into object header and PyLongValue struct. (GH-101292) 2023-01-30 10:03:04 +00:00
Guido van Rossum f5a3d91b6c
gh-98831: Support conditional effects; use for LOAD_ATTR (#101333) 2023-01-29 17:28:39 -08:00
Steve Dower 8d18d1ffd5
gh-99834: Update bundled copy of Tcl/Tk to 8.6.13.0 on Windows (GH-101307) 2023-01-26 20:47:24 +00:00
Guido van Rossum 9f2c479eaf
gh-98831: Fix two bugs in case generator (#101349)
Fix two bugs in case generator

- UndefinedLocalError when generating metadata for an 'op'
- Accidental newline inserted in test_generator.py
2023-01-26 09:15:05 -08:00
Irit Katriel b400219df5
gh-98831: rewrite RAISE_VARARGS in the instruction definition DSL (#101306) 2023-01-25 22:29:56 +00:00
Irit Katriel 19f90d6b97
gh-98831: add variable stack effect support to cases generator (#101309) 2023-01-25 20:41:03 +00:00
Guido van Rossum 1417712812
Add advice how to freeze fewer modules (#101298)
(And fix a bug that only occurs when you follow the advice.)
2023-01-25 09:28:14 -08:00
Guido van Rossum 395871e511
GH-98831: Elaborate some cases_generator tests (#101299)
* Make macro test more elaborate
* Add test for 'register inst()'
2023-01-25 08:55:46 -08:00
Irit Katriel 1a9d8c750b
gh-98831: rewrite pattern matching opcodes in the instruction definition DSL (#101287) 2023-01-24 22:39:13 +00:00
Erlend E. Aasland 00d092caa8
GH-91375: Purge `asyncio` static variables from globals-to-fix list (#101288) 2023-01-24 20:37:27 +05:30
Erlend E. Aasland 13566a37c2
GH-92123: Move _elementtree heap types to module state (#101187) 2023-01-21 12:01:15 +01:00
Irit Katriel 8a2d4f4e8e
gh-98831: register instructions have 0 pushes and pops (#101163) 2023-01-19 21:02:56 +00:00
Irit Katriel e9ccfe4a63
gh-100712: make it possible to disable specialization (for debugging) (#100713) 2023-01-19 18:14:55 +00:00
Kushal Das 75c8133efe
gh-100340: Allows -Wno-int-conversion for wasm (#100341)
Fixes #100340 allows -Wno-int-conversion for wasm
2023-01-18 19:43:49 +01:00
Zac Hatfield-Dodds 48ec678287
Fix typo and old link in wasm readme (#101096) 2023-01-17 16:06:38 -08:00
Guido van Rossum 80e3e3423c
GH-98831: Implement array support in cases generator (#100912)
You can now write things like this:
```
inst(BUILD_STRING, (pieces[oparg] -- str)) { ... }
inst(LIST_APPEND, (list, unused[oparg-1], v -- list, unused[oparg-1])) { ... }
```
Note that array output effects are only partially supported (they must be named `unused` or correspond to an input effect).
2023-01-17 15:59:19 -08:00
Pablo Galindo Salgado 1de4395f62
gh-101046: Fix a potential memory leak in the parser when raising MemoryError (#101051) 2023-01-16 18:45:37 +00:00
Mark Shannon 7b14c2ef19
GH-100982: Add `COMPARE_AND_BRANCH` instruction (GH-100983) 2023-01-16 12:35:21 +00:00
Guido van Rossum 5134ef4878
GH-98831: Identify instructions that don't use oparg (#100957)
For these the instr_format field uses IX instead of IB.
Register instructions use IX, IB, IBBX, IBBB, etc.

Also: Include the closing '}' in Block.tokens, for completeness
2023-01-13 17:06:45 -08:00
Mark Shannon c00eb1eae6
Don't double count misses. (GH-100984) 2023-01-13 15:57:39 +00:00
Eric Snow b511d3512b
GH-100997: Implement Multi-Phase Init for the _testinternalcapi Module (gh-100998)
_testinternalcapi is an internal module used for testing.

https://github.com/python/cpython/issues/100997
2023-01-12 13:42:03 -07:00
Guido van Rossum 3f3c78e32f
GH-98831: Refactor instr format code and change to enum (#100895) 2023-01-09 15:53:01 -08:00
Guido van Rossum 9ffbc58f5c
GH-98831: Add some tests for generate_cases.py (#100763)
- This doesn't cover everything (far from it) but it's a start.
- This uses pytest, which isn't ideal, but was quickest to get started.

Co-authored-by: Kumar Aditya <59607654+kumaraditya303@users.noreply.github.com>
2023-01-06 08:04:20 -08:00
Guido van Rossum 14b7f00fdf
GH-98831: Update generate_cases.py: register inst, opcode_metadata.h (#100735)
(These aren't used yet, but may be coming soon,
and it's easier to keep this tool the same between branches.)

Added a sanity check for all this to compile.c.

Co-authored-by: Irit Katriel <iritkatriel@yahoo.com>
2023-01-05 13:01:07 -08:00
Mark Shannon 15aecf8dd7
GH-100719: Remove the `co_nplaincellvars` field from code objects. (GH-100721) 2023-01-04 15:41:39 +00:00
Irit Katriel c31e356a10
gh-100720: refactor calculation of number of frame slots for a code object into the new function _PyFrame_NumSlotsForCodeObject (#100722) 2023-01-04 13:37:06 +00:00
Eli Schwartz 8586949833
gh-89419: gdb: fix bug causing `AttributeError` in py-locals when no frame is available (#100611) 2023-01-03 20:26:57 +05:30
Zachary Ware 2df82db485
gh-100540: Remove obsolete Modules/_ctypes/darwin/ dlfcn shim (GH-100541)
As far as I can tell, this hasn't been actually used since Mac OS X 10.2.
2022-12-29 16:13:28 -06:00
Zachary Ware 025b5c3780
gh-100540: Remove unused Modules/_ctypes/libffi_osx/ (GH-100543)
It was an ancient, modified copy of libffi that has not been in use
since GH-22855.
2022-12-28 13:38:52 -06:00
Nikita Sobolev 3da71ff0f2
Fix copy-paste error in `Tools/clinic.py` (#100560) 2022-12-28 09:57:57 +05:30
Kumar Aditya 7cf164ad5e
GH-100342: check for allocation failure in AC `*args` parsing (#100343) 2022-12-28 09:46:28 +05:30
Guido van Rossum 08e5594cf3
GH-98831: Modernize a ton of simpler instructions (#100545)
* load_const and load_fast aren't families for now
* Don't decref unmoved names
* Modernize GET_ANEXT
* Modernize GET_AWAITABLE
* Modernize ASYNC_GEN_WRAP
* Modernize YIELD_VALUE
* Modernize POP_EXCEPT (in more than one way)
* Modernize PREP_RERAISE_STAR
* Modernize LOAD_ASSERTION_ERROR
* Modernize LOAD_BUILD_CLASS
* Modernize STORE_NAME
* Modernize LOAD_NAME
* Modernize LOAD_CLASSDEREF
* Modernize LOAD_DEREF
* Modernize STORE_DEREF
* Modernize COPY_FREE_VARS (mark it as done)
* Modernize LIST_TO_TUPLE
* Modernize LIST_EXTEND
* Modernize SET_UPDATE
* Modernize SETUP_ANNOTATIONS
* Modernize DICT_UPDATE
* Modernize DICT_MERGE
* Modernize MAP_ADD
* Modernize IS_OP
* Modernize CONTAINS_OP
* Modernize CHECK_EXC_MATCH
* Modernize IMPORT_NAME
* Modernize IMPORT_STAR
* Modernize IMPORT_FROM
* Modernize JUMP_FORWARD (mark it as done)
* Modernize JUMP_BACKWARD (mark it as done)
2022-12-27 17:11:03 -08:00
Nikita Sobolev b0ea28913e
Remove `NoneType` redefinition from `clinic.py` (#100551) 2022-12-27 21:41:39 +05:30
Illia Volochii a23cb72ac8
gh-100454: Start running SSL tests with OpenSSL 3.1.0-beta1 (#100456) 2022-12-23 18:04:20 -08:00
penguin_wwy a02161286a
GH-99770: Make the correct call specialization fail kind show up in the stats (GH-99771) 2022-12-22 12:44:59 +00:00
Guido van Rossum 9cdd2fa63b
GH-98831: Add DECREF_INPUTS(), expanding to DECREF() each stack input (#100205)
The presence of this macro indicates that a particular instruction
may be considered for conversion to a register-based format
(see https://github.com/faster-cpython/ideas/issues/485).

An invariant (currently unchecked) is that `DEOPT_IF()` may only
occur *before* `DECREF_INPUTS()`, and `ERROR_IF()` may only occur
*after* it. One reason not to check this is that there are a few
places where we insert *two* `DECREF_INPUTS()` calls, in different
branches of the code. The invariant checking would have to be able
to do some flow control analysis to understand this.

Note that many instructions, especially specialized ones,
can't be converted to use this macro straightforwardly.
This is because the generator currently only generates plain
`Py_DECREF(variable)` statements, and cannot generate
things like `_Py_DECREF_SPECIALIZED()` let alone deal with
`_PyList_AppendTakeRef()`.
2022-12-16 20:45:55 -08:00
Eric Snow 0415cf895f
gh-81057: Move the Cached Parser Dummy Name to _PyRuntimeState (#100277) 2022-12-16 13:48:03 +00:00
Mark Shannon 5693f45b19
Assorted minor fixes for specialization stats. (GH-100219) 2022-12-14 15:50:02 +00:00
Mark Shannon 6997e77bdf
GH-100222: Redefine _Py_CODEUNIT as a union to clarify structure of code unit. (GH-100223) 2022-12-14 11:12:53 +00:00
Eric Snow 0e081a089e
gh-81057: Move _Py_RefTotal to the "Ignored Globals" List (gh-100203)
We can't move it to _PyRuntimeState because the symbol is exposed in the stable ABI. We'll have to sort that out before a per-interpreter GIL, but it shouldn't be too hard.

https://github.com/python/cpython/issues/81057
2022-12-12 16:51:27 -07:00
Eric Snow 5eb28bca9f
gh-81057: Move Signal-Related Globals to _PyRuntimeState (gh-100085)
https://github.com/python/cpython/issues/81057
2022-12-12 16:50:19 -07:00
Eric Snow 53d9cd95cd
gh-81057: Move faulthandler Globals to _PyRuntimeState (gh-100152)
https://github.com/python/cpython/issues/81057
2022-12-12 09:58:46 -07:00
Eric Snow 8790d4d31f
gh-81057: Move tracemalloc Globals to _PyRuntimeState (gh-100151)
https://github.com/python/cpython/issues/81057
2022-12-12 08:44:23 -07:00
Michael Droettboom 1583c6e326
GH-100143: Improve collecting pystats for parts of runs (GH-100144)
* pystats off by default

* Add -Xpystats flag

* Always dump pystats, even if turned off
2022-12-12 14:50:43 +00:00
Shantanu 621a1790c4
gh-100176: remove incorrect version compatibility check from argument clinic (#100190) 2022-12-12 17:22:12 +05:30
Mark Shannon fb713b2183
GH-98522: Add version number to code objects. (GH-98525)
* Add version number to code object for better versioning of functions.

* Improves specialization for closures and list comprehensions.
2022-12-09 12:18:45 +00:00
Eric Snow 1160001b34
gh-81057: Move Threading-Related Globals to _PyRuntimeState (#100084)
https://github.com/python/cpython/issues/81057
2022-12-08 17:50:58 -07:00
Eric Snow bc8cdf8c3d
gh-81057: Move Ceval Trampoline Globals to _PyRuntimeState (gh-100083)
https://github.com/python/cpython/issues/81057
2022-12-08 17:17:20 -07:00
Guido van Rossum 1cfa704f64
GH-98831: Generate things in the input order (#100123)
This makes it easier to see what changed in the generated code
when converting an instruction to super or macro.
2022-12-08 15:54:07 -08:00
Eric Snow 8a3f06c54b
gh-81057: Move time Globals to _PyRuntimeState (gh-100122)
https://github.com/python/cpython/issues/81057
2022-12-08 16:46:09 -07:00
Eric Snow cda9f0236f
gh-81057: Move OS-Related Globals to _PyRuntimeState (gh-100082)
https://github.com/python/cpython/issues/81057
2022-12-08 15:38:06 -07:00
Guido van Rossum c85be734d1
GH-98831: Typed stack effects, and more instructions converted (#99764)
Stack effects can now have a type, e.g. `inst(X, (left, right -- jump/uint64_t)) { ... }`.

Instructions converted to the non-legacy format:

* COMPARE_OP
* COMPARE_OP_FLOAT_JUMP
* COMPARE_OP_INT_JUMP
* COMPARE_OP_STR_JUMP
* STORE_ATTR
* DELETE_ATTR
* STORE_GLOBAL
* STORE_ATTR_INSTANCE_VALUE
* STORE_ATTR_WITH_HINT
* STORE_ATTR_SLOT, and complete the store_attr family
* Complete the store_subscr family: STORE_SUBSCR{,DICT,LIST_INT}
  (STORE_SUBSCR was alread half converted,
  but wasn't using cache effects yet.)
* DELETE_SUBSCR
* PRINT_EXPR
* INTERPRETER_EXIT (a bit weird, ends in return)
* RETURN_VALUE
* GET_AITER (had to restructure it some)
  The original had mysterious `SET_TOP(NULL)` before `goto error`.
  I assume those just account for `obj` having been decref'ed,
  so I got rid of them in favor of the cleanup implied by `ERROR_IF()`.
* LIST_APPEND (a bit unhappy with it)
* SET_ADD (also a bit unhappy with it)

Various other improvements/refactorings as well.
2022-12-08 13:31:27 -08:00
Eric Snow 91a8e002c2
gh-81057: Move More Globals to _PyRuntimeState (gh-100092)
https://github.com/python/cpython/issues/81057
2022-12-07 15:56:31 -07:00
Eric Snow d47ffeb9e3
gh-90110: Clean Up the C-analyzer Globals Lists (gh-100091)
https://github.com/python/cpython/issues/90110
2022-12-07 15:02:47 -07:00
Michael Droettboom 9dc787ea96
GH-100026: Include the number of raw input files in summarize_stats.py (GH-100027) 2022-12-06 11:14:47 +00:00
Eric Snow 530cc9dbb6
gh-99741: Implement Multi-Phase Init for the _xxsubinterpreters Module (gh-99742)
_xxsubinterpreters is an internal module used for testing.

https://github.com/python/cpython/issues/99741
2022-12-05 13:40:20 -07:00
Guido van Rossum acf9184e6b
GH-98831: Support cache effects in super- and macro instructions (#99601) 2022-12-02 19:57:30 -08:00
Michael Droettboom bf94c653f4
GH-99905: Fix output of misses in summarize_stats.py execution counts (GH-99906)
This was an indentation error introduced in 2844aa6a
2022-11-30 19:17:08 +00:00
colorfulappl 0da728387c
gh-64490: Fix bugs in argument clinic varargs processing (#32092) 2022-11-24 20:56:50 +01:00
colorfulappl 8dbe08eb7c
gh-99240: Fix double-free bug in Argument Clinic str_converter generated code (GH-99241)
Fix double-free bug mentioned at https://github.com/python/cpython/issues/99240,
by moving memory clean up out of "exit" label.

Automerge-Triggered-By: GH:erlend-aasland
2022-11-24 06:01:26 -08:00
colorfulappl 69f6cc77d0
gh-64490: Fix refcount error when arguments are packed to tuple in argument clinic (#99233) 2022-11-24 17:54:18 +05:30
Guido van Rossum 8f18ac04d3
GH-98831: Add `macro` and `op` and their implementation to DSL (#99495)
Newly supported interpreter definition syntax:
- `op(NAME, (input_stack_effects -- output_stack_effects)) { ... }`
- `macro(NAME) = OP1 + OP2;`

Also some other random improvements:
- Convert `WITH_EXCEPT_START` to use stack effects
- Fix lexer to balk at unrecognized characters, e.g. `@`
- Fix moved output names; support object pointers in cache
- Introduce `error()` method to print errors
- Introduce read_uint16(p) as equivalent to `*p`

Co-authored-by: Brandt Bucher <brandtbucher@gmail.com>
2022-11-22 16:04:57 -08:00
colorfulappl c450c8c9ed
gh-96002: Add functional test for Argument Clinic (#96178)
Co-authored-by: Kumar Aditya <59607654+kumaraditya303@users.noreply.github.com>
Co-authored-by: Erlend E. Aasland <erlend.aasland@protonmail.com>
2022-11-21 15:08:45 +01:00
Guido van Rossum 4f5e1cb00a
GH-98831: Refactor and fix cases generator (#99526)
Also complete cache effects for BINARY_SUBSCR family.
2022-11-17 17:06:07 -08:00
Eric Snow 9db1e17c80
gh-81057: Move the global Dict-Related Versions to _PyRuntimeState (gh-99497)
We also move the global func version.

https://github.com/python/cpython/issues/81057
2022-11-16 10:37:29 -07:00
Eric Snow 01fa907aa8
gh-81057: Move contextvars-related Globals to _PyRuntimeState (gh-99400)
This is part of the effort to consolidate global variables, to make them easier to manage (and make it easier to later move some of them to PyInterpreterState).

https://github.com/python/cpython/issues/81057
2022-11-16 09:54:28 -07:00
Eric Snow 5f55067e23
gh-81057: Move More Globals in Core Code to _PyRuntimeState (gh-99516)
https://github.com/python/cpython/issues/81057
2022-11-16 09:37:14 -07:00
Guido van Rossum e37744f289
GH-98831: Implement basic cache effects (#99313) 2022-11-15 19:59:19 -08:00
Eric Snow 3c57971a2d
gh-81057: Move Globals in Core Code to _PyRuntimeState (gh-99496)
This is the first of several changes to consolidate non-object globals in core code.

https://github.com/python/cpython/issues/81057
2022-11-15 09:45:11 -07:00
Eric Snow e874c2f198
gh-81057: Move the Remaining Import State Globals to _PyRuntimeState (gh-99488)
https://github.com/python/cpython/issues/81057
2022-11-14 15:56:16 -07:00
Eric Snow a088290f9d
gh-81057: Move Global Variables Holding Objects to _PyRuntimeState. (gh-99487)
This moves nearly all remaining object-holding globals in core code (other than static types).

https://github.com/python/cpython/issues/81057
2022-11-14 13:50:56 -07:00
Eric Snow 7f3a4b967c
gh-81057: Move PyImport_Inittab to _PyRuntimeState (gh-99402)
We actually don't move PyImport_Inittab.  Instead, we make a copy that we keep on _PyRuntimeState and use only that after Py_Initialize().  We also prevent folks from modifying PyImport_Inittab (the best we can) after that point.

https://github.com/python/cpython/issues/81057
2022-11-11 17:06:05 -07:00
Eric Snow 67807cfc87
gh-81057: Move the Allocators to _PyRuntimeState (gh-99217)
The global allocators were stored in 3 static global variables: _PyMem_Raw, _PyMem, and _PyObject.  State for the "small block" allocator was stored in another 13.  That makes a total of 16 global variables. We are moving all 16 to the _PyRuntimeState struct as part of the work for gh-81057.  (If PEP 684 is accepted then we will follow up by moving them all to PyInterpreterState.)

https://github.com/python/cpython/issues/81057
2022-11-11 16:30:46 -07:00
Eric Snow f531b6879b
gh-81057: Add PyInterpreterState.static_objects (gh-99397)
As we consolidate global variables, we find some objects that are almost suitable to add to _PyRuntimeState.global_objects, but have some small/sneaky bit of per-interpreter state (e.g. a weakref list). We're adding PyInterpreterState.static_objects so we can move such objects there. (We'll removed the _not_used field once we've added others.)

https://github.com/python/cpython/issues/81057
2022-11-11 14:24:18 -07:00
Eric Snow dd36b71fa6
gh-81057: Move the Extension Modules Cache to _PyRuntimeState (gh-99355)
We also move the closely related max_module_number and add comments documenting the group of struct members.

https://github.com/python/cpython/issues/81057
2022-11-11 14:16:28 -07:00
Eric Snow fe55ff3f68
gh-81057: Generate a Separate Initializer For Each Part of the Global Objects Initializer (gh-99389)
Up until now we had a single generated initializer macro for all the statically declared global objects in _PyRuntimeState, including several one-offs (e.g. the empty tuple). The one-offs don't need to be generated, but were because we had one big initializer. Having separate initializers for set of generated global objects allows us to generate only the ones we need to.  This allows us to add initializers for one-off global objects without having to generate them.

https://github.com/python/cpython/issues/81057
2022-11-11 13:23:41 -07:00
Ikko Ashimine 8fe1a88e6a
Fix typo in Tools/wasm/README.md (GH-99384)
Noticable -> Noticeable

Automerge-Triggered-By: GH:AlexWaygood
2022-11-11 08:38:09 -08:00
Gregory P. Smith e00d730efe
Update github CI to use OpenSSL 1.1.1s and 3.0.7. (#99364) 2022-11-11 01:33:55 -08:00