History

Guido van Rossum 7e135a48d6 gh-111520: Integrate the Tier 2 interpreter in the Tier 1 interpreter (#111428 ) - There is no longer a separate Python/executor.c file. - Conventions in Python/bytecodes.c are slightly different -- don't use `goto error`, you must use `GOTO_ERROR(error)` (same for others like `unused_local_error`). - The `TIER_ONE` and `TIER_TWO` symbols are only valid in the generated (.c.h) files. - In Lib/test/support/__init__.py, `Py_C_RECURSION_LIMIT` is imported from `_testcapi`. - On Windows, in debug mode, stack allocation grows from 8MiB to 12MiB. - Beware! This changes the env vars to enable uops and their debugging to `PYTHON_UOPS` and `PYTHON_LLTRACE`.		2023-11-01 13:13:02 -07:00
..
README.md	Update README for the cases generator (#107826 )	2023-08-10 01:05:51 +00:00
_typing_backports.py	gh-104504: cases generator: Add `--warn-unreachable` to the mypy config (#108112 )	2023-08-21 00:40:41 +01:00
analysis.py	GH-111485: Use micro-ops to split specialization code from base action (GH-111561)	2023-11-01 10:53:27 +00:00
flags.py	gh-107557: Remove unnecessary SAVE_IP instructions (#108583 )	2023-08-29 16:51:51 +00:00
formatting.py	gh-104504: Run mypy on cases_generator in CI (and blacken the code) (gh-108090)	2023-08-18 22:42:45 +09:00
generate_cases.py	gh-111520: Integrate the Tier 2 interpreter in the Tier 1 interpreter (#111428 )	2023-11-01 13:13:02 -07:00
instructions.py	gh-111520: Integrate the Tier 2 interpreter in the Tier 1 interpreter (#111428 )	2023-11-01 13:13:02 -07:00
interpreter_definition.md	gh-106812: Small stack effect fixes (#107759 )	2023-08-07 21:32:42 -07:00
lexer.py	gh-104504: Run mypy on cases_generator in CI (and blacken the code) (gh-108090)	2023-08-18 22:42:45 +09:00
mypy.ini	gh-104504: Cases generator: enable mypy's `possibly-undefined` error code (#108454 )	2023-08-25 18:08:29 +01:00
parsing.py	GH-108614: Add `RESUME_CHECK` instruction (GH-108630)	2023-09-07 14:39:03 +01:00
plexer.py	gh-106812: Refactor cases_generator to allow uops with array stack effects (#107564 )	2023-08-04 09:35:56 -07:00
stacking.py	GH-111485: Increment `next_instr` consistently at the start of the instruction. (GH-111486)	2023-10-31 10:09:54 +00:00

README.md

Tooling to generate interpreters

Documentation for the instruction definitions in Python/bytecodes.c ("the DSL") is here.

What's currently here:

lexer.py: lexer for C, originally written by Mark Shannon
plexer.py: OO interface on top of lexer.py; main class: PLexer
parsing.py: Parser for instruction definition DSL; main class Parser
generate_cases.py: driver script to read Python/bytecodes.c and write Python/generated_cases.c.h (and several other files)
analysis.py: Analyzer class used to read the input files
flags.py: abstractions related to metadata flags for instructions
formatting.py: Formatter class used to write the output files
instructions.py: classes to analyze and write instructions
stacking.py: code to handle generalized stack effects

Note that there is some dummy C code at the top and bottom of Python/bytecodes.c to fool text editors like VS Code into believing this is valid C code.

A bit about the parser

The parser class uses a pretty standard recursive descent scheme, but with unlimited backtracking. The PLexer class tokenizes the entire input before parsing starts. We do not run the C preprocessor. Each parsing method returns either an AST node (a Node instance) or None, or raises SyntaxError (showing the error in the C source).

Most parsing methods are decorated with @contextual, which automatically resets the tokenizer input position when None is returned. Parsing methods may also raise SyntaxError, which is irrecoverable. When a parsing method returns None, it is possible that after backtracking a different parsing method returns a valid AST.

Neither the lexer nor the parsers are complete or fully correct. Most known issues are tersely indicated by # TODO: comments. We plan to fix issues as they become relevant.