History

Mark Shannon ac10947ba7 GH-112354: `_GUARD_IS_TRUE_POP` side-exits to target the next instruction, not themselves. (GH-114078)		2024-01-15 11:41:06 +00:00
..
README.md	Update README for the cases generator (#107826 )	2023-08-10 01:05:51 +00:00
_typing_backports.py	gh-104504: cases generator: Add `--warn-unreachable` to the mypy config (#108112 )	2023-08-21 00:40:41 +01:00
analyzer.py	GH-112354: `_GUARD_IS_TRUE_POP` side-exits to target the next instruction, not themselves. (GH-114078)	2024-01-15 11:41:06 +00:00
cwriter.py	GH-111485: Generate instruction and uop metadata (GH-113287)	2023-12-20 14:27:25 +00:00
generators_common.py	GH-112354: `_GUARD_IS_TRUE_POP` side-exits to target the next instruction, not themselves. (GH-114078)	2024-01-15 11:41:06 +00:00
interpreter_definition.md	GH-112354: `_GUARD_IS_TRUE_POP` side-exits to target the next instruction, not themselves. (GH-114078)	2024-01-15 11:41:06 +00:00
lexer.py	gh-113710: Add types to the interpreter DSL (#113711 )	2024-01-13 01:30:27 +08:00
mypy.ini	GH-111485: Separate out parsing, analysis and code-gen phases of tier 1 code generator (GH-112299)	2023-12-07 12:49:40 +00:00
opcode_id_generator.py	GH-111485: Generate instruction and uop metadata (GH-113287)	2023-12-20 14:27:25 +00:00
opcode_metadata_generator.py	gh-113710: Add types to the interpreter DSL (#113711 )	2024-01-13 01:30:27 +08:00
parser.py	GH-111485: Delete the old generator code. (GH-113321)	2023-12-21 12:46:28 +00:00
parsing.py	gh-113710: Add types to the interpreter DSL (#113711 )	2024-01-13 01:30:27 +08:00
plexer.py	gh-106812: Refactor cases_generator to allow uops with array stack effects (#107564 )	2023-08-04 09:35:56 -07:00
py_metadata_generator.py	GH-111485: Generate instruction and uop metadata (GH-113287)	2023-12-20 14:27:25 +00:00
stack.py	GH-112354: `_GUARD_IS_TRUE_POP` side-exits to target the next instruction, not themselves. (GH-114078)	2024-01-15 11:41:06 +00:00
target_generator.py	GH-111485: Generate `TARGET` table for computed goto dispatch. (GH-113319)	2023-12-20 15:09:12 +00:00
tier1_generator.py	GH-111485: Generate instruction and uop metadata (GH-113287)	2023-12-20 14:27:25 +00:00
tier2_generator.py	GH-111485: Generate instruction and uop metadata (GH-113287)	2023-12-20 14:27:25 +00:00
uop_id_generator.py	GH-111485: Generate instruction and uop metadata (GH-113287)	2023-12-20 14:27:25 +00:00
uop_metadata_generator.py	GH-111485: Generate instruction and uop metadata (GH-113287)	2023-12-20 14:27:25 +00:00

README.md

Tooling to generate interpreters

Documentation for the instruction definitions in Python/bytecodes.c ("the DSL") is here.

What's currently here:

lexer.py: lexer for C, originally written by Mark Shannon
plexer.py: OO interface on top of lexer.py; main class: PLexer
parsing.py: Parser for instruction definition DSL; main class Parser
generate_cases.py: driver script to read Python/bytecodes.c and write Python/generated_cases.c.h (and several other files)
analysis.py: Analyzer class used to read the input files
flags.py: abstractions related to metadata flags for instructions
formatting.py: Formatter class used to write the output files
instructions.py: classes to analyze and write instructions
stacking.py: code to handle generalized stack effects

Note that there is some dummy C code at the top and bottom of Python/bytecodes.c to fool text editors like VS Code into believing this is valid C code.

A bit about the parser

The parser class uses a pretty standard recursive descent scheme, but with unlimited backtracking. The PLexer class tokenizes the entire input before parsing starts. We do not run the C preprocessor. Each parsing method returns either an AST node (a Node instance) or None, or raises SyntaxError (showing the error in the C source).

Most parsing methods are decorated with @contextual, which automatically resets the tokenizer input position when None is returned. Parsing methods may also raise SyntaxError, which is irrecoverable. When a parsing method returns None, it is possible that after backtracking a different parsing method returns a valid AST.

Neither the lexer nor the parsers are complete or fully correct. Most known issues are tersely indicated by # TODO: comments. We plan to fix issues as they become relevant.