Mirror
/
cpython
mirror of https://github.com/python/cpython
4
1
Fork 0
Code Issues Releases Wiki Activity
cpython/Tools/cases_generator
History
Mark Shannon e24eccbc1c
GH-111485: Sort metadata tables for easier checking of future diffs (GH-113101) 		2023-12-14 16:41:52 +00:00
..
README.md Update README for the cases generator (#107826) 2023-08-10 01:05:51 +00:00
_typing_backports.py gh-104504: cases generator: Add `--warn-unreachable` to the mypy config (#108112) 2023-08-21 00:40:41 +01:00
analysis.py gh-112519: Make it possible to specify instruction flags for pseudo instructions in bytecodes.c (#112520) 2023-11-30 11:03:30 +00:00
analyzer.py GH-111485: Factor out tier 2 code generation from the rest of the interpreter code generator (GH-112968) 2023-12-12 12:12:17 +00:00
cwriter.py GH-111485: Factor out opcode ID generator from the main cases generator. (GH-112831) 2023-12-08 11:48:30 +00:00
flags.py A few more cases_generator cleanups (#112220) 2023-11-17 22:36:37 +00:00
formatting.py gh-104504: Run mypy on cases_generator in CI (and blacken the code) (gh-108090) 2023-08-18 22:42:45 +09:00
generate_cases.py GH-111485: Sort metadata tables for easier checking of future diffs (GH-113101) 2023-12-14 16:41:52 +00:00
generators_common.py GH-111485: Factor out tier 2 code generation from the rest of the interpreter code generator (GH-112968) 2023-12-12 12:12:17 +00:00
instructions.py gh-112287: Speed up Tier 2 (uop) interpreter a little (#112286) 2023-11-20 11:25:32 -08:00
interpreter_definition.md gh-106812: Small stack effect fixes (#107759) 2023-08-07 21:32:42 -07:00
lexer.py GH-111485: Separate out parsing, analysis and code-gen phases of tier 1 code generator (GH-112299) 2023-12-07 12:49:40 +00:00
mypy.ini GH-111485: Separate out parsing, analysis and code-gen phases of tier 1 code generator (GH-112299) 2023-12-07 12:49:40 +00:00
opcode_id_generator.py GH-111485: Factor out tier 2 code generation from the rest of the interpreter code generator (GH-112968) 2023-12-12 12:12:17 +00:00
parser.py GH-111485: Separate out parsing, analysis and code-gen phases of tier 1 code generator (GH-112299) 2023-12-07 12:49:40 +00:00
parsing.py GH-111485: Separate out parsing, analysis and code-gen phases of tier 1 code generator (GH-112299) 2023-12-07 12:49:40 +00:00
plexer.py gh-106812: Refactor cases_generator to allow uops with array stack effects (#107564) 2023-08-04 09:35:56 -07:00
stack.py GH-111485: Factor out tier 2 code generation from the rest of the interpreter code generator (GH-112968) 2023-12-12 12:12:17 +00:00
stacking.py GH-111485: Increment `next_instr` consistently at the start of the instruction. (GH-111486) 2023-10-31 10:09:54 +00:00
tier1_generator.py GH-111485: Factor out tier 2 code generation from the rest of the interpreter code generator (GH-112968) 2023-12-12 12:12:17 +00:00
tier2_generator.py Fix whitespace in generated code 2023-12-13 12:31:41 +00:00
uop_id_generator.py GH-111485: Factor out tier 2 code generation from the rest of the interpreter code generator (GH-112968) 2023-12-12 12:12:17 +00:00

README.md

Tooling to generate interpreters

Documentation for the instruction definitions in Python/bytecodes.c ("the DSL") is here.

What's currently here:

  • lexer.py: lexer for C, originally written by Mark Shannon
  • plexer.py: OO interface on top of lexer.py; main class: PLexer
  • parsing.py: Parser for instruction definition DSL; main class Parser
  • generate_cases.py: driver script to read Python/bytecodes.c and write Python/generated_cases.c.h (and several other files)
  • analysis.py: Analyzer class used to read the input files
  • flags.py: abstractions related to metadata flags for instructions
  • formatting.py: Formatter class used to write the output files
  • instructions.py: classes to analyze and write instructions
  • stacking.py: code to handle generalized stack effects

Note that there is some dummy C code at the top and bottom of Python/bytecodes.c to fool text editors like VS Code into believing this is valid C code.

A bit about the parser

The parser class uses a pretty standard recursive descent scheme, but with unlimited backtracking. The PLexer class tokenizes the entire input before parsing starts. We do not run the C preprocessor. Each parsing method returns either an AST node (a Node instance) or None, or raises SyntaxError (showing the error in the C source).

Most parsing methods are decorated with @contextual, which automatically resets the tokenizer input position when None is returned. Parsing methods may also raise SyntaxError, which is irrecoverable. When a parsing method returns None, it is possible that after backtracking a different parsing method returns a valid AST.

Neither the lexer nor the parsers are complete or fully correct. Most known issues are tersely indicated by # TODO: comments. We plan to fix issues as they become relevant.