1218910860
This restores a corner case: when the generator is run with working directory set to Tools/cases_generator, the source filenames listed in the generated provenance header should be relative to the repo root directory. |
||
---|---|---|
.. | ||
README.md | ||
generate_cases.py | ||
interpreter_definition.md | ||
lexer.py | ||
parser.py | ||
plexer.py |
README.md
Tooling to generate interpreters
Documentation for the instruction definitions in Python/bytecodes.c
("the DSL") is here.
What's currently here:
lexer.py
: lexer for C, originally written by Mark Shannonplexer.py
: OO interface on top of lexer.py; main class:PLexer
parser.py
: Parser for instruction definition DSL; main classParser
generate_cases.py
: driver script to readPython/bytecodes.c
and writePython/generated_cases.c.h
(and several other files)test_generator.py
: tests, require manual running usingpytest
Note that there is some dummy C code at the top and bottom of
Python/bytecodes.c
to fool text editors like VS Code into believing this is valid C code.
A bit about the parser
The parser class uses a pretty standard recursive descent scheme,
but with unlimited backtracking.
The PLexer
class tokenizes the entire input before parsing starts.
We do not run the C preprocessor.
Each parsing method returns either an AST node (a Node
instance)
or None
, or raises SyntaxError
(showing the error in the C source).
Most parsing methods are decorated with @contextual
, which automatically
resets the tokenizer input position when None
is returned.
Parsing methods may also raise SyntaxError
, which is irrecoverable.
When a parsing method returns None
, it is possible that after backtracking
a different parsing method returns a valid AST.
Neither the lexer nor the parsers are complete or fully correct.
Most known issues are tersely indicated by # TODO:
comments.
We plan to fix issues as they become relevant.