032f480909
This mostly extracts a whole bunch of stuff out of generate_cases.py into separate files, but there are a few other things going on here. - analysis.py: `Analyzer` etc. - instructions.py: `Instruction` etc. - flags.py: `InstructionFlags`, `variable_used`, `variable_used_unspecialized` - formatting.py: `Formatter` etc. - Rename parser.py to parsing.py, to avoid conflict with stdlib parser.py - Blackify most things - Fix most mypy errors - Remove output filenames from Generator state, add them to `write_instructions()` etc. - Fix unit tests |
||
|---|---|---|
| .. | ||
| README.md | ||
| analysis.py | ||
| flags.py | ||
| formatting.py | ||
| generate_cases.py | ||
| instructions.py | ||
| interpreter_definition.md | ||
| lexer.py | ||
| parsing.py | ||
| plexer.py | ||
README.md
Tooling to generate interpreters
Documentation for the instruction definitions in Python/bytecodes.c
("the DSL") is here.
What's currently here:
lexer.py: lexer for C, originally written by Mark Shannonplexer.py: OO interface on top of lexer.py; main class:PLexerparser.py: Parser for instruction definition DSL; main classParsergenerate_cases.py: driver script to readPython/bytecodes.cand writePython/generated_cases.c.h(and several other files)test_generator.py: tests, require manual running usingpytest
Note that there is some dummy C code at the top and bottom of
Python/bytecodes.c
to fool text editors like VS Code into believing this is valid C code.
A bit about the parser
The parser class uses a pretty standard recursive descent scheme,
but with unlimited backtracking.
The PLexer class tokenizes the entire input before parsing starts.
We do not run the C preprocessor.
Each parsing method returns either an AST node (a Node instance)
or None, or raises SyntaxError (showing the error in the C source).
Most parsing methods are decorated with @contextual, which automatically
resets the tokenizer input position when None is returned.
Parsing methods may also raise SyntaxError, which is irrecoverable.
When a parsing method returns None, it is possible that after backtracking
a different parsing method returns a valid AST.
Neither the lexer nor the parsers are complete or fully correct.
Most known issues are tersely indicated by # TODO: comments.
We plan to fix issues as they become relevant.