8ac658114d
"Include/token.h", "Lib/token.py" (containing now some data moved from "Lib/tokenize.py") and new files "Parser/token.c" (containing the code moved from "Parser/tokenizer.c") and "Doc/library/token-list.inc" (included in "Doc/library/token.rst") are now generated from "Grammar/Tokens" by "Tools/scripts/generate_token.py". The script overwrites files only if needed and can be used on the read-only sources tree. "Lib/symbol.py" is now generated by "Tools/scripts/generate_symbol_py.py" instead of been executable itself. Added new make targets "regen-token" and "regen-symbol" which are now dependencies of "regen-all". The documentation contains now strings for operators and punctuation tokens. |
||
|---|---|---|
| .. | ||
| Python.asdl | ||
| acceler.c | ||
| asdl.py | ||
| asdl_c.py | ||
| bitset.c | ||
| firstsets.c | ||
| grammar.c | ||
| grammar1.c | ||
| listnode.c | ||
| metagrammar.c | ||
| myreadline.c | ||
| node.c | ||
| parser.c | ||
| parser.h | ||
| parsetok.c | ||
| parsetok_pgen.c | ||
| pgen.c | ||
| pgenmain.c | ||
| printgrammar.c | ||
| token.c | ||
| tokenizer.c | ||
| tokenizer.h | ||
| tokenizer_pgen.c | ||