mirror of https://github.com/python/cpython
6715f91edc
This commit replaces the Python implementation of the tokenize module with an implementation that reuses the real C tokenizer via a private extension module. The tokenize module now implements a compatibility layer that transforms tokens from the C tokenizer into Python tokenize tokens for backward compatibility. As the C tokenizer does not emit some tokens that the Python tokenizer provides (such as comments and non-semantic newlines), a new special mode has been added to the C tokenizer mode that currently is only used via the extension module that exposes it to the Python layer. This new mode forces the C tokenizer to emit these new extra tokens and add the appropriate metadata that is needed to match the old Python implementation. Co-authored-by: Pablo Galindo <pablogsal@gmail.com> |
||
|---|---|---|
| .. | ||
| NEWS.d | ||
| rhel7 | ||
| ACKS | ||
| HISTORY | ||
| Porting | ||
| README | ||
| README.AIX | ||
| README.coverity | ||
| README.valgrind | ||
| SpecialBuilds.txt | ||
| coverity_model.c | ||
| indent.pro | ||
| python-config.in | ||
| python-config.sh.in | ||
| python-embed.pc.in | ||
| python.man | ||
| python.pc.in | ||
| requirements-test.txt | ||
| stable_abi.toml | ||
| svnmap.txt | ||
| valgrind-python.supp | ||
| vgrindefs | ||
README
Python Misc subdirectory
========================
This directory contains files that wouldn't fit in elsewhere. Some
documents are only of historic importance.
Files found here
----------------
ACKS Acknowledgements
HISTORY News from previous releases -- oldest last
indent.pro GNU indent profile approximating my C style
NEWS News for this release (for some meaning of "this")
Porting Mini-FAQ on porting to new platforms
python-config.in Python script template for python-config
python.man UNIX man page for the python interpreter
python.pc.in Package configuration info template for pkg-config
README The file you're reading now
README.AIX Information about using Python on AIX
README.coverity Information about running Coverity's Prevent on Python
README.valgrind Information for Valgrind users, see valgrind-python.supp
SpecialBuilds.txt Describes extra symbols you can set for debug builds
svnmap.txt Map of old SVN revs and branches to hg changeset ids,
help history-digging
valgrind-python.supp Valgrind suppression file, see README.valgrind
vgrindefs Python configuration for vgrind (a generic pretty printer)