Mirror
/
cpython
mirror of https://github.com/python/cpython
4
1
Fork 0
Code Issues Releases Wiki Activity

Clarify the supported cases in the tokenize module (#105569)

This commit is contained in:
Pablo Galindo Salgado 2023-06-09 16:59:37 +01:00 committed by GitHub
parent 59f009e589
commit 1dd267af64
No known key found for this signature in database
GPG Key ID: 4AEE18F83AFDEB23
1 changed files with 9 additions and 0 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

9
Doc/library/tokenize.rst
View File

@ -22,6 +22,15 @@ the generic :data:`~token.OP` token type. The exact
type can be determined by checking the ``exact_type`` property on the
:term:`named tuple` returned from :func:`tokenize.tokenize`.
.. warning::
Note that the functions in this module are only designed to parse
syntactically valid Python code (code that does not raise when parsed
using :func:`ast.parse`). The behavior of the functions in this module is
**undefined** when providing invalid Python code and it can change at any
point.
Tokenizing Input
----------------