From 1dd267af642ed6df05a1c106e9dafb8252d826e6 Mon Sep 17 00:00:00 2001 From: Pablo Galindo Salgado Date: Fri, 9 Jun 2023 16:59:37 +0100 Subject: [PATCH] Clarify the supported cases in the tokenize module (#105569) --- Doc/library/tokenize.rst | 9 +++++++++ 1 file changed, 9 insertions(+) diff --git a/Doc/library/tokenize.rst b/Doc/library/tokenize.rst index 41222a771d1..bffe93006ed 100644 --- a/Doc/library/tokenize.rst +++ b/Doc/library/tokenize.rst @@ -22,6 +22,15 @@ the generic :data:`~token.OP` token type. The exact type can be determined by checking the ``exact_type`` property on the :term:`named tuple` returned from :func:`tokenize.tokenize`. + +.. warning:: + + Note that the functions in this module are only designed to parse + syntactically valid Python code (code that does not raise when parsed + using :func:`ast.parse`). The behavior of the functions in this module is + **undefined** when providing invalid Python code and it can change at any + point. + Tokenizing Input ----------------