traverseda
/
cpython
1
0
Fork 0
Code Issues Pull Requests Releases Wiki Activity

bpo-5028: Fix up rest of documentation for tokenize documenting line (GH-13686) 

https://bugs.python.org/issue5028
This commit is contained in:
Anthony Sottile 2019-05-30 15:06:32 -07:00 committed by Miss Islington (bot)
parent eea47e0939
commit 2a58b0636d
3 changed files with 4 additions and 4 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

4
Doc/library/tokenize.rst
View File

@ -39,8 +39,8 @@ The primary entry point is a :term:`generator`:
column where the token begins in the source; a 2-tuple ``(erow, ecol)`` of column where the token begins in the source; a 2-tuple ``(erow, ecol)`` of
ints specifying the row and column where the token ends in the source; and ints specifying the row and column where the token ends in the source; and
the line on which the token was found. The line passed (the last tuple item) the line on which the token was found. The line passed (the last tuple item)
is the *physical* line; continuation lines are included. The 5 tuple is is the *physical* line. The 5 tuple is returned as a :term:`named tuple`
returned as a :term:`named tuple` with the field names: with the field names:
``type string start end line``. ``type string start end line``.
The returned :term:`named tuple` has an additional property named The returned :term:`named tuple` has an additional property named

2
Lib/lib2to3/pgen2/tokenize.py
View File

@ -346,7 +346,7 @@ def generate_tokens(readline):
column where the token begins in the source; a 2-tuple (erow, ecol) of column where the token begins in the source; a 2-tuple (erow, ecol) of
ints specifying the row and column where the token ends in the source; ints specifying the row and column where the token ends in the source;
and the line on which the token was found. The line passed is the and the line on which the token was found. The line passed is the
physical line; continuation lines are included. physical line.
""" """
lnum = parenlev = continued = 0 lnum = parenlev = continued = 0
contstr, needcont = '', 0 contstr, needcont = '', 0

2
Lib/tokenize.py
View File

@ -415,7 +415,7 @@ def tokenize(readline):
column where the token begins in the source; a 2-tuple (erow, ecol) of column where the token begins in the source; a 2-tuple (erow, ecol) of
ints specifying the row and column where the token ends in the source; ints specifying the row and column where the token ends in the source;
and the line on which the token was found. The line passed is the and the line on which the token was found. The line passed is the
physical line; continuation lines are included. physical line.
The first token sequence will always be an ENCODING token The first token sequence will always be an ENCODING token
which tells you which encoding was used to decode the bytes stream. which tells you which encoding was used to decode the bytes stream.