Mirror
/
cpython
mirror of https://github.com/python/cpython
4
1
Fork 0
Code Issues Releases Wiki Activity

gh-105017: Fix including additional NL token when using CRLF (#105022) 

Co-authored-by: Pablo Galindo Salgado <Pablogsal@gmail.com>
This commit is contained in:
Marta Gómez Macías 2023-05-27 17:50:43 +01:00 committed by GitHub
parent 6e62eb2e70
commit 86d8f48935
No known key found for this signature in database
GPG Key ID: 4AEE18F83AFDEB23
3 changed files with 10 additions and 1 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

8
Lib/test/test_tokenize.py
View File

@ -84,6 +84,14 @@ class TokenizeTest(TestCase):
NEWLINE '\\n' (4, 26) (4, 27)
DEDENT '' (5, 0) (5, 0)
""")
self.check_tokenize("foo='bar'\r\n", """\
NAME 'foo' (1, 0) (1, 3)
OP '=' (1, 3) (1, 4)
STRING "'bar'" (1, 4) (1, 9)
NEWLINE '\\n' (1, 9) (1, 10)
""")
indent_error_file = b"""\
def k(x):
x += 2

1
Misc/NEWS.d/next/Core and Builtins/2023-05-27-16-23-16.gh-issue-105017.KQrsC0.rst Normal file
View File

@ -0,0 +1 @@
Do not include an additional final ``NL`` token when parsing files having CRLF lines. Patch by Marta Gómez.

2
Parser/tokenizer.c
View File

@ -800,7 +800,7 @@ translate_newlines(const char *s, int exec_input, struct tok_state *tok) {
}
/* If this is exec input, add a newline to the end of the string if
there isn't one already. */
if (exec_input && c != '\n') {
if (exec_input && c != '\n' && c != '\0') {
*current = '\n';
current++;
}