Mirror
/
cpython
mirror of https://github.com/python/cpython
4
1
Fork 0
Code Issues Releases Wiki Activity
98,820 Commits 6 Branches 592 Tags 2.8 GiB
Commit Graph

87 Commits

Author SHA1 Message Date
Brett Cannon a721abac29 Issue #26331: Implement the parsing part of PEP 515. 
Thanks to Georg Brandl for the patch.
 2016-09-09 14:57:09 -07:00
Zachary Ware 724f6a67f2 Rename test_pep####.py files 2016-09-09 12:55:37 -07:00
Zachary Ware a0154c0f0e Fix running test_tokenize directly 2016-09-09 12:55:14 -07:00
Eric V. Smith 1c8222c80a Issue 25311: Add support for f-strings to tokenize.py. Also added some comments to explain what's happening, since it's not so obvious. 2015-10-26 04:37:55 -04:00
Eric V. Smith 6731774216 Issue 25422: Add tests for multi-line string tokenization. Also remove truncated tokens. 2015-10-16 20:45:53 -04:00
Serhiy Storchaka 6f5175de15 Issue #25317: Converted doctests in test_tokenize to unittests. 
Made test_tokenize discoverable.
 2015-10-06 18:23:12 +03:00
Serhiy Storchaka 5f6fa82617 Issue #25317: Converted doctests in test_tokenize to unittests. 
Made test_tokenize discoverable.
 2015-10-06 18:16:28 +03:00
Yury Selivanov 96ec934e75 Issue #24619: Simplify async/await tokenization. 
This commit simplifies async/await tokenization in tokenizer.c,
tokenize.py & lib2to3/tokenize.py.  Previous solution was to keep
a stack of async-def & def blocks, whereas the new approach is just
to remember position of the outermost async-def block.

This change won't bring any parsing performance improvements, but
it makes the code much easier to read and validate.
 2015-07-23 15:01:58 +03:00
Yury Selivanov 8fb307cd65 Issue #24619: New approach for tokenizing async/await. 
This commit fixes how one-line async-defs and defs are tracked
by tokenizer.  It allows to correctly parse invalid code such
as:

>>> async def f():
...     def g(): pass
...     async = 10

and valid code such as:

>>> async def f():
...     async def g(): pass
...     await z

As a consequence, is is now possible to have one-line
'async def foo(): await ..' functions:

>>> async def foo(): return await bar()
 2015-07-22 13:33:45 +03:00
Jason R. Coombs a95a476b3a Issue #20387: Merge test and patch from 3.4.4 2015-06-28 11:13:30 -04:00
Jason R. Coombs b6d1cdda8e Issue #20387: Correct test to properly capture expectation. 2015-06-25 22:42:24 -04:00
Jason R. Coombs 5713b3c5bf Issue #20387: Add test capturing failure to roundtrip indented code in tokenize module. 2015-06-20 19:52:22 -04:00
Jason R. Coombs 7cf36387e4 Remove unused import and remove doctest-only import into doctests. 2015-06-20 19:13:50 -04:00
Victor Stinner 24d262af0b (Merge 3.5) Issue #23840: tokenize.open() now closes the temporary binary file 
on error to fix a resource warning.
 2015-05-26 00:46:44 +02:00
Victor Stinner 387729e183 Issue #23840: tokenize.open() now closes the temporary binary file on error to 
fix a resource warning.
 2015-05-26 00:43:58 +02:00
Yury Selivanov 8085b80c18 Issue 24226: Fix parsing of many sequential one-line 'def' statements. 2015-05-18 12:50:52 -04:00
Yury Selivanov 7544508f02 PEP 0492 -- Coroutines with async and await syntax. Issue #24017. 2015-05-11 22:57:16 -04:00
Serhiy Storchaka ee4c0b9dcf Issue #23681: Fixed Python 2 to 3 poring bugs. 
Indexing bytes retiurns an integer, not bytes.
 2015-03-20 16:48:02 +02:00
Serhiy Storchaka 74a49ac3f5 Issue #23681: Fixed Python 2 to 3 poring bugs. 
Indexing bytes retiurns an integer, not bytes.
 2015-03-20 16:46:19 +02:00
Benjamin Peterson d51374ed78 PEP 465: a dedicated infix operator for matrix multiplication (closes #21176) 2014-04-09 23:55:56 -04:00
Terry Jan Reedy 9dc3a36c84 Issue #9974: When untokenizing, use row info to insert backslash+newline. 
Original patches by A. Kuchling and G. Rees (#12691).
 2014-02-23 23:33:08 -05:00
Terry Jan Reedy 938ba685dc Issue #20750, Enable roundtrip tests for new 5-tuple untokenize. The 
constructed examples and all but 7 of the test/test_*.py files (run with -ucpu)
pass. Remove those that fail the new test from the selection list.
Patch partly based on patches by G. Brandl (#8478) and G. Rees (#12691).
 2014-02-23 18:00:31 -05:00
Terry Jan Reedy 5b8d2c3af7 Issue #8478: Untokenizer.compat now processes first token from iterator input. 
Patch based on lines from Georg Brandl, Eric Snow, and Gareth Rees.
 2014-02-17 23:12:16 -05:00
Terry Jan Reedy 58edfd9ff1 whitespace 2014-02-17 16:49:06 -05:00
Terry Jan Reedy 5e6db31368 Untokenize: An logically incorrect assert tested user input validity. 
Replace it with correct logic that raises ValueError for bad input.
Issues #8478 and #12691 reported the incorrect logic.
Add an Untokenize test case and an initial test method.
 2014-02-17 16:45:48 -05:00
Serhiy Storchaka 768c16ce02 Issue #18960: Fix bugs with Python source code encoding in the second line. 
* The first line of Python script could be executed twice when the source
encoding (not equal to 'utf-8') was specified on the second line.

* Now the source encoding declaration on the second line isn't effective if
the first line contains anything except a comment.

* As a consequence, 'python -x' works now again with files with the source
encoding declarations specified on the second file, and can be used again
to make Python batch files on Windows.

* The tokenize module now ignore the source encoding declaration on the second
line if the first line contains anything except a comment.

* IDLE now ignores the source encoding declaration on the second line if the
first line contains anything except a comment.

* 2to3 and the findnocoding.py script now ignore the source encoding
declaration on the second line if the first line contains anything except
a comment.
 2014-01-09 18:36:09 +02:00
Serhiy Storchaka dafea85190 Issue #18873: The tokenize module, IDLE, 2to3, and the findnocoding.py script 
now detect Python source code encoding only in comment lines.
 2013-09-16 23:51:56 +03:00
Ezio Melotti fafa8b7797 #16152: merge with 3.2. 2012-11-03 17:46:51 +02:00
Ezio Melotti 2cc3b4ba9f #16152: fix tokenize to ignore whitespace at the end of the code when no newline is found. Patch by Ned Batchelder. 2012-11-03 17:38:43 +02:00
Florent Xicluna fed2c51eea Merge branch 2012-07-07 12:26:56 +02:00
Florent Xicluna 11f0b41e9d Issue #14990: tokenize: correctly fail with SyntaxError on invalid encoding declaration. 2012-07-07 12:13:35 +02:00
Christian Heimes 0b3847de6d Issue #15096: Drop support for the ur string prefix 2012-06-20 11:17:58 +02:00
Meador Inge 8d5c0b8c19 Issue #15054: Fix incorrect tokenization of 'b' string literals. 
Patch by Serhiy Storchaka.
 2012-06-16 21:49:08 -05:00
Brett Cannon c33f3f2339 Issue #14629: Mention the filename in SyntaxError exceptions from 
tokenizer.detect_encoding() (when available).
 2012-04-20 13:23:54 -04:00
Martin v. Löwis 63c39fe38e merge 3.2: issue 14629 2012-04-20 14:37:17 +02:00
Martin v. Löwis 63674f4b52 Issue #14629: Raise SyntaxError in tokenizer.detect_encoding 
if the first two lines have non-UTF-8 characters without an encoding declaration.
 2012-04-20 14:36:47 +02:00
Armin Ronacher c0eaecafe9 Updated tokenize to support the inverse byte literals new in 3.3 2012-03-04 13:07:57 +00:00
Meador Inge 00c7f85298 Issue #2134: Add support for tokenize.TokenInfo.exact_type. 2012-01-19 00:44:45 -06:00
Ezio Melotti d8b509b192 #13012: use splitlines(keepends=True/False) instead of splitlines(0/1). 2011-09-28 17:37:55 +03:00
Benjamin Peterson 963e40256a tokenize is just broken on test_pep3131.py 2011-08-13 00:33:21 -05:00
Ned Deily 2ea6fccf64 Issue #12587: Correct faulty test file and reference in test_tokenize. 
(Patch by Robert Xiao)
 2011-07-19 16:15:27 -07:00
Ezio Melotti b3aedd4862 #9424: Replace deprecated assert* methods in the Python test suite. 2010-11-20 19:04:17 +00:00
Victor Stinner 92665ab8c7 test_tokenize: use self.assertEqual() instead of plain assert 2010-11-09 01:11:31 +00:00
Victor Stinner 58c0752a33 Issue #10335: Add tokenize.open(), detect the file encoding using 
tokenize.detect_encoding() and open it in read only mode.
 2010-11-09 01:08:59 +00:00
Brian Curtin 9f5f65c736 Fix #10258 - clean up resource warning 2010-10-30 21:35:28 +00:00
Antoine Pitrou 5bc4fa7a49 Replace the "compiler" resource with the more generic "cpu", so 
as to mark CPU-heavy tests.
 2010-10-14 15:34:31 +00:00
Benjamin Peterson 33856de84d handle names starting with non-ascii characters correctly #9712 2010-08-30 14:41:20 +00:00
Benjamin Peterson e992a7da01 remove pointless coding cookie 2010-08-30 13:57:28 +00:00
Mark Dickinson 388122d43b Issue #9337: Make float.__str__ identical to float.__repr__. 
(And similarly for complex numbers.)
 2010-08-04 20:56:28 +00:00
Mark Dickinson 3c0b317e36 Merged revisions 82356 via svnmerge from 
svn+ssh://pythondev@svn.python.org/python/trunk

........
  r82356 | mark.dickinson | 2010-06-29 08:37:25 +0100 (Tue, 29 Jun 2010) | 1 line

  Spelling.
........
 2010-06-29 07:38:37 +00:00