Commit Graph

84 Commits

Author SHA1 Message Date
Eric V. Smith 1c8222c80a Issue 25311: Add support for f-strings to tokenize.py. Also added some comments to explain what's happening, since it's not so obvious. 2015-10-26 04:37:55 -04:00
Eric V. Smith 6731774216 Issue 25422: Add tests for multi-line string tokenization. Also remove truncated tokens. 2015-10-16 20:45:53 -04:00
Serhiy Storchaka 6f5175de15 Issue #25317: Converted doctests in test_tokenize to unittests.
Made test_tokenize discoverable.
2015-10-06 18:23:12 +03:00
Serhiy Storchaka 5f6fa82617 Issue #25317: Converted doctests in test_tokenize to unittests.
Made test_tokenize discoverable.
2015-10-06 18:16:28 +03:00
Yury Selivanov 96ec934e75 Issue #24619: Simplify async/await tokenization.
This commit simplifies async/await tokenization in tokenizer.c,
tokenize.py & lib2to3/tokenize.py.  Previous solution was to keep
a stack of async-def & def blocks, whereas the new approach is just
to remember position of the outermost async-def block.

This change won't bring any parsing performance improvements, but
it makes the code much easier to read and validate.
2015-07-23 15:01:58 +03:00
Yury Selivanov 8fb307cd65 Issue #24619: New approach for tokenizing async/await.
This commit fixes how one-line async-defs and defs are tracked
by tokenizer.  It allows to correctly parse invalid code such
as:

>>> async def f():
...     def g(): pass
...     async = 10

and valid code such as:

>>> async def f():
...     async def g(): pass
...     await z

As a consequence, is is now possible to have one-line
'async def foo(): await ..' functions:

>>> async def foo(): return await bar()
2015-07-22 13:33:45 +03:00
Jason R. Coombs a95a476b3a Issue #20387: Merge test and patch from 3.4.4 2015-06-28 11:13:30 -04:00
Jason R. Coombs b6d1cdda8e Issue #20387: Correct test to properly capture expectation. 2015-06-25 22:42:24 -04:00
Jason R. Coombs 5713b3c5bf Issue #20387: Add test capturing failure to roundtrip indented code in tokenize module. 2015-06-20 19:52:22 -04:00
Jason R. Coombs 7cf36387e4 Remove unused import and remove doctest-only import into doctests. 2015-06-20 19:13:50 -04:00
Victor Stinner 24d262af0b (Merge 3.5) Issue #23840: tokenize.open() now closes the temporary binary file
on error to fix a resource warning.
2015-05-26 00:46:44 +02:00
Victor Stinner 387729e183 Issue #23840: tokenize.open() now closes the temporary binary file on error to
fix a resource warning.
2015-05-26 00:43:58 +02:00
Yury Selivanov 8085b80c18 Issue 24226: Fix parsing of many sequential one-line 'def' statements. 2015-05-18 12:50:52 -04:00
Yury Selivanov 7544508f02 PEP 0492 -- Coroutines with async and await syntax. Issue #24017. 2015-05-11 22:57:16 -04:00
Serhiy Storchaka ee4c0b9dcf Issue #23681: Fixed Python 2 to 3 poring bugs.
Indexing bytes retiurns an integer, not bytes.
2015-03-20 16:48:02 +02:00
Serhiy Storchaka 74a49ac3f5 Issue #23681: Fixed Python 2 to 3 poring bugs.
Indexing bytes retiurns an integer, not bytes.
2015-03-20 16:46:19 +02:00
Benjamin Peterson d51374ed78 PEP 465: a dedicated infix operator for matrix multiplication (closes #21176) 2014-04-09 23:55:56 -04:00
Terry Jan Reedy 9dc3a36c84 Issue #9974: When untokenizing, use row info to insert backslash+newline.
Original patches by A. Kuchling and G. Rees (#12691).
2014-02-23 23:33:08 -05:00
Terry Jan Reedy 938ba685dc Issue #20750, Enable roundtrip tests for new 5-tuple untokenize. The
constructed examples and all but 7 of the test/test_*.py files (run with -ucpu)
pass. Remove those that fail the new test from the selection list.
Patch partly based on patches by G. Brandl (#8478) and G. Rees (#12691).
2014-02-23 18:00:31 -05:00
Terry Jan Reedy 5b8d2c3af7 Issue #8478: Untokenizer.compat now processes first token from iterator input.
Patch based on lines from Georg Brandl, Eric Snow, and Gareth Rees.
2014-02-17 23:12:16 -05:00
Terry Jan Reedy 58edfd9ff1 whitespace 2014-02-17 16:49:06 -05:00
Terry Jan Reedy 5e6db31368 Untokenize: An logically incorrect assert tested user input validity.
Replace it with correct logic that raises ValueError for bad input.
Issues #8478 and #12691 reported the incorrect logic.
Add an Untokenize test case and an initial test method.
2014-02-17 16:45:48 -05:00
Serhiy Storchaka 768c16ce02 Issue #18960: Fix bugs with Python source code encoding in the second line.
* The first line of Python script could be executed twice when the source
encoding (not equal to 'utf-8') was specified on the second line.

* Now the source encoding declaration on the second line isn't effective if
the first line contains anything except a comment.

* As a consequence, 'python -x' works now again with files with the source
encoding declarations specified on the second file, and can be used again
to make Python batch files on Windows.

* The tokenize module now ignore the source encoding declaration on the second
line if the first line contains anything except a comment.

* IDLE now ignores the source encoding declaration on the second line if the
first line contains anything except a comment.

* 2to3 and the findnocoding.py script now ignore the source encoding
declaration on the second line if the first line contains anything except
a comment.
2014-01-09 18:36:09 +02:00
Serhiy Storchaka dafea85190 Issue #18873: The tokenize module, IDLE, 2to3, and the findnocoding.py script
now detect Python source code encoding only in comment lines.
2013-09-16 23:51:56 +03:00
Ezio Melotti fafa8b7797 #16152: merge with 3.2. 2012-11-03 17:46:51 +02:00
Ezio Melotti 2cc3b4ba9f #16152: fix tokenize to ignore whitespace at the end of the code when no newline is found. Patch by Ned Batchelder. 2012-11-03 17:38:43 +02:00
Florent Xicluna fed2c51eea Merge branch 2012-07-07 12:26:56 +02:00
Florent Xicluna 11f0b41e9d Issue #14990: tokenize: correctly fail with SyntaxError on invalid encoding declaration. 2012-07-07 12:13:35 +02:00
Christian Heimes 0b3847de6d Issue #15096: Drop support for the ur string prefix 2012-06-20 11:17:58 +02:00
Meador Inge 8d5c0b8c19 Issue #15054: Fix incorrect tokenization of 'b' string literals.
Patch by Serhiy Storchaka.
2012-06-16 21:49:08 -05:00
Brett Cannon c33f3f2339 Issue #14629: Mention the filename in SyntaxError exceptions from
tokenizer.detect_encoding() (when available).
2012-04-20 13:23:54 -04:00
Martin v. Löwis 63c39fe38e merge 3.2: issue 14629 2012-04-20 14:37:17 +02:00
Martin v. Löwis 63674f4b52 Issue #14629: Raise SyntaxError in tokenizer.detect_encoding
if the first two lines have non-UTF-8 characters without an encoding declaration.
2012-04-20 14:36:47 +02:00
Armin Ronacher c0eaecafe9 Updated tokenize to support the inverse byte literals new in 3.3 2012-03-04 13:07:57 +00:00
Meador Inge 00c7f85298 Issue #2134: Add support for tokenize.TokenInfo.exact_type. 2012-01-19 00:44:45 -06:00
Ezio Melotti d8b509b192 #13012: use splitlines(keepends=True/False) instead of splitlines(0/1). 2011-09-28 17:37:55 +03:00
Benjamin Peterson 963e40256a tokenize is just broken on test_pep3131.py 2011-08-13 00:33:21 -05:00
Ned Deily 2ea6fccf64 Issue #12587: Correct faulty test file and reference in test_tokenize.
(Patch by Robert Xiao)
2011-07-19 16:15:27 -07:00
Ezio Melotti b3aedd4862 #9424: Replace deprecated assert* methods in the Python test suite. 2010-11-20 19:04:17 +00:00
Victor Stinner 92665ab8c7 test_tokenize: use self.assertEqual() instead of plain assert 2010-11-09 01:11:31 +00:00
Victor Stinner 58c0752a33 Issue #10335: Add tokenize.open(), detect the file encoding using
tokenize.detect_encoding() and open it in read only mode.
2010-11-09 01:08:59 +00:00
Brian Curtin 9f5f65c736 Fix #10258 - clean up resource warning 2010-10-30 21:35:28 +00:00
Antoine Pitrou 5bc4fa7a49 Replace the "compiler" resource with the more generic "cpu", so
as to mark CPU-heavy tests.
2010-10-14 15:34:31 +00:00
Benjamin Peterson 33856de84d handle names starting with non-ascii characters correctly #9712 2010-08-30 14:41:20 +00:00
Benjamin Peterson e992a7da01 remove pointless coding cookie 2010-08-30 13:57:28 +00:00
Mark Dickinson 388122d43b Issue #9337: Make float.__str__ identical to float.__repr__.
(And similarly for complex numbers.)
2010-08-04 20:56:28 +00:00
Mark Dickinson 3c0b317e36 Merged revisions 82356 via svnmerge from
svn+ssh://pythondev@svn.python.org/python/trunk

........
  r82356 | mark.dickinson | 2010-06-29 08:37:25 +0100 (Tue, 29 Jun 2010) | 1 line

  Spelling.
........
2010-06-29 07:38:37 +00:00
Benjamin Peterson 689a558098 in tokenize.detect_encoding(), return utf-8-sig when a BOM is found 2010-03-18 22:29:52 +00:00
Benjamin Peterson a0dfa82eca Merged revisions 75149,75260-75263,75265-75267,75292,75300,75376,75405,75429-75433,75437,75445,75501,75551,75572,75589-75591,75657,75742,75868,75952-75957,76057,76105,76139,76143,76162,76223 via svnmerge from
svn+ssh://pythondev@svn.python.org/python/trunk

........
  r75149 | gregory.p.smith | 2009-09-29 16:56:31 -0500 (Tue, 29 Sep 2009) | 3 lines

  Mention issue6972 in extractall docs about overwriting things outside of
  the supplied path.
........
  r75260 | andrew.kuchling | 2009-10-05 16:24:20 -0500 (Mon, 05 Oct 2009) | 1 line

  Wording fix
........
  r75261 | andrew.kuchling | 2009-10-05 16:24:35 -0500 (Mon, 05 Oct 2009) | 1 line

  Fix narkup
........
  r75262 | andrew.kuchling | 2009-10-05 16:25:03 -0500 (Mon, 05 Oct 2009) | 1 line

  Document 'skip' parameter to constructor
........
  r75263 | andrew.kuchling | 2009-10-05 16:25:35 -0500 (Mon, 05 Oct 2009) | 1 line

  Note side benefit of socket.create_connection()
........
  r75265 | andrew.kuchling | 2009-10-05 17:31:11 -0500 (Mon, 05 Oct 2009) | 1 line

  Reword sentence
........
  r75266 | andrew.kuchling | 2009-10-05 17:32:48 -0500 (Mon, 05 Oct 2009) | 1 line

  Use standard comma punctuation; reword some sentences in the docs
........
  r75267 | andrew.kuchling | 2009-10-05 17:42:56 -0500 (Mon, 05 Oct 2009) | 1 line

  Backport r73983: Document the thousands separator.
........
  r75292 | benjamin.peterson | 2009-10-08 22:11:36 -0500 (Thu, 08 Oct 2009) | 1 line

  death to old CVS keyword
........
  r75300 | benjamin.peterson | 2009-10-09 16:48:14 -0500 (Fri, 09 Oct 2009) | 1 line

  fix some coding style
........
  r75376 | benjamin.peterson | 2009-10-11 20:26:07 -0500 (Sun, 11 Oct 2009) | 1 line

  platform we don't care about
........
  r75405 | neil.schemenauer | 2009-10-14 12:17:14 -0500 (Wed, 14 Oct 2009) | 4 lines

  Issue #1754094: Improve the stack depth calculation in the compiler.
  There should be no other effect than a small decrease in memory use.
  Patch by Christopher Tur Lesniewski-Laas.
........
  r75429 | benjamin.peterson | 2009-10-14 20:47:28 -0500 (Wed, 14 Oct 2009) | 1 line

  pep8ify if blocks
........
  r75430 | benjamin.peterson | 2009-10-14 20:49:37 -0500 (Wed, 14 Oct 2009) | 1 line

  use floor division and add a test that exercises the tabsize codepath
........
  r75431 | benjamin.peterson | 2009-10-14 20:56:25 -0500 (Wed, 14 Oct 2009) | 1 line

  change test to what I intended
........
  r75432 | benjamin.peterson | 2009-10-14 22:05:39 -0500 (Wed, 14 Oct 2009) | 1 line

  some cleanups
........
  r75433 | benjamin.peterson | 2009-10-14 22:06:55 -0500 (Wed, 14 Oct 2009) | 1 line

  make inspect.isabstract() always return a boolean; add a test for it, too #7069
........
  r75437 | benjamin.peterson | 2009-10-15 10:44:46 -0500 (Thu, 15 Oct 2009) | 1 line

  only clear a module's __dict__ if the module is the only one with a reference to it #7140
........
  r75445 | vinay.sajip | 2009-10-16 09:06:44 -0500 (Fri, 16 Oct 2009) | 1 line

  Issue #7120: logging: Removed import of multiprocessing which is causing crash in GAE.
........
  r75501 | antoine.pitrou | 2009-10-18 13:37:11 -0500 (Sun, 18 Oct 2009) | 3 lines

  Add a comment about unreachable code, and fix a typo
........
  r75551 | benjamin.peterson | 2009-10-19 22:14:10 -0500 (Mon, 19 Oct 2009) | 1 line

  use property api
........
  r75572 | benjamin.peterson | 2009-10-20 16:55:17 -0500 (Tue, 20 Oct 2009) | 1 line

  clarify buffer arg #7178
........
  r75589 | benjamin.peterson | 2009-10-21 21:26:47 -0500 (Wed, 21 Oct 2009) | 1 line

  whitespace
........
  r75590 | benjamin.peterson | 2009-10-21 21:36:47 -0500 (Wed, 21 Oct 2009) | 1 line

  rewrite to be nice to other implementations
........
  r75591 | benjamin.peterson | 2009-10-21 21:50:38 -0500 (Wed, 21 Oct 2009) | 4 lines

  rewrite for style, clarify, and comments

  Also, use the hasattr() like scheme of allowing BaseException exceptions through.
........
  r75657 | antoine.pitrou | 2009-10-24 07:41:27 -0500 (Sat, 24 Oct 2009) | 3 lines

  Fix compilation error in debug mode.
........
  r75742 | benjamin.peterson | 2009-10-26 17:51:16 -0500 (Mon, 26 Oct 2009) | 1 line

  use 'is' instead of id()
........
  r75868 | benjamin.peterson | 2009-10-27 15:59:18 -0500 (Tue, 27 Oct 2009) | 1 line

  test expect base classes
........
  r75952 | georg.brandl | 2009-10-29 15:38:32 -0500 (Thu, 29 Oct 2009) | 1 line

  Use the correct function name in docstring.
........
  r75953 | georg.brandl | 2009-10-29 15:39:50 -0500 (Thu, 29 Oct 2009) | 1 line

  Remove mention of the old -X command line switch.
........
  r75954 | georg.brandl | 2009-10-29 15:53:00 -0500 (Thu, 29 Oct 2009) | 1 line

  Use constants instead of magic integers for test result.  Do not re-run with --verbose3 for environment changing tests.
........
  r75955 | georg.brandl | 2009-10-29 15:54:03 -0500 (Thu, 29 Oct 2009) | 1 line

  Use a single style for all the docstrings in the math module.
........
  r75956 | georg.brandl | 2009-10-29 16:16:34 -0500 (Thu, 29 Oct 2009) | 1 line

  I do not think the "railroad" program mentioned is still available.
........
  r75957 | georg.brandl | 2009-10-29 16:44:56 -0500 (Thu, 29 Oct 2009) | 1 line

  Fix constant name.
........
  r76057 | benjamin.peterson | 2009-11-02 09:06:45 -0600 (Mon, 02 Nov 2009) | 1 line

  prevent a rather unlikely segfault
........
  r76105 | georg.brandl | 2009-11-04 01:38:12 -0600 (Wed, 04 Nov 2009) | 1 line

  #7259: show correct equivalent for operator.i* operations in docstring; fix minor issues in operator docs.
........
  r76139 | benjamin.peterson | 2009-11-06 19:04:38 -0600 (Fri, 06 Nov 2009) | 1 line

  spelling
........
  r76143 | georg.brandl | 2009-11-07 02:26:07 -0600 (Sat, 07 Nov 2009) | 1 line

  #7271: fix typo.
........
  r76162 | benjamin.peterson | 2009-11-08 22:10:53 -0600 (Sun, 08 Nov 2009) | 1 line

  discuss how to use -p
........
  r76223 | georg.brandl | 2009-11-12 02:29:46 -0600 (Thu, 12 Nov 2009) | 1 line

  Give the profile module a module directive.
........
2009-11-13 02:25:08 +00:00
Benjamin Peterson d3afadaa49 normalize latin-1 and utf-8 variant encodings like the builtin tokenizer does 2009-10-09 21:43:09 +00:00