Commit Graph

116 Commits

Author SHA1 Message Date
Jason R. Coombs a95a476b3a Issue #20387: Merge test and patch from 3.4.4 2015-06-28 11:13:30 -04:00
Dingyuan Wang e411b6629f Issue #20387: Restore retention of indentation during untokenize. 2015-06-22 10:01:12 +08:00
Victor Stinner 24d262af0b (Merge 3.5) Issue #23840: tokenize.open() now closes the temporary binary file
on error to fix a resource warning.
2015-05-26 00:46:44 +02:00
Victor Stinner 387729e183 Issue #23840: tokenize.open() now closes the temporary binary file on error to
fix a resource warning.
2015-05-26 00:43:58 +02:00
Yury Selivanov 7544508f02 PEP 0492 -- Coroutines with async and await syntax. Issue #24017. 2015-05-11 22:57:16 -04:00
Serhiy Storchaka ca8b64461d Issue #23615: Modules bz2, tarfile and tokenize now can be reloaded with
imp.reload().  Patch by Thomas Kluyver.
2015-03-11 17:31:33 +02:00
Serhiy Storchaka cf4a2f29ad Issue #23615: Modules bz2, tarfile and tokenize now can be reloaded with
imp.reload().  Patch by Thomas Kluyver.
2015-03-11 17:18:03 +02:00
Serhiy Storchaka 845b14cc8e Removed duplicated dict entries. 2015-01-11 12:48:17 +02:00
Victor Stinner 969175091c Issue #22599: Enhance tokenize.open() to be able to call it during Python
finalization.

Before the module kept a reference to the builtins module, but the module
attributes are cleared during Python finalization. Instead, keep directly a
reference to the open() function.

This enhancement is not perfect, calling tokenize.open() can still fail if
called very late during Python finalization.  Usually, the function is called
by the linecache module which is called to display a traceback or emit a
warning.
2014-12-05 10:17:10 +01:00
Victor Stinner 9d279b87d8 (Merge 3.4) Issue #22599: Enhance tokenize.open() to be able to call it during
Python finalization.

Before the module kept a reference to the builtins module, but the module
attributes are cleared during Python finalization. Instead, keep directly a
reference to the open() function.

This enhancement is not perfect, calling tokenize.open() can still fail if
called very late during Python finalization.  Usually, the function is called
by the linecache module which is called to display a traceback or emit a
warning.
2014-12-05 10:18:30 +01:00
Benjamin Peterson d51374ed78 PEP 465: a dedicated infix operator for matrix multiplication (closes #21176) 2014-04-09 23:55:56 -04:00
Terry Jan Reedy 58719a7de7 Merge with 3.3 2014-02-23 23:40:16 -05:00
Terry Jan Reedy f106f8f29c whitespace 2014-02-23 23:39:57 -05:00
Terry Jan Reedy 40f8c6774b Merge with 3.3 2014-02-23 23:33:44 -05:00
Terry Jan Reedy 9dc3a36c84 Issue #9974: When untokenizing, use row info to insert backslash+newline.
Original patches by A. Kuchling and G. Rees (#12691).
2014-02-23 23:33:08 -05:00
Terry Jan Reedy 79bf89986c Merge with 3.3 2014-02-17 23:12:37 -05:00
Terry Jan Reedy 5b8d2c3af7 Issue #8478: Untokenizer.compat now processes first token from iterator input.
Patch based on lines from Georg Brandl, Eric Snow, and Gareth Rees.
2014-02-17 23:12:16 -05:00
Terry Jan Reedy 8c8d77254f Untokenize, bad assert: Merge with 3.3 2014-02-17 16:46:43 -05:00
Terry Jan Reedy 5e6db31368 Untokenize: An logically incorrect assert tested user input validity.
Replace it with correct logic that raises ValueError for bad input.
Issues #8478 and #12691 reported the incorrect logic.
Add an Untokenize test case and an initial test method.
2014-02-17 16:45:48 -05:00
Serhiy Storchaka 7282ff6d5b Issue #18960: Fix bugs with Python source code encoding in the second line.
* The first line of Python script could be executed twice when the source
encoding (not equal to 'utf-8') was specified on the second line.

* Now the source encoding declaration on the second line isn't effective if
the first line contains anything except a comment.

* As a consequence, 'python -x' works now again with files with the source
encoding declarations specified on the second file, and can be used again
to make Python batch files on Windows.

* The tokenize module now ignore the source encoding declaration on the second
line if the first line contains anything except a comment.

* IDLE now ignores the source encoding declaration on the second line if the
first line contains anything except a comment.

* 2to3 and the findnocoding.py script now ignore the source encoding
declaration on the second line if the first line contains anything except
a comment.
2014-01-09 18:41:59 +02:00
Serhiy Storchaka 768c16ce02 Issue #18960: Fix bugs with Python source code encoding in the second line.
* The first line of Python script could be executed twice when the source
encoding (not equal to 'utf-8') was specified on the second line.

* Now the source encoding declaration on the second line isn't effective if
the first line contains anything except a comment.

* As a consequence, 'python -x' works now again with files with the source
encoding declarations specified on the second file, and can be used again
to make Python batch files on Windows.

* The tokenize module now ignore the source encoding declaration on the second
line if the first line contains anything except a comment.

* IDLE now ignores the source encoding declaration on the second line if the
first line contains anything except a comment.

* 2to3 and the findnocoding.py script now ignore the source encoding
declaration on the second line if the first line contains anything except
a comment.
2014-01-09 18:36:09 +02:00
Ezio Melotti 5833c00427 #19620: merge with 3.3. 2013-11-25 05:16:09 +02:00
Ezio Melotti 4bcc796acc #19620: Fix typo in docstring (noticed by Christopher Welborn). 2013-11-25 05:14:51 +02:00
Serhiy Storchaka 935349406a Issue #18873: The tokenize module, IDLE, 2to3, and the findnocoding.py script
now detect Python source code encoding only in comment lines.
2013-09-16 23:57:00 +03:00
Serhiy Storchaka dafea85190 Issue #18873: The tokenize module, IDLE, 2to3, and the findnocoding.py script
now detect Python source code encoding only in comment lines.
2013-09-16 23:51:56 +03:00
Andrew Svetlov f7a17b48d7 Replace IOError with OSError (#16715) 2012-12-25 16:47:37 +02:00
Ezio Melotti fafa8b7797 #16152: merge with 3.2. 2012-11-03 17:46:51 +02:00
Ezio Melotti 2cc3b4ba9f #16152: fix tokenize to ignore whitespace at the end of the code when no newline is found. Patch by Ned Batchelder. 2012-11-03 17:38:43 +02:00
Florent Xicluna fed2c51eea Merge branch 2012-07-07 12:26:56 +02:00
Florent Xicluna 11f0b41e9d Issue #14990: tokenize: correctly fail with SyntaxError on invalid encoding declaration. 2012-07-07 12:13:35 +02:00
Christian Heimes 0b3847de6d Issue #15096: Drop support for the ur string prefix 2012-06-20 11:17:58 +02:00
Meador Inge 8d5c0b8c19 Issue #15054: Fix incorrect tokenization of 'b' string literals.
Patch by Serhiy Storchaka.
2012-06-16 21:49:08 -05:00
Brett Cannon c33f3f2339 Issue #14629: Mention the filename in SyntaxError exceptions from
tokenizer.detect_encoding() (when available).
2012-04-20 13:23:54 -04:00
Martin v. Löwis 63c39fe38e merge 3.2: issue 14629 2012-04-20 14:37:17 +02:00
Martin v. Löwis 63674f4b52 Issue #14629: Raise SyntaxError in tokenizer.detect_encoding
if the first two lines have non-UTF-8 characters without an encoding declaration.
2012-04-20 14:36:47 +02:00
Armin Ronacher c0eaecafe9 Updated tokenize to support the inverse byte literals new in 3.3 2012-03-04 13:07:57 +00:00
Armin Ronacher 6ecf77b3f8 Basic support for PEP 414 without docs or tests. 2012-03-04 12:04:06 +00:00
Meador Inge 00c7f85298 Issue #2134: Add support for tokenize.TokenInfo.exact_type. 2012-01-19 00:44:45 -06:00
Antoine Pitrou 10a99b024d Issue #13150: The tokenize module doesn't compile large regular expressions at startup anymore.
Instead, the re module's standard caching does its work.
2011-10-11 15:45:56 +02:00
Meador Inge 14c0f03b58 Issue #12943: python -m tokenize support has been added to tokenize. 2011-10-07 08:53:38 -05:00
Brett Cannon 45b96d373e Merged revisions 88498 via svnmerge from
svn+ssh://pythondev@svn.python.org/python/branches/py3k

........
  r88498 | brett.cannon | 2011-02-21 19:25:12 -0800 (Mon, 21 Feb 2011) | 8 lines

  Issue #11074: Make 'tokenize' so it can be reloaded.

  The module stored away the 'open' object as found in the global namespace
  (which fell through to the built-in namespace) since it defined its own 'open'.
  Problem is that if you reloaded the module it then grabbed the 'open' defined
  in the previous load, leading to code that infinite recursed. Switched to
  simply call builtins.open directly.
........
2011-02-22 03:35:18 +00:00
Brett Cannon f3042782af Issue #11074: Make 'tokenize' so it can be reloaded.
The module stored away the 'open' object as found in the global namespace
(which fell through to the built-in namespace) since it defined its own 'open'.
Problem is that if you reloaded the module it then grabbed the 'open' defined
in the previous load, leading to code that infinite recursed. Switched to
simply call builtins.open directly.
2011-02-22 03:25:12 +00:00
Alexander Belopolsky b9d10d08c4 Issue #10386: Added __all__ to token module; this simplifies importing
in tokenize module and prevents leaking of private names through
import *.
2010-11-11 14:07:41 +00:00
Victor Stinner 58c0752a33 Issue #10335: Add tokenize.open(), detect the file encoding using
tokenize.detect_encoding() and open it in read only mode.
2010-11-09 01:08:59 +00:00
Raymond Hettinger a0e79408bc A little bit more readable repr method. 2010-09-09 08:29:05 +00:00
Raymond Hettinger 3fb79c747b Experiment: Let collections.namedtuple() do the work. This should work now that _collections is pre-built. The buildbots will tell us shortly. 2010-09-09 07:15:18 +00:00
Raymond Hettinger 6c60d099e5 Improve the repr for the TokenInfo named tuple. 2010-09-09 04:32:39 +00:00
Florent Xicluna 43e4ea1b17 Remove unused import, fix typo and rewrap docstrings. 2010-09-03 19:54:02 +00:00
Benjamin Peterson 33856de84d handle names starting with non-ascii characters correctly #9712 2010-08-30 14:41:20 +00:00
Benjamin Peterson 1613ed8108 fix for files with coding cookies and BOMs 2010-03-18 22:34:15 +00:00