Commit Graph

22 Commits

Author SHA1 Message Date
Miss Islington (bot) 51dbae867e
closes bpo-34515: Support non-ASCII identifiers in lib2to3. (GH-8950)
(cherry picked from commit 10a428b64b)

Co-authored-by: Monson Shao <holymonson@gmail.com>
2018-09-15 10:52:57 -07:00
Miss Islington (bot) dd6bdf610b Add support for all string literals to lib2to3 (GH-6457) (GH-6488)
(cherry picked from commit 8f37e84664)

Co-authored-by: Zsolt Dollenstein <zsol.zsol@gmail.com>
2018-04-16 15:16:57 -07:00
Miss Islington (bot) a90df5085b
Revert "bpo-30406: Make async and await proper keywords (GH-1669)" (GH-6143)
This reverts commit ac317700ce.

(Reverts only the lib2to3 part.)
(cherry picked from commit f64aae46da)

Co-authored-by: Jelle Zijlstra <jelle.zijlstra@gmail.com>
2018-03-18 12:15:52 -07:00
Jelle Zijlstra ac317700ce bpo-30406: Make async and await proper keywords (#1669)
Per PEP 492, 'async' and 'await' should become proper keywords in 3.7.
2017-10-05 23:24:46 -04:00
Łukasz Langa 1b9530c536 bpo-23894: make lib2to3 recognize f-strings (#1733)
Note: this doesn't unpack f-strings into the underlying JoinedStr AST.

Ideally we'd fully implement JoinedStr here but given its additional
complexity, I think this is worth bandaiding as is. This unblocks tools like
https://github.com/google/yapf to format 3.6 syntax using f-strings.
2017-05-22 16:35:48 -07:00
Łukasz Langa 0c4aca54dc Make rb'' strings work in lib2to3 (#1724)
This partially solves bpo-23894.
2017-05-22 15:19:09 -07:00
Nevada Sanchez a6e395dffa bpo-29869: Allow underscores in numeric literals in lib2to3. (GH-1119)
* Allow underscores in numeric literals in lib2to3.
* Stricter literal parsing for Python 3.6 in lib2to3.pgen2.tokenize.
* Add test case for underscores in literals in Python 3.
2017-04-13 10:32:54 -07:00
Serhiy Storchaka e431d3c9aa Issue #26581: Use the first coding cookie on a line, not the last one. 2016-03-20 23:36:29 +02:00
Yury Selivanov 96ec934e75 Issue #24619: Simplify async/await tokenization.
This commit simplifies async/await tokenization in tokenizer.c,
tokenize.py & lib2to3/tokenize.py.  Previous solution was to keep
a stack of async-def & def blocks, whereas the new approach is just
to remember position of the outermost async-def block.

This change won't bring any parsing performance improvements, but
it makes the code much easier to read and validate.
2015-07-23 15:01:58 +03:00
Yury Selivanov 8fb307cd65 Issue #24619: New approach for tokenizing async/await.
This commit fixes how one-line async-defs and defs are tracked
by tokenizer.  It allows to correctly parse invalid code such
as:

>>> async def f():
...     def g(): pass
...     async = 10

and valid code such as:

>>> async def f():
...     async def g(): pass
...     await z

As a consequence, is is now possible to have one-line
'async def foo(): await ..' functions:

>>> async def foo(): return await bar()
2015-07-22 13:33:45 +03:00
Yury Selivanov 7544508f02 PEP 0492 -- Coroutines with async and await syntax. Issue #24017. 2015-05-11 22:57:16 -04:00
Benjamin Peterson 4ab92c800a add matrix multiplication operator support to 2to3 2014-04-10 00:12:47 -04:00
Serhiy Storchaka 768c16ce02 Issue #18960: Fix bugs with Python source code encoding in the second line.
* The first line of Python script could be executed twice when the source
encoding (not equal to 'utf-8') was specified on the second line.

* Now the source encoding declaration on the second line isn't effective if
the first line contains anything except a comment.

* As a consequence, 'python -x' works now again with files with the source
encoding declarations specified on the second file, and can be used again
to make Python batch files on Windows.

* The tokenize module now ignore the source encoding declaration on the second
line if the first line contains anything except a comment.

* IDLE now ignores the source encoding declaration on the second line if the
first line contains anything except a comment.

* 2to3 and the findnocoding.py script now ignore the source encoding
declaration on the second line if the first line contains anything except
a comment.
2014-01-09 18:36:09 +02:00
Ezio Melotti 4bcc796acc #19620: Fix typo in docstring (noticed by Christopher Welborn). 2013-11-25 05:14:51 +02:00
Serhiy Storchaka dafea85190 Issue #18873: The tokenize module, IDLE, 2to3, and the findnocoding.py script
now detect Python source code encoding only in comment lines.
2013-09-16 23:51:56 +03:00
Benjamin Peterson 8d26b0be6d Merged revisions 80934 via svnmerge from
svn+ssh://pythondev@svn.python.org/python/trunk

................
  r80934 | benjamin.peterson | 2010-05-07 13:58:23 -0500 (Fri, 07 May 2010) | 69 lines

  Merged revisions 79911,79916-79917,80018,80418,80572-80573,80635-80639,80668,80922 via svnmerge from
  svn+ssh://pythondev@svn.python.org/sandbox/trunk/2to3/lib2to3

  ........
    r79911 | benjamin.peterson | 2010-04-09 15:38:53 -0500 (Fri, 09 Apr 2010) | 1 line

    use absolute import
  ........
    r79916 | benjamin.peterson | 2010-04-09 16:05:21 -0500 (Fri, 09 Apr 2010) | 1 line

    generalize detection of __future__ imports and attach them to the tree
  ........
    r79917 | benjamin.peterson | 2010-04-09 16:11:44 -0500 (Fri, 09 Apr 2010) | 1 line

    don't try to 'fix' relative imports when absolute_import is enabled #8858
  ........
    r80018 | benjamin.peterson | 2010-04-12 16:12:12 -0500 (Mon, 12 Apr 2010) | 4 lines

    prevent diffs from being mangled is multiprocess mode #6409

    Patch by George Boutsioukis.
  ........
    r80418 | benjamin.peterson | 2010-04-23 16:00:03 -0500 (Fri, 23 Apr 2010) | 1 line

    remove unhelpful description
  ........
    r80572 | benjamin.peterson | 2010-04-27 20:33:54 -0500 (Tue, 27 Apr 2010) | 1 line

    use unicode literals
  ........
    r80573 | jeffrey.yasskin | 2010-04-27 23:08:27 -0500 (Tue, 27 Apr 2010) | 6 lines

    Don't transform imports that are already relative.  2to3 turned
      from . import refactor
    into
      from .. import refactor
    which broke the transformation of 2to3 itself.
  ........
    r80635 | benjamin.peterson | 2010-04-29 16:02:23 -0500 (Thu, 29 Apr 2010) | 1 line

    remove imports
  ........
    r80636 | benjamin.peterson | 2010-04-29 16:02:41 -0500 (Thu, 29 Apr 2010) | 1 line

    unicode literal
  ........
    r80637 | benjamin.peterson | 2010-04-29 16:03:42 -0500 (Thu, 29 Apr 2010) | 1 line

    must pass a string to Number
  ........
    r80638 | benjamin.peterson | 2010-04-29 16:05:34 -0500 (Thu, 29 Apr 2010) | 1 line

    unicode literals
  ........
    r80639 | benjamin.peterson | 2010-04-29 16:06:09 -0500 (Thu, 29 Apr 2010) | 1 line

    pass string to Number
  ........
    r80668 | jeffrey.yasskin | 2010-04-30 18:02:47 -0500 (Fri, 30 Apr 2010) | 4 lines

    Make 2to3 run under Python 2.5 so that the benchmark suite at
    http://hg.python.org/benchmarks/ can use it and still run on implementations
    that haven't gotten to 2.6 yet.  Fixes issue 8566.
  ........
    r80922 | benjamin.peterson | 2010-05-07 11:06:25 -0500 (Fri, 07 May 2010) | 1 line

    prevent xrange transformation from wrapping range calls it produces in list
  ........
................
2010-05-07 19:10:11 +00:00
Benjamin Peterson 0af9398d27 Merged revisions 79306,79311,79325 via svnmerge from
svn+ssh://pythondev@svn.python.org/python/trunk

................
  r79306 | benjamin.peterson | 2010-03-22 17:40:06 -0500 (Mon, 22 Mar 2010) | 21 lines

  Merged revisions 79077,79137,79304-79305 via svnmerge from
  svn+ssh://pythondev@svn.python.org/sandbox/trunk/2to3/lib2to3

  ........
    r79077 | benjamin.peterson | 2010-03-18 18:05:29 -0500 (Thu, 18 Mar 2010) | 1 line

    port detect_encoding improvements from py3k
  ........
    r79137 | benjamin.peterson | 2010-03-20 11:12:53 -0500 (Sat, 20 Mar 2010) | 1 line

    add a fixer for setting sys.exitfunc #2356
  ........
    r79304 | benjamin.peterson | 2010-03-22 17:20:22 -0500 (Mon, 22 Mar 2010) | 1 line

    fix test_parser when it's run in a path with spaces #7666
  ........
    r79305 | benjamin.peterson | 2010-03-22 17:27:07 -0500 (Mon, 22 Mar 2010) | 1 line

    normalize whitespace
  ........
................
  r79311 | benjamin.peterson | 2010-03-22 17:54:42 -0500 (Mon, 22 Mar 2010) | 9 lines

  Merged revisions 79309 via svnmerge from
  svn+ssh://pythondev@svn.python.org/sandbox/trunk/2to3/lib2to3

  ........
    r79309 | benjamin.peterson | 2010-03-22 17:50:47 -0500 (Mon, 22 Mar 2010) | 1 line

    pass correct symbol in
  ........
................
  r79325 | benjamin.peterson | 2010-03-22 22:03:55 -0500 (Mon, 22 Mar 2010) | 13 lines

  Merged revisions 79313,79324 via svnmerge from
  svn+ssh://pythondev@svn.python.org/sandbox/trunk/2to3/lib2to3

  ........
    r79313 | benjamin.peterson | 2010-03-22 17:59:57 -0500 (Mon, 22 Mar 2010) | 1 line

    another case where a symbol is needed
  ........
    r79324 | benjamin.peterson | 2010-03-22 21:59:47 -0500 (Mon, 22 Mar 2010) | 1 line

    use unicode literals
  ........
................
2010-03-23 03:22:05 +00:00
Benjamin Peterson 2021100d6d Merged revisions 76259,76326,76376-76377,76430,76471,76517 via svnmerge from
svn+ssh://pythondev@svn.python.org/python/trunk

................
  r76259 | georg.brandl | 2009-11-14 05:50:51 -0600 (Sat, 14 Nov 2009) | 1 line

  Fix terminology.
................
  r76326 | georg.brandl | 2009-11-16 10:44:05 -0600 (Mon, 16 Nov 2009) | 1 line

  #7302: fix link.
................
  r76376 | georg.brandl | 2009-11-18 13:39:14 -0600 (Wed, 18 Nov 2009) | 1 line

  upcase Python
................
  r76377 | georg.brandl | 2009-11-18 14:05:15 -0600 (Wed, 18 Nov 2009) | 1 line

  Fix markup.
................
  r76430 | r.david.murray | 2009-11-20 07:29:43 -0600 (Fri, 20 Nov 2009) | 2 lines

  Issue 7363: fix indentation in socketserver udpserver example.
................
  r76471 | georg.brandl | 2009-11-23 13:53:19 -0600 (Mon, 23 Nov 2009) | 1 line

  #7345: fix arguments of formatyear().
................
  r76517 | benjamin.peterson | 2009-11-25 12:16:46 -0600 (Wed, 25 Nov 2009) | 29 lines

  Merged revisions 76160-76161,76250,76252,76447,76506 via svnmerge from
  svn+ssh://pythondev@svn.python.org/sandbox/trunk/2to3/lib2to3

  ........
    r76160 | benjamin.peterson | 2009-11-08 18:53:48 -0600 (Sun, 08 Nov 2009) | 1 line

    undeprecate the -p option; it's useful for converting python3 sources
  ........
    r76161 | benjamin.peterson | 2009-11-08 19:05:37 -0600 (Sun, 08 Nov 2009) | 1 line

    simplify condition
  ........
    r76250 | benjamin.peterson | 2009-11-13 16:56:48 -0600 (Fri, 13 Nov 2009) | 1 line

    fix handling of a utf-8 bom #7313
  ........
    r76252 | benjamin.peterson | 2009-11-13 16:58:36 -0600 (Fri, 13 Nov 2009) | 1 line

    remove pdb turd
  ........
    r76447 | benjamin.peterson | 2009-11-22 18:17:40 -0600 (Sun, 22 Nov 2009) | 1 line

    #7375 fix nested transformations in fix_urllib
  ........
    r76506 | benjamin.peterson | 2009-11-24 18:34:31 -0600 (Tue, 24 Nov 2009) | 1 line

    use generator expressions in any()
  ........
................
2009-11-25 18:34:42 +00:00
Benjamin Peterson d9af52ba7e Merged revisions 76062 via svnmerge from
svn+ssh://pythondev@svn.python.org/python/trunk

................
  r76062 | benjamin.peterson | 2009-11-02 12:12:12 -0600 (Mon, 02 Nov 2009) | 70 lines

  Merged revisions 74359,75081,75088,75213,75278,75303,75427-75428,75734-75736,75865,76059-76061 via svnmerge from
  svn+ssh://pythondev@svn.python.org/sandbox/trunk/2to3/lib2to3

  ........
    r74359 | benjamin.peterson | 2009-08-12 17:23:13 -0500 (Wed, 12 Aug 2009) | 1 line

    don't pass the deprecated print_function option
  ........
    r75081 | benjamin.peterson | 2009-09-26 22:02:57 -0500 (Sat, 26 Sep 2009) | 1 line

    let 2to3 work with extended iterable unpacking
  ........
    r75088 | benjamin.peterson | 2009-09-27 11:25:21 -0500 (Sun, 27 Sep 2009) | 1 line

    look on the type only for __call__
  ........
    r75213 | benjamin.peterson | 2009-10-03 10:09:46 -0500 (Sat, 03 Oct 2009) | 5 lines

    revert 75212; it's not correct

    People can use isinstance(x, collections.Callable) if they expect objects with
    __call__ in their instance dictionaries.
  ........
    r75278 | benjamin.peterson | 2009-10-07 16:25:56 -0500 (Wed, 07 Oct 2009) | 4 lines

    fix whitespace problems with fix_idioms #3563

    Patch by Joe Amenta.
  ........
    r75303 | benjamin.peterson | 2009-10-09 16:59:11 -0500 (Fri, 09 Oct 2009) | 1 line

    port latin-1 and utf-8 cookie improvements
  ........
    r75427 | benjamin.peterson | 2009-10-14 20:35:57 -0500 (Wed, 14 Oct 2009) | 1 line

    force floor division
  ........
    r75428 | benjamin.peterson | 2009-10-14 20:39:21 -0500 (Wed, 14 Oct 2009) | 1 line

    silence -3 warnings about __hash__
  ........
    r75734 | benjamin.peterson | 2009-10-26 16:25:53 -0500 (Mon, 26 Oct 2009) | 2 lines

    warn on map(None, ...) with more than 2 arguments #7203
  ........
    r75735 | benjamin.peterson | 2009-10-26 16:28:25 -0500 (Mon, 26 Oct 2009) | 1 line

    remove unused result
  ........
    r75736 | benjamin.peterson | 2009-10-26 16:29:02 -0500 (Mon, 26 Oct 2009) | 1 line

    using get() here is a bit pointless
  ........
    r75865 | benjamin.peterson | 2009-10-27 15:49:00 -0500 (Tue, 27 Oct 2009) | 1 line

    explain reason for warning
  ........
    r76059 | benjamin.peterson | 2009-11-02 11:43:47 -0600 (Mon, 02 Nov 2009) | 1 line

    tuples are no longer used for children
  ........
    r76060 | benjamin.peterson | 2009-11-02 11:55:40 -0600 (Mon, 02 Nov 2009) | 1 line

    revert r76059; apparently some fixers rely on Leaf no () for children
  ........
    r76061 | benjamin.peterson | 2009-11-02 12:06:17 -0600 (Mon, 02 Nov 2009) | 1 line

    make fix_tuple_params keep the tree valid #7253
  ........
................
2009-11-02 18:16:28 +00:00
Benjamin Peterson d481e3d791 Merged revisions 72494 via svnmerge from
svn+ssh://pythondev@svn.python.org/python/trunk

................
  r72494 | benjamin.peterson | 2009-05-08 20:01:14 -0500 (Fri, 08 May 2009) | 21 lines

  Merged revisions 72491-72493 via svnmerge from
  svn+ssh://pythondev@svn.python.org/sandbox/trunk/2to3/lib2to3

  ........
    r72491 | benjamin.peterson | 2009-05-08 19:33:27 -0500 (Fri, 08 May 2009) | 7 lines

    make 2to3 use unicode internally on 2.x

    This started out as a fix for #2660, but became this large refactoring
    when I realized the dire state this was in. 2to3 now uses
    tokenize.detect_encoding to decode the files correctly into unicode.
  ........
    r72492 | benjamin.peterson | 2009-05-08 19:35:38 -0500 (Fri, 08 May 2009) | 1 line

    remove compat code
  ........
    r72493 | benjamin.peterson | 2009-05-08 19:54:15 -0500 (Fri, 08 May 2009) | 1 line

    add a test for \r\n newlines
  ........
................
2009-05-09 19:42:23 +00:00
Martin v. Löwis 8a5f8ca33b Run 2to3 on this library. 2008-03-19 05:33:36 +00:00
Martin v. Löwis ef04c44e29 Merged revisions 61596-61597 via svnmerge from
svn+ssh://pythondev@svn.python.org/python/trunk

........
  r61596 | martin.v.loewis | 2008-03-18 23:43:46 -0500 (Di, 18 Mär 2008) | 2 lines

  Import lib2to3.
........
  r61597 | martin.v.loewis | 2008-03-18 23:58:04 -0500 (Di, 18 Mär 2008) | 3 lines

  Initialized merge tracking via "svnmerge" with revisions "1-61595" from
  svn+ssh://pythondev@svn.python.org/sandbox/trunk/2to3/lib2to3
........
2008-03-19 05:04:44 +00:00