Mirror
/
cpython
mirror of https://github.com/python/cpython
4
1
Fork 0
Code Issues Releases Wiki Activity
111,754 Commits 6 Branches 593 Tags 2.9 GiB
Commit Graph

63 Commits

Author SHA1 Message Date
Zsolt Dollenstein 149addd496
make lib2to3 parse async generators everywhere (GH-6588) 2021-08-10 11:31:32 +02:00
Inada Naoki c0ec4486dc
bpo-43651: Fix EncodingWarning in lib2to3/pgen2/pgen.py (GH-25127) 2021-04-02 08:59:15 +09:00
Tim Hatch 3c3aa4516c
lib2to3: Support named assignment expressions (GH-12702) 
There are two copies of the grammar -- the one used by Python itself as
Grammar/Grammar, and the one used by lib2to3 which has necessarily diverged at
Lib/lib2to3/Grammar.txt because it needs to support older syntax an we want it
to be reasonable stable to avoid requiring fixer rewrites.

This brings suport for syntax like `if x:= foo():` to match what the live
Python grammar does.

This should've been added at the time of the walrus operator itself, but lib2to3 being
independent is often overlooked.  So we do consider this a bugfix rather than enhancement.
 2020-04-02 15:34:54 -07:00
Anthony Sottile 2a58b0636d bpo-5028: Fix up rest of documentation for tokenize documenting line (GH-13686) 
https://bugs.python.org/issue5028
 2019-05-30 15:06:32 -07:00
Andrew Carr 1e36f75d63 bpo-5028: fix doc bug for tokenize (GH-11683) 
https://bugs.python.org/issue5028
 2019-05-30 12:31:51 -07:00
penguindustin 9646630895 bpo-36766: Typos in docs and code comments (GH-13116) 2019-05-06 14:57:17 -04:00
Anthony Sottile c57e6e2e52 bpo-35312: Make lib2to3.pgen2.parse.ParseError round-trip pickle-able. (GH-10710) 2018-11-27 20:39:49 +02:00
Srinivas Thatiparthy (శ్రీనివాస్ తాటిపర్తి) b9498e2367 bpo-35202: Remove unused imports in Lib directory. (GH-10446) 2018-11-10 09:22:02 +02:00
Monson Shao 10a428b64b closes bpo-34515: Support non-ASCII identifiers in lib2to3. (GH-8950) 2018-09-15 10:32:29 -07:00
Łukasz Langa 76618061b9
[lib2to3] Make grammar pickling faster (#6491) 
* Now uses pickle protocol 4

* Doesn't wrap the grammar's `__dict__` in ordered dictionaries anymore as
dictionaries in Python 3.6+ are ordered by default

This still produces deterministic pickles (that hash the same with MD5).
Tested with different PYTHONHASHSEED values.
 2018-04-16 17:33:59 -07:00
Zsolt Dollenstein 8f37e84664 Add support for all string literals to lib2to3 (#6457) 2018-04-16 14:33:31 -07:00
Jelle Zijlstra f64aae46da Revert "bpo-30406: Make async and await proper keywords (#1669)" (GH-6143) 
This reverts commit ac317700ce.

(Reverts only the lib2to3 part.)
 2018-03-18 09:54:33 -07:00
Benjamin Peterson d11e8e0d11
correct wording (#4983) 2017-12-22 20:48:13 -08:00
Benjamin Peterson 8a5877165e
bpo-24960: use pkgutil.get_data in lib2to3 to read pickled grammar files (#4977) 
This is more complicated than it should be because we need to preserve the
useful mtime-based regeneration feature that lib2to3.pgen2.driver.load_grammar
has. We only look for the pickled grammar file with pkgutil.get_data and only if
the source file does not exist.
 2017-12-22 12:18:33 -08:00
Jelle Zijlstra ac317700ce bpo-30406: Make async and await proper keywords (#1669) 
Per PEP 492, 'async' and 'await' should become proper keywords in 3.7.
 2017-10-05 23:24:46 -04:00
Victor Stinner 272d888c7b bpo-29783: Replace codecs.open() with io.open() (#599) 2017-06-16 08:59:01 +02:00
Łukasz Langa 1b9530c536 bpo-23894: make lib2to3 recognize f-strings (#1733) 
Note: this doesn't unpack f-strings into the underlying JoinedStr AST.

Ideally we'd fully implement JoinedStr here but given its additional
complexity, I think this is worth bandaiding as is. This unblocks tools like
https://github.com/google/yapf to format 3.6 syntax using f-strings.
 2017-05-22 16:35:48 -07:00
Łukasz Langa 0c4aca54dc Make rb'' strings work in lib2to3 (#1724) 
This partially solves bpo-23894.
 2017-05-22 15:19:09 -07:00
Nevada Sanchez a6e395dffa bpo-29869: Allow underscores in numeric literals in lib2to3. (GH-1119) 
* Allow underscores in numeric literals in lib2to3.
* Stricter literal parsing for Python 3.6 in lib2to3.pgen2.tokenize.
* Add test case for underscores in literals in Python 3.
 2017-04-13 10:32:54 -07:00
Serhiy Storchaka 5affd23e6f bpo-29762: More use "raise from None". (#569) 
This hides unwanted implementation details from tracebacks.
 2017-04-05 09:37:24 +03:00
Victor Stinner d6debb24e0 bpo-29919: Remove unused imports found by pyflakes (#137) 
Make also minor PEP8 coding style fixes on modified imports.
 2017-03-27 16:05:26 +02:00
Gregory P. Smith ext:(%20%5BGoogle%20Inc.%5D) 9719161858 Fix indentation (reindent.py). 2016-09-08 00:48:07 +00:00
Gregory P. Smith ext:(%20%5BGoogle%20Inc.%5D) dd1c638b92 lib2to3.pgen3.driver.load_grammar() now creates a stable cache file 
between runs given the same Grammar.txt input regardless of the hash
randomization setting.
 2016-09-08 00:40:07 +00:00
Serhiy Storchaka e431d3c9aa Issue #26581: Use the first coding cookie on a line, not the last one. 2016-03-20 23:36:29 +02:00
Yury Selivanov 96ec934e75 Issue #24619: Simplify async/await tokenization. 
This commit simplifies async/await tokenization in tokenizer.c,
tokenize.py & lib2to3/tokenize.py.  Previous solution was to keep
a stack of async-def & def blocks, whereas the new approach is just
to remember position of the outermost async-def block.

This change won't bring any parsing performance improvements, but
it makes the code much easier to read and validate.
 2015-07-23 15:01:58 +03:00
Yury Selivanov 8fb307cd65 Issue #24619: New approach for tokenizing async/await. 
This commit fixes how one-line async-defs and defs are tracked
by tokenizer.  It allows to correctly parse invalid code such
as:

>>> async def f():
...     def g(): pass
...     async = 10

and valid code such as:

>>> async def f():
...     async def g(): pass
...     await z

As a consequence, is is now possible to have one-line
'async def foo(): await ..' functions:

>>> async def foo(): return await bar()
 2015-07-22 13:33:45 +03:00
Yury Selivanov 7544508f02 PEP 0492 -- Coroutines with async and await syntax. Issue #24017. 2015-05-11 22:57:16 -04:00
Benjamin Peterson 4ab92c800a add matrix multiplication operator support to 2to3 2014-04-10 00:12:47 -04:00
Serhiy Storchaka 7282ff6d5b Issue #18960: Fix bugs with Python source code encoding in the second line. 
* The first line of Python script could be executed twice when the source
encoding (not equal to 'utf-8') was specified on the second line.

* Now the source encoding declaration on the second line isn't effective if
the first line contains anything except a comment.

* As a consequence, 'python -x' works now again with files with the source
encoding declarations specified on the second file, and can be used again
to make Python batch files on Windows.

* The tokenize module now ignore the source encoding declaration on the second
line if the first line contains anything except a comment.

* IDLE now ignores the source encoding declaration on the second line if the
first line contains anything except a comment.

* 2to3 and the findnocoding.py script now ignore the source encoding
declaration on the second line if the first line contains anything except
a comment.
 2014-01-09 18:41:59 +02:00
Serhiy Storchaka 768c16ce02 Issue #18960: Fix bugs with Python source code encoding in the second line. 
* The first line of Python script could be executed twice when the source
encoding (not equal to 'utf-8') was specified on the second line.

* Now the source encoding declaration on the second line isn't effective if
the first line contains anything except a comment.

* As a consequence, 'python -x' works now again with files with the source
encoding declarations specified on the second file, and can be used again
to make Python batch files on Windows.

* The tokenize module now ignore the source encoding declaration on the second
line if the first line contains anything except a comment.

* IDLE now ignores the source encoding declaration on the second line if the
first line contains anything except a comment.

* 2to3 and the findnocoding.py script now ignore the source encoding
declaration on the second line if the first line contains anything except
a comment.
 2014-01-09 18:36:09 +02:00
Ezio Melotti 5833c00427 #19620: merge with 3.3. 2013-11-25 05:16:09 +02:00
Ezio Melotti 4bcc796acc #19620: Fix typo in docstring (noticed by Christopher Welborn). 2013-11-25 05:14:51 +02:00
Serhiy Storchaka 935349406a Issue #18873: The tokenize module, IDLE, 2to3, and the findnocoding.py script 
now detect Python source code encoding only in comment lines.
 2013-09-16 23:57:00 +03:00
Serhiy Storchaka dafea85190 Issue #18873: The tokenize module, IDLE, 2to3, and the findnocoding.py script 
now detect Python source code encoding only in comment lines.
 2013-09-16 23:51:56 +03:00
Terry Jan Reedy 16b5c13668 Merge 3.3, issue #17047: remove doubled words found in 2.7 to 
3.4 Lib/*, as reported by Serhiy Storchaka and Matthew Barnett.
 2013-03-11 17:59:07 -04:00
Terry Jan Reedy c30b7b16ea Issue #17047: remove doubled words found in 2.7 to 3.4 Lib/*, 
as reported by Serhiy Storchaka and Matthew Barnett.
 2013-03-11 17:57:08 -04:00
Giampaolo Rodola' 2f50aaf2ff modernize some modules' code by using with statement around open() 2013-02-12 02:04:27 +01:00
Andrew Svetlov f7a17b48d7 Replace IOError with OSError (#16715) 2012-12-25 16:47:37 +02:00
Ned Deily 24ec054ea1 Issue #15822: Fix installation of lib2to3 grammar pickles to ensure 
they are created in the install locations and with the proper timestamp.
(Solution suggested by MvL)
 2012-09-08 19:04:47 -07:00
Ned Deily 47bcfff26b Issue #15822: Fix installation of lib2to3 grammar pickles to ensure 
they are created in the install locations and with the proper timestamp.
(Solution suggested by MvL)
 2012-09-08 18:50:56 -07:00
Barry Warsaw 78f89d8c38 - Issue #11250: Back port fix from 3.3 branch, so that 2to3 can handle files 
with line feeds.  This was ported from the sandbox to the 3.3 branch, but
  didn't make it into 3.2.

- Re-enable lib2to3's test_parser.py tests, though with an expected failure
  (see issue 13125).
 2011-10-07 14:44:49 -04:00
Ezio Melotti 3b3499ba69 #11565: Merge with 3.1. 2011-03-16 11:35:38 +02:00
Ezio Melotti 13925008dc #11565: Fix several typos. Patch by Piotr Kasprzyk. 2011-03-16 11:05:33 +02:00
Ezio Melotti b88ed1549e #11565: Merge with 3.2. 2011-03-16 11:38:59 +02:00
Benjamin Peterson 2e7965e8b0 Merged revisions 88661 via svnmerge from 
svn+ssh://pythondev@svn.python.org/sandbox/trunk/2to3/lib2to3

........
  r88661 | benjamin.peterson | 2011-02-26 16:06:24 -0600 (Sat, 26 Feb 2011) | 6 lines

  fix refactoring on formfeed characters #11250

  This is because text.splitlines() is not the same as
  list(StringIO.StringIO(text)).
........
 2011-02-26 22:12:10 +00:00
Benjamin Peterson c9e833faa0 Merged revisions 80936 via svnmerge from 
svn+ssh://pythondev@svn.python.org/python/branches/py3k

................
  r80936 | benjamin.peterson | 2010-05-07 14:10:11 -0500 (Fri, 07 May 2010) | 76 lines

  Merged revisions 80934 via svnmerge from
  svn+ssh://pythondev@svn.python.org/python/trunk

  ................
    r80934 | benjamin.peterson | 2010-05-07 13:58:23 -0500 (Fri, 07 May 2010) | 69 lines

    Merged revisions 79911,79916-79917,80018,80418,80572-80573,80635-80639,80668,80922 via svnmerge from
    svn+ssh://pythondev@svn.python.org/sandbox/trunk/2to3/lib2to3

    ........
      r79911 | benjamin.peterson | 2010-04-09 15:38:53 -0500 (Fri, 09 Apr 2010) | 1 line

      use absolute import
    ........
      r79916 | benjamin.peterson | 2010-04-09 16:05:21 -0500 (Fri, 09 Apr 2010) | 1 line

      generalize detection of __future__ imports and attach them to the tree
    ........
      r79917 | benjamin.peterson | 2010-04-09 16:11:44 -0500 (Fri, 09 Apr 2010) | 1 line

      don't try to 'fix' relative imports when absolute_import is enabled #8858
    ........
      r80018 | benjamin.peterson | 2010-04-12 16:12:12 -0500 (Mon, 12 Apr 2010) | 4 lines

      prevent diffs from being mangled is multiprocess mode #6409

      Patch by George Boutsioukis.
    ........
      r80418 | benjamin.peterson | 2010-04-23 16:00:03 -0500 (Fri, 23 Apr 2010) | 1 line

      remove unhelpful description
    ........
      r80572 | benjamin.peterson | 2010-04-27 20:33:54 -0500 (Tue, 27 Apr 2010) | 1 line

      use unicode literals
    ........
      r80573 | jeffrey.yasskin | 2010-04-27 23:08:27 -0500 (Tue, 27 Apr 2010) | 6 lines

      Don't transform imports that are already relative.  2to3 turned
        from . import refactor
      into
        from .. import refactor
      which broke the transformation of 2to3 itself.
    ........
      r80635 | benjamin.peterson | 2010-04-29 16:02:23 -0500 (Thu, 29 Apr 2010) | 1 line

      remove imports
    ........
      r80636 | benjamin.peterson | 2010-04-29 16:02:41 -0500 (Thu, 29 Apr 2010) | 1 line

      unicode literal
    ........
      r80637 | benjamin.peterson | 2010-04-29 16:03:42 -0500 (Thu, 29 Apr 2010) | 1 line

      must pass a string to Number
    ........
      r80638 | benjamin.peterson | 2010-04-29 16:05:34 -0500 (Thu, 29 Apr 2010) | 1 line

      unicode literals
    ........
      r80639 | benjamin.peterson | 2010-04-29 16:06:09 -0500 (Thu, 29 Apr 2010) | 1 line

      pass string to Number
    ........
      r80668 | jeffrey.yasskin | 2010-04-30 18:02:47 -0500 (Fri, 30 Apr 2010) | 4 lines

      Make 2to3 run under Python 2.5 so that the benchmark suite at
      http://hg.python.org/benchmarks/ can use it and still run on implementations
      that haven't gotten to 2.6 yet.  Fixes issue 8566.
    ........
      r80922 | benjamin.peterson | 2010-05-07 11:06:25 -0500 (Fri, 07 May 2010) | 1 line

      prevent xrange transformation from wrapping range calls it produces in list
    ........
  ................
................
 2010-05-08 15:46:00 +00:00
Benjamin Peterson 8d26b0be6d Merged revisions 80934 via svnmerge from 
svn+ssh://pythondev@svn.python.org/python/trunk

................
  r80934 | benjamin.peterson | 2010-05-07 13:58:23 -0500 (Fri, 07 May 2010) | 69 lines

  Merged revisions 79911,79916-79917,80018,80418,80572-80573,80635-80639,80668,80922 via svnmerge from
  svn+ssh://pythondev@svn.python.org/sandbox/trunk/2to3/lib2to3

  ........
    r79911 | benjamin.peterson | 2010-04-09 15:38:53 -0500 (Fri, 09 Apr 2010) | 1 line

    use absolute import
  ........
    r79916 | benjamin.peterson | 2010-04-09 16:05:21 -0500 (Fri, 09 Apr 2010) | 1 line

    generalize detection of __future__ imports and attach them to the tree
  ........
    r79917 | benjamin.peterson | 2010-04-09 16:11:44 -0500 (Fri, 09 Apr 2010) | 1 line

    don't try to 'fix' relative imports when absolute_import is enabled #8858
  ........
    r80018 | benjamin.peterson | 2010-04-12 16:12:12 -0500 (Mon, 12 Apr 2010) | 4 lines

    prevent diffs from being mangled is multiprocess mode #6409

    Patch by George Boutsioukis.
  ........
    r80418 | benjamin.peterson | 2010-04-23 16:00:03 -0500 (Fri, 23 Apr 2010) | 1 line

    remove unhelpful description
  ........
    r80572 | benjamin.peterson | 2010-04-27 20:33:54 -0500 (Tue, 27 Apr 2010) | 1 line

    use unicode literals
  ........
    r80573 | jeffrey.yasskin | 2010-04-27 23:08:27 -0500 (Tue, 27 Apr 2010) | 6 lines

    Don't transform imports that are already relative.  2to3 turned
      from . import refactor
    into
      from .. import refactor
    which broke the transformation of 2to3 itself.
  ........
    r80635 | benjamin.peterson | 2010-04-29 16:02:23 -0500 (Thu, 29 Apr 2010) | 1 line

    remove imports
  ........
    r80636 | benjamin.peterson | 2010-04-29 16:02:41 -0500 (Thu, 29 Apr 2010) | 1 line

    unicode literal
  ........
    r80637 | benjamin.peterson | 2010-04-29 16:03:42 -0500 (Thu, 29 Apr 2010) | 1 line

    must pass a string to Number
  ........
    r80638 | benjamin.peterson | 2010-04-29 16:05:34 -0500 (Thu, 29 Apr 2010) | 1 line

    unicode literals
  ........
    r80639 | benjamin.peterson | 2010-04-29 16:06:09 -0500 (Thu, 29 Apr 2010) | 1 line

    pass string to Number
  ........
    r80668 | jeffrey.yasskin | 2010-04-30 18:02:47 -0500 (Fri, 30 Apr 2010) | 4 lines

    Make 2to3 run under Python 2.5 so that the benchmark suite at
    http://hg.python.org/benchmarks/ can use it and still run on implementations
    that haven't gotten to 2.6 yet.  Fixes issue 8566.
  ........
    r80922 | benjamin.peterson | 2010-05-07 11:06:25 -0500 (Fri, 07 May 2010) | 1 line

    prevent xrange transformation from wrapping range calls it produces in list
  ........
................
 2010-05-07 19:10:11 +00:00
Benjamin Peterson 5fbccff118 Merged revisions 79327 via svnmerge from 
svn+ssh://pythondev@svn.python.org/python/branches/py3k

................
  r79327 | benjamin.peterson | 2010-03-22 22:22:05 -0500 (Mon, 22 Mar 2010) | 54 lines

  Merged revisions 79306,79311,79325 via svnmerge from
  svn+ssh://pythondev@svn.python.org/python/trunk

  ................
    r79306 | benjamin.peterson | 2010-03-22 17:40:06 -0500 (Mon, 22 Mar 2010) | 21 lines

    Merged revisions 79077,79137,79304-79305 via svnmerge from
    svn+ssh://pythondev@svn.python.org/sandbox/trunk/2to3/lib2to3

    ........
      r79077 | benjamin.peterson | 2010-03-18 18:05:29 -0500 (Thu, 18 Mar 2010) | 1 line

      port detect_encoding improvements from py3k
    ........
      r79137 | benjamin.peterson | 2010-03-20 11:12:53 -0500 (Sat, 20 Mar 2010) | 1 line

      add a fixer for setting sys.exitfunc #2356
    ........
      r79304 | benjamin.peterson | 2010-03-22 17:20:22 -0500 (Mon, 22 Mar 2010) | 1 line

      fix test_parser when it's run in a path with spaces #7666
    ........
      r79305 | benjamin.peterson | 2010-03-22 17:27:07 -0500 (Mon, 22 Mar 2010) | 1 line

      normalize whitespace
    ........
  ................
    r79311 | benjamin.peterson | 2010-03-22 17:54:42 -0500 (Mon, 22 Mar 2010) | 9 lines

    Merged revisions 79309 via svnmerge from
    svn+ssh://pythondev@svn.python.org/sandbox/trunk/2to3/lib2to3

    ........
      r79309 | benjamin.peterson | 2010-03-22 17:50:47 -0500 (Mon, 22 Mar 2010) | 1 line

      pass correct symbol in
    ........
  ................
    r79325 | benjamin.peterson | 2010-03-22 22:03:55 -0500 (Mon, 22 Mar 2010) | 13 lines

    Merged revisions 79313,79324 via svnmerge from
    svn+ssh://pythondev@svn.python.org/sandbox/trunk/2to3/lib2to3

    ........
      r79313 | benjamin.peterson | 2010-03-22 17:59:57 -0500 (Mon, 22 Mar 2010) | 1 line

      another case where a symbol is needed
    ........
      r79324 | benjamin.peterson | 2010-03-22 21:59:47 -0500 (Mon, 22 Mar 2010) | 1 line

      use unicode literals
    ........
  ................
................
 2010-03-23 03:29:23 +00:00
Benjamin Peterson 0af9398d27 Merged revisions 79306,79311,79325 via svnmerge from 
svn+ssh://pythondev@svn.python.org/python/trunk

................
  r79306 | benjamin.peterson | 2010-03-22 17:40:06 -0500 (Mon, 22 Mar 2010) | 21 lines

  Merged revisions 79077,79137,79304-79305 via svnmerge from
  svn+ssh://pythondev@svn.python.org/sandbox/trunk/2to3/lib2to3

  ........
    r79077 | benjamin.peterson | 2010-03-18 18:05:29 -0500 (Thu, 18 Mar 2010) | 1 line

    port detect_encoding improvements from py3k
  ........
    r79137 | benjamin.peterson | 2010-03-20 11:12:53 -0500 (Sat, 20 Mar 2010) | 1 line

    add a fixer for setting sys.exitfunc #2356
  ........
    r79304 | benjamin.peterson | 2010-03-22 17:20:22 -0500 (Mon, 22 Mar 2010) | 1 line

    fix test_parser when it's run in a path with spaces #7666
  ........
    r79305 | benjamin.peterson | 2010-03-22 17:27:07 -0500 (Mon, 22 Mar 2010) | 1 line

    normalize whitespace
  ........
................
  r79311 | benjamin.peterson | 2010-03-22 17:54:42 -0500 (Mon, 22 Mar 2010) | 9 lines

  Merged revisions 79309 via svnmerge from
  svn+ssh://pythondev@svn.python.org/sandbox/trunk/2to3/lib2to3

  ........
    r79309 | benjamin.peterson | 2010-03-22 17:50:47 -0500 (Mon, 22 Mar 2010) | 1 line

    pass correct symbol in
  ........
................
  r79325 | benjamin.peterson | 2010-03-22 22:03:55 -0500 (Mon, 22 Mar 2010) | 13 lines

  Merged revisions 79313,79324 via svnmerge from
  svn+ssh://pythondev@svn.python.org/sandbox/trunk/2to3/lib2to3

  ........
    r79313 | benjamin.peterson | 2010-03-22 17:59:57 -0500 (Mon, 22 Mar 2010) | 1 line

    another case where a symbol is needed
  ........
    r79324 | benjamin.peterson | 2010-03-22 21:59:47 -0500 (Mon, 22 Mar 2010) | 1 line

    use unicode literals
  ........
................
 2010-03-23 03:22:05 +00:00
Benjamin Peterson 90f5ba538b convert shebang lines: python -> python3 2010-03-11 22:53:45 +00:00