Commit Graph

159 Commits

Author SHA1 Message Date
Tim Hatch 3c3aa4516c
lib2to3: Support named assignment expressions (GH-12702)
There are two copies of the grammar -- the one used by Python itself as
Grammar/Grammar, and the one used by lib2to3 which has necessarily diverged at
Lib/lib2to3/Grammar.txt because it needs to support older syntax an we want it
to be reasonable stable to avoid requiring fixer rewrites.

This brings suport for syntax like `if x:= foo():` to match what the live
Python grammar does.

This should've been added at the time of the walrus operator itself, but lib2to3 being
independent is often overlooked.  So we do consider this a bugfix rather than enhancement.
2020-04-02 15:34:54 -07:00
José Roberto Meza Cabrera 276a84a0a6
bpo-38080: Added "getproxies" to urllib fixes in the 2to3 tool (GH-16167) 2020-03-11 23:51:20 +00:00
Vlad Emelianov 768d739c1c
bpo-38641: Add lib2to3 support for starred expressions in return/yield statements (GH-16994) 2020-03-01 19:59:26 +00:00
Batuhan Taşkaya 61b14151cc bpo-39313: Add an option to RefactoringTool for using exec as a function (GH-17967)
https://bugs.python.org/issue39313


Automerge-Triggered-By: @pablogsal
2020-01-12 14:13:31 -08:00
Dong-hee Na b821173b54 bpo-38871: Fix lib2to3 for filter-based statements that contain lambda (GH-17780)
Correctly parenthesize filter-based statements that contain lambda
expressions in lib2to3.
2020-01-07 18:30:54 +01:00
Zsolt Dollenstein 96b06aefe2 bpo-33348: parse expressions after * and ** in lib2to3 (GH-6586)
These are valid even in python 2.7


https://bugs.python.org/issue33348



Automerge-Triggered-By: @gpshead
2019-10-23 23:19:07 -07:00
Min ho Kim c4cacc8c5e Fix typos in comments, docs and test names (#15018)
* Fix typos in comments, docs and test names

* Update test_pyparse.py

account for change in string length

* Apply suggestion: splitable -> splittable

Co-Authored-By: Terry Jan Reedy <tjreedy@udel.edu>

* Apply suggestion: splitable -> splittable

Co-Authored-By: Terry Jan Reedy <tjreedy@udel.edu>

* Apply suggestion: Dealloccte -> Deallocate

Co-Authored-By: Terry Jan Reedy <tjreedy@udel.edu>

* Update posixmodule checksum.

* Reverse idlelib changes.
2019-07-30 18:16:13 -04:00
Anthony Sottile c57e6e2e52 bpo-35312: Make lib2to3.pgen2.parse.ParseError round-trip pickle-able. (GH-10710) 2018-11-27 20:39:49 +02:00
Zackery Spytz d4d60134b2 bpo-16965: 2to3 now rewrites execfile() to open with 'rb'. (GH-8569) 2018-10-13 12:27:30 +03:00
Monson Shao 10a428b64b closes bpo-34515: Support non-ASCII identifiers in lib2to3. (GH-8950) 2018-09-15 10:32:29 -07:00
Serhiy Storchaka 4b8a7f51da
Revert "closes bpo-27494: Fix 2to3 handling of trailing comma after a generator expression (GH-3771)" (#8241)
This reverts commit af810b35b4.

This is not valid syntax (see bpo-32012).
2018-07-31 09:34:30 +03:00
Berker Peksag 7a3056fa7d
bpo-21446: Update reload fixer to use importlib (GH-8391) 2018-07-23 09:49:08 +03:00
Jason R. Coombs cafaf0447b
bpo-34108: Fix double carriage return in 2to3 on Windows (#8271)
* Add test capturing failure.
* Honor newlines as present in the original file.
2018-07-13 11:26:03 -04:00
Aaron Ang c127a86e18 bpo-11594: Ensure line-endings are respected when using 2to3 (GH-6483) 2018-04-17 14:34:14 -07:00
Zsolt Dollenstein 8f37e84664 Add support for all string literals to lib2to3 (#6457) 2018-04-16 14:33:31 -07:00
Jelle Zijlstra f64aae46da Revert "bpo-30406: Make async and await proper keywords (#1669)" (GH-6143)
This reverts commit ac317700ce.

(Reverts only the lib2to3 part.)
2018-03-18 09:54:33 -07:00
Łukasz Langa 74f56878cd
lib2to3: Add more tests (#6101) 2018-03-13 10:53:22 -07:00
Łukasz Langa b51f5de711
bpo-33064: lib2to3: support trailing comma after *args and **kwargs (#6096)
New tests also added.

I also made the comments in line with the builtin Grammar/Grammar. PEP 306 was
withdrawn, Kees Blom's railroad program has been lost to the sands of time for
at least 16 years now (I found a python-dev post from people looking for it).
2018-03-13 00:44:49 -07:00
Benjamin Peterson 1e17d4aaff
compare with difflib not diff(1) (GH-5450) 2018-01-30 09:03:12 -08:00
Eric Appelt 14e976e00e closes bpo-30117: fix lib2to3 ParserIdempotency test (GH-1242)
Fix two (in my opinion) spurious failure conditions in the lib2to3.tests.test_parser.TestParserIdempotency test_parser test.

    Use the same encoding found in the initial file to write a temp file for a diff. This retains the BOM if the encoding was initially utf-8-sig.

    If the file cannot be parsed using the normal grammar, try again with no print statement which should succeed for valid files using future print_function

For case (1), the driver was correctly handling a BOM in a utf-8 file, but then the test was not writing a comparison file using 'utf-8-sig' to diff against, so the BOM got removed. I don't think that is the fault of the parser, and lib2to3 will retain the BOM.

For case (2), lib2to3 pre-detects the use of from __future__ import print_function or allows the user to force this interpretation with a -p flag, and then selects a different grammar with the print statement removed. That makes the test cases unfair to this test as the driver itself doesn't know which grammar to use. As a minimal fix, the test will try using a grammar with the print statement, and if that fails fall back on a grammar without it. A more thorough handling of the idempotency test would to be to parse all files using both grammars and ignore if one of the two failed but otherwise check both. I didn't think this was necessary but can change.
2018-01-29 23:13:03 -08:00
Benjamin Peterson 8a5877165e
bpo-24960: use pkgutil.get_data in lib2to3 to read pickled grammar files (#4977)
This is more complicated than it should be because we need to preserve the
useful mtime-based regeneration feature that lib2to3.pgen2.driver.load_grammar
has. We only look for the pickled grammar file with pkgutil.get_data and only if
the source file does not exist.
2017-12-22 12:18:33 -08:00
Dong-hee Na a489599793 bpo-32046: Update 2to3 when converts operator.isCallable(obj). (#4417) 2017-11-28 11:26:56 -05:00
Serhiy Storchaka 0a2abdfca2
bpo-30143: 2to3 now generates a code that uses abstract collection classes (#1262)
from collections.abc rather than collections.
2017-11-16 09:16:24 +02:00
Jelle Zijlstra ac317700ce bpo-30406: Make async and await proper keywords (#1669)
Per PEP 492, 'async' and 'await' should become proper keywords in 3.7.
2017-10-05 23:24:46 -04:00
Jakub Stasiak af810b35b4 closes bpo-27494: Fix 2to3 handling of trailing comma after a generator expression (#3771) 2017-10-05 00:10:09 -07:00
Łukasz Langa 1b9530c536 bpo-23894: make lib2to3 recognize f-strings (#1733)
Note: this doesn't unpack f-strings into the underlying JoinedStr AST.

Ideally we'd fully implement JoinedStr here but given its additional
complexity, I think this is worth bandaiding as is. This unblocks tools like
https://github.com/google/yapf to format 3.6 syntax using f-strings.
2017-05-22 16:35:48 -07:00
Łukasz Langa 0c4aca54dc Make rb'' strings work in lib2to3 (#1724)
This partially solves bpo-23894.
2017-05-22 15:19:09 -07:00
Nevada Sanchez a6e395dffa bpo-29869: Allow underscores in numeric literals in lib2to3. (GH-1119)
* Allow underscores in numeric literals in lib2to3.
* Stricter literal parsing for Python 3.6 in lib2to3.pgen2.tokenize.
* Add test case for underscores in literals in Python 3.
2017-04-13 10:32:54 -07:00
Stuart Berg 93b4b47e3a bpo-28837: Fix lib2to3 handling of map/zip/filter calls when followed with a 'trailer', e.g. zip()[x] (#24) 2017-04-05 22:19:40 -07:00
Serhiy Storchaka 70d28a184c Remove unused imports. 2016-12-16 20:00:15 +02:00
Gregory P. Smith ext:(%20%5BGoogle%20Inc.%5D) 3b822d6c89 Issue #25969: Update the lib2to3 grammar to handle the unpacking
generalizations added in 3.5.
2016-09-09 18:19:51 -07:00
Gregory P. Smith ext:(%20%5BGoogle%20Inc.%5D) 28325749c0 Issue #25969: Update the lib2to3 grammar to handle the unpacking
generalizations added in 3.5.
2016-09-09 18:18:52 -07:00
Gregory P. Smith dbdf029a55 remove fix_callable - callable() was readded many releases ago. 2016-09-09 18:14:33 -07:00
Gregory P. Smith 223f12c9eb Remove 2to3's fix_callable... We reintroduced the callable built-in
pretty early on in the 3.x series (3.1 or 3.2?).
2016-09-09 17:03:58 -07:00
Gregory P. Smith fa40ca811f issue27985 - fix the incorrect duplicate class name in the lib2to3
test.  call it TestVarAnnotations instead.
2016-09-09 14:48:08 -07:00
Yury Selivanov 52c4e7cc84 Issue #28008: Implement PEP 530 -- asynchronous comprehensions. 2016-09-09 10:36:01 -07:00
Yury Selivanov f8cb8a16a3 Issue #27985: Implement PEP 526 -- Syntax for Variable Annotations.
Patch by Ivan Levkivskyi.
2016-09-08 20:50:03 -07:00
Gregory P. Smith ext:(%20%5BGoogle%20Inc.%5D) dd1c638b92 lib2to3.pgen3.driver.load_grammar() now creates a stable cache file
between runs given the same Grammar.txt input regardless of the hash
randomization setting.
2016-09-08 00:40:07 +00:00
Gregory P. Smith ext:(%20%5BGoogle%20Inc.%5D) a68c1bca7b Remove legacy "from __future__ import with_statement" lines. 2016-09-08 13:47:41 -07:00
Gregory P. Smith ext:(%20%5BGoogle%20Inc.%5D) 0c578d62fc lib2to3.pgen3.driver.load_grammar() now creates a stable cache file
between runs given the same Grammar.txt input regardless of the hash
randomization setting.
2016-09-08 00:46:26 +00:00
Serhiy Storchaka 64204de04c Issue #27095: Simplified MAKE_FUNCTION and removed MAKE_CLOSURE opcodes.
Patch by Demur Rumed.
2016-06-12 17:36:24 +03:00
Martin Panter 50d4c15ba7 Issue #27030: Merge RE fix from 3.5 2016-06-12 06:08:29 +00:00
Martin Panter d4704803b9 Fix buggy RE “\parrot_example.py”, uncovered by Issue #27030 2016-06-12 06:07:35 +00:00
Terry Jan Reedy cc20d63e64 Merge with 3.5 2016-05-06 15:58:13 -04:00
Terry Jan Reedy 823c626b43 Issue 26911: fix import (other problems remain). 2016-05-06 15:57:57 -04:00
Serhiy Storchaka a6f26c1d34 Remove more unused imports in tests. 2016-04-25 00:05:30 +03:00
Serhiy Storchaka e437a10d15 Issue #23277: Remove unused imports in tests. 2016-04-24 21:41:02 +03:00
Serhiy Storchaka c7f44aa99a Issue #23277: Remove more unused sys and os imports. 2016-04-24 13:25:01 +03:00
Yury Selivanov 38d3876bfb Merge 3.5 (Issue #24619) 2015-07-23 15:02:25 +03:00
Yury Selivanov 96ec934e75 Issue #24619: Simplify async/await tokenization.
This commit simplifies async/await tokenization in tokenizer.c,
tokenize.py & lib2to3/tokenize.py.  Previous solution was to keep
a stack of async-def & def blocks, whereas the new approach is just
to remember position of the outermost async-def block.

This change won't bring any parsing performance improvements, but
it makes the code much easier to read and validate.
2015-07-23 15:01:58 +03:00