Commit Graph

152 Commits

Author SHA1 Message Date
Anthony Sottile c57e6e2e52 bpo-35312: Make lib2to3.pgen2.parse.ParseError round-trip pickle-able. (GH-10710) 2018-11-27 20:39:49 +02:00
Zackery Spytz d4d60134b2 bpo-16965: 2to3 now rewrites execfile() to open with 'rb'. (GH-8569) 2018-10-13 12:27:30 +03:00
Monson Shao 10a428b64b closes bpo-34515: Support non-ASCII identifiers in lib2to3. (GH-8950) 2018-09-15 10:32:29 -07:00
Serhiy Storchaka 4b8a7f51da
Revert "closes bpo-27494: Fix 2to3 handling of trailing comma after a generator expression (GH-3771)" (#8241)
This reverts commit af810b35b4.

This is not valid syntax (see bpo-32012).
2018-07-31 09:34:30 +03:00
Berker Peksag 7a3056fa7d
bpo-21446: Update reload fixer to use importlib (GH-8391) 2018-07-23 09:49:08 +03:00
Jason R. Coombs cafaf0447b
bpo-34108: Fix double carriage return in 2to3 on Windows (#8271)
* Add test capturing failure.
* Honor newlines as present in the original file.
2018-07-13 11:26:03 -04:00
Aaron Ang c127a86e18 bpo-11594: Ensure line-endings are respected when using 2to3 (GH-6483) 2018-04-17 14:34:14 -07:00
Zsolt Dollenstein 8f37e84664 Add support for all string literals to lib2to3 (#6457) 2018-04-16 14:33:31 -07:00
Jelle Zijlstra f64aae46da Revert "bpo-30406: Make async and await proper keywords (#1669)" (GH-6143)
This reverts commit ac317700ce.

(Reverts only the lib2to3 part.)
2018-03-18 09:54:33 -07:00
Łukasz Langa 74f56878cd
lib2to3: Add more tests (#6101) 2018-03-13 10:53:22 -07:00
Łukasz Langa b51f5de711
bpo-33064: lib2to3: support trailing comma after *args and **kwargs (#6096)
New tests also added.

I also made the comments in line with the builtin Grammar/Grammar. PEP 306 was
withdrawn, Kees Blom's railroad program has been lost to the sands of time for
at least 16 years now (I found a python-dev post from people looking for it).
2018-03-13 00:44:49 -07:00
Benjamin Peterson 1e17d4aaff
compare with difflib not diff(1) (GH-5450) 2018-01-30 09:03:12 -08:00
Eric Appelt 14e976e00e closes bpo-30117: fix lib2to3 ParserIdempotency test (GH-1242)
Fix two (in my opinion) spurious failure conditions in the lib2to3.tests.test_parser.TestParserIdempotency test_parser test.

    Use the same encoding found in the initial file to write a temp file for a diff. This retains the BOM if the encoding was initially utf-8-sig.

    If the file cannot be parsed using the normal grammar, try again with no print statement which should succeed for valid files using future print_function

For case (1), the driver was correctly handling a BOM in a utf-8 file, but then the test was not writing a comparison file using 'utf-8-sig' to diff against, so the BOM got removed. I don't think that is the fault of the parser, and lib2to3 will retain the BOM.

For case (2), lib2to3 pre-detects the use of from __future__ import print_function or allows the user to force this interpretation with a -p flag, and then selects a different grammar with the print statement removed. That makes the test cases unfair to this test as the driver itself doesn't know which grammar to use. As a minimal fix, the test will try using a grammar with the print statement, and if that fails fall back on a grammar without it. A more thorough handling of the idempotency test would to be to parse all files using both grammars and ignore if one of the two failed but otherwise check both. I didn't think this was necessary but can change.
2018-01-29 23:13:03 -08:00
Benjamin Peterson 8a5877165e
bpo-24960: use pkgutil.get_data in lib2to3 to read pickled grammar files (#4977)
This is more complicated than it should be because we need to preserve the
useful mtime-based regeneration feature that lib2to3.pgen2.driver.load_grammar
has. We only look for the pickled grammar file with pkgutil.get_data and only if
the source file does not exist.
2017-12-22 12:18:33 -08:00
Dong-hee Na a489599793 bpo-32046: Update 2to3 when converts operator.isCallable(obj). (#4417) 2017-11-28 11:26:56 -05:00
Serhiy Storchaka 0a2abdfca2
bpo-30143: 2to3 now generates a code that uses abstract collection classes (#1262)
from collections.abc rather than collections.
2017-11-16 09:16:24 +02:00
Jelle Zijlstra ac317700ce bpo-30406: Make async and await proper keywords (#1669)
Per PEP 492, 'async' and 'await' should become proper keywords in 3.7.
2017-10-05 23:24:46 -04:00
Jakub Stasiak af810b35b4 closes bpo-27494: Fix 2to3 handling of trailing comma after a generator expression (#3771) 2017-10-05 00:10:09 -07:00
Łukasz Langa 1b9530c536 bpo-23894: make lib2to3 recognize f-strings (#1733)
Note: this doesn't unpack f-strings into the underlying JoinedStr AST.

Ideally we'd fully implement JoinedStr here but given its additional
complexity, I think this is worth bandaiding as is. This unblocks tools like
https://github.com/google/yapf to format 3.6 syntax using f-strings.
2017-05-22 16:35:48 -07:00
Łukasz Langa 0c4aca54dc Make rb'' strings work in lib2to3 (#1724)
This partially solves bpo-23894.
2017-05-22 15:19:09 -07:00
Nevada Sanchez a6e395dffa bpo-29869: Allow underscores in numeric literals in lib2to3. (GH-1119)
* Allow underscores in numeric literals in lib2to3.
* Stricter literal parsing for Python 3.6 in lib2to3.pgen2.tokenize.
* Add test case for underscores in literals in Python 3.
2017-04-13 10:32:54 -07:00
Stuart Berg 93b4b47e3a bpo-28837: Fix lib2to3 handling of map/zip/filter calls when followed with a 'trailer', e.g. zip()[x] (#24) 2017-04-05 22:19:40 -07:00
Serhiy Storchaka 70d28a184c Remove unused imports. 2016-12-16 20:00:15 +02:00
Gregory P. Smith ext:(%20%5BGoogle%20Inc.%5D) 3b822d6c89 Issue #25969: Update the lib2to3 grammar to handle the unpacking
generalizations added in 3.5.
2016-09-09 18:19:51 -07:00
Gregory P. Smith ext:(%20%5BGoogle%20Inc.%5D) 28325749c0 Issue #25969: Update the lib2to3 grammar to handle the unpacking
generalizations added in 3.5.
2016-09-09 18:18:52 -07:00
Gregory P. Smith dbdf029a55 remove fix_callable - callable() was readded many releases ago. 2016-09-09 18:14:33 -07:00
Gregory P. Smith 223f12c9eb Remove 2to3's fix_callable... We reintroduced the callable built-in
pretty early on in the 3.x series (3.1 or 3.2?).
2016-09-09 17:03:58 -07:00
Gregory P. Smith fa40ca811f issue27985 - fix the incorrect duplicate class name in the lib2to3
test.  call it TestVarAnnotations instead.
2016-09-09 14:48:08 -07:00
Yury Selivanov 52c4e7cc84 Issue #28008: Implement PEP 530 -- asynchronous comprehensions. 2016-09-09 10:36:01 -07:00
Yury Selivanov f8cb8a16a3 Issue #27985: Implement PEP 526 -- Syntax for Variable Annotations.
Patch by Ivan Levkivskyi.
2016-09-08 20:50:03 -07:00
Gregory P. Smith ext:(%20%5BGoogle%20Inc.%5D) dd1c638b92 lib2to3.pgen3.driver.load_grammar() now creates a stable cache file
between runs given the same Grammar.txt input regardless of the hash
randomization setting.
2016-09-08 00:40:07 +00:00
Gregory P. Smith ext:(%20%5BGoogle%20Inc.%5D) a68c1bca7b Remove legacy "from __future__ import with_statement" lines. 2016-09-08 13:47:41 -07:00
Gregory P. Smith ext:(%20%5BGoogle%20Inc.%5D) 0c578d62fc lib2to3.pgen3.driver.load_grammar() now creates a stable cache file
between runs given the same Grammar.txt input regardless of the hash
randomization setting.
2016-09-08 00:46:26 +00:00
Serhiy Storchaka 64204de04c Issue #27095: Simplified MAKE_FUNCTION and removed MAKE_CLOSURE opcodes.
Patch by Demur Rumed.
2016-06-12 17:36:24 +03:00
Martin Panter 50d4c15ba7 Issue #27030: Merge RE fix from 3.5 2016-06-12 06:08:29 +00:00
Martin Panter d4704803b9 Fix buggy RE “\parrot_example.py”, uncovered by Issue #27030 2016-06-12 06:07:35 +00:00
Terry Jan Reedy cc20d63e64 Merge with 3.5 2016-05-06 15:58:13 -04:00
Terry Jan Reedy 823c626b43 Issue 26911: fix import (other problems remain). 2016-05-06 15:57:57 -04:00
Serhiy Storchaka a6f26c1d34 Remove more unused imports in tests. 2016-04-25 00:05:30 +03:00
Serhiy Storchaka e437a10d15 Issue #23277: Remove unused imports in tests. 2016-04-24 21:41:02 +03:00
Serhiy Storchaka c7f44aa99a Issue #23277: Remove more unused sys and os imports. 2016-04-24 13:25:01 +03:00
Yury Selivanov 38d3876bfb Merge 3.5 (Issue #24619) 2015-07-23 15:02:25 +03:00
Yury Selivanov 96ec934e75 Issue #24619: Simplify async/await tokenization.
This commit simplifies async/await tokenization in tokenizer.c,
tokenize.py & lib2to3/tokenize.py.  Previous solution was to keep
a stack of async-def & def blocks, whereas the new approach is just
to remember position of the outermost async-def block.

This change won't bring any parsing performance improvements, but
it makes the code much easier to read and validate.
2015-07-23 15:01:58 +03:00
Robert Collins d4f4a6dddb Issue #13938: 2to3 converts StringTypes to a tuple. Patch from Mark Hammond. 2015-07-23 06:19:18 +12:00
Yury Selivanov 7544508f02 PEP 0492 -- Coroutines with async and await syntax. Issue #24017. 2015-05-11 22:57:16 -04:00
Zachary Ware 2acbae8016 Issue #22173: Update lib2to3 tests to use unittest test discovery. 2014-10-29 12:24:59 -05:00
Berker Peksag f23530f569 Issue #22186: Fix typos in Lib/.
Patch by Févry Thibault.
2014-10-19 18:04:38 +03:00
Benjamin Peterson 0654be18b3 teach 2to3 about 'yield from' 2014-04-10 00:23:18 -04:00
Benjamin Peterson 4ab92c800a add matrix multiplication operator support to 2to3 2014-04-10 00:12:47 -04:00
Serhiy Storchaka 8f8ec92de8 Issue #19936: Added executable bits or shebang lines to Python scripts which
requires them.  Disable executable bits and shebang lines in test and
benchmark files in order to prevent using a random system python, and in
source files of modules which don't provide command line interface.  Fixed
shebang lines in the unittestgui and checkpip scripts.
2014-01-16 17:33:23 +02:00