Commit Graph

259 Commits

Author SHA1 Message Date
Anthony Sottile c57e6e2e52 bpo-35312: Make lib2to3.pgen2.parse.ParseError round-trip pickle-able. (GH-10710) 2018-11-27 20:39:49 +02:00
Srinivas Thatiparthy (శ్రీనివాస్ తాటిపర్తి) b9498e2367 bpo-35202: Remove unused imports in Lib directory. (GH-10446) 2018-11-10 09:22:02 +02:00
Zackery Spytz d4d60134b2 bpo-16965: 2to3 now rewrites execfile() to open with 'rb'. (GH-8569) 2018-10-13 12:27:30 +03:00
Monson Shao 10a428b64b closes bpo-34515: Support non-ASCII identifiers in lib2to3. (GH-8950) 2018-09-15 10:32:29 -07:00
Serhiy Storchaka 4b8a7f51da
Revert "closes bpo-27494: Fix 2to3 handling of trailing comma after a generator expression (GH-3771)" (#8241)
This reverts commit af810b35b4.

This is not valid syntax (see bpo-32012).
2018-07-31 09:34:30 +03:00
Berker Peksag 7a3056fa7d
bpo-21446: Update reload fixer to use importlib (GH-8391) 2018-07-23 09:49:08 +03:00
Jason R. Coombs cafaf0447b
bpo-34108: Fix double carriage return in 2to3 on Windows (#8271)
* Add test capturing failure.
* Honor newlines as present in the original file.
2018-07-13 11:26:03 -04:00
Denis Osipov e3a523a0fa bpo-31583: Fix 2to3 for using with --add-suffix option (GH-3758) 2018-04-18 16:50:25 -07:00
Aaron Ang c127a86e18 bpo-11594: Ensure line-endings are respected when using 2to3 (GH-6483) 2018-04-17 14:34:14 -07:00
Łukasz Langa 76618061b9
[lib2to3] Make grammar pickling faster (#6491)
* Now uses pickle protocol 4

* Doesn't wrap the grammar's `__dict__` in ordered dictionaries anymore as
dictionaries in Python 3.6+ are ordered by default

This still produces deterministic pickles (that hash the same with MD5).
Tested with different PYTHONHASHSEED values.
2018-04-16 17:33:59 -07:00
Zsolt Dollenstein 8f37e84664 Add support for all string literals to lib2to3 (#6457) 2018-04-16 14:33:31 -07:00
Jelle Zijlstra f64aae46da Revert "bpo-30406: Make async and await proper keywords (#1669)" (GH-6143)
This reverts commit ac317700ce.

(Reverts only the lib2to3 part.)
2018-03-18 09:54:33 -07:00
Łukasz Langa 74f56878cd
lib2to3: Add more tests (#6101) 2018-03-13 10:53:22 -07:00
Łukasz Langa b51f5de711
bpo-33064: lib2to3: support trailing comma after *args and **kwargs (#6096)
New tests also added.

I also made the comments in line with the builtin Grammar/Grammar. PEP 306 was
withdrawn, Kees Blom's railroad program has been lost to the sands of time for
at least 16 years now (I found a python-dev post from people looking for it).
2018-03-13 00:44:49 -07:00
Benjamin Peterson 1e17d4aaff
compare with difflib not diff(1) (GH-5450) 2018-01-30 09:03:12 -08:00
Eric Appelt 14e976e00e closes bpo-30117: fix lib2to3 ParserIdempotency test (GH-1242)
Fix two (in my opinion) spurious failure conditions in the lib2to3.tests.test_parser.TestParserIdempotency test_parser test.

    Use the same encoding found in the initial file to write a temp file for a diff. This retains the BOM if the encoding was initially utf-8-sig.

    If the file cannot be parsed using the normal grammar, try again with no print statement which should succeed for valid files using future print_function

For case (1), the driver was correctly handling a BOM in a utf-8 file, but then the test was not writing a comparison file using 'utf-8-sig' to diff against, so the BOM got removed. I don't think that is the fault of the parser, and lib2to3 will retain the BOM.

For case (2), lib2to3 pre-detects the use of from __future__ import print_function or allows the user to force this interpretation with a -p flag, and then selects a different grammar with the print statement removed. That makes the test cases unfair to this test as the driver itself doesn't know which grammar to use. As a minimal fix, the test will try using a grammar with the print statement, and if that fails fall back on a grammar without it. A more thorough handling of the idempotency test would to be to parse all files using both grammars and ignore if one of the two failed but otherwise check both. I didn't think this was necessary but can change.
2018-01-29 23:13:03 -08:00
Benjamin Peterson e325608740
remove unused import (#5040) 2017-12-28 23:24:40 -08:00
Benjamin Peterson e5f7dccefa
make PatternCompiler use the packaged grammar if possible (more bpo-24960) (#5034) 2017-12-28 17:54:12 -08:00
Benjamin Peterson d11e8e0d11
correct wording (#4983) 2017-12-22 20:48:13 -08:00
Benjamin Peterson 8a5877165e
bpo-24960: use pkgutil.get_data in lib2to3 to read pickled grammar files (#4977)
This is more complicated than it should be because we need to preserve the
useful mtime-based regeneration feature that lib2to3.pgen2.driver.load_grammar
has. We only look for the pickled grammar file with pkgutil.get_data and only if
the source file does not exist.
2017-12-22 12:18:33 -08:00
Dong-hee Na a489599793 bpo-32046: Update 2to3 when converts operator.isCallable(obj). (#4417) 2017-11-28 11:26:56 -05:00
Serhiy Storchaka 0a2abdfca2
bpo-30143: 2to3 now generates a code that uses abstract collection classes (#1262)
from collections.abc rather than collections.
2017-11-16 09:16:24 +02:00
Jelle Zijlstra ac317700ce bpo-30406: Make async and await proper keywords (#1669)
Per PEP 492, 'async' and 'await' should become proper keywords in 3.7.
2017-10-05 23:24:46 -04:00
Jakub Stasiak af810b35b4 closes bpo-27494: Fix 2to3 handling of trailing comma after a generator expression (#3771) 2017-10-05 00:10:09 -07:00
Victor Stinner 272d888c7b bpo-29783: Replace codecs.open() with io.open() (#599) 2017-06-16 08:59:01 +02:00
Łukasz Langa 1b9530c536 bpo-23894: make lib2to3 recognize f-strings (#1733)
Note: this doesn't unpack f-strings into the underlying JoinedStr AST.

Ideally we'd fully implement JoinedStr here but given its additional
complexity, I think this is worth bandaiding as is. This unblocks tools like
https://github.com/google/yapf to format 3.6 syntax using f-strings.
2017-05-22 16:35:48 -07:00
Łukasz Langa 0c4aca54dc Make rb'' strings work in lib2to3 (#1724)
This partially solves bpo-23894.
2017-05-22 15:19:09 -07:00
Serhiy Storchaka 2e576f5aec bpo-30144: Import collections ABC from collections.abc rather than collections. (#1263) 2017-04-24 09:05:00 +03:00
Nevada Sanchez a6e395dffa bpo-29869: Allow underscores in numeric literals in lib2to3. (GH-1119)
* Allow underscores in numeric literals in lib2to3.
* Stricter literal parsing for Python 3.6 in lib2to3.pgen2.tokenize.
* Add test case for underscores in literals in Python 3.
2017-04-13 10:32:54 -07:00
Stuart Berg 93b4b47e3a bpo-28837: Fix lib2to3 handling of map/zip/filter calls when followed with a 'trailer', e.g. zip()[x] (#24) 2017-04-05 22:19:40 -07:00
Serhiy Storchaka 5affd23e6f bpo-29762: More use "raise from None". (#569)
This hides unwanted implementation details from tracebacks.
2017-04-05 09:37:24 +03:00
Michael Selik 11fa3c7cd1 bpo-29957: change LBYL key lookup to dict.setdefault (#938)
* change LBYL key lookup to dict.setdefault

The ``results`` was constructed as a defaultdict and we could simply
delete the check ``if key not in results``. However, I think it's safer
to use dict.setdefault as I'm not sure whether the caller expects a
regular dict or defaultdict.

* add name to the acknowledgements file

* use defaultdict to make the key-lookup cleaner
2017-04-01 23:02:31 -07:00
Victor Stinner d6debb24e0 bpo-29919: Remove unused imports found by pyflakes (#137)
Make also minor PEP8 coding style fixes on modified imports.
2017-03-27 16:05:26 +02:00
Serhiy Storchaka bdf6b910f9 bpo-29776: Use decorator syntax for properties. (#585) 2017-03-19 08:40:32 +02:00
Serhiy Storchaka 70d28a184c Remove unused imports. 2016-12-16 20:00:15 +02:00
Benjamin Peterson 3d3e9ffa8e merge 3.5 2016-12-05 22:31:12 -08:00
Benjamin Peterson 41fef63076 rm unused import 2016-12-05 22:30:55 -08:00
Martin Panter fd1df3fef2 Issue #28394: Merge typo fixes from 3.5 into 3.6 2016-10-10 00:28:38 +00:00
Martin Panter d2a584b47b Issue #28394: Spelling and typo fixes in code comments and changelog
Includes patch by Ville Skyttä.
2016-10-10 00:24:34 +00:00
Gregory P. Smith ext:(%20%5BGoogle%20Inc.%5D) fc3f7d5677 Remove the debug print that i accidentally left in. 2016-09-10 20:39:50 -07:00
Gregory P. Smith ext:(%20%5BGoogle%20Inc.%5D) 3a95f45a38 Remove the debug print that i accidentally left in. 2016-09-10 20:39:36 -07:00
Gregory P. Smith ext:(%20%5BGoogle%20Inc.%5D) 28325749c0 Issue #25969: Update the lib2to3 grammar to handle the unpacking
generalizations added in 3.5.
2016-09-09 18:18:52 -07:00
Gregory P. Smith dbdf029a55 remove fix_callable - callable() was readded many releases ago. 2016-09-09 18:14:33 -07:00
Gregory P. Smith ext:(%20%5BGoogle%20Inc.%5D) 3b822d6c89 Issue #25969: Update the lib2to3 grammar to handle the unpacking
generalizations added in 3.5.
2016-09-09 18:19:51 -07:00
Gregory P. Smith 223f12c9eb Remove 2to3's fix_callable... We reintroduced the callable built-in
pretty early on in the 3.x series (3.1 or 3.2?).
2016-09-09 17:03:58 -07:00
Gregory P. Smith fa40ca811f issue27985 - fix the incorrect duplicate class name in the lib2to3
test.  call it TestVarAnnotations instead.
2016-09-09 14:48:08 -07:00
Yury Selivanov 52c4e7cc84 Issue #28008: Implement PEP 530 -- asynchronous comprehensions. 2016-09-09 10:36:01 -07:00
Yury Selivanov f8cb8a16a3 Issue #27985: Implement PEP 526 -- Syntax for Variable Annotations.
Patch by Ivan Levkivskyi.
2016-09-08 20:50:03 -07:00
Gregory P. Smith ext:(%20%5BGoogle%20Inc.%5D) 9719161858 Fix indentation (reindent.py). 2016-09-08 00:48:07 +00:00
Gregory P. Smith ext:(%20%5BGoogle%20Inc.%5D) dd1c638b92 lib2to3.pgen3.driver.load_grammar() now creates a stable cache file
between runs given the same Grammar.txt input regardless of the hash
randomization setting.
2016-09-08 00:40:07 +00:00