This adds a `feature_version` flag to `ast.parse()` (documented) and `compile()` (hidden) that allow tweaking the parser to support older versions of the grammar. In particular if `feature_version` is 5 or 6, the hacks for the `async` and `await` keyword from PEP 492 are reinstated. (For 7 or higher, these are unconditionally treated as keywords, but they are still special tokens rather than `NAME` tokens that the parser driver recognizes.)
https://bugs.python.org/issue35975
The majority of this PR is tediously passing `end_lineno` and `end_col_offset` everywhere. Here are non-trivial points:
* It is not possible to reconstruct end positions in AST "on the fly", some information is lost after an AST node is constructed, so we need two more attributes for every AST node `end_lineno` and `end_col_offset`.
* I add end position information to both CST and AST. Although it may be technically possible to avoid adding end positions to CST, the code becomes more cumbersome and less efficient.
* Since the end position is not known for non-leaf CST nodes while the next token is added, this requires a bit of extra care (see `_PyNode_FinalizeEndPos`). Unless I made some mistake, the algorithm should be linear.
* For statements, I "trim" the end position of suites to not include the terminal newlines and dedent (this seems to be what people would expect), for example in
```python
class C:
pass
pass
```
the end line and end column for the class definition is (2, 8).
* For `end_col_offset` I use the common Python convention for indexing, for example for `pass` the `end_col_offset` is 4 (not 3), so that `[0:4]` gives one the source code that corresponds to the node.
* I added a helper function `ast.get_source_segment()`, to get source text segment corresponding to a given AST node. It is also useful for testing.
An (inevitable) downside of this PR is that AST now takes almost 25% more memory. I think however it is probably justified by the benefits.
Remove the docstring attribute of AST types and restore docstring
expression as a first stmt in their body.
Co-authored-by: INADA Naoki <methane@users.noreply.github.com>
* bpo-29463: Add docstring field to some AST nodes.
ClassDef, ModuleDef, FunctionDef, and AsyncFunctionDef has docstring
field for now. It was first statement of there body.
* fix document. thanks travis!
* doc fixes
svn+ssh://pythondev@svn.python.org/python/trunk
........
r67162 | benjamin.peterson | 2008-11-08 10:55:33 -0600 (Sat, 08 Nov 2008) | 1 line
a few compile() and ast doc improvements
........
svn+ssh://pythondev@svn.python.org/python/trunk
........
r64434 | andrew.kuchling | 2008-06-20 18:13:58 -0500 (Fri, 20 Jun 2008) | 1 line
Remove request for e-mail; it's unlikely these classes will be saved
........
r64435 | andrew.kuchling | 2008-06-20 18:14:32 -0500 (Fri, 20 Jun 2008) | 1 line
Grammar fixes
........
r64440 | andrew.kuchling | 2008-06-21 08:29:12 -0500 (Sat, 21 Jun 2008) | 1 line
Docstring typo
........
r64441 | andrew.kuchling | 2008-06-21 08:47:20 -0500 (Sat, 21 Jun 2008) | 1 line
Use repr() for bad input strings; this makes the empty string or binary characters more visible
........
r64442 | andrew.kuchling | 2008-06-21 08:48:38 -0500 (Sat, 21 Jun 2008) | 1 line
Docstring correction
........
r64443 | georg.brandl | 2008-06-21 09:26:19 -0500 (Sat, 21 Jun 2008) | 2 lines
Documentation fix.
........
r64445 | facundo.batista | 2008-06-21 12:30:06 -0500 (Sat, 21 Jun 2008) | 3 lines
Reviewed and updated the documentation. Fixes#3017.
........
r64447 | facundo.batista | 2008-06-21 13:58:04 -0500 (Sat, 21 Jun 2008) | 6 lines
Now a from submitted via POST that also has a query string
will contain both FieldStorage and MiniFieldStorage items.
Fixes#1817.
........
r64448 | facundo.batista | 2008-06-21 14:48:19 -0500 (Sat, 21 Jun 2008) | 5 lines
In the deprecated functions I added an alert to review
specially a section of the subprocess documentation
that helps with the replacing of those functionss.
........
r64450 | georg.brandl | 2008-06-22 04:05:29 -0500 (Sun, 22 Jun 2008) | 2 lines
Turn section references into proper cross-references.
........
r64452 | facundo.batista | 2008-06-22 08:36:20 -0500 (Sun, 22 Jun 2008) | 5 lines
Issue #2722. Now the char buffer to support the path string has
not fixed length, it mallocs memory if needed. As a result, we
don't have a maximum for the getcwd() method.
........
r64455 | facundo.batista | 2008-06-22 10:27:10 -0500 (Sun, 22 Jun 2008) | 4 lines
Issue 3164. Small fix to don't repeat a comparation
without necessity.
........
r64461 | georg.brandl | 2008-06-22 13:11:52 -0500 (Sun, 22 Jun 2008) | 2 lines
#3085: Fix syntax error.
........
r64464 | georg.brandl | 2008-06-22 13:31:54 -0500 (Sun, 22 Jun 2008) | 2 lines
Expand docstrings of sqlite3 functions.
........
r64466 | georg.brandl | 2008-06-22 14:07:59 -0500 (Sun, 22 Jun 2008) | 2 lines
Write out "phi" consistently.
........
r64468 | facundo.batista | 2008-06-22 14:35:24 -0500 (Sun, 22 Jun 2008) | 4 lines
Just returning nothing instead of rising TestSkipped, because
it makes the test fail in the trunk.loewis-sun buildbot.
........
svn+ssh://pythondev@svn.python.org/python/trunk
........
r64089 | armin.ronacher | 2008-06-10 22:37:02 +0200 (mar., 10 juin 2008) | 3 lines
Fix a formatting error in the ast documentation.
........
r64098 | raymond.hettinger | 2008-06-11 02:25:29 +0200 (mer., 11 juin 2008) | 6 lines
Mini-PEP: Simplifying numbers.py
* Convert binary methods in Integral to mixin methods
* Remove three-arg __pow__ as a required method
* Make __int__ the root method instead of __long__.
........
r64100 | raymond.hettinger | 2008-06-11 02:28:51 +0200 (mer., 11 juin 2008) | 1 line
Update numbers doc for the Integral simplification.
........
r64101 | raymond.hettinger | 2008-06-11 02:44:47 +0200 (mer., 11 juin 2008) | 3 lines
Handle the case with zero arguments.
........
r64102 | benjamin.peterson | 2008-06-11 03:31:28 +0200 (mer., 11 juin 2008) | 4 lines
convert test_struct to a unittest thanks to Giampaolo Rodola
I had to disable one test because it was functioning incorrectly, see #1530559
I also removed the debugging prints
........
r64113 | thomas.heller | 2008-06-11 09:10:43 +0200 (mer., 11 juin 2008) | 2 lines
Fix markup.
Document the new 'offset' parameter for the 'ctypes.byref' function.
........
r64115 | raymond.hettinger | 2008-06-11 12:30:54 +0200 (mer., 11 juin 2008) | 1 line
Multi-arg form for set.difference() and set.difference_update().
........
r64116 | raymond.hettinger | 2008-06-11 14:06:49 +0200 (mer., 11 juin 2008) | 1 line
Issue 3051: Let heapq work with either __lt__ or __le__.
........
r64118 | raymond.hettinger | 2008-06-11 14:39:09 +0200 (mer., 11 juin 2008) | 1 line
Optimize previous checkin for heapq.
........
r64120 | raymond.hettinger | 2008-06-11 15:14:50 +0200 (mer., 11 juin 2008) | 1 line
Add test for heapq using both __lt__ and __le__.
........
r64132 | gregory.p.smith | 2008-06-11 20:00:52 +0200 (mer., 11 juin 2008) | 3 lines
Correct an incorrect comment about our #include of stddef.h.
(see Doug Evans' comment on python-dev 2008-06-10)
........
r64342 | guido.van.rossum | 2008-06-17 19:38:02 +0200 (mar., 17 juin 2008) | 3 lines
Roll back Raymond's -r64098 while we think of something better.
(See issue 3056 -- we're close to a resolution but need unittests.)
........
Guido van Rossum
Ivan Levkivskyi
Serhiy Storchaka
Brett Cannon
Sanyam Khurana
INADA Naoki
Martin Panter
Senthil Kumaran
Georg Brandl
Benjamin Peterson
Terry Reedy
Raymond Hettinger
Éric Araujo
Alexander Belopolsky
Amaury Forgeot d'Arc