This adds a `feature_version` flag to `ast.parse()` (documented) and `compile()` (hidden) that allow tweaking the parser to support older versions of the grammar. In particular if `feature_version` is 5 or 6, the hacks for the `async` and `await` keyword from PEP 492 are reinstated. (For 7 or higher, these are unconditionally treated as keywords, but they are still special tokens rather than `NAME` tokens that the parser driver recognizes.)
https://bugs.python.org/issue35975
The majority of this PR is tediously passing `end_lineno` and `end_col_offset` everywhere. Here are non-trivial points:
* It is not possible to reconstruct end positions in AST "on the fly", some information is lost after an AST node is constructed, so we need two more attributes for every AST node `end_lineno` and `end_col_offset`.
* I add end position information to both CST and AST. Although it may be technically possible to avoid adding end positions to CST, the code becomes more cumbersome and less efficient.
* Since the end position is not known for non-leaf CST nodes while the next token is added, this requires a bit of extra care (see `_PyNode_FinalizeEndPos`). Unless I made some mistake, the algorithm should be linear.
* For statements, I "trim" the end position of suites to not include the terminal newlines and dedent (this seems to be what people would expect), for example in
```python
class C:
pass
pass
```
the end line and end column for the class definition is (2, 8).
* For `end_col_offset` I use the common Python convention for indexing, for example for `pass` the `end_col_offset` is 4 (not 3), so that `[0:4]` gives one the source code that corresponds to the node.
* I added a helper function `ast.get_source_segment()`, to get source text segment corresponding to a given AST node. It is also useful for testing.
An (inevitable) downside of this PR is that AST now takes almost 25% more memory. I think however it is probably justified by the benefits.
Remove the docstring attribute of AST types and restore docstring
expression as a first stmt in their body.
Co-authored-by: INADA Naoki <methane@users.noreply.github.com>
* bpo-29463: Add docstring field to some AST nodes.
ClassDef, ModuleDef, FunctionDef, and AsyncFunctionDef has docstring
field for now. It was first statement of there body.
* fix document. thanks travis!
* doc fixes
Issue #26146: Add a new kind of AST node: ast.Constant. It can be used by
external AST optimizers, but the compiler does not emit directly such node.
An optimizer can replace the following AST nodes with ast.Constant:
* ast.NameConstant: None, False, True
* ast.Num: int, float, complex
* ast.Str: str
* ast.Bytes: bytes
* ast.Tuple if items are constants too: tuple
* frozenset
Update code to accept ast.Constant instead of ast.Num and/or ast.Str:
* compiler
* docstrings
* ast.literal_eval()
* Tools/parser/unparse.py
svn+ssh://pythondev@svn.python.org/python/trunk
........
r68450 | jeffrey.yasskin | 2009-01-09 10:47:07 -0600 (Fri, 09 Jan 2009) | 3 lines
Fix issue 4884, preventing a crash in the socket code when python is compiled
with llvm-gcc and run with a glibc <2.10.
........
r68480 | vinay.sajip | 2009-01-10 07:38:04 -0600 (Sat, 10 Jan 2009) | 1 line
Minor documentation changes cross-referencing NullHandler to the documentation on configuring logging in a library.
........
r68481 | vinay.sajip | 2009-01-10 07:42:04 -0600 (Sat, 10 Jan 2009) | 1 line
Corrected an incorrect self-reference.
........
r68493 | benjamin.peterson | 2009-01-10 11:18:55 -0600 (Sat, 10 Jan 2009) | 1 line
rewrite verbose conditionals
........
r68495 | benjamin.peterson | 2009-01-10 11:36:44 -0600 (Sat, 10 Jan 2009) | 1 line
tp_iter only exists with Py_TPFLAGS_HAVE_ITER #4901
........
r68501 | vinay.sajip | 2009-01-10 13:22:57 -0600 (Sat, 10 Jan 2009) | 1 line
Corrected minor typo and added .currentmodule directives to fix missing cross-references.
........
r68512 | benjamin.peterson | 2009-01-10 16:42:10 -0600 (Sat, 10 Jan 2009) | 1 line
make tests fail if they can't be imported
........
r68514 | benjamin.peterson | 2009-01-10 17:41:59 -0600 (Sat, 10 Jan 2009) | 1 line
move seealso to a more appropiate place
........
r68515 | benjamin.peterson | 2009-01-10 17:49:08 -0600 (Sat, 10 Jan 2009) | 1 line
macos 9 isn't supported
........
r68534 | gregory.p.smith | 2009-01-11 11:53:33 -0600 (Sun, 11 Jan 2009) | 2 lines
correct email address
........
r68535 | gregory.p.smith | 2009-01-11 11:57:54 -0600 (Sun, 11 Jan 2009) | 9 lines
Update the documentation for binascii and zlib crc32/adler32 functions
to better describe the signed vs unsigned return value behavior on
different platforms and versions of python. Mention the workaround to
make them all return the same thing by using & 0xffffffff.
Fixes issue4903.
Also needs to be merged into release26-maint, release30-maint, & py3k.
........
r68536 | benjamin.peterson | 2009-01-11 13:48:15 -0600 (Sun, 11 Jan 2009) | 1 line
add email addresses
........
r68552 | vinay.sajip | 2009-01-12 14:36:18 -0600 (Mon, 12 Jan 2009) | 1 line
Minor changes/corrections in markup.
........
r68563 | benjamin.peterson | 2009-01-12 19:49:10 -0600 (Mon, 12 Jan 2009) | 1 line
small logic correction
........
r68570 | raymond.hettinger | 2009-01-13 03:08:32 -0600 (Tue, 13 Jan 2009) | 5 lines
Issue 4922: Incorrect comments for MutableSet.add() and MutableSet.discard().
Needs to be backported to 2.6 and forward ported to 3.0 and 3.1.
........
r68571 | armin.ronacher | 2009-01-13 05:52:23 -0600 (Tue, 13 Jan 2009) | 3 lines
ast.literal_eval can properly evaluate complex numbers now. This fixes issue4907.
........
r68572 | andrew.kuchling | 2009-01-13 07:40:54 -0600 (Tue, 13 Jan 2009) | 1 line
Note that first coord. is left alone
........
r68575 | thomas.heller | 2009-01-13 11:32:28 -0600 (Tue, 13 Jan 2009) | 1 line
Fix refcount leak in error cases. Bug found by coverity.
........
r68582 | georg.brandl | 2009-01-13 16:14:01 -0600 (Tue, 13 Jan 2009) | 2 lines
Use assertRaises.
........
r68596 | amaury.forgeotdarc | 2009-01-13 17:39:22 -0600 (Tue, 13 Jan 2009) | 3 lines
#1162154: inspect.getmembers() now skips attributes that raise AttributeError,
e.g. a __slots__ attribute which has not been set.
........
r68623 | vinay.sajip | 2009-01-15 16:48:13 -0600 (Thu, 15 Jan 2009) | 1 line
Made minor changes/corrections in markup. Added a couple of section headings.
........
r68624 | vinay.sajip | 2009-01-15 17:04:47 -0600 (Thu, 15 Jan 2009) | 1 line
Minor changes/corrections in markup.
........
r68628 | benjamin.peterson | 2009-01-15 20:55:24 -0600 (Thu, 15 Jan 2009) | 1 line
compare with == not is #4946
........
svn+ssh://pythondev@svn.python.org/python/trunk
........
r64434 | andrew.kuchling | 2008-06-20 18:13:58 -0500 (Fri, 20 Jun 2008) | 1 line
Remove request for e-mail; it's unlikely these classes will be saved
........
r64435 | andrew.kuchling | 2008-06-20 18:14:32 -0500 (Fri, 20 Jun 2008) | 1 line
Grammar fixes
........
r64440 | andrew.kuchling | 2008-06-21 08:29:12 -0500 (Sat, 21 Jun 2008) | 1 line
Docstring typo
........
r64441 | andrew.kuchling | 2008-06-21 08:47:20 -0500 (Sat, 21 Jun 2008) | 1 line
Use repr() for bad input strings; this makes the empty string or binary characters more visible
........
r64442 | andrew.kuchling | 2008-06-21 08:48:38 -0500 (Sat, 21 Jun 2008) | 1 line
Docstring correction
........
r64443 | georg.brandl | 2008-06-21 09:26:19 -0500 (Sat, 21 Jun 2008) | 2 lines
Documentation fix.
........
r64445 | facundo.batista | 2008-06-21 12:30:06 -0500 (Sat, 21 Jun 2008) | 3 lines
Reviewed and updated the documentation. Fixes#3017.
........
r64447 | facundo.batista | 2008-06-21 13:58:04 -0500 (Sat, 21 Jun 2008) | 6 lines
Now a from submitted via POST that also has a query string
will contain both FieldStorage and MiniFieldStorage items.
Fixes#1817.
........
r64448 | facundo.batista | 2008-06-21 14:48:19 -0500 (Sat, 21 Jun 2008) | 5 lines
In the deprecated functions I added an alert to review
specially a section of the subprocess documentation
that helps with the replacing of those functionss.
........
r64450 | georg.brandl | 2008-06-22 04:05:29 -0500 (Sun, 22 Jun 2008) | 2 lines
Turn section references into proper cross-references.
........
r64452 | facundo.batista | 2008-06-22 08:36:20 -0500 (Sun, 22 Jun 2008) | 5 lines
Issue #2722. Now the char buffer to support the path string has
not fixed length, it mallocs memory if needed. As a result, we
don't have a maximum for the getcwd() method.
........
r64455 | facundo.batista | 2008-06-22 10:27:10 -0500 (Sun, 22 Jun 2008) | 4 lines
Issue 3164. Small fix to don't repeat a comparation
without necessity.
........
r64461 | georg.brandl | 2008-06-22 13:11:52 -0500 (Sun, 22 Jun 2008) | 2 lines
#3085: Fix syntax error.
........
r64464 | georg.brandl | 2008-06-22 13:31:54 -0500 (Sun, 22 Jun 2008) | 2 lines
Expand docstrings of sqlite3 functions.
........
r64466 | georg.brandl | 2008-06-22 14:07:59 -0500 (Sun, 22 Jun 2008) | 2 lines
Write out "phi" consistently.
........
r64468 | facundo.batista | 2008-06-22 14:35:24 -0500 (Sun, 22 Jun 2008) | 4 lines
Just returning nothing instead of rising TestSkipped, because
it makes the test fail in the trunk.loewis-sun buildbot.
........
Guido van Rossum
Ivan Levkivskyi
Anthony Sottile
Serhiy Storchaka
Matthias Bussonnier
INADA Naoki
Victor Stinner
Yury Selivanov
Benjamin Peterson
Éric Araujo
Terry Reedy
Georg Brandl
Raymond Hettinger
Florent Xicluna