The majority of this PR is tediously passing `end_lineno` and `end_col_offset` everywhere. Here are non-trivial points:
* It is not possible to reconstruct end positions in AST "on the fly", some information is lost after an AST node is constructed, so we need two more attributes for every AST node `end_lineno` and `end_col_offset`.
* I add end position information to both CST and AST. Although it may be technically possible to avoid adding end positions to CST, the code becomes more cumbersome and less efficient.
* Since the end position is not known for non-leaf CST nodes while the next token is added, this requires a bit of extra care (see `_PyNode_FinalizeEndPos`). Unless I made some mistake, the algorithm should be linear.
* For statements, I "trim" the end position of suites to not include the terminal newlines and dedent (this seems to be what people would expect), for example in
```python
class C:
pass
pass
```
the end line and end column for the class definition is (2, 8).
* For `end_col_offset` I use the common Python convention for indexing, for example for `pass` the `end_col_offset` is 4 (not 3), so that `[0:4]` gives one the source code that corresponds to the node.
* I added a helper function `ast.get_source_segment()`, to get source text segment corresponding to a given AST node. It is also useful for testing.
An (inevitable) downside of this PR is that AST now takes almost 25% more memory. I think however it is probably justified by the benefits.
* ast.h now includes Python-ast.h and node.h
* parsetok.h now includes node.h and grammar.h
* symtable.h now includes Python-ast.h
* Modify asdl_c.py to enhance Python-ast.h:
* Add #ifndef/#define Py_PYTHON_AST_H to be able to include the header
twice
* Add "extern { ... }" for C++
* Undefine "Yield" macro conflicting with winbase.h
* Remove "#undef Yield" from C files, it's now done in Python-ast.h
* Remove now useless includes in C files
bpo-29463 added optional "docstring" field to 4 AST types.
While it is optional, it breaks backward compatibility because AST constructor
requires number of positional argument is same to number of fields.
AST types accepts empty arguments, and incomplete keyword arguments.
But it's not big problem because field can be filled after creation, and checked when compiling.
So stop requiring complete set of fields for positional arguments too.
obj2ast_constant() code is baesd on obj2ast_object() which has a special case
for Py_None. But in practice, we don't need to have a special case for
constants.
Issue noticed by Joseph Jevnik on a review.
Issue #26146: Add a new kind of AST node: ast.Constant. It can be used by
external AST optimizers, but the compiler does not emit directly such node.
An optimizer can replace the following AST nodes with ast.Constant:
* ast.NameConstant: None, False, True
* ast.Num: int, float, complex
* ast.Str: str
* ast.Bytes: bytes
* ast.Tuple if items are constants too: tuple
* frozenset
Update code to accept ast.Constant instead of ast.Num and/or ast.Str:
* compiler
* docstrings
* ast.literal_eval()
* Tools/parser/unparse.py
The new parser does not rely on Spark (which is now removed from our repo),
uses modern 3.x idioms and is significantly smaller and simpler.
It generates exactly the same AST files (.h and .c), so in practice no builds
should be affected.
svn+ssh://pythondev@svn.python.org/python/trunk
........
r81465 | georg.brandl | 2010-05-22 06:29:19 -0500 (Sat, 22 May 2010) | 2 lines
Issue #3924: Ignore cookies with invalid "version" field in cookielib.
........
r81466 | georg.brandl | 2010-05-22 06:31:16 -0500 (Sat, 22 May 2010) | 1 line
Underscore the name of an internal utility function.
........
r81468 | georg.brandl | 2010-05-22 06:43:25 -0500 (Sat, 22 May 2010) | 1 line
#8635: document enumerate() start parameter in docstring.
........
r81679 | benjamin.peterson | 2010-06-03 16:21:03 -0500 (Thu, 03 Jun 2010) | 1 line
use a set for membership testing
........
r81735 | michael.foord | 2010-06-05 06:46:59 -0500 (Sat, 05 Jun 2010) | 1 line
Extract error message truncating into a method (unittest.TestCase._truncateMessage).
........
r81760 | michael.foord | 2010-06-05 14:38:42 -0500 (Sat, 05 Jun 2010) | 1 line
Issue 8302. SkipTest exception is setUpClass or setUpModule is now reported as a skip rather than an error.
........
r81868 | benjamin.peterson | 2010-06-09 14:45:04 -0500 (Wed, 09 Jun 2010) | 1 line
fix code formatting
........
r82183 | benjamin.peterson | 2010-06-23 15:29:26 -0500 (Wed, 23 Jun 2010) | 1 line
cpython only gc tests
........
svn+ssh://pythondev@svn.python.org/python/trunk
........
r62039 | georg.brandl | 2008-03-29 06:24:23 -0700 (Sat, 29 Mar 2008) | 3 lines
Properly check for consistency with the third argument of
compile() when compiling an AST node.
........
r62040 | amaury.forgeotdarc | 2008-03-29 06:47:05 -0700 (Sat, 29 Mar 2008) | 5 lines
The buildbot "x86 W2k8 trunk" seems to hang in test_socket.
http://www.python.org/dev/buildbot/trunk/x86%20W2k8%20trunk/builds/255/step-test/0
Temporarily increase verbosity of this test.
........
r62042 | amaury.forgeotdarc | 2008-03-29 07:53:05 -0700 (Sat, 29 Mar 2008) | 3 lines
Still investigating on the hanging test_socket.
the test itself doesn't do anything on windows, focus on setUp and tearDown.
........
svn+ssh://pythondev@svn.python.org/python/trunk
........
r60990 | eric.smith | 2008-02-23 17:05:26 +0100 (Sat, 23 Feb 2008) | 1 line
Removed duplicate Py_CHARMASK define. It's already defined in Python.h.
........
r60991 | andrew.kuchling | 2008-02-23 17:23:05 +0100 (Sat, 23 Feb 2008) | 4 lines
#1330538: Improve comparison of xmlrpclib.DateTime and datetime instances.
Remove automatic handling of datetime.date and datetime.time.
This breaks backward compatibility, but python-dev discussion was strongly
against this automatic conversion; see the bug for a link.
........
r60994 | andrew.kuchling | 2008-02-23 17:39:43 +0100 (Sat, 23 Feb 2008) | 1 line
#835521: Add index entries for various pickle-protocol methods and attributes
........
r60995 | andrew.kuchling | 2008-02-23 18:10:46 +0100 (Sat, 23 Feb 2008) | 2 lines
#1433694: minidom's .normalize() failed to set .nextSibling for last element.
Fix by Malte Helmert
........
r61000 | christian.heimes | 2008-02-23 18:40:11 +0100 (Sat, 23 Feb 2008) | 1 line
Patch #2167 from calvin: Remove unused imports
........
r61001 | christian.heimes | 2008-02-23 18:42:31 +0100 (Sat, 23 Feb 2008) | 1 line
Patch #1957: syslogmodule: Release GIL when calling syslog(3)
........
r61002 | christian.heimes | 2008-02-23 18:52:07 +0100 (Sat, 23 Feb 2008) | 2 lines
Issue #2051 and patch from Alexander Belopolsky:
Permission for pyc and pyo files are inherited from the py file.
........
svn+ssh://pythondev@svn.python.org/python/branches/p3yk
........
r55077 | guido.van.rossum | 2007-05-02 11:54:37 -0700 (Wed, 02 May 2007) | 2 lines
Use the new print syntax, at least.
........
r55142 | fred.drake | 2007-05-04 21:27:30 -0700 (Fri, 04 May 2007) | 1 line
remove old cruftiness
........
r55143 | fred.drake | 2007-05-04 21:52:16 -0700 (Fri, 04 May 2007) | 1 line
make this work with the new Python
........
r55162 | neal.norwitz | 2007-05-06 22:29:18 -0700 (Sun, 06 May 2007) | 1 line
Get asdl code gen working with Python 2.3. Should continue to work with 3.0
........
r55164 | neal.norwitz | 2007-05-07 00:00:38 -0700 (Mon, 07 May 2007) | 1 line
Verify checkins to p3yk (sic) branch go to 3000 list.
........
r55166 | neal.norwitz | 2007-05-07 00:12:35 -0700 (Mon, 07 May 2007) | 1 line
Fix this test so it runs again by importing warnings_test properly.
........
r55167 | neal.norwitz | 2007-05-07 01:03:22 -0700 (Mon, 07 May 2007) | 8 lines
So long xrange. range() now supports values that are outside
-sys.maxint to sys.maxint. floats raise a TypeError.
This has been sitting for a long time. It probably has some problems and
needs cleanup. Objects/rangeobject.c now uses 4-space indents since
it is almost completely new.
........
r55171 | guido.van.rossum | 2007-05-07 10:21:26 -0700 (Mon, 07 May 2007) | 4 lines
Fix two tests that were previously depending on significant spaces
at the end of a line (and before that on Python 2.x print behavior
that has no exact equivalent in 3.0).
........
Guido van Rossum
Ivan Levkivskyi
Victor Stinner
Serhiy Storchaka
Antoine Pitrou
INADA Naoki
Yuan Chao Chou
INADA Naoki
Victor Stinner
Benjamin Peterson
Eli Bendersky
Antoine Pitrou
Ezio Melotti
Christian Heimes
Matthias Klose
Amaury Forgeot d'Arc
Meador Inge
Martin v. Löwis
Éric Araujo
Mark Dickinson
Neal Norwitz