traverseda
/
cpython
1
0
Fork 0
Code Issues Pull Requests Releases Wiki Activity
cpython/Include
History
Ivan Levkivskyi 9932a22897
bpo-33416: Add end positions to Python AST (GH-11605) 
The majority of this PR is tediously passing `end_lineno` and `end_col_offset` everywhere. Here are non-trivial points:
* It is not possible to reconstruct end positions in AST "on the fly", some information is lost after an AST node is constructed, so we need two more attributes for every AST node `end_lineno` and `end_col_offset`.
* I add end position information to both CST and AST.  Although it may be technically possible to avoid adding end positions to CST, the code becomes more cumbersome and less efficient.
* Since the end position is not known for non-leaf CST nodes while the next token is added, this requires a bit of extra care (see `_PyNode_FinalizeEndPos`). Unless I made some mistake, the algorithm should be linear.
* For statements, I "trim" the end position of suites to not include the terminal newlines and dedent (this seems to be what people would expect), for example in
  ```python
  class C:
      pass

  pass
  ```
  the end line and end column for the class definition is (2, 8).
* For `end_col_offset` I use the common Python convention for indexing, for example for `pass` the `end_col_offset` is 4 (not 3), so that `[0:4]` gives one the source code that corresponds to the node.
* I added a helper function `ast.get_source_segment()`, to get source text segment corresponding to a given AST node. It is also useful for testing.

An (inevitable) downside of this PR is that AST now takes almost 25% more memory. I think however it is probably justified by the benefits.
 		2019-01-22 11:18:22 +00:00
..
cpython bpo-35444: Unify and optimize the helper for getting a builtin object. (GH-11047) 2018-12-11 08:28:18 +02:00
internal bpo-35758: Fix building on ARM + MSVC (gh-11531) 2019-01-21 21:49:40 +01:00
Python-ast.h bpo-33416: Add end positions to Python AST (GH-11605) 2019-01-22 11:18:22 +00:00
Python.h bpo-28503: Use crypt_r() when available instead of crypt() (GH-11373) 2018-12-30 15:42:32 -08:00
abstract.h bpo-35134: Create Include/cpython/abstract.h (GH-10728) 2018-11-26 22:42:04 +01:00
asdl.h bpo-18407: ast.c uses Py_ssize_t for asdl_seq_LEN() iterator (GH-10655) 2018-11-22 14:45:16 +01:00
ast.h bpo-35177: Add dependencies between header files (GH-10361) 2018-11-12 00:56:19 +01:00
bitset.h bpo-32150: Expand tabs to spaces in C files. (#4583) 2017-11-28 17:56:10 +02:00
bltinmodule.h
…
boolobject.h
…
bytearrayobject.h fix comment 2013-10-05 16:28:04 -04:00
bytes_methods.h bpo-32677: Add .isascii() to str, bytes and bytearray (GH-5342) 2018-01-27 14:06:21 +09:00
bytesobject.h bpo-32150: Expand tabs to spaces in C files. (#4583) 2017-11-28 17:56:10 +02:00
cellobject.h bpo-32150: Expand tabs to spaces in C files. (#4583) 2017-11-28 17:56:10 +02:00
ceval.h bpo-35444: Unify and optimize the helper for getting a builtin object. (GH-11047) 2018-12-11 08:28:18 +02:00
classobject.h bpo-32150: Expand tabs to spaces in C files. (#4583) 2017-11-28 17:56:10 +02:00
code.h String annotations [PEP 563] (#4390) 2018-01-26 08:20:18 -08:00
codecs.h Issue #29058: All stable API extensions added after Python 3.2 are now 2016-12-27 14:57:39 +02:00
compile.h bpo-32455: Add jump parameter to dis.stack_effect(). (GH-6610) 2018-09-18 09:54:26 +03:00
complexobject.h Issue #21803: remove macro indirections in complexobject.h 2014-07-07 18:49:30 -04:00
context.h bpo-34762: Fix contextvars C API to use PyObject* pointer types. (GH-9473) 2018-09-21 15:33:56 -04:00
coreconfig.h bpo-35233: InitConfigTests tests more config vars (GH-10541) 2018-11-14 17:39:45 +01:00
datetime.h bpo-35081: Remove Py_BUILD_CORE from datetime.h (GH-10416) 2018-11-13 09:02:25 +01:00
descrobject.h bpo-32240: Add the const qualifier to declarations of PyObject* array arguments. (#4746) 2017-12-15 13:11:11 +02:00
dictobject.h bpo-35134: Create Include/cpython/dictobject.h (GH-10732) 2018-11-27 00:12:26 +01:00
dtoa.h Issue #14521: Make result of float('nan') and float('-nan') more consistent across platforms. Further, don't rely on Py_HUGE_VAL for float('inf'). 2012-04-29 15:31:56 +01:00
dynamic_annotations.h fix instances of consecutive articles (closes #23221) 2015-01-13 09:17:24 -05:00
enumobject.h
…
errcode.h bpo-32150: Expand tabs to spaces in C files. (#4583) 2017-11-28 17:56:10 +02:00
eval.h bpo-32240: Add the const qualifier to declarations of PyObject* array arguments. (#4746) 2017-12-15 13:11:11 +02:00
fileobject.h bpo-29240: PEP 540: Add a new UTF-8 Mode (#855) 2017-12-13 12:29:09 +01:00
fileutils.h bpo-28604: Fix localeconv() for different LC_MONETARY (GH-10606) 2018-11-20 16:20:16 +01:00
floatobject.h Issue #11734: Add support for IEEE 754 half-precision floats to the struct module. Original patch by Eli Stevens. 2016-09-03 17:21:29 +01:00
frameobject.h Move exc state to generator. Fixes bpo-25612 (#1773) 2017-10-22 23:41:51 +02:00
funcobject.h bpo-32240: Add the const qualifier to declarations of PyObject* array arguments. (#4746) 2017-12-15 13:11:11 +02:00
genobject.h bpo-32591: Add native coroutine origin tracking (#5250) 2018-01-21 09:44:07 -05:00
graminit.h bpo-30406: Make async and await proper keywords (#1669) 2017-10-05 23:24:46 -04:00
grammar.h bpo-32150: Expand tabs to spaces in C files. (#4583) 2017-11-28 17:56:10 +02:00
import.h Fix redundant declaration of _PyImport_AddModuleObject (GH-7992) 2018-06-29 16:47:18 -06:00
intrcheck.h Add missed details of the C API introduced in 3.7. (GH-7047) 2018-05-22 20:59:42 +03:00
iterobject.h Rename _PyIter_GetBuiltin to _PyObject_GetBuiltin, and do not include it in the stable ABI. 2012-04-05 00:04:20 +02:00
listobject.h bpo-29867: Add asserts in PyTuple_GET_SIZE, PyList_GET_SIZE and PySet_GET_SIZE. (#751) 2017-04-22 01:48:11 +02:00
longintrepr.h bpo-32150: Expand tabs to spaces in C files. (#4583) 2017-11-28 17:56:10 +02:00
longobject.h bpo-20260: Implement non-bitwise unsigned int converters for Argument Clinic. (GH-8434) 2018-07-26 13:22:16 +03:00
marshal.h Issue #1772673: The type of `char*` arguments now changed to `const char*`. 2013-10-19 21:03:34 +03:00
memoryobject.h Issue #29058: All stable API extensions added after Python 3.2 are now 2016-12-27 14:57:39 +02:00
metagrammar.h
…
methodobject.h bpo-32240: Add the const qualifier to declarations of PyObject* array arguments. (#4746) 2017-12-15 13:11:11 +02:00
modsupport.h bpo-35582: Argument Clinic: inline parsing code for positional parameters. (GH-11313) 2019-01-11 16:01:14 +02:00
moduleobject.h bpo-33237: Improve AttributeError message for partially initialized module. (GH-6398) 2018-10-30 13:19:51 +02:00
namespaceobject.h Issue #26900: Excluded underscored names and other private API from limited API. 2016-09-11 11:03:14 +03:00
node.h bpo-33416: Add end positions to Python AST (GH-11605) 2019-01-22 11:18:22 +00:00
object.h bpo-35134: Don't define types twice in header files (GH-10754) 2018-11-27 23:54:59 +01:00
objimpl.h bpo-35134: Create Include/cpython/ subdirectory (GH-10624) 2018-11-23 17:00:00 +01:00
odictobject.h bpo-35059: Add _PyObject_CAST() macro (GH-10645) 2018-11-22 02:57:29 +01:00
opcode.h Update opcode.h header comment to mention the source data file (GH-9935) 2018-10-18 14:53:18 -05:00
osdefs.h Issue #22591: Drop support of MS-DOS 2014-10-10 11:55:41 +02:00
osmodule.h Issue #29058: All stable API extensions added after Python 3.2 are now 2016-12-27 14:57:39 +02:00
parsetok.h bpo-35177: Add dependencies between header files (GH-10361) 2018-11-12 00:56:19 +01:00
patchlevel.h Start of 3.8.0a0 2018-01-31 17:44:09 -05:00
pgen.h
…
pgenheaders.h bpo-32150: Expand tabs to spaces in C files. (#4583) 2017-11-28 17:56:10 +02:00
py_curses.h bpo-20171: Convert the _curses and _curses_panel modules to Argument Clinic. (GH-4251) 2018-05-10 11:27:23 +03:00
pyarena.h
…
pycapsule.h Merged revisions 88517 via svnmerge from 2011-02-22 23:43:57 +00:00
pyctype.h Try to fix linking failures under Windows 2011-10-04 14:43:47 +02:00
pydebug.h bpo-32125: Remove Py_UseClassExceptionsFlag flag (#4544) 2017-11-24 22:06:38 +01:00
pydtrace.d bpo-31574: importlib dtrace (#3749) 2017-09-29 15:53:19 -07:00
pydtrace.h bpo-35059: Remove Py_STATIC_INLINE() macro (GH-10216) 2018-10-29 14:49:24 +01:00
pyerrors.h bpo-35134: Add Include/cpython/pyerrors.h (GH-10727) 2018-11-26 22:11:25 +01:00
pyexpat.h bpo-34623: Use XML_SetHashSalt in _elementtree (GH-9146) 2018-09-18 05:38:58 -07:00
pyfpe.h bpo-29137: Remove fpectl module (#4789) 2018-01-05 23:15:34 -08:00
pyhash.h Correct the location of a function mentioned in a comment (GH-4327) 2017-11-09 19:55:34 +03:00
pylifecycle.h bpo-35259: Limit `Py_FinalizeEx()` to `Py_LIMITED_API >= 0x03060000`. (GH-10620) 2018-12-20 23:11:03 +02:00
pymacconfig.h Minor spelling fixes 2016-08-20 08:03:06 +00:00
pymacro.h bpo-31338 (#3374) 2017-09-14 18:13:16 -07:00
pymath.h bpo-31373: fix undefined floating-point demotions (#3396) 2017-09-07 11:13:59 -07:00
pymem.h bpo-35081: Move Py_BUILD_CORE code to internal/mem.h (GH-10249) 2018-10-31 20:19:24 +01:00
pyport.h bpo-35758: Fix building on ARM + MSVC (gh-11531) 2019-01-21 21:49:40 +01:00
pystate.h bpo-35134: Don't define types twice in header files (GH-10754) 2018-11-27 23:54:59 +01:00
pystrcmp.h
…
pystrhex.h Issue #26900: Excluded underscored names and other private API from limited API. 2016-09-11 11:03:14 +03:00
pystrtod.h Issue #26331: Implement the parsing part of PEP 515. 2016-09-09 14:57:09 -07:00
pythonrun.h bpo-22257: Small changes for PEP 432. (#1728) 2017-05-22 21:36:03 -07:00
pythread.h bpo-30768: Recompute timeout on interrupted lock (GH-4103) 2017-10-24 16:53:32 -07:00
pytime.h bpo-31784: Implement PEP 564: add time.time_ns() (#3989) 2017-11-02 07:28:27 -07:00
rangeobject.h
…
setobject.h bpo-29867: Add asserts in PyTuple_GET_SIZE, PyList_GET_SIZE and PySet_GET_SIZE. (#751) 2017-04-22 01:48:11 +02:00
sliceobject.h bpo-32150: Expand tabs to spaces in C files. (#4583) 2017-11-28 17:56:10 +02:00
structmember.h Issue #28761: The fields name and doc of structures PyMemberDef, PyGetSetDef, 2016-11-22 07:58:08 +02:00
structseq.h Issue #28761: The fields name and doc of structures PyMemberDef, PyGetSetDef, 2016-11-22 07:58:08 +02:00
symtable.h bpo-35177: Add dependencies between header files (GH-10361) 2018-11-12 00:56:19 +01:00
sysmodule.h bpo-32030: Make _PySys_AddXOptionWithError() private (GH-10236) 2018-10-30 14:31:42 +01:00
token.h bpo-30455: Generate all token related code and docs from Grammar/Tokens. (GH-10370) 2018-12-22 11:18:40 +02:00
traceback.h Issue #26900: Excluded underscored names and other private API from limited API. 2016-09-11 11:03:14 +03:00
tracemalloc.h bpo-35053: Define _PyTraceMalloc_NewReference in object.h (GH-10107) 2018-10-26 00:01:56 +02:00
tupleobject.h bpo-35134: Create Include/cpython/tupleobject.h (GH-10764) 2018-11-28 13:01:32 +01:00
typeslots.h Issue 24365: Conditionalize PEP 489 additions to the stable ABI 2015-06-02 19:06:47 -04:00
ucnhash.h Removed unintentional trailing spaces in non-external and non-generated C files. 2015-03-18 21:53:15 +02:00
unicodeobject.h bpo-35134: Create Include/cpython/unicodeobject.h (GH-10680) 2018-11-26 17:29:38 +01:00
warnings.h bpo-32591: Add native coroutine origin tracking (#5250) 2018-01-21 09:44:07 -05:00
weakrefobject.h Closes #13867: remove untrue comment about PyWeakref_Check(). 2013-10-08 19:50:26 +02:00