Commit Graph

34 Commits

Author SHA1 Message Date
Ivan Levkivskyi 9932a22897
bpo-33416: Add end positions to Python AST (GH-11605)
The majority of this PR is tediously passing `end_lineno` and `end_col_offset` everywhere. Here are non-trivial points:
* It is not possible to reconstruct end positions in AST "on the fly", some information is lost after an AST node is constructed, so we need two more attributes for every AST node `end_lineno` and `end_col_offset`.
* I add end position information to both CST and AST.  Although it may be technically possible to avoid adding end positions to CST, the code becomes more cumbersome and less efficient.
* Since the end position is not known for non-leaf CST nodes while the next token is added, this requires a bit of extra care (see `_PyNode_FinalizeEndPos`). Unless I made some mistake, the algorithm should be linear.
* For statements, I "trim" the end position of suites to not include the terminal newlines and dedent (this seems to be what people would expect), for example in
  ```python
  class C:
      pass

  pass
  ```
  the end line and end column for the class definition is (2, 8).
* For `end_col_offset` I use the common Python convention for indexing, for example for `pass` the `end_col_offset` is 4 (not 3), so that `[0:4]` gives one the source code that corresponds to the node.
* I added a helper function `ast.get_source_segment()`, to get source text segment corresponding to a given AST node. It is also useful for testing.

An (inevitable) downside of this PR is that AST now takes almost 25% more memory. I think however it is probably justified by the benefits.
2019-01-22 11:18:22 +00:00
Serhiy Storchaka 598ceae876
bpo-32150: Expand tabs to spaces in C files. (#4583) 2017-11-28 17:56:10 +02:00
Serhiy Storchaka f402775e5d Removed trailing tabs. 2015-03-21 09:25:53 +02:00
Serhiy Storchaka 60fe569673 Issue #18637: Fixed an error in _PyNode_SizeOf declaration.
Patch by Roumen Petrov.
2014-11-18 17:30:15 +02:00
Jesus Cea 88ca04e6a8 MERGE: Closes #15512: Correct __sizeof__ support for parser 2012-08-03 14:29:26 +02:00
Jesus Cea e9c5318967 Closes #15512: Correct __sizeof__ support for parser 2012-08-03 14:28:37 +02:00
Benjamin Peterson c0beabc2a5 move LINENO define to where it actually belongs 2012-01-16 17:29:05 -05:00
Martin v. Löwis 49c5da1d88 Patch #1440601: Add col_offset attribute to AST nodes. 2006-03-01 22:49:05 +00:00
Anthony Baxter c2a5a63654 PEP-0318, @decorator-style. In Guido's words:
"@ seems the syntax that everybody can hate equally"
Implementation by Mark Russell, from SF #979728.
2004-08-02 06:10:11 +00:00
Guido van Rossum e5de77800e Back out the previous change. It doesn't save any space on 64-bit
platforms.
2002-09-17 03:27:02 +00:00
Guido van Rossum 73d66925d9 Rearrange the members of struct _node to put shorter fields towards
the end, in the hope of saving some bytes on 64-bit machines.  (Too
bad n_nchildren can't be made an unsigned short, but
test/test_longexp.py specifically tests for more than 2**16 subtrees
at one level.)

I don't expect any binary compatibility issues here, unless someone
has an old binary of parsermodule.so saved away.
2002-09-17 03:24:09 +00:00
Mark Hammond 91a681debf Excise DL_EXPORT from Include.
Thanks to Skip Montanaro and Kalle Svensson for the patches.
2002-08-12 07:21:58 +00:00
Guido van Rossum 5aace07fe0 Use an assert() for the REQ() macro instead of making up our own
assertion.
2001-10-15 17:23:13 +00:00
Guido van Rossum 8586991099 REMOVED all CWI, CNRI and BeOpen copyright markings.
This should match the situation in the 1.6b1 tree.
2000-09-01 23:29:29 +00:00
Fred Drake ef8ace3a6f Charles G. Waldman <cgw@fnal.gov>:
Add the EXTENDED_ARG opcode to the virtual machine, allowing 32-bit
arguments to opcodes instead of being forced to stick to the 16-bit
limit.  This is especially useful for machine-generated code, which
can be too long for the SET_LINENO parameter to fit into 16 bits.

This closes the implementation portion of SourceForge patch #100893.
2000-08-24 00:32:09 +00:00
Fred Drake 5eb6d4e3bf ANSI-fication and Py_PROTO extermination. 2000-07-08 23:37:28 +00:00
Guido van Rossum ffcc3813d8 Change copyright notice - 2nd try. 2000-06-30 23:58:06 +00:00
Guido van Rossum fd71b9e9d4 Change copyright notice. 2000-06-30 23:50:40 +00:00
Jeremy Hylton 94988067b9 Add new parser error code, E_OVERFLOW. This error is returned when
the number of children of a node exceeds the max possible value for
the short that is used to count them.  The Python runtime converts
this parser error into the SyntaxError "expression too long."
2000-06-20 19:10:44 +00:00
Guido van Rossum 43466ec7b0 Add DL_IMPORT(returntype) for all officially exported functions. 1998-12-04 18:48:25 +00:00
Guido van Rossum 408027ea46 Rename DEBUG macro to Py_DEBUG 1996-12-30 16:17:54 +00:00
Guido van Rossum d266eb460e New permission notice, includes CNRI. 1996-10-25 14:44:06 +00:00
Guido van Rossum caa6380886 The great renaming, phase two: all header files have been updated to
use the new names exclusively, and the linker will see the new names.
Files that import "Python.h" also only see the new names.  Files that
import "allobjects.h" will continue to be able to use the old names,
due to the inclusion (in allobjects.h) of "rename2.h".
1995-01-12 11:45:45 +00:00
Guido van Rossum 5799b52008 Added 1995 copyright.
object.h: made sizes and refcnts signed ints.
stringobject.h: make getstrsize() signed int.
methodobject.h: add METH_VARARGS and METH_FREENAME flag bit definitions.
1995-01-04 19:06:22 +00:00
Guido van Rossum b4f066da49 Include/node.h: make some fields short to save space during
parsing
1994-08-23 13:37:43 +00:00
Guido van Rossum b6775db241 Merge alpha100 branch back to main trunk 1994-08-01 11:34:53 +00:00
Guido van Rossum b73cc04e62 * ceval.c, longobject.c, methodobject.c, listnode.c, arraymodule.c,
pythonrun.c: added static forward declarations
* pythonrun.h, ceval.h, longobject.h, node.h: removed declarations of
  static routines
1993-11-01 16:28:59 +00:00
Guido van Rossum a3309960a5 * Added support for X11 modules.
* Makefile: change location of FORMS library.
* posixmodule.c: turn #if 0 into #ifdef MSDOS (stuff in unistd.h or not)
* Almost all .h files: added CPP magic to avoid duplicate inclusions and
  to support inclusion from C++.
1993-07-28 09:05:47 +00:00
Guido van Rossum 9bfef44d97 * Changed all copyright messages to include 1993.
* Stubs for faster implementation of local variables (not yet finished)
* Added function name to code object.  Print it for code and function
  objects.  THIS MAKES THE .PYC FILE FORMAT INCOMPATIBLE (the version
  number has changed accordingly)
* Print address of self for built-in methods
* New internal functions getattro and setattro (getattr/setattr with
  string object arg)
* Replaced "dictobject" with more powerful "mappingobject"
* New per-type functio tp_hash to implement arbitrary object hashing,
  and hashobject() to interface to it
* Added built-in functions hash(v) and hasattr(v, 'name')
* classobject: made some functions static that accidentally weren't;
  added __hash__ special instance method to implement hash()
* Added proper comparison for built-in methods and functions
1993-03-29 10:43:31 +00:00
Guido van Rossum 5113f5fd34 Copyright for 1992 added 1992-04-05 14:20:22 +00:00
Guido van Rossum f70e43a073 Added copyright notice. 1991-02-19 12:39:46 +00:00
Guido van Rossum 3f5da24ea3 "Compiling" version 1990-12-20 15:06:42 +00:00
Guido van Rossum cc398d1cbb Added prototype for new function freenode(). 1990-11-18 17:37:25 +00:00
Guido van Rossum 85a5fbbdfe Initial revision 1990-10-14 12:07:46 +00:00