Commit Graph

32 Commits

Author SHA1 Message Date
Guido van Rossum 495da29225 bpo-35975: Support parsing earlier minor versions of Python 3 (GH-12086)
This adds a `feature_version` flag to `ast.parse()` (documented) and `compile()` (hidden) that allow tweaking the parser to support older versions of the grammar. In particular if `feature_version` is 5 or 6, the hacks for the `async` and `await` keyword from PEP 492 are reinstated. (For 7 or higher, these are unconditionally treated as keywords, but they are still special tokens rather than `NAME` tokens that the parser driver recognizes.)



https://bugs.python.org/issue35975
2019-03-07 12:38:08 -08:00
Guido van Rossum dcfcd146f8 bpo-35766: Merge typed_ast back into CPython (GH-11645) 2019-01-31 12:40:27 +01:00
Ivan Levkivskyi 9932a22897
bpo-33416: Add end positions to Python AST (GH-11605)
The majority of this PR is tediously passing `end_lineno` and `end_col_offset` everywhere. Here are non-trivial points:
* It is not possible to reconstruct end positions in AST "on the fly", some information is lost after an AST node is constructed, so we need two more attributes for every AST node `end_lineno` and `end_col_offset`.
* I add end position information to both CST and AST.  Although it may be technically possible to avoid adding end positions to CST, the code becomes more cumbersome and less efficient.
* Since the end position is not known for non-leaf CST nodes while the next token is added, this requires a bit of extra care (see `_PyNode_FinalizeEndPos`). Unless I made some mistake, the algorithm should be linear.
* For statements, I "trim" the end position of suites to not include the terminal newlines and dedent (this seems to be what people would expect), for example in
  ```python
  class C:
      pass

  pass
  ```
  the end line and end column for the class definition is (2, 8).
* For `end_col_offset` I use the common Python convention for indexing, for example for `pass` the `end_col_offset` is 4 (not 3), so that `[0:4]` gives one the source code that corresponds to the node.
* I added a helper function `ast.get_source_segment()`, to get source text segment corresponding to a given AST node. It is also useful for testing.

An (inevitable) downside of this PR is that AST now takes almost 25% more memory. I think however it is probably justified by the benefits.
2019-01-22 11:18:22 +00:00
Serhiy Storchaka 913876d824
bpo-35054: Add yet more index entries for symbols. (GH-10121) 2018-10-28 13:41:26 +02:00
Serhiy Storchaka 3f22811fef
bpo-32892: Use ast.Constant instead of specific constant AST types. (GH-9445) 2018-09-27 17:42:37 +03:00
Serhiy Storchaka 73cbe7a01a
bpo-32911: Revert bpo-29463. (GH-7121) (GH-7197)
Remove the docstring attribute of AST types and restore docstring
expression as a first stmt in their body.

Co-authored-by: INADA Naoki <methane@users.noreply.github.com>
2018-05-29 12:04:55 +03:00
Brett Cannon 7a7f100eb3
bpo-32758: Warn that ast.parse() and ast.literal_eval() can segfault the interpreter (GH-5960) 2018-03-09 12:03:22 -08:00
Sanyam Khurana 338cd83c5d bpo-25910: Link redirections in docs (#1933)
Fixes some redirection links in docs.
2018-01-20 01:25:37 +01:00
INADA Naoki cb41b2766d bpo-29463: Add docstring field to some AST nodes. (#46)
* bpo-29463: Add docstring field to some AST nodes.

ClassDef, ModuleDef, FunctionDef, and AsyncFunctionDef has docstring
field for now.  It was first statement of there body.

* fix document.  thanks travis!

* doc fixes
2017-02-22 16:31:59 +01:00
Martin Panter 1050d2d0c7 Issue #26462: Doc: reduce literal_block warnings, fix syntax highlighting.
Patch by Julien Palard.
2016-07-26 11:18:21 +02:00
Senthil Kumaran f3695bfacf Issue16544 - Add a link to an external documentation resource in ast module docs. 2016-01-06 21:26:53 -08:00
Martin Panter 2e4571a456 Fix a few grammar problems in the documentation and comments 2015-11-14 01:07:43 +00:00
Georg Brandl b9b389efed Closes #22525: clarify documentation for ast.literal_eval(). 2014-11-05 20:20:28 +01:00
Serhiy Storchaka fbc1c26803 Issue #19795: Improved markup of True/False constants. 2013-11-29 12:17:13 +02:00
Benjamin Peterson 0bd152cd67 remove ast.__version__ (closes #12273) 2011-07-15 21:10:13 -05:00
Benjamin Peterson 6cb2b923e0 convert ast versioning to mercurial 2011-03-12 18:28:16 -06:00
Terry Reedy feac624827 Issue #11000 ast.parse parses source, not just expressions. 2011-01-24 21:36:03 +00:00
Raymond Hettinger 4f707fd316 Separate source link from main text. 2011-01-10 19:54:11 +00:00
Raymond Hettinger 1048094037 Move source links to consistent location and remove wordy, big yellow boxes. 2011-01-10 03:26:08 +00:00
Georg Brandl 619e7ba814 #10869: do not visit root node twice in ast.increment_lineno(). 2011-01-09 07:38:51 +00:00
Éric Araujo 6e6cb8e0e4 Provide links to Python source where the code is short, readable and
informative adjunct to the docs.  Forward-port of Raymond's r86225 and
r86245 using the new source reST role added in #10334.
2010-11-16 19:13:50 +00:00
Alexander Belopolsky f0a0d14c28 Issue #7061: Dropped "for Tk" from turtle module title and moved its
doc section under frameworks.  Also fixed a couple of markup issues
that affected TOC rendering.
2010-10-27 03:06:43 +00:00
Georg Brandl 85f2177f6d Add bytes in literal_eval doc. 2010-07-13 06:38:10 +00:00
Georg Brandl 492f3fc272 Allow set literals in literal_eval(). 2010-07-11 09:41:21 +00:00
Georg Brandl 22b3431426 Merged revisions 74209 via svnmerge from
svn+ssh://pythondev@svn.python.org/python/trunk

........
  r74209 | georg.brandl | 2009-07-26 16:37:28 +0200 (So, 26 Jul 2009) | 1 line

  builtin -> built-in.
........
2009-07-26 14:54:51 +00:00
Georg Brandl 48310cd3f2 Remove trailing whitespace. 2009-01-03 21:18:54 +00:00
Georg Brandl 36ab1ef8ee Merged revisions 68219 via svnmerge from
svn+ssh://pythondev@svn.python.org/python/trunk

........
  r68219 | georg.brandl | 2009-01-03 21:47:01 +0100 (Sat, 03 Jan 2009) | 2 lines

  Fix uses of the default role.
........
2009-01-03 21:17:04 +00:00
Benjamin Peterson ec9199be08 Merged revisions 67162 via svnmerge from
svn+ssh://pythondev@svn.python.org/python/trunk

........
  r67162 | benjamin.peterson | 2008-11-08 10:55:33 -0600 (Sat, 08 Nov 2008) | 1 line

  a few compile() and ast doc improvements
........
2008-11-08 17:05:00 +00:00
Benjamin Peterson 9abf93d6c9 murder some versionadded and versionchanged directives in their beds 2008-08-16 02:59:55 +00:00
Benjamin Peterson dcf97b98ec Merged revisions 64434-64435,64440-64443,64445,64447-64448,64450,64452,64455,64461,64464,64466,64468 via svnmerge from
svn+ssh://pythondev@svn.python.org/python/trunk

........
  r64434 | andrew.kuchling | 2008-06-20 18:13:58 -0500 (Fri, 20 Jun 2008) | 1 line

  Remove request for e-mail; it's unlikely these classes will be saved
........
  r64435 | andrew.kuchling | 2008-06-20 18:14:32 -0500 (Fri, 20 Jun 2008) | 1 line

  Grammar fixes
........
  r64440 | andrew.kuchling | 2008-06-21 08:29:12 -0500 (Sat, 21 Jun 2008) | 1 line

  Docstring typo
........
  r64441 | andrew.kuchling | 2008-06-21 08:47:20 -0500 (Sat, 21 Jun 2008) | 1 line

  Use repr() for bad input strings; this makes the empty string or binary characters more visible
........
  r64442 | andrew.kuchling | 2008-06-21 08:48:38 -0500 (Sat, 21 Jun 2008) | 1 line

  Docstring correction
........
  r64443 | georg.brandl | 2008-06-21 09:26:19 -0500 (Sat, 21 Jun 2008) | 2 lines

  Documentation fix.
........
  r64445 | facundo.batista | 2008-06-21 12:30:06 -0500 (Sat, 21 Jun 2008) | 3 lines


  Reviewed and updated the documentation. Fixes #3017.
........
  r64447 | facundo.batista | 2008-06-21 13:58:04 -0500 (Sat, 21 Jun 2008) | 6 lines


  Now a from submitted via POST that also has a query string
  will contain both FieldStorage and MiniFieldStorage items.

  Fixes #1817.
........
  r64448 | facundo.batista | 2008-06-21 14:48:19 -0500 (Sat, 21 Jun 2008) | 5 lines


  In the deprecated functions I added an alert to review
  specially a section of the subprocess documentation
  that helps with the replacing of those functionss.
........
  r64450 | georg.brandl | 2008-06-22 04:05:29 -0500 (Sun, 22 Jun 2008) | 2 lines

  Turn section references into proper cross-references.
........
  r64452 | facundo.batista | 2008-06-22 08:36:20 -0500 (Sun, 22 Jun 2008) | 5 lines


  Issue #2722. Now the char buffer to support the path string has
  not fixed length, it mallocs memory if needed. As a result, we
  don't have a maximum for the getcwd() method.
........
  r64455 | facundo.batista | 2008-06-22 10:27:10 -0500 (Sun, 22 Jun 2008) | 4 lines


  Issue 3164. Small fix to don't repeat a comparation
  without necessity.
........
  r64461 | georg.brandl | 2008-06-22 13:11:52 -0500 (Sun, 22 Jun 2008) | 2 lines

  #3085: Fix syntax error.
........
  r64464 | georg.brandl | 2008-06-22 13:31:54 -0500 (Sun, 22 Jun 2008) | 2 lines

  Expand docstrings of sqlite3 functions.
........
  r64466 | georg.brandl | 2008-06-22 14:07:59 -0500 (Sun, 22 Jun 2008) | 2 lines

  Write out "phi" consistently.
........
  r64468 | facundo.batista | 2008-06-22 14:35:24 -0500 (Sun, 22 Jun 2008) | 4 lines


  Just returning nothing instead of rising TestSkipped, because
  it makes the test fail in the trunk.loewis-sun buildbot.
........
2008-07-02 17:30:14 +00:00
Amaury Forgeot d'Arc fdfe62d887 Merged revisions 64089,64098,64100-64102,64113,64115-64116,64118,64120,64132,64342 via svnmerge from
svn+ssh://pythondev@svn.python.org/python/trunk

........
  r64089 | armin.ronacher | 2008-06-10 22:37:02 +0200 (mar., 10 juin 2008) | 3 lines

  Fix a formatting error in the ast documentation.
........
  r64098 | raymond.hettinger | 2008-06-11 02:25:29 +0200 (mer., 11 juin 2008) | 6 lines

  Mini-PEP: Simplifying numbers.py
  * Convert binary methods in Integral to mixin methods
  * Remove three-arg __pow__ as a required method
  * Make __int__ the root method instead of __long__.
........
  r64100 | raymond.hettinger | 2008-06-11 02:28:51 +0200 (mer., 11 juin 2008) | 1 line

  Update numbers doc for the Integral simplification.
........
  r64101 | raymond.hettinger | 2008-06-11 02:44:47 +0200 (mer., 11 juin 2008) | 3 lines

  Handle the case with zero arguments.
........
  r64102 | benjamin.peterson | 2008-06-11 03:31:28 +0200 (mer., 11 juin 2008) | 4 lines

  convert test_struct to a unittest thanks to Giampaolo Rodola
  I had to disable one test because it was functioning incorrectly, see #1530559
  I also removed the debugging prints
........
  r64113 | thomas.heller | 2008-06-11 09:10:43 +0200 (mer., 11 juin 2008) | 2 lines

  Fix markup.
  Document the new 'offset' parameter for the 'ctypes.byref' function.
........
  r64115 | raymond.hettinger | 2008-06-11 12:30:54 +0200 (mer., 11 juin 2008) | 1 line

  Multi-arg form for set.difference() and set.difference_update().
........
  r64116 | raymond.hettinger | 2008-06-11 14:06:49 +0200 (mer., 11 juin 2008) | 1 line

  Issue 3051:  Let heapq work with either __lt__ or __le__.
........
  r64118 | raymond.hettinger | 2008-06-11 14:39:09 +0200 (mer., 11 juin 2008) | 1 line

  Optimize previous checkin for heapq.
........
  r64120 | raymond.hettinger | 2008-06-11 15:14:50 +0200 (mer., 11 juin 2008) | 1 line

  Add test for heapq using both __lt__ and __le__.
........
  r64132 | gregory.p.smith | 2008-06-11 20:00:52 +0200 (mer., 11 juin 2008) | 3 lines

  Correct an incorrect comment about our #include of stddef.h.
  (see Doug Evans' comment on python-dev 2008-06-10)
........
  r64342 | guido.van.rossum | 2008-06-17 19:38:02 +0200 (mar., 17 juin 2008) | 3 lines

  Roll back Raymond's -r64098 while we think of something better.
  (See issue 3056 -- we're close to a resolution but need unittests.)
........
2008-06-17 20:36:03 +00:00
Georg Brandl 0c77a82c3d Merged revisions 63829-63831,63858,63865,63879,63882,63948,63970-63972,63976,63989,64014-64015,64021-64022,64063-64065,64067 via svnmerge from
svn+ssh://pythondev@svn.python.org/python/trunk

........
  r63829 | mark.summerfield | 2008-05-31 15:05:34 +0200 (Sat, 31 May 2008) | 4 lines

  Added a note to [] that special forms & special chars lose their meaning
  and backrefs can't be used inside []
........
  r63830 | georg.brandl | 2008-05-31 16:40:09 +0200 (Sat, 31 May 2008) | 2 lines

  #3010: clarification about stdin/use_rawinput.
........
  r63831 | georg.brandl | 2008-05-31 16:45:55 +0200 (Sat, 31 May 2008) | 2 lines

  #3005: add explaining sentence to easydialogs docs.
........
  r63858 | georg.brandl | 2008-06-01 18:41:31 +0200 (Sun, 01 Jun 2008) | 2 lines

  Add plain text make target.
........
  r63865 | georg.brandl | 2008-06-01 21:24:36 +0200 (Sun, 01 Jun 2008) | 2 lines

  Spaces vs. tabs.
........
  r63879 | gregory.p.smith | 2008-06-02 00:57:47 +0200 (Mon, 02 Jun 2008) | 3 lines

  Make the _H #define's match the header file names.  Fix comments to
  mention the correct type names.
........
  r63882 | gregory.p.smith | 2008-06-02 01:48:47 +0200 (Mon, 02 Jun 2008) | 3 lines

  Adds a Thread.getIdent() method to provide the _get_ident() value for
  any given threading.Thread object.  feature request issue 2871.
........
  r63948 | alexandre.vassalotti | 2008-06-04 22:41:44 +0200 (Wed, 04 Jun 2008) | 2 lines

  Fixed complex.__getnewargs__() to not emit another complex object.
........
  r63970 | andrew.kuchling | 2008-06-06 01:33:54 +0200 (Fri, 06 Jun 2008) | 1 line

  Document 'utc' parameter
........
  r63971 | andrew.kuchling | 2008-06-06 01:35:31 +0200 (Fri, 06 Jun 2008) | 1 line

  Add various items
........
  r63972 | andrew.kuchling | 2008-06-06 01:35:48 +0200 (Fri, 06 Jun 2008) | 1 line

  Grammar fix
........
  r63976 | georg.brandl | 2008-06-06 09:34:50 +0200 (Fri, 06 Jun 2008) | 2 lines

  Markup fix.
........
  r63989 | thomas.heller | 2008-06-06 20:42:11 +0200 (Fri, 06 Jun 2008) | 2 lines

  Add a reminder for the maintainer of whatsnew.
........
  r64014 | georg.brandl | 2008-06-07 17:59:10 +0200 (Sat, 07 Jun 2008) | 3 lines

  Factor out docstring dedenting from inspect.getdoc() into inspect.cleandoc()
  to ease standalone use of the algorithm.
........
  r64015 | georg.brandl | 2008-06-07 18:04:01 +0200 (Sat, 07 Jun 2008) | 2 lines

  Revert unwanted changes.
........
  r64021 | georg.brandl | 2008-06-07 20:16:12 +0200 (Sat, 07 Jun 2008) | 2 lines

  X-ref to numbers module.
........
  r64022 | georg.brandl | 2008-06-07 20:17:37 +0200 (Sat, 07 Jun 2008) | 3 lines

  Document the "st" API, to avoid confusion with the "new" AST.
  Add a note about using the new AST module.
........
  r64063 | martin.v.loewis | 2008-06-10 07:03:35 +0200 (Tue, 10 Jun 2008) | 2 lines

  Add Gregor Lingl.
........
  r64064 | georg.brandl | 2008-06-10 09:45:28 +0200 (Tue, 10 Jun 2008) | 2 lines

  Add the "ast" module, containing helpers to ease use of the "_ast" classes.
........
  r64065 | raymond.hettinger | 2008-06-10 09:57:15 +0200 (Tue, 10 Jun 2008) | 1 line

  Add Arnaud for his efforts on multi-arg set operations.
........
  r64067 | georg.brandl | 2008-06-10 14:46:39 +0200 (Tue, 10 Jun 2008) | 2 lines

  #2536: fix itertools.permutations and itertools.combinations docstrings.
........
2008-06-10 16:37:50 +00:00