Commit Graph

27 Commits

Author SHA1 Message Date
Guido van Rossum 495da29225 bpo-35975: Support parsing earlier minor versions of Python 3 (GH-12086)
This adds a `feature_version` flag to `ast.parse()` (documented) and `compile()` (hidden) that allow tweaking the parser to support older versions of the grammar. In particular if `feature_version` is 5 or 6, the hacks for the `async` and `await` keyword from PEP 492 are reinstated. (For 7 or higher, these are unconditionally treated as keywords, but they are still special tokens rather than `NAME` tokens that the parser driver recognizes.)



https://bugs.python.org/issue35975
2019-03-07 12:38:08 -08:00
Guido van Rossum dcfcd146f8 bpo-35766: Merge typed_ast back into CPython (GH-11645) 2019-01-31 12:40:27 +01:00
Ivan Levkivskyi 9932a22897
bpo-33416: Add end positions to Python AST (GH-11605)
The majority of this PR is tediously passing `end_lineno` and `end_col_offset` everywhere. Here are non-trivial points:
* It is not possible to reconstruct end positions in AST "on the fly", some information is lost after an AST node is constructed, so we need two more attributes for every AST node `end_lineno` and `end_col_offset`.
* I add end position information to both CST and AST.  Although it may be technically possible to avoid adding end positions to CST, the code becomes more cumbersome and less efficient.
* Since the end position is not known for non-leaf CST nodes while the next token is added, this requires a bit of extra care (see `_PyNode_FinalizeEndPos`). Unless I made some mistake, the algorithm should be linear.
* For statements, I "trim" the end position of suites to not include the terminal newlines and dedent (this seems to be what people would expect), for example in
  ```python
  class C:
      pass

  pass
  ```
  the end line and end column for the class definition is (2, 8).
* For `end_col_offset` I use the common Python convention for indexing, for example for `pass` the `end_col_offset` is 4 (not 3), so that `[0:4]` gives one the source code that corresponds to the node.
* I added a helper function `ast.get_source_segment()`, to get source text segment corresponding to a given AST node. It is also useful for testing.

An (inevitable) downside of this PR is that AST now takes almost 25% more memory. I think however it is probably justified by the benefits.
2019-01-22 11:18:22 +00:00
Anthony Sottile 7417622617 bpo-35733: Make isinstance(ast.Constant(boolean), ast.Num) be false. (GH-11547) 2019-01-18 21:30:28 +02:00
Serhiy Storchaka 6015cc50bc
bpo-32892: Support subclasses of base types in isinstance checks for AST constants. (GH-9934)
Some projects (e.g. Chameleon) create ast.Str containing an instance
of the str subclass.
2018-10-28 13:43:03 +02:00
Serhiy Storchaka 3f22811fef
bpo-32892: Use ast.Constant instead of specific constant AST types. (GH-9445) 2018-09-27 17:42:37 +03:00
Serhiy Storchaka 08f127a3ca
bpo-33851: Fix ast.get_docstring() for a node that lacks a docstring. (GH-7682) 2018-06-15 11:05:15 +03:00
Serhiy Storchaka 73cbe7a01a
bpo-32911: Revert bpo-29463. (GH-7121) (GH-7197)
Remove the docstring attribute of AST types and restore docstring
expression as a first stmt in their body.

Co-authored-by: INADA Naoki <methane@users.noreply.github.com>
2018-05-29 12:04:55 +03:00
Serhiy Storchaka d8ac4d1d5a
bpo-31778: Make ast.literal_eval() more strict. (#4035)
Addition and subtraction of arbitrary numbers no longer allowed.
2018-01-04 11:15:39 +02:00
Matthias Bussonnier 41cea70aa3 bpo-29637: clean docstring only if not None (GH-267) 2017-02-24 15:44:19 +09:00
INADA Naoki cb41b2766d bpo-29463: Add docstring field to some AST nodes. (#46)
* bpo-29463: Add docstring field to some AST nodes.

ClassDef, ModuleDef, FunctionDef, and AsyncFunctionDef has docstring
field for now.  It was first statement of there body.

* fix document.  thanks travis!

* doc fixes
2017-02-22 16:31:59 +01:00
Victor Stinner f2c1aa1661 Add ast.Constant
Issue #26146: Add a new kind of AST node: ast.Constant. It can be used by
external AST optimizers, but the compiler does not emit directly such node.

An optimizer can replace the following AST nodes with ast.Constant:

* ast.NameConstant: None, False, True
* ast.Num: int, float, complex
* ast.Str: str
* ast.Bytes: bytes
* ast.Tuple if items are constants too: tuple
* frozenset

Update code to accept ast.Constant instead of ast.Num and/or ast.Str:

* compiler
* docstrings
* ast.literal_eval()
* Tools/parser/unparse.py
2016-01-26 00:40:57 +01:00
Victor Stinner 6fbd7741db Issue #25670: Remove duplicate getattr() from ast.NodeTransformer 2015-11-20 08:49:58 +01:00
Yury Selivanov 2f07a66ded Issue #24688: ast.get_docstring() for 'async def' functions. 2015-07-23 08:54:35 +03:00
Benjamin Peterson 442f20996d create NameConstant AST class for None, True, and False literals (closes #16619) 2012-12-06 17:41:04 -05:00
Benjamin Peterson 0bd152cd67 remove ast.__version__ (closes #12273) 2011-07-15 21:10:13 -05:00
Éric Araujo 2a83cc61d6 Add missing types to docstring of ast.literal_eval.
The reST doc was updated but not the docstring.
2011-04-17 19:10:27 +02:00
Terry Reedy feac624827 Issue #11000 ast.parse parses source, not just expressions. 2011-01-24 21:36:03 +00:00
Georg Brandl 619e7ba814 #10869: do not visit root node twice in ast.increment_lineno(). 2011-01-09 07:38:51 +00:00
Raymond Hettinger bc95973b51 Fix handling on negative numbers in ast.literal_eval(). 2010-10-08 00:47:45 +00:00
Florent Xicluna e01de8f2f3 remove pointless coding cookies 2010-08-30 14:05:50 +00:00
Benjamin Peterson 5ef96e5fac allow byte literals 2010-07-11 23:06:06 +00:00
Georg Brandl 492f3fc272 Allow set literals in literal_eval(). 2010-07-11 09:41:21 +00:00
Benjamin Peterson 058e31ef2a Merged revisions 68450,68480-68481,68493,68495,68501,68512,68514-68515,68534-68536,68552,68563,68570-68572,68575,68582,68596,68623-68624,68628 via svnmerge from
svn+ssh://pythondev@svn.python.org/python/trunk

........
  r68450 | jeffrey.yasskin | 2009-01-09 10:47:07 -0600 (Fri, 09 Jan 2009) | 3 lines

  Fix issue 4884, preventing a crash in the socket code when python is compiled
  with llvm-gcc and run with a glibc <2.10.
........
  r68480 | vinay.sajip | 2009-01-10 07:38:04 -0600 (Sat, 10 Jan 2009) | 1 line

  Minor documentation changes cross-referencing NullHandler to the documentation on configuring logging in a library.
........
  r68481 | vinay.sajip | 2009-01-10 07:42:04 -0600 (Sat, 10 Jan 2009) | 1 line

  Corrected an incorrect self-reference.
........
  r68493 | benjamin.peterson | 2009-01-10 11:18:55 -0600 (Sat, 10 Jan 2009) | 1 line

  rewrite verbose conditionals
........
  r68495 | benjamin.peterson | 2009-01-10 11:36:44 -0600 (Sat, 10 Jan 2009) | 1 line

  tp_iter only exists with Py_TPFLAGS_HAVE_ITER #4901
........
  r68501 | vinay.sajip | 2009-01-10 13:22:57 -0600 (Sat, 10 Jan 2009) | 1 line

  Corrected minor typo and added .currentmodule directives to fix missing cross-references.
........
  r68512 | benjamin.peterson | 2009-01-10 16:42:10 -0600 (Sat, 10 Jan 2009) | 1 line

  make tests fail if they can't be imported
........
  r68514 | benjamin.peterson | 2009-01-10 17:41:59 -0600 (Sat, 10 Jan 2009) | 1 line

  move seealso to a more appropiate place
........
  r68515 | benjamin.peterson | 2009-01-10 17:49:08 -0600 (Sat, 10 Jan 2009) | 1 line

  macos 9 isn't supported
........
  r68534 | gregory.p.smith | 2009-01-11 11:53:33 -0600 (Sun, 11 Jan 2009) | 2 lines

  correct email address
........
  r68535 | gregory.p.smith | 2009-01-11 11:57:54 -0600 (Sun, 11 Jan 2009) | 9 lines

  Update the documentation for binascii and zlib crc32/adler32 functions
  to better describe the signed vs unsigned return value behavior on
  different platforms and versions of python.  Mention the workaround to
  make them all return the same thing by using & 0xffffffff.

  Fixes issue4903.

  Also needs to be merged into release26-maint, release30-maint, & py3k.
........
  r68536 | benjamin.peterson | 2009-01-11 13:48:15 -0600 (Sun, 11 Jan 2009) | 1 line

  add email addresses
........
  r68552 | vinay.sajip | 2009-01-12 14:36:18 -0600 (Mon, 12 Jan 2009) | 1 line

  Minor changes/corrections in markup.
........
  r68563 | benjamin.peterson | 2009-01-12 19:49:10 -0600 (Mon, 12 Jan 2009) | 1 line

  small logic correction
........
  r68570 | raymond.hettinger | 2009-01-13 03:08:32 -0600 (Tue, 13 Jan 2009) | 5 lines

  Issue 4922: Incorrect comments for MutableSet.add() and MutableSet.discard().

  Needs to be backported to 2.6 and forward ported to 3.0 and 3.1.
........
  r68571 | armin.ronacher | 2009-01-13 05:52:23 -0600 (Tue, 13 Jan 2009) | 3 lines

  ast.literal_eval can properly evaluate complex numbers now.  This fixes issue4907.
........
  r68572 | andrew.kuchling | 2009-01-13 07:40:54 -0600 (Tue, 13 Jan 2009) | 1 line

  Note that first coord. is left alone
........
  r68575 | thomas.heller | 2009-01-13 11:32:28 -0600 (Tue, 13 Jan 2009) | 1 line

  Fix refcount leak in error cases.  Bug found by coverity.
........
  r68582 | georg.brandl | 2009-01-13 16:14:01 -0600 (Tue, 13 Jan 2009) | 2 lines

  Use assertRaises.
........
  r68596 | amaury.forgeotdarc | 2009-01-13 17:39:22 -0600 (Tue, 13 Jan 2009) | 3 lines

  #1162154: inspect.getmembers() now skips attributes that raise AttributeError,
  e.g. a __slots__ attribute which has not been set.
........
  r68623 | vinay.sajip | 2009-01-15 16:48:13 -0600 (Thu, 15 Jan 2009) | 1 line

  Made minor changes/corrections in markup. Added a couple of section headings.
........
  r68624 | vinay.sajip | 2009-01-15 17:04:47 -0600 (Thu, 15 Jan 2009) | 1 line

  Minor changes/corrections in markup.
........
  r68628 | benjamin.peterson | 2009-01-15 20:55:24 -0600 (Thu, 15 Jan 2009) | 1 line

  compare with == not is #4946
........
2009-01-16 03:54:08 +00:00
Benjamin Peterson 1a6e0d0831 Merged revisions 66974,66977,66984,66989,66992,66994-66996,66998-67000,67007,67015 via svnmerge from
svn+ssh://pythondev@svn.python.org/python/trunk

........
  r66974 | benjamin.peterson | 2008-10-19 08:59:01 -0500 (Sun, 19 Oct 2008) | 1 line

  fix compiler warning
........
  r66977 | benjamin.peterson | 2008-10-19 14:39:16 -0500 (Sun, 19 Oct 2008) | 1 line

  mention -n
........
  r66984 | armin.ronacher | 2008-10-20 16:29:08 -0500 (Mon, 20 Oct 2008) | 3 lines

  Fixed #4062, added import for _ast.__version__ to ast to match the documented behavior.
........
  r66989 | matthias.klose | 2008-10-21 04:12:25 -0500 (Tue, 21 Oct 2008) | 2 lines

  - install versioned manpage
........
  r66992 | benjamin.peterson | 2008-10-21 15:51:13 -0500 (Tue, 21 Oct 2008) | 1 line

  make sure to call iteritems()
........
  r66994 | amaury.forgeotdarc | 2008-10-21 17:01:38 -0500 (Tue, 21 Oct 2008) | 6 lines

  #4157 move two test functions out of platform.py.

  Turn them into unit tests, and correct an obvious typo:
      (("a", "b") ("c", "d") ("e", "f"))
  compiles even with the missing commas, but does not execute very well...
........
  r66995 | benjamin.peterson | 2008-10-21 17:18:29 -0500 (Tue, 21 Oct 2008) | 1 line

  return ArgInfo from inspect.getargvalues #4092
........
  r66996 | benjamin.peterson | 2008-10-21 17:20:31 -0500 (Tue, 21 Oct 2008) | 1 line

  add NEWs note for last change
........
  r66998 | benjamin.peterson | 2008-10-22 15:57:43 -0500 (Wed, 22 Oct 2008) | 1 line

  fix a few typos
........
  r66999 | benjamin.peterson | 2008-10-22 16:05:30 -0500 (Wed, 22 Oct 2008) | 1 line

  and another typo...
........
  r67000 | benjamin.peterson | 2008-10-22 16:16:34 -0500 (Wed, 22 Oct 2008) | 1 line

  fix #4150: pdb's up command didn't work for generators in post-mortem
........
  r67007 | benjamin.peterson | 2008-10-23 16:43:48 -0500 (Thu, 23 Oct 2008) | 1 line

  only nonempty __slots__ don't work
........
  r67015 | georg.brandl | 2008-10-25 02:00:52 -0500 (Sat, 25 Oct 2008) | 2 lines

  Typo fix.
........
2008-10-25 15:49:17 +00:00
Benjamin Peterson dcf97b98ec Merged revisions 64434-64435,64440-64443,64445,64447-64448,64450,64452,64455,64461,64464,64466,64468 via svnmerge from
svn+ssh://pythondev@svn.python.org/python/trunk

........
  r64434 | andrew.kuchling | 2008-06-20 18:13:58 -0500 (Fri, 20 Jun 2008) | 1 line

  Remove request for e-mail; it's unlikely these classes will be saved
........
  r64435 | andrew.kuchling | 2008-06-20 18:14:32 -0500 (Fri, 20 Jun 2008) | 1 line

  Grammar fixes
........
  r64440 | andrew.kuchling | 2008-06-21 08:29:12 -0500 (Sat, 21 Jun 2008) | 1 line

  Docstring typo
........
  r64441 | andrew.kuchling | 2008-06-21 08:47:20 -0500 (Sat, 21 Jun 2008) | 1 line

  Use repr() for bad input strings; this makes the empty string or binary characters more visible
........
  r64442 | andrew.kuchling | 2008-06-21 08:48:38 -0500 (Sat, 21 Jun 2008) | 1 line

  Docstring correction
........
  r64443 | georg.brandl | 2008-06-21 09:26:19 -0500 (Sat, 21 Jun 2008) | 2 lines

  Documentation fix.
........
  r64445 | facundo.batista | 2008-06-21 12:30:06 -0500 (Sat, 21 Jun 2008) | 3 lines


  Reviewed and updated the documentation. Fixes #3017.
........
  r64447 | facundo.batista | 2008-06-21 13:58:04 -0500 (Sat, 21 Jun 2008) | 6 lines


  Now a from submitted via POST that also has a query string
  will contain both FieldStorage and MiniFieldStorage items.

  Fixes #1817.
........
  r64448 | facundo.batista | 2008-06-21 14:48:19 -0500 (Sat, 21 Jun 2008) | 5 lines


  In the deprecated functions I added an alert to review
  specially a section of the subprocess documentation
  that helps with the replacing of those functionss.
........
  r64450 | georg.brandl | 2008-06-22 04:05:29 -0500 (Sun, 22 Jun 2008) | 2 lines

  Turn section references into proper cross-references.
........
  r64452 | facundo.batista | 2008-06-22 08:36:20 -0500 (Sun, 22 Jun 2008) | 5 lines


  Issue #2722. Now the char buffer to support the path string has
  not fixed length, it mallocs memory if needed. As a result, we
  don't have a maximum for the getcwd() method.
........
  r64455 | facundo.batista | 2008-06-22 10:27:10 -0500 (Sun, 22 Jun 2008) | 4 lines


  Issue 3164. Small fix to don't repeat a comparation
  without necessity.
........
  r64461 | georg.brandl | 2008-06-22 13:11:52 -0500 (Sun, 22 Jun 2008) | 2 lines

  #3085: Fix syntax error.
........
  r64464 | georg.brandl | 2008-06-22 13:31:54 -0500 (Sun, 22 Jun 2008) | 2 lines

  Expand docstrings of sqlite3 functions.
........
  r64466 | georg.brandl | 2008-06-22 14:07:59 -0500 (Sun, 22 Jun 2008) | 2 lines

  Write out "phi" consistently.
........
  r64468 | facundo.batista | 2008-06-22 14:35:24 -0500 (Sun, 22 Jun 2008) | 4 lines


  Just returning nothing instead of rising TestSkipped, because
  it makes the test fail in the trunk.loewis-sun buildbot.
........
2008-07-02 17:30:14 +00:00
Georg Brandl 0c77a82c3d Merged revisions 63829-63831,63858,63865,63879,63882,63948,63970-63972,63976,63989,64014-64015,64021-64022,64063-64065,64067 via svnmerge from
svn+ssh://pythondev@svn.python.org/python/trunk

........
  r63829 | mark.summerfield | 2008-05-31 15:05:34 +0200 (Sat, 31 May 2008) | 4 lines

  Added a note to [] that special forms & special chars lose their meaning
  and backrefs can't be used inside []
........
  r63830 | georg.brandl | 2008-05-31 16:40:09 +0200 (Sat, 31 May 2008) | 2 lines

  #3010: clarification about stdin/use_rawinput.
........
  r63831 | georg.brandl | 2008-05-31 16:45:55 +0200 (Sat, 31 May 2008) | 2 lines

  #3005: add explaining sentence to easydialogs docs.
........
  r63858 | georg.brandl | 2008-06-01 18:41:31 +0200 (Sun, 01 Jun 2008) | 2 lines

  Add plain text make target.
........
  r63865 | georg.brandl | 2008-06-01 21:24:36 +0200 (Sun, 01 Jun 2008) | 2 lines

  Spaces vs. tabs.
........
  r63879 | gregory.p.smith | 2008-06-02 00:57:47 +0200 (Mon, 02 Jun 2008) | 3 lines

  Make the _H #define's match the header file names.  Fix comments to
  mention the correct type names.
........
  r63882 | gregory.p.smith | 2008-06-02 01:48:47 +0200 (Mon, 02 Jun 2008) | 3 lines

  Adds a Thread.getIdent() method to provide the _get_ident() value for
  any given threading.Thread object.  feature request issue 2871.
........
  r63948 | alexandre.vassalotti | 2008-06-04 22:41:44 +0200 (Wed, 04 Jun 2008) | 2 lines

  Fixed complex.__getnewargs__() to not emit another complex object.
........
  r63970 | andrew.kuchling | 2008-06-06 01:33:54 +0200 (Fri, 06 Jun 2008) | 1 line

  Document 'utc' parameter
........
  r63971 | andrew.kuchling | 2008-06-06 01:35:31 +0200 (Fri, 06 Jun 2008) | 1 line

  Add various items
........
  r63972 | andrew.kuchling | 2008-06-06 01:35:48 +0200 (Fri, 06 Jun 2008) | 1 line

  Grammar fix
........
  r63976 | georg.brandl | 2008-06-06 09:34:50 +0200 (Fri, 06 Jun 2008) | 2 lines

  Markup fix.
........
  r63989 | thomas.heller | 2008-06-06 20:42:11 +0200 (Fri, 06 Jun 2008) | 2 lines

  Add a reminder for the maintainer of whatsnew.
........
  r64014 | georg.brandl | 2008-06-07 17:59:10 +0200 (Sat, 07 Jun 2008) | 3 lines

  Factor out docstring dedenting from inspect.getdoc() into inspect.cleandoc()
  to ease standalone use of the algorithm.
........
  r64015 | georg.brandl | 2008-06-07 18:04:01 +0200 (Sat, 07 Jun 2008) | 2 lines

  Revert unwanted changes.
........
  r64021 | georg.brandl | 2008-06-07 20:16:12 +0200 (Sat, 07 Jun 2008) | 2 lines

  X-ref to numbers module.
........
  r64022 | georg.brandl | 2008-06-07 20:17:37 +0200 (Sat, 07 Jun 2008) | 3 lines

  Document the "st" API, to avoid confusion with the "new" AST.
  Add a note about using the new AST module.
........
  r64063 | martin.v.loewis | 2008-06-10 07:03:35 +0200 (Tue, 10 Jun 2008) | 2 lines

  Add Gregor Lingl.
........
  r64064 | georg.brandl | 2008-06-10 09:45:28 +0200 (Tue, 10 Jun 2008) | 2 lines

  Add the "ast" module, containing helpers to ease use of the "_ast" classes.
........
  r64065 | raymond.hettinger | 2008-06-10 09:57:15 +0200 (Tue, 10 Jun 2008) | 1 line

  Add Arnaud for his efforts on multi-arg set operations.
........
  r64067 | georg.brandl | 2008-06-10 14:46:39 +0200 (Tue, 10 Jun 2008) | 2 lines

  #2536: fix itertools.permutations and itertools.combinations docstrings.
........
2008-06-10 16:37:50 +00:00