Commit Graph

107 Commits

Author SHA1 Message Date
Guido van Rossum dcfcd146f8 bpo-35766: Merge typed_ast back into CPython (GH-11645) 2019-01-31 12:40:27 +01:00
Ivan Levkivskyi 9932a22897
bpo-33416: Add end positions to Python AST (GH-11605)
The majority of this PR is tediously passing `end_lineno` and `end_col_offset` everywhere. Here are non-trivial points:
* It is not possible to reconstruct end positions in AST "on the fly", some information is lost after an AST node is constructed, so we need two more attributes for every AST node `end_lineno` and `end_col_offset`.
* I add end position information to both CST and AST.  Although it may be technically possible to avoid adding end positions to CST, the code becomes more cumbersome and less efficient.
* Since the end position is not known for non-leaf CST nodes while the next token is added, this requires a bit of extra care (see `_PyNode_FinalizeEndPos`). Unless I made some mistake, the algorithm should be linear.
* For statements, I "trim" the end position of suites to not include the terminal newlines and dedent (this seems to be what people would expect), for example in
  ```python
  class C:
      pass

  pass
  ```
  the end line and end column for the class definition is (2, 8).
* For `end_col_offset` I use the common Python convention for indexing, for example for `pass` the `end_col_offset` is 4 (not 3), so that `[0:4]` gives one the source code that corresponds to the node.
* I added a helper function `ast.get_source_segment()`, to get source text segment corresponding to a given AST node. It is also useful for testing.

An (inevitable) downside of this PR is that AST now takes almost 25% more memory. I think however it is probably justified by the benefits.
2019-01-22 11:18:22 +00:00
Anthony Sottile 7417622617 bpo-35733: Make isinstance(ast.Constant(boolean), ast.Num) be false. (GH-11547) 2019-01-18 21:30:28 +02:00
Serhiy Storchaka 3bcbedc9f1
bpo-34850: Emit a warning for "is" and "is not" with a literal. (GH-9642) 2019-01-18 07:47:48 +02:00
Anthony Sottile 995d9b9297 bpo-16806: Fix `lineno` and `col_offset` for multi-line string tokens (GH-10021) 2019-01-13 13:05:13 +09:00
Serhiy Storchaka b619b09792
bpo-31241: Fix AST node position for list and generator comprehensions. (GH-10633)
The lineno and col_offset attributes of AST nodes for list comprehensions,
generator expressions and tuples are now point to the opening parenthesis or
square brace. For tuples without parenthesis they point to the position
of the first item.
2018-11-27 09:40:29 +02:00
Serhiy Storchaka 95b6acf951
bpo-34876: Change the lineno of the AST for decorated function and class. (GH-9731)
It was overridden by the lineno of the first decorator. Now it is
the lineno of 'def' or 'class'.
2018-10-30 13:16:02 +02:00
Serhiy Storchaka 6015cc50bc
bpo-32892: Support subclasses of base types in isinstance checks for AST constants. (GH-9934)
Some projects (e.g. Chameleon) create ast.Str containing an instance
of the str subclass.
2018-10-28 13:43:03 +02:00
Serhiy Storchaka 3f22811fef
bpo-32892: Use ast.Constant instead of specific constant AST types. (GH-9445) 2018-09-27 17:42:37 +03:00
guoci 90fc8980bb closes bpo-31902: Fix the col_offset attribute for ast.Async* nodes to point to the "async" keyword. (GH-4175)
Previously, col_offset points to the keyword after "async".
2018-09-11 14:45:45 -07:00
Serhiy Storchaka 08f127a3ca
bpo-33851: Fix ast.get_docstring() for a node that lacks a docstring. (GH-7682) 2018-06-15 11:05:15 +03:00
Serhiy Storchaka 73cbe7a01a
bpo-32911: Revert bpo-29463. (GH-7121) (GH-7197)
Remove the docstring attribute of AST types and restore docstring
expression as a first stmt in their body.

Co-authored-by: INADA Naoki <methane@users.noreply.github.com>
2018-05-29 12:04:55 +03:00
Serhiy Storchaka d8ac4d1d5a
bpo-31778: Make ast.literal_eval() more strict. (#4035)
Addition and subtraction of arbitrary numbers no longer allowed.
2018-01-04 11:15:39 +02:00
Oren Milman 7dc46d8cf5 bpo-31592: Fix an assertion failure in Python parser in case of a bad unicodedata.normalize(). (#3767) 2017-09-30 20:16:24 +03:00
Matthias Bussonnier 41cea70aa3 bpo-29637: clean docstring only if not None (GH-267) 2017-02-24 15:44:19 +09:00
INADA Naoki 4c78c527d2 bpo-29622: Make AST constructor to accept less than enough number of positional arguments (GH-249)
bpo-29463 added optional "docstring" field to 4 AST types.
While it is optional, it breaks backward compatibility because AST constructor
requires number of positional argument is same to number of fields.

AST types accepts empty arguments, and incomplete keyword arguments.
But it's not big problem because field can be filled after creation, and checked when compiling.
So stop requiring complete set of fields for positional arguments too.
2017-02-24 02:48:17 +09:00
INADA Naoki cb41b2766d bpo-29463: Add docstring field to some AST nodes. (#46)
* bpo-29463: Add docstring field to some AST nodes.

ClassDef, ModuleDef, FunctionDef, and AsyncFunctionDef has docstring
field for now.  It was first statement of there body.

* fix document.  thanks travis!

* doc fixes
2017-02-22 16:31:59 +01:00
Yury Selivanov 52c4e7cc84 Issue #28008: Implement PEP 530 -- asynchronous comprehensions. 2016-09-09 10:36:01 -07:00
Serhiy Storchaka 7de2840508 Issue #27352: Fixed an error message in a test. 2016-06-27 23:40:43 +03:00
Berker Peksag 0a5bd51dd3 Issue #13436: Add a test to make sure that ast.ImportFrom(level=None) works 2016-04-29 19:50:02 +03:00
Victor Stinner 15a3095d64 compiler: don't emit SyntaxWarning on const stmt
Issue #26204: the compiler doesn't emit SyntaxWarning warnings anymore when
constant statements are ignored.
2016-02-08 22:45:06 +01:00
Victor Stinner a2724095cd compiler now ignores constant statements
The compile ignores constant statements and emit a SyntaxWarning warning.

Don't emit the warning for string statement because triple quoted string is a
common syntax for multiline comments.

Don't emit the warning on ellipis neither: 'def f(): ...' is a legit syntax for
abstract functions.

Changes:

* test_ast: ignore SyntaxWarning when compiling test statements. Modify
  test_load_const() to use assignment expressions rather than constant
  expression.
* test_code: add more kinds of constant statements, ignore SyntaxWarning when
  testing that the compiler removes constant statements.
* test_grammar: ignore SyntaxWarning on the statement "1"
2016-02-08 18:17:58 +01:00
Victor Stinner f089196beb Simplify main() of test_ast
* Use ast.parse() to get the AST for a statement
* Use str%args syntax for format a line

Issue #26204.
2016-02-08 17:15:21 +01:00
Victor Stinner be59d1489b Issue #26146: enhance ast.Constant error message
Mention the name of the invalid type in error message of AST validation for
constants.

Suggestion made by Joseph Jevnik on a review.
2016-01-27 00:39:12 +01:00
Victor Stinner f2c1aa1661 Add ast.Constant
Issue #26146: Add a new kind of AST node: ast.Constant. It can be used by
external AST optimizers, but the compiler does not emit directly such node.

An optimizer can replace the following AST nodes with ast.Constant:

* ast.NameConstant: None, False, True
* ast.Num: int, float, complex
* ast.Str: str
* ast.Bytes: bytes
* ast.Tuple if items are constants too: tuple
* frozenset

Update code to accept ast.Constant instead of ast.Num and/or ast.Str:

* compiler
* docstrings
* ast.literal_eval()
* Tools/parser/unparse.py
2016-01-26 00:40:57 +01:00
Benjamin Peterson 58b53953f8 make opening brace of container literals and comprehensions correspond to the line number and col offset of the AST node (closes #25131) 2015-09-25 22:44:43 -07:00
Yury Selivanov b3d531348c Issue #24975: Fix AST compilation for PEP 448 syntax. 2015-09-01 16:10:49 -04:00
Yury Selivanov 2f07a66ded Issue #24688: ast.get_docstring() for 'async def' functions. 2015-07-23 08:54:35 +03:00
Yury Selivanov 7544508f02 PEP 0492 -- Coroutines with async and await syntax. Issue #24017. 2015-05-11 22:57:16 -04:00
Benjamin Peterson 025e9ebd0a PEP 448: additional unpacking generalizations (closes #2292)
Patch by Neil Girdhar.
2015-05-05 20:16:41 -04:00
Benjamin Peterson 7a66fc22ad revert lineno and col_offset changes from #16795 (closes #21295) 2015-02-02 10:51:20 -05:00
Benjamin Peterson e84fde981d set line and column numbers for keyword-only arg nodes (closes #20619) 2014-02-13 19:22:14 -05:00
Ezio Melotti a7e7497d88 #18466: merge with 3.3. 2013-08-17 16:58:13 +03:00
Ezio Melotti 85a8629d21 #18466: fix more typos. Patch by Févry Thibault. 2013-08-17 16:57:41 +03:00
Brett Cannon 3e9a9ae09d Update various test modules to use unittest.main() for test discovery
instead of manually listing tests for test.support.run_unittest().
2013-06-12 21:25:59 -04:00
Victor Stinner d502a07fac test_ast: pass the filename to ast.parse() 2013-03-22 00:06:20 +01:00
Benjamin Peterson cda75be02a unify some ast.argument's attrs; change Attribute column offset (closes #16795)
Patch from Sven Brauch.
2013-03-18 10:48:58 -07:00
Benjamin Peterson 442f20996d create NameConstant AST class for None, True, and False literals (closes #16619) 2012-12-06 17:41:04 -05:00
Mark Dickinson ded35aeb9d Issue #16546: make ast.YieldFrom argument mandatory. 2012-11-25 14:36:26 +00:00
Benjamin Peterson bd0df50fb6 get rid of ast_error_finish by passing the compiling struct to ast_error 2012-09-02 15:04:51 -04:00
Benjamin Peterson c7f447c89f merge 3.2 (#15846) 2012-09-02 14:24:44 -04:00
Benjamin Peterson 2e2c903700 prevert ast errors from being normalized before ast_error_finish is called (closes #15846) 2012-09-02 14:23:15 -04:00
Benjamin Peterson 9ed37433c4 test AST base type garbage collection 2012-07-08 11:13:36 -07:00
Benjamin Peterson 98ba753432 merge 3.2 (#14378) 2012-03-22 08:19:50 -04:00
Benjamin Peterson a4e4e35783 check by equality for __future__ not identity (closes #14378) 2012-03-22 08:19:04 -04:00
Benjamin Peterson 7e0dbfbbde give the AST class a __dict__ 2012-03-12 09:46:44 -07:00
Benjamin Peterson 61a4161de4 merge 3.2 2012-03-12 09:43:27 -07:00
Benjamin Peterson ef87f06798 allow AST objects to be like its subclasses 2012-03-12 09:43:09 -07:00
Benjamin Peterson 09b4bd03f6 merge 3.2 2012-03-12 09:28:36 -07:00
Benjamin Peterson d279e286f2 every other ast object has a dict, so I think AST should, too 2012-03-12 09:27:36 -07:00