Commit Graph

142 Commits

Author SHA1 Message Date
Pablo Galindo 55e0836849
[3.9] bpo-41631: _ast module uses again a global state (GH-21961) (GH-22258)
Partially revert commit ac46eb4ad6662cf6d771b20d8963658b2186c48c:
"bpo-38113: Update the Python-ast.c generator to PEP384 (gh-15957)".

Using a module state per module instance is causing subtle practical
problems.

For example, the Mercurial project replaces the __import__() function
to implement lazy import, whereas Python expected that "import _ast"
always return a fully initialized _ast module.

Add _PyAST_Fini() to clear the state at exit.

The _ast module has no state (set _astmodule.m_size to 0). Remove
astmodule_traverse(), astmodule_clear() and astmodule_free()
functions..
(cherry picked from commit e5fbe0cbd4)

Co-authored-by: Victor Stinner <vstinner@python.org>
2020-09-15 20:32:56 +02:00
Miss Islington (bot) a1320989f5
bpo-40726: handle uninitalized end_lineno on ast.increment_lineno (GH-20312)
(cherry picked from commit 8f4380d2f5)

Co-authored-by: Batuhan Taskaya <batuhanosmantaskaya@gmail.com>
2020-08-05 06:52:29 -07:00
Miss Islington (bot) 90ee51f1cd
bpo-40870: Invalidate usage of some constants with ast.Name (GH-20649)
(cherry picked from commit 68874a8502)

Co-authored-by: Batuhan Taskaya <isidentical@gmail.com>
2020-06-06 10:04:38 -07:00
Pablo Galindo 9b83829e7d
[3.9] bpo-40614: Respect feature version for f-string debug expressions (GH-20196) (GH-20464)
Co-authored-by: Lysandros Nikolaou <lisandrosnik@gmail.com>
Co-authored-by: Pablo Galindo <pablogsal@gmail.com>
(cherry picked from commit c116c94)

Co-authored-by: Shantanu <hauntsaninja@users.noreply.github.com>
2020-05-27 22:01:11 +01:00
Miss Islington (bot) 1a4e9e6f35
bpo-36290: Fix keytword collision handling in AST node constructors (GH-12382)
(cherry picked from commit c73914a562)

Co-authored-by: Rémi Lapeyre <remi.lapeyre@lenstra.fr>
2020-05-24 14:32:32 -07:00
Irit Katriel e6578a226d
bpo-40662: Fixed ast.get_source_segment for ast nodes that have incomplete location information (GH-20157)
Co-authored-by: Pablo Galindo <Pablogsal@gmail.com>
2020-05-18 19:14:12 +01:00
Batuhan Taskaya 091951a67c
bpo-40528: Improve and clear several aspects of the ASDL definition code for the AST (GH-19952) 2020-05-06 15:29:32 +01:00
Curtis Bucher c21c51235a
bpo-40355: Improve error messages in ast.literal_eval with malformed Dict nodes (GH-19868)
Co-authored-by: Pablo Galindo <Pablogsal@gmail.com>
2020-05-05 20:40:56 +01:00
Pablo Galindo 40cf35c5b0
bpo-40141: Include the value in the column position for keyword AST nodes (GH-19348) 2020-04-03 21:02:26 +01:00
Pablo Galindo 168660b547
bpo-40141: Add line and column information to ast.keyword nodes (GH-19283) 2020-04-02 00:47:39 +01:00
Serhiy Storchaka bace59d8b8
bpo-39999: Improve compatibility of the ast module. (GH-19056)
* Re-add removed classes Suite, slice, Param, AugLoad and AugStore.
* Add docstrings for dummy classes.
* Add docstrings for attribute aliases.
* Set __module__ to "ast" instead of "_ast".
2020-03-22 20:33:34 +02:00
Batuhan Taşkaya 0ac59f93c0
bpo-40000: Improve error messages when validating invalid ast.Constant nodes (GH-19055)
Co-authored-by: Pablo Galindo <Pablogsal@gmail.com>
2020-03-19 11:32:28 +00:00
Batuhan Taşkaya 4ab362cec6
bpo-39638: Keep ASDL signatures in the AST nodes (GH-18515) 2020-03-16 10:12:53 +02:00
Serhiy Storchaka 13d52c2686
bpo-34822: Simplify AST for subscription. (GH-9605)
* Remove the slice type.
* Make Slice a kind of the expr type instead of the slice type.
* Replace ExtSlice(slices) with Tuple(slices, Load()).
* Replace Index(value) with a value itself.

All non-terminal nodes in AST for expressions are now of the expr type.
2020-03-10 18:52:34 +02:00
Serhiy Storchaka b7e9525f9c
bpo-36287: Make ast.dump() not output optional fields and attributes with default values. (GH-18843)
The default values for optional fields and attributes of AST nodes are now set
as class attributes (e.g. Constant.kind is set to None).
2020-03-10 00:07:47 +02:00
Batuhan Taşkaya 397b96f6d7
bpo-38870: Implement a precedence algorithm in ast.unparse (GH-17377)
Implement a simple precedence algorithm for ast.unparse in order to avoid redundant
parenthesis for nested structures in the final output.
2020-03-01 20:12:17 +00:00
Serhiy Storchaka 6e619c48b8
bpo-39474: Fix AST pos for expressions like (a)(b), (a)[b] and (a).b. (GH-18477) 2020-02-12 22:37:49 +02:00
Lysandros Nikolaou d2e1098641
bpo-39579: Fix Attribute end_col_offset to point at the current node (GH-18405) 2020-02-07 15:36:32 -08:00
Serhiy Storchaka 850a8856e1
bpo-39235: Check end_lineno and end_col_offset of AST nodes. (GH-17926) 2020-01-10 10:12:55 +02:00
Raymond Hettinger 4fcf5c12a3
bpo-39158: ast.literal_eval() doesn't support empty sets (GH-17742) 2020-01-02 22:21:18 -07:00
Lysandros Nikolaou 50d4f12958 bpo-39080: Starred Expression's column offset fix when inside a CALL (GH-17645)
Co-Authored-By: Pablo Galindo <Pablogsal@gmail.com>
2019-12-18 00:20:55 +00:00
Lysandros Nikolaou 5936a4ce91 Fix elif start column offset when there is an else following (GH-17596) 2019-12-14 10:24:57 +00:00
Lysandros Nikolaou 025a602af7 bpo-39031: Include elif keyword when producing lineno/col-offset info for if_stmt (GH-17582)
When parsing an "elif" node, lineno and col_offset of the node now point to the "elif" keyword and not to its condition, making it consistent with the "if" node.


https://bugs.python.org/issue39031



Automerge-Triggered-By: @pablogsal
2019-12-12 13:40:21 -08:00
Serhiy Storchaka 26ae9f6d3d
bpo-38535: Fix positions for AST nodes for calls without arguments in decorators. (GH-16861) 2019-10-26 16:46:05 +03:00
Serhiy Storchaka 850573b836
bpo-37995: Add an option to ast.dump() to produce a multiline output. (GH-15631) 2019-09-09 19:33:13 +03:00
Serhiy Storchaka e64f948e76
bpo-37950: Fix ast.dump() when call with incompletely initialized node. (GH-15510) 2019-08-29 09:30:23 +03:00
Serhiy Storchaka c3ea41e9bf
bpo-36917: Add default implementation of ast.NodeVisitor.visit_Constant(). (GH-15490)
It emits a deprecation warning and calls corresponding method
visit_Num(), visit_Str(), etc.
2019-08-26 10:13:19 +03:00
Min ho Kim c4cacc8c5e Fix typos in comments, docs and test names (#15018)
* Fix typos in comments, docs and test names

* Update test_pyparse.py

account for change in string length

* Apply suggestion: splitable -> splittable

Co-Authored-By: Terry Jan Reedy <tjreedy@udel.edu>

* Apply suggestion: splitable -> splittable

Co-Authored-By: Terry Jan Reedy <tjreedy@udel.edu>

* Apply suggestion: Dealloccte -> Deallocate

Co-Authored-By: Terry Jan Reedy <tjreedy@udel.edu>

* Update posixmodule checksum.

* Reverse idlelib changes.
2019-07-30 18:16:13 -04:00
Pablo Galindo cd6e83b481 bpo-37593: Swap the positions of posonlyargs and args in the constructor of ast.parameters nodes (GH-14778)
https://bugs.python.org/issue37593
2019-07-14 16:32:18 -07:00
Carl Friedrich Bolz-Tereick 430a9f44fe bpo-18374: fix tests to check the correct thing about line numbers (GH-14659) 2019-07-09 13:20:01 +01:00
Carl Friedrich Bolz-Tereick 110a47c4f4 bpo-18374: fix wrong col_offset of some ast.BinOp instances (GH-14607)
Nested BinOp instances (e.g. a+b+c) had a wrong col_offset for the
second BinOp (e.g. 2 instead of 0 in the example). Fix it by using the
correct st node to copy the line and col_offset from in ast.c.
2019-07-08 22:17:56 +01:00
Pablo Galindo 2f58a84104
bpo-37112: Allow compile to work on AST with positional only arguments with defaults (GH-13697) 2019-05-31 14:09:49 +01:00
Pablo Galindo 8c77b8cb91
bpo-36540: PEP 570 -- Implementation (GH-12701)
This commit contains the implementation of PEP570: Python positional-only parameters.

* Update Grammar/Grammar with new typedarglist and varargslist

* Regenerate grammar files

* Update and regenerate AST related files

* Update code object

* Update marshal.c

* Update compiler and symtable

* Regenerate importlib files

* Update callable objects

* Implement positional-only args logic in ceval.c

* Regenerate frozen data

* Update standard library to account for positional-only args

* Add test file for positional-only args

* Update other test files to account for positional-only args

* Add News entry

* Update inspect module and related tests
2019-04-29 13:36:57 +01:00
Pablo Galindo 0c9258a6d2
bpo-36332: Allow compile() to handle AST objects with assignment expressions (GH-12398) 2019-03-18 13:51:53 +00:00
Guido van Rossum 10f8ce6688 bpo-36280: Add Constant.kind field (GH-12295)
The value is a string for string and byte literals, None otherwise.
It is 'u' for u"..." literals, 'b' for b"..." literals, '' for "..." literals.
The 'r' (raw) prefix is ignored.
Does not apply to f-strings.

This appears sufficient to make mypy capable of using the stdlib ast module instead of typed_ast (assuming a mypy patch I'm working on).

WIP: I need to make the tests pass. @ilevkivskyi @serhiy-storchaka 



https://bugs.python.org/issue36280
2019-03-13 13:00:46 -07:00
Guido van Rossum dcfcd146f8 bpo-35766: Merge typed_ast back into CPython (GH-11645) 2019-01-31 12:40:27 +01:00
Ivan Levkivskyi 9932a22897
bpo-33416: Add end positions to Python AST (GH-11605)
The majority of this PR is tediously passing `end_lineno` and `end_col_offset` everywhere. Here are non-trivial points:
* It is not possible to reconstruct end positions in AST "on the fly", some information is lost after an AST node is constructed, so we need two more attributes for every AST node `end_lineno` and `end_col_offset`.
* I add end position information to both CST and AST.  Although it may be technically possible to avoid adding end positions to CST, the code becomes more cumbersome and less efficient.
* Since the end position is not known for non-leaf CST nodes while the next token is added, this requires a bit of extra care (see `_PyNode_FinalizeEndPos`). Unless I made some mistake, the algorithm should be linear.
* For statements, I "trim" the end position of suites to not include the terminal newlines and dedent (this seems to be what people would expect), for example in
  ```python
  class C:
      pass

  pass
  ```
  the end line and end column for the class definition is (2, 8).
* For `end_col_offset` I use the common Python convention for indexing, for example for `pass` the `end_col_offset` is 4 (not 3), so that `[0:4]` gives one the source code that corresponds to the node.
* I added a helper function `ast.get_source_segment()`, to get source text segment corresponding to a given AST node. It is also useful for testing.

An (inevitable) downside of this PR is that AST now takes almost 25% more memory. I think however it is probably justified by the benefits.
2019-01-22 11:18:22 +00:00
Anthony Sottile 7417622617 bpo-35733: Make isinstance(ast.Constant(boolean), ast.Num) be false. (GH-11547) 2019-01-18 21:30:28 +02:00
Serhiy Storchaka 3bcbedc9f1
bpo-34850: Emit a warning for "is" and "is not" with a literal. (GH-9642) 2019-01-18 07:47:48 +02:00
Anthony Sottile 995d9b9297 bpo-16806: Fix `lineno` and `col_offset` for multi-line string tokens (GH-10021) 2019-01-13 13:05:13 +09:00
Serhiy Storchaka b619b09792
bpo-31241: Fix AST node position for list and generator comprehensions. (GH-10633)
The lineno and col_offset attributes of AST nodes for list comprehensions,
generator expressions and tuples are now point to the opening parenthesis or
square brace. For tuples without parenthesis they point to the position
of the first item.
2018-11-27 09:40:29 +02:00
Serhiy Storchaka 95b6acf951
bpo-34876: Change the lineno of the AST for decorated function and class. (GH-9731)
It was overridden by the lineno of the first decorator. Now it is
the lineno of 'def' or 'class'.
2018-10-30 13:16:02 +02:00
Serhiy Storchaka 6015cc50bc
bpo-32892: Support subclasses of base types in isinstance checks for AST constants. (GH-9934)
Some projects (e.g. Chameleon) create ast.Str containing an instance
of the str subclass.
2018-10-28 13:43:03 +02:00
Serhiy Storchaka 3f22811fef
bpo-32892: Use ast.Constant instead of specific constant AST types. (GH-9445) 2018-09-27 17:42:37 +03:00
guoci 90fc8980bb closes bpo-31902: Fix the col_offset attribute for ast.Async* nodes to point to the "async" keyword. (GH-4175)
Previously, col_offset points to the keyword after "async".
2018-09-11 14:45:45 -07:00
Serhiy Storchaka 08f127a3ca
bpo-33851: Fix ast.get_docstring() for a node that lacks a docstring. (GH-7682) 2018-06-15 11:05:15 +03:00
Serhiy Storchaka 73cbe7a01a
bpo-32911: Revert bpo-29463. (GH-7121) (GH-7197)
Remove the docstring attribute of AST types and restore docstring
expression as a first stmt in their body.

Co-authored-by: INADA Naoki <methane@users.noreply.github.com>
2018-05-29 12:04:55 +03:00
Serhiy Storchaka d8ac4d1d5a
bpo-31778: Make ast.literal_eval() more strict. (#4035)
Addition and subtraction of arbitrary numbers no longer allowed.
2018-01-04 11:15:39 +02:00
Oren Milman 7dc46d8cf5 bpo-31592: Fix an assertion failure in Python parser in case of a bad unicodedata.normalize(). (#3767) 2017-09-30 20:16:24 +03:00
Matthias Bussonnier 41cea70aa3 bpo-29637: clean docstring only if not None (GH-267) 2017-02-24 15:44:19 +09:00