Commit Graph

129 Commits

Author SHA1 Message Date
Emmanuel Arias ed5e29cba5 bpo-36385: Add ``elif`` sentence on to avoid multiple ``if`` (GH-12478)
Currently, when arguments on Parser/asdl_c.py are parsed
``ìf`` sentence is used. This PR Propose to use ``elif``
to avoid multiple evaluting of the ifs.





https://bugs.python.org/issue36385
2019-03-20 21:39:17 -07:00
Guido van Rossum 495da29225 bpo-35975: Support parsing earlier minor versions of Python 3 (GH-12086)
This adds a `feature_version` flag to `ast.parse()` (documented) and `compile()` (hidden) that allow tweaking the parser to support older versions of the grammar. In particular if `feature_version` is 5 or 6, the hacks for the `async` and `await` keyword from PEP 492 are reinstated. (For 7 or higher, these are unconditionally treated as keywords, but they are still special tokens rather than `NAME` tokens that the parser driver recognizes.)



https://bugs.python.org/issue35975
2019-03-07 12:38:08 -08:00
Guido van Rossum 3a32e3bf88
bpo-35766 follow-up: Kill half-support for FunctionType in PyAST_obj2mod (#11714)
See 229874c612 (r252631862)

https://bugs.python.org/issue35766
2019-02-01 11:37:34 -08:00
Guido van Rossum dcfcd146f8 bpo-35766: Merge typed_ast back into CPython (GH-11645) 2019-01-31 12:40:27 +01:00
Ivan Levkivskyi 9932a22897
bpo-33416: Add end positions to Python AST (GH-11605)
The majority of this PR is tediously passing `end_lineno` and `end_col_offset` everywhere. Here are non-trivial points:
* It is not possible to reconstruct end positions in AST "on the fly", some information is lost after an AST node is constructed, so we need two more attributes for every AST node `end_lineno` and `end_col_offset`.
* I add end position information to both CST and AST.  Although it may be technically possible to avoid adding end positions to CST, the code becomes more cumbersome and less efficient.
* Since the end position is not known for non-leaf CST nodes while the next token is added, this requires a bit of extra care (see `_PyNode_FinalizeEndPos`). Unless I made some mistake, the algorithm should be linear.
* For statements, I "trim" the end position of suites to not include the terminal newlines and dedent (this seems to be what people would expect), for example in
  ```python
  class C:
      pass

  pass
  ```
  the end line and end column for the class definition is (2, 8).
* For `end_col_offset` I use the common Python convention for indexing, for example for `pass` the `end_col_offset` is 4 (not 3), so that `[0:4]` gives one the source code that corresponds to the node.
* I added a helper function `ast.get_source_segment()`, to get source text segment corresponding to a given AST node. It is also useful for testing.

An (inevitable) downside of this PR is that AST now takes almost 25% more memory. I think however it is probably justified by the benefits.
2019-01-22 11:18:22 +00:00
Victor Stinner 3bb183d7fb
bpo-35177, Python-ast.h: Fix "Yield" compiler warning (GH-10664)
Partially revert commit 5f2df88b63e50d23914e97ec778861a52abdeaad:
add "#undef Yield" to .c files after including Python-ast.h.

Fix the warning:

    winbase.h(102): warning C4005: 'Yield': macro redefinition
2018-11-22 18:38:38 +01:00
Victor Stinner 5f2df88b63
bpo-35177: Add dependencies between header files (GH-10361)
* ast.h now includes Python-ast.h and node.h
* parsetok.h now includes node.h and grammar.h
* symtable.h now includes Python-ast.h
* Modify asdl_c.py to enhance Python-ast.h:

  * Add #ifndef/#define Py_PYTHON_AST_H to be able to include the header
    twice
  * Add "extern { ... }" for C++
  * Undefine "Yield" macro conflicting with winbase.h

* Remove "#undef Yield" from C files, it's now done in Python-ast.h
* Remove now useless includes in C files
2018-11-12 00:56:19 +01:00
Serhiy Storchaka 3f22811fef
bpo-32892: Use ast.Constant instead of specific constant AST types. (GH-9445) 2018-09-27 17:42:37 +03:00
Serhiy Storchaka a5c42284e6
bpo-33677: Fix signatures of tp_clear handlers for AST and deque. (GH-7196) 2018-05-31 07:34:34 +03:00
Serhiy Storchaka f320be77ff bpo-32571: Avoid raising unneeded AttributeError and silencing it in C code (GH-5222)
Add two new private APIs: _PyObject_LookupAttr() and _PyObject_LookupAttrId()
2018-01-25 17:49:40 +09:00
Serhiy Storchaka bba2239c17
bpo-31572: Get rid of _PyObject_HasAttrId() in the ASDL parser. (#3725)
Silence only expected AttributeError.
2017-11-11 16:41:32 +02:00
Antoine Pitrou b091bec824 bpo-31536: Avoid wholesale rebuild after `make regen-all` (#3678)
* bpo-31536: Avoid wholesale rebuild after `make regen-all`

* Add NEWS
2017-09-20 14:57:56 -07:00
Serhiy Storchaka 5d84cb368c bpo-31464: asdl_c.py no longer emits trailing spaces in Python-ast.h. (#3568) 2017-09-14 20:28:22 -07:00
INADA Naoki a6296d34a4 bpo-31095: fix potential crash during GC (GH-2974) 2017-08-24 14:55:17 +09:00
Yuan Chao Chou 2af565baf4 Fix a shadow-compatible-local warning (#2180)
Change the shadowing naming, 'value' (Python-ast.c:4652), to 'val'
to prevent the variables from being misused.
2017-08-04 10:53:12 -07:00
INADA Naoki 4c78c527d2 bpo-29622: Make AST constructor to accept less than enough number of positional arguments (GH-249)
bpo-29463 added optional "docstring" field to 4 AST types.
While it is optional, it breaks backward compatibility because AST constructor
requires number of positional argument is same to number of fields.

AST types accepts empty arguments, and incomplete keyword arguments.
But it's not big problem because field can be filled after creation, and checked when compiling.
So stop requiring complete set of fields for positional arguments too.
2017-02-24 02:48:17 +09:00
INADA Naoki fc489082c8 Issue #29369: Use Py_IDENTIFIER in Python-ast.c 2017-01-25 22:33:43 +09:00
Serhiy Storchaka 5e80855af3 Issue #24098: Fixed possible crash when AST is changed in process of
compiling it.
2016-10-07 21:55:49 +03:00
Serhiy Storchaka cf3806026b Issue #24098: Fixed possible crash when AST is changed in process of
compiling it.
2016-10-07 21:51:28 +03:00
Victor Stinner 25219f596a Issue #26146: remove useless code
obj2ast_constant() code is baesd on obj2ast_object() which has a special case
for Py_None. But in practice, we don't need to have a special case for
constants.

Issue noticed by Joseph Jevnik on a review.
2016-01-27 00:37:59 +01:00
Victor Stinner f2c1aa1661 Add ast.Constant
Issue #26146: Add a new kind of AST node: ast.Constant. It can be used by
external AST optimizers, but the compiler does not emit directly such node.

An optimizer can replace the following AST nodes with ast.Constant:

* ast.NameConstant: None, False, True
* ast.Num: int, float, complex
* ast.Str: str
* ast.Bytes: bytes
* ast.Tuple if items are constants too: tuple
* frozenset

Update code to accept ast.Constant instead of ast.Num and/or ast.Str:

* compiler
* docstrings
* ast.literal_eval()
* Tools/parser/unparse.py
2016-01-26 00:40:57 +01:00
Victor Stinner c106c68aeb Issue #25555: Fix parser and AST: fill lineno and col_offset of "arg" node when
compiling AST from Python objects.
2015-11-06 17:01:48 +01:00
Benjamin Peterson 860c8a404a merge 3.5 (#25502) 2015-10-28 23:15:22 -07:00
Benjamin Peterson 669ff66c32 remove duplicated imports (closes #25502) 2015-10-28 23:15:13 -07:00
Serhiy Storchaka fc632e3912 Merge with 3.5. 2015-10-06 18:52:52 +03:00
Serhiy Storchaka 481d3af82e Make asdl_c.py to generate Python-ast.c changed in issue #15989. 2015-09-06 23:29:04 +03:00
Eli Bendersky 5e3d338a74 Issue #19655: Replace the ASDL parser carried with CPython
The new parser does not rely on Spark (which is now removed from our repo),
uses modern 3.x idioms and is significantly smaller and simpler.

It generates exactly the same AST files (.h and .c), so in practice no builds
should be affected.
2014-05-09 17:58:22 -07:00
Benjamin Peterson 42ec031fe7 merge 3.3 (#20588) 2014-02-10 22:41:40 -05:00
Benjamin Peterson c2f665e721 don't put runtime values in array initializer for C89 compliance (closes #20588) 2014-02-10 22:19:02 -05:00
Antoine Pitrou d01d396e7f Issue #4555: All exported C symbols are now prefixed with either "Py" or "_Py".
("make smelly" now clean)
2013-10-12 22:52:43 +02:00
Ezio Melotti d640fe2af5 #18803: merge with 3.3. 2013-08-26 01:33:30 +03:00
Ezio Melotti 7c4a7e6f3c #18803: fix more typos. Patch by Févry Thibault. 2013-08-26 01:32:56 +03:00
Christian Heimes b7f1b38dea Issue #18552: Check return value of PyArena_AddPyObject() in obj2ast_object(). 2013-07-27 00:33:35 +02:00
Christian Heimes 70c94e7896 Issue #18552: Check return value of PyArena_AddPyObject() in obj2ast_object(). 2013-07-27 00:33:13 +02:00
Victor Stinner b318990cac (Merge 3.3) Parser/asdl_c.py: use Py_CLEAR() 2013-07-27 00:04:42 +02:00
Victor Stinner 1acc129d48 Parser/asdl_c.py: use Py_CLEAR() 2013-07-27 00:03:47 +02:00
Victor Stinner ee4b59c0f8 (Merge 3.3) According to the PEP 7, C code must "use 4-space indents"
Replace 8 spaces with 4.
2013-07-27 00:01:35 +02:00
Victor Stinner ce72e1ce6c According to the PEP 7, C code must "use 4-space indents"
Replace 8 spaces with 4.
2013-07-27 00:00:36 +02:00
Victor Stinner bdf630c4a7 Issue #18408: Fix Python-ast.c: handle init_types() failure (ex: MemoryError) 2013-07-17 00:17:15 +02:00
Benjamin Peterson 8d89c2aaba change AST codegen to use PyModule_AddIntMacro 2013-05-20 10:28:48 -07:00
Benjamin Peterson 7654ab9ef0 placate msvc 2013-03-18 23:39:53 -07:00
Benjamin Peterson b72406b8fa refactor to fix refleaks 2013-03-18 23:24:41 -07:00
Benjamin Peterson cda75be02a unify some ast.argument's attrs; change Attribute column offset (closes #16795)
Patch from Sven Brauch.
2013-03-18 10:48:58 -07:00
Benjamin Peterson 442f20996d create NameConstant AST class for None, True, and False literals (closes #16619) 2012-12-06 17:41:04 -05:00
Matthias Klose aee3c76acf - Issue #16262: fix out-of-src-tree builds, if mercurial is not installed. 2012-10-21 23:12:35 +02:00
Ezio Melotti 8a9cc526fe #15923: merge with 3.2. 2012-09-30 22:47:47 +03:00
Ezio Melotti cb2916a714 #15923: fix a mistake in asdl_c.py that resulted in a TypeError after 2801bf875a24 (see #15801). 2012-09-30 22:41:37 +03:00
Benjamin Peterson 8107176f9b add gc support to the AST base type (closes #15293) 2012-07-08 11:03:46 -07:00
Antoine Pitrou 507507473e Issue #15291: Fix a memory leak where AST nodes where not properly deallocated. 2012-07-08 12:43:32 +02:00
Benjamin Peterson 77fa9379e2 use Py_ssize_t for ast sequence lengths 2012-05-15 10:10:27 -07:00