Commit Graph

202 Commits

Author SHA1 Message Date
Serhiy Storchaka 1f21eaa15e
bpo-15999: Clean up of handling boolean arguments. (GH-15610)
* Use the 'p' format unit instead of manually called PyObject_IsTrue().
* Pass boolean value instead 0/1 integers to functions that needs boolean.
* Convert some arguments to boolean only once.
2019-09-01 12:16:51 +03:00
Pablo Galindo 9211e2fd81 bpo-37268: Add deprecation notice and a DeprecationWarning for the parser module (GH-15017)
Deprecate the parser module and add a deprecation warning triggered on import and a warning block in the documentation.





https://bugs.python.org/issue37268



Automerge-Triggered-By: @pablogsal
2019-07-30 04:04:01 -07:00
Victor Stinner 37d66d7d4b
bpo-37253: Add _PyCompilerFlags_INIT macro (GH-14018)
Add a new _PyCompilerFlags_INIT macro to initialize PyCompilerFlags
variables, rather than initializing cf_flags and cf_feature_version
explicitly in each variable.
2019-06-13 02:16:41 +02:00
Jeroen Demeyer 530f506ac9 bpo-36974: tp_print -> tp_vectorcall_offset and tp_reserved -> tp_as_async (GH-13464)
Automatically replace
tp_print -> tp_vectorcall_offset
tp_compare -> tp_as_async
tp_reserved -> tp_as_async
2019-05-30 19:13:39 -07:00
Inada Naoki 09415ff0eb
fix warnings by adding more const (GH-12924) 2019-04-23 20:39:37 +09:00
tyomitch cb0748d393 bpo-36440: include node names in ParserError messages, instead of numeric IDs (GH-12565)
The error messages in the parser module are referring to numeric IDs for the nodes. To improve readability, use the node names when reporting errors.
2019-04-03 01:12:07 -04:00
Pablo Galindo 9a0000d15d
bpo-36256: Fix bug in parsermodule when parsing if statements (GH-12477)
bpo-36256: Fix bug in parsermodule when parsing if statements

In the parser module, when validating nodes before starting the parsing with to create a ST in "parser_newstobject" there is a problem that appears when two arcs in the same DFA state has transitions with labels with the same type. For example, the DFA for if_stmt has a state with
two labels with the same type: "elif" and "else" (type NAME). The algorithm tries one by one the arcs until the label that starts the arc transition has a label with the same type of the current child label we are trying to accept. In this case, the arc for "elif" comes before the arc for "else"and passes this test (because the current child label is "else" and has the same type as "elif"). This lead to expecting a namedexpr_test (305) instead of a colon (11). The solution is to compare also the string representation (in case there is one) of the labels to see if the transition that we have is the correct one.
2019-03-21 23:33:02 +00:00
tyomitch 1b304f992d Remove d_initial from the parser as it is unused (GH-12212)
d_initial, the first state of a particular DFA in the parser has always been initialized to 0 in the old pgen as well as the new pgen. As this value is not used and the first state of each DFA is assumed to be the first element in the array representing it, remove d_initial from the parser to reduce complexity.
2019-03-09 15:35:50 +00:00
Guido van Rossum 495da29225 bpo-35975: Support parsing earlier minor versions of Python 3 (GH-12086)
This adds a `feature_version` flag to `ast.parse()` (documented) and `compile()` (hidden) that allow tweaking the parser to support older versions of the grammar. In particular if `feature_version` is 5 or 6, the hacks for the `async` and `await` keyword from PEP 492 are reinstated. (For 7 or higher, these are unconditionally treated as keywords, but they are still special tokens rather than `NAME` tokens that the parser driver recognizes.)



https://bugs.python.org/issue35975
2019-03-07 12:38:08 -08:00
Guido van Rossum dcfcd146f8 bpo-35766: Merge typed_ast back into CPython (GH-11645) 2019-01-31 12:40:27 +01:00
Ivan Levkivskyi 9932a22897
bpo-33416: Add end positions to Python AST (GH-11605)
The majority of this PR is tediously passing `end_lineno` and `end_col_offset` everywhere. Here are non-trivial points:
* It is not possible to reconstruct end positions in AST "on the fly", some information is lost after an AST node is constructed, so we need two more attributes for every AST node `end_lineno` and `end_col_offset`.
* I add end position information to both CST and AST.  Although it may be technically possible to avoid adding end positions to CST, the code becomes more cumbersome and less efficient.
* Since the end position is not known for non-leaf CST nodes while the next token is added, this requires a bit of extra care (see `_PyNode_FinalizeEndPos`). Unless I made some mistake, the algorithm should be linear.
* For statements, I "trim" the end position of suites to not include the terminal newlines and dedent (this seems to be what people would expect), for example in
  ```python
  class C:
      pass

  pass
  ```
  the end line and end column for the class definition is (2, 8).
* For `end_col_offset` I use the common Python convention for indexing, for example for `pass` the `end_col_offset` is 4 (not 3), so that `[0:4]` gives one the source code that corresponds to the node.
* I added a helper function `ast.get_source_segment()`, to get source text segment corresponding to a given AST node. It is also useful for testing.

An (inevitable) downside of this PR is that AST now takes almost 25% more memory. I think however it is probably justified by the benefits.
2019-01-22 11:18:22 +00:00
Serhiy Storchaka 62be74290a
bpo-33012: Fix invalid function cast warnings with gcc 8. (GH-6749)
Fix invalid function cast warnings with gcc 8
for method conventions different from METH_NOARGS, METH_O and
METH_VARARGS excluding Argument Clinic generated code.
2018-11-27 13:27:31 +02:00
Victor Stinner 3bb183d7fb
bpo-35177, Python-ast.h: Fix "Yield" compiler warning (GH-10664)
Partially revert commit 5f2df88b63e50d23914e97ec778861a52abdeaad:
add "#undef Yield" to .c files after including Python-ast.h.

Fix the warning:

    winbase.h(102): warning C4005: 'Yield': macro redefinition
2018-11-22 18:38:38 +01:00
Victor Stinner 5f2df88b63
bpo-35177: Add dependencies between header files (GH-10361)
* ast.h now includes Python-ast.h and node.h
* parsetok.h now includes node.h and grammar.h
* symtable.h now includes Python-ast.h
* Modify asdl_c.py to enhance Python-ast.h:

  * Add #ifndef/#define Py_PYTHON_AST_H to be able to include the header
    twice
  * Add "extern { ... }" for C++
  * Undefine "Yield" macro conflicting with winbase.h

* Remove "#undef Yield" from C files, it's now done in Python-ast.h
* Remove now useless includes in C files
2018-11-12 00:56:19 +01:00
Serhiy Storchaka e5362eaa75
bpo-33308: Fix a crash in the parser module when convert an ST object. (#6519)
Converting with line_info=False and col_info=True crashed before.
2018-04-19 01:55:37 +03:00
stratakis e8b1965639 bpo-23699: Use a macro to reduce boilerplate code in rich comparison functions (GH-793) 2017-11-02 20:32:54 +10:00
Serhiy Storchaka a79f4c2195 bpo-30070: Fixed leaks and crashes in errors handling in the parser module. (#1131) 2017-04-19 21:09:21 +03:00
Serhiy Storchaka 228b12edcc Issue #28999: Use Py_RETURN_NONE, Py_RETURN_TRUE and Py_RETURN_FALSE wherever
possible.  Patch is writen with Coccinelle.
2017-01-23 09:47:21 +02:00
Serhiy Storchaka 06515833fe Replaced outdated macros _PyUnicode_AsString and _PyUnicode_AsStringAndSize
with PyUnicode_AsUTF8 and PyUnicode_AsUTF8AndSize.
2016-11-20 09:13:07 +02:00
Benjamin Peterson 53595c4879 replace custom validation logic in the parse module with a simple DFA validator (closes #26526)
Patch from A. Skrobov.
2016-06-02 11:30:18 -07:00
Serhiy Storchaka ef1585eb9a Issue #25923: Added more const qualifiers to signatures of static and private functions. 2015-12-25 20:01:53 +02:00
Serhiy Storchaka 2d06e84455 Issue #25923: Added the const qualifier to static constant arrays. 2015-12-25 19:53:18 +02:00
Serhiy Storchaka 5c4064e8bd Issue #25421: __sizeof__ methods of builtin types now use dynamic basic size.
This allows sys.getsize() to work correctly with their subclasses with
__slots__ defined.
2015-12-19 20:05:25 +02:00
Yury Selivanov 14acf5f41d Issue #24791: Fix grammar regression for call syntax: 'g(*a or b)'. 2015-08-05 17:54:10 -04:00
Benjamin Peterson de12b79cd6 allow test node after ** in calls (closes #24176) 2015-05-16 09:44:45 -04:00
Yury Selivanov 7544508f02 PEP 0492 -- Coroutines with async and await syntax. Issue #24017. 2015-05-11 22:57:16 -04:00
Benjamin Peterson 025e9ebd0a PEP 448: additional unpacking generalizations (closes #2292)
Patch by Neil Girdhar.
2015-05-05 20:16:41 -04:00
Serhiy Storchaka 009b811d67 Removed unintentional trailing spaces in non-external and non-generated C files. 2015-03-18 21:53:15 +02:00
Victor Stinner f4c68db162 (Merge 3.3) parser: fix usage of Py_BuildValue() to build a parser error
Fix typo: "os" format => "Os"
2014-01-02 11:50:10 +01:00
Victor Stinner 5f8d485982 parser: fix usage of Py_BuildValue() to build a parser error
Fix typo: "os" format => "Os"
2014-01-02 11:49:27 +01:00
Victor Stinner 14e461d5b9 Close #11619: The parser and the import machinery do not encode Unicode
filenames anymore on Windows.
2013-08-26 22:28:21 +02:00
Victor Stinner 6684bdf73d Issue #18408: Fix typo in build_node_tree() of the parser module
Type "o" format of Py_BuildValue() is invalid: it must be "O".
2013-07-17 00:13:52 +02:00
Victor Stinner df4572cc71 Issue #18408: parser module: fix error handling in node2tuple()
Handle PyLong_FromLong() and PyUnicode_FromString() failures
2013-07-12 01:35:10 +02:00
Victor Stinner 3bd6abd129 Issue #18408: Fix parser.sequence2st() and parser.tuple2st(): raise MemoryError
on memory allocation failure

Instead of ignoring the memory allocation failure and create invalid objects.
2013-07-12 01:33:59 +02:00
Serhiy Storchaka 9101e23ff6 Issue #15989: Fix several occurrences of integer overflow
when result of PyLong_AsLong() narrowed to int without checks.

This is a backport of changesets 13e2e44db99d and 525407d89277.
2013-01-19 12:41:45 +02:00
Serhiy Storchaka 7898043868 Issue #15989: Fix several occurrences of integer overflow
when result of PyLong_AsLong() narrowed to int without checks.
2013-01-15 01:12:17 +02:00
Andrew Svetlov 5b89840d9c Issue #16714: use 'raise' exceptions, don't 'throw'.
Patch by Serhiy Storchaka.
2012-12-18 21:26:36 +02:00
Andrew Svetlov 737fb89dd1 Issue #16714: use 'raise' exceptions, don't 'throw'.
Patch by Serhiy Storchaka.
2012-12-18 21:14:22 +02:00
Antoine Pitrou 721738fbee Issue #15604: Update uses of PyObject_IsTrue() to check for and handle errors correctly.
Patch by Serhiy Storchaka.
2012-08-15 23:20:39 +02:00
Antoine Pitrou 6f430e4963 Issue #15604: Update uses of PyObject_IsTrue() to check for and handle errors correctly.
Patch by Serhiy Storchaka.
2012-08-15 23:18:25 +02:00
Jesus Cea 88ca04e6a8 MERGE: Closes #15512: Correct __sizeof__ support for parser 2012-08-03 14:29:26 +02:00
Jesus Cea e9c5318967 Closes #15512: Correct __sizeof__ support for parser 2012-08-03 14:28:37 +02:00
Mark Dickinson 943cab2fec Issue #14741: Merge fix from 3.2. 2012-05-07 17:25:14 +01:00
Mark Dickinson da029fb293 Issue #14741: Fix missing support for ellipsis in parser module. 2012-05-07 17:24:04 +01:00
Mark Dickinson 72f6095d4f Issue #14697: Merge fix from 3.2. 2012-05-07 16:36:33 +01:00
Mark Dickinson 11c1dee183 Issue #14697: Fix missing parser module support for set displays and set comprehensions. 2012-05-07 16:34:34 +01:00
Mark Dickinson cf360b9209 Issue #14701: Add missing support for 'raise ... from' in parser module. 2012-05-07 12:01:27 +01:00
Mark Dickinson 9fad160411 Issue #14701: Merge fix from 3.2. 2012-05-07 12:03:11 +01:00
Mark Dickinson 57404891a0 Issue #14696: Merge from 3.2 2012-04-29 22:20:01 +01:00
Mark Dickinson 407b3bd89b Issue #14696: Fix parser module to understand 'nonlocal' declarations. 2012-04-29 22:18:31 +01:00