This program can segfault the parser by stack overflow:
```
import ast
code = "f(" + ",".join(['a' for _ in range(100000)]) + ")"
print("Ready!")
ast.parse(code)
```
the reason is that the rule for arguments has a simple recursion when collecting args:
args[expr_ty]:
[...]
| a=named_expression b=[',' c=args { c }] {
[...] }.
(cherry picked from commit 4a97b1517a)
Co-authored-by: Pablo Galindo <Pablogsal@gmail.com>
This will improve the debug experience if something fails in the produced AST. Previously, errors in the produced AST can be felt much later like in the garbage collector or the compiler, making debugging them much more difficult..
(cherry picked from commit 1332226b32)
Co-authored-by: Pablo Galindo <Pablogsal@gmail.com>
GCC says
```
../cpython/Parser/string_parser.c: In function ‘fstring_find_expr’:
../cpython/Parser/string_parser.c:404:93: warning: ‘cols’ may be used uninitialized in this function [-Wmaybe-uninitialized]
404 | p2->starting_col_offset = p->tok->first_lineno == p->tok->lineno ? t->col_offset + cols : cols;
| ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~^~~~~~
../cpython/Parser/string_parser.c:384:16: note: ‘cols’ was declared here
384 | int lines, cols;
| ^~~~
../cpython/Parser/string_parser.c:403:45: warning: ‘lines’ may be used uninitialized in this function [-Wmaybe-uninitialized]
403 | p2->starting_lineno = t->lineno + lines - 1;
| ~~~~~~~~~~~~~~~~~~^~~
../cpython/Parser/string_parser.c:384:9: note: ‘lines’ was declared here
384 | int lines, cols;
| ^~~~~
```
and, indeed, if `PyBytes_AsString` somehow fails, lines & cols will not be initialized.
(cherry picked from commit 2ad7e9c011)
Co-authored-by: Benjamin Peterson <benjamin@python.org>
This commit changes the parsing of f-string expressions with the new parser. The parser gets pre-fed with the location of the expression itself (not the f-string, which was what we were doing before). This allows us to completely skip the shifting of the AST nodes after the parsing is completed..
(cherry picked from commit 1f0f4abb11)
Prefix the error message with `fstring: `, when parsing an f-string expression throws a `SyntaxError`.
(cherry picked from commit 2e0a920e9e)
Co-authored-by: Lysandros Nikolaou <lisandrosnik@gmail.com>
`GET_INVALID_TARGET` might unexpectedly return `NULL`, which if not
caught will cause a SEGFAULT. Therefore, this commit introduces a new
inline function `RAISE_SYNTAX_ERROR_INVALID_TARGET` that always
checks for `GET_INVALID_TARGET` returning NULL and can be used in
the grammar, replacing the long C ternary operation used till now.
(cherry picked from commit 6c4e0bd974)
Automerge-Triggered-By: @pablogsal
* bpo-40334: Produce better error messages on invalid targets (GH-20106)
The following error messages get produced:
- `cannot delete ...` for invalid `del` targets
- `... is an illegal 'for' target` for invalid targets in for
statements
- `... is an illegal 'with' target` for invalid targets in
with statements
Additionally, a few `cut`s were added in various places before the
invocation of the `invalid_*` rule, in order to speed things
up.
Co-authored-by: Pablo Galindo <Pablogsal@gmail.com>
(cherry picked from commit 01ece63d42)
When a `SyntaxError` in the expression part of a fstring is found,
the filename attribute of the `SyntaxError` is always `<fstring>`.
With this commit, it gets changed to always have the name of the file
the fstring resides in.
Co-authored-by: Pablo Galindo <Pablogsal@gmail.com>.
(cherry picked from commit f7b1e46156)
The error message, generated for a non-parenthesized generator expression
in function calls, was still the generic `invalid syntax`, when the generator expression wasn't appearing as the first argument in the call. With this patch, even on input like `f(a, b, c for c in d, e)`, the correct error message gets produced.
(cherry picked from commit ae14583302)
Co-authored-by: Lysandros Nikolaou <lisandrosnik@gmail.com>
The following improvements are implemented in this commit:
- `p->error_indicator` is set, in case malloc or realloc fail.
- Avoid memory leaks in the case that realloc fails.
- Call `PyErr_NoMemory()` instead of `PyErr_Format()`, because it requires no memory.
Co-authored-by: Pablo Galindo <Pablogsal@gmail.com>
This commit fixes the new parser to disallow invalid targets in the
following scenarios:
- Augmented assignments must only accept a single target (Name,
Attribute or Subscript), but no tuples or lists.
- `except` clauses should only accept a single `Name` as a target.
Co-authored-by: Pablo Galindo <Pablogsal@gmail.com>
This commit fixes SyntaxError locations when the caret is not displayed,
by doing the following:
- `col_number` always gets set to the location of the offending
node/expr. When no caret is to be displayed, this gets achieved
by setting the object holding the error line to None.
- Introduce a new function `_PyPegen_raise_error_known_location`,
which can be called, when an arbitrary `lineno`/`col_offset`
needs to be passed. This function then gets used in the grammar
(through some new macros and inline functions) so that SyntaxError
locations of the new parser match that of the old.
With the new parser, the error message contains always the trailing
newlines, causing the comparison of the repr of the error messages
in codeop to fail. This commit makes the new parser mirror the old parser's
behaviour regarding trailing newlines.
This is for the C generator:
- Disallow rule and variable names starting with `_`
- Rename most local variable names generated by the parser to start with `_`
Exceptions:
- Renaming `p` to `_p` will be a separate PR
- There are still some names that might clash, e.g.
- anything starting with `Py`
- C reserved words (`if` etc.)
- Macros like `EXTRA` and `CHECK`
When parsing something like `f(g()=2)`, where the name of a default arg
is not a NAME, but an arbitrary expression, a specialised error message
is emitted.
When parsing a string with an invalid escape, the old parser used to
point to the beginning of the invalid string. This commit changes the new
parser to match that behaviour, since it's currently pointing to the
end of the string (or to be more precise, to the beginning of the next
token).
Pablo Galindo
Miss Islington (bot)
Guido van Rossum
Lysandros Nikolaou
Serhiy Storchaka
Shantanu