Fix a leak and subsequent crash in parsetok.c caused by realloc misuse on a rare codepath.
Realloc returns a null pointer on failure, and then growable_comment_array_deallocate crashes later when it dereferences it.
* Re-add removed classes Suite, slice, Param, AugLoad and AugStore.
* Add docstrings for dummy classes.
* Add docstrings for attribute aliases.
* Set __module__ to "ast" instead of "_ast".
* Remove the slice type.
* Make Slice a kind of the expr type instead of the slice type.
* Replace ExtSlice(slices) with Tuple(slices, Load()).
* Replace Index(value) with a value itself.
All non-terminal nodes in AST for expressions are now of the expr type.
The Py_FatalError() function is replaced with a macro which logs
automatically the name of the current function, unless the
Py_LIMITED_API macro is defined.
Changes:
* Add _Py_FatalErrorFunc() function.
* Remove the function name from the message of Py_FatalError() calls
which included the function name.
* Update tests.
The AST "Suite" node is no longer used and it can be removed from the ASDL definition and related structures (compiler, visitors, ...).
Co-Authored-By: Victor Stinner <vstinner@python.org>
Co-authored-by: Brett Cannon <54418+brettcannon@users.noreply.github.com>
Co-authored-by: Pablo Galindo <Pablogsal@gmail.com>
The function PyTokenizer_FromUTF8 from Parser/tokenizer.c had a comment:
/* XXX: constify members. */
This patch addresses that.
In the tok_state struct:
* end and start were non-const but could be made const
* str and input were const but should have been non-const
Changes to support this include:
* decode_str() now returns a char * since it is allocated.
* PyTokenizer_FromString() and PyTokenizer_FromUTF8() each creates a
new char * for an allocate string instead of reusing the input
const char *.
* PyTokenizer_Get() and tok_get() now take const char ** arguments.
* Various local vars are const or non-const accordingly.
I was able to remove five casts that cast away constness.
Summary: This mostly migrates Python-ast.c to PEP384 and removes all statics from the whole file. This modifies the generator itself that generates the Python-ast.c. It leaves in the usage of _PyObject_LookupAttr even though it's not fully PEP384 compatible (this could always be shimmed in by anyone who needs it).
This is the converse of GH-15353 -- in addition to plenty of
scripts in the tree that are marked with the executable bit
(and so can be directly executed), there are a few that have
a leading `#!` which could let them be executed, but it doesn't
do anything because they don't have the executable bit set.
Here's a command which finds such files and marks them. The
first line finds files in the tree with a `#!` line *anywhere*;
the next-to-last step checks that the *first* line is actually of
that form. In between we filter out files that already have the
bit set, and some files that are meant as fragments to be
consumed by one or another kind of preprocessor.
$ git grep -l '^#!' \
| grep -vxFf <( \
git ls-files --stage \
| perl -lane 'print $F[3] if (!/^100644/)' \
) \
| grep -ve '\.in$' -e '^Doc/includes/' \
| while read f; do
head -c2 "$f" | grep -qxF '#!' \
&& chmod a+x "$f"; \
done
* Refactor Parser/pgen and add documentation and explanations
To improve the readability and maintainability of the parser
generator perform the following transformations:
* Separate the metagrammar parser in its own class to simplify
the parser generator logic.
* Create separate classes for DFAs and NFAs and move methods that
act exclusively on them from the parser generator to these
classes.
* Add docstrings and comment documenting the process to go from
the grammar file into NFAs and then DFAs. Detail some of the
algorithms and give some background explanations of some concepts
that will helps readers not familiar with the parser generation
process.
* Select more descriptive names for some variables and variables.
* PEP8 formatting and quote-style homogenization.
The output of the parser generator remains the same (Include/graminit.h
and Python/graminit.c remain untouched by running the new parser generator).
When using the "=" debug functionality of f-strings, use another Constant node (or a merged constant node) instead of adding expr_text to the FormattedValue node.
This disallows things like `# type: ignoreé`, which seems wrong.
Also switch to using Py_ISALNUM for the alnum check, for consistency
with other code (and maybe correctness re: locale issues?).
https://bugs.python.org/issue36878
GH-13238 made extra text after a # type: ignore accepted by the parser.
This finishes the job and actually plumbs the extra text through the
parser and makes it available in the AST.
This makes the parser consistent with the tokenize module (already the case
in `pypy`).
sample
------
```python
x = 5\
```
before
------
```console
$ python3 t.py
$ python3 -mtokenize t.py
t.py:2:0: error: EOF in multi-line statement
```
after
-----
```console
$ ./python t.py
File "t.py", line 3
x = 5\
^
SyntaxError: unexpected EOF while parsing
$ ./python -m tokenize t.py
t.py:2:0: error: EOF in multi-line statement
```
https://bugs.python.org/issue2180
In the parser, when using the type_comments=True option, recognize
a TYPE_IGNORE as anything containing `# type: ignore` followed by
a non-alphanumeric character. This is to allow ignores such as
`# type: ignore[E1000]`.
If a "=" is specified a the end of an f-string expression, the f-string will evaluate to the text of the expression, followed by '=', followed by the repr of the value of the expression.
This commit contains the implementation of PEP570: Python positional-only parameters.
* Update Grammar/Grammar with new typedarglist and varargslist
* Regenerate grammar files
* Update and regenerate AST related files
* Update code object
* Update marshal.c
* Update compiler and symtable
* Regenerate importlib files
* Update callable objects
* Implement positional-only args logic in ceval.c
* Regenerate frozen data
* Update standard library to account for positional-only args
* Add test file for positional-only args
* Update other test files to account for positional-only args
* Add News entry
* Update inspect module and related tests
Pablo Galindo
Alexander Riccio
Victor Stinner
Serhiy Storchaka
Batuhan Taşkaya
xatier
Andy Lester
Brandt Bucher
Emmanuel Arias
Alex Henrie
Steve Dower
Vinay Sajip
Max Bernstein
Dong-hee Na
Eddie Elizondo
Zackery Spytz
Dino Viehland
Greg Price
Shashi Ranjan
Hansraj Das
Anthony Sottile
Victor Stinner
Jeroen Demeyer
Eric V. Smith
Steve Dower
Michael J. Sullivan
Matthias Bussonnier