Commit Graph

1267 Commits

Author SHA1 Message Date
Victor Stinner 4a943c3251
gh-125196: Use PyUnicodeWriter in parser (#125271)
Replace the private _PyUnicodeWriter API with the public
PyUnicodeWriter API in _PyPegen_concatenate_strings().
2024-10-12 09:28:34 +02:00
Sam Gross 427dcf24de
gh-125268: Use static string for "1e309" in AST (#125272)
When formatting the AST as a string, infinite values are replaced by
1e309, which evaluates to infinity. The initialization of this string
replacement was not thread-safe in the free threading build.
2024-10-10 16:21:29 -04:00
Victor Stinner b9a8ca0a6a
gh-115754: Use Py_GetConstant(Py_CONSTANT_EMPTY_STR) (#125194)
Replace PyUnicode_New(0, 0), PyUnicode_FromString("")
and PyUnicode_FromStringAndSize("", 0)
with Py_GetConstant(Py_CONSTANT_EMPTY_STR).
2024-10-09 17:15:23 +02:00
Victor Stinner 6a39e96ab8
gh-115754: Use Py_GetConstant(Py_CONSTANT_EMPTY_BYTES) (#125195)
Replace PyBytes_FromString("") and PyBytes_FromStringAndSize("", 0)
with Py_GetConstant(Py_CONSTANT_EMPTY_BYTES).
2024-10-09 17:12:11 +02:00
Tomas R. a1be83dae3
gh-125010: Fix `use-after-free` in AST `repr()` (#125015) 2024-10-06 12:46:03 -07:00
Rigel Di Scala 39c859f6ff
gh-122951: Simplify the grammar of the assignment rule (#124998) 2024-10-06 11:55:56 +02:00
Victor Stinner 6c7d5c6415
gh-111178: Fix function signatures in Python-ast.c (#124942) 2024-10-04 11:59:08 +02:00
efimov-mikhail 1f9025a4e7
gh-124889: Remove redundant artificial rules in PEG parser (#124893)
Cache in C PEG-generator reworked:
we save artificial rules in cache by Node string representation as a key instead of Node object itself.
As a result total count of artificial rules in parsers.c is lowered from 283 to 170.
More natural number ordering is used for the names of artificial rules.

Auxiliary method CCallMakerVisitor._generate_artificial_rule_call is added.
Its purpose is abstracting work with artificial rules cache.

Explicit using of "is_repeat1" kwarg is added to visit_Repeat0 and visit_Repeat1 methods.
Its slightly improve code readabitily.
2024-10-03 13:58:56 +01:00
Tomas R 21d2a9ab2f
gh-116022: Improve `repr()` of AST nodes (#117046)
Co-authored-by: AN Long <aisk@users.noreply.github.com>
Co-authored-by: Jelle Zijlstra <jelle.zijlstra@gmail.com>
Co-authored-by: Alex Waygood <Alex.Waygood@Gmail.com>
Co-authored-by: Bénédikt Tran <10796600+picnixz@users.noreply.github.com>
2024-09-18 10:28:22 -07:00
Victor Stinner f9fa6ba4f8
gh-124064: Fix -Wconversion warnings in Parser/string_parser.c (#124204)
Fix integer overflow check in decode_unicode_with_escapes(): use
PY_SSIZE_T_MAX instead of SIZE_MAX.
2024-09-18 19:10:56 +02:00
Victor Stinner 3aff1d0260
gh-124064: Fix -Wconversion warnings in Parser/pegen.c (#124181) 2024-09-17 15:58:43 +00:00
Sam Gross 0c080d7c77
gh-123321: Make Parser/myreadline.c locking safe in free-threaded build (#123690)
Use a `PyMutex` to avoid the race in mutex initialization. Use relaxed
atomics to avoid the data race on reading `_PyOS_ReadlineTState` when
checking for re-entrant calls.
2024-09-06 15:07:08 -04:00
Bar Harel a4562fedad
gh-123321: Fix Parser/myreadline.c to prevent a segfault during a multi-threaded race (#123323) 2024-09-04 17:21:30 +02:00
sobolevn 23f159ae71
gh-123562: Improve `SyntaxError` message for `case ... as a.b` (#123563) 2024-09-02 13:11:44 +02:00
sobolevn e451a8937d
gh-123440: Improve error message for `except as` used with not a name (#123442) 2024-08-30 17:21:59 +01:00
Pablo Galindo Salgado adc5190014
gh-123229: Fix valgrind warning by initializing the f-string buffers to 0 in the tokenizer (#123263)
Signed-off-by: Pablo Galindo <pablogsal@gmail.com>
2024-08-23 12:33:45 +00:00
Lysandros Nikolaou ce0d66c8d2
gh-122581: Avoid data races when collecting parser statistics (#122694) 2024-08-06 13:29:57 +02:00
Pablo Galindo Salgado db2d8b6db1
gh-122300: Preserve AST nodes for format specifiers with single elements (#122308) 2024-07-26 16:29:41 +00:00
Serhiy Storchaka 6c09b8de5c
gh-122270: Fix typos in the Py_DEBUG macro name (GH-122271) 2024-07-25 14:04:22 +03:00
Xie Yanbo 2a5d1eb707
Fix typos in comments and exception message (#122147) 2024-07-23 14:34:14 +05:30
Pablo Galindo Salgado 2009e25e26
gh-122026: Fix identification of mismatched parentheses inside f-strings (#122028) 2024-07-19 19:08:08 +02:00
Serhiy Storchaka 1a0c7b9ba4
gh-121905: Consistently use "floating-point" instead of "floating point" (GH-121907) 2024-07-19 08:06:02 +00:00
Pablo Galindo Salgado c46d64e0ef
gh-121130: Fix f-string format specifiers with debug expressions (#121150) 2024-07-16 19:57:22 +01:00
Jelle Zijlstra 58e8cf2bb6
gh-121332: Make AST node constructor check _attributes instead of hardcoding attributes (#121334) 2024-07-11 14:34:53 +00:00
Bénédikt Tran 9728ead361
gh-121141: add support for `copy.replace` to AST nodes (#121162) 2024-07-03 20:10:54 -07:00
Victor Stinner 769aea3329
gh-120155: Fix Coverity issue in parse_string() (#120997) 2024-06-25 17:53:24 +01:00
Jelle Zijlstra 42b2c9d78d
gh-120108: Fix deepcopying of AST trees with .parent attributes (#120114) 2024-06-25 08:12:11 -07:00
Lysandros Nikolaou 348184845a
gh-120956: Avoid comparison of int to Py_ssize_t in parser (#120959) 2024-06-24 18:13:02 +02:00
Steve Dower e731554337
Fixes loop variables to be the same types as their limit (GH-120958) 2024-06-24 17:11:47 +01:00
Pablo Galindo Salgado ac61d58db0
gh-119521: Rename IncompleteInputError to _IncompleteInputError and remove from public API/ABI (GH-119680)
Signed-off-by: Pablo Galindo <pablogsal@gmail.com>
Co-authored-by: Petr Viktorin <encukou@gmail.com>
2024-06-24 14:08:12 +02:00
Petr Viktorin 6f1d448bc1
gh-113993: Allow interned strings to be mortal, and fix related issues (GH-120520)
* Add an InternalDocs file describing how interning should work and how to use it.

* Add internal functions to *explicitly* request what kind of interning is done:
  - `_PyUnicode_InternMortal`
  - `_PyUnicode_InternImmortal`
  - `_PyUnicode_InternStatic`

* Switch uses of `PyUnicode_InternInPlace` to those.

* Disallow using `_Py_SetImmortal` on strings directly.
  You should use `_PyUnicode_InternImmortal` instead:
  - Strings should be interned before immortalization, otherwise you're possibly
    interning a immortalizing copy.
  - `_Py_SetImmortal` doesn't handle the `SSTATE_INTERNED_MORTAL` to
    `SSTATE_INTERNED_IMMORTAL` update, and those flags can't be changed in
    backports, as they are now part of public API and version-specific ABI.

* Add private `_only_immortal` argument for `sys.getunicodeinternedsize`, used in refleak test machinery.

* Make sure the statically allocated string singletons are unique. This means these sets are now disjoint:
  - `_Py_ID`
  - `_Py_STR` (including the empty string)
  - one-character latin-1 singletons

  Now, when you intern a singleton, that exact singleton will be interned.

* Add a `_Py_LATIN1_CHR` macro, use it instead of `_Py_ID`/`_Py_STR` for one-character latin-1 singletons everywhere (including Clinic).

* Intern `_Py_STR` singletons at startup.

* For free-threaded builds, intern `_Py_LATIN1_CHR` singletons at startup.

* Beef up the tests. Cover internal details (marked with `@cpython_only`).

* Add lots of assertions

Co-Authored-By: Eric Snow <ericsnowcurrently@gmail.com>
2024-06-21 17:19:31 +02:00
Mark Jason Dominus (陶敏修) bd8c1f97e1
gh-94808: Reorganize _make_posargs and mark unused code (GH-119227)
* Reorganize four-way if-elsif-elsif-elsif as nested if-elses
* Mark unused branch in _make_posargs

`names_with_default` is never `NULL`, even if there are no names with
defaults.  In that case it points to a structure with `size` zero.

Rather than eliminating the branch, we leave it behind with an `assert(0)`
in case a future change to the grammar exercises the branch.
2024-06-04 14:59:56 +02:00
Petr Viktorin 31a4fb3c74
gh-119724: Revert "bpo-45759: Better error messages for non-matching 'elif'/'else' statements (#29513)" (#119974)
This reverts commit 1c8f912ebd.
2024-06-03 18:10:15 -07:00
Petr Viktorin 48f21b3631
gh-118235: Move RAISE_SYNTAX_ERROR actions to invalid rules and make sure they stay there (GH-119731)
The Full Grammar specification in the docs omits rule actions, so grammar rules that raise a syntax error looked like valid syntax.
This was solved in ef940de by hiding those rules in the custom syntax highlighter.

This moves all syntax-error alternatives to invalid rules, adds a validator that ensures that actions containing RAISE_SYNTAX_ERROR are in invalid rules, and reverts the syntax highlighter hack.
2024-05-30 09:27:32 +02:00
Lysandros Nikolaou d87b015106
gh-119118: Fix performance regression in tokenize module (#119615)
* gh-119118: Fix performance regression in tokenize module

- Cache line object to avoid creating a Unicode object
  for all of the tokens in the same line.
- Speed up byte offset to column offset conversion by using the
  smallest buffer possible to measure the difference.

Co-authored-by: Pablo Galindo <pablogsal@gmail.com>
2024-05-28 19:17:49 +00:00
Jelle Zijlstra 68fbc00dc8
gh-118851: Default ctx arguments to AST constructors to Load() (#118854)
Co-authored-by: Alex Waygood <Alex.Waygood@Gmail.com>
2024-05-09 15:30:14 -07:00
Nikita Sobolev b60d4c0d53
gh-118090: Improve error message for empty type param brackets (GH-118091) 2024-05-07 14:01:06 +02:00
Jelle Zijlstra e0422198fb
gh-117486: Improve behavior for user-defined AST subclasses (#118212)
Now, such classes will no longer require changes in Python 3.13 in the normal case.
The test suite for robotframework passes with no DeprecationWarnings under this PR.

I also added a new DeprecationWarning for the case where `_field_types` exists
but is incomplete, since that seems likely to indicate a user mistake.
2024-05-06 15:57:27 -07:00
Brett Simmers c2627d6eea
gh-116322: Add Py_mod_gil module slot (#116882)
This PR adds the ability to enable the GIL if it was disabled at
interpreter startup, and modifies the multi-phase module initialization
path to enable the GIL when loading a module, unless that module's spec
includes a slot indicating it can run safely without the GIL.

PEP 703 called the constant for the slot `Py_mod_gil_not_used`; I went
with `Py_MOD_GIL_NOT_USED` for consistency with gh-104148.

A warning will be issued up to once per interpreter for the first
GIL-using module that is loaded. If `-v` is given, a shorter message
will be printed to stderr every time a GIL-using module is loaded
(including the first one that issues a warning).
2024-05-03 11:30:55 -04:00
Jelle Zijlstra ca269e58c2
gh-116126: Implement PEP 696 (#116129)
Co-authored-by: Alex Waygood <Alex.Waygood@Gmail.com>
Co-authored-by: Bénédikt Tran <10796600+picnixz@users.noreply.github.com>
Co-authored-by: Shantanu <12621235+hauntsaninja@users.noreply.github.com>
2024-05-03 06:17:32 -07:00
David Rubin 9b280ab0ab
gh-116988: Remove duplicates of `annotated_rhs` in the Grammar (#117004) 2024-04-24 18:16:06 +01:00
Nikita Sobolev de1f686827
gh-118082: Improve `import` without names syntax error message (#118083) 2024-04-23 13:00:52 +01:00
Grigoriev Semyon c97d3af239
gh-109120: Fix syntax error in handlinh of incorrect star expressions (#117444) 2024-04-02 11:42:58 +01:00
Jelle Zijlstra 4c71d51a4b
gh-117266: Fix crashes on user-created AST subclasses (GH-117276)
Fix crashes on user-created AST subclasses
2024-03-28 11:30:31 +01:00
Pablo Galindo Salgado 61599a48f5
bpo-24612: Improve syntax error for 'not' after an operator (GH-28170)
Co-authored-by: Lysandros Nikolaou <lisandrosnik@gmail.com>
2024-03-26 10:30:46 +01:00
Serhiy Storchaka 72d3cc94cd
gh-116437: Use new C API PyDict_Pop() to simplify the code (GH-116438) 2024-03-07 11:21:08 +02:00
Jelle Zijlstra ed4dfd8825
gh-105858: Improve AST node constructors (#105880)
Demonstration:

>>> ast.FunctionDef.__annotations__
{'name': <class 'str'>, 'args': <class 'ast.arguments'>, 'body': list[ast.stmt], 'decorator_list': list[ast.expr], 'returns': ast.expr | None, 'type_comment': str | None, 'type_params': list[ast.type_param]}
>>> ast.FunctionDef()
<stdin>:1: DeprecationWarning: FunctionDef.__init__ missing 1 required positional argument: 'name'. This will become an error in Python 3.15.
<stdin>:1: DeprecationWarning: FunctionDef.__init__ missing 1 required positional argument: 'args'. This will become an error in Python 3.15.
<ast.FunctionDef object at 0x101959460>
>>> node = ast.FunctionDef(name="foo", args=ast.arguments())
>>> node.decorator_list
[]
>>> ast.FunctionDef(whatever="you want", name="x", args=ast.arguments())
<stdin>:1: DeprecationWarning: FunctionDef.__init__ got an unexpected keyword argument 'whatever'. Support for arbitrary keyword arguments is deprecated and will be removed in Python 3.15.
<ast.FunctionDef object at 0x1019581f0>
2024-02-27 18:13:03 -08:00
Pablo Galindo Salgado 015b97d19a
gh-115823: Calculate correctly error locations when dealing with implicit encodings (#115824) 2024-02-26 12:57:09 +00:00
Alex Waygood 7a3518e43a
gh-115881: Ensure `ast.parse()` parses conditional context managers even with low `feature_version` passed (#115920) 2024-02-26 09:22:09 +00:00
Pablo Galindo Salgado 39d102c2ee
gh-113744: Add a new IncompleteInputError exception to improve incomplete input detection in the codeop module (#113745)
Signed-off-by: Pablo Galindo <pablogsal@gmail.com>
2024-01-30 16:21:30 +00:00