Commit Graph

206 Commits

Author SHA1 Message Date
Jelle Zijlstra 42b2c9d78d
gh-120108: Fix deepcopying of AST trees with .parent attributes (#120114) 2024-06-25 08:12:11 -07:00
Kirill Podoprigora b8fb369193
gh-102797: Add more code snippets in test_ast (#102798)
Co-authored-by: Terry Jan Reedy <tjreedy@udel.edu>
Co-authored-by: Nineteendo <nineteendo19d0@gmail.com>
2024-06-19 09:51:43 +02:00
Alex Waygood 008bc04dcb
gh-119562: Remove AST nodes deprecated since Python 3.8 (#119563) 2024-05-26 12:34:48 +00:00
Batuhan Taskaya d065edfb66
gh-60191: Implement ast.compare (#19211)
* bpo-15987: Implement ast.compare

Add a compare() function that compares two ASTs for structural equality. There are two set of attributes on AST node objects, fields and attributes. The fields are always compared, since they represent the actual structure of the code. The attributes can be optionally be included in the comparison. Attributes capture things like line numbers of column offsets, so comparing them involves test whether the layout of the program text is the same. Since whitespace seems inessential for comparing ASTs, the default is to compare fields but not attributes.

ASTs are just Python objects that can be modified in arbitrary ways. The API for ASTs is under-specified in the presence of user modifications to objects. The comparison respects modifications to fields and attributes, and to _fields and _attributes attributes. A user could create obviously malformed objects, and the code will probably fail with an AttributeError when that happens. (For example, adding "spam" to _fields but not adding a "spam" attribute to the object.) 

Co-authored-by: Jeremy Hylton <jeremy@alum.mit.edu>
2024-05-22 01:39:26 +00:00
Jelle Zijlstra 68fbc00dc8
gh-118851: Default ctx arguments to AST constructors to Load() (#118854)
Co-authored-by: Alex Waygood <Alex.Waygood@Gmail.com>
2024-05-09 15:30:14 -07:00
Jelle Zijlstra e0422198fb
gh-117486: Improve behavior for user-defined AST subclasses (#118212)
Now, such classes will no longer require changes in Python 3.13 in the normal case.
The test suite for robotframework passes with no DeprecationWarnings under this PR.

I also added a new DeprecationWarning for the case where `_field_types` exists
but is incomplete, since that seems likely to indicate a user mistake.
2024-05-06 15:57:27 -07:00
Jelle Zijlstra ca269e58c2
gh-116126: Implement PEP 696 (#116129)
Co-authored-by: Alex Waygood <Alex.Waygood@Gmail.com>
Co-authored-by: Bénédikt Tran <10796600+picnixz@users.noreply.github.com>
Co-authored-by: Shantanu <12621235+hauntsaninja@users.noreply.github.com>
2024-05-03 06:17:32 -07:00
Kirill Podoprigora aa8f6d2708
gh-118374: test_ast: Add ``ctx`` argument to ``ast.Name`` calls (#118375) 2024-04-29 05:38:46 +00:00
Nikita Sobolev 692e902c74
gh-116023: Add `show_empty=False` to `ast.dump` (#116037)
Co-authored-by: Carl Meyer <carl@oddbird.net>
2024-04-24 11:02:38 +03:00
Erlend E. Aasland ca62ffd1a5
gh-116303: Skip tests if C recursion limit is unavailable (GH-117368)
The test suite fetches the C recursion limit from the _testcapi
extension module. Test extension modules can be disabled using the
--disable-test-modules configure option.
2024-04-08 14:45:25 +02:00
Jelle Zijlstra 4c71d51a4b
gh-117266: Fix crashes on user-created AST subclasses (GH-117276)
Fix crashes on user-created AST subclasses
2024-03-28 11:30:31 +01:00
Jelle Zijlstra ed4dfd8825
gh-105858: Improve AST node constructors (#105880)
Demonstration:

>>> ast.FunctionDef.__annotations__
{'name': <class 'str'>, 'args': <class 'ast.arguments'>, 'body': list[ast.stmt], 'decorator_list': list[ast.expr], 'returns': ast.expr | None, 'type_comment': str | None, 'type_params': list[ast.type_param]}
>>> ast.FunctionDef()
<stdin>:1: DeprecationWarning: FunctionDef.__init__ missing 1 required positional argument: 'name'. This will become an error in Python 3.15.
<stdin>:1: DeprecationWarning: FunctionDef.__init__ missing 1 required positional argument: 'args'. This will become an error in Python 3.15.
<ast.FunctionDef object at 0x101959460>
>>> node = ast.FunctionDef(name="foo", args=ast.arguments())
>>> node.decorator_list
[]
>>> ast.FunctionDef(whatever="you want", name="x", args=ast.arguments())
<stdin>:1: DeprecationWarning: FunctionDef.__init__ got an unexpected keyword argument 'whatever'. Support for arbitrary keyword arguments is deprecated and will be removed in Python 3.15.
<ast.FunctionDef object at 0x1019581f0>
2024-02-27 18:13:03 -08:00
Alex Waygood 7a3518e43a
gh-115881: Ensure `ast.parse()` parses conditional context managers even with low `feature_version` passed (#115920) 2024-02-26 09:22:09 +00:00
Mark Shannon 17b73ab99e
GH-113655: Lower the C recursion limit on various platforms (GH-113944) 2024-01-16 09:32:01 +00:00
Victor Stinner bd89bca9e2
gh-111798: Use lower Py_C_RECURSION_LIMIT in debug mode (#112124)
* Run again test_ast_recursion_limit() on WASI platform.
* Add _testinternalcapi.get_c_recursion_remaining().
* Fix test_ast and test_sys_settrace: test_ast_recursion_limit() and
  test_trace_unpack_long_sequence() now adjust the maximum recursion
  depth depending on the the remaining C recursion.
2023-11-16 13:52:33 +00:00
Tomas R 453e96e302
gh-111420: Allow type comments in parenthesized `with` statements (#111468) 2023-10-31 21:02:42 +00:00
Serhiy Storchaka f3ba0a74cd
gh-108416: Mark slow test methods with @requires_resource('cpu') (GH-108421)
Only mark tests which spend significant system or user time,
by itself or in subprocesses.
2023-09-02 07:45:34 +03:00
Irit Katriel 2dfbd4f36d
gh-108113: Make it possible to optimize an AST (#108282) 2023-08-23 09:01:17 +01:00
Irit Katriel 10a91d7e98
gh-108113: Make it possible to create an optimized AST (#108154) 2023-08-21 16:31:30 +00:00
Mark Shannon fa45958450
GH-107263: Increase C stack limit for most functions, except `_PyEval_EvalFrameDefault()` (GH-107535)
* Set C recursion limit to 1500, set cost of eval loop to 2 frames, and compiler mutliply to 2.
2023-08-04 10:10:29 +01:00
Brandt Bucher a4056c8f9c
GH-105588: Add missing error checks to some obj2ast_* converters (GH-105589) 2023-06-15 15:45:13 -07:00
Jelle Zijlstra 77d2579586
gh-104799: Default missing lists in AST to the empty list (#104834)
Co-authored-by: Alex Waygood <Alex.Waygood@Gmail.com>
2023-06-01 18:39:39 -07:00
Jelle Zijlstra ba73473f4c
gh-104799: Move location of type_params AST fields (#104828)
Co-authored-by: Alex Waygood <Alex.Waygood@Gmail.com>
2023-05-26 05:54:37 -07:00
Jelle Zijlstra 24d8b88420
gh-103763: Implement PEP 695 (#103764)
This implements PEP 695, Type Parameter Syntax. It adds support for:

- Generic functions (def func[T](): ...)
- Generic classes (class X[T](): ...)
- Type aliases (type X = ...)
- New scoping when the new syntax is used within a class body
- Compiler and interpreter changes to support the new syntax and scoping rules 

Co-authored-by: Marc Mueller <30130371+cdce8p@users.noreply.github.com>
Co-authored-by: Eric Traut <eric@traut.com>
Co-authored-by: Larry Hastings <larry@hastings.org>
Co-authored-by: Alex Waygood <Alex.Waygood@Gmail.com>
2023-05-15 20:36:23 -07:00
Irit Katriel 8a3702f0c7
gh-104482: Fix error handling bugs in ast.c (#104483) 2023-05-15 21:53:55 +01:00
Alex Waygood 376137f6ec
gh-90953: Emit deprecation warnings for `ast` features deprecated in Python 3.8 (#104199)
`ast.Num`, `ast.Str`, `ast.Bytes`, `ast.Ellipsis` and `ast.NameConstant` now all emit deprecation warnings on import, access, instantation or `isinstance()` checks.

Co-authored-by: Serhiy Storchaka <storchaka@gmail.com>
2023-05-06 16:48:07 +00:00
Tian Gao 36860134a9
gh-103285: Rewrite _splitlines_no_ff to improve performance (#103307) 2023-04-23 23:03:49 -06:00
Pablo Galindo Salgado 1ef61cf71a
gh-102856: Initial implementation of PEP 701 (#102855)
Co-authored-by: Lysandros Nikolaou <lisandrosnik@gmail.com>
Co-authored-by: Batuhan Taskaya <isidentical@gmail.com>
Co-authored-by: Marta Gómez Macías <mgmacias@google.com>
Co-authored-by: sunmy2019 <59365878+sunmy2019@users.noreply.github.com>
2023-04-19 11:18:16 -05:00
Nikita Sobolev bb396eece4
gh-101821: Test coverage for `ast.main` function (#101822) 2023-03-11 11:10:52 -08:00
penguin_wwy 753fc8a5d6
gh-101632: Add the new RETURN_CONST opcode (#101633) 2023-02-07 22:32:21 +00:00
Eclips4 28db978d7f
gh-101229: Add tests for aliases of imported names (#101230) 2023-01-30 11:49:06 -08:00
Nikita Sobolev c1c5882359
gh-100518: Add tests for `ast.NodeTransformer` (#100521) 2023-01-21 21:44:41 +00:00
Nikita Sobolev 2e80c2a976
gh-100882: Improve `test_pickling` case in `test_ast.py` (#100883) 2023-01-09 19:20:25 +00:00
Victor Stinner cd67c1bb30
test_ast uses infinite_recursion() to prevent crash (#100104)
test.test_ast_recursion_limit() now uses infinite_recursion() of
test.support to prevent crashes on debug builds.

Before this change, the test crashed on ARM64 Windows 3.x buildbot
worker which builds Python in debug mode.
2022-12-08 09:26:38 +01:00
Batuhan Taskaya 1acdfec359
gh-99341: Cover type ignore nodes when incrementing line numbers (GH-99422) 2022-11-22 02:41:14 -08:00
Mark Shannon 76449350b3
GH-91079: Decouple C stack overflow checks from Python recursion checks. (GH-96510) 2022-10-05 01:34:03 +01:00
Pablo Galindo Salgado aab01e3524
gh-96670: Raise SyntaxError when parsing NULL bytes (#97594) 2022-09-27 23:23:42 +01:00
Nikita Sobolev 2c7d2e8d46
gh-96587: Raise `SyntaxError` for PEP654 on older `feature_version` (#96588) 2022-09-05 17:54:09 +01:00
Gregory P. Smith 511ca94520
gh-95778: CVE-2020-10735: Prevent DoS by very large int() (#96499)
Integer to and from text conversions via CPython's bignum `int` type is not safe against denial of service attacks due to malicious input. Very large input strings with hundred thousands of digits can consume several CPU seconds.

This PR comes fresh from a pile of work done in our private PSRT security response team repo.

Signed-off-by: Christian Heimes [Red Hat] <christian@python.org>
Tons-of-polishing-up-by: Gregory P. Smith [Google] <greg@krypto.org>
Reviews via the private PSRT repo via many others (see the NEWS entry in the PR).

<!-- gh-issue-number: gh-95778 -->
* Issue: gh-95778
<!-- /gh-issue-number -->

I wrote up [a one pager for the release managers](https://docs.google.com/document/d/1KjuF_aXlzPUxTK4BMgezGJ2Pn7uevfX7g0_mvgHlL7Y/edit#). Much of that text wound up in the Issue. Backports PRs already exist. See the issue for links.
2022-09-02 09:35:08 -07:00
Shantanu a965db37f2
gh-94996: Disallow lambda pos only params with feature_version < (3, 8) (GH-95934) 2022-08-12 20:41:02 +02:00
Shantanu b5e3ea2862
gh-94996: Disallow parsing pos only params with feature_version < (3, 8) (GH-94997) 2022-08-12 19:27:50 +02:00
Shantanu a5dde0fe4f
gh-95066: ast: Replace assert with ValueError (GH-95072)
Co-authored-by: blurb-it[bot] <43283697+blurb-it[bot]@users.noreply.github.com>
2022-07-26 11:43:09 +02:00
Pablo Galindo Salgado 0047447294
gh-95185: Check recursion depth in the AST constructor (#95186)
Co-authored-by: Serhiy Storchaka <storchaka@gmail.com>
2022-07-24 15:58:52 +01:00
Shantanu 0daba82221
gh-94949: Disallow parsing parenthesised ctx mgr with old feature_version (#94950)
* gh-94949: Disallow parsing parenthesised ctx manager with old feature_version

* 📜🤖 Added by blurb_it.

* Allow it with feature_version=(3, 9) as well

Co-authored-by: blurb-it[bot] <43283697+blurb-it[bot]@users.noreply.github.com>
2022-07-18 22:10:49 +01:00
Shantanu ae0be5a53b
gh-94947: Disallow parsing walrus with feature_version < (3, 8) (#94948)
* gh-94947: Disallow parsing walrus with feature_version < (3, 8)

* oops, commit the parser

* 📜🤖 Added by blurb_it.

Co-authored-by: blurb-it[bot] <43283697+blurb-it[bot]@users.noreply.github.com>
2022-07-18 10:20:12 +01:00
Victor Stinner e87ada48a9
Run Tools/scripts/reindent.py (#94225)
Reindent files which were not properly formatted (PEP 8: 4 spaces).

Remove also some trailing spaces.
2022-06-26 10:34:06 +02:00
Pablo Galindo Salgado 705eaec28f
gh-92597: Ensure that AST nodes without explicit end positions can be compiled (GH-93359) 2022-06-01 00:00:47 +01:00
Pablo Galindo Salgado 5893b5db98
gh-93351: Ensure the position information in AST nodes created by the parser is always consistent (GH-93352) 2022-05-30 19:30:15 +01:00
Matthew Rahtz e8e737bcf6
bpo-43224: Implement PEP 646 grammar changes (GH-31018)
Co-authored-by: Jelle Zijlstra <jelle.zijlstra@gmail.com>
2022-03-26 09:55:35 -07:00
Pablo Galindo Salgado a0efc0c196
bpo-46091: Correctly calculate indentation levels for whitespace lines with continuation characters (GH-30130) 2022-01-25 22:12:14 +00:00