Commit Graph

86 Commits

Author SHA1 Message Date
Pablo Galindo Salgado 1ef61cf71a
gh-102856: Initial implementation of PEP 701 (#102855)
Co-authored-by: Lysandros Nikolaou <lisandrosnik@gmail.com>
Co-authored-by: Batuhan Taskaya <isidentical@gmail.com>
Co-authored-by: Marta Gómez Macías <mgmacias@google.com>
Co-authored-by: sunmy2019 <59365878+sunmy2019@users.noreply.github.com>
2023-04-19 11:18:16 -05:00
Pablo Galindo Salgado f533f216e6
gh-102416: Do not memoize incorrectly loop rules in the parser (#102467) 2023-03-06 14:41:53 +01:00
Pablo Galindo Salgado 1de4395f62
gh-101046: Fix a potential memory leak in the parser when raising MemoryError (#101051) 2023-01-16 18:45:37 +00:00
Victor Stinner 4ce2a202c7
gh-99300: Use Py_NewRef() in Parser/ directory (#99330)
Replace Py_INCREF() with Py_NewRef() in C files of the Parser/
directory and in the PEG generator.
2022-11-10 15:30:05 +01:00
Victor Stinner 47e35625ff
gh-84623: Remove unused imports (#94132) 2022-06-22 19:14:27 +02:00
Victor Stinner 616fa3465d
gh-54781: Move Lib/lib2to3/tests/ to Lib/test/test_lib2to3/ (#94049)
* Move Lib/lib2to3/tests/ to Lib/test/test_lib2to3/.
* Remove Lib/test/test_lib2to3.py.
* Update imports.
* all_project_files(): use different paths and sort files
  to make the tests more reproducible.
* Update references to tests.
2022-06-21 15:21:22 +02:00
Victor Stinner 5115a16831
gh-93103: Parser uses PyConfig.parser_debug instead of Py_DebugFlag (#93106)
* Replace deprecated Py_DebugFlag with PyConfig.parser_debug in the
  parser.
* Add Parser.debug member.
* Add tok_state.debug member.
* Py_FrozenMain(): Replace Py_VerboseFlag with PyConfig.verbose.
2022-05-24 22:35:08 +02:00
Christian Heimes 137fd3d88a
gh-90473: Decrease recursion limit and skip tests on WASI (GH-92803) 2022-05-19 12:43:16 +02:00
Victor Stinner 804f2529d8
gh-91320: Use _PyCFunction_CAST() (#92251)
Replace "(PyCFunction)(void(*)(void))func" cast with
_PyCFunction_CAST(func).

Change generated by the command:

sed -i -e \
  's!(PyCFunction)(void(\*)(void)) *\([A-Za-z0-9_]\+\)!_PyCFunction_CAST(\1)!g' \
  $(find -name "*.c")
2022-05-03 21:42:14 +02:00
Jeremy Kloth 612e422c6e
bpo-46576: Speed up test_peg_generator by using a static library for shared sources (GH-32338)
Speed up test_peg_generator by using a static library for shared sources to avoid recompiling as much code.
2022-04-06 14:55:58 -07:00
Luca Chiodini 7f07b5ee9c
Update grammar_grapher with the new forced (&&) directive (#31704) 2022-03-06 15:26:28 +01:00
Pablo Galindo Salgado 390459de6d
Allow the parser to avoid nested processing of invalid rules (GH-31252) 2022-02-10 13:12:14 +00:00
Gregory P. Smith 164a017e13
bpo-46576: bpo-46524: Disable compiler optimization within test_peg_generator. (#31015)
Disable compiler optimization within test_peg_generator.

This speed up test_peg_generator by always disabling compiler
optimizations by using -O0 or equivalent when the test is building its
own C extensions.

A build not using --with-pydebug in order to speed up test execution
winds up with this test taking a very long time as it would do
repeated compilation of parser C code using the same optimization
flags as CPython was built with.

This speeds the test up 6-8x on gps-raspbian.

Also incorporate's #31017's win32 conditional and flags.

Co-authored-by: Kumar Aditya kumaraditya303
2022-02-02 12:15:16 -08:00
Pablo Galindo Salgado dd6c35761a
bpo-46110: Restore commit e9898bf153
This restores commit e9898bf153 .
2022-01-03 19:54:06 +00:00
Pablo Galindo Salgado 9d35dedc5e
Revert "bpo-46110: Add a recursion check to avoid stack overflow in the PEG parser (GH-30177)" (GH-30363)
This reverts commit e9898bf153 temporarily as we want to confirm if this commit is the cause of a slowdown at startup time.
2022-01-03 18:29:18 +00:00
Pablo Galindo Salgado e9898bf153
bpo-46110: Add a recursion check to avoid stack overflow in the PEG parser (GH-30177)
Co-authored-by: Batuhan Taskaya <isidentical@gmail.com>
2021-12-20 15:43:26 +00:00
Victor Stinner 253b7a0a9f
bpo-45866: pegen strips directory of "generated from" header (GH-29777)
"make regen-all" now produces the same output when run from a
directory other than the source tree: when building Python out of the
source tree.
2021-11-26 11:50:34 +01:00
Pablo Galindo Salgado c9c4444d9f
Refactor parser compilation units into specific components (GH-29676) 2021-11-21 01:08:50 +00:00
wim glenn 762173c670
bpo-44257: fix "assigment_expr" typo + regenerate the grammar, and remove unused imports (GH-29393)
Co-authored-by: Wim Glenn <wglenn@jumptrading.com>
2021-11-03 20:57:40 +00:00
Pablo Galindo Salgado a106343f63
bpo-45494: Fix parser crash when reporting errors involving invalid continuation characters (GH-28993)
There are two errors that this commit fixes:

* The parser was not correctly computing the offset and the string
  source for E_LINECONT errors due to the incorrect usage of strtok().
* The parser was not correctly unwinding the call stack when a tokenizer
  exception happened in rules involving optionals ('?', [...]) as we
  always make them return valid results by using the comma operator. We
  need to check first if we don't have an error before continuing.
2021-10-19 21:24:12 +02:00
Christian Clauss 682aecfdeb
Fix typos in the Tools directory (GH-28769)
Like #28744 but for the Tools directory.

[skip issue] Opening a related issue is pending python/psf-infra-meta#130

Automerge-Triggered-By: GH:pablogsal
2021-10-06 10:55:16 -07:00
Ikko Ashimine 48fadb1f19
[Tools/peg_generator/pegen/parser.py] Fix typo: s/wether/whether/ (GH-28739) 2021-10-05 23:06:19 +02:00
Pablo Galindo Salgado b01fd533fe
Extract visitors from the grammar nodes and call makers in the peg generator (GH-28172)
Simplify the peg generator logic by extracting as much visitors as possible to disentangle the flow and separate concerns.
2021-09-05 14:58:52 +01:00
Pablo Galindo Salgado 953d27261e
Update pegen to use the latest upstream developments (GH-27586) 2021-08-12 17:37:30 +01:00
Akira Nonaka aef1b58dc8
bpo-44345: Fix 'generated by' comment in parser.c (GH-26615) 2021-06-09 16:38:53 +02:00
Pablo Galindo c878a97968
bpo-44180: Fix edge cases in invalid assigment rules in the parser (GH-26283)
The invalid assignment rules are very delicate since the parser can
easily raise an invalid assignment when a keyword argument is provided.
As they are very deep into the grammar tree, is very difficult to
specify in which contexts these rules can be used and in which don't.
For that, we need to use a different version of the rule that doesn't do
error checking in those situations where we don't want the rule to raise
(keyword arguments and generator expressions).

We also need to check if we are in left-recursive rule, as those can try
to eagerly advance the parser even if the parse will fail at the end of
the expression. Failing to do this allows the parser to start parsing a
call as a tuple and incorrectly identify a keyword argument as an
invalid assignment, before it realizes that it was not a tuple after all.
2021-05-21 18:34:54 +01:00
Pablo Galindo b0544ba77c
bpo-38605: Revert making 'from __future__ import annotations' the default (GH-25490)
This reverts commits 044a1048ca and 1be456ae9d, adapting the code to changes that happened after it.
2021-04-21 12:41:19 +01:00
Pablo Galindo b280248be8
bpo-43822: Improve syntax errors for missing commas (GH-25377) 2021-04-15 21:38:45 +01:00
Pablo Galindo 58bafe42ab
Sanitize macros and debug functions in pegen.c (GH-25291) 2021-04-09 01:17:31 +01:00
Victor Stinner 8370e07e1e
bpo-43244: Remove the pyarena.h header (GH-25007)
Remove the pyarena.h header file with functions:

* PyArena_New()
* PyArena_Free()
* PyArena_Malloc()
* PyArena_AddPyObject()

These functions were undocumented, excluded from the limited C API,
and were only used internally by the compiler.

Add pycore_pyarena.h header. Rename functions:

* PyArena_New() => _PyArena_New()
* PyArena_Free() => _PyArena_Free()
* PyArena_Malloc() => _PyArena_Malloc()
* PyArena_AddPyObject() => _PyArena_AddPyObject()
2021-03-24 02:23:01 +01:00
Victor Stinner 57364ce34e
bpo-43244: Remove parser_interface.h header file (GH-25001)
Remove parser functions using the "struct _mod" type, because the
AST C API was removed:

* PyParser_ASTFromFile()
* PyParser_ASTFromFileObject()
* PyParser_ASTFromFilename()
* PyParser_ASTFromString()
* PyParser_ASTFromStringObject()

These functions were undocumented and excluded from the limited C
API.

Add pycore_parser.h internal header file. Rename functions:

* PyParser_ASTFromFileObject() => _PyParser_ASTFromFile()
* PyParser_ASTFromStringObject() => _PyParser_ASTFromString()

These functions are no longer exported (replace PyAPI_FUNC() with
extern).

Remove also _PyPegen_run_parser_from_file() function. Update
test_peg_generator to use _PyPegen_run_parser_from_file_pointer()
instead.
2021-03-24 01:29:09 +01:00
Victor Stinner a81fca6ec8
bpo-43244: Add pycore_compile.h header file (GH-25000)
Remove the compiler functions using "struct _mod" type, because the
public AST C API was removed:

* PyAST_Compile()
* PyAST_CompileEx()
* PyAST_CompileObject()
* PyFuture_FromAST()
* PyFuture_FromASTObject()

These functions were undocumented and excluded from the limited C API.

Rename functions:

* PyAST_CompileObject() => _PyAST_Compile()
* PyFuture_FromASTObject() => _PyFuture_FromAST()

Moreover, _PyFuture_FromAST() is no longer exported (replace
PyAPI_FUNC() with extern). _PyAST_Compile() remains exported for
test_peg_generator.

Remove also compatibility functions:

* PyAST_Compile()
* PyAST_CompileEx()
* PyFuture_FromAST()
2021-03-24 00:51:50 +01:00
Victor Stinner 6af528b4ab
bpo-43244: Fix test_peg_generators on Windows (GH-24913)
Don't redefine Py_DebugFlag, it's already defined in pydebug.h which
is included by Python.h
2021-03-18 09:54:13 +01:00
Victor Stinner e0bf70d08c
bpo-43244: Fix test_peg_generator for PyAST_Validate() (GH-24912)
test_peg_generator now defines _Py_TEST_PEGEN macro when building C
code to not call PyAST_Validate() in Parser/pegen.c. Moreover, it
defines Py_BUILD_CORE_MODULE macro to get access to the internal
C API.

Remove "global_ast_state" from Python-ast.c when it's built by
test_peg_generator: always get the AST state from the current interpreter.
2021-03-18 02:46:06 +01:00
Elisha Hollander e272528bbd
Remove unnecessary imports in the grammar parser (GH-24904) 2021-03-17 22:07:17 +00:00
Jozef Grajciar c994ffe695
bpo-11717: fix ssize_t redefinition error when targeting 32bit Windows app (GH-24479) 2021-03-01 11:18:33 +00:00
Pablo Galindo 58fb156edd
bpo-42997: Improve error message for missing : before suites (GH-24292)
* Add to the peg generator a new directive ('&&') that allows to expect
  a token and hard fail the parsing if the token is not found. This
  allows to quickly emmit syntax errors for missing tokens.

* Use the new grammar element to hard-fail if the ':' is missing before
  suites.
2021-02-02 19:54:22 +00:00
Pablo Galindo 3bcc4ead3f
Add small validator utility for PEG grammars (GH-23519) 2020-12-26 19:11:29 +00:00
Lysandros Nikolaou 02cdfc93f8
bpo-42218: Correctly handle errors in left-recursive rules (GH-23065)
Left-recursive rules need to check for errors explicitly, since
even if the rule returns NULL, the parsing might continue and lead
to long-distance failures.

Co-authored-by: Pablo Galindo <Pablogsal@gmail.com>
2020-10-31 20:31:41 +02:00
Lysandros Nikolaou bca7014032
bpo-42123: Run the parser two times and only enable invalid rules on the second run (GH-22111)
* Implement running the parser a second time for the errors messages

The first parser run is only responsible for detecting whether
there is a `SyntaxError` or not. If there isn't the AST gets returned.
Otherwise, the parser is run a second time with all the `invalid_*`
rules enabled so that all the customized error messages get produced.
2020-10-27 00:42:04 +02:00
Andre Delfino e8a2076e14
Revert "Fix all Python Cookbook links (#22205)" (GH-22424)
This commit reverts commit ac0333e1e1 as the original links are working again and they provide extended features such as comments and alternative versions.
2020-09-27 01:47:25 +01:00
Pablo Galindo a5634c4067
bpo-41746: Add type information to asdl_seq objects (GH-22223)
* Add new capability to the PEG parser to type variable assignments. For instance:
```
       | a[asdl_stmt_seq*]=';'.small_stmt+ [';'] NEWLINE { a }
```

* Add new sequence types from the asdl definition (automatically generated)
* Make `asdl_seq` type a generic aliasing pointer type.
* Create a new `asdl_generic_seq` for the generic case using `void*`.
* The old `asdl_seq_GET`/`ast_seq_SET` macros now are typed.
* New `asdl_seq_GET_UNTYPED`/`ast_seq_SET_UNTYPED` macros for dealing with generic sequences.
* Changes all possible `asdl_seq` types to use specific versions everywhere.
2020-09-16 19:42:00 +01:00
Andre Delfino ac0333e1e1
Fix all Python Cookbook links (#22205) 2020-09-15 21:13:26 +01:00
Pablo Galindo e55a0e971b
Fix 'gather' rules in the python parser generator (GH-22021)
Currently, empty sequences in gather rules make the conditional for
gather rules fail as empty sequences evaluate as "False". We need to
explicitly check for "None" (the failure condition) to avoid false
negatives.
2020-09-03 15:29:55 +01:00
Guido van Rossum 508ed2d912
Delete remaining references to Grammar/Grammar from docs (#21624)
(Ironically, the file itself remains, see https://github.com/we-like-parsers/cpython/issues/135.)
2020-07-26 08:27:52 -07:00
Pablo Galindo 1ac0cbca36
bpo-41215: Don't use NULL by default in the PEG parser keyword list (GH-21355)
Automerge-Triggered-By: @lysnikolaou
2020-07-06 12:31:16 -07:00
Batuhan Taskaya 55460ee6dc
bpo-41044: Generate valid PEG python parsers for opt+seq rules (GH-20995)
Co-authored-by: Pablo Galindo <Pablogsal@gmail.com>
2020-06-20 18:40:06 +01:00
Serhiy Storchaka 9355868458
bpo-41043: Escape literal part of the path for glob(). (GH-20994) 2020-06-20 11:10:31 +03:00
Pablo Galindo 5fc4f8ae68
Fix trailing whitespace in keyword.py (GH-20881) 2020-06-15 04:33:33 +01:00
Pablo Galindo 78319e373d
Include soft keywords in keyword.py (GH-20877) 2020-06-15 03:55:15 +01:00