86617518c4
gh-108179: Add error message for parser stack overflows ( #108256 )
2023-08-22 08:41:50 +01:00
0dd3fc2a64
gh-108216: Cleanup #include in internal header files ( #108228 )
...
* Add missing includes.
* Remove unused includes.
* Update old include/symbol names to newer names.
* Mention at least one included symbol.
* Sort includes.
* Update Tools/cases_generator/generate_cases.py used to generated
pycore_opcode_metadata.h.
* Update Parser/asdl_c.py used to generate pycore_ast.h.
* Cleanup also includes in _testcapimodule.c and _testinternalcapi.c.
2023-08-21 18:05:59 +00:00
10a91d7e98
gh-108113: Make it possible to create an optimized AST ( #108154 )
2023-08-21 16:31:30 +00:00
d66bc9e8a7
gh-107967: Fix infinite recursion on invalid escape sequence warning ( #107968 )
2023-08-15 11:26:42 +00:00
fa45958450
GH-107263: Increase C stack limit for most functions, except `_PyEval_EvalFrameDefault()` (GH-107535)
...
* Set C recursion limit to 1500, set cost of eval loop to 2 frames, and compiler mutliply to 2.
2023-08-04 10:10:29 +01:00
da8f87b7ea
gh-107015: Remove async_hacks from the tokenizer ( #107018 )
2023-07-26 16:34:15 +01:00
1a3faba9f1
gh-106869: Use new PyMemberDef constant names ( #106871 )
...
* Remove '#include "structmember.h"'.
* If needed, add <stddef.h> to get offsetof() function.
* Update Parser/asdl_c.py to regenerate Python/Python-ast.c.
* Replace:
* T_SHORT => Py_T_SHORT
* T_INT => Py_T_INT
* T_LONG => Py_T_LONG
* T_FLOAT => Py_T_FLOAT
* T_DOUBLE => Py_T_DOUBLE
* T_STRING => Py_T_STRING
* T_OBJECT => _Py_T_OBJECT
* T_CHAR => Py_T_CHAR
* T_BYTE => Py_T_BYTE
* T_UBYTE => Py_T_UBYTE
* T_USHORT => Py_T_USHORT
* T_UINT => Py_T_UINT
* T_ULONG => Py_T_ULONG
* T_STRING_INPLACE => Py_T_STRING_INPLACE
* T_BOOL => Py_T_BOOL
* T_OBJECT_EX => Py_T_OBJECT_EX
* T_LONGLONG => Py_T_LONGLONG
* T_ULONGLONG => Py_T_ULONGLONG
* T_PYSSIZET => Py_T_PYSSIZET
* T_NONE => _Py_T_NONE
* READONLY => Py_READONLY
* PY_AUDIT_READ => Py_AUDIT_READ
* READ_RESTRICTED => Py_AUDIT_READ
* PY_WRITE_RESTRICTED => _Py_WRITE_RESTRICTED
* RESTRICTED => (READ_RESTRICTED | _Py_WRITE_RESTRICTED)
2023-07-25 15:28:30 +02:00
7d41ead919
gh-106320: Remove _PyBytes_Join() C API ( #107144 )
...
Move private _PyBytes functions to the internal C API
(pycore_bytesobject.h):
* _PyBytes_DecodeEscape()
* _PyBytes_FormatEx()
* _PyBytes_FromHex()
* _PyBytes_Join()
No longer export these functions.
2023-07-23 20:10:12 +00:00
d228825e08
gh-106320: Remove _PyOS_ReadlineTState API ( #107034 )
...
Remove _PyOS_ReadlineTState variable from the public C API.
The symbol is still exported for the readline shared extension.
2023-07-22 14:45:56 +00:00
76e20c361c
gh-106989: Remove tok report warnings ( #106993 )
2023-07-22 14:23:23 +02:00
be1b968dc1
gh-106521: Remove _PyObject_LookupAttr() function (GH-106642)
2023-07-12 08:57:10 +03:00
dfe4de2038
gh-106396: Special-case empty format spec to gen empty JoinedStr node ( #106401 )
2023-07-04 14:19:08 +02:00
d8c5d76da2
gh-106320: Remove private _PyUnicode codecs C API functions ( #106385 )
...
Remove private _PyUnicode codecs C API functions: move them to the
internal C API (pycore_unicodeobject.h). No longer export most of
these functions.
2023-07-04 07:29:52 +00:00
c5afc97fc2
gh-106320: Remove private _PyErr C API functions ( #106356 )
...
Remove private _PyErr C API functions: move them to the internal
C API (pycore_pyerrors.h).
2023-07-03 10:48:50 +00:00
d5bd32fb48
gh-104922: remove PY_SSIZE_T_CLEAN ( #106315 )
2023-07-02 15:07:46 +09:00
46c1097868
gh-106145: Make `end_{lineno,col_offset}` required on `type_param` nodes ( #106224 )
2023-06-30 23:45:08 +00:00
8c5f74fc89
gh-106023: Update code using _PyObject_FastCall() ( #106257 )
...
Replace _PyObject_FastCall() calls with PyObject_Vectorcall().
2023-06-30 01:05:01 +00:00
13237a2da8
gh-98931: Add custom error messages to invalid import/from with multiple targets ( #105985 )
...
Co-authored-by: Alex Waygood <Alex.Waygood@Gmail.com>
2023-06-22 15:56:40 +00:00
6586cee27f
gh-105938: Emit a SyntaxWarning for escaped braces in an f-string ( #105939 )
2023-06-20 12:38:46 +00:00
a4056c8f9c
GH-105588: Add missing error checks to some obj2ast_* converters (GH-105589)
2023-06-15 15:45:13 -07:00
d382ad4915
gh-105820: Fix tok_mode expression buffer in file & readline tokenizer ( #105828 )
2023-06-15 16:21:24 +00:00
12b6d844d8
gh-105800: Issue SyntaxWarning in f-strings for invalid escape sequences ( #105801 )
2023-06-15 01:08:12 +01:00
abfbab6415
gh-105718: Fix buffer allocation in tokenizer with readline ( #105728 )
2023-06-13 16:18:11 +01:00
b047fa5e56
gh-105549: Tokenize separately NUMBER and NAME tokens and allow 0-prefixed literals ( #105555 )
2023-06-09 21:39:01 +01:00
c0a6ed3934
gh-105259: Ensure we don't show newline characters for trailing NEWLINE tokens ( #105364 )
2023-06-06 12:52:16 +01:00
41de54378d
gh-105194: Fix format specifier escaped characters in f-strings ( #105231 )
2023-06-02 13:33:26 +02:00
77d2579586
gh-104799: Default missing lists in AST to the empty list ( #104834 )
...
Co-authored-by: Alex Waygood <Alex.Waygood@Gmail.com>
2023-06-01 18:39:39 -07:00
ef300937c2
gh-92536: Remove PyUnicode_READY() calls ( #105210 )
...
Since Python 3.12, PyUnicode_READY() does nothing and always
returns 0.
2023-06-02 01:33:17 +02:00
70f315c2d6
gh-105042: Disable unmatched parens syntax error in python tokenize ( #105061 )
2023-05-30 22:52:52 +01:00
9216e69a87
gh-105069: Add a readline-like callable to the tokenizer to consume input iteratively ( #105070 )
2023-05-30 22:43:34 +01:00
96fff35325
gh-105017: Include CRLF lines in strings and column numbers ( #105030 )
...
Co-authored-by: Pablo Galindo <pablogsal@gmail.com>
2023-05-28 15:15:53 +01:00
86d8f48935
gh-105017: Fix including additional NL token when using CRLF ( #105022 )
...
Co-authored-by: Pablo Galindo Salgado <Pablogsal@gmail.com>
2023-05-27 16:50:43 +00:00
6e62eb2e70
Fix indentation in Parser/tokenizer.c ( #105012 )
2023-05-27 12:41:50 +01:00
ba73473f4c
gh-104799: Move location of type_params AST fields ( #104828 )
...
Co-authored-by: Alex Waygood <Alex.Waygood@Gmail.com>
2023-05-26 05:54:37 -07:00
705e387dd8
Fix typo in the tokenizer ( #104950 )
2023-05-26 02:50:33 +00:00
c90a862cdc
gh-104866: Tokenize should emit NEWLINE after exiting block with comment ( #104870 )
2023-05-24 17:18:17 +01:00
357bed0bcd
GH-104668: Don't call PyOS_* hooks in subinterpreters (GH-104674)
2023-05-22 19:34:34 +00:00
0a7796052a
gh-102856: Allow comments inside multi-line f-string expresions ( #104006 )
2023-05-22 10:30:07 +00:00
a5f244d627
gh-104656: Rename typeparams AST node to type_params ( #104657 )
2023-05-21 21:25:09 -07:00
f3466bc040
gh-98836: Extend PyUnicode_FromFormat() (GH-98838)
...
* Support for conversion specifiers o (octal) and X (uppercase hexadecimal).
* Support for length modifiers j (intmax_t) and t (ptrdiff_t).
* Length modifiers are now applied to all integer conversions.
* Support for wchar_t C strings (%ls and %lV).
* Support for variable width and precision (*).
* Support for flag - (left alignment).
2023-05-22 00:32:39 +03:00
6715f91edc
gh-102856: Python tokenizer implementation for PEP 701 ( #104323 )
...
This commit replaces the Python implementation of the tokenize module with an implementation
that reuses the real C tokenizer via a private extension module. The tokenize module now implements
a compatibility layer that transforms tokens from the C tokenizer into Python tokenize tokens for backward
compatibility.
As the C tokenizer does not emit some tokens that the Python tokenizer provides (such as comments and non-semantic newlines), a new special mode has been added to the C tokenizer mode that currently is only used via
the extension module that exposes it to the Python layer. This new mode forces the C tokenizer to emit these new extra tokens and add the appropriate metadata that is needed to match the old Python implementation.
Co-authored-by: Pablo Galindo <pablogsal@gmail.com>
2023-05-21 01:03:02 +01:00
ff7f731632
gh-104658: Fix location of unclosed quote error for multiline f-strings ( #104660 )
2023-05-20 14:07:05 +01:00
24d8b88420
gh-103763: Implement PEP 695 ( #103764 )
...
This implements PEP 695, Type Parameter Syntax. It adds support for:
- Generic functions (def func[T](): ...)
- Generic classes (class X[T](): ...)
- Type aliases (type X = ...)
- New scoping when the new syntax is used within a class body
- Compiler and interpreter changes to support the new syntax and scoping rules
Co-authored-by: Marc Mueller <30130371+cdce8p@users.noreply.github.com>
Co-authored-by: Eric Traut <eric@traut.com>
Co-authored-by: Larry Hastings <larry@hastings.org>
Co-authored-by: Alex Waygood <Alex.Waygood@Gmail.com>
2023-05-15 20:36:23 -07:00
d513ddee94
Trim trailing whitespace and test on CI ( #104275 )
...
Co-authored-by: Alex Waygood <Alex.Waygood@Gmail.com>
2023-05-08 17:03:52 +03:00
a9c6e0618f
gh-99113: Add Py_MOD_PER_INTERPRETER_GIL_SUPPORTED (gh-104205)
...
Here we are doing no more than adding the value for Py_mod_multiple_interpreters and using it for stdlib modules. We will start checking for it in gh-104206 (once PyInterpreterState.ceval.own_gil is added in gh-104204).
2023-05-05 21:11:27 +00:00
eba64d2afb
gh-104169: Ensure the tokenizer doesn't overwrite previous errors ( #104170 )
2023-05-04 15:15:26 +01:00
ef0df5284f
gh-97556: Raise null bytes syntax error upon null in multiline string (GH-104136)
2023-05-04 14:26:23 +02:00
5078eedc5b
gh-104016: Fixed off by 1 error in f string tokenizer ( #104047 )
...
Co-authored-by: sunmy2019 <59365878+sunmy2019@users.noreply.github.com>
Co-authored-by: Ken Jin <kenjin@python.org>
Co-authored-by: Pablo Galindo <pablogsal@gmail.com>
2023-05-01 19:15:47 +00:00
d5a97074d2
gh-103824: fix use-after-free error in Parser/tokenizer.c ( #103993 )
2023-05-01 15:26:43 +00:00
9169a56fad
gh-103656: Transfer f-string buffers to parser to avoid use-after-free (GH-103896)
...
Co-authored-by: Pablo Galindo <pablogsal@gmail.com>
2023-04-27 01:33:31 +00:00
Dennis Sweeney
Victor Stinner
Irit Katriel
Lysandros Nikolaou
Mark Shannon
Pablo Galindo Salgado
Menelaos Kotoglou
Serhiy Storchaka
Inada Naoki
Nikita Sobolev
Brandt Bucher
Jelle Zijlstra
Marta Gómez Macías
Petr Vaněk
Stepfen Shawn
Cristián Maureira-Fredes
Hugo van Kemenade
Eric Snow
jx124
chgnrdv