d66bc9e8a7
gh-107967: Fix infinite recursion on invalid escape sequence warning ( #107968 )
2023-08-15 11:26:42 +00:00
fa45958450
GH-107263: Increase C stack limit for most functions, except `_PyEval_EvalFrameDefault()` (GH-107535)
...
* Set C recursion limit to 1500, set cost of eval loop to 2 frames, and compiler mutliply to 2.
2023-08-04 10:10:29 +01:00
da8f87b7ea
gh-107015: Remove async_hacks from the tokenizer ( #107018 )
2023-07-26 16:34:15 +01:00
1a3faba9f1
gh-106869: Use new PyMemberDef constant names ( #106871 )
...
* Remove '#include "structmember.h"'.
* If needed, add <stddef.h> to get offsetof() function.
* Update Parser/asdl_c.py to regenerate Python/Python-ast.c.
* Replace:
* T_SHORT => Py_T_SHORT
* T_INT => Py_T_INT
* T_LONG => Py_T_LONG
* T_FLOAT => Py_T_FLOAT
* T_DOUBLE => Py_T_DOUBLE
* T_STRING => Py_T_STRING
* T_OBJECT => _Py_T_OBJECT
* T_CHAR => Py_T_CHAR
* T_BYTE => Py_T_BYTE
* T_UBYTE => Py_T_UBYTE
* T_USHORT => Py_T_USHORT
* T_UINT => Py_T_UINT
* T_ULONG => Py_T_ULONG
* T_STRING_INPLACE => Py_T_STRING_INPLACE
* T_BOOL => Py_T_BOOL
* T_OBJECT_EX => Py_T_OBJECT_EX
* T_LONGLONG => Py_T_LONGLONG
* T_ULONGLONG => Py_T_ULONGLONG
* T_PYSSIZET => Py_T_PYSSIZET
* T_NONE => _Py_T_NONE
* READONLY => Py_READONLY
* PY_AUDIT_READ => Py_AUDIT_READ
* READ_RESTRICTED => Py_AUDIT_READ
* PY_WRITE_RESTRICTED => _Py_WRITE_RESTRICTED
* RESTRICTED => (READ_RESTRICTED | _Py_WRITE_RESTRICTED)
2023-07-25 15:28:30 +02:00
7d41ead919
gh-106320: Remove _PyBytes_Join() C API ( #107144 )
...
Move private _PyBytes functions to the internal C API
(pycore_bytesobject.h):
* _PyBytes_DecodeEscape()
* _PyBytes_FormatEx()
* _PyBytes_FromHex()
* _PyBytes_Join()
No longer export these functions.
2023-07-23 20:10:12 +00:00
d228825e08
gh-106320: Remove _PyOS_ReadlineTState API ( #107034 )
...
Remove _PyOS_ReadlineTState variable from the public C API.
The symbol is still exported for the readline shared extension.
2023-07-22 14:45:56 +00:00
76e20c361c
gh-106989: Remove tok report warnings ( #106993 )
2023-07-22 14:23:23 +02:00
be1b968dc1
gh-106521: Remove _PyObject_LookupAttr() function (GH-106642)
2023-07-12 08:57:10 +03:00
dfe4de2038
gh-106396: Special-case empty format spec to gen empty JoinedStr node ( #106401 )
2023-07-04 14:19:08 +02:00
d8c5d76da2
gh-106320: Remove private _PyUnicode codecs C API functions ( #106385 )
...
Remove private _PyUnicode codecs C API functions: move them to the
internal C API (pycore_unicodeobject.h). No longer export most of
these functions.
2023-07-04 07:29:52 +00:00
c5afc97fc2
gh-106320: Remove private _PyErr C API functions ( #106356 )
...
Remove private _PyErr C API functions: move them to the internal
C API (pycore_pyerrors.h).
2023-07-03 10:48:50 +00:00
d5bd32fb48
gh-104922: remove PY_SSIZE_T_CLEAN ( #106315 )
2023-07-02 15:07:46 +09:00
46c1097868
gh-106145: Make `end_{lineno,col_offset}` required on `type_param` nodes ( #106224 )
2023-06-30 23:45:08 +00:00
8c5f74fc89
gh-106023: Update code using _PyObject_FastCall() ( #106257 )
...
Replace _PyObject_FastCall() calls with PyObject_Vectorcall().
2023-06-30 01:05:01 +00:00
13237a2da8
gh-98931: Add custom error messages to invalid import/from with multiple targets ( #105985 )
...
Co-authored-by: Alex Waygood <Alex.Waygood@Gmail.com>
2023-06-22 15:56:40 +00:00
6586cee27f
gh-105938: Emit a SyntaxWarning for escaped braces in an f-string ( #105939 )
2023-06-20 12:38:46 +00:00
a4056c8f9c
GH-105588: Add missing error checks to some obj2ast_* converters (GH-105589)
2023-06-15 15:45:13 -07:00
d382ad4915
gh-105820: Fix tok_mode expression buffer in file & readline tokenizer ( #105828 )
2023-06-15 16:21:24 +00:00
12b6d844d8
gh-105800: Issue SyntaxWarning in f-strings for invalid escape sequences ( #105801 )
2023-06-15 01:08:12 +01:00
abfbab6415
gh-105718: Fix buffer allocation in tokenizer with readline ( #105728 )
2023-06-13 16:18:11 +01:00
b047fa5e56
gh-105549: Tokenize separately NUMBER and NAME tokens and allow 0-prefixed literals ( #105555 )
2023-06-09 21:39:01 +01:00
c0a6ed3934
gh-105259: Ensure we don't show newline characters for trailing NEWLINE tokens ( #105364 )
2023-06-06 12:52:16 +01:00
41de54378d
gh-105194: Fix format specifier escaped characters in f-strings ( #105231 )
2023-06-02 13:33:26 +02:00
77d2579586
gh-104799: Default missing lists in AST to the empty list ( #104834 )
...
Co-authored-by: Alex Waygood <Alex.Waygood@Gmail.com>
2023-06-01 18:39:39 -07:00
ef300937c2
gh-92536: Remove PyUnicode_READY() calls ( #105210 )
...
Since Python 3.12, PyUnicode_READY() does nothing and always
returns 0.
2023-06-02 01:33:17 +02:00
70f315c2d6
gh-105042: Disable unmatched parens syntax error in python tokenize ( #105061 )
2023-05-30 22:52:52 +01:00
9216e69a87
gh-105069: Add a readline-like callable to the tokenizer to consume input iteratively ( #105070 )
2023-05-30 22:43:34 +01:00
96fff35325
gh-105017: Include CRLF lines in strings and column numbers ( #105030 )
...
Co-authored-by: Pablo Galindo <pablogsal@gmail.com>
2023-05-28 15:15:53 +01:00
86d8f48935
gh-105017: Fix including additional NL token when using CRLF ( #105022 )
...
Co-authored-by: Pablo Galindo Salgado <Pablogsal@gmail.com>
2023-05-27 16:50:43 +00:00
6e62eb2e70
Fix indentation in Parser/tokenizer.c ( #105012 )
2023-05-27 12:41:50 +01:00
ba73473f4c
gh-104799: Move location of type_params AST fields ( #104828 )
...
Co-authored-by: Alex Waygood <Alex.Waygood@Gmail.com>
2023-05-26 05:54:37 -07:00
705e387dd8
Fix typo in the tokenizer ( #104950 )
2023-05-26 02:50:33 +00:00
c90a862cdc
gh-104866: Tokenize should emit NEWLINE after exiting block with comment ( #104870 )
2023-05-24 17:18:17 +01:00
357bed0bcd
GH-104668: Don't call PyOS_* hooks in subinterpreters (GH-104674)
2023-05-22 19:34:34 +00:00
0a7796052a
gh-102856: Allow comments inside multi-line f-string expresions ( #104006 )
2023-05-22 10:30:07 +00:00
a5f244d627
gh-104656: Rename typeparams AST node to type_params ( #104657 )
2023-05-21 21:25:09 -07:00
f3466bc040
gh-98836: Extend PyUnicode_FromFormat() (GH-98838)
...
* Support for conversion specifiers o (octal) and X (uppercase hexadecimal).
* Support for length modifiers j (intmax_t) and t (ptrdiff_t).
* Length modifiers are now applied to all integer conversions.
* Support for wchar_t C strings (%ls and %lV).
* Support for variable width and precision (*).
* Support for flag - (left alignment).
2023-05-22 00:32:39 +03:00
6715f91edc
gh-102856: Python tokenizer implementation for PEP 701 ( #104323 )
...
This commit replaces the Python implementation of the tokenize module with an implementation
that reuses the real C tokenizer via a private extension module. The tokenize module now implements
a compatibility layer that transforms tokens from the C tokenizer into Python tokenize tokens for backward
compatibility.
As the C tokenizer does not emit some tokens that the Python tokenizer provides (such as comments and non-semantic newlines), a new special mode has been added to the C tokenizer mode that currently is only used via
the extension module that exposes it to the Python layer. This new mode forces the C tokenizer to emit these new extra tokens and add the appropriate metadata that is needed to match the old Python implementation.
Co-authored-by: Pablo Galindo <pablogsal@gmail.com>
2023-05-21 01:03:02 +01:00
ff7f731632
gh-104658: Fix location of unclosed quote error for multiline f-strings ( #104660 )
2023-05-20 14:07:05 +01:00
24d8b88420
gh-103763: Implement PEP 695 ( #103764 )
...
This implements PEP 695, Type Parameter Syntax. It adds support for:
- Generic functions (def func[T](): ...)
- Generic classes (class X[T](): ...)
- Type aliases (type X = ...)
- New scoping when the new syntax is used within a class body
- Compiler and interpreter changes to support the new syntax and scoping rules
Co-authored-by: Marc Mueller <30130371+cdce8p@users.noreply.github.com>
Co-authored-by: Eric Traut <eric@traut.com>
Co-authored-by: Larry Hastings <larry@hastings.org>
Co-authored-by: Alex Waygood <Alex.Waygood@Gmail.com>
2023-05-15 20:36:23 -07:00
d513ddee94
Trim trailing whitespace and test on CI ( #104275 )
...
Co-authored-by: Alex Waygood <Alex.Waygood@Gmail.com>
2023-05-08 17:03:52 +03:00
a9c6e0618f
gh-99113: Add Py_MOD_PER_INTERPRETER_GIL_SUPPORTED (gh-104205)
...
Here we are doing no more than adding the value for Py_mod_multiple_interpreters and using it for stdlib modules. We will start checking for it in gh-104206 (once PyInterpreterState.ceval.own_gil is added in gh-104204).
2023-05-05 21:11:27 +00:00
eba64d2afb
gh-104169: Ensure the tokenizer doesn't overwrite previous errors ( #104170 )
2023-05-04 15:15:26 +01:00
ef0df5284f
gh-97556: Raise null bytes syntax error upon null in multiline string (GH-104136)
2023-05-04 14:26:23 +02:00
5078eedc5b
gh-104016: Fixed off by 1 error in f string tokenizer ( #104047 )
...
Co-authored-by: sunmy2019 <59365878+sunmy2019@users.noreply.github.com>
Co-authored-by: Ken Jin <kenjin@python.org>
Co-authored-by: Pablo Galindo <pablogsal@gmail.com>
2023-05-01 19:15:47 +00:00
d5a97074d2
gh-103824: fix use-after-free error in Parser/tokenizer.c ( #103993 )
2023-05-01 15:26:43 +00:00
9169a56fad
gh-103656: Transfer f-string buffers to parser to avoid use-after-free (GH-103896)
...
Co-authored-by: Pablo Galindo <pablogsal@gmail.com>
2023-04-27 01:33:31 +00:00
57f8f9a66d
gh-103718: Correctly set f-string buffers in all cases (GH-103815)
...
Turns out we always need to remember/restore fstring buffers in all of
the stack of tokenizer modes, cause they might change to
`TOK_REGULAR_MODE` and have newlines inside the braces (which is when we
need to reallocate the buffer and restore the fstring ones).
2023-04-25 01:31:21 +00:00
cb157a1a35
GH-103727: Avoid advancing tokenizer too far in f-string mode (GH-103775)
2023-04-24 12:30:21 -06:00
05b3ce7339
GH-103718: Correctly cache and restore f-string buffers when needed (GH-103719)
2023-04-23 13:06:10 -06:00
Lysandros Nikolaou
Mark Shannon
Pablo Galindo Salgado
Victor Stinner
Menelaos Kotoglou
Serhiy Storchaka
Inada Naoki
Nikita Sobolev
Brandt Bucher
Jelle Zijlstra
Marta Gómez Macías
Petr Vaněk
Stepfen Shawn
Cristián Maureira-Fredes
Hugo van Kemenade
Eric Snow
jx124
chgnrdv