Commit Graph

1170 Commits

Author SHA1 Message Date
Serhiy Storchaka be1b968dc1
gh-106521: Remove _PyObject_LookupAttr() function (GH-106642) 2023-07-12 08:57:10 +03:00
Lysandros Nikolaou dfe4de2038
gh-106396: Special-case empty format spec to gen empty JoinedStr node (#106401) 2023-07-04 14:19:08 +02:00
Victor Stinner d8c5d76da2
gh-106320: Remove private _PyUnicode codecs C API functions (#106385)
Remove private _PyUnicode codecs C API functions: move them to the
internal C API (pycore_unicodeobject.h). No longer export most of
these functions.
2023-07-04 07:29:52 +00:00
Victor Stinner c5afc97fc2
gh-106320: Remove private _PyErr C API functions (#106356)
Remove private _PyErr C API functions: move them to the internal
C API (pycore_pyerrors.h).
2023-07-03 10:48:50 +00:00
Inada Naoki d5bd32fb48
gh-104922: remove PY_SSIZE_T_CLEAN (#106315) 2023-07-02 15:07:46 +09:00
Nikita Sobolev 46c1097868
gh-106145: Make `end_{lineno,col_offset}` required on `type_param` nodes (#106224) 2023-06-30 23:45:08 +00:00
Victor Stinner 8c5f74fc89
gh-106023: Update code using _PyObject_FastCall() (#106257)
Replace _PyObject_FastCall() calls with PyObject_Vectorcall().
2023-06-30 01:05:01 +00:00
Pablo Galindo Salgado 13237a2da8
gh-98931: Add custom error messages to invalid import/from with multiple targets (#105985)
Co-authored-by: Alex Waygood <Alex.Waygood@Gmail.com>
2023-06-22 15:56:40 +00:00
Lysandros Nikolaou 6586cee27f
gh-105938: Emit a SyntaxWarning for escaped braces in an f-string (#105939) 2023-06-20 12:38:46 +00:00
Brandt Bucher a4056c8f9c
GH-105588: Add missing error checks to some obj2ast_* converters (GH-105589) 2023-06-15 15:45:13 -07:00
Lysandros Nikolaou d382ad4915
gh-105820: Fix tok_mode expression buffer in file & readline tokenizer (#105828) 2023-06-15 16:21:24 +00:00
Pablo Galindo Salgado 12b6d844d8
gh-105800: Issue SyntaxWarning in f-strings for invalid escape sequences (#105801) 2023-06-15 01:08:12 +01:00
Lysandros Nikolaou abfbab6415
gh-105718: Fix buffer allocation in tokenizer with readline (#105728) 2023-06-13 16:18:11 +01:00
Pablo Galindo Salgado b047fa5e56
gh-105549: Tokenize separately NUMBER and NAME tokens and allow 0-prefixed literals (#105555) 2023-06-09 21:39:01 +01:00
Pablo Galindo Salgado c0a6ed3934
gh-105259: Ensure we don't show newline characters for trailing NEWLINE tokens (#105364) 2023-06-06 12:52:16 +01:00
Pablo Galindo Salgado 41de54378d
gh-105194: Fix format specifier escaped characters in f-strings (#105231) 2023-06-02 13:33:26 +02:00
Jelle Zijlstra 77d2579586
gh-104799: Default missing lists in AST to the empty list (#104834)
Co-authored-by: Alex Waygood <Alex.Waygood@Gmail.com>
2023-06-01 18:39:39 -07:00
Victor Stinner ef300937c2
gh-92536: Remove PyUnicode_READY() calls (#105210)
Since Python 3.12, PyUnicode_READY() does nothing and always
returns 0.
2023-06-02 01:33:17 +02:00
Lysandros Nikolaou 70f315c2d6
gh-105042: Disable unmatched parens syntax error in python tokenize (#105061) 2023-05-30 22:52:52 +01:00
Pablo Galindo Salgado 9216e69a87
gh-105069: Add a readline-like callable to the tokenizer to consume input iteratively (#105070) 2023-05-30 22:43:34 +01:00
Marta Gómez Macías 96fff35325
gh-105017: Include CRLF lines in strings and column numbers (#105030)
Co-authored-by: Pablo Galindo <pablogsal@gmail.com>
2023-05-28 15:15:53 +01:00
Marta Gómez Macías 86d8f48935
gh-105017: Fix including additional NL token when using CRLF (#105022)
Co-authored-by: Pablo Galindo Salgado <Pablogsal@gmail.com>
2023-05-27 16:50:43 +00:00
Petr Vaněk 6e62eb2e70
Fix indentation in Parser/tokenizer.c (#105012) 2023-05-27 12:41:50 +01:00
Jelle Zijlstra ba73473f4c
gh-104799: Move location of type_params AST fields (#104828)
Co-authored-by: Alex Waygood <Alex.Waygood@Gmail.com>
2023-05-26 05:54:37 -07:00
Stepfen Shawn 705e387dd8
Fix typo in the tokenizer (#104950) 2023-05-26 02:50:33 +00:00
Lysandros Nikolaou c90a862cdc
gh-104866: Tokenize should emit NEWLINE after exiting block with comment (#104870) 2023-05-24 17:18:17 +01:00
Brandt Bucher 357bed0bcd
GH-104668: Don't call PyOS_* hooks in subinterpreters (GH-104674) 2023-05-22 19:34:34 +00:00
Cristián Maureira-Fredes 0a7796052a
gh-102856: Allow comments inside multi-line f-string expresions (#104006) 2023-05-22 10:30:07 +00:00
Jelle Zijlstra a5f244d627
gh-104656: Rename typeparams AST node to type_params (#104657) 2023-05-21 21:25:09 -07:00
Serhiy Storchaka f3466bc040
gh-98836: Extend PyUnicode_FromFormat() (GH-98838)
* Support for conversion specifiers o (octal) and X (uppercase hexadecimal).
* Support for length modifiers j (intmax_t) and t (ptrdiff_t).
* Length modifiers are now applied to all integer conversions.
* Support for wchar_t C strings (%ls and %lV).
* Support for variable width and precision (*).
* Support for flag - (left alignment).
2023-05-22 00:32:39 +03:00
Marta Gómez Macías 6715f91edc
gh-102856: Python tokenizer implementation for PEP 701 (#104323)
This commit replaces the Python implementation of the tokenize module with an implementation
that reuses the real C tokenizer via a private extension module. The tokenize module now implements
a compatibility layer that transforms tokens from the C tokenizer into Python tokenize tokens for backward
compatibility.

As the C tokenizer does not emit some tokens that the Python tokenizer provides (such as comments and non-semantic newlines), a new special mode has been added to the C tokenizer mode that currently is only used via
the extension module that exposes it to the Python layer. This new mode forces the C tokenizer to emit these new extra tokens and add the appropriate metadata that is needed to match the old Python implementation.

Co-authored-by: Pablo Galindo <pablogsal@gmail.com>
2023-05-21 01:03:02 +01:00
Pablo Galindo Salgado ff7f731632
gh-104658: Fix location of unclosed quote error for multiline f-strings (#104660) 2023-05-20 14:07:05 +01:00
Jelle Zijlstra 24d8b88420
gh-103763: Implement PEP 695 (#103764)
This implements PEP 695, Type Parameter Syntax. It adds support for:

- Generic functions (def func[T](): ...)
- Generic classes (class X[T](): ...)
- Type aliases (type X = ...)
- New scoping when the new syntax is used within a class body
- Compiler and interpreter changes to support the new syntax and scoping rules 

Co-authored-by: Marc Mueller <30130371+cdce8p@users.noreply.github.com>
Co-authored-by: Eric Traut <eric@traut.com>
Co-authored-by: Larry Hastings <larry@hastings.org>
Co-authored-by: Alex Waygood <Alex.Waygood@Gmail.com>
2023-05-15 20:36:23 -07:00
Hugo van Kemenade d513ddee94
Trim trailing whitespace and test on CI (#104275)
Co-authored-by: Alex Waygood <Alex.Waygood@Gmail.com>
2023-05-08 17:03:52 +03:00
Eric Snow a9c6e0618f
gh-99113: Add Py_MOD_PER_INTERPRETER_GIL_SUPPORTED (gh-104205)
Here we are doing no more than adding the value for Py_mod_multiple_interpreters and using it for stdlib modules.  We will start checking for it in gh-104206 (once PyInterpreterState.ceval.own_gil is added in gh-104204).
2023-05-05 21:11:27 +00:00
Pablo Galindo Salgado eba64d2afb
gh-104169: Ensure the tokenizer doesn't overwrite previous errors (#104170) 2023-05-04 15:15:26 +01:00
Lysandros Nikolaou ef0df5284f
gh-97556: Raise null bytes syntax error upon null in multiline string (GH-104136) 2023-05-04 14:26:23 +02:00
jx124 5078eedc5b
gh-104016: Fixed off by 1 error in f string tokenizer (#104047)
Co-authored-by: sunmy2019 <59365878+sunmy2019@users.noreply.github.com>
Co-authored-by: Ken Jin <kenjin@python.org>
Co-authored-by: Pablo Galindo <pablogsal@gmail.com>
2023-05-01 19:15:47 +00:00
chgnrdv d5a97074d2
gh-103824: fix use-after-free error in Parser/tokenizer.c (#103993) 2023-05-01 15:26:43 +00:00
Lysandros Nikolaou 9169a56fad
gh-103656: Transfer f-string buffers to parser to avoid use-after-free (GH-103896)
Co-authored-by: Pablo Galindo <pablogsal@gmail.com>
2023-04-27 01:33:31 +00:00
Lysandros Nikolaou 57f8f9a66d
gh-103718: Correctly set f-string buffers in all cases (GH-103815)
Turns out we always need to remember/restore fstring buffers in all of
the stack of tokenizer modes, cause they might change to
`TOK_REGULAR_MODE` and have newlines inside the braces (which is when we
need to reallocate the buffer and restore the fstring ones).
2023-04-25 01:31:21 +00:00
Lysandros Nikolaou cb157a1a35
GH-103727: Avoid advancing tokenizer too far in f-string mode (GH-103775) 2023-04-24 12:30:21 -06:00
Lysandros Nikolaou 05b3ce7339
GH-103718: Correctly cache and restore f-string buffers when needed (GH-103719) 2023-04-23 13:06:10 -06:00
Nikita Sobolev 0fd3891758
gh-102310: Change error range for invalid bytes literals (#103663) 2023-04-22 18:08:27 -06:00
Pablo Galindo Salgado d4aa8578b1
gh-102856: Clean some of the PEP 701 tokenizer implementation (#103634) 2023-04-19 14:51:31 -06:00
Pablo Galindo Salgado 1ef61cf71a
gh-102856: Initial implementation of PEP 701 (#102855)
Co-authored-by: Lysandros Nikolaou <lisandrosnik@gmail.com>
Co-authored-by: Batuhan Taskaya <isidentical@gmail.com>
Co-authored-by: Marta Gómez Macías <mgmacias@google.com>
Co-authored-by: sunmy2019 <59365878+sunmy2019@users.noreply.github.com>
2023-04-19 11:18:16 -05:00
Chenxi Mao 7703def37e
GH-102711: Fix warnings found by clang (#102712)
There are some warnings if build python via clang:

Parser/pegen.c:812:31: warning: a function declaration without a prototype is deprecated in all versions of C [-Wstrict-prototypes]
_PyPegen_clear_memo_statistics()
                              ^
                               void

Parser/pegen.c:820:29: warning: a function declaration without a prototype is deprecated in all versions of C [-Wstrict-prototypes]
_PyPegen_get_memo_statistics()
                            ^
                             void

Fix it to make clang happy.

Signed-off-by: Chenxi Mao <chenxi.mao@suse.com>
2023-03-28 10:52:22 +02:00
Max Bachmann c6858d1e7f
gh-102255: Improve build support for Windows API partitions (GH-102256)
Add `MS_WINDOWS_DESKTOP`, `MS_WINDOWS_APPS`, `MS_WINDOWS_SYSTEM` and `MS_WINDOWS_GAMES` preprocessor definitions to allow switching off functionality missing from particular API partitions ("partitions" are used in Windows to identify overlapping subsets of APIs).
CPython only officially supports `MS_WINDOWS_DESKTOP` and `MS_WINDOWS_SYSTEM` (APPS is included by normal desktop builds, but APPS without DESKTOP is not covered). Other configurations are a convenience for people building their own runtimes.
`MS_WINDOWS_GAMES` is for the Xbox subset of the Windows API, which is also available on client OS, but is restricted compared to `MS_WINDOWS_DESKTOP`. These restrictions may change over time, as they relate to the build headers rather than the OS support, and so we assume that Xbox builds will use the latest available version of the GDK.
2023-03-09 21:09:12 +00:00
Pablo Galindo Salgado f533f216e6
gh-102416: Do not memoize incorrectly loop rules in the parser (#102467) 2023-03-06 14:41:53 +01:00
Eric Snow 880437d4ec
gh-100227: Move _str_replace_inf to PyInterpreterState (gh-102333)
https://github.com/python/cpython/issues/100227
2023-02-28 14:16:39 -07:00