Commit Graph

1196 Commits

Author SHA1 Message Date
Pablo Galindo Salgado aab01e3524
gh-96670: Raise SyntaxError when parsing NULL bytes (#97594) 2022-09-27 23:23:42 +01:00
Lysandros Nikolaou 7e36abbb78
gh-91210: Improve error message when non-default param follows default (GH-95933)
- Improve error message when parameter without a default follows one with a default
- Show same error message when positional-only params precede the default/non-default sequence
2022-09-17 10:09:28 -07:00
Matthias Görgens 81e36f350b
gh-96678: Fix UB of null pointer arithmetic (GH-96782)
Automerge-Triggered-By: GH:pablogsal
2022-09-13 06:14:35 -07:00
Michael Droettboom 8bc356a7dd
gh-96268: Fix loading invalid UTF-8 (#96270)
This makes tokenizer.c:valid_utf8 match stringlib/codecs.h:decode_utf8.

It also fixes an off-by-one error introduced in 3.10 for the line number when the tokenizer reports bad UTF8.
2022-09-07 14:23:54 -07:00
Michael Droettboom 05692c67c5
gh-96611: Fix error message for invalid UTF-8 in mid-multiline string (#96623) 2022-09-07 00:12:16 +01:00
Nikita Sobolev 2c7d2e8d46
gh-96587: Raise `SyntaxError` for PEP654 on older `feature_version` (#96588) 2022-09-05 17:54:09 +01:00
Gregory P. Smith 511ca94520
gh-95778: CVE-2020-10735: Prevent DoS by very large int() (#96499)
Integer to and from text conversions via CPython's bignum `int` type is not safe against denial of service attacks due to malicious input. Very large input strings with hundred thousands of digits can consume several CPU seconds.

This PR comes fresh from a pile of work done in our private PSRT security response team repo.

Signed-off-by: Christian Heimes [Red Hat] <christian@python.org>
Tons-of-polishing-up-by: Gregory P. Smith [Google] <greg@krypto.org>
Reviews via the private PSRT repo via many others (see the NEWS entry in the PR).

<!-- gh-issue-number: gh-95778 -->
* Issue: gh-95778
<!-- /gh-issue-number -->

I wrote up [a one pager for the release managers](https://docs.google.com/document/d/1KjuF_aXlzPUxTK4BMgezGJ2Pn7uevfX7g0_mvgHlL7Y/edit#). Much of that text wound up in the Issue. Backports PRs already exist. See the issue for links.
2022-09-02 09:35:08 -07:00
Shantanu a965db37f2
gh-94996: Disallow lambda pos only params with feature_version < (3, 8) (GH-95934) 2022-08-12 20:41:02 +02:00
Shantanu b5e3ea2862
gh-94996: Disallow parsing pos only params with feature_version < (3, 8) (GH-94997) 2022-08-12 19:27:50 +02:00
Christian Heimes b4c857d0fd
gh-95876: Fix format string in pegen error location code (#95877) 2022-08-11 09:55:57 +01:00
Honglin Zhu b946f529ef
gh-95355: Check tokens[0] after allocating memory (GH-95356)
#95355

Automerge-Triggered-By: GH:pablogsal
2022-07-28 03:00:34 -07:00
Pablo Galindo Salgado 0047447294
gh-95185: Check recursion depth in the AST constructor (#95186)
Co-authored-by: Serhiy Storchaka <storchaka@gmail.com>
2022-07-24 15:58:52 +01:00
Shantanu 0daba82221
gh-94949: Disallow parsing parenthesised ctx mgr with old feature_version (#94950)
* gh-94949: Disallow parsing parenthesised ctx manager with old feature_version

* 📜🤖 Added by blurb_it.

* Allow it with feature_version=(3, 9) as well

Co-authored-by: blurb-it[bot] <43283697+blurb-it[bot]@users.noreply.github.com>
2022-07-18 22:10:49 +01:00
Shantanu ae0be5a53b
gh-94947: Disallow parsing walrus with feature_version < (3, 8) (#94948)
* gh-94947: Disallow parsing walrus with feature_version < (3, 8)

* oops, commit the parser

* 📜🤖 Added by blurb_it.

Co-authored-by: blurb-it[bot] <43283697+blurb-it[bot]@users.noreply.github.com>
2022-07-18 10:20:12 +01:00
Pablo Galindo Salgado 2e9da8e352
gh-94869: Fix the location in some expressions for multi-line f-string ast nodes (#94895) 2022-07-16 19:51:53 +01:00
Paul m. p. Peny bbb2ab70b6
[3.11] bpo-14916: interactive fd is not tied to stdin [type-bug] (#91469)
* bpo-14916: interactive fd is not always stdin

related to https://github.com/python/cpython/pull/31006 merged bugfix

following https://bugs.python.org/issue14916

* 📜🤖 Added by blurb_it.

Co-authored-by: blurb-it[bot] <43283697+blurb-it[bot]@users.noreply.github.com>
2022-07-16 09:35:19 +01:00
Pablo Galindo Salgado 36fcde61ba
gh-94360: Fix a tokenizer crash when reading encoded files with syntax errors from stdin (#94386)
* gh-94360: Fix a tokenizer crash when reading encoded files with syntax errors from stdin

Signed-off-by: Pablo Galindo <pablogsal@gmail.com>

* nitty nit

Co-authored-by: Łukasz Langa <lukasz@langa.pl>
2022-07-05 17:39:21 +01:00
wookie184 8c237a7a71
gh-94192: Fix error for dictionary literals with invalid expression as value. (#94304)
* Fix error for dictionary literals with invalid expression as value.

* Remove trailing whitespace
2022-06-26 19:42:47 +01:00
wookie184 2fc83ac3af
gh-92858: Improve error message for some suites with syntax error before ':' (#92894) 2022-06-23 17:31:09 +01:00
Victor Stinner 47e35625ff
gh-84623: Remove unused imports (#94132) 2022-06-22 19:14:27 +02:00
Victor Stinner cfb986a1a2
gh-93937: PyOS_StdioReadline() uses PyConfig.legacy_windows_stdio (#94024)
On Windows, PyOS_StdioReadline() now gets
PyConfig.legacy_windows_stdio from _PyOS_ReadlineTState, rather than
using the deprecated global Py_LegacyWindowsStdioFlag variable.

Fix also a compiler warning in Py_SetStandardStreamEncoding().
2022-06-20 16:10:47 +02:00
Serhiy Storchaka 6fd4c8ec77
gh-93741: Add private C API _PyImport_GetModuleAttrString() (GH-93742)
It combines PyImport_ImportModule() and PyObject_GetAttrString()
and saves 4-6 lines of code on every use.

Add also _PyImport_GetModuleAttr() which takes Python strings as arguments.
2022-06-14 07:15:26 +03:00
Pablo Galindo Salgado 53a8b17895
gh-93671: Avoid exponential backtracking in deeply nested sequence patterns in match statements (GH-93680)
Co-authored-by: Łukasz Langa <lukasz@langa.pl>
2022-06-10 17:56:45 +02:00
Eric V. Smith ee70c70aa9
gh-93418: Fix an assert when an f-string expression is followed by an '=', but no closing brace. (gh-93419) 2022-06-01 19:20:06 -04:00
Pablo Galindo Salgado 705eaec28f
gh-92597: Ensure that AST nodes without explicit end positions can be compiled (GH-93359) 2022-06-01 00:00:47 +01:00
Serhiy Storchaka 07df8d5b2c
gh-93283: Improve error message for f-string with invalid conversion character (GH-93349) 2022-05-31 20:38:29 +03:00
Kumar Aditya cb04a09d2d
GH-93207: Remove HAVE_STDARG_PROTOTYPES configure check for stdarg.h (#93215) 2022-05-27 13:30:45 +02:00
Victor Stinner 5115a16831
gh-93103: Parser uses PyConfig.parser_debug instead of Py_DebugFlag (#93106)
* Replace deprecated Py_DebugFlag with PyConfig.parser_debug in the
  parser.
* Add Parser.debug member.
* Add tok_state.debug member.
* Py_FrozenMain(): Replace Py_VerboseFlag with PyConfig.verbose.
2022-05-24 22:35:08 +02:00
Christian Heimes 137fd3d88a
gh-90473: Decrease recursion limit and skip tests on WASI (GH-92803) 2022-05-19 12:43:16 +02:00
Victor Stinner da5727a120
gh-92651: Remove the Include/token.h header file (#92652)
Remove the token.h header file. There was never any public tokenizer
C API. The token.h header file was only designed to be used by Python
internals.

Move Include/token.h to Include/internal/pycore_token.h. Including
this header file now requires that the Py_BUILD_CORE macro is
defined. It no longer checks for the Py_LIMITED_API macro.

Rename functions:

* PyToken_OneChar() => _PyToken_OneChar()
* PyToken_TwoChars() => _PyToken_TwoChars()
* PyToken_ThreeChars() => _PyToken_ThreeChars()
2022-05-11 23:22:50 +02:00
Victor Stinner d716a0dfe2
Use static inline function Py_EnterRecursiveCall() (#91988)
Currently, calling Py_EnterRecursiveCall() and
Py_LeaveRecursiveCall() may use a function call or a static inline
function call, depending if the internal pycore_ceval.h header file
is included or not. Use a different name for the static inline
function to ensure that the static inline function is always used in
Python internals for best performance. Similar approach than
PyThreadState_GET() (function call) and _PyThreadState_GET() (static
inline function).

* Rename _Py_EnterRecursiveCall() to _Py_EnterRecursiveCallTstate()
* Rename _Py_LeaveRecursiveCall() to _Py_LeaveRecursiveCallTstate()
* pycore_ceval.h: Rename Py_EnterRecursiveCall() to
  _Py_EnterRecursiveCall() and Py_LeaveRecursiveCall() and
  _Py_LeaveRecursiveCall()
2022-05-04 13:30:23 +02:00
Serhiy Storchaka 3483299a24
gh-81548: Deprecate octal escape sequences with value larger than 0o377 (GH-91668) 2022-04-30 13:16:27 +03:00
Serhiy Storchaka 43a8bf1ea4
gh-87999: Change warning type for numeric literal followed by keyword (GH-91980)
The warning emitted by the Python parser for a numeric literal
immediately followed by keyword has been changed from deprecation
warning to syntax warning.
2022-04-27 20:15:14 +03:00
Matthieu Dartiailh aa0f056a00
bpo-47212: Improve error messages for un-parenthesized generator expressions (GH-32302) 2022-04-05 14:47:13 +01:00
Christian Heimes 3df0e63aab
bpo-46315: Use fopencookie only on Emscripten 3.x and newer (GH-32266) 2022-04-02 23:11:38 +02:00
Hugo van Kemenade 6881ea936e
bpo-47126: Update to canonical PEP URLs specified by PEP 676 (GH-32124) 2022-03-30 12:00:27 +01:00
Maciej Górski 7b44ade018
bpo-47129: Add more informative messages to f-string syntax errors (32127)
* Add more informative messages to f-string syntax errors

* 📜🤖 Added by blurb_it.

* Fix whitespaces

* Change error message

* Remove the 'else' statement (as sugested in review)

Co-authored-by: blurb-it[bot] <43283697+blurb-it[bot]@users.noreply.github.com>
2022-03-28 17:08:36 -04:00
Matthew Rahtz e8e737bcf6
bpo-43224: Implement PEP 646 grammar changes (GH-31018)
Co-authored-by: Jelle Zijlstra <jelle.zijlstra@gmail.com>
2022-03-26 09:55:35 -07:00
Pablo Galindo Salgado 26cca8067b
bpo-47117: Don't crash if we fail to decode characters when the tokenizer buffers are uninitialized (GH-32129)
Automerge-Triggered-By: GH:pablogsal
2022-03-26 09:29:02 -07:00
Christian Heimes 9b889b5bda
bpo-46315: Use fopencookie() to avoid dup() in _PyTokenizer_FindEncodingFilename (GH-32033)
WASI does not have dup() and Emscripten's emulation is slow.
2022-03-22 17:08:51 +01:00
Pablo Galindo Salgado 7d810b6a4e
bpo-46838: Syntax error improvements for function definitions (GH-31590) 2022-03-22 11:38:41 +00:00
Oleg Iarygin 13b0412223
bpo-46920: Remove code that has explainers why it was disabled (GH-31813) 2022-03-14 17:04:22 +01:00
Oleg Iarygin a52f82baf2
bpo-46920: Remove disabled debug code added decades ago and likely unnecessary (GH-31812) 2022-03-14 17:03:21 +01:00
Serhiy Storchaka 090e5c4b94
bpo-46820: Fix a SyntaxError in a numeric literal followed by "not in" (GH-31479)
Fix parsing a numeric literal immediately (without spaces) followed by
"not in" keywords, like in "1not in x". Now the parser only emits
a warning, not a syntax error.
2022-02-22 09:51:51 +02:00
Eric V. Smith ffd9f8ff84
bpo-46762: Fix an assert failure in f-strings where > or < is the last character if the f-string is missing a trailing right brace. (#31365) 2022-02-16 05:54:09 -05:00
Pablo Galindo Salgado e19059ecd8
Don't print rejected tokens when using the debug flags in the parser (GH-31258) 2022-02-10 14:38:27 +00:00
Pablo Galindo Salgado 390459de6d
Allow the parser to avoid nested processing of invalid rules (GH-31252) 2022-02-10 13:12:14 +00:00
Pablo Galindo Salgado b71dc71905
bpo-46707: Avoid potential exponential backtracking in some syntax errors (GH-31241) 2022-02-10 03:37:17 +00:00
Eric Snow 81c72044a1
bpo-46541: Replace core use of _Py_IDENTIFIER() with statically initialized global objects. (gh-30928)
We're no longer using _Py_IDENTIFIER() (or _Py_static_string()) in any core CPython code.  It is still used in a number of non-builtin stdlib modules.

The replacement is: PyUnicodeObject (not pointer) fields under _PyRuntimeState, statically initialized as part of _PyRuntime.  A new _Py_GET_GLOBAL_IDENTIFIER() macro facilitates lookup of the fields (along with _Py_GET_GLOBAL_STRING() for non-identifier strings).

https://bugs.python.org/issue46541#msg411799 explains the rationale for this change.

The core of the change is in:

* (new) Include/internal/pycore_global_strings.h - the declarations for the global strings, along with the macros
* Include/internal/pycore_runtime_init.h - added the static initializers for the global strings
* Include/internal/pycore_global_objects.h - where the struct in pycore_global_strings.h is hooked into _PyRuntimeState
* Tools/scripts/generate_global_objects.py - added generation of the global string declarations and static initializers

I've also added a --check flag to generate_global_objects.py (along with make check-global-objects) to check for unused global strings.  That check is added to the PR CI config.

The remainder of this change updates the core code to use _Py_GET_GLOBAL_IDENTIFIER() instead of _Py_IDENTIFIER() and the related _Py*Id functions (likewise for _Py_GET_GLOBAL_STRING() instead of _Py_static_string()).  This includes adding a few functions where there wasn't already an alternative to _Py*Id(), replacing the _Py_Identifier * parameter with PyObject *.

The following are not changed (yet):

* stop using _Py_IDENTIFIER() in the stdlib modules
* (maybe) get rid of _Py_IDENTIFIER(), etc. entirely -- this may not be doable as at least one package on PyPI using this (private) API
* (maybe) intern the strings during runtime init

https://bugs.python.org/issue46541
2022-02-08 13:39:07 -07:00
Pablo Galindo Salgado 69e10976b2
bpo-46521: Fix codeop to use a new partial-input mode of the parser (GH-31010) 2022-02-08 11:54:37 +00:00
Paul m. p. P 89b13042fc
bpo-14916: use specified tokenizer fd for file input (GH-31006)
@pablogsal, sorry i failed to rebase to main, so i recreated https://github.com/python/cpython/pull/22190#issuecomment-1024633392

> PyRun_InteractiveOne\*() functions allow to explicitily set fd instead of stdin.
but stdin was hardcoded in readline call.

> This patch does not fix target file for prompt unlike original bpo one : prompt fd is unrelated to tokenizer source which could be read only. It is more of a bugfix regarding the docs :  actual documentation say "prompt the user" so one would expect prompt to go on stdout not a file for both PyRun_InteractiveOne\*() and PyRun_InteractiveLoop\*().

Automerge-Triggered-By: GH:pablogsal
2022-02-01 14:33:52 -08:00
Pablo Galindo Salgado a0efc0c196
bpo-46091: Correctly calculate indentation levels for whitespace lines with continuation characters (GH-30130) 2022-01-25 22:12:14 +00:00
Eric V. Smith 0daf72194b
bpo-46503: Prevent an assert from firing when parsing some invalid \N sequences in f-strings. (GH-30865)
* bpo-46503: Prevent an assert from firing.  Also fix one nearby tiny PEP-7 nit.

* Added blurb.
2022-01-24 21:53:27 -05:00
Pablo Galindo Salgado 650720a0cf
Fix the caret position in some syntax errors in interactive mode (GH-30718) 2022-01-20 15:34:13 +00:00
Pablo Galindo Salgado 8c2fd09f36
bpo-46339: Include clarification on assert in 'get_error_line_from_tokenizer_buffers' (#30545) 2022-01-18 11:13:00 +00:00
Pablo Galindo Salgado cedec19be8
bpo-46339: Fix crash in the parser when computing error text for multi-line f-strings (GH-30529)
Automerge-Triggered-By: GH:pablogsal
2022-01-11 08:30:39 -08:00
Pablo Galindo Salgado 6fa8b2ceee
bpo-46237: Fix the line number of tokenizer errors inside f-strings (GH-30463) 2022-01-08 00:23:40 +00:00
Batuhan Taskaya d382f7ee0b
bpo-46289: Make conversion of FormattedValue not optional on ASDL (GH-30467)
Automerge-Triggered-By: GH:isidentical
2022-01-07 13:05:28 -08:00
Pablo Galindo Salgado 70f415fb8b
bpo-46240: Correct the error for unclosed parentheses when the tokenizer is not finished (GH-30378) 2022-01-04 10:41:22 +00:00
Pablo Galindo Salgado dd6c35761a
bpo-46110: Restore commit e9898bf153
This restores commit e9898bf153 .
2022-01-03 19:54:06 +00:00
Pablo Galindo Salgado 9d35dedc5e
Revert "bpo-46110: Add a recursion check to avoid stack overflow in the PEG parser (GH-30177)" (GH-30363)
This reverts commit e9898bf153 temporarily as we want to confirm if this commit is the cause of a slowdown at startup time.
2022-01-03 18:29:18 +00:00
Pablo Galindo Salgado e9898bf153
bpo-46110: Add a recursion check to avoid stack overflow in the PEG parser (GH-30177)
Co-authored-by: Batuhan Taskaya <isidentical@gmail.com>
2021-12-20 15:43:26 +00:00
Irit Katriel d60457a667
bpo-45292: [PEP-654] add except* (GH-29581) 2021-12-14 16:48:15 +00:00
Kumar Aditya 41026c3155
bpo-45855: Replaced deprecated `PyImport_ImportModuleNoBlock` with PyImport_ImportModule (GH-30046) 2021-12-12 10:45:20 +02:00
Pablo Galindo Salgado 4325a766f5
bpo-46054: Fix parsing error when parsing non-utf8 characters in source files (GH-30068) 2021-12-12 07:06:50 +00:00
Weipeng Hong 28179aac79
bpo-42918: Improve build-in function compile() in mode 'single' (GH-29934)
Co-authored-by: Alex Waygood <Alex.Waygood@Gmail.com>
2021-12-11 00:44:26 +01:00
Pablo Galindo Salgado 1c7a1c3be0
bpo-46004: Fix error location for loops with invalid targets (GH-29959) 2021-12-07 13:02:15 +00:00
Victor Stinner 253b7a0a9f
bpo-45866: pegen strips directory of "generated from" header (GH-29777)
"make regen-all" now produces the same output when run from a
directory other than the source tree: when building Python out of the
source tree.
2021-11-26 11:50:34 +01:00
Pablo Galindo Salgado 24c10d2943
bpo-45727: Only trigger the 'did you forgot a comma' error suggestion if inside parentheses (GH-29757) 2021-11-24 22:21:23 +00:00
Pablo Galindo Salgado 4f006a789a
Ensure the str member of the tokenizer is always initialised (GH-29681) 2021-11-21 02:06:39 +00:00
Pablo Galindo Salgado c9c4444d9f
Refactor parser compilation units into specific components (GH-29676) 2021-11-21 01:08:50 +00:00
Pablo Galindo Salgado 81f4e116ef
bpo-45811: Improve error message when source code contains invisible control characters (GH-29654) 2021-11-20 18:28:28 +00:00
Pablo Galindo Salgado 7a1d932528
bpo-45450: Improve syntax error for parenthesized arguments (GH-28906) 2021-11-20 18:27:40 +00:00
Pablo Galindo Salgado 79ff0d1687
bpo-45494: Fix error location in EOF tokenizer errors (GH-29108) 2021-11-20 17:40:59 +00:00
Pablo Galindo Salgado fdcc46d955
bpo-45848: Allow the parser to get error lines from encoded files (GH-29646) 2021-11-20 15:36:07 +01:00
Pablo Galindo Salgado 546cefcda7
bpo-45727: Make the syntax error for missing comma more consistent (GH-29427) 2021-11-19 23:11:57 +00:00
Pablo Galindo Salgado da20d7401d
bpo-45822: Respect PEP 263's coding cookies in the parser even if flags are not provided (GH-29582) 2021-11-16 12:30:47 -08:00
Pablo Galindo Salgado df4ae55e66
bpo-45820: Fix a segfault when the parser fails without reading any input (GH-29580) 2021-11-16 19:51:52 +00:00
Pablo Galindo Salgado 25835c518a
bpo-45738: Fix computation of error location for invalid continuation (GH-29550)
characters in the parser
2021-11-14 01:06:41 +00:00
Carl Friedrich Bolz-Tereick 2819e98d10
bpo-45764: improve error message when missing '(' after 'def' (GH-29484)
to achieve this, change the grammar to expect the '(' token after 'def' NAME.

Automerge-Triggered-By: GH:pablogsal
2021-11-09 06:03:32 -08:00
Pablo Galindo Salgado e2d65630f3
bpo-45716: Improve the error message when using True/False/None as keywords in a call (GH-29413) 2021-11-05 13:54:55 +00:00
wim glenn 762173c670
bpo-44257: fix "assigment_expr" typo + regenerate the grammar, and remove unused imports (GH-29393)
Co-authored-by: Wim Glenn <wglenn@jumptrading.com>
2021-11-03 20:57:40 +00:00
Pablo Galindo Salgado cdc7a58277
bpo-45562: Ensure all tokenizer debug messages are printed to stderr (GH-29270) 2021-10-28 18:06:15 +01:00
Pablo Galindo Salgado 10bbd41ba8
bpo-45562: Print tokenizer debug messages to stderr (GH-29250) 2021-10-27 14:27:34 -07:00
Nikita Sobolev 4bc5473a42
bpo-45574: fix warning about `print_escape` being unused (GH-29172)
It used to be like this:
<img width="1232" alt="Снимок экрана 2021-10-22 в 23 07 40" src="https://user-images.githubusercontent.com/4660275/138516608-fef6ec01-a96a-40f4-81ef-52265b0f536b.png">

Quick `grep` tells that it is just used in one place under `Py_DEBUG`: f6e8b80d20/Parser/tokenizer.c (L1047-L1051)
<img width="752" alt="Снимок экрана 2021-10-22 в 23 08 09" src="https://user-images.githubusercontent.com/4660275/138516684-ea503136-1e92-48a5-95bb-419e190d5866.png">

I am not sure, but it also looks like a private thing, it should not affect other users.

Automerge-Triggered-By: GH:pablogsal
2021-10-22 14:57:24 -07:00
Pablo Galindo Salgado 86dfb55d2e
bpo-45562: Only show debug output from the parser in debug builds (GH-29140) 2021-10-22 01:52:24 -07:00
Pablo Galindo Salgado a106343f63
bpo-45494: Fix parser crash when reporting errors involving invalid continuation characters (GH-28993)
There are two errors that this commit fixes:

* The parser was not correctly computing the offset and the string
  source for E_LINECONT errors due to the incorrect usage of strtok().
* The parser was not correctly unwinding the call stack when a tokenizer
  exception happened in rules involving optionals ('?', [...]) as we
  always make them return valid results by using the comma operator. We
  need to check first if we don't have an error before continuing.
2021-10-19 21:24:12 +02:00
Serhiy Storchaka c96d1546b1
bpo-45461: Fix IncrementalDecoder and StreamReader in the "unicode-escape" codec (GH-28939)
They support now splitting escape sequences between input chunks.

Add the third parameter "final" in codecs.unicode_escape_decode().
It is True by default to match the former behavior.
2021-10-14 13:17:00 +03:00
Victor Stinner 713bb19356
bpo-45434: Mark the PyTokenizer C API as private (GH-28924)
Rename PyTokenize functions to mark them as private:

* PyTokenizer_FindEncodingFilename() => _PyTokenizer_FindEncodingFilename()
* PyTokenizer_FromString() => _PyTokenizer_FromString()
* PyTokenizer_FromFile() => _PyTokenizer_FromFile()
* PyTokenizer_FromUTF8() => _PyTokenizer_FromUTF8()
* PyTokenizer_Free() => _PyTokenizer_Free()
* PyTokenizer_Get() => _PyTokenizer_Get()

Remove the unused PyTokenizer_FindEncoding() function.

import.c: remove unused #include "errcode.h".
2021-10-13 17:22:14 +02:00
Victor Stinner 97308dfcdc
bpo-45434: Move _Py_BEGIN_SUPPRESS_IPH to pycore_fileutils.h (GH-28922) 2021-10-13 15:03:35 +02:00
Victor Stinner d943d19172
bpo-45439: Move _PyObject_CallNoArgs() to pycore_call.h (GH-28895)
* Move _PyObject_CallNoArgs() to pycore_call.h (internal C API).
* _ssl, _sqlite and _testcapi extensions now call the public
  PyObject_CallNoArgs() function, rather than _PyObject_CallNoArgs().
* _lsprof extension is now built with Py_BUILD_CORE_MODULE macro
  defined to get access to internal _PyObject_CallNoArgs().
2021-10-12 08:38:19 +02:00
Victor Stinner ce3489cfdb
bpo-45439: Rename _PyObject_CallNoArg() to _PyObject_CallNoArgs() (GH-28891)
Fix typo in the private _PyObject_CallNoArg() function name: rename
it to _PyObject_CallNoArgs() to be consistent with the public
function PyObject_CallNoArgs().
2021-10-12 00:42:23 +02:00
Pablo Galindo Salgado 0219017df7
bpo-45408: Don't override previous tokenizer errors in the second parser pass (GH-28812) 2021-10-07 22:33:05 +01:00
Christian Clauss 5f401f1040
Fix typos in the Objects directory (GH-28766) 2021-10-06 16:57:10 -07:00
Rajendra arora a47d67cf46
Optimized code format (GH-28599)
Automerge-Triggered-By: GH:pablogsal
2021-09-28 04:51:39 -07:00
Rajendra arora 8b7427b554
Fixed typo in "decclarations" (GH-28578) 2021-09-28 13:56:41 +03:00
Pablo Galindo Salgado e5f13ce5b4
bpo-43914: Correctly highlight SyntaxError exceptions for invalid generator expression in function calls (GH-28576) 2021-09-27 14:37:43 +01:00
Pablo Galindo Salgado b01fd533fe
Extract visitors from the grammar nodes and call makers in the peg generator (GH-28172)
Simplify the peg generator logic by extracting as much visitors as possible to disentangle the flow and separate concerns.
2021-09-05 14:58:52 +01:00
Pablo Galindo Salgado b2f68b1900
bpo-44947: Refine the syntax error for trailing commas in import statements (GH-27814) 2021-08-18 22:09:21 +02:00
Pablo Galindo Salgado 953d27261e
Update pegen to use the latest upstream developments (GH-27586) 2021-08-12 17:37:30 +01:00
Pablo Galindo Salgado 8e832fb2a2
bpo-44885: Correct the ast locations of f-strings with format specs and repeated expressions (GH-27729) 2021-08-12 17:13:30 +01:00
Serhiy Storchaka 058fb35b57
bpo-44854: Remove trailing whitespaces (GH-27689) 2021-08-09 21:32:54 +03:00
Pablo Galindo Salgado f5cbea6b1b
bpo-44838: Refine the custom syntax errors for invalid 'if' expressions (GH-27615) 2021-08-05 18:28:57 +01:00
Miguel Brito 28b6dc9dd5
bpo-44792: Improve syntax errors for if expressions (GH-27506) 2021-08-02 18:11:37 +01:00
Pablo Galindo Salgado 208a7e957b
bpo-34013: Don't consider a grouped expression when reporting legacy print syntax errors (GH-27521) 2021-08-01 02:10:50 +01:00
Pablo Galindo Salgado 302cf35505
Divide the grammar into sections to improve readability (GH-27502) 2021-07-31 13:04:37 +01:00
Noah Kantrowitz be42c06bb0
Update URLs in comments and metadata to use HTTPS (GH-27458) 2021-07-30 15:54:46 +02:00
Pablo Galindo Salgado ecc3c8e421
bpo-34013: Move the Python 2 hints from the exception constructor to the parser (GH-27392) 2021-07-27 21:30:32 +01:00
Pablo Galindo Salgado 6948964ecf
bpo-34013: Generalize the invalid legacy statement error message (GH-27389) 2021-07-27 17:19:22 +01:00
Batuhan Taskaya fbc349ff79
bpo-43950: Distinguish errors happening on character offset decoding (GH-27217) 2021-07-20 16:42:12 +01:00
Pablo Galindo Salgado a283ef116b
Remove unnecessary 'invalid_primary' rule in the parser (GH-27186) 2021-07-16 13:20:09 +01:00
Pablo Galindo Salgado f24777c2b3
bpo-44317: Improve tokenizer errors with more informative locations (GH-26555) 2021-07-10 01:29:29 +01:00
Ammar Askar 5644c7b3ff
bpo-43950: Print columns in tracebacks (PEP 657) (GH-26958)
The traceback.c and traceback.py mechanisms now utilize the newly added code.co_positions and PyCode_Addr2Location
to print carets on the specific expressions involved in a traceback.

Co-authored-by: Pablo Galindo <Pablogsal@gmail.com>
Co-authored-by: Ammar Askar <ammar@ammaraskar.com>
Co-authored-by: Batuhan Taskaya <batuhanosmantaskaya@gmail.com>
2021-07-05 00:14:33 +01:00
Batuhan Taskaya 107a2c59c9
bpo-40528: fix is_simple(sum)s behavior for attributes (GH-26918)
This is something I noticed while (now discontinued) experimenting
with the idea of annotating operators with location information. Unfortunately
without this addition, adding any `attributes` to stuff like `unaryop`
doesn't change anything since the code assumes they are singletons and
caches all instances. This patch fixes this assumption with including
the attributes as well as constructor fields.
2021-06-27 17:58:32 +03:00
Pablo Galindo 0acc258fe6
bpo-44456: Improve the syntax error when mixing keyword and positional patterns (GH-26793) 2021-06-24 16:09:57 +01:00
Batuhan Taskaya 6c76df2b86
bpo-40528: move asdl identifier collection to the new metadata system (GH-26858) 2021-06-24 15:16:00 +03:00
Batuhan Taskaya 35ad425866
bpo-40528: Implement a metadata system for ASDL Generator (GH-20193)
ASDL Generator was lack of proper annotation related to generated
module. This patch implements a MetadataVisitor that produces a
metadata object to pass to other visitors that are visiting that
same module. For the inital patch, it dynamically retrieves int
sequences (like cmpop), that was previously hardcoded. It offers
an interface that is easy to extend.
2021-06-22 19:29:42 +03:00
Pablo Galindo 507ed6fa1d
bpo-44409: Fix error location in tokenizer errors that happen during initialization (GH-26712) 2021-06-14 17:46:11 +01:00
Binbin 17b16e13bb
Fix typos in multiple files (GH-26689)
Co-authored-by: Terry Jan Reedy <tjreedy@udel.edu>
2021-06-12 22:47:44 -04:00
Pablo Galindo a342cc5891
bpo-44396: Update multi-line-start location when reallocating tokenizer buffers (GH-26676)
Automerge-Triggered-By: GH:pablogsal
2021-06-12 10:53:49 -07:00
Serhiy Storchaka be8b631b7a
Add more const modifiers. (GH-26691) 2021-06-12 16:11:59 +03:00
Pablo Galindo 05073036dc
bpo-44368: Improve syntax errors with invalid as pattern targets (GH-26632) 2021-06-10 23:50:32 +01:00
Lysandros Nikolaou e7b4644607
bpo-44385: Remove unused grammar rules (GH-26655)
Automerge-Triggered-By: GH:lysnikolaou
2021-06-10 15:05:06 -07:00
Pablo Galindo 457ce60fc7
bpo-44368: Ensure we don't raise incorrect custom syntax errors with soft keywords (GH-26630) 2021-06-09 22:20:01 +01:00
Akira Nonaka aef1b58dc8
bpo-44345: Fix 'generated by' comment in parser.c (GH-26615) 2021-06-09 16:38:53 +02:00
Pablo Galindo 9fd21f649d
bpo-44349: Fix edge case when displaying text from files with encoding in syntax errors (GH-26611) 2021-06-09 00:54:29 +01:00
Serhiy Storchaka 2ea6d89028
bpo-43833: Emit warnings for numeric literals followed by keyword (GH-25466)
Emit a deprecation warning if the numeric literal is immediately followed by
one of keywords: and, else, for, if, in, is, or. Raise a syntax error with
more informative message if it is immediately followed by other keyword or
identifier.

Automerge-Triggered-By: GH:pablogsal
2021-06-08 16:31:10 -07:00
Pablo Galindo bafe0aade5
bpo-44335: Ensure the tokenizer doesn't go into Python with the error set (GH-26608) 2021-06-08 20:02:03 +01:00
Pablo Galindo d334c73b56
bpo-44335: Fix a regression when identifying invalid characters in syntax errors (GH-26589) 2021-06-08 12:25:22 +01:00
Ned Batchelder ffd87b7093
fix: use unambiguous punction in 'invalid escape sequence' message (GH-26582) 2021-06-08 01:15:46 +01:00
Pablo Galindo b250f89bb7
bpo-44305: Improve syntax error for try blocks without except or finally (GH-26523) 2021-06-03 23:52:12 +01:00
Batuhan Taskaya f3491242e4
bpo-11105: Do not crash when compiling recursive ASTs (GH-20594)
When compiling an AST object with a direct / indirect reference
cycles, on the conversion phase because of exceeding amount of
calls, a segfault was raised. This patch adds recursion guards to
places for preventing user inputs to not to crash AST but instead
raise a RecursionError.
2021-06-03 21:01:02 +01:00
Serhiy Storchaka 39dd141a4b
bpo-44273: Improve syntax error message for assigning to "..." (GH-26477)
Use "ellipsis" instead of "Ellipsis" in syntax error messages to eliminate confusion with built-in variable Ellipsis.
2021-06-01 12:07:05 +01:00
Pablo Galindo bd7476dae3
bpo-44201: Avoid side effects of "invalid_*" rules in the REPL (GH-26298)
When the parser does a second pass to check for errors, these rules can
have some small side-effects as they may advance the parser more than
the point reached in the first pass. This can cause the tokenizer to ask
for extra tokens in interactive mode causing the tokenizer to show the
prompt instead of failing instantly.

To avoid this, add a new mode to the tokenizer that is activated in the
second pass and deactivates asking for new tokens when the interactive
line is finished. As the parsing should have reached the last line in
the first pass, the second pass should not need to ask for more tokens.
2021-05-22 23:05:00 +01:00
Pablo Galindo c878a97968
bpo-44180: Fix edge cases in invalid assigment rules in the parser (GH-26283)
The invalid assignment rules are very delicate since the parser can
easily raise an invalid assignment when a keyword argument is provided.
As they are very deep into the grammar tree, is very difficult to
specify in which contexts these rules can be used and in which don't.
For that, we need to use a different version of the rule that doesn't do
error checking in those situations where we don't want the rule to raise
(keyword arguments and generator expressions).

We also need to check if we are in left-recursive rule, as those can try
to eagerly advance the parser even if the parse will fail at the end of
the expression. Failing to do this allows the parser to start parsing a
call as a tuple and incorrectly identify a keyword argument as an
invalid assignment, before it realizes that it was not a tuple after all.
2021-05-21 18:34:54 +01:00
Pablo Galindo b51081c1a8
bpo-44180: Report generic syntax errors in the furthest position reached in the first parser pass (GH-26253) 2021-05-21 16:09:51 +01:00
Pablo Galindo 33c0c90dea
bpo-44168: Fix error message in the parser for keyword arguments for invalid expressions (GH-26210) 2021-05-19 19:03:04 +01:00
Pablo Galindo 80b089179f
bpo-44143: Fix crash in the parser when raising tokenizer errors with an exception set (GH-26144) 2021-05-15 17:58:02 +01:00
Pablo Galindo 6692dc1ca9
bpo-43149: Correct the syntax error message for multiple exception types (GH-25996)
Automerge-Triggered-By: GH:pablogsal
2021-05-08 11:24:41 -07:00
Pablo Galindo 9142088e74
bpo-43822: Prioritize tokenizer errors over custom syntax errors when raising parser exceptions (GH-25866) 2021-05-04 01:32:46 +01:00
Brandt Bucher dbe60ee09d
bpo-43892: Validate the first term of complex literal value patterns (GH-25735) 2021-04-29 17:19:28 -07:00
Nick Coghlan 1e7b858575
bpo-43892: Make match patterns explicit in the AST (GH-25585)
Co-authored-by: Brandt Bucher <brandtbucher@gmail.com>
2021-04-28 22:58:44 -07:00
Segev Finer 5e437fb872
bpo-30555: Fix WindowsConsoleIO fails in the presence of fd redirection (GH-1927)
This works by not caching the handle and instead getting the handle from
the file descriptor each time, so that if the actual handle changes by
fd redirection closing/opening the console handle beneath our feet, we
will keep working correctly.
2021-04-23 23:00:27 +01:00
Pablo Galindo a77aac4fca
bpo-43914: Highlight invalid ranges in SyntaxErrors (#25525)
To improve the user experience understanding what part of the error messages associated with SyntaxErrors is wrong, we can highlight the whole error range and not only place the caret at the first character. In this way:

>>> foo(x, z for z in range(10), t, w)
  File "<stdin>", line 1
    foo(x, z for z in range(10), t, w)
           ^
SyntaxError: Generator expression must be parenthesized

becomes

>>> foo(x, z for z in range(10), t, w)
  File "<stdin>", line 1
    foo(x, z for z in range(10), t, w)
           ^^^^^^^^^^^^^^^^^^^^
SyntaxError: Generator expression must be parenthesized
2021-04-23 14:27:05 +01:00
Pablo Galindo 56c95dfe27
bpo-43859: Improve the error message for IndentationError exceptions (GH-25431) 2021-04-21 15:28:21 +01:00
Pablo Galindo b5b98bd8f8
bpo-43823: Fix location of one of the errors for invalid dictionary literals (GH-25427) 2021-04-16 00:45:42 +01:00
Pablo Galindo b280248be8
bpo-43822: Improve syntax errors for missing commas (GH-25377) 2021-04-15 21:38:45 +01:00
Pablo Galindo da74350174
bpo-43823: Improve syntax errors for invalid dictionary literals (GH-25378) 2021-04-15 14:06:39 +01:00
Pablo Galindo 30ed93bfec
bpo-43797: Handle correctly invalid assignments inside function calls and generators (GH-25390) 2021-04-13 17:51:21 +01:00
Pablo Galindo d9151cb453
Ensure that early = are not matched by the parser as invalid comparisons (GH-25375) 2021-04-13 02:32:33 +01:00