6fa8b2ceee
bpo-46237: Fix the line number of tokenizer errors inside f-strings (GH-30463)
2022-01-08 00:23:40 +00:00
d382f7ee0b
bpo-46289: Make conversion of FormattedValue not optional on ASDL (GH-30467)
...
Automerge-Triggered-By: GH:isidentical
2022-01-07 13:05:28 -08:00
70f415fb8b
bpo-46240: Correct the error for unclosed parentheses when the tokenizer is not finished (GH-30378)
2022-01-04 10:41:22 +00:00
dd6c35761a
bpo-46110: Restore commit e9898bf153
...
This restores commit e9898bf153
2022-01-03 19:54:06 +00:00
9d35dedc5e
Revert "bpo-46110: Add a recursion check to avoid stack overflow in the PEG parser (GH-30177)" (GH-30363)
...
This reverts commit e9898bf153
2022-01-03 18:29:18 +00:00
e9898bf153
bpo-46110: Add a recursion check to avoid stack overflow in the PEG parser (GH-30177)
...
Co-authored-by: Batuhan Taskaya <isidentical@gmail.com>
2021-12-20 15:43:26 +00:00
d60457a667
bpo-45292: [PEP-654] add except* (GH-29581)
2021-12-14 16:48:15 +00:00
41026c3155
bpo-45855: Replaced deprecated `PyImport_ImportModuleNoBlock` with PyImport_ImportModule (GH-30046)
2021-12-12 10:45:20 +02:00
4325a766f5
bpo-46054: Fix parsing error when parsing non-utf8 characters in source files (GH-30068)
2021-12-12 07:06:50 +00:00
28179aac79
bpo-42918: Improve build-in function compile() in mode 'single' (GH-29934)
...
Co-authored-by: Alex Waygood <Alex.Waygood@Gmail.com>
2021-12-11 00:44:26 +01:00
1c7a1c3be0
bpo-46004: Fix error location for loops with invalid targets (GH-29959)
2021-12-07 13:02:15 +00:00
253b7a0a9f
bpo-45866: pegen strips directory of "generated from" header (GH-29777)
...
"make regen-all" now produces the same output when run from a
directory other than the source tree: when building Python out of the
source tree.
2021-11-26 11:50:34 +01:00
24c10d2943
bpo-45727: Only trigger the 'did you forgot a comma' error suggestion if inside parentheses (GH-29757)
2021-11-24 22:21:23 +00:00
4f006a789a
Ensure the str member of the tokenizer is always initialised (GH-29681)
2021-11-21 02:06:39 +00:00
c9c4444d9f
Refactor parser compilation units into specific components (GH-29676)
2021-11-21 01:08:50 +00:00
81f4e116ef
bpo-45811: Improve error message when source code contains invisible control characters (GH-29654)
2021-11-20 18:28:28 +00:00
7a1d932528
bpo-45450: Improve syntax error for parenthesized arguments (GH-28906)
2021-11-20 18:27:40 +00:00
79ff0d1687
bpo-45494: Fix error location in EOF tokenizer errors (GH-29108)
2021-11-20 17:40:59 +00:00
fdcc46d955
bpo-45848: Allow the parser to get error lines from encoded files (GH-29646)
2021-11-20 15:36:07 +01:00
546cefcda7
bpo-45727: Make the syntax error for missing comma more consistent (GH-29427)
2021-11-19 23:11:57 +00:00
da20d7401d
bpo-45822: Respect PEP 263's coding cookies in the parser even if flags are not provided (GH-29582)
2021-11-16 12:30:47 -08:00
df4ae55e66
bpo-45820: Fix a segfault when the parser fails without reading any input (GH-29580)
2021-11-16 19:51:52 +00:00
25835c518a
bpo-45738: Fix computation of error location for invalid continuation (GH-29550)
...
characters in the parser
2021-11-14 01:06:41 +00:00
2819e98d10
bpo-45764: improve error message when missing '(' after 'def' (GH-29484)
...
to achieve this, change the grammar to expect the '(' token after 'def' NAME.
Automerge-Triggered-By: GH:pablogsal
2021-11-09 06:03:32 -08:00
e2d65630f3
bpo-45716: Improve the error message when using True/False/None as keywords in a call (GH-29413)
2021-11-05 13:54:55 +00:00
762173c670
bpo-44257: fix "assigment_expr" typo + regenerate the grammar, and remove unused imports (GH-29393)
...
Co-authored-by: Wim Glenn <wglenn@jumptrading.com>
2021-11-03 20:57:40 +00:00
cdc7a58277
bpo-45562: Ensure all tokenizer debug messages are printed to stderr (GH-29270)
2021-10-28 18:06:15 +01:00
10bbd41ba8
bpo-45562: Print tokenizer debug messages to stderr (GH-29250)
2021-10-27 14:27:34 -07:00
4bc5473a42
bpo-45574: fix warning about `print_escape` being unused (GH-29172)
...
It used to be like this:
<img width="1232" alt="Снимок экрана 2021-10-22 в 23 07 40" src="https://user-images.githubusercontent.com/4660275/138516608-fef6ec01-a96a-40f4-81ef-52265b0f536b.png ">
Quick `grep` tells that it is just used in one place under `Py_DEBUG`: f6e8b80d20/Parser/tokenizer.c (L1047-L1051)https://user-images.githubusercontent.com/4660275/138516684-ea503136-1e92-48a5-95bb-419e190d5866.png ">
I am not sure, but it also looks like a private thing, it should not affect other users.
Automerge-Triggered-By: GH:pablogsal
2021-10-22 14:57:24 -07:00
86dfb55d2e
bpo-45562: Only show debug output from the parser in debug builds (GH-29140)
2021-10-22 01:52:24 -07:00
a106343f63
bpo-45494: Fix parser crash when reporting errors involving invalid continuation characters (GH-28993)
...
There are two errors that this commit fixes:
* The parser was not correctly computing the offset and the string
source for E_LINECONT errors due to the incorrect usage of strtok().
* The parser was not correctly unwinding the call stack when a tokenizer
exception happened in rules involving optionals ('?', [...]) as we
always make them return valid results by using the comma operator. We
need to check first if we don't have an error before continuing.
2021-10-19 21:24:12 +02:00
c96d1546b1
bpo-45461: Fix IncrementalDecoder and StreamReader in the "unicode-escape" codec (GH-28939)
...
They support now splitting escape sequences between input chunks.
Add the third parameter "final" in codecs.unicode_escape_decode().
It is True by default to match the former behavior.
2021-10-14 13:17:00 +03:00
713bb19356
bpo-45434: Mark the PyTokenizer C API as private (GH-28924)
...
Rename PyTokenize functions to mark them as private:
* PyTokenizer_FindEncodingFilename() => _PyTokenizer_FindEncodingFilename()
* PyTokenizer_FromString() => _PyTokenizer_FromString()
* PyTokenizer_FromFile() => _PyTokenizer_FromFile()
* PyTokenizer_FromUTF8() => _PyTokenizer_FromUTF8()
* PyTokenizer_Free() => _PyTokenizer_Free()
* PyTokenizer_Get() => _PyTokenizer_Get()
Remove the unused PyTokenizer_FindEncoding() function.
import.c: remove unused #include "errcode.h".
2021-10-13 17:22:14 +02:00
97308dfcdc
bpo-45434: Move _Py_BEGIN_SUPPRESS_IPH to pycore_fileutils.h (GH-28922)
2021-10-13 15:03:35 +02:00
d943d19172
bpo-45439: Move _PyObject_CallNoArgs() to pycore_call.h (GH-28895)
...
* Move _PyObject_CallNoArgs() to pycore_call.h (internal C API).
* _ssl, _sqlite and _testcapi extensions now call the public
PyObject_CallNoArgs() function, rather than _PyObject_CallNoArgs().
* _lsprof extension is now built with Py_BUILD_CORE_MODULE macro
defined to get access to internal _PyObject_CallNoArgs().
2021-10-12 08:38:19 +02:00
ce3489cfdb
bpo-45439: Rename _PyObject_CallNoArg() to _PyObject_CallNoArgs() (GH-28891)
...
Fix typo in the private _PyObject_CallNoArg() function name: rename
it to _PyObject_CallNoArgs() to be consistent with the public
function PyObject_CallNoArgs().
2021-10-12 00:42:23 +02:00
0219017df7
bpo-45408: Don't override previous tokenizer errors in the second parser pass (GH-28812)
2021-10-07 22:33:05 +01:00
5f401f1040
Fix typos in the Objects directory (GH-28766)
2021-10-06 16:57:10 -07:00
a47d67cf46
Optimized code format (GH-28599)
...
Automerge-Triggered-By: GH:pablogsal
2021-09-28 04:51:39 -07:00
8b7427b554
Fixed typo in "decclarations" (GH-28578)
2021-09-28 13:56:41 +03:00
e5f13ce5b4
bpo-43914: Correctly highlight SyntaxError exceptions for invalid generator expression in function calls (GH-28576)
2021-09-27 14:37:43 +01:00
b01fd533fe
Extract visitors from the grammar nodes and call makers in the peg generator (GH-28172)
...
Simplify the peg generator logic by extracting as much visitors as possible to disentangle the flow and separate concerns.
2021-09-05 14:58:52 +01:00
b2f68b1900
bpo-44947: Refine the syntax error for trailing commas in import statements (GH-27814)
2021-08-18 22:09:21 +02:00
953d27261e
Update pegen to use the latest upstream developments (GH-27586)
2021-08-12 17:37:30 +01:00
8e832fb2a2
bpo-44885: Correct the ast locations of f-strings with format specs and repeated expressions (GH-27729)
2021-08-12 17:13:30 +01:00
058fb35b57
bpo-44854: Remove trailing whitespaces (GH-27689)
2021-08-09 21:32:54 +03:00
f5cbea6b1b
bpo-44838: Refine the custom syntax errors for invalid 'if' expressions (GH-27615)
2021-08-05 18:28:57 +01:00
28b6dc9dd5
bpo-44792: Improve syntax errors for if expressions (GH-27506)
2021-08-02 18:11:37 +01:00
208a7e957b
bpo-34013: Don't consider a grouped expression when reporting legacy print syntax errors (GH-27521)
2021-08-01 02:10:50 +01:00
302cf35505
Divide the grammar into sections to improve readability (GH-27502)
2021-07-31 13:04:37 +01:00
be42c06bb0
Update URLs in comments and metadata to use HTTPS (GH-27458)
2021-07-30 15:54:46 +02:00
ecc3c8e421
bpo-34013: Move the Python 2 hints from the exception constructor to the parser (GH-27392)
2021-07-27 21:30:32 +01:00
6948964ecf
bpo-34013: Generalize the invalid legacy statement error message (GH-27389)
2021-07-27 17:19:22 +01:00
fbc349ff79
bpo-43950: Distinguish errors happening on character offset decoding (GH-27217)
2021-07-20 16:42:12 +01:00
a283ef116b
Remove unnecessary 'invalid_primary' rule in the parser (GH-27186)
2021-07-16 13:20:09 +01:00
f24777c2b3
bpo-44317: Improve tokenizer errors with more informative locations (GH-26555)
2021-07-10 01:29:29 +01:00
5644c7b3ff
bpo-43950: Print columns in tracebacks (PEP 657) (GH-26958)
...
The traceback.c and traceback.py mechanisms now utilize the newly added code.co_positions and PyCode_Addr2Location
to print carets on the specific expressions involved in a traceback.
Co-authored-by: Pablo Galindo <Pablogsal@gmail.com>
Co-authored-by: Ammar Askar <ammar@ammaraskar.com>
Co-authored-by: Batuhan Taskaya <batuhanosmantaskaya@gmail.com>
2021-07-05 00:14:33 +01:00
107a2c59c9
bpo-40528: fix is_simple(sum)s behavior for attributes (GH-26918)
...
This is something I noticed while (now discontinued) experimenting
with the idea of annotating operators with location information. Unfortunately
without this addition, adding any `attributes` to stuff like `unaryop`
doesn't change anything since the code assumes they are singletons and
caches all instances. This patch fixes this assumption with including
the attributes as well as constructor fields.
2021-06-27 17:58:32 +03:00
0acc258fe6
bpo-44456: Improve the syntax error when mixing keyword and positional patterns (GH-26793)
2021-06-24 16:09:57 +01:00
6c76df2b86
bpo-40528: move asdl identifier collection to the new metadata system (GH-26858)
2021-06-24 15:16:00 +03:00
35ad425866
bpo-40528: Implement a metadata system for ASDL Generator (GH-20193)
...
ASDL Generator was lack of proper annotation related to generated
module. This patch implements a MetadataVisitor that produces a
metadata object to pass to other visitors that are visiting that
same module. For the inital patch, it dynamically retrieves int
sequences (like cmpop), that was previously hardcoded. It offers
an interface that is easy to extend.
2021-06-22 19:29:42 +03:00
507ed6fa1d
bpo-44409: Fix error location in tokenizer errors that happen during initialization (GH-26712)
2021-06-14 17:46:11 +01:00
17b16e13bb
Fix typos in multiple files (GH-26689)
...
Co-authored-by: Terry Jan Reedy <tjreedy@udel.edu>
2021-06-12 22:47:44 -04:00
a342cc5891
bpo-44396: Update multi-line-start location when reallocating tokenizer buffers (GH-26676)
...
Automerge-Triggered-By: GH:pablogsal
2021-06-12 10:53:49 -07:00
be8b631b7a
Add more const modifiers. (GH-26691)
2021-06-12 16:11:59 +03:00
05073036dc
bpo-44368: Improve syntax errors with invalid as pattern targets (GH-26632)
2021-06-10 23:50:32 +01:00
e7b4644607
bpo-44385: Remove unused grammar rules (GH-26655)
...
Automerge-Triggered-By: GH:lysnikolaou
2021-06-10 15:05:06 -07:00
457ce60fc7
bpo-44368: Ensure we don't raise incorrect custom syntax errors with soft keywords (GH-26630)
2021-06-09 22:20:01 +01:00
aef1b58dc8
bpo-44345: Fix 'generated by' comment in parser.c (GH-26615)
2021-06-09 16:38:53 +02:00
9fd21f649d
bpo-44349: Fix edge case when displaying text from files with encoding in syntax errors (GH-26611)
2021-06-09 00:54:29 +01:00
2ea6d89028
bpo-43833: Emit warnings for numeric literals followed by keyword (GH-25466)
...
Emit a deprecation warning if the numeric literal is immediately followed by
one of keywords: and, else, for, if, in, is, or. Raise a syntax error with
more informative message if it is immediately followed by other keyword or
identifier.
Automerge-Triggered-By: GH:pablogsal
2021-06-08 16:31:10 -07:00
bafe0aade5
bpo-44335: Ensure the tokenizer doesn't go into Python with the error set (GH-26608)
2021-06-08 20:02:03 +01:00
d334c73b56
bpo-44335: Fix a regression when identifying invalid characters in syntax errors (GH-26589)
2021-06-08 12:25:22 +01:00
ffd87b7093
fix: use unambiguous punction in 'invalid escape sequence' message (GH-26582)
2021-06-08 01:15:46 +01:00
b250f89bb7
bpo-44305: Improve syntax error for try blocks without except or finally (GH-26523)
2021-06-03 23:52:12 +01:00
f3491242e4
bpo-11105: Do not crash when compiling recursive ASTs (GH-20594)
...
When compiling an AST object with a direct / indirect reference
cycles, on the conversion phase because of exceeding amount of
calls, a segfault was raised. This patch adds recursion guards to
places for preventing user inputs to not to crash AST but instead
raise a RecursionError.
2021-06-03 21:01:02 +01:00
39dd141a4b
bpo-44273: Improve syntax error message for assigning to "..." (GH-26477)
...
Use "ellipsis" instead of "Ellipsis" in syntax error messages to eliminate confusion with built-in variable Ellipsis.
2021-06-01 12:07:05 +01:00
bd7476dae3
bpo-44201: Avoid side effects of "invalid_*" rules in the REPL (GH-26298)
...
When the parser does a second pass to check for errors, these rules can
have some small side-effects as they may advance the parser more than
the point reached in the first pass. This can cause the tokenizer to ask
for extra tokens in interactive mode causing the tokenizer to show the
prompt instead of failing instantly.
To avoid this, add a new mode to the tokenizer that is activated in the
second pass and deactivates asking for new tokens when the interactive
line is finished. As the parsing should have reached the last line in
the first pass, the second pass should not need to ask for more tokens.
2021-05-22 23:05:00 +01:00
c878a97968
bpo-44180: Fix edge cases in invalid assigment rules in the parser (GH-26283)
...
The invalid assignment rules are very delicate since the parser can
easily raise an invalid assignment when a keyword argument is provided.
As they are very deep into the grammar tree, is very difficult to
specify in which contexts these rules can be used and in which don't.
For that, we need to use a different version of the rule that doesn't do
error checking in those situations where we don't want the rule to raise
(keyword arguments and generator expressions).
We also need to check if we are in left-recursive rule, as those can try
to eagerly advance the parser even if the parse will fail at the end of
the expression. Failing to do this allows the parser to start parsing a
call as a tuple and incorrectly identify a keyword argument as an
invalid assignment, before it realizes that it was not a tuple after all.
2021-05-21 18:34:54 +01:00
b51081c1a8
bpo-44180: Report generic syntax errors in the furthest position reached in the first parser pass (GH-26253)
2021-05-21 16:09:51 +01:00
33c0c90dea
bpo-44168: Fix error message in the parser for keyword arguments for invalid expressions (GH-26210)
2021-05-19 19:03:04 +01:00
80b089179f
bpo-44143: Fix crash in the parser when raising tokenizer errors with an exception set (GH-26144)
2021-05-15 17:58:02 +01:00
6692dc1ca9
bpo-43149: Correct the syntax error message for multiple exception types (GH-25996)
...
Automerge-Triggered-By: GH:pablogsal
2021-05-08 11:24:41 -07:00
9142088e74
bpo-43822: Prioritize tokenizer errors over custom syntax errors when raising parser exceptions (GH-25866)
2021-05-04 01:32:46 +01:00
dbe60ee09d
bpo-43892: Validate the first term of complex literal value patterns (GH-25735)
2021-04-29 17:19:28 -07:00
1e7b858575
bpo-43892: Make match patterns explicit in the AST (GH-25585)
...
Co-authored-by: Brandt Bucher <brandtbucher@gmail.com>
2021-04-28 22:58:44 -07:00
5e437fb872
bpo-30555: Fix WindowsConsoleIO fails in the presence of fd redirection (GH-1927)
...
This works by not caching the handle and instead getting the handle from
the file descriptor each time, so that if the actual handle changes by
fd redirection closing/opening the console handle beneath our feet, we
will keep working correctly.
2021-04-23 23:00:27 +01:00
a77aac4fca
bpo-43914: Highlight invalid ranges in SyntaxErrors ( #25525 )
...
To improve the user experience understanding what part of the error messages associated with SyntaxErrors is wrong, we can highlight the whole error range and not only place the caret at the first character. In this way:
>>> foo(x, z for z in range(10), t, w)
File "<stdin>", line 1
foo(x, z for z in range(10), t, w)
^
SyntaxError: Generator expression must be parenthesized
becomes
>>> foo(x, z for z in range(10), t, w)
File "<stdin>", line 1
foo(x, z for z in range(10), t, w)
^^^^^^^^^^^^^^^^^^^^
SyntaxError: Generator expression must be parenthesized
2021-04-23 14:27:05 +01:00
56c95dfe27
bpo-43859: Improve the error message for IndentationError exceptions (GH-25431)
2021-04-21 15:28:21 +01:00
b5b98bd8f8
bpo-43823: Fix location of one of the errors for invalid dictionary literals (GH-25427)
2021-04-16 00:45:42 +01:00
b280248be8
bpo-43822: Improve syntax errors for missing commas (GH-25377)
2021-04-15 21:38:45 +01:00
da74350174
bpo-43823: Improve syntax errors for invalid dictionary literals (GH-25378)
2021-04-15 14:06:39 +01:00
30ed93bfec
bpo-43797: Handle correctly invalid assignments inside function calls and generators (GH-25390)
2021-04-13 17:51:21 +01:00
d9151cb453
Ensure that early = are not matched by the parser as invalid comparisons (GH-25375)
2021-04-13 02:32:33 +01:00
b86ed8e3bb
bpo-43797: Improve syntax error for invalid comparisons ( #25317 )
...
* bpo-43797: Improve syntax error for invalid comparisons
* Update Lib/test/test_fstring.py
Co-authored-by: Guido van Rossum <gvanrossum@gmail.com>
* Apply review comments
* can't -> cannot
Co-authored-by: Guido van Rossum <gvanrossum@gmail.com>
2021-04-12 16:59:30 +01:00
75a06f067b
bpo-43798: Add source location attributes to alias (GH-25324)
...
* Add source location attributes to alias.
* Move alias star construction to pegen helper.
Co-authored-by: blurb-it[bot] <43283697+blurb-it[bot]@users.noreply.github.com>
Co-authored-by: Pablo Galindo <Pablogsal@gmail.com>
2021-04-10 22:56:28 +02:00
d00a449d6d
Simplify _PyPegen_fill_token in pegen.c (GH-25295)
2021-04-09 01:32:25 +01:00
58bafe42ab
Sanitize macros and debug functions in pegen.c (GH-25291)
2021-04-09 01:17:31 +01:00
4f642dae4e
Break down some complex functions in pegen.c for readability (GH-25292)
2021-04-09 00:48:53 +01:00
c0e11a3ceb
Fix possible refleak involving _PyArena_AddPyObject (GH-25289)
2021-04-09 00:05:44 +01:00
Pablo Galindo Salgado
Batuhan Taskaya
Irit Katriel
Kumar Aditya
Weipeng Hong
Victor Stinner
Carl Friedrich Bolz-Tereick
wim glenn
Nikita Sobolev
Serhiy Storchaka
Christian Clauss
Rajendra arora
Miguel Brito
Noah Kantrowitz
Ammar Askar
Binbin
Lysandros Nikolaou
Akira Nonaka
Ned Batchelder
Brandt Bucher
Nick Coghlan
Segev Finer
Matthew Suozzo
Erlend Egeberg Aasland
