Commit Graph

1009 Commits

Author SHA1 Message Date
Serhiy Storchaka c96d1546b1
bpo-45461: Fix IncrementalDecoder and StreamReader in the "unicode-escape" codec (GH-28939)
They support now splitting escape sequences between input chunks.

Add the third parameter "final" in codecs.unicode_escape_decode().
It is True by default to match the former behavior.
2021-10-14 13:17:00 +03:00
Victor Stinner 713bb19356
bpo-45434: Mark the PyTokenizer C API as private (GH-28924)
Rename PyTokenize functions to mark them as private:

* PyTokenizer_FindEncodingFilename() => _PyTokenizer_FindEncodingFilename()
* PyTokenizer_FromString() => _PyTokenizer_FromString()
* PyTokenizer_FromFile() => _PyTokenizer_FromFile()
* PyTokenizer_FromUTF8() => _PyTokenizer_FromUTF8()
* PyTokenizer_Free() => _PyTokenizer_Free()
* PyTokenizer_Get() => _PyTokenizer_Get()

Remove the unused PyTokenizer_FindEncoding() function.

import.c: remove unused #include "errcode.h".
2021-10-13 17:22:14 +02:00
Victor Stinner 97308dfcdc
bpo-45434: Move _Py_BEGIN_SUPPRESS_IPH to pycore_fileutils.h (GH-28922) 2021-10-13 15:03:35 +02:00
Victor Stinner d943d19172
bpo-45439: Move _PyObject_CallNoArgs() to pycore_call.h (GH-28895)
* Move _PyObject_CallNoArgs() to pycore_call.h (internal C API).
* _ssl, _sqlite and _testcapi extensions now call the public
  PyObject_CallNoArgs() function, rather than _PyObject_CallNoArgs().
* _lsprof extension is now built with Py_BUILD_CORE_MODULE macro
  defined to get access to internal _PyObject_CallNoArgs().
2021-10-12 08:38:19 +02:00
Victor Stinner ce3489cfdb
bpo-45439: Rename _PyObject_CallNoArg() to _PyObject_CallNoArgs() (GH-28891)
Fix typo in the private _PyObject_CallNoArg() function name: rename
it to _PyObject_CallNoArgs() to be consistent with the public
function PyObject_CallNoArgs().
2021-10-12 00:42:23 +02:00
Pablo Galindo Salgado 0219017df7
bpo-45408: Don't override previous tokenizer errors in the second parser pass (GH-28812) 2021-10-07 22:33:05 +01:00
Christian Clauss 5f401f1040
Fix typos in the Objects directory (GH-28766) 2021-10-06 16:57:10 -07:00
Rajendra arora a47d67cf46
Optimized code format (GH-28599)
Automerge-Triggered-By: GH:pablogsal
2021-09-28 04:51:39 -07:00
Rajendra arora 8b7427b554
Fixed typo in "decclarations" (GH-28578) 2021-09-28 13:56:41 +03:00
Pablo Galindo Salgado e5f13ce5b4
bpo-43914: Correctly highlight SyntaxError exceptions for invalid generator expression in function calls (GH-28576) 2021-09-27 14:37:43 +01:00
Pablo Galindo Salgado b01fd533fe
Extract visitors from the grammar nodes and call makers in the peg generator (GH-28172)
Simplify the peg generator logic by extracting as much visitors as possible to disentangle the flow and separate concerns.
2021-09-05 14:58:52 +01:00
Pablo Galindo Salgado b2f68b1900
bpo-44947: Refine the syntax error for trailing commas in import statements (GH-27814) 2021-08-18 22:09:21 +02:00
Pablo Galindo Salgado 953d27261e
Update pegen to use the latest upstream developments (GH-27586) 2021-08-12 17:37:30 +01:00
Pablo Galindo Salgado 8e832fb2a2
bpo-44885: Correct the ast locations of f-strings with format specs and repeated expressions (GH-27729) 2021-08-12 17:13:30 +01:00
Serhiy Storchaka 058fb35b57
bpo-44854: Remove trailing whitespaces (GH-27689) 2021-08-09 21:32:54 +03:00
Pablo Galindo Salgado f5cbea6b1b
bpo-44838: Refine the custom syntax errors for invalid 'if' expressions (GH-27615) 2021-08-05 18:28:57 +01:00
Miguel Brito 28b6dc9dd5
bpo-44792: Improve syntax errors for if expressions (GH-27506) 2021-08-02 18:11:37 +01:00
Pablo Galindo Salgado 208a7e957b
bpo-34013: Don't consider a grouped expression when reporting legacy print syntax errors (GH-27521) 2021-08-01 02:10:50 +01:00
Pablo Galindo Salgado 302cf35505
Divide the grammar into sections to improve readability (GH-27502) 2021-07-31 13:04:37 +01:00
Noah Kantrowitz be42c06bb0
Update URLs in comments and metadata to use HTTPS (GH-27458) 2021-07-30 15:54:46 +02:00
Pablo Galindo Salgado ecc3c8e421
bpo-34013: Move the Python 2 hints from the exception constructor to the parser (GH-27392) 2021-07-27 21:30:32 +01:00
Pablo Galindo Salgado 6948964ecf
bpo-34013: Generalize the invalid legacy statement error message (GH-27389) 2021-07-27 17:19:22 +01:00
Batuhan Taskaya fbc349ff79
bpo-43950: Distinguish errors happening on character offset decoding (GH-27217) 2021-07-20 16:42:12 +01:00
Pablo Galindo Salgado a283ef116b
Remove unnecessary 'invalid_primary' rule in the parser (GH-27186) 2021-07-16 13:20:09 +01:00
Pablo Galindo Salgado f24777c2b3
bpo-44317: Improve tokenizer errors with more informative locations (GH-26555) 2021-07-10 01:29:29 +01:00
Ammar Askar 5644c7b3ff
bpo-43950: Print columns in tracebacks (PEP 657) (GH-26958)
The traceback.c and traceback.py mechanisms now utilize the newly added code.co_positions and PyCode_Addr2Location
to print carets on the specific expressions involved in a traceback.

Co-authored-by: Pablo Galindo <Pablogsal@gmail.com>
Co-authored-by: Ammar Askar <ammar@ammaraskar.com>
Co-authored-by: Batuhan Taskaya <batuhanosmantaskaya@gmail.com>
2021-07-05 00:14:33 +01:00
Batuhan Taskaya 107a2c59c9
bpo-40528: fix is_simple(sum)s behavior for attributes (GH-26918)
This is something I noticed while (now discontinued) experimenting
with the idea of annotating operators with location information. Unfortunately
without this addition, adding any `attributes` to stuff like `unaryop`
doesn't change anything since the code assumes they are singletons and
caches all instances. This patch fixes this assumption with including
the attributes as well as constructor fields.
2021-06-27 17:58:32 +03:00
Pablo Galindo 0acc258fe6
bpo-44456: Improve the syntax error when mixing keyword and positional patterns (GH-26793) 2021-06-24 16:09:57 +01:00
Batuhan Taskaya 6c76df2b86
bpo-40528: move asdl identifier collection to the new metadata system (GH-26858) 2021-06-24 15:16:00 +03:00
Batuhan Taskaya 35ad425866
bpo-40528: Implement a metadata system for ASDL Generator (GH-20193)
ASDL Generator was lack of proper annotation related to generated
module. This patch implements a MetadataVisitor that produces a
metadata object to pass to other visitors that are visiting that
same module. For the inital patch, it dynamically retrieves int
sequences (like cmpop), that was previously hardcoded. It offers
an interface that is easy to extend.
2021-06-22 19:29:42 +03:00
Pablo Galindo 507ed6fa1d
bpo-44409: Fix error location in tokenizer errors that happen during initialization (GH-26712) 2021-06-14 17:46:11 +01:00
Binbin 17b16e13bb
Fix typos in multiple files (GH-26689)
Co-authored-by: Terry Jan Reedy <tjreedy@udel.edu>
2021-06-12 22:47:44 -04:00
Pablo Galindo a342cc5891
bpo-44396: Update multi-line-start location when reallocating tokenizer buffers (GH-26676)
Automerge-Triggered-By: GH:pablogsal
2021-06-12 10:53:49 -07:00
Serhiy Storchaka be8b631b7a
Add more const modifiers. (GH-26691) 2021-06-12 16:11:59 +03:00
Pablo Galindo 05073036dc
bpo-44368: Improve syntax errors with invalid as pattern targets (GH-26632) 2021-06-10 23:50:32 +01:00
Lysandros Nikolaou e7b4644607
bpo-44385: Remove unused grammar rules (GH-26655)
Automerge-Triggered-By: GH:lysnikolaou
2021-06-10 15:05:06 -07:00
Pablo Galindo 457ce60fc7
bpo-44368: Ensure we don't raise incorrect custom syntax errors with soft keywords (GH-26630) 2021-06-09 22:20:01 +01:00
Akira Nonaka aef1b58dc8
bpo-44345: Fix 'generated by' comment in parser.c (GH-26615) 2021-06-09 16:38:53 +02:00
Pablo Galindo 9fd21f649d
bpo-44349: Fix edge case when displaying text from files with encoding in syntax errors (GH-26611) 2021-06-09 00:54:29 +01:00
Serhiy Storchaka 2ea6d89028
bpo-43833: Emit warnings for numeric literals followed by keyword (GH-25466)
Emit a deprecation warning if the numeric literal is immediately followed by
one of keywords: and, else, for, if, in, is, or. Raise a syntax error with
more informative message if it is immediately followed by other keyword or
identifier.

Automerge-Triggered-By: GH:pablogsal
2021-06-08 16:31:10 -07:00
Pablo Galindo bafe0aade5
bpo-44335: Ensure the tokenizer doesn't go into Python with the error set (GH-26608) 2021-06-08 20:02:03 +01:00
Pablo Galindo d334c73b56
bpo-44335: Fix a regression when identifying invalid characters in syntax errors (GH-26589) 2021-06-08 12:25:22 +01:00
Ned Batchelder ffd87b7093
fix: use unambiguous punction in 'invalid escape sequence' message (GH-26582) 2021-06-08 01:15:46 +01:00
Pablo Galindo b250f89bb7
bpo-44305: Improve syntax error for try blocks without except or finally (GH-26523) 2021-06-03 23:52:12 +01:00
Batuhan Taskaya f3491242e4
bpo-11105: Do not crash when compiling recursive ASTs (GH-20594)
When compiling an AST object with a direct / indirect reference
cycles, on the conversion phase because of exceeding amount of
calls, a segfault was raised. This patch adds recursion guards to
places for preventing user inputs to not to crash AST but instead
raise a RecursionError.
2021-06-03 21:01:02 +01:00
Serhiy Storchaka 39dd141a4b
bpo-44273: Improve syntax error message for assigning to "..." (GH-26477)
Use "ellipsis" instead of "Ellipsis" in syntax error messages to eliminate confusion with built-in variable Ellipsis.
2021-06-01 12:07:05 +01:00
Pablo Galindo bd7476dae3
bpo-44201: Avoid side effects of "invalid_*" rules in the REPL (GH-26298)
When the parser does a second pass to check for errors, these rules can
have some small side-effects as they may advance the parser more than
the point reached in the first pass. This can cause the tokenizer to ask
for extra tokens in interactive mode causing the tokenizer to show the
prompt instead of failing instantly.

To avoid this, add a new mode to the tokenizer that is activated in the
second pass and deactivates asking for new tokens when the interactive
line is finished. As the parsing should have reached the last line in
the first pass, the second pass should not need to ask for more tokens.
2021-05-22 23:05:00 +01:00
Pablo Galindo c878a97968
bpo-44180: Fix edge cases in invalid assigment rules in the parser (GH-26283)
The invalid assignment rules are very delicate since the parser can
easily raise an invalid assignment when a keyword argument is provided.
As they are very deep into the grammar tree, is very difficult to
specify in which contexts these rules can be used and in which don't.
For that, we need to use a different version of the rule that doesn't do
error checking in those situations where we don't want the rule to raise
(keyword arguments and generator expressions).

We also need to check if we are in left-recursive rule, as those can try
to eagerly advance the parser even if the parse will fail at the end of
the expression. Failing to do this allows the parser to start parsing a
call as a tuple and incorrectly identify a keyword argument as an
invalid assignment, before it realizes that it was not a tuple after all.
2021-05-21 18:34:54 +01:00
Pablo Galindo b51081c1a8
bpo-44180: Report generic syntax errors in the furthest position reached in the first parser pass (GH-26253) 2021-05-21 16:09:51 +01:00
Pablo Galindo 33c0c90dea
bpo-44168: Fix error message in the parser for keyword arguments for invalid expressions (GH-26210) 2021-05-19 19:03:04 +01:00