This commit removes the old parser, the deprecated parser module, the old parser compatibility flags and environment variables and all associated support code and documentation.
This fixes both the traceback.py module and the C code for formatting syntax errors (in Python/pythonrun.c). They now both consistently do the following:
- Suppress caret if it points left of text
- Allow caret pointing just past end of line
- If caret points past end of line, clip to *just* past end of line
The syntax error formatting code in traceback.py was mostly rewritten; small, subtle changes were applied to the C code in pythonrun.c.
There's still a difference when the text contains embedded newlines. Neither handles these very well, and I don't think the case occurs in practice.
Automerge-Triggered-By: @gvanrossum
Due to backwards compatibility concerns regarding keywords immediately followed by a string without whitespace between them (like in `bg="#d00" if clear else"#fca"`) will fail to parse,
commit 41d5b94af4 has to be reverted.
This commit also allows to pass flags to the new parser in all interfaces and fixes a bug in the parser generator that was causing to inline rules with actions, making them disappear.
* Rename PyConfig.use_peg to _use_peg_parser
* Document PyConfig._use_peg_parser and mark it a deprecated
* Mark -X oldparser option and PYTHONOLDPARSER env var as deprecated
in the documentation.
* Add use_old_parser() and skip_if_new_parser() to test.support
* Remove sys.flags.use_peg: use_old_parser() uses
_testinternalcapi.get_configs() instead.
* Enhance test_embed tests
* subprocess._args_from_interpreter_flags() copies -X oldparser
Rename _PyInterpreterState_GET_UNSAFE() to _PyInterpreterState_GET()
for consistency with _PyThreadState_GET() and to have a shorter name
(help to fit into 80 columns).
Add also "assert(tstate != NULL);" to the function.
Don't access PyInterpreterState.config member directly anymore, but
use new functions:
* _PyInterpreterState_GetConfig()
* _PyInterpreterState_SetConfig()
* _Py_GetConfig()
Replace _PyInterpreterState_Get() function call with
_PyInterpreterState_GET_UNSAFE() macro which is more efficient but
don't check if tstate or interp is NULL.
_Py_GetConfigsAsDict() now uses _PyThreadState_GET().
Move the following functions from the public C API to the internal C
API:
* _PyDebug_PrintTotalRefs(),
* _Py_PrintReferenceAddresses()
* _Py_PrintReferences()
Fix sys.excepthook() and PyErr_Display() if a filename is a bytes
string. For example, for a SyntaxError exception where the filename
attribute is a bytes string.
Cleanup also test_sys:
* Sort imports.
* Rename numruns global var to INTERN_NUMRUNS.
* Add DisplayHookTest and ExceptHookTest test case classes.
* Don't save/restore sys.stdout and sys.displayhook using
setUp()/tearDown(): do it in each test method.
* Test error case (call hook with no argument) after the success case.
Add a new _PyCompilerFlags_INIT macro to initialize PyCompilerFlags
variables, rather than initializing cf_flags and cf_feature_version
explicitly in each variable.
sys.excepthook() and sys.unraisablehook() now explicitly flush the
file (usually sys.stderr).
If file.flush() fails, sys.excepthook() silently ignores the error,
whereas sys.unraisablehook() logs the new exception.
Add a new threading.excepthook() function which handles uncaught
Thread.run() exception. It can be overridden to control how uncaught
exceptions are handled.
threading.ExceptHookArgs is not documented on purpose: it should not
be used directly.
* threading.excepthook() and threading.ExceptHookArgs.
* Add _PyErr_Display(): similar to PyErr_Display(), but accept a
'file' parameter.
* Add _thread._excepthook(): C implementation of the exception hook
calling _PyErr_Display().
* Add _thread._ExceptHookArgs: structseq type.
* Add threading._invoke_excepthook_wrapper() which handles the gory
details to ensure that everything remains alive during Python
shutdown.
* Add unit tests.
Py_Main() and _Py_RunMain() now return the exitcode rather than
calling Py_Exit(exitcode) when calling PyErr_Print() if the current
exception type is SystemExit.
* Add _Py_HandleSystemExit().
* Add pymain_exit_err_print().
* Add pymain_exit_print().
* Add a private _Py_InitializeMain() function.
* Add again _PyCoreConfig._init_main.
* _Py_InitializeFromConfig() now uses _init_main to decide
if _Py_InitializeMainInterpreter() should be called.
* _PyCoreConfig: rename _frozen to pathconfig_warnings, its value is
now the opposite of Py_FrozenFlag.
* Add an unit test for _init_main=0 and _Py_InitializeMain().
This adds a `feature_version` flag to `ast.parse()` (documented) and `compile()` (hidden) that allow tweaking the parser to support older versions of the grammar. In particular if `feature_version` is 5 or 6, the hacks for the `async` and `await` keyword from PEP 492 are reinstated. (For 7 or higher, these are unconditionally treated as keywords, but they are still special tokens rather than `NAME` tokens that the parser driver recognizes.)
https://bugs.python.org/issue35975
Explicitly reinitialize this every eval *just in case* someone is
calling into an embedded Python where they don't care about an uncaught
KeyboardInterrupt exception (why didn't they leave
`config.install_signal_handlers` set to `0`?!?) but then later call
`Py_Main()` itself (which *checks* this flag and dies with a signal after
its interpreter exits). We don't want a previous embedded interpreter's
uncaught exception to trigger an unexplained signal exit from a future
`Py_Main()` based one.
* bpo-1054041: Exit properly by a signal after a ^C.
An uncaught KeyboardInterrupt exception means the user pressed ^C and
our code did not handle it. Programs that install SIGINT handlers are
supposed to reraise the SIGINT signal to the SIG_DFL handler in order
to exit in a manner that their calling process can detect that they
died due to a Ctrl-C. https://www.cons.org/cracauer/sigint.html
After this change on POSIX systems
while true; do python -c 'import time; time.sleep(23)'; done
can be stopped via a simple Ctrl-C instead of the shell infinitely
restarting a new python process.
What to do on Windows, or if anything needs to be done there has not
yet been determined. That belongs in its own PR.
TODO(gpshead): A unittest for this behavior is still needed.
* Do the unhandled ^C check after pymain_free.
* Return STATUS_CONTROL_C_EXIT on Windows.
* Fix ifdef around unistd.h include.
* 📜🤖 Added by blurb_it.
* Add STATUS_CTRL_C_EXIT to the os module on Windows
* Add unittests.
* Don't send CTRL_C_EVENT in the Windows test.
It was causing CI systems to bail out of the entire test suite.
See https://dev.azure.com/Python/cpython/_build/results?buildId=37980
for example.
* Correct posix test (fail on macOS?) check.
* STATUS_CONTROL_C_EXIT must be unsigned.
* Improve the error message.
* test typo :)
* Skip if the bash version is too old.
...and rename the windows test to reflect what it does.
* min bash version is 4.4, detect no bash.
* restore a blank line i didn't mean to delete.
* PyErr_Occurred() before the Py_DECREF(co);
* Don't add os.STATUS_CONTROL_C_EXIT as a constant.
* Update the Windows test comment.
* Refactor common logic into a run_eval_code_obj fn.
* ast.h now includes Python-ast.h and node.h
* parsetok.h now includes node.h and grammar.h
* symtable.h now includes Python-ast.h
* Modify asdl_c.py to enhance Python-ast.h:
* Add #ifndef/#define Py_PYTHON_AST_H to be able to include the header
twice
* Add "extern { ... }" for C++
* Undefine "Yield" macro conflicting with winbase.h
* Remove "#undef Yield" from C files, it's now done in Python-ast.h
* Remove now useless includes in C files
Serhiy Storchaka
Lysandros Nikolaou
Pablo Galindo
Ammar Askar
Guido van Rossum
Victor Stinner
Steve Dower
Benjamin Peterson
Vinay Sajip
Jeroen Demeyer
Victor Stinner
Dino Viehland
Steve Dower
Gregory P. Smith
INADA Naoki
Eric V. Smith