- drop TargetScopeError in favour of raising SyntaxError directly
as per the updated PEP 572
- comprehension iteration variables are explicitly local, but
named expression targets in comprehensions are nonlocal or
global. Raise SyntaxError as specified in PEP 572
- named expression targets in the outermost iterable of a
comprehension have an ambiguous target scope. Avoid resolving
that question now by raising SyntaxError. PEP 572
originally required this only for cases where the bound name
conflicts with the iteration variable in the comprehension,
but CPython can't easily restrict the exception to that case
(as it doesn't know the target variable names when visiting
the outermost iterator expression)
(cherry picked from commit 5dbe0f59b7)
Fix compilation of "break" and "continue" in the
"finally" block when the corresponding "try" block
contains "return" with a non-constant value.
(cherry picked from commit ef61c524dd)
PyConfig_Read() is now responsible to handle early calls to
PySys_AddXOption() and PySys_AddWarnOption().
Options added by PySys_AddXOption() are now handled the same way than
PyConfig.xoptions and command line -X options.
For example, PySys_AddXOption(L"faulthandler") enables faulthandler
as expected.
(cherry picked from commit 120b707a6d)
empty_argv is no longer static in Python 3.8, but it is declared in
a temporary scope, whereas argv keeps a reference to it.
empty_argv memory (allocated on the stack) is reused by
make_sys_argv() code which is inlined when using gcc -O3.
Define empty_argv in PySys_SetArgvEx() body, to ensure
that it remains valid for the whole lifetime of
the PySys_SetArgvEx() call.
(cherry picked from commit c48682509d)
bpo-37834: Normalise handling of reparse points on Windows
* ntpath.realpath() and nt.stat() will traverse all supported reparse points (previously was mixed)
* nt.lstat() will let the OS traverse reparse points that are not name surrogates (previously would not traverse any reparse point)
* nt.[l]stat() will only set S_IFLNK for symlinks (previous behaviour)
* nt.readlink() will read destinations for symlinks and junction points only
bpo-1311: os.path.exists('nul') now returns True on Windows
* nt.stat('nul').st_mode is now S_IFCHR (previously was an error)
* bpo-32912: Revert warnings for invalid escape sequences.
DeprecationWarning will continue to be emitted for invalid escape sequences in string and bytes literals in 3.8 just as it did in 3.7.
SyntaxWarning may be emitted in the future. But per mailing list discussion, we don't yet know when because we haven't settled on how to do so in a non-disruptive manner.
This changeset increases the default size of the stack
for threads on macOS to the size of the stack
of the main thread and reenables the relevant
recursion test.
(cherry picked from commit 1a057bab0f)
Co-authored-by: Ronald Oussoren <ronaldoussoren@mac.com>
https://bugs.python.org/issue37500
Add a new field to the compiler structure that allows to be configured
so no bytecode is emitted. In this way is possible to detect errors by
walking the nodes while preserving optimizations.
https://bugs.python.org/issue37500
(cherry picked from commit 18c5f9d44d)
Co-authored-by: Pablo Galindo <Pablogsal@gmail.com>
* bpo-37697: Sync with importlib_metadata 0.19
* Run make regen-importlib
* 📜🤖 Added by blurb_it.
(cherry picked from commit 049460da9c)
Co-authored-by: Jason R. Coombs <jaraco@jaraco.com>
Nested BinOp instances (e.g. a+b+c) had a wrong col_offset for the
second BinOp (e.g. 2 instead of 0 in the example). Fix it by using the
correct st node to copy the line and col_offset from in ast.c.
(cherry picked from commit 110a47c4f4)
Co-authored-by: Carl Friedrich Bolz-Tereick <cfbolz@gmx.de>
Fix sys.excepthook() and PyErr_Display() if a filename is a bytes
string. For example, for a SyntaxError exception where the filename
attribute is a bytes string.
Cleanup also test_sys:
* Sort imports.
* Rename numruns global var to INTERN_NUMRUNS.
* Add DisplayHookTest and ExceptHookTest test case classes.
* Don't save/restore sys.stdout and sys.displayhook using
setUp()/tearDown(): do it in each test method.
* Test error case (call hook with no argument) after the success case.
(cherry picked from commit f9b7457bd7)
Co-authored-by: Victor Stinner <vstinner@redhat.com>
Add PyCode_NewEx to be used internally and set PyCode_New as a compatibility wrapper
(cherry picked from commit 4a2edc34a4)
Co-authored-by: Pablo Galindo <Pablogsal@gmail.com>
sys._base_executable is now always defined on all platforms, and can be overridden through configuration.
Also adds test.support.PythonSymlink to encapsulate platform-specific logic for symlinking sys.executable
… as proposed in PEP 572; key is now evaluated before value.
https://bugs.python.org/issue35224
(cherry picked from commit c8a35417db)
Co-authored-by: Jörn Heissler <joernheissler@users.noreply.github.com>
On Windows, os.dup() no longer creates an inheritable fd when handling a
character file.
(cherry picked from commit 28fca0c422)
Co-authored-by: Zackery Spytz <zspytz@gmail.com>
Fix a regression introduced by af8646c805 that was causing code of the form:
if True and False:
do_something()
to be optimized incorrectly, eliminating the block.
(cherry picked from commit 05f8318655)
Co-authored-by: Pablo Galindo <Pablogsal@gmail.com>
The peephole optimizer was not optimizing correctly bytecode after negative deltas were introduced. This is due to the fact that some special values (255) were being searched for in both instruction pointer delta and line number deltas.
(cherry picked from commit 3498c642f4)
Co-authored-by: Pablo Galindo <Pablogsal@gmail.com>
PyAST_obj2mod_ex() is similar to PyAST_obj2mod() with an additional
'feature_version' parameter which is unused.
(cherry picked from commit 022ac0a497)
Co-authored-by: Victor Stinner <vstinner@redhat.com>
Add a new _PyCompilerFlags_INIT macro to initialize PyCompilerFlags
variables, rather than initializing cf_flags and cf_feature_version
explicitly in each variable.
(cherry picked from commit 37d66d7d4b)
Co-authored-by: Victor Stinner <vstinner@redhat.com>
Rename compile() feature_version parameter to _feature_version and
convert it to a keyword-only parameter.
Update also test_type_comments to pass feature_version as a tuple.
(cherry picked from commit efdf6ca90f)
* bpo-22385: Support output separators in hex methods.
Also in binascii.hexlify aka b2a_hex.
The underlying implementation behind all hex generation in CPython uses the
same pystrhex.c implementation. This adds support to bytes, bytearray,
and memoryview objects.
The binascii module functions exist rather than being slated for deprecation
because they return bytes rather than requiring an intermediate step through a
str object.
This change was inspired by MicroPython which supports sep in its binascii
implementation (and does not yet support the .hex methods).
https://bugs.python.org/issue22385
sys.excepthook() and sys.unraisablehook() now explicitly flush the
file (usually sys.stderr).
If file.flush() fails, sys.excepthook() silently ignores the error,
whereas sys.unraisablehook() logs the new exception.
It has been documented as deprecated and to be removed in 3.8;
From a comment on another thread – which I can't find ; leave get_coro_wrapper() for now, but always return `None`.
https://bugs.python.org/issue36933
Add a new threading.excepthook() function which handles uncaught
Thread.run() exception. It can be overridden to control how uncaught
exceptions are handled.
threading.ExceptHookArgs is not documented on purpose: it should not
be used directly.
* threading.excepthook() and threading.ExceptHookArgs.
* Add _PyErr_Display(): similar to PyErr_Display(), but accept a
'file' parameter.
* Add _thread._excepthook(): C implementation of the exception hook
calling _PyErr_Display().
* Add _thread._ExceptHookArgs: structseq type.
* Add threading._invoke_excepthook_wrapper() which handles the gory
details to ensure that everything remains alive during Python
shutdown.
* Add unit tests.
When using the "=" debug functionality of f-strings, use another Constant node (or a merged constant node) instead of adding expr_text to the FormattedValue node.
* sys.unraisablehook: add 'err_msg' field to UnraisableHookArgs.
* Use _PyErr_WriteUnraisableMsg() in _ctypes _DictRemover_call()
and gc delete_garbage().
* _PyPreConfig_InitCompatConfig() sets utf8_mode to 0.
* Change Py_UTF8Mode default value to 0.
* Fix _PyPreConfig_Copy(): copy also _config_init attrbibute.
* _PyPreConfig_AsDict() exports _config_init
* Fix _PyPreConfig_GetGlobalConfig(): use Py_UTF8Mode if it's greater
than 0, even if utf8_mode >= 0.
* Add unit tests on environment variables using Python API.
PyErr_WriteUnraisable() now creates a traceback object if there is no
current traceback. Moreover, call PyErr_NormalizeException() and
PyException_SetTraceback() to normalize the exception value. Ignore
silently any error.
* _PyPreConfig_GetGlobalConfig() and _PyCoreConfig_GetGlobalConfig()
now do nothing if the configuration was not initialized with
_PyPreConfig_InitCompatConfig() and _PyCoreConfig_InitCompatConfig()
* Remove utf8_mode=-2 special case: use utf8_mode=-1 instead.
* Fix _PyPreConfig_InitPythonConfig():
* isolated = 0 instead of -1
* use_environment = 1 instead of -1
* Rename _PyConfig_INIT to _PyConfig_INIT_COMPAT
* Rename _PyPreConfig_Init() to _PyPreConfig_InitCompatConfig()
* Rename _PyCoreConfig_Init() to _PyCoreConfig_InitCompatConfig()
* PyInterpreterState_New() now uses _PyCoreConfig_InitPythonConfig()
as default configuration, but it's very quickly overriden anyway.
* _freeze_importlib.c uses _PyCoreConfig_SetString() to set
program_name.
* Cleanup preconfig_init_utf8_mode(): cmdline is always non-NULL.
GH-13238 made extra text after a # type: ignore accepted by the parser.
This finishes the job and actually plumbs the extra text through the
parser and makes it available in the AST.
Add new sys.unraisablehook() function which can be overridden to
control how "unraisable exceptions" are handled. It is called when an
exception has occurred but there is no way for Python to handle it.
For example, when a destructor raises an exception or during garbage
collection (gc.collect()).
Changes:
* Add an internal UnraisableHookArgs type used to pass arguments to
sys.unraisablehook.
* Add _PyErr_WriteUnraisableDefaultHook().
* The default hook now ignores exception on writing the traceback.
* test_sys now uses unittest.main() to automatically discover tests:
remove test_main().
* Add _PyErr_Init().
* Fix PyErr_WriteUnraisable(): hold a strong reference to sys.stderr
while using it
_PyRuntime.preconfig.coerce_c_locale can now be used to
check if the C locale has been coerced.
* Fix _Py_LegacyLocaleDetected(): don't attempt to coerce the
C locale if LC_ALL environment variable is set. Add 'warn'
parameter: emit_stderr_warning_for_legacy_locale() must not
the LC_ALL env var.
* _PyPreConfig_Write() sets coerce_c_locale to 0 if
_Py_CoerceLegacyLocale() fails.
* Add _PyPreConfig.parse_argv
* Add _PyCoreConfig._config_init field and _PyCoreConfigInitEnum enum
type
* Initialization functions: reject preconfig=NULL and config=NULL
* Add config parameter to _PyCoreConfig_DecodeLocaleErr(): pass
config->argv to _Py_PreInitializeFromPyArgv(), to parse config
command line arguments in preinitialization.
* Add config parameter to _PyCoreConfig_SetString(). It now
preinitializes Python.
* _PyCoreConfig_SetPyArgv() now also preinitializes Python for wide
argv
* Fix _Py_PreInitializeFromCoreConfig(): don't pass args to
_Py_PreInitializeFromPyArgv() if config.parse_argv=0.
* Use "char * const *" and "wchar_t * const *" types for 'argv'
parameters and _PyArgv.argv.
* Add unit test on preinitialization from argv.
* _PyPreConfig.allocator type becomes int
* Add _PyPreConfig_InitFromPreConfig() and
_PyPreConfig_InitFromCoreConfig() helper functions
_PyPreConfig_InitPythonConfig() and _PyCoreConfig_InitPythonConfig()
no longer inherit their values from global configuration variables.
Changes:
* _PyPreCmdline_Read() now ignores -X dev and PYTHONDEVMODE
if dev_mode is already set.
* Inline _PyPreConfig_INIT macro into _PyPreConfig_Init() function.
* Inline _PyCoreConfig_INIT macro into _PyCoreConfig_Init() function.
* Replace _PyCoreConfig_Init() with _PyCoreConfig_InitPythonConfig()
in most tests of _testembed.c.
* Replace _PyCoreConfig_Init() with _PyCoreConfig_InitIsolatedConfig()
in _freeze_importlib.c.
* Move some initialization functions from the internal
to the private API.
* The type of initlization function 'argc' parameters becomes
Py_ssize_t, instead of int.
* Change _PyPreConfig_Copy() return type to void, instead of int.
The function cannot fail anymore.
* Fix compilation warnings on Windows.
* Add _PyInitError functions:
* _PyInitError_Ok()
* _PyInitError_Error()
* _PyInitError_NoMemory()
* _PyInitError_Exit()
* _PyInitError_IsError()
* _PyInitError_IsExit()
* _PyInitError_Failed()
* frozenmain.c and _testembed.c now use functions rather than macros.
* Move _Py_INIT_xxx() macros to the internal API.
* Move _PyWstrList_INIT macro to the internal API.
Py_Main() and _Py_RunMain() now return the exitcode rather than
calling Py_Exit(exitcode) when calling PyErr_Print() if the current
exception type is SystemExit.
* Add _Py_HandleSystemExit().
* Add pymain_exit_err_print().
* Add pymain_exit_print().
_PyPreConfig_InitIsolatedConfig() sets configure_locale to 0 to
prevent Python to modify the LC_CTYPE locale. In that case,
coerce_c_locale an coerce_c_locale_warn are set to 0 as well.
Add new functions to get the Python interpreter behavior:
* _PyPreConfig_InitPythonConfig()
* _PyCoreConfig_InitPythonConfig()
Add new functions to get an isolated configuration:
* _PyPreConfig_InitIsolatedConfig()
* _PyCoreConfig_InitIsolatedConfig()
Replace _PyPreConfig_INIT and _PyCoreConfig_INIT with new functions
_PyPreConfig_Init() and _PyCoreConfig_Init().
_PyCoreConfig: set configure_c_stdio and parse_argv to 0 by default
to behave as Python 3.6 in the default configuration.
_PyCoreConfig_Read() no longer sets coerce_c_locale_warn to 1 if it's
equal to 0. coerce_c_locale_warn must now be set to -1 (ex: using
_PyCoreConfig_InitPythonConfig()) to enable C locale coercion
warning.
Add unit tests for _PyCoreConfig_InitPythonConfig()
and _PyCoreConfig_InitIsolatedConfig().
Changes:
* Rename _PyCoreConfig_GetCoreConfig() to _PyPreConfig_GetCoreConfig()
* Fix core_read_precmdline(): handle parse_argv=0
* Fix _Py_PreInitializeFromCoreConfig(): pass coreconfig.argv
to _Py_PreInitializeFromPyArgv(), except if parse_argv=0
* Add PyMemAllocatorName enum
* _PyPreConfig.allocator type becomes PyMemAllocatorName, instead of
char*
* Remove _PyPreConfig_Clear()
* Add _PyMem_GetAllocatorName()
* Rename _PyMem_GetAllocatorsName() to
_PyMem_GetCurrentAllocatorName()
* Remove _PyPreConfig_SetAllocator(): just call
_PyMem_SetupAllocators() directly, we don't have do reallocate the
configuration with the new allocator anymore!
* _PyPreConfig_Write() parameter becomes const, as it should be in
the first place!
Move the check for dead conditionals (if 0) to the peephole optimizer
and make sure that the code block is still compiled to report any
existing syntax errors within.
* Add a private _Py_InitializeMain() function.
* Add again _PyCoreConfig._init_main.
* _Py_InitializeFromConfig() now uses _init_main to decide
if _Py_InitializeMainInterpreter() should be called.
* _PyCoreConfig: rename _frozen to pathconfig_warnings, its value is
now the opposite of Py_FrozenFlag.
* Add an unit test for _init_main=0 and _Py_InitializeMain().
* _PyCoreConfig_Read() doesn't parse nor update argv
if parse_argv is 0.
* Move path configuration fields in _PyCoreConfig.
* Add an unit test for parse_argv=0.
* Remove unused "done": label in _Py_RunMain().
We cannot use "unsigned int" for exitcode on Windows, since
Py_Main() and _Py_RunMain() always return an "int".
Changes:
* _PyPathConfig_ComputeSysPath0() now returns -1 if an exception is
raised.
* pymain_run_python() no longer uses _PyInitError but display the
exception and set exitcode to 1 in case of error.
* Fix _Py_RunMain(): return an exitcode rather than calling
exit() on pymain_run_python() failure.
* _Py_ExitInitError() no longer uses ExitProcess() on Windows, use
exit() on all platforms.
* _Py_ExitInitError() now fails with a fatal error if 'err' is not an
error not an exit.
Remove the PyEval_ReInitThreads() function from the Python C API.
It should not be called explicitly: use PyOS_AfterFork_Child()
instead.
Rename PyEval_ReInitThreads() to _PyEval_ReInitThreads() and add a
'runtime' parameter.
Add "struct _ceval_runtime_state *ceval = &_PyRuntime.ceval;" local
variables to function to better highlight the dependency on the
global variable _PyRuntime and to point directly to _PyRuntime.ceval
field rather than on the larger _PyRuntime.
Changes:
* Add _PyRuntimeState_GetThreadState(runtime) macro.
* Add _PyEval_AddPendingCall(ceval, ...) and
_PyThreadState_Swap(gilstate, ...) functions.
* _PyThreadState_GET() macro now calls
_PyRuntimeState_GetThreadState() using &_PyRuntime.
* Add 'ceval' parameter to COMPUTE_EVAL_BREAKER(),
SIGNAL_PENDING_SIGNALS(), _PyEval_SignalAsyncExc(),
_PyEval_SignalReceived() and _PyEval_FiniThreads() macros and
functions.
* Add 'tstate' parameter to call_function(), do_call_core() and
do_raise().
* Add 'runtime' parameter to _Py_CURRENTLY_FINALIZING(),
_Py_FinishPendingCalls() and _PyThreadState_DeleteExcept()
macros and functions.
* Declare 'runtime', 'tstate', 'ceval' and 'eval_breaker' variables
as constant.
Add new trashcan macros to deal with a double deallocation that could occur when the `tp_dealloc` of a subclass calls the `tp_dealloc` of a base class and that base class uses the trashcan mechanism.
Patch by Jeroen Demeyer.
If a "=" is specified a the end of an f-string expression, the f-string will evaluate to the text of the expression, followed by '=', followed by the repr of the value of the expression.
_PyCoreConfig_SetPyArgv() and _PyCoreConfig_SetWideString() now
pre-initialize Python if needed to ensure that the locale encoding is
properly configured.
* Add _Py_PreInitializeFromPyArgv() internal function.
* Add 'args' parameter to _Py_PreInitializeFromCoreConfig()
_PyCoreConfig: Change filesystem_encoding, filesystem_errors,
stdio_encoding and stdio_errors fields type from char* to wchar_t*.
Changes:
* PyInterpreterState: replace fscodec_initialized (int) with fs_codec
structure.
* Add get_error_handler_wide() and unicode_encode_utf8() helper
functions.
* Add error_handler parameter to unicode_encode_locale()
and unicode_decode_locale().
* Remove _PyCoreConfig_SetString().
* Rename _PyCoreConfig_SetWideString() to _PyCoreConfig_SetString().
* Rename _PyCoreConfig_SetWideStringFromString()
to _PyCoreConfig_DecodeLocale().
Add _Py_FORCE_UTF8_LOCALE and _Py_FORCE_UTF8_FS_ENCODING macros to
avoid factorize "#if defined(__ANDROID__) || defined(__VXWORKS__)"
and "#if defined(__APPLE__)".
Cleanup also config_init_fs_encoding().
* Remove _PyInitError.user_err field and _Py_INIT_USER_ERR() macro:
use _Py_INIT_ERR() instead. _Py_ExitInitError() now longer calls
abort() on error: exit with exit code 1 instead.
* Add _PyInitError._type private field.
* exitcode field type is now unsigned int on Windows.
* Rename prefix field to _func.
* Rename msg field to err_msg.
This commit contains the implementation of PEP570: Python positional-only parameters.
* Update Grammar/Grammar with new typedarglist and varargslist
* Regenerate grammar files
* Update and regenerate AST related files
* Update code object
* Update marshal.c
* Update compiler and symtable
* Regenerate importlib files
* Update callable objects
* Implement positional-only args logic in ceval.c
* Regenerate frozen data
* Update standard library to account for positional-only args
* Add test file for positional-only args
* Update other test files to account for positional-only args
* Add News entry
* Update inspect module and related tests
* Add _PyEval_FiniThreads2(). _PyEval_FiniThreads() now only clears
the pending lock, whereas _PyEval_FiniThreads2() destroys the GIL.
* pymain_free() now calls _PyEval_FiniThreads2().
* Py_FinalizeEx() now calls _PyEval_FiniThreads().
PyEval_AcquireLock() and PyEval_AcquireThread() now
terminate the current thread if called while the interpreter is
finalizing, making them consistent with PyEval_RestoreThread(),
Py_END_ALLOW_THREADS, and PyGILState_Ensure().
Py_FinalizeEx() now clears _PyRuntime.warnings variables and
_PyRuntime.exitfuncs.
Changes:
* Add _PyWarnings_Fini(): called by Py_FinalizeEx()
* call_ll_exitfuncs() now clears _PyRuntime.exitfuncs while iterating
on it (on backward order).
Py_InitializeEx() now uses a runtime variable passed to subfunctions,
rather than working directly on the global variable _PyRuntime.
Add 'runtime' parameter to _PyCoreConfig_Write(), _PySys_Create(),
_PySys_InitMain(), _PyGILState_Init(),
emit_stderr_warning_for_legacy_locale() and other subfunctions.
* Add a 'runtime' variable to Py_FinalizeEx() rather than working
directly on the global variable _PyRuntime
* Add a 'runtime' parameter to _PyGC_Fini(), _PyGILState_Fini()
and call_ll_exitfuncs()
The PyOS_AfterFork_Child() function now pass a 'runtime' parameter to
subfunctions.
* Fix _PyRuntimeState_ReInitThreads(): use the correct memory allocator
* Add runtime parameter to _PyRuntimeState_ReInitThreads(),
_PyGILState_Reinit() and _PyInterpreterState_DeleteExceptMain()
* Move _PyGILState_Reinit() to the internal C API.
* Add 'runtime' parameter to _PyThreadState_Init()
* Add 'gilstate' parameter to _PyGILState_NoteThreadState()
* Move _PyThreadState_Init() and _PyThreadState_DeleteExcept()
to the internal C API.
Change PyAPI_FUNC(type), PyAPI_DATA(type) and PyMODINIT_FUNC macros
of pyport.h when Py_BUILD_CORE_MODULE is defined.
The Py_BUILD_CORE_MODULE define must be now be used to build a C
extension as a dynamic library accessing Python internals: export the
PyInit_xxx() function in DLL exports on Windows.
Changes:
* Py_BUILD_CORE_BUILTIN and Py_BUILD_CORE_MODULE now imply
Py_BUILD_CORE directy in pyport.h.
* ceval.c compilation now fails with an error if Py_BUILD_CORE is not
defined, just to ensure that Python is build with the correct
defines.
* setup.py now compiles _pickle.c with Py_BUILD_CORE_MODULE define.
* setup.py compiles _json.c with Py_BUILD_CORE_MODULE define, rather
than Py_BUILD_CORE_BUILTIN define
* PCbuild/pythoncore.vcxproj: Add Py_BUILD_CORE_BUILTIN define.
Fix Python Initialization code on FreeBSD to detect properly when
stdin file descriptor (fd 0) is invalid.
On FreeBSD, fstat() must be used to check if stdin (fd 0) is valid.
dup(0) doesn't fail if stdin is invalid in some cases.
This is effectively an un-revert of #11617 and #12024 (reverted in #12159). Portions of those were merged in other PRs (with lower risk) and this represents the remainder. Note that I found 3 different bugs in the original PRs and have fixed them here.
Fix time.mktime() error handling on AIX for year before 1970.
Other changes:
* mktime(): rename variable 'buf' to 'tm'.
* _PyTime_localtime():
* Use "localtime" rather than "ctime" in the error message
(specific to AIX).
* Always initialize errno to 0 just in case if localtime_r()
doesn't set errno on error.
* On AIX, avoid abs() which is limited to int type.
* EINVAL constant is now always available.