Commit Graph

6631 Commits

Author SHA1 Message Date
Victor Stinner 7be4e350aa
bpo-40513: Per-interpreter GIL (GH-19943)
In the experimental isolated subinterpreters build mode, the GIL is
now per-interpreter.

Move gil from _PyRuntimeState.ceval to PyInterpreterState.ceval.

new_interpreter() always get the config from the main interpreter.
2020-05-05 20:27:47 +02:00
Victor Stinner 0dd5e7a718
bpo-40513: new_interpreter() init GIL earlier (GH-19942)
Fix also code to handle init_interp_main() failure.
2020-05-05 20:16:37 +02:00
Victor Stinner e838a9324c
bpo-40522: _PyThreadState_Swap() sets autoTSSkey (GH-19939)
In the experimental isolated subinterpreters build mode,
_PyThreadState_GET() gets the autoTSSkey variable and
_PyThreadState_Swap() sets the autoTSSkey variable.

* Add _PyThreadState_GetTSS()
* _PyRuntimeState_GetThreadState() and _PyThreadState_GET()
  return _PyThreadState_GetTSS()
* PyEval_SaveThread() sets the autoTSSkey variable to current Python
  thread state rather than NULL.
* eval_frame_handle_pending() doesn't check that
  _PyThreadState_Swap() result is NULL.
* _PyThreadState_Swap() gets the current Python thread state with
  _PyThreadState_GetTSS() rather than
  _PyRuntimeGILState_GetThreadState().
* PyGILState_Ensure() no longer checks _PyEval_ThreadsInitialized()
  since it cannot access the current interpreter.
2020-05-05 19:56:48 +02:00
Victor Stinner 299b8c61e9
Revert "bpo-40513: Per-interpreter signals pending (GH-19924)" (GH-19932)
This reverts commit 4e01946caf.
2020-05-05 17:40:18 +02:00
Victor Stinner b0be6b3b94
bpo-29587: _PyErr_ChainExceptions() checks exception (GH-19902)
_PyErr_ChainExceptions() now ensures that the first parameter is an
exception type, as done by _PyErr_SetObject().

* The following function now check PyExceptionInstance_Check() in an
  assertion using a new _PyBaseExceptionObject_cast() helper
  function:

  * PyException_GetTraceback(), PyException_SetTraceback()
  * PyException_GetCause(), PyException_SetCause()
  * PyException_GetContext(), PyException_SetContext()

* PyExceptionClass_Name() now checks PyExceptionClass_Check() with an
  assertion.

* Remove XXX comment and add gi_exc_state variable to _gen_throw().

* Remove comment from test_generators
2020-05-05 17:07:41 +02:00
Victor Stinner 4e30ed3af0
bpo-40513: Per-interpreter recursion_limit (GH-19929)
Move recursion_limit member from _PyRuntimeState.ceval to
PyInterpreterState.ceval.

* Py_SetRecursionLimit() now only sets _Py_CheckRecursionLimit
  of ceval.c if the current Python thread is part of the main
  interpreter.
* Inline _Py_MakeEndRecCheck() into _Py_LeaveRecursiveCall().
* Convert _Py_RecursionLimitLowerWaterMark() macro into a static
  inline function.
2020-05-05 16:52:52 +02:00
Victor Stinner c5fa364f4e
bpo-40514: Add --with-experimental-isolated-subinterpreters (GH-19926)
Add --with-experimental-isolated-subinterpreters build option to
configure: better isolate subinterpreters, experimental build mode.

When used, force the usage of the libc malloc() memory allocator,
since pymalloc relies on the unique global interpreter lock (GIL).
2020-05-05 16:41:11 +02:00
Victor Stinner 0b1e3307e2
bpo-40513: Per-interpreter gil_drop_request (GH-19927)
Move gil_drop_request member from _PyRuntimeState.ceval to
PyInterpreterState.ceval.
2020-05-05 16:14:31 +02:00
Victor Stinner 4e01946caf
bpo-40513: Per-interpreter signals pending (GH-19924)
Move signals_pending from _PyRuntime.ceval to
PyInterpreterState.ceval.
2020-05-05 15:43:37 +02:00
Lysandros Nikolaou 846d8b28ab
bpo-40246: Revert reporting of invalid string prefixes (GH-19888)
Due to backwards compatibility concerns regarding keywords immediately followed by a string without whitespace between them (like in `bg="#d00" if clear else"#fca"`) will fail to parse,
commit 41d5b94af4 has to be reverted.
2020-05-04 12:32:18 +01:00
Robert Rouhani f40bd466bf
bpo-40417: Fix deprecation warning in PyImport_ReloadModule (GH-19750)
I can add another commit with the new test case I wrote to verify that the warning was being printed before my change, stopped printing after my change, and that the function does not return null after my change.

Automerge-Triggered-By: @brettcannon
2020-05-01 16:28:06 -07:00
Gregory Szorc 64224a4727
bpo-40412: Nullify inittab_copy during finalization (GH-19746)
Otherwise we leave a dangling pointer to free'd memory. If we
then initialize a new interpreter in the same process and call
PyImport_ExtendInittab, we will (likely) crash when calling
PyMem_RawRealloc(inittab_copy, ...) since the pointer address
is bogus.

Automerge-Triggered-By: @brettcannon
2020-05-01 11:07:54 -07:00
Victor Stinner 252346acd9
bpo-40453: Add PyConfig._isolated_subinterpreter (GH-19820)
An isolated subinterpreter cannot spawn threads, spawn a child
process or call os.fork().

* Add private _Py_NewInterpreter(isolated_subinterpreter) function.
* Add isolated=True keyword-only parameter to
  _xxsubinterpreters.create().
* Allow again os.fork() in "non-isolated" subinterpreters.
2020-05-01 11:33:44 +02:00
Guido van Rossum c001c09e90
bpo-40334: Support type comments (GH-19780)
This implements full support for # type: <type> comments, # type: ignore <stuff> comments, and the func_type parsing mode for ast.parse() and compile().

Closes https://github.com/we-like-parsers/cpython/issues/95.

(For now, you need to use the master branch of mypy, since another issue unique to 3.9 had to be fixed there, and there's no mypy release yet.)

The only thing missing is `feature_version=N`, which is being tracked in https://github.com/we-like-parsers/cpython/issues/124.
2020-04-30 12:12:19 -07:00
Victor Stinner 7036477323
bpo-40421: Add PyFrame_GetBack() function (GH-19765)
New PyFrame_GetBack() function: get the frame next outer frame.

Replace frame->f_back with PyFrame_GetBack(frame) in most code but
frameobject.c, ceval.c and genobject.c.
2020-04-29 03:28:46 +02:00
Victor Stinner 4386b9045e
bpo-40429: PyThreadState_GetFrame() returns a strong ref (GH-19781)
The PyThreadState_GetFrame() function now returns a strong reference
to the frame.
2020-04-29 03:01:43 +02:00
Victor Stinner ae00a5a885
bpo-40428: Remove PyTuple_ClearFreeList() function (GH-19769)
Remove the following function from the C API:

* PyAsyncGen_ClearFreeLists()
* PyContext_ClearFreeList()
* PyDict_ClearFreeList()
* PyFloat_ClearFreeList()
* PyFrame_ClearFreeList()
* PyList_ClearFreeList()
* PySet_ClearFreeList()
* PyTuple_ClearFreeList()

Make these functions private, move them to the internal C API and
change their return type to void.

Call explicitly PyGC_Collect() to free all free lists.

Note: PySet_ClearFreeList() did nothing.
2020-04-29 02:29:20 +02:00
Victor Stinner 8852ad4208
bpo-40429: PyFrame_GetCode() now returns a strong reference (GH-19773) 2020-04-29 01:28:13 +02:00
Victor Stinner 6d86a2331e
bpo-40429: PyFrame_GetCode() result cannot be NULL (GH-19772)
Add frame_nslots() to factorize duplicate code.
2020-04-29 00:56:58 +02:00
Victor Stinner a42ca74fa3
bpo-40421: Add PyFrame_GetCode() function (GH-19757)
PyFrame_GetCode(frame): return a borrowed reference to the frame
code.

Replace frame->f_code with PyFrame_GetCode(frame) in most code,
except in frameobject.c, genobject.c and ceval.c.

Also add PyFrame_GetLineNumber() to the limited C API.
2020-04-28 19:01:31 +02:00
Pablo Galindo 2b74c835a7
bpo-40334: Support CO_FUTURE_BARRY_AS_BDFL in the new parser (GH-19721)
This commit also allows to pass flags to the new parser in all interfaces and fixes a bug in the parser generator that was causing to inline rules with actions, making them disappear.
2020-04-27 18:02:07 +01:00
Victor Stinner 25104949a5
bpo-40048: Fix _PyCode_InitOpcache() error path (GH-19691)
If _PyCode_InitOpcache() fails in _PyEval_EvalFrameDefault(), use
"goto exit_eval_frame;" rather than "return NULL;" to exit the
function in a consistent state. For example, tstate->frame is now
reset properly.
2020-04-24 02:43:18 +02:00
Guido van Rossum bc28805570
bpo-40334: Use old compiler when compile mode is func_type (GH-19692)
This is invoked by mypy, using ast.parse(source, "<func_type>", "func_type"). Since the new grammar doesn't yet support the func_type_input start symbol we must use the old compiler in this case to prevent a crash.

https://bugs.python.org/issue40334
2020-04-23 15:42:56 -07:00
Lysandros Nikolaou ebebb6429c
bpo-40334: Improve various PEG-Parser related stuff (GH-19669)
The changes in this commit are all related to @vstinner's original review comments of the initial PEP 617 implementation PR.
2020-04-23 16:36:06 +01:00
Pablo Galindo ee40e4b856
bpo-40334: Don't downcast from Py_ssize_t to int (GH-19671) 2020-04-23 03:43:08 +01:00
Victor Stinner 1def7754b7
bpo-40334: Rename PyConfig.use_peg to _use_peg_parser (GH-19670)
* Rename PyConfig.use_peg to _use_peg_parser
* Document PyConfig._use_peg_parser and mark it a deprecated
* Mark -X oldparser option and PYTHONOLDPARSER env var as deprecated
  in the documentation.
* Add use_old_parser() and skip_if_new_parser() to test.support
* Remove sys.flags.use_peg: use_old_parser() uses
  _testinternalcapi.get_configs() instead.
* Enhance test_embed tests
* subprocess._args_from_interpreter_flags() copies -X oldparser
2020-04-23 03:03:24 +02:00
Pablo Galindo c5fc156852
bpo-40334: PEP 617 implementation: New PEG parser for CPython (GH-19503)
Co-authored-by: Guido van Rossum <guido@python.org>
Co-authored-by: Lysandros Nikolaou <lisandrosnik@gmail.com>
2020-04-22 23:29:27 +01:00
Batuhan Taşkaya 4454057269
bpo-39562: Prevent collision of future and compiler flags (GH-19230)
The constant values of future flags in the __future__ module
is updated in order to prevent collision with compiler flags.
Previously PyCF_ALLOW_TOP_LEVEL_AWAIT was clashing
with CO_FUTURE_DIVISION.
2020-04-22 18:09:03 +02:00
sweeneyde 6a9e80a931
bpo-40313: speed up bytes.hex() (GH-19594)
Automerge-Triggered-By: @gpshead
2020-04-20 17:17:52 -07:00
Hakan Çelik ce578831a4
Add spaces around the ":=" operator in ast_unparse.c (GH-19568) 2020-04-18 17:17:19 +01:00
Victor Stinner 1a1bd2e238
bpo-40302: Replace PY_INT64_T with int64_t (GH-19573)
* Replace PY_INT64_T with int64_t
* Replace PY_UINT32_T with uint32_t
* Replace PY_UINT64_T with uint64_t

sha3module.c no longer checks if PY_UINT64_T is defined since it's
always defined and uint64_t is always available on platforms
supported by Python.
2020-04-17 19:13:06 +02:00
Victor Stinner 455df97798
Optimize _Py_strhex_impl() (GH-19535)
Avoid a temporary buffer to create a bytes string: use
PyBytes_FromStringAndSize() to directly allocate a bytes object.

Cleanup also the code: PEP 7 formatting, move variable definitions
closer to where they are used. Fix assertion checking "j" index.
2020-04-15 14:05:24 +02:00
Victor Stinner 4f98f465f1
bpo-40268: Remove unused imports in pylifecycle.c (GH-19533)
Remove unused imports in files:

* initconfig.c
* main.c
* preconfig.h
* pylifecycle.c
* python.c
* pythonrun.c
2020-04-15 04:01:58 +02:00
Victor Stinner 361dcdcefc
bpo-40268: Remove unused osdefs.h includes (GH-19532)
When the include is needed, add required symbol in a comment.
2020-04-15 03:24:57 +02:00
Victor Stinner d9ea5cae1d
bpo-40268: Remove unused pycore_pymem.h includes (GH-19531) 2020-04-15 02:57:50 +02:00
Victor Stinner 4a21e57fe5
bpo-40268: Remove unused structmember.h includes (GH-19530)
If only offsetof() is needed: include stddef.h instead.

When structmember.h is used, add a comment explaining that
PyMemberDef is used.
2020-04-15 02:35:41 +02:00
Victor Stinner 62183b8d6d
bpo-40268: Remove explicit pythread.h includes (#19529)
Remove explicit pythread.h includes: it is always included
by Python.h.
2020-04-15 02:04:42 +02:00
Ethan Smith d01628e411
bpo-39481: PEP 585 for dataclasses, mailbox, contextvars (GH-19425) 2020-04-14 16:14:15 -07:00
Pablo Galindo 33986465bd
bpo-39522: Always initialise kind attribute in constant ast nodes (GH-19525) 2020-04-14 21:40:41 +01:00
Batuhan Taşkaya 43aeefa419
bpo-39522: Use _PyUnicodeWriter_WriteStr instead of PyUnicode_AS_DATA (GH-19523) 2020-04-14 21:21:22 +01:00
Batuhan Taşkaya aade1cc453
bpo-395222: Correctly unparse unicode prefix in ast_unparse.c (GH-19512) 2020-04-14 19:55:01 +01:00
Hai Shi 675d9a3d7a
bpo-40170: Convert PyObject_IS_GC() macro to a function (GH-19464) 2020-04-14 20:11:20 +02:00
Victor Stinner e560f90602
bpo-40268: Move struct _gc_runtime_state to pycore_gc.h (GH-19515) 2020-04-14 18:30:41 +02:00
Dong-hee Na 62f75fe3dd
bpo-40232: Update PyOS_AfterFork_Child() to use _PyThread_at_fork_reinit() (GH-19450) 2020-04-15 01:16:24 +09:00
Victor Stinner e5014be049
bpo-40268: Remove a few pycore_pystate.h includes (GH-19510) 2020-04-14 17:52:15 +02:00
Victor Stinner 81a7be3fa2
bpo-40268: Rename _PyInterpreterState_GET_UNSAFE() (GH-19509)
Rename _PyInterpreterState_GET_UNSAFE() to _PyInterpreterState_GET()
for consistency with _PyThreadState_GET() and to have a shorter name
(help to fit into 80 columns).

Add also "assert(tstate != NULL);" to the function.
2020-04-14 15:14:01 +02:00
Victor Stinner 4a3fe08353
bpo-40268: Include explicitly pycore_interp.h (GH-19505)
pycore_pystate.h no longer includes pycore_interp.h:
it's now included explicitly in files accessing PyInterpreterState.
2020-04-14 14:26:24 +02:00
Batuhan Taşkaya 258f5179f9
bpo-32894: Support unparsing of infinity numbers in ast_unparser.c (GH-17426) 2020-04-14 00:51:31 +02:00
Victor Stinner 1c4cbdf94d
bpo-40268: Add pycore_runtime.h header file (GH-19493)
Move PyRuntimeState from pycore_pystate.h to pycore_runtime.h.

Remove _PyGILState_check_enabled macro: access directly
_PyRuntime.gilstate.check_enabled.
2020-04-13 11:45:21 +02:00
Victor Stinner da7933ecc3
bpo-40268: Add _PyInterpreterState_GetConfig() (GH-19492)
Don't access PyInterpreterState.config member directly anymore, but
use new functions:

* _PyInterpreterState_GetConfig()
* _PyInterpreterState_SetConfig()
* _Py_GetConfig()
2020-04-13 03:04:28 +02:00
Lysandros Nikolaou 41d5b94af4
bpo-40246: Report a better error message for invalid string prefixes (GH-19476) 2020-04-12 19:21:00 +01:00
Serhiy Storchaka 8f87eefe7f
bpo-39943: Add the const qualifier to pointers on non-mutable PyBytes data. (GH-19472) 2020-04-12 14:58:27 +03:00
Serhiy Storchaka cd8295ff75
bpo-39943: Add the const qualifier to pointers on non-mutable PyUnicode data. (GH-19345) 2020-04-11 10:48:40 +03:00
Victor Stinner b54a99d643
bpo-40082: trip_signal() uses the main interpreter (GH-19441)
Fix the signal handler: it now always uses the main interpreter,
rather than trying to get the current Python thread state.

The following function now accepts an interpreter, instead of a
Python thread state:

* _PyEval_SignalReceived()
* _Py_ThreadCanHandleSignals()
* _PyEval_AddPendingCall()
* COMPUTE_EVAL_BREAKER()
* SET_GIL_DROP_REQUEST(), RESET_GIL_DROP_REQUEST()
* SIGNAL_PENDING_CALLS(), UNSIGNAL_PENDING_CALLS()
* SIGNAL_PENDING_SIGNALS(), UNSIGNAL_PENDING_SIGNALS()
* SIGNAL_ASYNC_EXC(), UNSIGNAL_ASYNC_EXC()

Py_AddPendingCall() now uses the main interpreter if it fails to the
current Python thread state.

Convert _PyThreadState_GET() and PyInterpreterState_GET_UNSAFE()
macros to static inline functions.
2020-04-08 23:35:05 +02:00
Victor Stinner cfc3c2f8b3
bpo-37127: Remove _pending_calls.finishing (GH-19439) 2020-04-08 22:10:53 +02:00
Victor Stinner dda5d6e071
bpo-40226: PyInterpreterState_Delete() deletes pending calls (GH-19436)
PyInterpreterState_New() is now responsible to create pending calls,
PyInterpreterState_Delete() now deletes pending calls.

* Rename _PyEval_InitThreads() to _PyEval_InitGIL() and rename
  _PyEval_InitGIL() to _PyEval_FiniGIL().
* _PyEval_InitState() and PyEval_FiniState() now create and delete
  pending calls. _PyEval_InitState() now returns -1 on memory
  allocation failure.
* Add init_interp_create_gil() helper function: code shared by
  Py_NewInterpreter() and Py_InitializeFromConfig().
* init_interp_create_gil() now also calls _PyEval_FiniGIL(),
  _PyEval_InitGIL() and _PyGILState_Init() in subinterpreters, but
  these functions now do nothing when called from a subinterpreter.
2020-04-08 17:54:59 +02:00
Victor Stinner a15e260b70
bpo-40170: Add _PyIndex_Check() internal function (GH-19426)
Add _PyIndex_Check() function to the internal C API: fast inlined
verson of PyIndex_Check().

Add Include/internal/pycore_abstract.h header file.

Replace PyIndex_Check() with _PyIndex_Check() in C files of Objects
and Python subdirectories.
2020-04-08 02:01:56 +02:00
Victor Stinner 87255be696
bpo-40089: Add _at_fork_reinit() method to locks (GH-19195)
Add a private _at_fork_reinit() method to _thread.Lock,
_thread.RLock, threading.RLock and threading.Condition classes:
reinitialize the lock after fork in the child process; reset the lock
to the unlocked state.

Rename also the private _reset_internal_locks() method of
threading.Event to _at_fork_reinit().

* Add _PyThread_at_fork_reinit() private function. It is excluded
  from the limited C API.
* threading.Thread._reset_internal_locks() now calls
  _at_fork_reinit() on self._tstate_lock rather than creating a new
  Python lock object.
2020-04-07 23:11:49 +02:00
Zackery Spytz 08050e959e
bpo-40147: Fix a compiler warning on Windows in Python/compile.c (GH-19389)
Change the type of nkeywords to Py_ssize_t.
2020-04-06 07:47:47 +01:00
Batuhan Taşkaya 1b21573a89
closes bpo-40184: Only define pysiphash if the hash algorithm is SIPHASH24. (GH-19369) 2020-04-04 16:25:12 -05:00
Pablo Galindo 40cf35c5b0
bpo-40141: Include the value in the column position for keyword AST nodes (GH-19348) 2020-04-03 21:02:26 +01:00
Pablo Galindo 254ec78341
bpo-40147: Move the check for duplicate keywords to the compiler (GH-19289) 2020-04-03 20:37:13 +01:00
Pablo Galindo 168660b547
bpo-40141: Add line and column information to ast.keyword nodes (GH-19283) 2020-04-02 00:47:39 +01:00
Serhiy Storchaka 2c003eff8f
bpo-39943: Clean up marshal.c. (GH-19236)
* Add consts.
* Remove redundant casts and checks.
* Use concrete C API macros.
* Avoid raising and silencing OverflowError for ints.
2020-03-31 23:23:21 +03:00
Victor Stinner 400e1dbcad
Document most common signals (GH-19245)
Document individual signals (only the most common signals):
description, default action, availability.
2020-03-31 19:13:10 +02:00
Victor Stinner 8d84adcd73
bpo-32591: _PyErr_WarnUnawaitedCoroutine() sets source (GH-19247)
The _PyErr_WarnUnawaitedCoroutine() fallback now also sets the
coroutine object as the source of the warning, as done by the Python
implementation warnings._warn_unawaited_coroutine().

Moreover, don't truncate the coroutine name: Python supports
arbitrary string length to format the message.
2020-03-31 17:25:12 +02:00
Andy Lester fc2d8d62af
bpo-39943: Remove unnecessary casts in import.c that remove constness (GH-19209) 2020-03-30 13:19:14 -07:00
Victor Stinner 1c1e68cf3e
bpo-38644: Use _PySys_Audit(): pass tstate explicitly (GH-19183)
Add the dependency to tstate more explicit.
2020-03-27 15:11:45 +01:00
Victor Stinner 71a3522ef8
bpo-38644: Make tstate more explicit inside pystate.c (GH-19182)
Fix PyInterpreterState_New(): Don't call PyErr_SetString() when there
is no current Python thread state (if tstate is NULL).
2020-03-26 22:46:14 +01:00
Victor Stinner 728189884e
bpo-38644: Pass tstate explicitly in signalmodule.c (GH-19184)
PyOS_InterruptOccurred() now checks _Py_ThreadCanHandleSignals()
before checking if SIGINT is tripped.
2020-03-26 22:28:11 +01:00
Victor Stinner 08faf0016e
bpo-38644: Add _PySys_Audit() which takes tstate (GH-19180)
Add _PySys_Audit() function to the internal C API: similar to
PySys_Audit(), but requires a mandatory tstate parameter.

Cleanup sys_audit_tstate() code: remove code path for NULL tstate,
since the function exits at entry if tstate is NULL. Remove also code
path for NULL tstate->interp: should_audit() now ensures that it is
not NULL (even if tstate->interp cannot be NULL in practice).

PySys_AddAuditHook() now checks if tstate is not NULL to decide if
tstate can be used or not, and tstate is set to NULL if the runtime
is not initialized yet.

Use _PySys_Audit() in sysmodule.c.
2020-03-26 18:57:32 +01:00
Zackery Spytz 79ceccd1ec
bpo-38410: Properly handle PySys_Audit() failures (GH-16657) 2020-03-26 12:11:13 +00:00
Furkan Önder cb6534e1a8
bpo-40067: Improve error messages for multiple star expressions in assignments (GH-19168)
Co-Authored-By: Batuhan Taşkaya <isidentical@gmail.com>
Co-Authored-By: Pablo Galindo <Pablogsal@gmail.com>
2020-03-26 01:54:31 +00:00
Victor Stinner 5c3cda0d1a
bpo-39947: Add PyThreadState_GetID() function (GH-19163)
Add PyThreadState_GetID() function: get the unique identifier of a
Python thread state.
2020-03-25 21:23:53 +01:00
Victor Stinner 0e427c6d15
bpo-39947: Add _PyThreadState_GetDict() function (GH-19160) 2020-03-25 21:22:55 +01:00
Victor Stinner 87d3b9db4a
bpo-39882: Add _Py_FatalErrorFormat() function (GH-19157) 2020-03-25 19:27:36 +01:00
Andy Lester 7668a8bc93
Use calloc-based functions, not malloc. (GH-19152) 2020-03-24 23:26:44 -05:00
Victor Stinner 83d46e0622
bpo-40050: Fix importlib._bootstrap_external (GH-19135)
Remove two unused imports: _thread and _weakref. Avoid creating a new
winreg builtin module if it's already available in sys.modules.

The winreg module is now stored as "winreg" rather than "_winreg".
2020-03-24 18:03:34 +01:00
Victor Stinner 5804f878e7
bpo-20526: Fix PyThreadState_Clear(): don't decref frame (GH-19120)
PyThreadState.frame is a borrowed reference, not a strong reference:
PyThreadState_Clear() must not call Py_CLEAR(tstate->frame).

Remove test_threading.test_warnings_at_exit(): we cannot warranty
that the Python thread state of daemon threads is cleared in a
reliable way during Python shutdown.
2020-03-24 16:32:26 +01:00
Serhiy Storchaka bace59d8b8
bpo-39999: Improve compatibility of the ast module. (GH-19056)
* Re-add removed classes Suite, slice, Param, AugLoad and AugStore.
* Add docstrings for dummy classes.
* Add docstrings for attribute aliases.
* Set __module__ to "ast" instead of "_ast".
2020-03-22 20:33:34 +02:00
Ronald Oussoren 044cf94f61
bpo-22490: Remove __PYVENV_LAUNCHER__ from environment during launch (GH-9516)
* bpo-22490: Remove "__PYVENV_LAUNCHER__" from the shell environment on macOS

This changeset removes the environment varialbe "__PYVENV_LAUNCHER__"
during interpreter launch as it is only needed to communicate between
the stub executable in framework installs and the actual interpreter.

Leaving the environment variable present may lead to misbehaviour when
launching other scripts.

* Actually commit the changes for issue 22490...

* Correct typo

Co-Authored-By: Nicola Soranzo <nicola.soranzo@gmail.com>

* Run make patchcheck

Co-authored-by: Jason R. Coombs <jaraco@jaraco.com>
Co-authored-by: Nicola Soranzo <nicola.soranzo@gmail.com>
2020-03-22 14:31:46 -04:00
Dong-hee Na 05e4a296ec
bpo-40024: Add PyModule_AddType() helper function (GH-19088) 2020-03-22 17:17:34 +01:00
Victor Stinner 6723e933c4
bpo-39946: Remove _PyThreadState_GetFrame (GH-19094)
Remove _PyRuntime.getframe hook and remove _PyThreadState_GetFrame
macro which was an alias to _PyRuntime.getframe. They were only
exposed by the internal C API. Remove also PyThreadFrameGetter type.
2020-03-20 17:46:56 +01:00
Victor Stinner fd1e1a18fa
bpo-39947: Add PyThreadState_GetFrame() function (GH-19092)
Add PyThreadState_GetFrame() function: get the current frame
of a Python thread state.
2020-03-20 15:51:45 +01:00
Victor Stinner d83168854e
bpo-40010: Optimize pending calls in multithreaded applications (GH-19091)
If a thread different than the main thread schedules a pending call
(Py_AddPendingCall()), the bytecode evaluation loop is no longer
interrupted at each bytecode instruction to check for pending calls
which cannot be executed. Only the main thread can execute pending
calls.

Previously, the bytecode evaluation loop was interrupted at each
instruction until the main thread executes pending calls.

* Add _Py_ThreadCanHandlePendingCalls() function.
* SIGNAL_PENDING_CALLS() now only sets eval_breaker to 1 if the
  current thread can execute pending calls. Only the main thread can
  execute pending calls.
2020-03-20 14:50:35 +01:00
Victor Stinner d2a8e5b42c
bpo-40010: COMPUTE_EVAL_BREAKER() checks for subinterpreter (GH-19087)
COMPUTE_EVAL_BREAKER() now also checks if the Python thread state
belongs to the main interpreter. Don't break the evaluation loop if
there are pending signals but the Python thread state it belongs to a
subinterpeter.

* Add _Py_IsMainThread() function.
* Add _Py_ThreadCanHandleSignals() function.
2020-03-20 13:38:58 +01:00
Victor Stinner da2914db4b
bpo-40010: Pass tstate to ceval GIL functions (GH-19077)
* Add eval_frame_handle_pending() helper function: cold code path.
* Fix PyEval_ReleaseLock(): don't dereference tstate if it's NULL.
2020-03-20 09:29:08 +01:00
Victor Stinner a36adfa6bb
bpo-39877: 4th take_gil() fix for daemon threads (GH-19080)
bpo-39877, bpo-40010: Add a third tstate_must_exit() check in
take_gil() to prevent using tstate which has been freed.
2020-03-19 19:48:25 +01:00
Victor Stinner 5a3a71dddb
bpo-40010: Optimize signal handling in multithreaded applications (GH-19067)
If a thread different than the main thread gets a signal, the
bytecode evaluation loop is no longer interrupted at each bytecode
instruction to check for pending signals which cannot be handled.
Only the main thread of the main interpreter can handle signals.

Previously, the bytecode evaluation loop was interrupted at each
instruction until the main thread handles signals.

Changes:

* COMPUTE_EVAL_BREAKER() and SIGNAL_PENDING_SIGNALS() no longer set
  eval_breaker to 1 if the current thread cannot handle signals.
* take_gil() now always recomputes eval_breaker.
2020-03-19 17:40:12 +01:00
Batuhan Taşkaya 9052f7a41b
bpo-39562: Allow executing asynchronous comprehensions in the asyncio REPL (GH-18968)
Co-authored-by: Pablo Galindo <Pablogsal@gmail.com>
2020-03-19 11:35:44 +00:00
Batuhan Taşkaya 0ac59f93c0
bpo-40000: Improve error messages when validating invalid ast.Constant nodes (GH-19055)
Co-authored-by: Pablo Galindo <Pablogsal@gmail.com>
2020-03-19 11:32:28 +00:00
Victor Stinner 50e6e99178
bpo-39984: Move pending calls to PyInterpreterState (GH-19066)
If Py_AddPendingCall() is called in a subinterpreter, the function is
now scheduled to be called from the subinterpreter, rather than being
called from the main interpreter.

Each subinterpreter now has its own list of scheduled calls.

* Move pending and eval_breaker fields from _PyRuntimeState.ceval
  to PyInterpreterState.ceval.
* new_interpreter() now calls _PyEval_InitThreads() to create
  pending calls lock.
* Fix Py_AddPendingCall() for subinterpreters. It now calls
  _PyThreadState_GET() which works in a subinterpreter if the
  caller holds the GIL, and only falls back on
  PyGILState_GetThisThreadState() if _PyThreadState_GET()
  returns NULL.
2020-03-19 02:41:21 +01:00
Pablo Galindo d112c600ab
bpo-39220: Do not optimise annotation if 'from __future__ import annotations' is used (GH-17866)
Do not apply AST-based optimizations if 'from __future__ import annotations' is used in order to
prevent information lost in the final version of the annotations.
2020-03-18 23:02:09 +00:00
Victor Stinner 8849e5962b
bpo-39984: trip_signal() uses PyGILState_GetThisThreadState() (GH-19061)
bpo-37127, bpo-39984:

* trip_signal() and Py_AddPendingCall() now get the current Python
  thread state using PyGILState_GetThisThreadState() rather than
  _PyRuntimeState_GetThreadState() to be able to get it even if the
  GIL is released.
* _PyEval_SignalReceived() now expects tstate rather than ceval.
* Remove ceval parameter of _PyEval_AddPendingCall(): ceval is now
  get from tstate parameter.
2020-03-18 19:28:53 +01:00
Victor Stinner 56bfdebfb1
bpo-39984: Pass tstate to _PyEval_SignalAsyncExc() (GH-19049)
_PyEval_SignalAsyncExc() and _PyEval_FiniThreads() now expect tstate,
instead of ceval.
2020-03-18 09:26:25 +01:00
Victor Stinner 29356e03d4
bpo-39877: Fix take_gil() for daemon threads (GH-19054)
bpo-39877, bpo-39984: If the thread must exit, don't access tstate to
prevent a potential crash: tstate memory has been freed.
2020-03-18 03:04:33 +01:00
Victor Stinner 23ef89db7a
bpo-39984: _PyThreadState_DeleteCurrent() takes tstate (GH-19051)
* _PyThreadState_DeleteCurrent() now takes tstate rather than
  runtime.
* Add ensure_tstate_not_null() helper to pystate.c.
* Add _PyEval_ReleaseLock() function.
* _PyThreadState_DeleteCurrent() now calls
  _PyEval_ReleaseLock(tstate) and frees PyThreadState memory after
  this call, not before.
* PyGILState_Release(): rename "tcur" variable to "tstate".
2020-03-18 02:26:04 +01:00
Victor Stinner d7fabc1162
bpo-39984: Pass tstate to handle_signals() (GH-19050)
handle_signals() and make_pending_calls() now expect tstate rather
than runtime.
2020-03-18 01:56:21 +01:00
Serhiy Storchaka 6b97598fb6
bpo-39988: Remove ast.AugLoad and ast.AugStore node classes. (GH-19038) 2020-03-17 23:41:08 +02:00
Victor Stinner dab8423d22
bpo-39984: Add PyInterpreterState.ceval (GH-19047)
subinterpreters: Move _PyRuntimeState.ceval.tracing_possible to
PyInterpreterState.ceval.tracing_possible: each interpreter now has
its own variable.

Changes:

* Add _ceval_state structure.
* Add PyInterpreterState.ceval field.
* _PyEval_EvalFrameDefault(): add ceval2 variable (struct _ceval_state*).
* Rename _PyEval_Initialize() to _PyEval_InitRuntimeState().
* Add _PyEval_InitState().
* Don't export internal _Py_FinishPendingCalls() and
  _PyEval_FiniThreads() functions anymore.
2020-03-17 18:56:44 +01:00
Serhiy Storchaka 61cb3d02b8
bpo-39987: Simplify setting lineno in the compiler. (GH-19037) 2020-03-17 18:07:30 +02:00
Victor Stinner f6a5850782
bpo-35370: PyEval_SetTrace() logs unraisable error (GH-18977)
If PySys_Audit() fails in PyEval_SetProfile() or PyEval_SetTrace(),
log the error as an unraisable exception.
2020-03-16 17:41:44 +01:00
Dong-hee Na 87ec86c425
bpo-37207: Add _PyArg_NoKwnames() helper function (GH-18980) 2020-03-16 15:06:20 +01:00
Batuhan Taşkaya 4ab362cec6
bpo-39638: Keep ASDL signatures in the AST nodes (GH-18515) 2020-03-16 10:12:53 +02:00
Batuhan Taşkaya 8689209e03
bpo-39969: Remove ast.Param node class as is no longer used (GH-19020) 2020-03-15 19:32:17 +00:00
Pablo Galindo 90235810ec
bpo-39965: Correctly raise SyntaxError if await is used outside async functions when PyCF_ALLOW_TOP_LEVEL_AWAIT is set (GH-19010) 2020-03-15 04:29:22 +00:00
Hai Shi 57c7810489
bpo-39585: Delete a pending comment in _warning.c (GH-18414) 2020-03-14 15:40:58 +02:00
Victor Stinner 8fb02b6e19
bpo-39947: Add PyThreadState_GetInterpreter() (GH-18981)
Add PyThreadState_GetInterpreter(tstate): get the interpreter of a
Python thread state.
2020-03-13 23:38:08 +01:00
Victor Stinner be79373a78
bpo-39947: Add PyInterpreterState_Get() function (GH-18979)
* Rename _PyInterpreterState_Get() to PyInterpreterState_Get() and
  move it the limited C API.
* Add _PyInterpreterState_Get() alias to PyInterpreterState_Get() for
  backward compatibility with Python 3.8.
2020-03-13 18:15:33 +01:00
Victor Stinner ff4584caca
bpo-39947: Use _PyInterpreterState_GET_UNSAFE() (GH-18978)
Replace _PyInterpreterState_Get() function call with
_PyInterpreterState_GET_UNSAFE() macro which is more efficient but
don't check if tstate or interp is NULL.

_Py_GetConfigsAsDict() now uses _PyThreadState_GET().
2020-03-13 18:03:56 +01:00
Victor Stinner 309d7cc5df
bpo-35370: Add _PyEval_SetTrace() function (GH-18975)
* sys.settrace(), sys.setprofile() and _lsprof.Profiler.enable() now
  properly report PySys_Audit() error if "sys.setprofile" or
  "sys.settrace" audit event is denied.
* Add _PyEval_SetProfile() and _PyEval_SetTrace() function: similar
  to PyEval_SetProfile() and PyEval_SetTrace() but take a tstate
  parameter and return -1 on error.
* Add _PyObject_FastCallTstate() function.
2020-03-13 16:39:12 +01:00
Victor Stinner 0b72b23fb0
bpo-38500: Add _PyInterpreterState_SetEvalFrameFunc() (GH-17340)
PyInterpreterState.eval_frame function now requires a tstate (Python
thread state) parameter.

Add private functions to the C API to get and set the frame
evaluation function:

* Add tstate parameter to _PyFrameEvalFunction function type.
* Add _PyInterpreterState_GetEvalFrameFunc() and
  _PyInterpreterState_SetEvalFrameFunc() functions.
* Add tstate parameter to _PyEval_EvalFrameDefault().
2020-03-12 23:18:39 +01:00
Andy Lester 75cd5bf428
Remove unused args from two pylifecycle.c functions (GH-18867)
* Remove unused tstate argument from init_set_builtins_open()
* Replace fd (int) with tstate (PyThreadState *) in in _Py_FatalError_PrintExc()
2020-03-12 08:49:05 +01:00
bobince 64838ce717
bpo-39847: EnterNonRecursiveMutex() uses GetTickCount64() (GH-18780)
The 32-bit (49-day) TickCount relied on in EnterNonRecursiveMutex can overflow
in the gap between the 'target' time and the 'now' time WaitForSingleObjectEx
returns, causing the loop to think it needs to wait another 49 days. This is
most likely to happen when the machine is hibernated during
WaitForSingleObjectEx.

This makes acquiring a lock/event/etc from the _thread or threading module
appear to never timeout.

Replace with GetTickCount64 - this is OK now Python no longer supports XP which
lacks it, and is in use for time.monotonic().

Co-authored-by: And Clover <and.clover@bromium.com>
2020-03-12 00:39:02 +01:00
Andy Lester 76d5877b72
closes bpo-39922: Remove unused args from four functions. (GH-18893) 2020-03-10 19:18:12 -07:00
Victor Stinner 4e53abb0f4
bpo-38631: _PyGILState_Init() returns PyStatus (GH-18908)
_PyGILState_Init() now returns PyStatus rather than calling
Py_FatalError() on failure.
2020-03-10 23:49:16 +01:00
Serhiy Storchaka 13d52c2686
bpo-34822: Simplify AST for subscription. (GH-9605)
* Remove the slice type.
* Make Slice a kind of the expr type instead of the slice type.
* Replace ExtSlice(slices) with Tuple(slices, Load()).
* Replace Index(value) with a value itself.

All non-terminal nodes in AST for expressions are now of the expr type.
2020-03-10 18:52:34 +02:00
Victor Stinner 8510f43078
bpo-1294959: Add sys.platlibdir attribute (GH-18381)
Add --with-platlibdir option to the configure script: name of the
platform-specific library directory, stored in the new sys.platlitdir
attribute. It is used to build the path of platform-specific dynamic
libraries and the path of the standard library.

It is equal to "lib" on most platforms. On Fedora and SuSE, it is
equal to "lib64" on 64-bit systems.

Co-Authored-By: Jan Matějek <jmatejek@suse.com>
Co-Authored-By: Matěj Cepl <mcepl@cepl.eu>
Co-Authored-By: Charalampos Stratakis <cstratak@redhat.com>
2020-03-10 09:53:09 +01:00
Victor Stinner b4698ecfdb
bpo-39877: Deprecate PyEval_InitThreads() (GH-18892)
Deprecated PyEval_InitThreads() and PyEval_ThreadsInitialized().
Calling PyEval_InitThreads() now does nothing.
2020-03-10 01:28:54 +01:00
Victor Stinner 175a704abf
bpo-39877: PyGILState_Ensure() don't call PyEval_InitThreads() (GH-18891)
PyGILState_Ensure() doesn't call PyEval_InitThreads() anymore when a
new Python thread state is created. The GIL is created by
Py_Initialize() since Python 3.7, it's not needed to call
PyEval_InitThreads() explicitly.

Add an assertion to ensure that the GIL is already created.
2020-03-10 00:37:48 +01:00
Victor Stinner 9ad58acbe8
bpo-19466: Py_Finalize() clears daemon threads earlier (GH-18848)
Clear the frames of daemon threads earlier during the Python shutdown to
call objects destructors. So "unclosed file" resource warnings are now
emitted for daemon threads in a more reliable way.

Cleanup _PyThreadState_DeleteExcept() code: rename "garbage" to
"list".
2020-03-09 23:37:49 +01:00
Victor Stinner 9229eeee10
bpo-39877: take_gil() checks tstate_must_exit() twice (GH-18890)
take_gil() now also checks tstate_must_exit() after acquiring
the GIL: exit the thread if Py_Finalize() has been called.
2020-03-09 23:10:53 +01:00
Serhiy Storchaka b7e9525f9c
bpo-36287: Make ast.dump() not output optional fields and attributes with default values. (GH-18843)
The default values for optional fields and attributes of AST nodes are now set
as class attributes (e.g. Constant.kind is set to None).
2020-03-10 00:07:47 +02:00
Victor Stinner 85f5a69ae1
bpo-39877: Refactor take_gil() function (GH-18885)
* Remove ceval parameter of take_gil(): get it from tstate.
* Move exit_thread_if_finalizing() call inside take_gil(). Replace
  exit_thread_if_finalizing() with tstate_must_exit(): the caller is
  now responsible to call PyThread_exit_thread().
* Move is_tstate_valid() assertion inside take_gil(). Remove
  is_tstate_valid(): inline code into take_gil().
* Move gil_created() assertion inside take_gil().
2020-03-09 22:12:04 +01:00
Victor Stinner 111e4ee52a
bpo-39877: Py_Initialize() pass tstate to PyEval_InitThreads() (GH-18884) 2020-03-09 21:24:14 +01:00
Victor Stinner 3225b9f973
bpo-39877: Remove useless PyEval_InitThreads() calls (GH-18883)
Py_Initialize() calls PyEval_InitThreads() since Python 3.7. It's no
longer needed to call it explicitly.
2020-03-09 20:56:57 +01:00
Serhiy Storchaka eebaa9bfc5
bpo-38249: Expand Py_UNREACHABLE() to __builtin_unreachable() in the release mode. (GH-16329)
Co-authored-by: Victor Stinner <vstinner@python.org>
2020-03-09 20:49:52 +02:00
idomic fc72ab6913
bpo-38691: importlib ignores PYTHONCASEOK if -E is used (GH-18627)
The importlib module now ignores the PYTHONCASEOK
environment variable when the -E or -I command line
options are being used.
2020-03-09 12:57:53 +01:00
Andy Lester 28ca43b7e3
closes bpo-39898: Remove unused arg from append_formattedvalue. (GH-18840) 2020-03-08 09:53:59 -07:00
Victor Stinner eb4e2ae2b8
bpo-39877: Fix PyEval_RestoreThread() for daemon threads (GH-18811)
* exit_thread_if_finalizing() does now access directly _PyRuntime
  variable, rather than using tstate->interp->runtime since tstate
  can be a dangling pointer after Py_Finalize() has been called.
* exit_thread_if_finalizing() is now called *before* calling
  take_gil(). _PyRuntime.finalizing is an atomic variable,
  we don't need to hold the GIL to access it.
* Add ensure_tstate_not_null() function to check that tstate is not
  NULL at runtime. Check tstate earlier. take_gil() does not longer
  check if tstate is NULL.

Cleanup:

* PyEval_RestoreThread() no longer saves/restores errno: it's already
  done inside take_gil().
* PyEval_AcquireLock(), PyEval_AcquireThread(),
  PyEval_RestoreThread() and _PyEval_EvalFrameDefault() now check if
  tstate is valid with the new is_tstate_valid() function which uses
  _PyMem_IsPtrFreed().
2020-03-08 11:57:45 +01:00
Brandt Bucher d5aa2e941c
bpo-39890: Don't mutate the AST when compiling starred assignments (GH-18833) 2020-03-08 03:44:18 +00:00
Andy Lester aa450a0364
closes bpo-39886: Remove unused arg from config_get_stdio_errors. (GH-18823) 2020-03-07 09:36:04 -08:00
Andy Lester ad0c775ea2
closes bpo-39878: Remove unused arguments from static functions. (GH-18822)
calc_number_widths -> PyObject *number

fill_number -> Py_ssize_t d_end
2020-03-07 09:29:10 -08:00
Victor Stinner 9e5d30cc99
bpo-39882: Py_FatalError() logs the function name (GH-18819)
The Py_FatalError() function is replaced with a macro which logs
automatically the name of the current function, unless the
Py_LIMITED_API macro is defined.

Changes:

* Add _Py_FatalErrorFunc() function.
* Remove the function name from the message of Py_FatalError() calls
  which included the function name.
* Update tests.
2020-03-07 00:54:20 +01:00
Victor Stinner 7b3c252dc7
bpo-39877: _PyRuntimeState.finalizing becomes atomic (GH-18816)
Convert _PyRuntimeState.finalizing field to an atomic variable:

* Rename it to _finalizing
* Change its type to _Py_atomic_address
* Add _PyRuntimeState_GetFinalizing() and _PyRuntimeState_SetFinalizing()
  functions
* Remove _Py_CURRENTLY_FINALIZING() function: replace it with testing
  directly _PyRuntimeState_GetFinalizing() value

Convert _PyRuntimeState_GetThreadState() to static inline function.
2020-03-07 00:24:23 +01:00
Andy Lester 557287075c
bpo-39573: Use Py_IS_TYPE() macro to check for types (GH-18809)
Co-authored-by: Victor Stinner <vstinner@python.org>
2020-03-06 23:53:17 +01:00
Andy Lester 9566842e89
closes bpo-39872: Remove unused args from symtable_exit_block and symtable_visit_annotations. (GH-18800) 2020-03-06 07:46:04 -08:00
Andy Lester da4d656e95
closes bpo-39870: Remove unused arg from sys_displayhook_unencodable. (GH-18796)
Also move int err to its innermost scope.
2020-03-05 20:34:36 -08:00
Victor Stinner efc28bbbee
Add a comment to _Py_RestoreSignals() (GH-18792)
subprocess _posix_spawn() should stay in sync with _Py_RestoreSignals().
2020-03-05 18:13:56 +01:00
Batuhan Taşkaya d82e469048
bpo-39639: Remove the AST "Suite" node and associated code (GH-18513)
The AST "Suite" node is no longer used and it can be removed from the ASDL definition and related structures (compiler, visitors, ...).

Co-Authored-By: Victor Stinner <vstinner@python.org>
Co-authored-by: Brett Cannon <54418+brettcannon@users.noreply.github.com>
Co-authored-by: Pablo Galindo <Pablogsal@gmail.com>
2020-03-04 16:16:46 +00:00
Andy Lester dffe4c0709
bpo-39573: Finish converting to new Py_IS_TYPE() macro (GH-18601) 2020-03-04 14:15:20 +01:00
Brandt Bucher be501ca241
bpo-39702: Relax grammar restrictions on decorators (PEP 614) (GH-18570) 2020-03-03 14:25:44 -08:00
Serhiy Storchaka ae75a29435
bpo-39831: Remove outdated comment. (GH-18764) 2020-03-03 19:43:29 +02:00
Armin Rigo 6daa37fd42
bpo-38091: Import deadlock detection causes deadlock (GH-17518)
Automerge-Triggered-By: @brettcannon
2020-03-02 17:37:25 -08:00
Stefan Krah b3b9ade4a3
bpo-39776: Lock ++interp->tstate_next_unique_id. (GH-18746) (#18746)
- Threads created by PyGILState_Ensure() could have a duplicate tstate->id.
2020-03-02 21:22:36 +01:00
Serhiy Storchaka 2d2f85517f
bpo-39831: Fix a reference leak in PyErr_WarnEx(). (GH-18750) 2020-03-02 22:05:08 +02:00
Victor Stinner 66b7973c1b
bpo-39796: Fix _warnings module initialization (GH-18739)
* Add _PyWarnings_InitState() which only initializes the _warnings
  module state (tstate->interp->warnings) without creating a module
  object
* Py_InitializeFromConfig() now calls _PyWarnings_InitState() instead
  of _PyWarnings_Init()
* Rename also private functions of _warnings.c to avoid confusion
  between the public C API and the private C API.
2020-03-02 15:02:18 +01:00
Serhiy Storchaka 28d0bcac8b
bpo-38913: Fix segfault in Py_BuildValue("(s#O)", ...) if entered with exception raised. (GH-18656) 2020-03-02 08:42:39 +02:00
Andy Lester 3fe9117779
closes bpo-39803: Remove unused str from _PyLong_FormatAdvancedWriter. (GH-18709) 2020-03-01 13:26:43 -08:00
Batuhan Taşkaya 185903de12
bpo-39520: Fix un-parsing of ext slices with no dimensions (GH-18304) 2020-03-01 20:07:22 +00:00