Commit Graph

354 Commits

Author SHA1 Message Date
An Long 29c117202e
bpo-40834: Fix truncate when sending str object with channel (GH-20555) 2020-06-13 05:26:01 -07:00
Victor Stinner 26881c8fae
PyOS_AfterFork_Child() uses PyStatus (GH-20596)
PyOS_AfterFork_Child() helper functions now return a PyStatus:
PyOS_AfterFork_Child() is now responsible to handle errors.

* Move _PySignal_AfterFork() to the internal C API
* Add #ifdef HAVE_FORK on _PyGILState_Reinit(), _PySignal_AfterFork()
  and _PyInterpreterState_DeleteExceptMain().
2020-06-02 15:51:37 +02:00
Victor Stinner 3026cad59b
bpo-40826: Add _Py_EnsureTstateNotNULL() macro (GH-20571)
Add _Py_EnsureTstateNotNULL(tstate) macro: call Py_FatalError() if
tstate is NULL, the error message contains the current function name.
2020-06-01 16:02:40 +02:00
Victor Stinner e838a9324c
bpo-40522: _PyThreadState_Swap() sets autoTSSkey (GH-19939)
In the experimental isolated subinterpreters build mode,
_PyThreadState_GET() gets the autoTSSkey variable and
_PyThreadState_Swap() sets the autoTSSkey variable.

* Add _PyThreadState_GetTSS()
* _PyRuntimeState_GetThreadState() and _PyThreadState_GET()
  return _PyThreadState_GetTSS()
* PyEval_SaveThread() sets the autoTSSkey variable to current Python
  thread state rather than NULL.
* eval_frame_handle_pending() doesn't check that
  _PyThreadState_Swap() result is NULL.
* _PyThreadState_Swap() gets the current Python thread state with
  _PyThreadState_GetTSS() rather than
  _PyRuntimeGILState_GetThreadState().
* PyGILState_Ensure() no longer checks _PyEval_ThreadsInitialized()
  since it cannot access the current interpreter.
2020-05-05 19:56:48 +02:00
Victor Stinner 4386b9045e
bpo-40429: PyThreadState_GetFrame() returns a strong ref (GH-19781)
The PyThreadState_GetFrame() function now returns a strong reference
to the frame.
2020-04-29 03:01:43 +02:00
Victor Stinner 1a1bd2e238
bpo-40302: Replace PY_INT64_T with int64_t (GH-19573)
* Replace PY_INT64_T with int64_t
* Replace PY_UINT32_T with uint32_t
* Replace PY_UINT64_T with uint64_t

sha3module.c no longer checks if PY_UINT64_T is defined since it's
always defined and uint64_t is always available on platforms
supported by Python.
2020-04-17 19:13:06 +02:00
Victor Stinner d9ea5cae1d
bpo-40268: Remove unused pycore_pymem.h includes (GH-19531) 2020-04-15 02:57:50 +02:00
Dong-hee Na 62f75fe3dd
bpo-40232: Update PyOS_AfterFork_Child() to use _PyThread_at_fork_reinit() (GH-19450) 2020-04-15 01:16:24 +09:00
Victor Stinner e5014be049
bpo-40268: Remove a few pycore_pystate.h includes (GH-19510) 2020-04-14 17:52:15 +02:00
Victor Stinner 81a7be3fa2
bpo-40268: Rename _PyInterpreterState_GET_UNSAFE() (GH-19509)
Rename _PyInterpreterState_GET_UNSAFE() to _PyInterpreterState_GET()
for consistency with _PyThreadState_GET() and to have a shorter name
(help to fit into 80 columns).

Add also "assert(tstate != NULL);" to the function.
2020-04-14 15:14:01 +02:00
Victor Stinner 1c4cbdf94d
bpo-40268: Add pycore_runtime.h header file (GH-19493)
Move PyRuntimeState from pycore_pystate.h to pycore_runtime.h.

Remove _PyGILState_check_enabled macro: access directly
_PyRuntime.gilstate.check_enabled.
2020-04-13 11:45:21 +02:00
Victor Stinner da7933ecc3
bpo-40268: Add _PyInterpreterState_GetConfig() (GH-19492)
Don't access PyInterpreterState.config member directly anymore, but
use new functions:

* _PyInterpreterState_GetConfig()
* _PyInterpreterState_SetConfig()
* _Py_GetConfig()
2020-04-13 03:04:28 +02:00
Victor Stinner dda5d6e071
bpo-40226: PyInterpreterState_Delete() deletes pending calls (GH-19436)
PyInterpreterState_New() is now responsible to create pending calls,
PyInterpreterState_Delete() now deletes pending calls.

* Rename _PyEval_InitThreads() to _PyEval_InitGIL() and rename
  _PyEval_InitGIL() to _PyEval_FiniGIL().
* _PyEval_InitState() and PyEval_FiniState() now create and delete
  pending calls. _PyEval_InitState() now returns -1 on memory
  allocation failure.
* Add init_interp_create_gil() helper function: code shared by
  Py_NewInterpreter() and Py_InitializeFromConfig().
* init_interp_create_gil() now also calls _PyEval_FiniGIL(),
  _PyEval_InitGIL() and _PyGILState_Init() in subinterpreters, but
  these functions now do nothing when called from a subinterpreter.
2020-04-08 17:54:59 +02:00
Victor Stinner 71a3522ef8
bpo-38644: Make tstate more explicit inside pystate.c (GH-19182)
Fix PyInterpreterState_New(): Don't call PyErr_SetString() when there
is no current Python thread state (if tstate is NULL).
2020-03-26 22:46:14 +01:00
Victor Stinner 5c3cda0d1a
bpo-39947: Add PyThreadState_GetID() function (GH-19163)
Add PyThreadState_GetID() function: get the unique identifier of a
Python thread state.
2020-03-25 21:23:53 +01:00
Victor Stinner 0e427c6d15
bpo-39947: Add _PyThreadState_GetDict() function (GH-19160) 2020-03-25 21:22:55 +01:00
Victor Stinner 87d3b9db4a
bpo-39882: Add _Py_FatalErrorFormat() function (GH-19157) 2020-03-25 19:27:36 +01:00
Andy Lester 7668a8bc93
Use calloc-based functions, not malloc. (GH-19152) 2020-03-24 23:26:44 -05:00
Victor Stinner 5804f878e7
bpo-20526: Fix PyThreadState_Clear(): don't decref frame (GH-19120)
PyThreadState.frame is a borrowed reference, not a strong reference:
PyThreadState_Clear() must not call Py_CLEAR(tstate->frame).

Remove test_threading.test_warnings_at_exit(): we cannot warranty
that the Python thread state of daemon threads is cleared in a
reliable way during Python shutdown.
2020-03-24 16:32:26 +01:00
Victor Stinner 6723e933c4
bpo-39946: Remove _PyThreadState_GetFrame (GH-19094)
Remove _PyRuntime.getframe hook and remove _PyThreadState_GetFrame
macro which was an alias to _PyRuntime.getframe. They were only
exposed by the internal C API. Remove also PyThreadFrameGetter type.
2020-03-20 17:46:56 +01:00
Victor Stinner fd1e1a18fa
bpo-39947: Add PyThreadState_GetFrame() function (GH-19092)
Add PyThreadState_GetFrame() function: get the current frame
of a Python thread state.
2020-03-20 15:51:45 +01:00
Victor Stinner d2a8e5b42c
bpo-40010: COMPUTE_EVAL_BREAKER() checks for subinterpreter (GH-19087)
COMPUTE_EVAL_BREAKER() now also checks if the Python thread state
belongs to the main interpreter. Don't break the evaluation loop if
there are pending signals but the Python thread state it belongs to a
subinterpeter.

* Add _Py_IsMainThread() function.
* Add _Py_ThreadCanHandleSignals() function.
2020-03-20 13:38:58 +01:00
Victor Stinner 56bfdebfb1
bpo-39984: Pass tstate to _PyEval_SignalAsyncExc() (GH-19049)
_PyEval_SignalAsyncExc() and _PyEval_FiniThreads() now expect tstate,
instead of ceval.
2020-03-18 09:26:25 +01:00
Victor Stinner 23ef89db7a
bpo-39984: _PyThreadState_DeleteCurrent() takes tstate (GH-19051)
* _PyThreadState_DeleteCurrent() now takes tstate rather than
  runtime.
* Add ensure_tstate_not_null() helper to pystate.c.
* Add _PyEval_ReleaseLock() function.
* _PyThreadState_DeleteCurrent() now calls
  _PyEval_ReleaseLock(tstate) and frees PyThreadState memory after
  this call, not before.
* PyGILState_Release(): rename "tcur" variable to "tstate".
2020-03-18 02:26:04 +01:00
Victor Stinner dab8423d22
bpo-39984: Add PyInterpreterState.ceval (GH-19047)
subinterpreters: Move _PyRuntimeState.ceval.tracing_possible to
PyInterpreterState.ceval.tracing_possible: each interpreter now has
its own variable.

Changes:

* Add _ceval_state structure.
* Add PyInterpreterState.ceval field.
* _PyEval_EvalFrameDefault(): add ceval2 variable (struct _ceval_state*).
* Rename _PyEval_Initialize() to _PyEval_InitRuntimeState().
* Add _PyEval_InitState().
* Don't export internal _Py_FinishPendingCalls() and
  _PyEval_FiniThreads() functions anymore.
2020-03-17 18:56:44 +01:00
Victor Stinner 8fb02b6e19
bpo-39947: Add PyThreadState_GetInterpreter() (GH-18981)
Add PyThreadState_GetInterpreter(tstate): get the interpreter of a
Python thread state.
2020-03-13 23:38:08 +01:00
Victor Stinner be79373a78
bpo-39947: Add PyInterpreterState_Get() function (GH-18979)
* Rename _PyInterpreterState_Get() to PyInterpreterState_Get() and
  move it the limited C API.
* Add _PyInterpreterState_Get() alias to PyInterpreterState_Get() for
  backward compatibility with Python 3.8.
2020-03-13 18:15:33 +01:00
Victor Stinner 0b72b23fb0
bpo-38500: Add _PyInterpreterState_SetEvalFrameFunc() (GH-17340)
PyInterpreterState.eval_frame function now requires a tstate (Python
thread state) parameter.

Add private functions to the C API to get and set the frame
evaluation function:

* Add tstate parameter to _PyFrameEvalFunction function type.
* Add _PyInterpreterState_GetEvalFrameFunc() and
  _PyInterpreterState_SetEvalFrameFunc() functions.
* Add tstate parameter to _PyEval_EvalFrameDefault().
2020-03-12 23:18:39 +01:00
Victor Stinner 4e53abb0f4
bpo-38631: _PyGILState_Init() returns PyStatus (GH-18908)
_PyGILState_Init() now returns PyStatus rather than calling
Py_FatalError() on failure.
2020-03-10 23:49:16 +01:00
Victor Stinner 175a704abf
bpo-39877: PyGILState_Ensure() don't call PyEval_InitThreads() (GH-18891)
PyGILState_Ensure() doesn't call PyEval_InitThreads() anymore when a
new Python thread state is created. The GIL is created by
Py_Initialize() since Python 3.7, it's not needed to call
PyEval_InitThreads() explicitly.

Add an assertion to ensure that the GIL is already created.
2020-03-10 00:37:48 +01:00
Victor Stinner 9ad58acbe8
bpo-19466: Py_Finalize() clears daemon threads earlier (GH-18848)
Clear the frames of daemon threads earlier during the Python shutdown to
call objects destructors. So "unclosed file" resource warnings are now
emitted for daemon threads in a more reliable way.

Cleanup _PyThreadState_DeleteExcept() code: rename "garbage" to
"list".
2020-03-09 23:37:49 +01:00
Victor Stinner 9e5d30cc99
bpo-39882: Py_FatalError() logs the function name (GH-18819)
The Py_FatalError() function is replaced with a macro which logs
automatically the name of the current function, unless the
Py_LIMITED_API macro is defined.

Changes:

* Add _Py_FatalErrorFunc() function.
* Remove the function name from the message of Py_FatalError() calls
  which included the function name.
* Update tests.
2020-03-07 00:54:20 +01:00
Victor Stinner 7b3c252dc7
bpo-39877: _PyRuntimeState.finalizing becomes atomic (GH-18816)
Convert _PyRuntimeState.finalizing field to an atomic variable:

* Rename it to _finalizing
* Change its type to _Py_atomic_address
* Add _PyRuntimeState_GetFinalizing() and _PyRuntimeState_SetFinalizing()
  functions
* Remove _Py_CURRENTLY_FINALIZING() function: replace it with testing
  directly _PyRuntimeState_GetFinalizing() value

Convert _PyRuntimeState_GetThreadState() to static inline function.
2020-03-07 00:24:23 +01:00
Stefan Krah b3b9ade4a3
bpo-39776: Lock ++interp->tstate_next_unique_id. (GH-18746) (#18746)
- Threads created by PyGILState_Ensure() could have a duplicate tstate->id.
2020-03-02 21:22:36 +01:00
Victor Stinner 4d96b4635a
bpo-39511: PyThreadState_Clear() calls on_delete (GH-18296)
PyThreadState.on_delete is a callback used to notify Python when a
thread completes. _thread._set_sentinel() function creates a lock
which is released when the thread completes. It sets on_delete
callback to the internal release_sentinel() function. This lock is
known as Threading._tstate_lock in the threading module.

The release_sentinel() function uses the Python C API. The problem is
that on_delete is called late in the Python finalization, when the C
API is no longer fully working.

The PyThreadState_Clear() function now calls the
PyThreadState.on_delete callback. Previously, that happened in
PyThreadState_Delete().

The release_sentinel() function is now called when the C API is still
fully working.
2020-02-01 02:30:25 +01:00
Victor Stinner 82c83bd907
bpo-38858: _PyImport_FixupExtensionObject() handles subinterpreters (GH-17350)
If _PyImport_FixupExtensionObject() is called from a subinterpreter,
leave extensions unchanged and don't copy the module dictionary
into def->m_base.m_copy.
2019-11-22 18:52:27 +01:00
Victor Stinner fff7bbfdb6
bpo-38858: Add _Py_IsMainInterpreter(tstate) (GH-17293) 2019-11-20 17:34:39 +01:00
Victor Stinner 7247407c35
bpo-36854: Move _PyRuntimeState.gc to PyInterpreterState (GH-17287)
* Rename _PyGC_InitializeRuntime() to _PyGC_InitState()
* finalize_interp_clear() now also calls _PyGC_Fini() in
  subinterpreters (clear the GC state).
2019-11-20 12:25:50 +01:00
Victor Stinner 9da7430675
bpo-36854: Clear the current thread later (GH-17279)
Clear the current thread later in the Python finalization.

* The PyInterpreterState_Delete() function is now responsible
  to call PyThreadState_Swap(NULL).
* The tstate_delete_common() function is now responsible to clear the
  "autoTSSKey" thread local storage and it only clears it once the
  thread state is fully cleared. It allows to still get the current
  thread from TSS in tstate_delete_common().
2019-11-20 11:17:17 +01:00
Victor Stinner 7eee5beaf8
bpo-38858: Factorize Py_EndInterpreter() code (GH-17273)
* Factorize code in common between Py_FinalizeEx() and
  Py_EndInterpreter().
* Py_EndInterpreter() now also calls _PyWarnings_Fini().
* Call _PyExc_Fini() and _PyGC_Fini() later in the finalization.
2019-11-20 10:38:34 +01:00
Victor Stinner 01b1cc12e7
bpo-36710: Add PyInterpreterState.runtime field (GH-17270)
Add PyInterpreterState.runtime field: reference to the _PyRuntime
global variable. This field exists to not have to pass runtime in
addition to tstate to a function.  Get runtime from tstate:
tstate->interp->runtime.

Remove "_PyRuntimeState *runtime" parameter from functions already
taking a "PyThreadState *tstate" parameter.

_PyGC_Init() first parameter becomes "PyThreadState *tstate".
2019-11-20 02:27:56 +01:00
Victor Stinner 444b39bb64
bpo-38631: Avoid Py_FatalError() in handle_legacy_finalizers() (GH-17266)
* Rename _PyGC_Initialize() to _PyGC_InitializeRuntime()
* Add _PyGC_Init(): initialize _PyRuntime.gc.garbage list
* Call _PyGC_Init() before _PyTypes_Init()
2019-11-20 01:18:11 +01:00
Joannah Nanjekye 8855e47d09 bpo-38266: Revert bpo-37878: Make PyThreadState_DeleteCurrent() Internal (GH-16558)
Revert the removal of PyThreadState_DeleteCurrent() with documentation.
2019-10-04 13:35:42 +02:00
Victor Stinner 8462a4936b
bpo-38304: PyConfig_InitPythonConfig() cannot fail anymore (GH-16509)
PyConfig_InitPythonConfig() and PyConfig_InitIsolatedConfig() no
longer return PyStatus: they cannot fail anymore.
2019-10-01 12:06:16 +02:00
Victor Stinner 3c30a76f3d
bpo-38304: Remove PyConfig.struct_size (GH-16500) (GH-16508)
For now, we'll rely on the fact that the config structures aren't covered by the stable ABI.

We may revisit this in the future if we further explore the idea of offering a stable embedding API.

(cherry picked from commit bdace21b76)
2019-10-01 10:56:37 +02:00
Victor Stinner 441b10cf28
bpo-38304: Add PyConfig.struct_size (GH-16451)
Add a new struct_size field to PyPreConfig and PyConfig structures to
allow to modify these structures in the future without breaking the
backward compatibility.

* Replace private _config_version field with public struct_size field
  in PyPreConfig and PyConfig.
* Public PyPreConfig_InitIsolatedConfig() and
  PyPreConfig_InitPythonConfig()
  return type becomes PyStatus, instead of void.
* Internal _PyConfig_InitCompatConfig(),
  _PyPreConfig_InitCompatConfig(), _PyPreConfig_InitFromConfig(),
  _PyPreConfig_InitFromPreConfig() return type becomes PyStatus,
  instead of void.
* Remove _Py_CONFIG_VERSION
* Update the Initialization Configuration documentation.
2019-09-28 04:28:35 +02:00
Benjamin Peterson 39de95b746
closes bpo-38124: Fix bounds check in PyState_AddModule. (GH-16007)
The >=, checking whether a module index was in already in the module-by-index list, needed to be strict.

Also, fold nested ifs into one and fix some bad spacing.
2019-09-12 00:43:22 +01:00
Joannah Nanjekye 2bc43cdc01 bpo-37878: Remove PyThreadState_DeleteCurrent() function (GH-15315)
* Rename PyThreadState_DeleteCurrent()
  to _PyThreadState_DeleteCurrent()
* Move it to the internal C API

Co-Authored-By: Carol Willing <carolcode@willingconsulting.com>
2019-09-05 18:06:49 +02:00
Min ho Kim 39d87b5471 Fix typos mostly in comments, docs and test names (GH-15209) 2019-08-30 16:21:19 -04:00
Min ho Kim c4cacc8c5e Fix typos in comments, docs and test names (#15018)
* Fix typos in comments, docs and test names

* Update test_pyparse.py

account for change in string length

* Apply suggestion: splitable -> splittable

Co-Authored-By: Terry Jan Reedy <tjreedy@udel.edu>

* Apply suggestion: splitable -> splittable

Co-Authored-By: Terry Jan Reedy <tjreedy@udel.edu>

* Apply suggestion: Dealloccte -> Deallocate

Co-Authored-By: Terry Jan Reedy <tjreedy@udel.edu>

* Update posixmodule checksum.

* Reverse idlelib changes.
2019-07-30 18:16:13 -04:00
Victor Stinner 36456df138
bpo-37392: Remove sys.setcheckinterval() (GH-14355)
Remove sys.getcheckinterval() and sys.setcheckinterval() functions.
They were deprecated since Python 3.2. Use sys.getswitchinterval()
and sys.setswitchinterval() instead.

Remove also check_interval field of the PyInterpreterState structure.
2019-06-25 03:01:08 +02:00
Victor Stinner b45d259bdd
bpo-36710: Use tstate in pylifecycle.c (GH-14249)
In pylifecycle.c: pass tstate argument, rather than interp argument,
to functions.
2019-06-20 00:05:23 +02:00
Victor Stinner 0a28f8d379
bpo-36710: Add tstate parameter in import.c (GH-14218)
* Add 'tstate' parameter to many internal import.c functions.
* _PyImportZip_Init() now gets 'tstate' parameter rather than
  'interp'.
* Add 'interp' parameter to _PyState_ClearModules() and rename it
  to _PyInterpreterState_ClearModules().
* Move private _PyImport_FindBuiltin() to the internal C API; add
  'tstate' parameter to it.
* Remove private _PyImport_AddModuleObject() from the C API:
  use public PyImport_AddModuleObject() instead.
* Remove private _PyImport_FindExtensionObjectEx() from the C API:
  use private _PyImport_FindExtensionObject() instead.
2019-06-19 02:54:39 +02:00
Victor Stinner 0fd2c300c2
Revert "bpo-36818: Add PyInterpreterState.runtime field. (gh-13129)" (GH-13795)
This reverts commit 396e0a8d9d.
2019-06-04 03:15:09 +02:00
Victor Stinner e225bebc14
Revert "bpo-33608: Factor out a private, per-interpreter _Py_AddPendingCall(). (gh-13714)" (GH-13780)
This reverts commit 6a150bcaeb.
2019-06-03 18:14:24 +02:00
Eric Snow 6a150bcaeb
bpo-33608: Factor out a private, per-interpreter _Py_AddPendingCall(). (gh-13714) 2019-06-01 15:39:46 -06:00
Eric Snow 396e0a8d9d
bpo-36818: Add PyInterpreterState.runtime field. (gh-13129)
https://bugs.python.org/issue36818
2019-05-31 21:16:47 -06:00
Matthias Bussonnier 3880f263d2 bpo-36933: Remove sys.set_coroutine_wrapper (marked for removal in 3.8) (GH-13577)
It has been documented as deprecated and to be removed in 3.8; 

From a comment on another thread – which I can't find ; leave get_coro_wrapper() for now, but always return `None`.


https://bugs.python.org/issue36933
2019-05-28 00:10:59 -07:00
Victor Stinner 331a6a56e9
bpo-36763: Implement the PEP 587 (GH-13592)
* Add a whole new documentation page:
  "Python Initialization Configuration"
* PyWideStringList_Append() return type is now PyStatus,
  instead of int
* PyInterpreterState_New() now calls PyConfig_Clear() if
  PyConfig_InitPythonConfig() fails.
* Rename files:

  * Python/coreconfig.c => Python/initconfig.c
  * Include/cpython/coreconfig.h => Include/cpython/initconfig.h
  * Include/internal/: pycore_coreconfig.h => pycore_initconfig.h

* Rename structures

  * _PyCoreConfig => PyConfig
  * _PyPreConfig => PyPreConfig
  * _PyInitError => PyStatus
  * _PyWstrList => PyWideStringList

* Rename PyConfig fields:

  * use_module_search_paths => module_search_paths_set
  * module_search_path_env => pythonpath_env

* Rename PyStatus field: _func => func
* PyInterpreterState: rename core_config field to config
* Rename macros and functions:

  * _PyCoreConfig_SetArgv() => PyConfig_SetBytesArgv()
  * _PyCoreConfig_SetWideArgv() => PyConfig_SetArgv()
  * _PyCoreConfig_DecodeLocale() => PyConfig_SetBytesString()
  * _PyInitError_Failed() => PyStatus_Exception()
  * _Py_INIT_ERROR_TYPE_xxx enums => _PyStatus_TYPE_xxx
  * _Py_UnixMain() => Py_BytesMain()
  * _Py_ExitInitError() => Py_ExitStatusException()
  * _Py_PreInitializeFromArgs() => Py_PreInitializeFromBytesArgs()
  * _Py_PreInitializeFromWideArgs() => Py_PreInitializeFromArgs()
  * _Py_PreInitialize() => Py_PreInitialize()
  * _Py_RunMain() => Py_RunMain()
  * _Py_InitializeFromConfig() => Py_InitializeFromConfig()
  * _Py_INIT_XXX() => _PyStatus_XXX()
  * _Py_INIT_FAILED() => _PyStatus_EXCEPTION()

* Rename 'err' PyStatus variables to 'status'
* Convert RUN_CODE() macro to config_run_code() static inline function
* Remove functions:

  * _Py_InitializeFromArgs()
  * _Py_InitializeFromWideArgs()
  * _PyInterpreterState_GetCoreConfig()
2019-05-27 16:39:22 +02:00
Steve Dower b82e17e626
bpo-36842: Implement PEP 578 (GH-12613)
Adds sys.audit, sys.addaudithook, io.open_code, and associated C APIs.
2019-05-23 08:45:22 -07:00
Victor Stinner 022be02dcf
bpo-36763: Add _PyPreConfig._config_init (GH-13481)
* _PyPreConfig_GetGlobalConfig() and  _PyCoreConfig_GetGlobalConfig()
  now do nothing if the configuration was not initialized with
  _PyPreConfig_InitCompatConfig() and _PyCoreConfig_InitCompatConfig()
* Remove utf8_mode=-2 special case: use utf8_mode=-1 instead.
* Fix _PyPreConfig_InitPythonConfig():

  * isolated = 0 instead of -1
  * use_environment = 1 instead of -1

* Rename _PyConfig_INIT to  _PyConfig_INIT_COMPAT
* Rename _PyPreConfig_Init() to _PyPreConfig_InitCompatConfig()
* Rename _PyCoreConfig_Init() to _PyCoreConfig_InitCompatConfig()
* PyInterpreterState_New() now uses _PyCoreConfig_InitPythonConfig()
  as default configuration, but it's very quickly overriden anyway.
* _freeze_importlib.c uses _PyCoreConfig_SetString() to set
  program_name.
* Cleanup preconfig_init_utf8_mode(): cmdline is always non-NULL.
2019-05-22 23:58:50 +02:00
Victor Stinner cab5d0741e
bpo-36763: Add _PyCoreConfig_InitPythonConfig() (GH-13388)
Add new functions to get the Python interpreter behavior:

* _PyPreConfig_InitPythonConfig()
* _PyCoreConfig_InitPythonConfig()

Add new functions to get an isolated configuration:

* _PyPreConfig_InitIsolatedConfig()
* _PyCoreConfig_InitIsolatedConfig()

Replace _PyPreConfig_INIT and _PyCoreConfig_INIT with new functions
_PyPreConfig_Init() and _PyCoreConfig_Init().

_PyCoreConfig: set configure_c_stdio and parse_argv to 0 by default
to behave as Python 3.6 in the default configuration.

_PyCoreConfig_Read() no longer sets coerce_c_locale_warn to 1 if it's
equal to 0. coerce_c_locale_warn must now be set to -1 (ex: using
_PyCoreConfig_InitPythonConfig()) to enable C locale coercion
warning.

Add unit tests for _PyCoreConfig_InitPythonConfig()
and _PyCoreConfig_InitIsolatedConfig().

Changes:

* Rename _PyCoreConfig_GetCoreConfig() to _PyPreConfig_GetCoreConfig()
* Fix core_read_precmdline(): handle parse_argv=0
* Fix _Py_PreInitializeFromCoreConfig(): pass coreconfig.argv
  to _Py_PreInitializeFromPyArgv(), except if parse_argv=0
2019-05-17 19:01:14 +02:00
Victor Stinner b16b4e4592
bpo-36763: Add PyMemAllocatorName (GH-13387)
* Add PyMemAllocatorName enum
* _PyPreConfig.allocator type becomes PyMemAllocatorName, instead of
  char*
* Remove _PyPreConfig_Clear()
* Add _PyMem_GetAllocatorName()
* Rename _PyMem_GetAllocatorsName() to
  _PyMem_GetCurrentAllocatorName()
* Remove _PyPreConfig_SetAllocator(): just call
  _PyMem_SetupAllocators() directly, we don't have do reallocate the
  configuration with the new allocator anymore!
* _PyPreConfig_Write() parameter becomes const, as it should be in
  the first place!
2019-05-17 15:20:52 +02:00
Victor Stinner 09532feeec
bpo-36710: Add 'ceval' local variable to ceval.c (GH-12934)
Add "struct _ceval_runtime_state *ceval = &_PyRuntime.ceval;" local
variables to function to better highlight the dependency on the
global variable _PyRuntime and to point directly to _PyRuntime.ceval
field rather than on the larger _PyRuntime.

Changes:

* Add _PyRuntimeState_GetThreadState(runtime) macro.
* Add _PyEval_AddPendingCall(ceval, ...) and
  _PyThreadState_Swap(gilstate, ...) functions.
* _PyThreadState_GET() macro now calls
  _PyRuntimeState_GetThreadState() using &_PyRuntime.
* Add 'ceval' parameter to COMPUTE_EVAL_BREAKER(),
  SIGNAL_PENDING_SIGNALS(), _PyEval_SignalAsyncExc(),
  _PyEval_SignalReceived() and _PyEval_FiniThreads() macros and
  functions.
* Add 'tstate' parameter to call_function(), do_call_core() and
  do_raise().
* Add 'runtime' parameter to _Py_CURRENTLY_FINALIZING(),
  _Py_FinishPendingCalls() and _PyThreadState_DeleteExcept()
  macros and functions.
* Declare 'runtime', 'tstate', 'ceval' and 'eval_breaker' variables
  as constant.
2019-05-10 23:39:09 +02:00
Eric Snow 86ea58149c
bpo-36737: Use the module state C-API for warnings. (gh-13159) 2019-05-10 13:29:55 -04:00
Victor Stinner 99e69d44f4
bpo-36710: Fix compiler warning on PyThreadState_Delete() (GH-12962)
_PyThreadState_Delete() has no return value.
2019-04-26 05:48:51 +02:00
Victor Stinner 10c8e6af91
bpo-36710: Add runtime variable in pystate.c (GH-12956)
Add 'gilstate', 'runtime' or 'xidregistry' parameter to many
functions on pystate.c to avoid lying on _PyRuntime global.
2019-04-26 01:53:18 +02:00
Victor Stinner 43125224d6
bpo-36710: Add runtime variable to Py_InitializeEx() (GH-12939)
Py_InitializeEx() now uses a runtime variable passed to subfunctions,
rather than working directly on the global variable _PyRuntime.

Add 'runtime' parameter to _PyCoreConfig_Write(), _PySys_Create(),
_PySys_InitMain(), _PyGILState_Init(),
emit_stderr_warning_for_legacy_locale() and other subfunctions.
2019-04-24 18:23:53 +02:00
Victor Stinner 8e91c246e4
bpo-36710: Add runtime variable to Py_FinalizeEx() (GH-12937)
* Add a 'runtime' variable to Py_FinalizeEx() rather than working
  directly on the global variable _PyRuntime
* Add a 'runtime' parameter to _PyGC_Fini(), _PyGILState_Fini()
  and call_ll_exitfuncs()
2019-04-24 17:24:01 +02:00
Victor Stinner b930a2d2b1
bpo-36710: PyOS_AfterFork_Child() pass runtime parameter (GH-12936)
The PyOS_AfterFork_Child() function now pass a 'runtime' parameter to
subfunctions.

* Fix _PyRuntimeState_ReInitThreads(): use the correct memory allocator
* Add runtime parameter to _PyRuntimeState_ReInitThreads(),
  _PyGILState_Reinit() and _PyInterpreterState_DeleteExceptMain()
* Move _PyGILState_Reinit() to the internal C API.
2019-04-24 17:14:33 +02:00
Victor Stinner 8bb3230149
bpo-36710: Add runtime parameter to _PyThreadState_Init() (GH-12935)
* Add 'runtime' parameter to _PyThreadState_Init()
* Add 'gilstate' parameter to _PyGILState_NoteThreadState()
* Move _PyThreadState_Init() and _PyThreadState_DeleteExcept()
   to the internal C API.
2019-04-24 16:47:40 +02:00
Eric Snow b75b1a3504
bpo-33608: Revert "Factor out a private, per-interpreter _Py_AddPendingCall()." (gh-12806)
This reverts commit f13c5c8b94 (gh-12360).
2019-04-12 10:20:10 -06:00
Eric Snow f13c5c8b94
bpo-33608: Factor out a private, per-interpreter _Py_AddPendingCall(). (gh-12360)
This is effectively an un-revert of #11617 and #12024 (reverted in #12159). Portions of those were merged in other PRs (with lower risk) and this represents the remainder. Note that I found 3 different bugs in the original PRs and have fixed them here.
2019-04-12 09:18:16 -06:00
Victor Stinner 8b9dbc017a
bpo-36444: Remove _PyMainInterpreterConfig (GH-12571) 2019-03-27 01:36:16 +01:00
Victor Stinner 6d5ee973f0
bpo-36301: Add _PyRuntimeState.preconfig (GH-12506)
_PyPreConfig_Write() now writes the applied pre-configuration into
_PyRuntimeState.preconfig.
2019-03-23 12:05:43 +01:00
Stéphane Wirtel 943395fab9 bpo-36333: Fix leak _PyRuntimeState_Fini (GH-12400) 2019-03-19 11:51:32 +01:00
Eric Snow d2fdd1fedf
bpo-36124: Add PyInterpreterState.dict. (gh-12132) 2019-03-15 17:47:43 -06:00
Eric Snow c11183cdcf
bpo-36097: Use only public C-API in the_xxsubinterpreters module (adding as necessary). (gh-12359) 2019-03-15 16:35:46 -06:00
Eric Snow 8479a3426e
bpo-33608: Make sure locks in the runtime are properly re-created. (gh-12245) 2019-03-08 23:44:33 -07:00
Eric Snow 5be45a6105
bpo-33608: Minor cleanup related to pending calls. (gh-12247) 2019-03-08 22:47:07 -07:00
Victor Stinner 4d61e6e3b8
Revert: bpo-33608: Factor out a private, per-interpreter _Py_AddPendingCall(). (GH-11617) (GH-12159)
* Revert "bpo-36097: Use only public C-API in the_xxsubinterpreters module (adding as necessary). (#12003)"

This reverts commit bcfa450f21.

* Revert "bpo-33608: Simplify ceval's DISPATCH by hoisting eval_breaker ahead of time. (gh-12062)"

This reverts commit bda918bf65.

* Revert "bpo-33608: Use _Py_AddPendingCall() in _PyCrossInterpreterData_Release(). (gh-12024)"

This reverts commit b05b711a2c.

* Revert "bpo-33608: Factor out a private, per-interpreter _Py_AddPendingCall(). (GH-11617)"

This reverts commit ef4ac967e2.
2019-03-04 14:21:28 +01:00
Eric Snow bcfa450f21
bpo-36097: Use only public C-API in the_xxsubinterpreters module (adding as necessary). (#12003) 2019-03-01 16:50:31 -07:00
Eric Snow b05b711a2c
bpo-33608: Use _Py_AddPendingCall() in _PyCrossInterpreterData_Release(). (gh-12024) 2019-03-01 12:35:10 -07:00
Victor Stinner f684d83d86
bpo-36142: Exclude coreconfig.h from Py_LIMITED_API (GH-12111)
The whole coreconfig.h header is now excluded from Py_LIMITED_API.
Move functions definitions into a new internal pycore_coreconfig.h
header.

* Move Include/coreconfig.h to Include/cpython/coreconfig.h
* coreconfig.h header is now excluded from Py_LIMITED_API
* Move functions to pycore_coreconfig.h
2019-03-01 03:44:13 +01:00
Eric Snow ef4ac967e2
bpo-33608: Factor out a private, per-interpreter _Py_AddPendingCall(). (GH-11617)
This involves moving the global "pending calls" state to PyInterpreterState.

https://bugs.python.org/issue33608
2019-02-24 15:40:47 -08:00
Eric Snow be3b295838
bpo-35886: Make PyInterpreterState an opaque type in the public API. (GH-11731)
Move PyInterpreterState into the "internal" header files.
2019-02-23 11:35:52 -07:00
Stéphane Wirtel b5409dacc4 bpo-35993: Fix _PyInterpreterState_DeleteExceptMain() (GH-11852)
Fix a crash on fork when using subinterpreters.
2019-02-20 15:27:22 +01:00
Alexey Izbyshev 16f842da3c bpo-35972: _xxsubinterpreters: Fix potential integer truncation on 32-bit in channel_send() (gh-11822) 2019-02-12 09:06:43 -07:00
Michael Felt a909460a09 bpo-34569: Fix subinterpreter 32-bit ABI, pystate.c/_new_long_object() (gh-9127)
This fixes ShareableTypeTests.test_int() in Lib/test/test__xxsubinterpreters.py.
2019-01-11 11:17:03 -07:00
Zackery Spytz 2a893430c9 bpo-35414: Add a missing Py_INCREF(Py_None) in PyState_RemoveModule(). (GH-10914) 2018-12-05 09:14:00 +02:00
Victor Stinner 621cebe81b
bpo-35081: Rename internal headers (GH-10275)
Rename Include/internal/ headers:

* pycore_hash.h -> pycore_pyhash.h
* pycore_lifecycle.h -> pycore_pylifecycle.h
* pycore_mem.h -> pycore_pymem.h
* pycore_state.h -> pycore_pystate.h

Add missing headers to Makefile.pre.in and PCbuild:

* pycore_condvar.h.
* pycore_hamt.h
* pycore_pyhash.h
2018-11-12 16:53:38 +01:00
Victor Stinner 50b48572d9
bpo-35081: Add _PyThreadState_GET() internal macro (GH-10266)
If Py_BUILD_CORE is defined, the PyThreadState_GET() macro access
_PyRuntime which comes from the internal pycore_state.h header.
Public headers must not require internal headers.

Move PyThreadState_GET() and _PyInterpreterState_GET_UNSAFE() from
Include/pystate.h to Include/internal/pycore_state.h, and rename
PyThreadState_GET() to _PyThreadState_GET() there.

The PyThreadState_GET() macro of pystate.h is now redefined when
pycore_state.h is included, to use the fast _PyThreadState_GET().

Changes:

* Add _PyThreadState_GET() macro
* Replace "PyThreadState_GET()->interp" with
  _PyInterpreterState_GET_UNSAFE()
* Replace PyThreadState_GET() with _PyThreadState_GET() in internal C
  files (compiled with Py_BUILD_CORE defined), but keep
  PyThreadState_GET() in the public header files.
* _testcapimodule.c: replace PyThreadState_GET() with
  PyThreadState_Get(); the module is not compiled with Py_BUILD_CORE
  defined.
* pycore_state.h now requires Py_BUILD_CORE to be defined.
2018-11-01 01:51:40 +01:00
Victor Stinner 27e2d1f219
bpo-35081: Add pycore_ prefix to internal header files (GH-10263)
* Rename Include/internal/ header files:

  * pyatomic.h -> pycore_atomic.h
  * ceval.h -> pycore_ceval.h
  * condvar.h -> pycore_condvar.h
  * context.h -> pycore_context.h
  * pygetopt.h -> pycore_getopt.h
  * gil.h -> pycore_gil.h
  * hamt.h -> pycore_hamt.h
  * hash.h -> pycore_hash.h
  * mem.h -> pycore_mem.h
  * pystate.h -> pycore_state.h
  * warnings.h -> pycore_warnings.h

* PCbuild project, Makefile.pre.in, Modules/Setup: add the
  Include/internal/ directory to the search paths of header files.
* Update includes. For example, replace #include "internal/mem.h"
  with #include "pycore_mem.h".
2018-11-01 00:52:28 +01:00
Victor Stinner 2be00d987d
bpo-35081: Move Py_BUILD_CORE code to internal/mem.h (GH-10249)
* Add #include "internal/mem.h" to C files using
  _PyMem_SetDefaultAllocator().
* Include/internal/mem.h now requires Py_BUILD_CORE to be defined.
2018-10-31 20:19:24 +01:00
Victor Stinner 9204fb8623
bpo-35081: Cleanup pystate.c and pystate.h (GH-10240)
* Remove _PyThreadState_Current
* Replace GET_TSTATE() with PyThreadState_GET()
* Replace GET_INTERP_STATE() with _PyInterpreterState_GET_UNSAFE()
* Replace direct access to _PyThreadState_Current with
  PyThreadState_GET()
* Replace _PyThreadState_Current with
  _PyRuntime.gilstate.tstate_current
* Rename SET_TSTATE() to _PyThreadState_SET(), name more
  consistent with _PyThreadState_GET()
* Update outdated comments
2018-10-30 15:13:17 +01:00
Eric Snow 5903296045
bpo-34651: Only allow the main interpreter to fork. (gh-9279)
When os.fork() is called (on platforms that support it) all threads but the current one are destroyed in the child process. Consequently we must ensure that all but the associated interpreter are likewise destroyed. The main interpreter is critical for runtime operation, so we must ensure that fork only happens in the main interpreter.

https://bugs.python.org/issue34651
2018-09-14 14:17:20 -07:00
Pablo Galindo 95d630e221
bpo-34408: Prevent a null pointer dereference and resource leakage in `PyInterpreterState_New()` (GH-8767)
* A pointer in `PyInterpreterState_New()` could have been `NULL` when being dereferenced.

* Memory was leaked in `PyInterpreterState_New()` when taking some error-handling code path.
2018-08-31 22:49:29 +01:00
Victor Stinner caba55b3b7
bpo-34301: Add _PyInterpreterState_Get() helper function (GH-8592)
sys_setcheckinterval() now uses a local variable to parse arguments,
before writing into interp->check_interval.
2018-08-03 15:33:52 +02:00
Victor Stinner 53b7d4e402
bpo-34170: Add _PyCoreConfig.bytes_warning (GH-8447)
Add more fields to _PyCoreConfig:

* _check_hash_pycs_mode
* bytes_warning
* debug
* inspect
* interactive
* legacy_windows_fs_encoding
* legacy_windows_stdio
* optimization_level
* quiet
* unbuffered_stdio
* user_site_directory
* verbose
* write_bytecode

Changes:

* Remove pymain_get_global_config() and pymain_set_global_config()
  which became useless. These functions have been replaced by
  _PyCoreConfig_GetGlobalConfig() and
  _PyCoreConfig_SetGlobalConfig().
* sys.flags.dont_write_bytecode value is now restricted to 1 even if
  -B option is specified multiple times on the command line.
* PyThreadState_Clear() now uses the config from the current
  interpreter rather than using global Py_VerboseFlag
2018-07-25 01:37:05 +02:00
Eric Snow 63799136e6
bpo-33615: Re-enable a subinterpreter test. (gh-7251)
For bpo-32604 I added extra subinterpreter-related tests (see #6914), which caused a few buildbots to crash. This patch fixes the crash by ensuring that refcounts in channels are handled properly.
2018-06-01 18:45:20 -06:00
Victor Stinner b6dccf54fd
bpo-33612: Remove PyThreadState_Clear() assertion (#7069)
bpo-25612, bpo-33612: Remove an assertion from PyThreadState_Clear()
which failed at Python shutdown or on fork if a thread was running a
generator.
2018-05-23 18:00:55 +02:00
Eric Snow 6d2cd9036c
bpo-32604: Improve subinterpreter tests. (#6914)
Add more tests for subinterpreters. This patch also fixes a few small defects in the channel implementation.
2018-05-16 15:04:57 -04:00
Victor Stinner 5d92647102
bpo-33005: Fix _PyGILState_Reinit() (#6001)
Fix a crash on fork when using a custom memory allocator (ex: using
PYTHONMALLOC env var).

_PyGILState_Reinit() and _PyInterpreterState_Enable() now use the
default RAW memory allocator to allocate a new interpreters mutex on
fork.
2018-03-06 14:31:37 +01:00
Eric Snow f53d9f2778
bpo-32604: Swap threads only if the interpreter is different. (gh-5778)
The CPython runtime assumes that there is a one-to-one relationship (for a given interpreter) between PyThreadState and OS threads. Sending and receiving on a channel in the same interpreter was causing crashes because of this (specifically due to a check in PyThreadState_Swap()). The solution is to not switch threads if the interpreter is the same.
2018-02-20 16:30:17 -07:00
Eric Snow 4c6955e2b0
bpo-32604: Clean up created subinterpreters before runtime finalization. (gh-5709) 2018-02-16 18:53:40 -07:00
Eric Snow 4e9da0d163
bpo-32604: Fix memory leaks in the new _xxsubinterpreters module. (#5507) 2018-02-02 21:49:49 -07:00
Eric Snow 7f8bfc9b9a
bpo-32604: Expose the subinterpreters C-API in a "private" stdlib module. (gh-1748)
The module is primarily intended for internal use in the test suite.  Building the module under Windows will come in a follow-up PR.
2018-01-29 18:23:44 -07:00
Yury Selivanov f23746a934
bpo-32436: Implement PEP 567 (#5027) 2018-01-22 19:11:18 -05:00
Nathaniel J. Smith fc2f407829 bpo-32591: Add native coroutine origin tracking (#5250)
* Add coro.cr_origin and sys.set_coroutine_origin_tracking_depth
* Use coroutine origin information in the unawaited coroutine warning
* Stop using set_coroutine_wrapper in asyncio debug mode
* In BaseEventLoop.set_debug, enable debugging in the correct thread
2018-01-21 09:44:07 -05:00
Marcel Plch 776407fe89 bpo-31901: atexit callbacks should be run at subinterpreter shutdown (#4611)
Change atexit behavior and PEP-489 multiphase init support.
2017-12-20 11:17:58 +01:00
Victor Stinner da273412c4
bpo-32030: Add _PyCoreConfig_Copy() (#4874)
Each interpreter now has its core_config and main_config copy:

* Add _PyCoreConfig_Copy() and _PyMainInterpreterConfig_Copy()
* Move _PyCoreConfig_Read(), _PyCoreConfig_Clear() and
  _PyMainInterpreterConfig_Clear() from Python/pylifecycle.c to
  Modules/main.c
* Fix _Py_InitializeEx_Private(): call _PyCoreConfig_ReadEnv() before
  _Py_InitializeCore()
2017-12-15 01:46:02 +01:00
Victor Stinner b4d1e1f7c1
bpo-20891: Fix PyGILState_Ensure() (#4650)
When PyGILState_Ensure() is called in a non-Python thread before
PyEval_InitThreads(), only call PyEval_InitThreads() after calling
PyThreadState_New() to fix a crash.

Add an unit test in test_embed.
2017-11-30 22:05:00 +01:00
Victor Stinner 5d39e04290
bpo-32030: Rework memory allocators (#4625)
* Fix _PyMem_SetupAllocators("debug"): always restore allocators to
  the defaults, rather than only caling _PyMem_SetupDebugHooks().
* Add _PyMem_SetDefaultAllocator() helper to set the "default"
  allocator.
* Add _PyMem_GetAllocatorsName(): get the name of the allocators
* main() now uses debug hooks on memory allocators if Py_DEBUG is
  defined, rather than calling directly malloc()
* Document default memory allocators in C API documentation
* _Py_InitializeCore() now fails with a fatal user error if
  PYTHONMALLOC value is an unknown memory allocator, instead of
  failing with a fatal internal error.
* Add new tests on the PYTHONMALLOC environment variable
* Add support.with_pymalloc()
* Add the _testcapi.WITH_PYMALLOC constant and expose it as
   support.with_pymalloc().
* sysconfig.get_config_var('WITH_PYMALLOC') doesn't work on Windows, so
   replace it with support.with_pymalloc().
* pythoninfo: add _testcapi collector for pymem
2017-11-29 17:20:38 +01:00
Victor Stinner 9e87e7776f
bpo-32096: Remove obj and mem from _PyRuntime (#4532)
bpo-32096, bpo-30860:  Partially revert the commit
2ebc5ce42a8a9e047e790aefbf9a94811569b2b6:

* Move structures back from Include/internal/mem.h to
  Objects/obmalloc.c
* Remove _PyObject_Initialize() and _PyMem_Initialize()
* Remove Include/internal/pymalloc.h
* Add test_capi.test_pre_initialization_api():
   Make sure that it's possible to call Py_DecodeLocale(), and then call
   Py_SetProgramName() with the decoded string, before Py_Initialize().

PyMem_RawMalloc() and Py_DecodeLocale() can be called again before
_PyRuntimeState_Init().

Co-Authored-By: Eric Snow <ericsnowcurrently@gmail.com>
2017-11-24 12:09:24 +01:00
Victor Stinner d434110974
bpo-32030: Add _PyCoreConfig.module_search_path_env (#4504)
Changes:

* Py_Main() initializes _PyCoreConfig.module_search_path_env from
  the PYTHONPATH environment variable.
* PyInterpreterState_New() now initializes core_config and config
  fields
* Compute sys.path a little bit ealier in
  _Py_InitializeMainInterpreter() and new_interpreter()
* Add _Py_GetPathWithConfig() private function.
2017-11-23 00:12:09 +01:00
Victor Stinner ccb0442a33
bpo-32043: New "developer mode": "-X dev" option (#4413)
Add a new "developer mode": new "-X dev" command line option to
enable debug checks at runtime.

Changes:

* Add unit tests for -X dev
* test_cmd_line: replace test.support with support.
* Fix _PyRuntimeState_Fini(): Use the same memory allocator
   than _PyRuntimeState_Init().
* Fix _PyMem_GetDefaultRawAllocator()
2017-11-16 03:20:31 -08:00
Victor Stinner a7368ac636
bpo-32030: Enhance Py_Main() (#4412)
Parse more env vars in Py_Main():

* Add more options to _PyCoreConfig:

  * faulthandler
  * tracemalloc
  * importtime

* Move code to parse environment variables from _Py_InitializeCore()
  to Py_Main(). This change fixes a regression from Python 3.6:
  PYTHONUNBUFFERED is now read before calling pymain_init_stdio().
* _PyFaulthandler_Init() and _PyTraceMalloc_Init() now take an
  argument to decide if the module has to be enabled at startup.
* tracemalloc_start() is now responsible to check the maximum number
  of frames.

Other changes:

* Cleanup Py_Main():

  * Rename some pymain_xxx() subfunctions
  * Add pymain_run_python() subfunction

* Cleanup Py_NewInterpreter()
* _PyInterpreterState_Enable() now reports failure
* init_hash_secret() now considers pyurandom() failure as an "user
  error": don't fail with abort().
* pymain_optlist_append() and pymain_strdup() now sets err on memory
  allocation failure.
2017-11-15 18:11:45 -08:00
Victor Stinner f7e5b56c37
bpo-32030: Split Py_Main() into subfunctions (#4399)
* Don't use "Python runtime" anymore to parse command line options or
  to get environment variables: pymain_init() is now a strict
  separation.
* Use an error message rather than "crashing" directly with
  Py_FatalError(). Limit the number of calls to Py_FatalError(). It
  prepares the code to handle errors more nicely later.
* Warnings options (-W, PYTHONWARNINGS) and "XOptions" (-X) are now
  only added to the sys module once Python core is properly
  initialized.
* _PyMain is now the well identified owner of some important strings
  like: warnings options, XOptions, and the "program name". The
  program name string is now properly freed at exit.
  pymain_free() is now responsible to free the "command" string.
* Rename most methods in Modules/main.c to use a "pymain_" prefix to
  avoid conflits and ease debug.
* Replace _Py_CommandLineDetails_INIT with memset(0)
* Reorder a lot of code to fix the initialization ordering. For
  example, initializing standard streams now comes before parsing
  PYTHONWARNINGS.
* Py_Main() now handles errors when adding warnings options and
  XOptions.
* Add _PyMem_GetDefaultRawAllocator() private function.
* Cleanup _PyMem_Initialize(): remove useless global constants: move
  them into _PyMem_Initialize().
* Call _PyRuntime_Initialize() as soon as possible:
  _PyRuntime_Initialize() now returns an error message on failure.
* Add _PyInitError structure and following macros:

  * _Py_INIT_OK()
  * _Py_INIT_ERR(msg)
  * _Py_INIT_USER_ERR(msg): "user" error, don't abort() in that case
  * _Py_INIT_FAILED(err)
2017-11-15 15:48:08 -08:00
Serhiy Storchaka bdf4298ae2 Fix trailing whitespaces in C files. (#4130) 2017-10-26 16:59:40 +03:00
pdox 1896793520 bpo-31857: Make the behavior of USE_STACKCHECK deterministic (#4098) 2017-10-25 23:03:01 -07:00
Mark Shannon ae3087c638 Move exc state to generator. Fixes bpo-25612 (#1773)
Move exception state information from frame objects to coroutine (generator/thread) object where it belongs.
2017-10-22 23:41:51 +02:00
Masayuki Yamamoto 731e189014 bpo-25658: Implement PEP 539 for Thread Specific Storage (TSS) API (GH-1362)
See PEP 539 for details.

Highlights of changes:

- Add Thread Specific Storage (TSS) API
- Document the Thread Local Storage (TLS) API as deprecated
- Update code that used TLS API to use TSS API
2017-10-06 20:41:34 +10:00
Eric Snow dae0276bb6 bpo-30860: Fix a refleak. (#3567)
Resolves bpo-31420.

(This was accidentally reverted when in #3565.)
2017-09-14 00:35:58 -07:00
Eric Snow 93c92f7d1d bpo-31404: Revert "remove modules from Py_InterpreterState (#1638)" (#3565)
PR #1638, for bpo-28411, causes problems in some (very) edge cases. Until that gets sorted out, we're reverting the merge. PR #3506, a fix on top of #1638, is also getting reverted.
2017-09-13 23:46:04 -07:00
Eric Snow 8728018624 bpo-30860: Fix a refleak. (#3506)
* Drop warnoptions from PyInterpreterState.

* Drop xoptions from PyInterpreterState.

* Don't set warnoptions and _xoptions again.

* Decref after adding to sys.__dict__.

* Drop an unused macro.

* Check sys.xoptions *before* we delete it.
2017-09-11 17:59:22 -07:00
Eric Snow 2ebc5ce42a bpo-30860: Consolidate stateful runtime globals. (#3397)
* group the (stateful) runtime globals into various topical structs
* consolidate the topical structs under a single top-level _PyRuntimeState struct
* add a check-c-globals.py script that helps identify runtime globals

Other globals are excluded (see globals.txt and check-c-globals.py).
2017-09-07 23:51:28 -06:00
Antoine Pitrou a6a4dc816d bpo-31370: Remove support for threads-less builds (#3385)
* Remove Setup.config
* Always define WITH_THREAD for compatibility.
2017-09-07 18:56:24 +02:00
Eric Snow 05351c1bd8 Revert "bpo-30860: Consolidate stateful runtime globals." (#3379)
Windows buildbots started failing due to include-related errors.
2017-09-05 21:43:08 -07:00
Benjamin Peterson b0a9a5a6a4 correct initialization code (#3376)
Explicitly initialize struct members rather than relying on compiler extensions.
2017-09-05 20:19:12 -07:00
Eric Snow 76d5abc868 bpo-30860: Consolidate stateful runtime globals. (#2594)
* group the (stateful) runtime globals into various topical structs
* consolidate the topical structs under a single top-level _PyRuntimeState struct
* add a check-c-globals.py script that helps identify runtime globals

Other globals are excluded (see globals.txt and check-c-globals.py).
2017-09-05 18:26:16 -07:00
Eric Snow 86b7afdfee bpo-28411: Remove "modules" field from Py_InterpreterState. (#1638)
sys.modules is the one true source.
2017-09-04 17:54:09 -06:00
Dino Viehland f3cffd2b78 bpo-30604: clean up co_extra support (#2144)
bpo-30604: port fix from 3.6 dropping binary compatibility tweaks
2017-06-21 17:44:36 -04:00
Antoine Pitrou f7ecfac0c1 Doc nits for bpo-16500 (#1841)
* Doc nits for bpo-16500

* Fix more references
2017-05-28 11:35:14 +02:00
Antoine Pitrou 346cbd351e bpo-16500: Allow registering at-fork handlers (#1715)
* bpo-16500: Allow registering at-fork handlers

* Address Serhiy's comments

* Add doc for new C API

* Add doc for new Python-facing function

* Add NEWS entry + doc nit
2017-05-27 17:50:54 +02:00
Eric Snow 6b4be195cd bpo-22257: Small changes for PEP 432. (#1728)
PEP 432 specifies a number of large changes to interpreter startup code, including exposing a cleaner C-API. The major changes depend on a number of smaller changes. This patch includes all those smaller changes.
2017-05-22 21:36:03 -07:00
Eric Snow e377416c10 bpo-29102: Add a unique ID to PyInterpreterState. (#1639) 2017-05-22 19:46:40 -07:00
Jason Fried f82c951d1c bpo-30395 _PyGILState_Reinit deadlock fix (#1734)
head_lock could be held by another thread when fork happened. We should
reset it to avoid deadlock.
2017-05-22 16:58:55 -07:00
Serhiy Storchaka aefa7ebf0f bpo-6532: Make the thread id an unsigned integer. (#781)
* bpo-6532: Make the thread id an unsigned integer.

From C API side the type of results of PyThread_start_new_thread() and
PyThread_get_thread_ident(), the id parameter of
PyThreadState_SetAsyncExc(), and the thread_id field of PyThreadState
changed from "long" to "unsigned long".

* Restore a check in thread_get_ident().
2017-03-23 14:48:39 +01:00
Benjamin Peterson 4fd64b9a6a remove ceval timestamp support 2016-09-09 14:57:58 -07:00
Yury Selivanov eb6364557f Issue #28003: Implement PEP 525 -- Asynchronous Generators. 2016-09-08 22:01:51 -07:00
Brett Cannon 5c4de2863b Add the co_extra field and accompanying APIs to code objects.
This completes PEP 523.
2016-09-07 11:16:41 -07:00
Benjamin Peterson ca47063998 replace Py_(u)intptr_t with the c99 standard types 2016-09-06 13:47:26 -07:00
Brett Cannon 3cebf93872 Implement the frame evaluation API aspect of PEP 523. 2016-09-05 15:33:46 -07:00
Berker Peksag c98afb7a26 Issue #27587: Merge from 3.5 2016-08-22 18:07:02 +03:00
Berker Peksag 4b7b565c58 Issue #27587: Move null pointer check earlier in _PyState_AddModule()
This was found by PVS-Studio:

V595 The 'def' pointer was utilized before it was verified
against nullptr. Check lines: 286, 292. pystate.c 286

Initial patch by Christian Heimes.
2016-08-22 18:05:56 +03:00
Serhiy Storchaka 133138a284 Issue #22557: Now importing already imported modules is up to 2.5 times faster. 2016-08-02 22:51:21 +03:00
Serhiy Storchaka c2f7d87897 Issue #26932: Fixed support of RTLD_* constants defined as enum values,
not via macros (in particular on Android).  Patch by Chi Hsuan Yen.
2016-05-04 09:44:44 +03:00
Victor Stinner 861d9abfcf faulthandler now works in non-Python threads
Issue #26563:

* Add _PyGILState_GetInterpreterStateUnsafe() function: the single
  PyInterpreterState used by this process' GILState implementation.
* Enhance _Py_DumpTracebackThreads() to retrieve the interpreter state from
  autoInterpreterState in last resort. The function now accepts NULL for interp
  and current_tstate parameters.
* test_faulthandler: fix a ResourceWarning when test is interrupted by CTRL+c
2016-03-16 22:45:24 +01:00
Victor Stinner 8a1be61849 Add more checks on the GIL
Issue #10915, #15751, #26558:

* PyGILState_Check() now returns 1 (success) before the creation of the GIL and
  after the destruction of the GIL. It allows to use the function early in
  Python initialization and late in Python finalization.
* Add a flag to disable PyGILState_Check(). Disable PyGILState_Check() when
  Py_NewInterpreter() is called
* Add assert(PyGILState_Check()) to: _Py_dup(), _Py_fstat(), _Py_read()
  and _Py_write()
2016-03-14 22:07:55 +01:00
Victor Stinner b02ef715a3 Use Py_uintptr_t for atomic pointers
Issue #26161: Use Py_uintptr_t instead of void* for atomic pointers in
pyatomic.h. Use atomic_uintptr_t when <stdatomic.h> is used.

Using void* causes compilation warnings depending on which implementation of
atomic types is used.
2016-01-22 14:09:55 +01:00