Commit Graph

246 Commits

Author SHA1 Message Date
Pablo Galindo b96c6b0723 bpo-38962: Fix reference leak in new_interpreter() (GH-17453)
https://bugs.python.org/issue38962



Automerge-Triggered-By: @pablogsal
2019-12-04 03:19:59 -08:00
Victor Stinner 2ec1a1b307
bpo-38858: new_interpreter() uses pycore_init_import_warnings() (GH-17353) 2019-11-22 21:54:33 +01:00
Victor Stinner 2582d46fbc
bpo-38858: new_interpreter() reuses pycore_init_builtins() (GH-17351)
new_interpreter() now calls _PyBuiltin_Init() to create the builtins
module and calls _PyImport_FixupBuiltin(), rather than using
_PyImport_FindBuiltin(tstate, "builtins").

pycore_init_builtins() is now responsible to initialize
intepr->builtins_copy: inline _PyImport_Init() and remove this
function.
2019-11-22 19:24:49 +01:00
Victor Stinner b00513636c
bpo-38858: Add init_interp_main() subfunction (GH-17347)
Fix new_interpreter() error handling: undo it all if status is an
exception.
2019-11-22 17:52:42 +01:00
Victor Stinner e0c9ab8e26
bpo-38858: Add init_set_builtins_open() subfunction (GH-17346) 2019-11-22 16:19:14 +01:00
Victor Stinner 3d4833488a
bpo-38858: Call _PyUnicode_Fini() in Py_EndInterpreter() (GH-17330)
Py_EndInterpreter() now clears the filesystem codec.
2019-11-22 12:27:50 +01:00
Victor Stinner 310e2d2517
bpo-36854: Fix refleak in subinterpreter (GH-17331)
finalize_interp_clear() now explicitly clears the codec registry and
then trigger a GC collection to clear all references.
2019-11-22 10:58:00 +01:00
Victor Stinner b93f31fcd9
bpo-38858: Fix Py_Finalize() when called from a subinterpreter (GH-17297)
Use _Py_IsMainInterpreter() in Py_Initialize() and Py_Finalize() to
detect if the current interpreter is the main interpreter or not.
2019-11-20 18:39:12 +01:00
Victor Stinner 4dedd0f0dd
bpo-37340: Remove PyMethod_ClearFreeList() and PyCFunction_ClearFreeList() (GH-17284)
Remove PyMethod_ClearFreeList() and PyCFunction_ClearFreeList()
functions: the free lists of bound method objects have been removed.

Remove also _PyMethod_Fini() and _PyCFunction_Fini() functions.
2019-11-20 12:59:12 +01:00
Victor Stinner 7247407c35
bpo-36854: Move _PyRuntimeState.gc to PyInterpreterState (GH-17287)
* Rename _PyGC_InitializeRuntime() to _PyGC_InitState()
* finalize_interp_clear() now also calls _PyGC_Fini() in
  subinterpreters (clear the GC state).
2019-11-20 12:25:50 +01:00
Victor Stinner e7e699e4df
bpo-38858: Fix reference leak in pycore_init_types() (GH-17286)
Only call _PyGC_Init(), _PyExc_Init() and _PyErr_Init() in
new_interpreter().
2019-11-20 12:08:13 +01:00
Victor Stinner 9da7430675
bpo-36854: Clear the current thread later (GH-17279)
Clear the current thread later in the Python finalization.

* The PyInterpreterState_Delete() function is now responsible
  to call PyThreadState_Swap(NULL).
* The tstate_delete_common() function is now responsible to clear the
  "autoTSSKey" thread local storage and it only clears it once the
  thread state is fully cleared. It allows to still get the current
  thread from TSS in tstate_delete_common().
2019-11-20 11:17:17 +01:00
Victor Stinner 7eee5beaf8
bpo-38858: Factorize Py_EndInterpreter() code (GH-17273)
* Factorize code in common between Py_FinalizeEx() and
  Py_EndInterpreter().
* Py_EndInterpreter() now also calls _PyWarnings_Fini().
* Call _PyExc_Fini() and _PyGC_Fini() later in the finalization.
2019-11-20 10:38:34 +01:00
Victor Stinner 01b1cc12e7
bpo-36710: Add PyInterpreterState.runtime field (GH-17270)
Add PyInterpreterState.runtime field: reference to the _PyRuntime
global variable. This field exists to not have to pass runtime in
addition to tstate to a function.  Get runtime from tstate:
tstate->interp->runtime.

Remove "_PyRuntimeState *runtime" parameter from functions already
taking a "PyThreadState *tstate" parameter.

_PyGC_Init() first parameter becomes "PyThreadState *tstate".
2019-11-20 02:27:56 +01:00
Victor Stinner 444b39bb64
bpo-38631: Avoid Py_FatalError() in handle_legacy_finalizers() (GH-17266)
* Rename _PyGC_Initialize() to _PyGC_InitializeRuntime()
* Add _PyGC_Init(): initialize _PyRuntime.gc.garbage list
* Call _PyGC_Init() before _PyTypes_Init()
2019-11-20 01:18:11 +01:00
Victor Stinner ef5aa9af7c
bpo-38858: Reorganize pycore_init_types() (GH-17265)
* Call _PyLong_Init() and _PyExc_Init() earlier
* new_interpreter() reuses pycore_init_types()
2019-11-20 00:38:03 +01:00
Victor Stinner 8462a4936b
bpo-38304: PyConfig_InitPythonConfig() cannot fail anymore (GH-16509)
PyConfig_InitPythonConfig() and PyConfig_InitIsolatedConfig() no
longer return PyStatus: they cannot fail anymore.
2019-10-01 12:06:16 +02:00
Victor Stinner 3c30a76f3d
bpo-38304: Remove PyConfig.struct_size (GH-16500) (GH-16508)
For now, we'll rely on the fact that the config structures aren't covered by the stable ABI.

We may revisit this in the future if we further explore the idea of offering a stable embedding API.

(cherry picked from commit bdace21b76)
2019-10-01 10:56:37 +02:00
Victor Stinner 441b10cf28
bpo-38304: Add PyConfig.struct_size (GH-16451)
Add a new struct_size field to PyPreConfig and PyConfig structures to
allow to modify these structures in the future without breaking the
backward compatibility.

* Replace private _config_version field with public struct_size field
  in PyPreConfig and PyConfig.
* Public PyPreConfig_InitIsolatedConfig() and
  PyPreConfig_InitPythonConfig()
  return type becomes PyStatus, instead of void.
* Internal _PyConfig_InitCompatConfig(),
  _PyPreConfig_InitCompatConfig(), _PyPreConfig_InitFromConfig(),
  _PyPreConfig_InitFromPreConfig() return type becomes PyStatus,
  instead of void.
* Remove _Py_CONFIG_VERSION
* Update the Initialization Configuration documentation.
2019-09-28 04:28:35 +02:00
Victor Stinner 12f2f177fc
bpo-38234: Py_Initialize() sets global path configuration (GH-16421)
* Py_InitializeFromConfig() now writes PyConfig path configuration to
  the global path configuration (_Py_path_config).
* Add test_embed.test_get_pathconfig().
* Fix typo in _PyWideStringList_Join().
2019-09-26 15:51:50 +02:00
Victor Stinner c422167749
bpo-38234: Remove _PyPathConfig.dll_path (GH-16307)
The DLL path is not computed from any user configuration and cannot
be configured by PyConfig. Instead, add a new _Py_dll_path global variable.

Remove _PyConfig_SetPathConfig(): replaced with _PyPathConfig_Init().

Py_Initialize() now longer sets the "global path configuration",
but only initialize _Py_dll_path.
2019-09-21 01:02:56 +02:00
Victor Stinner 1ce16fb097
bpo-38070: Py_FatalError() logs runtime state (GH-16246) 2019-09-18 01:35:33 +02:00
Victor Stinner d3b904144e
bpo-38070: Add _PyRuntimeState.preinitializing (GH-16245)
Add _PyRuntimeState.preinitializing field: set to 1 while
Py_PreInitialize() is running.

_PyRuntimeState: rename also pre_initialized field to preinitialized.
2019-09-17 23:59:51 +02:00
Serhiy Storchaka 1f21eaa15e
bpo-15999: Clean up of handling boolean arguments. (GH-15610)
* Use the 'p' format unit instead of manually called PyObject_IsTrue().
* Pass boolean value instead 0/1 integers to functions that needs boolean.
* Convert some arguments to boolean only once.
2019-09-01 12:16:51 +03:00
Victor Stinner bed4817d52
Make PyXXX_Fini() functions private (GH-15531)
For example, rename PyTuple_Fini() to _PyTuple_Fini().

These functions are only declared in the internal C API.
2019-08-27 00:12:32 +02:00
Jeroen Demeyer 762f93ff2e bpo-37337: Add _PyObject_CallMethodNoArgs() (GH-14267) 2019-07-08 17:19:25 +09:00
Victor Stinner b45d259bdd
bpo-36710: Use tstate in pylifecycle.c (GH-14249)
In pylifecycle.c: pass tstate argument, rather than interp argument,
to functions.
2019-06-20 00:05:23 +02:00
Victor Stinner 987a0dcfa1
bpo-36710: Remove PyImport_Cleanup() function (GH-14221)
* Rename PyImport_Cleanup() to _PyImport_Cleanup() and move it to the
  internal C API. Add 'tstate' parameters.
* Remove documentation of _PyImport_Init(), PyImport_Cleanup(),
  _PyImport_Fini(). All three were documented as "For internal use
  only.".
2019-06-19 10:36:10 +02:00
Victor Stinner 0a28f8d379
bpo-36710: Add tstate parameter in import.c (GH-14218)
* Add 'tstate' parameter to many internal import.c functions.
* _PyImportZip_Init() now gets 'tstate' parameter rather than
  'interp'.
* Add 'interp' parameter to _PyState_ClearModules() and rename it
  to _PyInterpreterState_ClearModules().
* Move private _PyImport_FindBuiltin() to the internal C API; add
  'tstate' parameter to it.
* Remove private _PyImport_AddModuleObject() from the C API:
  use public PyImport_AddModuleObject() instead.
* Remove private _PyImport_FindExtensionObjectEx() from the C API:
  use private _PyImport_FindExtensionObject() instead.
2019-06-19 02:54:39 +02:00
Victor Stinner 838f26402d
bpo-36710: Pass explicitly tstate in sysmodule.c (GH-14060)
* Replace global var Py_VerboseFlag with interp->config.verbose.
* Add _PyErr_NoMemory(tstate) function.
* Add tstate parameter to _PyEval_SetCoroutineOriginTrackingDepth()
  and move the function to the internal API.
* Replace _PySys_InitMain(runtime, interp)
  with _PySys_InitMain(runtime, tstate).
2019-06-13 22:41:23 +02:00
Victor Stinner 0fd2c300c2
Revert "bpo-36818: Add PyInterpreterState.runtime field. (gh-13129)" (GH-13795)
This reverts commit 396e0a8d9d.
2019-06-04 03:15:09 +02:00
Victor Stinner e225bebc14
Revert "bpo-33608: Factor out a private, per-interpreter _Py_AddPendingCall(). (gh-13714)" (GH-13780)
This reverts commit 6a150bcaeb.
2019-06-03 18:14:24 +02:00
Inada Naoki 91234a1636
bpo-26219: per opcode cache for LOAD_GLOBAL (GH-12884)
This patch implements per opcode cache mechanism, and use it in
only LOAD_GLOBAL opcode.

Based on Yury's opcache3.patch in bpo-26219.
2019-06-03 21:30:58 +09:00
Eric Snow 6a150bcaeb
bpo-33608: Factor out a private, per-interpreter _Py_AddPendingCall(). (gh-13714) 2019-06-01 15:39:46 -06:00
Eric Snow 396e0a8d9d
bpo-36818: Add PyInterpreterState.runtime field. (gh-13129)
https://bugs.python.org/issue36818
2019-05-31 21:16:47 -06:00
Victor Stinner 331a6a56e9
bpo-36763: Implement the PEP 587 (GH-13592)
* Add a whole new documentation page:
  "Python Initialization Configuration"
* PyWideStringList_Append() return type is now PyStatus,
  instead of int
* PyInterpreterState_New() now calls PyConfig_Clear() if
  PyConfig_InitPythonConfig() fails.
* Rename files:

  * Python/coreconfig.c => Python/initconfig.c
  * Include/cpython/coreconfig.h => Include/cpython/initconfig.h
  * Include/internal/: pycore_coreconfig.h => pycore_initconfig.h

* Rename structures

  * _PyCoreConfig => PyConfig
  * _PyPreConfig => PyPreConfig
  * _PyInitError => PyStatus
  * _PyWstrList => PyWideStringList

* Rename PyConfig fields:

  * use_module_search_paths => module_search_paths_set
  * module_search_path_env => pythonpath_env

* Rename PyStatus field: _func => func
* PyInterpreterState: rename core_config field to config
* Rename macros and functions:

  * _PyCoreConfig_SetArgv() => PyConfig_SetBytesArgv()
  * _PyCoreConfig_SetWideArgv() => PyConfig_SetArgv()
  * _PyCoreConfig_DecodeLocale() => PyConfig_SetBytesString()
  * _PyInitError_Failed() => PyStatus_Exception()
  * _Py_INIT_ERROR_TYPE_xxx enums => _PyStatus_TYPE_xxx
  * _Py_UnixMain() => Py_BytesMain()
  * _Py_ExitInitError() => Py_ExitStatusException()
  * _Py_PreInitializeFromArgs() => Py_PreInitializeFromBytesArgs()
  * _Py_PreInitializeFromWideArgs() => Py_PreInitializeFromArgs()
  * _Py_PreInitialize() => Py_PreInitialize()
  * _Py_RunMain() => Py_RunMain()
  * _Py_InitializeFromConfig() => Py_InitializeFromConfig()
  * _Py_INIT_XXX() => _PyStatus_XXX()
  * _Py_INIT_FAILED() => _PyStatus_EXCEPTION()

* Rename 'err' PyStatus variables to 'status'
* Convert RUN_CODE() macro to config_run_code() static inline function
* Remove functions:

  * _Py_InitializeFromArgs()
  * _Py_InitializeFromWideArgs()
  * _PyInterpreterState_GetCoreConfig()
2019-05-27 16:39:22 +02:00
Steve Dower b82e17e626
bpo-36842: Implement PEP 578 (GH-12613)
Adds sys.audit, sys.addaudithook, io.open_code, and associated C APIs.
2019-05-23 08:45:22 -07:00
Victor Stinner 5edcf26358
bpo-36763: Rename private Python initialization functions (GH-13511)
* Rename private C functions:

  * _Py_Initialize_ReconfigureCore => pyinit_core_reconfigure
  * _Py_InitializeCore_impl => pyinit_core_config
  * _Py_InitializeCore = > pyinit_core
  * _Py_InitializeMainInterpreter => pyinit_main
  * init_python => pyinit_python

* Rename _testembed.c commands: add "test_" prefix.
2019-05-23 00:57:57 +02:00
Victor Stinner 022be02dcf
bpo-36763: Add _PyPreConfig._config_init (GH-13481)
* _PyPreConfig_GetGlobalConfig() and  _PyCoreConfig_GetGlobalConfig()
  now do nothing if the configuration was not initialized with
  _PyPreConfig_InitCompatConfig() and _PyCoreConfig_InitCompatConfig()
* Remove utf8_mode=-2 special case: use utf8_mode=-1 instead.
* Fix _PyPreConfig_InitPythonConfig():

  * isolated = 0 instead of -1
  * use_environment = 1 instead of -1

* Rename _PyConfig_INIT to  _PyConfig_INIT_COMPAT
* Rename _PyPreConfig_Init() to _PyPreConfig_InitCompatConfig()
* Rename _PyCoreConfig_Init() to _PyCoreConfig_InitCompatConfig()
* PyInterpreterState_New() now uses _PyCoreConfig_InitPythonConfig()
  as default configuration, but it's very quickly overriden anyway.
* _freeze_importlib.c uses _PyCoreConfig_SetString() to set
  program_name.
* Cleanup preconfig_init_utf8_mode(): cmdline is always non-NULL.
2019-05-22 23:58:50 +02:00
Victor Stinner ef9d9b6312
bpo-36829: Add sys.unraisablehook() (GH-13187)
Add new sys.unraisablehook() function which can be overridden to
control how "unraisable exceptions" are handled. It is called when an
exception has occurred but there is no way for Python to handle it.
For example, when a destructor raises an exception or during garbage
collection (gc.collect()).

Changes:

* Add an internal UnraisableHookArgs type used to pass arguments to
  sys.unraisablehook.
* Add _PyErr_WriteUnraisableDefaultHook().
* The default hook now ignores exception on writing the traceback.
* test_sys now uses unittest.main() to automatically discover tests:
  remove test_main().
* Add _PyErr_Init().
* Fix PyErr_WriteUnraisable(): hold a strong reference to sys.stderr
  while using it
2019-05-22 11:28:22 +02:00
Victor Stinner 0f72147ce2
bpo-36763: Fix _PyRuntime.preconfig.coerce_c_locale (GH-13444)
_PyRuntime.preconfig.coerce_c_locale can now be used to
check if the C locale has been coerced.

* Fix _Py_LegacyLocaleDetected(): don't attempt to coerce the
  C locale if LC_ALL environment variable is set. Add 'warn'
  parameter: emit_stderr_warning_for_legacy_locale() must not
  the LC_ALL env var.
* _PyPreConfig_Write() sets coerce_c_locale to 0 if
  _Py_CoerceLegacyLocale() fails.
2019-05-20 17:16:38 +02:00
Victor Stinner 6d1c46746e
bpo-36763: Fix Python preinitialization (GH-13432)
* Add _PyPreConfig.parse_argv
* Add _PyCoreConfig._config_init field and _PyCoreConfigInitEnum enum
  type
* Initialization functions: reject preconfig=NULL and config=NULL
* Add config parameter to _PyCoreConfig_DecodeLocaleErr(): pass
  config->argv to _Py_PreInitializeFromPyArgv(), to parse config
  command line arguments in preinitialization.
* Add config parameter to _PyCoreConfig_SetString(). It now
  preinitializes Python.
* _PyCoreConfig_SetPyArgv() now also preinitializes Python for wide
  argv
* Fix _Py_PreInitializeFromCoreConfig(): don't pass args to
  _Py_PreInitializeFromPyArgv() if config.parse_argv=0.
* Use "char * const *" and "wchar_t * const *" types for 'argv'
  parameters and _PyArgv.argv.
* Add unit test on preinitialization from argv.
* _PyPreConfig.allocator type becomes int
* Add _PyPreConfig_InitFromPreConfig() and
  _PyPreConfig_InitFromCoreConfig() helper functions
2019-05-20 11:02:00 +02:00
Victor Stinner ed48866c55
bpo-35134: Split traceback.h header (GH-13430)
Add new Include/cpython/traceback.h and Include/internal/traceback.h
header files.
2019-05-20 00:14:57 +02:00
Victor Stinner b594784272
bpo-36763: _Py_InitializeFromArgs() argc becomes Py_ssize_t (GH-13396)
* The type of initlization function 'argc' parameters becomes
  Py_ssize_t, instead of int.
* Change _PyPreConfig_Copy() return type to void, instead of int.
  The function cannot fail anymore.
* Fix compilation warnings on Windows.
2019-05-18 00:38:16 +02:00
Victor Stinner cab5d0741e
bpo-36763: Add _PyCoreConfig_InitPythonConfig() (GH-13388)
Add new functions to get the Python interpreter behavior:

* _PyPreConfig_InitPythonConfig()
* _PyCoreConfig_InitPythonConfig()

Add new functions to get an isolated configuration:

* _PyPreConfig_InitIsolatedConfig()
* _PyCoreConfig_InitIsolatedConfig()

Replace _PyPreConfig_INIT and _PyCoreConfig_INIT with new functions
_PyPreConfig_Init() and _PyCoreConfig_Init().

_PyCoreConfig: set configure_c_stdio and parse_argv to 0 by default
to behave as Python 3.6 in the default configuration.

_PyCoreConfig_Read() no longer sets coerce_c_locale_warn to 1 if it's
equal to 0. coerce_c_locale_warn must now be set to -1 (ex: using
_PyCoreConfig_InitPythonConfig()) to enable C locale coercion
warning.

Add unit tests for _PyCoreConfig_InitPythonConfig()
and _PyCoreConfig_InitIsolatedConfig().

Changes:

* Rename _PyCoreConfig_GetCoreConfig() to _PyPreConfig_GetCoreConfig()
* Fix core_read_precmdline(): handle parse_argv=0
* Fix _Py_PreInitializeFromCoreConfig(): pass coreconfig.argv
  to _Py_PreInitializeFromPyArgv(), except if parse_argv=0
2019-05-17 19:01:14 +02:00
Victor Stinner b16b4e4592
bpo-36763: Add PyMemAllocatorName (GH-13387)
* Add PyMemAllocatorName enum
* _PyPreConfig.allocator type becomes PyMemAllocatorName, instead of
  char*
* Remove _PyPreConfig_Clear()
* Add _PyMem_GetAllocatorName()
* Rename _PyMem_GetAllocatorsName() to
  _PyMem_GetCurrentAllocatorName()
* Remove _PyPreConfig_SetAllocator(): just call
  _PyMem_SetupAllocators() directly, we don't have do reallocate the
  configuration with the new allocator anymore!
* _PyPreConfig_Write() parameter becomes const, as it should be in
  the first place!
2019-05-17 15:20:52 +02:00
Victor Stinner 9ef5dcaa0b
bpo-36763: Add _Py_InitializeMain() (GH-13362)
* Add a private _Py_InitializeMain() function.
* Add again _PyCoreConfig._init_main.
* _Py_InitializeFromConfig() now uses _init_main to decide
  if _Py_InitializeMainInterpreter() should be called.
* _PyCoreConfig: rename _frozen to pathconfig_warnings, its value is
  now the opposite of Py_FrozenFlag.
* Add an unit test for _init_main=0 and _Py_InitializeMain().
2019-05-16 17:38:16 +02:00
Victor Stinner dbacfc2273
bpo-36763: _PyInitError always use int for exitcode (GH-13360)
We cannot use "unsigned int" for exitcode on Windows, since
Py_Main() and _Py_RunMain() always return an "int".

Changes:

* _PyPathConfig_ComputeSysPath0() now returns -1 if an exception is
  raised.
* pymain_run_python() no longer uses _PyInitError but display the
  exception and set exitcode to 1 in case of error.
* Fix _Py_RunMain(): return an exitcode rather than calling
  exit() on pymain_run_python() failure.
* _Py_ExitInitError() no longer uses ExitProcess() on Windows, use
  exit() on all platforms.
* _Py_ExitInitError() now fails with a fatal error if 'err' is not an
  error not an exit.
2019-05-16 16:39:26 +02:00
Victor Stinner c96be811fa
bpo-36900: Replace global conf vars with config (GH-13299)
Replace global configuration variables with core_config read from the
current interpreter.

Cleanup dynload_hpux.c.
2019-05-14 17:34:56 +02:00
Victor Stinner 410b85a7f7
bpo-36900: import.c uses PyInterpreterState.core_config (GH-13278)
Move _PyImportZip_Init() to the internal C API and add an 'interp'
parameter.
2019-05-13 17:12:45 +02:00