Commit Graph

403 Commits

Author SHA1 Message Date
Eric Snow 81c72044a1
bpo-46541: Replace core use of _Py_IDENTIFIER() with statically initialized global objects. (gh-30928)
We're no longer using _Py_IDENTIFIER() (or _Py_static_string()) in any core CPython code.  It is still used in a number of non-builtin stdlib modules.

The replacement is: PyUnicodeObject (not pointer) fields under _PyRuntimeState, statically initialized as part of _PyRuntime.  A new _Py_GET_GLOBAL_IDENTIFIER() macro facilitates lookup of the fields (along with _Py_GET_GLOBAL_STRING() for non-identifier strings).

https://bugs.python.org/issue46541#msg411799 explains the rationale for this change.

The core of the change is in:

* (new) Include/internal/pycore_global_strings.h - the declarations for the global strings, along with the macros
* Include/internal/pycore_runtime_init.h - added the static initializers for the global strings
* Include/internal/pycore_global_objects.h - where the struct in pycore_global_strings.h is hooked into _PyRuntimeState
* Tools/scripts/generate_global_objects.py - added generation of the global string declarations and static initializers

I've also added a --check flag to generate_global_objects.py (along with make check-global-objects) to check for unused global strings.  That check is added to the PR CI config.

The remainder of this change updates the core code to use _Py_GET_GLOBAL_IDENTIFIER() instead of _Py_IDENTIFIER() and the related _Py*Id functions (likewise for _Py_GET_GLOBAL_STRING() instead of _Py_static_string()).  This includes adding a few functions where there wasn't already an alternative to _Py*Id(), replacing the _Py_Identifier * parameter with PyObject *.

The following are not changed (yet):

* stop using _Py_IDENTIFIER() in the stdlib modules
* (maybe) get rid of _Py_IDENTIFIER(), etc. entirely -- this may not be doable as at least one package on PyPI using this (private) API
* (maybe) intern the strings during runtime init

https://bugs.python.org/issue46541
2022-02-08 13:39:07 -07:00
Steve Dower 99fcf15052
bpo-45582: Port getpath[p].c to Python (GH-29041)
The getpath.py file is frozen at build time and executed as code over a namespace. It is never imported, nor is it meant to be importable or reusable. However, it should be easier to read, modify, and patch than the previous code.

This commit attempts to preserve every previously tested quirk, but these may be changed in the future to better align platforms.
2021-12-03 00:08:42 +00:00
Victor Stinner d943d19172
bpo-45439: Move _PyObject_CallNoArgs() to pycore_call.h (GH-28895)
* Move _PyObject_CallNoArgs() to pycore_call.h (internal C API).
* _ssl, _sqlite and _testcapi extensions now call the public
  PyObject_CallNoArgs() function, rather than _PyObject_CallNoArgs().
* _lsprof extension is now built with Py_BUILD_CORE_MODULE macro
  defined to get access to internal _PyObject_CallNoArgs().
2021-10-12 08:38:19 +02:00
Victor Stinner ce3489cfdb
bpo-45439: Rename _PyObject_CallNoArg() to _PyObject_CallNoArgs() (GH-28891)
Fix typo in the private _PyObject_CallNoArg() function name: rename
it to _PyObject_CallNoArgs() to be consistent with the public
function PyObject_CallNoArgs().
2021-10-12 00:42:23 +02:00
borispopoff c7c3b7db29
Typo in comment (GH-24199)
Automerge-Triggered-By: GH:Mariatta
2021-01-25 14:51:48 -08:00
Victor Stinner a12491681f
bpo-32381: pymain_run_command() uses PyCF_IGNORE_COOKIE (GH-23724)
The coding cookie (ex: "# coding: latin1") is now ignored in the
command passed to the -c command line option.

Since pymain_run_command() uses UTF-8, pass PyCF_IGNORE_COOKIE
compiler flag to the parser.

pymain_run_python() no longer propages compiler flags between
function calls.
2020-12-23 19:16:56 +01:00
Victor Stinner ceb420251c
bpo-32381: pymain_run_file() uses PySys_FormatStderr() (GH-23778)
If config->run_filename doesn't exist, log the error into sys.stderr
using "%R" format, to escape properly unencodable characters (usually
with backslashreplace).
2020-12-15 16:25:27 +01:00
Victor Stinner a82f63f5af
bpo-32381: Add _PyRun_AnyFileObject() (GH-23723)
pymain_run_file() no longer encodes the filename: pass the filename
as an object to the new _PyRun_AnyFileObject() function.

Add new private functions:

* _PyRun_AnyFileObject()
* _PyRun_InteractiveLoopObject()
* _Py_FdIsInteractive()
2020-12-09 22:37:27 +01:00
Victor Stinner 550e4673be
bpo-32381: Add _PyRun_SimpleFileObject() (GH-23709)
pymain_run_startup() now pass the filename as a Python object to
_PyRun_SimpleFileObject().
2020-12-09 00:32:54 +01:00
Thomas Grainger a68a2ad19c
bpo-41602: raise SIGINT exit code on KeyboardInterrupt from pymain_run_module (#21956)
Closes bpo issue 41602
2020-09-22 08:53:03 -07:00
Serhiy Storchaka a7dc714701
bpo-41094: Additional fix for PYTHONSTARTUP. (GH-21119) 2020-06-24 19:46:30 +03:00
Serhiy Storchaka 6c6810d989
bpo-41094: Fix decoding errors with audit when open files. (GH-21095) 2020-06-24 08:46:05 +03:00
Victor Stinner 4f98f465f1
bpo-40268: Remove unused imports in pylifecycle.c (GH-19533)
Remove unused imports in files:

* initconfig.c
* main.c
* preconfig.h
* pylifecycle.c
* python.c
* pythonrun.c
2020-04-15 04:01:58 +02:00
Victor Stinner d9ea5cae1d
bpo-40268: Remove unused pycore_pymem.h includes (GH-19531) 2020-04-15 02:57:50 +02:00
Victor Stinner e5014be049
bpo-40268: Remove a few pycore_pystate.h includes (GH-19510) 2020-04-14 17:52:15 +02:00
Victor Stinner 81a7be3fa2
bpo-40268: Rename _PyInterpreterState_GET_UNSAFE() (GH-19509)
Rename _PyInterpreterState_GET_UNSAFE() to _PyInterpreterState_GET()
for consistency with _PyThreadState_GET() and to have a shorter name
(help to fit into 80 columns).

Add also "assert(tstate != NULL);" to the function.
2020-04-14 15:14:01 +02:00
Victor Stinner 4a3fe08353
bpo-40268: Include explicitly pycore_interp.h (GH-19505)
pycore_pystate.h no longer includes pycore_interp.h:
it's now included explicitly in files accessing PyInterpreterState.
2020-04-14 14:26:24 +02:00
Victor Stinner da7933ecc3
bpo-40268: Add _PyInterpreterState_GetConfig() (GH-19492)
Don't access PyInterpreterState.config member directly anymore, but
use new functions:

* _PyInterpreterState_GetConfig()
* _PyInterpreterState_SetConfig()
* _Py_GetConfig()
2020-04-13 03:04:28 +02:00
Victor Stinner 61691d8336
bpo-38353: Cleanup includes in the internal C API (GH-16548)
Use forward declaration of types to avoid includes in the internal C
API. Add also comment to justify other includes.
2019-10-02 23:51:20 +02:00
Victor Stinner 5e0ea7540f
bpo-37474: Don't call fedisableexcept() on FreeBSD (GH-16515)
On FreeBSD, Python no longer calls fedisableexcept() at startup to
control the floating point control mode. The call became useless
since FreeBSD 6: it became the default mode.
2019-10-01 13:12:29 +02:00
Victor Stinner 8462a4936b
bpo-38304: PyConfig_InitPythonConfig() cannot fail anymore (GH-16509)
PyConfig_InitPythonConfig() and PyConfig_InitIsolatedConfig() no
longer return PyStatus: they cannot fail anymore.
2019-10-01 12:06:16 +02:00
Victor Stinner 3c30a76f3d
bpo-38304: Remove PyConfig.struct_size (GH-16500) (GH-16508)
For now, we'll rely on the fact that the config structures aren't covered by the stable ABI.

We may revisit this in the future if we further explore the idea of offering a stable embedding API.

(cherry picked from commit bdace21b76)
2019-10-01 10:56:37 +02:00
Victor Stinner 441b10cf28
bpo-38304: Add PyConfig.struct_size (GH-16451)
Add a new struct_size field to PyPreConfig and PyConfig structures to
allow to modify these structures in the future without breaking the
backward compatibility.

* Replace private _config_version field with public struct_size field
  in PyPreConfig and PyConfig.
* Public PyPreConfig_InitIsolatedConfig() and
  PyPreConfig_InitPythonConfig()
  return type becomes PyStatus, instead of void.
* Internal _PyConfig_InitCompatConfig(),
  _PyPreConfig_InitCompatConfig(), _PyPreConfig_InitFromConfig(),
  _PyPreConfig_InitFromPreConfig() return type becomes PyStatus,
  instead of void.
* Remove _Py_CONFIG_VERSION
* Update the Initialization Configuration documentation.
2019-09-28 04:28:35 +02:00
Serhiy Storchaka 1f21eaa15e
bpo-15999: Clean up of handling boolean arguments. (GH-15610)
* Use the 'p' format unit instead of manually called PyObject_IsTrue().
* Pass boolean value instead 0/1 integers to functions that needs boolean.
* Convert some arguments to boolean only once.
2019-09-01 12:16:51 +03:00
Steve Dower e226e83d36
bpo-37363: Add audit events on startup for the run commands (GH-14524) 2019-07-01 16:03:53 -07:00
Victor Stinner 67310023f2
bpo-36763: Use PyConfig_Clear() (GH-14445)
Stop using "static PyConfig", PyConfig must now always use
dynamically allocated strings: use PyConfig_SetString(),
PyConfig_SetArgv() and PyConfig_Clear().
2019-07-01 19:52:45 +02:00
Victor Stinner 37d66d7d4b
bpo-37253: Add _PyCompilerFlags_INIT macro (GH-14018)
Add a new _PyCompilerFlags_INIT macro to initialize PyCompilerFlags
variables, rather than initializing cf_flags and cf_feature_version
explicitly in each variable.
2019-06-13 02:16:41 +02:00
Victor Stinner 331a6a56e9
bpo-36763: Implement the PEP 587 (GH-13592)
* Add a whole new documentation page:
  "Python Initialization Configuration"
* PyWideStringList_Append() return type is now PyStatus,
  instead of int
* PyInterpreterState_New() now calls PyConfig_Clear() if
  PyConfig_InitPythonConfig() fails.
* Rename files:

  * Python/coreconfig.c => Python/initconfig.c
  * Include/cpython/coreconfig.h => Include/cpython/initconfig.h
  * Include/internal/: pycore_coreconfig.h => pycore_initconfig.h

* Rename structures

  * _PyCoreConfig => PyConfig
  * _PyPreConfig => PyPreConfig
  * _PyInitError => PyStatus
  * _PyWstrList => PyWideStringList

* Rename PyConfig fields:

  * use_module_search_paths => module_search_paths_set
  * module_search_path_env => pythonpath_env

* Rename PyStatus field: _func => func
* PyInterpreterState: rename core_config field to config
* Rename macros and functions:

  * _PyCoreConfig_SetArgv() => PyConfig_SetBytesArgv()
  * _PyCoreConfig_SetWideArgv() => PyConfig_SetArgv()
  * _PyCoreConfig_DecodeLocale() => PyConfig_SetBytesString()
  * _PyInitError_Failed() => PyStatus_Exception()
  * _Py_INIT_ERROR_TYPE_xxx enums => _PyStatus_TYPE_xxx
  * _Py_UnixMain() => Py_BytesMain()
  * _Py_ExitInitError() => Py_ExitStatusException()
  * _Py_PreInitializeFromArgs() => Py_PreInitializeFromBytesArgs()
  * _Py_PreInitializeFromWideArgs() => Py_PreInitializeFromArgs()
  * _Py_PreInitialize() => Py_PreInitialize()
  * _Py_RunMain() => Py_RunMain()
  * _Py_InitializeFromConfig() => Py_InitializeFromConfig()
  * _Py_INIT_XXX() => _PyStatus_XXX()
  * _Py_INIT_FAILED() => _PyStatus_EXCEPTION()

* Rename 'err' PyStatus variables to 'status'
* Convert RUN_CODE() macro to config_run_code() static inline function
* Remove functions:

  * _Py_InitializeFromArgs()
  * _Py_InitializeFromWideArgs()
  * _PyInterpreterState_GetCoreConfig()
2019-05-27 16:39:22 +02:00
Erik Janssens 925af1d99b bpo-36965: Fix includes in main.c on Windows with non-MSC compilers (GH-13421)
Include windows.h rather than crtdbg.h to get STATUS_CONTROL_C_EXIT constant.
Moreover, include windows.h on Windows, not only when MSC is used.
2019-05-21 12:11:11 +02:00
Victor Stinner 12083284c5
bpo-36763: _Py_RunMain() doesn't call Py_Exit() anymore (GH-13390)
Py_Main() and _Py_RunMain() now return the exitcode rather than
calling Py_Exit(exitcode) when calling PyErr_Print() if the current
exception type is SystemExit.

* Add _Py_HandleSystemExit().
* Add pymain_exit_err_print().
* Add pymain_exit_print().
2019-05-17 23:05:29 +02:00
Victor Stinner cab5d0741e
bpo-36763: Add _PyCoreConfig_InitPythonConfig() (GH-13388)
Add new functions to get the Python interpreter behavior:

* _PyPreConfig_InitPythonConfig()
* _PyCoreConfig_InitPythonConfig()

Add new functions to get an isolated configuration:

* _PyPreConfig_InitIsolatedConfig()
* _PyCoreConfig_InitIsolatedConfig()

Replace _PyPreConfig_INIT and _PyCoreConfig_INIT with new functions
_PyPreConfig_Init() and _PyCoreConfig_Init().

_PyCoreConfig: set configure_c_stdio and parse_argv to 0 by default
to behave as Python 3.6 in the default configuration.

_PyCoreConfig_Read() no longer sets coerce_c_locale_warn to 1 if it's
equal to 0. coerce_c_locale_warn must now be set to -1 (ex: using
_PyCoreConfig_InitPythonConfig()) to enable C locale coercion
warning.

Add unit tests for _PyCoreConfig_InitPythonConfig()
and _PyCoreConfig_InitIsolatedConfig().

Changes:

* Rename _PyCoreConfig_GetCoreConfig() to _PyPreConfig_GetCoreConfig()
* Fix core_read_precmdline(): handle parse_argv=0
* Fix _Py_PreInitializeFromCoreConfig(): pass coreconfig.argv
  to _Py_PreInitializeFromPyArgv(), except if parse_argv=0
2019-05-17 19:01:14 +02:00
Victor Stinner fed02e15b3
bpo-36763: Remove _PyCoreConfig.program (GH-13373)
Use _PyCoreConfig.program_name instead.
2019-05-17 11:12:09 +02:00
Victor Stinner ae239f6b06
bpo-36763: Add _PyCoreConfig.parse_argv (GH-13361)
* _PyCoreConfig_Read() doesn't parse nor update argv
  if parse_argv is 0.
* Move path configuration fields in _PyCoreConfig.
* Add an unit test for parse_argv=0.
* Remove unused "done": label in _Py_RunMain().
2019-05-16 17:02:56 +02:00
Victor Stinner dbacfc2273
bpo-36763: _PyInitError always use int for exitcode (GH-13360)
We cannot use "unsigned int" for exitcode on Windows, since
Py_Main() and _Py_RunMain() always return an "int".

Changes:

* _PyPathConfig_ComputeSysPath0() now returns -1 if an exception is
  raised.
* pymain_run_python() no longer uses _PyInitError but display the
  exception and set exitcode to 1 in case of error.
* Fix _Py_RunMain(): return an exitcode rather than calling
  exit() on pymain_run_python() failure.
* _Py_ExitInitError() no longer uses ExitProcess() on Windows, use
  exit() on all platforms.
* _Py_ExitInitError() now fails with a fatal error if 'err' is not an
  error not an exit.
2019-05-16 16:39:26 +02:00
Victor Stinner c96be811fa
bpo-36900: Replace global conf vars with config (GH-13299)
Replace global configuration variables with core_config read from the
current interpreter.

Cleanup dynload_hpux.c.
2019-05-14 17:34:56 +02:00
Victor Stinner 70005ac0fd
bpo-36763: _PyCoreConfig_SetPyArgv() preinitializes Python (GH-13037)
_PyCoreConfig_SetPyArgv() and _PyCoreConfig_SetWideString() now
pre-initialize Python if needed to ensure that the locale encoding is
properly configured.

* Add _Py_PreInitializeFromPyArgv() internal function.
* Add 'args' parameter to _Py_PreInitializeFromCoreConfig()
2019-05-02 15:25:34 -04:00
Victor Stinner db71975431
bpo-36763: Rework _PyInitError API (GH-13031)
* Remove _PyInitError.user_err field and _Py_INIT_USER_ERR() macro:
  use _Py_INIT_ERR() instead. _Py_ExitInitError() now longer calls
  abort() on error: exit with exit code 1 instead.
* Add _PyInitError._type private field.
* exitcode field type is now unsigned int on Windows.
* Rename prefix field to _func.
* Rename msg field to err_msg.
2019-05-01 05:35:33 +02:00
Victor Stinner 99fcc616d4
Revert "bpo-36356: Destroy the GIL at exit (GH-12453)" (GH613006)
This reverts commit b36e5d627d.
2019-04-29 13:04:07 +02:00
Victor Stinner b36e5d627d
bpo-36356: Destroy the GIL at exit (GH-12453)
* Add _PyEval_FiniThreads2(). _PyEval_FiniThreads() now only clears
  the pending lock, whereas _PyEval_FiniThreads2() destroys the GIL.
* pymain_free() now calls _PyEval_FiniThreads2().
* Py_FinalizeEx() now calls _PyEval_FiniThreads().
2019-04-29 11:15:56 +02:00
Victor Stinner 4cb525a1f0
bpo-36356: pymain_exit_error() only call pymain_free() for exit (GH-12968)
Add _Py_INIT_HAS_EXITCODE() macro.
2019-04-26 13:05:47 +02:00
Inada Naoki 10654c19b5
bpo-20844: open script file with "rb" mode (GH-12616) 2019-04-01 18:35:20 +09:00
Victor Stinner 2f54908afc
bpo-36471: Add _Py_RunMain() (GH-12618)
* Add config_read_cmdline() subfunction. Remove _PyCmdline structure.
* _PyCoreConfig_Read() now also parses config->argv command line
  arguments
2019-03-29 15:13:46 +01:00
Victor Stinner d929f1838a
bpo-36443: Disable C locale coercion and UTF-8 Mode by default (GH-12589)
bpo-36443, bpo-36202: Since Python 3.7.0, calling Py_DecodeLocale()
before Py_Initialize() produces mojibake if the LC_CTYPE locale is
coerced and/or if the UTF-8 Mode is enabled by the user
configuration. This change fix the issue by disabling LC_CTYPE
coercion and UTF-8 Mode by default. They must now be enabled
explicitly (opt-in) using the new _Py_PreInitialize() API with
_PyPreConfig.

When embedding Python, set coerce_c_locale and utf8_mode attributes
of _PyPreConfig to -1 to enable automatically these parameters
depending on the LC_CTYPE locale, environment variables and command
line arguments

Alternative: Setting Py_UTF8Mode to 1 always explicitly enables the
UTF-8 Mode.

Changes:

* _PyPreConfig_INIT now sets coerce_c_locale and utf8_mode to 0 by
  default.
* _Py_InitializeFromArgs() and _Py_InitializeFromWideArgs() can now
  be called with config=NULL.
2019-03-27 18:28:46 +01:00
Victor Stinner 5ac27a50ff
bpo-36444: Rework _Py_InitializeFromConfig() API (GH-12576) 2019-03-27 13:40:14 +01:00
Victor Stinner 484f20d2ff
bpo-36444: Add _PyCoreConfig._init_main (GH-12572)
* Add _PyCoreConfig._init_main: if equals to zero,
  _Py_InitializeFromConfig() doesn't call
  _Py_InitializeMainInterpreter().
* Add interp_p parameter to _Py_InitializeFromConfig().
* pymain_init() now calls _Py_InitializeFromConfig().
* Make _Py_InitializeCore() private.
2019-03-27 02:04:16 +01:00
Victor Stinner 8b9dbc017a
bpo-36444: Remove _PyMainInterpreterConfig (GH-12571) 2019-03-27 01:36:16 +01:00
Victor Stinner f8ba6f5afc
bpo-36301: Cleanup preconfig.c and coreconfig.c (GH-12563)
* _PyCoreConfig_Write() now updates _PyRuntime.preconfig
* Remove _PyPreCmdline_Copy()
* _PyPreCmdline_Read() now accepts _PyPreConfig and _PyCoreConfig
  optional configurations.
* Rename _PyPreConfig_ReadFromArgv() to _PyPreConfig_Read(). Simplify
  the code.
* Calling _PyCoreConfig_Read() no longer adds the warning options
  twice: don't add a warning option if it's already in the list.
* Rename _PyCoreConfig_ReadFromArgv() to _PyCoreConfig_Read().
* Rename config_from_cmdline() to _PyCoreConfig_ReadFromArgv().
* Add more assertions on _PyCoreConfig in _PyCoreConfig_Read().
* Move some functions.
* Make some config functions private.
2019-03-26 16:58:50 +01:00
Victor Stinner 20004959d2
bpo-36301: Remove _PyCoreConfig.preconfig (GH-12546)
* Replace _PyCoreConfig.preconfig with 3 new fields in _PyCoreConfig:
  isolated, use_environment, dev_mode.
* Add _PyPreCmdline.dev_mode.
* Add _Py_PreInitializeFromPreConfigInPlace().
2019-03-26 02:31:11 +01:00
Victor Stinner f78a5e9ce8
bpo-36301: Add _Py_GetEnv() function (GH-12542)
* Make _PyPreConfig_GetEnv(), _PyCoreConfig_GetEnv() and
  _PyCoreConfig_GetEnvDup() private
* _Py_get_env_flag() first parameter becomes "int use_environment"
2019-03-26 00:03:15 +01:00
Victor Stinner f72346c475
bpo-36301: Cleanup preconfig code (GH-12535)
Prepare code to move some _PyPreConfig parameters into _PyPreCmdline.
Changes:

* _PyCoreConfig_ReadFromArgv(): remove preconfig parameter,
  use _PyRuntime.preconfig.
* Add _PyPreCmdline_GetPreConfig() (called by _PyPreConfig_Read()).
* Rename _PyPreCmdline_Init() to _PyPreCmdline_SetArgv()
* Factorize _Py_PreInitializeFromPreConfig() code: add
  pyinit_preinit().
* _PyPreConfig_Read() now sets coerce_c_locale to 2 if it must be
  coerced.
* Remove _PyCoreConfig_ReadPreConfig().
* _PyCoreConfig_Write() now copies updated preconfig into _PyRuntime.
2019-03-25 17:54:58 +01:00