Commit Graph

134 Commits

Author SHA1 Message Date
Sam Gross fdee7b7b3e
gh-112532: Require mimalloc in `--disable-gil` builds (gh-112883) 2023-12-12 09:04:48 +09:00
Donghee Na 0362cbf908
gh-109595: Add -Xcpu_count=<n> cmdline for container users (#109667)
---------

Co-authored-by: Victor Stinner <vstinner@python.org>
Co-authored-by: Gregory P. Smith [Google LLC] <greg@krypto.org>
2023-10-10 19:00:09 +09:00
Victor Stinner bb057b3370
gh-85283: Add PySys_AuditTuple() function (#108965)
sys.audit() now has assertions to check that the event argument is
not NULL and that the format argument does not use the "N" format.

Add tests on PySys_AuditTuple().
2023-10-05 21:59:35 +00:00
Victor Stinner a0773b89df
gh-108753: Enhance pystats (#108754)
Statistics gathering is now off by default. Use the "-X pystats"
command line option or set the new PYTHONSTATS environment variable
to 1 to turn statistics gathering on at Python startup.

Statistics are no longer dumped at exit if statistics gathering was
off or statistics have been cleared.

Changes:

* Add PYTHONSTATS environment variable.
* sys._stats_dump() now returns False if statistics are not dumped
  because they are all equal to zero.
* Add PyConfig._pystats member.
* Add tests on sys functions and on setting PyConfig._pystats to 1.
* Add Include/cpython/pystats.h and Include/internal/pycore_pystats.h
  header files.
* Rename '_py_stats' variable to '_Py_stats'.
* Exclude Include/cpython/pystats.h from the Py_LIMITED_API.
* Move pystats.h include from object.h to Python.h.
* Add _Py_StatsOn() and _Py_StatsOff() functions. Remove
  '_py_stats_struct' variable from the API: make it static in
  specialize.c.
* Document API in Include/pystats.h and Include/cpython/pystats.h.
* Complete pystats documentation in Doc/using/configure.rst.
* Don't write "all zeros" stats: if _stats_off() and _stats_clear()
  or _stats_dump() were called.
* _PyEval_Fini() now always call _Py_PrintSpecializationStats() which
  does nothing if stats are all zeros.

Co-authored-by: Michael Droettboom <mdboom@gmail.com>
2023-09-06 15:54:59 +00:00
Victor Stinner e35c722d22
gh-106659: Fix test_embed.test_forced_io_encoding() on Windows (#108010)
Use config.legacy_windows_stdio=1 to avoid _io._WindowsConsoleIO.
2023-08-16 11:56:56 +00:00
Victor Stinner 424049cc11
gh-105145: Remove old functions to config Python init (#105154)
Remove the following old functions to configure the Python
initialization, deprecated in Python 3.11:

* PySys_AddWarnOptionUnicode()
* PySys_AddWarnOption()
* PySys_AddXOption()
* PySys_HasWarnOptions()
* PySys_SetArgvEx()
* PySys_SetArgv()
* PySys_SetPath()
* Py_SetPath()
* Py_SetProgramName()
* Py_SetPythonHome()
* Py_SetStandardStreamEncoding()
* _Py_SetProgramFullPath()

Most of these functions are kept in the stable ABI, except:

* Py_SetStandardStreamEncoding()
* _Py_SetProgramFullPath()

Update Doc/extending/embedding.rst and Doc/extending/extending.rst to
use the new PyConfig API.

_testembed.c:

* check_stdio_details() now sets stdio_encoding and stdio_errors
  of PyConfig.
* Add definitions of functions removed from the API but kept in the
  stable ABI.
* test_init_from_config() and test_init_read_set() now use
  PyConfig_SetString() instead of PyConfig_SetBytesString().

Remove _Py_ClearStandardStreamEncoding() internal function.
2023-06-01 09:14:02 +02:00
Eddie Elizondo ea2c001650
gh-84436: Implement Immortal Objects (gh-19474)
This is the implementation of PEP683

Motivation:

The PR introduces the ability to immortalize instances in CPython which bypasses reference counting. Tagging objects as immortal allows up to skip certain operations when we know that the object will be around for the entire execution of the runtime.

Note that this by itself will bring a performance regression to the runtime due to the extra reference count checks. However, this brings the ability of having truly immutable objects that are useful in other contexts such as immutable data sharing between sub-interpreters.
2023-04-22 13:39:37 -06:00
chgnrdv 85ba8a3e03
gh-102650: Remove duplicate include directives from multiple source files (#102651)
Remove duplicate include directives from multiple source files
2023-03-13 13:25:17 -06:00
Kumar Aditya 6e3cc72afe
GH-90699: disallow `_Py_IDENTIFIER` in core code (GH-99210) 2022-11-09 08:53:21 -08:00
Nick Coghlan 05e48865be
gh-96853: Restore test coverage for Py_Initialize(Ex) (GH-98212)
* As most of `test_embed` now uses `Py_InitializeFromConfig`, add
  a specific test case to cover `Py_Initialize` (and `Py_InitializeEx`)
* Rename `_testembed` init helper to clarify the API used
* Add a `PyConfig_Clear` call in `Py_InitializeEx` to make
  the code more obviously correct (it already didn't leak as
  none of the dynamically allocated config fields were being
  populated, but it's clearer if the wrappers follow the
  documented API usage guidelines)
2022-10-30 22:01:30 +10:00
Eric Snow f32369480d
gh-98608: Change _Py_NewInterpreter() to _Py_NewInterpreterFromConfig() (gh-98609)
(see https://github.com/python/cpython/issues/98608)

This change does the following:

1. change the argument to a new `_PyInterpreterConfig` struct
2. rename the function to `_Py_NewInterpreterFromConfig()`, inspired by `Py_InitializeFromConfig()` (takes a `_PyInterpreterConfig`  instead of `isolated_subinterpreter`)
3. split up the boolean `isolated_subinterpreter` into the corresponding multiple granular settings
   * allow_fork
   * allow_subprocess
   * allow_threads
4. add `PyInterpreterState.feature_flags` to store those settings
5. add a function for checking if a feature is enabled on an opaque `PyInterpreterState *`
6. drop `PyConfig._isolated_interpreter`

The existing default (see `Py_NewInterpeter()` and `Py_Initialize*()`) allows fork, subprocess, and threads and the optional "isolated" interpreter (see the `_xxsubinterpreters` module) disables all three.  None of that changes here; the defaults are preserved.

Note that the given `_PyInterpreterConfig` will not be used outside `_Py_NewInterpreterFromConfig()`, nor preserved.  This contrasts with how `PyConfig` is currently preserved, used, and even modified outside `Py_InitializeFromConfig()`.  I'd rather just avoid that mess from the start for `_PyInterpreterConfig`.  We can preserve it later if we find an actual need.

This change allows us to follow up with a number of improvements (e.g. stop disallowing subprocess and support disallowing exec instead).

(Note that this PR adds "private" symbols.  We'll probably make them public, and add docs, in a separate change.)
2022-10-26 11:16:30 -06:00
Gregory P. Smith b0f89cb431
gh-96512: Move int_max_str_digits setting to PyConfig (#96944)
It had to live as a global outside of PyConfig for stable ABI reasons in
the pre-3.12 backports.

This removes the `_Py_global_config_int_max_str_digits` and gets rid of
the equivalent field in the internal `struct _is PyInterpreterState` as
code can just use the existing nested config struct within that.

Adds tests to verify unique settings and configs in subinterpreters.
2022-10-03 13:55:45 -07:00
Brandt Bucher 8fd2c3b75b
GH-96975: Skip incomplete frames in PyEval_GetFrame (GH-97003) 2022-09-22 09:16:52 -07:00
Matthias Görgens 6ba686d97f
gh-96678: Avoid undefined behaviour in test (#96672) 2022-09-13 13:15:11 +01:00
Pablo Galindo Salgado aa37ffda29
bpo-45445: Revert "bpo-45445: Fail if an invalid X-option is provided in the command line (GH-28823)" (#94745) 2022-07-31 16:33:56 +01:00
neonene 4a6bb30eb6
gh-90815: Fix test_embed for Windows PGO build with mimalloc (GH-94790)
Fixes the failure of PGO building with `mimalloc` on Windows, ensuring that `test_bpo20891` does not break profiling data (`python31*.pgc`).
2022-07-13 09:08:12 -07:00
neonene 38af903506
gh-91985: Ensure in-tree builds override platstdlib_dir in every path calculation (GH-93641) 2022-06-16 22:41:57 +01:00
Victor Stinner ada8b6d1b1
gh-57684: Add -P cmdline option and PYTHONSAFEPATH env var (#31542)
Add the -P command line option and the PYTHONSAFEPATH environment
variable to not prepend a potentially unsafe path to sys.path.

* Add sys.flags.safe_path flag.
* Add PyConfig.safe_path member.
* Programs/_bootstrap_python.c uses config.safe_path=0.
* Update subprocess._optim_args_from_interpreter_flags() to handle
  the -P command line option.
* Modules/getpath.py sets safe_path to 1 if a "._pth" file is
  present.
2022-05-06 01:34:11 +02:00
Eric Snow 81c72044a1
bpo-46541: Replace core use of _Py_IDENTIFIER() with statically initialized global objects. (gh-30928)
We're no longer using _Py_IDENTIFIER() (or _Py_static_string()) in any core CPython code.  It is still used in a number of non-builtin stdlib modules.

The replacement is: PyUnicodeObject (not pointer) fields under _PyRuntimeState, statically initialized as part of _PyRuntime.  A new _Py_GET_GLOBAL_IDENTIFIER() macro facilitates lookup of the fields (along with _Py_GET_GLOBAL_STRING() for non-identifier strings).

https://bugs.python.org/issue46541#msg411799 explains the rationale for this change.

The core of the change is in:

* (new) Include/internal/pycore_global_strings.h - the declarations for the global strings, along with the macros
* Include/internal/pycore_runtime_init.h - added the static initializers for the global strings
* Include/internal/pycore_global_objects.h - where the struct in pycore_global_strings.h is hooked into _PyRuntimeState
* Tools/scripts/generate_global_objects.py - added generation of the global string declarations and static initializers

I've also added a --check flag to generate_global_objects.py (along with make check-global-objects) to check for unused global strings.  That check is added to the PR CI config.

The remainder of this change updates the core code to use _Py_GET_GLOBAL_IDENTIFIER() instead of _Py_IDENTIFIER() and the related _Py*Id functions (likewise for _Py_GET_GLOBAL_STRING() instead of _Py_static_string()).  This includes adding a few functions where there wasn't already an alternative to _Py*Id(), replacing the _Py_Identifier * parameter with PyObject *.

The following are not changed (yet):

* stop using _Py_IDENTIFIER() in the stdlib modules
* (maybe) get rid of _Py_IDENTIFIER(), etc. entirely -- this may not be doable as at least one package on PyPI using this (private) API
* (maybe) intern the strings during runtime init

https://bugs.python.org/issue46541
2022-02-08 13:39:07 -07:00
Victor Stinner 6415e2ee49
bpo-46417: _testembed.c avoids Py_SetProgramName() (GH-30732)
* _testembed_Py_Initialize() now uses the PyConfig API, rather than
  deprecated Py_SetProgramName().
* Reduce INIT_LOOPS from 16 to 4: test_embed now takes 8.7 seconds
  rather than 14.7 seconds.
2022-01-21 02:12:18 +01:00
Victor Stinner e9e3eab0b8
bpo-46417: Finalize structseq types at exit (GH-30645)
Add _PyStructSequence_FiniType() and _PyStaticType_Dealloc()
functions to finalize a structseq static type in Py_Finalize().
Currrently, these functions do nothing if Python is built in release
mode.

Clear static types:

* AsyncGenHooksType: sys.set_asyncgen_hooks()
* FlagsType: sys.flags
* FloatInfoType: sys.float_info
* Hash_InfoType: sys.hash_info
* Int_InfoType: sys.int_info
* ThreadInfoType: sys.thread_info
* UnraisableHookArgsType: sys.unraisablehook
* VersionInfoType: sys.version
* WindowsVersionType: sys.getwindowsversion()
2022-01-21 01:42:25 +01:00
Dong-hee Na 196b53eb1e
bpo-45189: Drop the "list_frozen" command from _test_embed. (GH-30273) 2021-12-28 11:05:50 +09:00
Victor Stinner a6c3b0faa1
bpo-45954: Rename PyConfig.no_debug_ranges to code_debug_ranges (GH-29886)
Rename PyConfig.no_debug_ranges to PyConfig.code_debug_ranges and
invert the value.

Document -X no_debug_ranges and PYTHONNODEBUGRANGES env var in
PyConfig.code_debug_ranges documentation.
2021-12-02 11:43:37 +01:00
Nikita Sobolev 401d25e92f
bpo-45666: Use `%S` for MSVC and `%s` elsewhere for `swprintf` in `_testembed.c` (GH-29341)
Co-authored-by: Ken Jin <28750310+Fidget-Spinner@users.noreply.github.com>
2021-11-01 20:11:34 +08:00
Nikita Sobolev aad48062ef
bpo-45666: fixes warning with `swprintf` and `%s` (GH-29307) 2021-10-31 10:13:54 +08:00
Eric Snow 074fa57506
bpo-45395: Make custom frozen modules additions instead of replacements. (gh-28778)
Currently custom modules (the array set on PyImport_FrozenModules) replace all the frozen stdlib modules. That can be problematic and is unlikely to be what the user wants. This change treats the custom frozen modules as additions instead. They take precedence over all other frozen modules except for those needed to bootstrap the import system. If the "code" field of an entry in the custom array is NULL then that frozen module is treated as disabled, which allows a custom entry to disable a frozen stdlib module.

This change allows us to get rid of is_essential_frozen_module() and simplifies the logic for which frozen modules should be ignored.

https://bugs.python.org/issue45395
2021-10-28 15:04:33 -06:00
Eric Snow 6afb285ff0
bpo-45020: Add tests for the -X "frozen_modules" option. (gh-28997)
We hadn't explicitly added any tests for this, so here they are.

https://bugs.python.org/issue45020
2021-10-25 15:26:41 -06:00
Victor Stinner aac29af678
bpo-45434: pyport.h no longer includes <stdlib.h> (GH-28914)
Include <stdlib.h> explicitly in C files.

Python.h includes <wchar.h>.
2021-10-13 19:25:53 +02:00
Pablo Galindo Salgado db2b6a20cd
bpo-45445: Fail if an invalid X-option is provided in the command line (GH-28823) 2021-10-13 18:08:19 +01:00
Victor Stinner 5e2c32e08e
bpo-40413: test_embed tests calling Py_RunMain() multiple times (GH-28466)
Calling Py_InitializeFromConfig()+Py_RunMain() multiple times must
not crash.

Cleanup also test_get_argc_argv().
2021-09-20 10:30:02 +02:00
Ammar Askar 4823d9a512
bpo-43950: Add option to opt-out of PEP-657 (GH-27023)
Co-authored-by: Pablo Galindo <Pablogsal@gmail.com>
Co-authored-by: Batuhan Taskaya <batuhanosmantaskaya@gmail.com>
Co-authored-by: Ammar Askar <ammar@ammaraskar.com>
2021-07-07 20:07:12 +01:00
Victor Stinner 48e3a1d95a
bpo-44434: Remove useless calls to PyThread_exit_thread() (GH-26943)
Remove useless calls to PyThread_exit_thread() in two unit tests of
_testcapi and _testembed modules.

Co-authored-by: Alexey Izbyshev <izbyshev@ispras.ru>
2021-06-29 02:03:30 +02:00
Victor Stinner 489699ca05
bpo-44441: _PyImport_Fini2() resets PyImport_Inittab (GH-26874)
Py_RunMain() now resets PyImport_Inittab to its initial value at
exit. It must be possible to call PyImport_AppendInittab() or
PyImport_ExtendInittab() at each Python initialization.
2021-06-23 14:13:27 +02:00
Victor Stinner eaede0ded7
bpo-44131: Test Py_FrozenMain() (GH-26126)
* Add test_frozenmain to test_embed
* Add Programs/test_frozenmain.py
* Add Programs/freeze_test_frozenmain.py
* Add Programs/test_frozenmain.h
* Add make regen-test-frozenmain
* Add test_frozenmain command to Programs/_testembed
* _testembed.c: add error(msg) function
2021-05-17 23:48:35 +02:00
Victor Stinner 6cd0446ef7
bpo-44113: Deprecate old functions to config Python init (GH-26060)
Deprecate the following functions to configure the Python
initialization:

* PySys_AddWarnOption()
* PySys_AddWarnOptionUnicode()
* PySys_AddXOption()
* PySys_HasWarnOptions()
* Py_SetPath()
* Py_SetProgramName()
* Py_SetPythonHome()
* Py_SetStandardStreamEncoding()
* _Py_SetProgramFullPath()

Use the new PyConfig API of the Python Initialization Configuration
instead (PEP 587).
2021-05-12 23:59:25 +02:00
Victor Stinner 307745aa42
bpo-43445: Add frozen modules to sys.stdlib_module_names (GH-24798)
Add frozen modules to sys.stdlib_module_names. For example, add
"_frozen_importlib" and "_frozen_importlib_external" names.

Add "list_frozen" command to Programs/_testembed.
2021-03-10 11:14:07 +01:00
Victor Stinner 11d13e83ab
bpo-42882: Add test_embed.test_unicode_id_init() (GH-24198)
Test that _PyUnicode_FromId() works when Python is initialized
multiples times.
2021-01-12 11:26:26 +01:00
Victor Stinner 9e1b828265
bpo-42260: Compute the path config in the main init (GH-23211)
The path configuration is now computed in the "main" initialization.
The core initialization no longer computes it.

* Add _PyConfig_Read() function to read the configuration without
  computing the path configuration.
* pyinit_core() no longer computes the path configuration: it is now
  computed by init_interp_main().
* The path configuration output members of PyConfig are now optional:

  * executable
  * base_executable
  * prefix
  * base_prefix
  * exec_prefix
  * base_exec_prefix

* _PySys_UpdateConfig() now skips NULL strings in PyConfig.
* _testembed: Rename test_set_config() to test_init_set_config() for
  consistency with other tests.
2020-11-10 13:21:52 +01:00
Victor Stinner 048a35659a
bpo-42260: Add _PyInterpreterState_SetConfig() (GH-23158)
* Inline _PyInterpreterState_SetConfig(): replace it with
  _PyConfig_Copy().
* Add _PyErr_SetFromPyStatus()
* Add _PyInterpreterState_GetConfigCopy()
* Add a new _PyInterpreterState_SetConfig() function.
* Add an unit which gets, modifies, and sets the config.
2020-11-05 00:45:56 +01:00
Konge daa0fe03a5
bpo-41162: Clear audit hooks later during finalization (GH-21222) 2020-07-03 22:06:46 +01:00
Pablo Galindo 436b648910
bpo-40939: Remove some extra references to PYTHONOLDPARSER (GH-20815)
Automerge-Triggered-By: @pablogsal
2020-06-11 10:08:05 -07:00
Pablo Galindo 1ed83adb0e
bpo-40939: Remove the old parser (GH-20768)
This commit removes the old parser, the deprecated parser module, the old parser compatibility flags and environment variables and all associated support code and documentation.
2020-06-11 17:30:46 +01:00
Victor Stinner e81f6e687d
bpo-40910: Export Py_GetArgcArgv() function (GH-20721)
Export explicitly the Py_GetArgcArgv() function to the C API and
document the function. Previously, it was exported implicitly which
no longer works since Python is built with -fvisibility=hidden.

* Add PyConfig._orig_argv member.
* Py_InitializeFromConfig() no longer calls _PyConfig_Write() twice.
* PyConfig_Read() no longer initializes Py_GetArgcArgv(): it is now
  _PyConfig_Write() responsibility.
* _PyConfig_Write() result type becomes PyStatus instead of void.
* Write an unit test on Py_GetArgcArgv().
2020-06-08 18:12:59 +02:00
Sandro Mani 8f023a2f66
bpo-40854: Allow overriding sys.platlibdir via PYTHONPLATLIBDIR env-var (GH-20605) 2020-06-08 17:28:11 +02:00
Victor Stinner 252346acd9
bpo-40453: Add PyConfig._isolated_subinterpreter (GH-19820)
An isolated subinterpreter cannot spawn threads, spawn a child
process or call os.fork().

* Add private _Py_NewInterpreter(isolated_subinterpreter) function.
* Add isolated=True keyword-only parameter to
  _xxsubinterpreters.create().
* Allow again os.fork() in "non-isolated" subinterpreters.
2020-05-01 11:33:44 +02:00
Victor Stinner 1def7754b7
bpo-40334: Rename PyConfig.use_peg to _use_peg_parser (GH-19670)
* Rename PyConfig.use_peg to _use_peg_parser
* Document PyConfig._use_peg_parser and mark it a deprecated
* Mark -X oldparser option and PYTHONOLDPARSER env var as deprecated
  in the documentation.
* Add use_old_parser() and skip_if_new_parser() to test.support
* Remove sys.flags.use_peg: use_old_parser() uses
  _testinternalcapi.get_configs() instead.
* Enhance test_embed tests
* subprocess._args_from_interpreter_flags() copies -X oldparser
2020-04-23 03:03:24 +02:00
Pablo Galindo c5fc156852
bpo-40334: PEP 617 implementation: New PEG parser for CPython (GH-19503)
Co-authored-by: Guido van Rossum <guido@python.org>
Co-authored-by: Lysandros Nikolaou <lisandrosnik@gmail.com>
2020-04-22 23:29:27 +01:00
Victor Stinner 62183b8d6d
bpo-40268: Remove explicit pythread.h includes (#19529)
Remove explicit pythread.h includes: it is always included
by Python.h.
2020-04-15 02:04:42 +02:00
Victor Stinner e5014be049
bpo-40268: Remove a few pycore_pystate.h includes (GH-19510) 2020-04-14 17:52:15 +02:00
Victor Stinner 3225b9f973
bpo-39877: Remove useless PyEval_InitThreads() calls (GH-18883)
Py_Initialize() calls PyEval_InitThreads() since Python 3.7. It's no
longer needed to call it explicitly.
2020-03-09 20:56:57 +01:00