Commit Graph

190 Commits

Author SHA1 Message Date
Gregory P. Smith b0f89cb431
gh-96512: Move int_max_str_digits setting to PyConfig (#96944)
It had to live as a global outside of PyConfig for stable ABI reasons in
the pre-3.12 backports.

This removes the `_Py_global_config_int_max_str_digits` and gets rid of
the equivalent field in the internal `struct _is PyInterpreterState` as
code can just use the existing nested config struct within that.

Adds tests to verify unique settings and configs in subinterpreters.
2022-10-03 13:55:45 -07:00
Brandt Bucher 8fd2c3b75b
GH-96975: Skip incomplete frames in PyEval_GetFrame (GH-97003) 2022-09-22 09:16:52 -07:00
Matthias Görgens 6ba686d97f
gh-96678: Avoid undefined behaviour in test (#96672) 2022-09-13 13:15:11 +01:00
Pablo Galindo Salgado aa37ffda29
bpo-45445: Revert "bpo-45445: Fail if an invalid X-option is provided in the command line (GH-28823)" (#94745) 2022-07-31 16:33:56 +01:00
neonene 4a6bb30eb6
gh-90815: Fix test_embed for Windows PGO build with mimalloc (GH-94790)
Fixes the failure of PGO building with `mimalloc` on Windows, ensuring that `test_bpo20891` does not break profiling data (`python31*.pgc`).
2022-07-13 09:08:12 -07:00
Irit Katriel 324d01944d
gh-94485: Set line number of module's RESUME instruction to 0, as specified by PEP 626 (GH-94552)
Co-authored-by: Mark Shannon <mark@hotpy.org>
2022-07-05 14:38:44 +02:00
Dennis Sweeney 5fcfdd87c9
GH-91432: Specialize FOR_ITER (GH-91713)
* Adds FOR_ITER_LIST and FOR_ITER_RANGE specializations.

* Adds _PyLong_AssignValue() internal function to avoid temporary boxing of ints.
2022-06-21 11:19:26 +01:00
neonene 38af903506
gh-91985: Ensure in-tree builds override platstdlib_dir in every path calculation (GH-93641) 2022-06-16 22:41:57 +01:00
Ken Jin b083450f88
GH-93429: Merge `LOAD_METHOD` back into `LOAD_ATTR` (GH-93430) 2022-06-14 11:36:22 +01:00
Ken Jin d52ffc1d1f
gh-93382: Cache result of `PyCode_GetCode` in codeobject (GH-93383)
Co-authored-by: Kumar Aditya <59607654+kumaraditya303@users.noreply.github.com>
Co-authored-by: Dennis Sweeney <36520290+sweeneyde@users.noreply.github.com>
2022-06-04 00:41:18 +08:00
Mark Shannon e48ac9c100
GH-90690: Remove `PRECALL` instruction (GH-92925) 2022-05-19 11:05:26 +01:00
Victor Stinner ada8b6d1b1
gh-57684: Add -P cmdline option and PYTHONSAFEPATH env var (#31542)
Add the -P command line option and the PYTHONSAFEPATH environment
variable to not prepend a potentially unsafe path to sys.path.

* Add sys.flags.safe_path flag.
* Add PyConfig.safe_path member.
* Programs/_bootstrap_python.c uses config.safe_path=0.
* Update subprocess._optim_args_from_interpreter_flags() to handle
  the -P command line option.
* Modules/getpath.py sets safe_path to 1 if a "._pth" file is
  present.
2022-05-06 01:34:11 +02:00
Inada Naoki 6dcfd6c5e3
gh-78214: marshal: Stabilize FLAG_REF usage (GH-8226)
Use FLAG_REF always for interned strings.

Refcounts of interned string is very unstable.
When compiling same source, refcounts of interned string in the output may be 1 or >1.
It makes FLAG_REF usage unstable.

To help reproducible build, use FLAG_REF for interned string even if refcnt(obj)==1.
2022-05-04 10:01:15 +09:00
Mark Shannon 944fffee89
GH-88116: Use a compact format to represent end line and column offsets. (GH-91666)
* Stores all location info in linetable to conform to PEP 626.

* Remove column table from code objects.

* Remove end-line table from code objects.

* Document new location table format
2022-04-21 16:10:37 +01:00
Irit Katriel a00518d9ad
bpo-47120: Replace the JUMP_ABSOLUTE opcode by the relative JUMP_BACKWARD (GH-32115) 2022-03-31 14:14:15 +01:00
Brandt Bucher f193631387
bpo-46841: Use inline caching for calls (GH-31709) 2022-03-07 11:45:00 -08:00
Brandt Bucher 05a8bc1c94
bpo-46841: Use inline caching for attribute accesses (GH-31640) 2022-03-03 15:31:00 -08:00
Mark Shannon 3b0f1c5a71
bpo-46841: Use inline cache for `BINARY_SUBSCR`. (GH-31618) 2022-03-01 16:00:34 +00:00
Kumar Aditya 0d9b565e62
Propagate errors (however unlikely) from _Py_Deepfreeze_Init() (GH-31596) 2022-02-26 08:35:03 -08:00
Kumar Aditya 4dc746310b
bpo-46430: Fix memory leak in interned strings of deep-frozen modules (GH-31549) 2022-02-24 17:54:06 +01:00
Mark Shannon 59585d6b2e
bpo-46329: Streamline calling sequence a bit. (GH-31465)
* Move handling of bound-methods to PRECALL.

* Remove call_shape.postcall_shrink

* Remove call_shape.callable

* Remove call_shape.callable. Change CALL oparg to match PRECALL oparg.

* Move KW_NAMES before PRECALL.

* Update opcode docs in dis.rst
2022-02-21 18:26:47 +00:00
Mark Shannon cf345e945f
bpo-46329: Change calling sequence (again) (GH-31373)
* Change calling sequence: Add PUSH_NULL. Merge PRECALL_FUNCTION and PRECALL_METHOD into PRECALL.
2022-02-18 17:19:08 +00:00
Eric Snow 81c72044a1
bpo-46541: Replace core use of _Py_IDENTIFIER() with statically initialized global objects. (gh-30928)
We're no longer using _Py_IDENTIFIER() (or _Py_static_string()) in any core CPython code.  It is still used in a number of non-builtin stdlib modules.

The replacement is: PyUnicodeObject (not pointer) fields under _PyRuntimeState, statically initialized as part of _PyRuntime.  A new _Py_GET_GLOBAL_IDENTIFIER() macro facilitates lookup of the fields (along with _Py_GET_GLOBAL_STRING() for non-identifier strings).

https://bugs.python.org/issue46541#msg411799 explains the rationale for this change.

The core of the change is in:

* (new) Include/internal/pycore_global_strings.h - the declarations for the global strings, along with the macros
* Include/internal/pycore_runtime_init.h - added the static initializers for the global strings
* Include/internal/pycore_global_objects.h - where the struct in pycore_global_strings.h is hooked into _PyRuntimeState
* Tools/scripts/generate_global_objects.py - added generation of the global string declarations and static initializers

I've also added a --check flag to generate_global_objects.py (along with make check-global-objects) to check for unused global strings.  That check is added to the PR CI config.

The remainder of this change updates the core code to use _Py_GET_GLOBAL_IDENTIFIER() instead of _Py_IDENTIFIER() and the related _Py*Id functions (likewise for _Py_GET_GLOBAL_STRING() instead of _Py_static_string()).  This includes adding a few functions where there wasn't already an alternative to _Py*Id(), replacing the _Py_Identifier * parameter with PyObject *.

The following are not changed (yet):

* stop using _Py_IDENTIFIER() in the stdlib modules
* (maybe) get rid of _Py_IDENTIFIER(), etc. entirely -- this may not be doable as at least one package on PyPI using this (private) API
* (maybe) intern the strings during runtime init

https://bugs.python.org/issue46541
2022-02-08 13:39:07 -07:00
Mark Shannon 89fd7c3452
bpo-46329: Split calls into precall and call instructions. (GH-30855)
* Add PRECALL_FUNCTION opcode.

* Move 'call shape' varaibles into struct.

* Replace CALL_NO_KW and CALL_KW with KW_NAMES and CALL instructions.

* Specialize for builtin methods taking using the METH_FASTCALL | METH_KEYWORDS protocol.

* Allow kwnames for specialized calls to builtin types.

* Specialize calls to tuple(arg) and str(arg).
2022-01-28 12:42:30 +00:00
Kumar Aditya c7f810b34d
bpo-46476: Fix memory leak in code objects generated by deepfreeze (GH-30853)
Add _Py_Deepfreeze_Fini() and _PyStaticCode_Dealloc() functions.
2022-01-27 14:03:47 +01:00
Victor Stinner 6415e2ee49
bpo-46417: _testembed.c avoids Py_SetProgramName() (GH-30732)
* _testembed_Py_Initialize() now uses the PyConfig API, rather than
  deprecated Py_SetProgramName().
* Reduce INIT_LOOPS from 16 to 4: test_embed now takes 8.7 seconds
  rather than 14.7 seconds.
2022-01-21 02:12:18 +01:00
Victor Stinner e9e3eab0b8
bpo-46417: Finalize structseq types at exit (GH-30645)
Add _PyStructSequence_FiniType() and _PyStaticType_Dealloc()
functions to finalize a structseq static type in Py_Finalize().
Currrently, these functions do nothing if Python is built in release
mode.

Clear static types:

* AsyncGenHooksType: sys.set_asyncgen_hooks()
* FlagsType: sys.flags
* FloatInfoType: sys.float_info
* Hash_InfoType: sys.hash_info
* Int_InfoType: sys.int_info
* ThreadInfoType: sys.thread_info
* UnraisableHookArgsType: sys.unraisablehook
* VersionInfoType: sys.version
* WindowsVersionType: sys.getwindowsversion()
2022-01-21 01:42:25 +01:00
Victor Stinner ea1a54506b
bpo-46303: Move fileutils.h private functions to internal C API (GH-30484)
Move almost all private functions of Include/cpython/fileutils.h to
the internal C API Include/internal/pycore_fileutils.h.

Only keep _Py_fopen_obj() in Include/cpython/fileutils.h, since it's
used by _testcapi which must not use the internal C API.

Move EncodeLocaleEx() and DecodeLocaleEx() functions from _testcapi
to _testinternalcapi, since the C API moved to the internal C API.
2022-01-11 11:56:16 +01:00
Mark Shannon e028ae99ec
bpo-45923: Handle call events in bytecode (GH-30364)
* Add a RESUME instruction to handle "call" events.
2022-01-06 13:09:25 +00:00
Dong-hee Na 196b53eb1e
bpo-45189: Drop the "list_frozen" command from _test_embed. (GH-30273) 2021-12-28 11:05:50 +09:00
Mark Shannon 9f8f45144b
bpo-44525: Split calls into PRECALL and CALL (GH-30011)
* Add 3 new opcodes for calls: PRECALL_METHOD, CALL_NO_KW, CALL_KW.

* Update specialization to handle new CALL opcodes.

* Specialize call to method descriptors.

* Remove old CALL opcodes: CALL_FUNCTION, CALL_METHOD, CALL_METHOD_KW, CALL_FUNCTION_KW.
2021-12-14 18:22:44 +00:00
Christian Heimes eb483c46d6
bpo-45949: Pure Python freeze module for cross builds (GH-29899) 2021-12-13 20:48:46 +01:00
Christian Heimes 84ca1232b0
bpo-45950: Introduce Bootstrap Python again (#29859)
The build system now uses a :program:`_bootstrap_python` interpreter for
freezing and deepfreezing again. To speed up build process the build tools
:program:`_bootstrap_python` and :program:`_freeze_module` are no longer
build with LTO.

Cross building depends on a build Python interpreter, which must have same
version and bytecode as target host Python.
2021-12-03 16:01:11 +01:00
Victor Stinner a6c3b0faa1
bpo-45954: Rename PyConfig.no_debug_ranges to code_debug_ranges (GH-29886)
Rename PyConfig.no_debug_ranges to PyConfig.code_debug_ranges and
invert the value.

Document -X no_debug_ranges and PYTHONNODEBUGRANGES env var in
PyConfig.code_debug_ranges documentation.
2021-12-02 11:43:37 +01:00
Brandt Bucher 69a6173d64
Remove Programs/_freeze_importlib (#29357) 2021-11-01 18:09:26 -07:00
Nikita Sobolev 401d25e92f
bpo-45666: Use `%S` for MSVC and `%s` elsewhere for `swprintf` in `_testembed.c` (GH-29341)
Co-authored-by: Ken Jin <28750310+Fidget-Spinner@users.noreply.github.com>
2021-11-01 20:11:34 +08:00
Nikita Sobolev aad48062ef
bpo-45666: fixes warning with `swprintf` and `%s` (GH-29307) 2021-10-31 10:13:54 +08:00
Eric Snow 074fa57506
bpo-45395: Make custom frozen modules additions instead of replacements. (gh-28778)
Currently custom modules (the array set on PyImport_FrozenModules) replace all the frozen stdlib modules. That can be problematic and is unlikely to be what the user wants. This change treats the custom frozen modules as additions instead. They take precedence over all other frozen modules except for those needed to bootstrap the import system. If the "code" field of an entry in the custom array is NULL then that frozen module is treated as disabled, which allows a custom entry to disable a frozen stdlib module.

This change allows us to get rid of is_essential_frozen_module() and simplifies the logic for which frozen modules should be ignored.

https://bugs.python.org/issue45395
2021-10-28 15:04:33 -06:00
Brandt Bucher 82a662e521
bpo-44511: Improve the bytecode for class and mapping patterns (GH-26922)
* Refactor mapping patterns and speed up class patterns.

* Simplify MATCH_KEYS and MATCH_CLASS.

* Add COPY opcode.
2021-10-27 10:45:35 +01:00
Eric Snow 6afb285ff0
bpo-45020: Add tests for the -X "frozen_modules" option. (gh-28997)
We hadn't explicitly added any tests for this, so here they are.

https://bugs.python.org/issue45020
2021-10-25 15:26:41 -06:00
Victor Stinner aac29af678
bpo-45434: pyport.h no longer includes <stdlib.h> (GH-28914)
Include <stdlib.h> explicitly in C files.

Python.h includes <wchar.h>.
2021-10-13 19:25:53 +02:00
Pablo Galindo Salgado db2b6a20cd
bpo-45445: Fail if an invalid X-option is provided in the command line (GH-28823) 2021-10-13 18:08:19 +01:00
Eric Snow 08285d563e
bpo-45020: Identify which frozen modules are actually aliases. (gh-28655)
In the list of generated frozen modules at the top of Tools/scripts/freeze_modules.py, you will find that some of the modules have a different name than the module (or .py file) that is actually frozen. Let's call each case an "alias". Aliases do not come into play until we get to the (generated) list of modules in Python/frozen.c. (The tool for freezing modules, Programs/_freeze_module, is only concerned with the source file, not the module it will be used for.)

Knowledge of which frozen modules are aliases (and the identity of the original module) normally isn't important. However, this information is valuable when we go to set __file__ on frozen stdlib modules. This change updates Tools/scripts/freeze_modules.py to map aliases to the original module name (or None if not a stdlib module) in Python/frozen.c. We also add a helper function in Python/import.c to look up a frozen module's alias and add the result of that function to the frozen info returned from find_frozen().

https://bugs.python.org/issue45020
2021-10-05 11:26:37 -06:00
Victor Stinner 5e2c32e08e
bpo-40413: test_embed tests calling Py_RunMain() multiple times (GH-28466)
Calling Py_InitializeFromConfig()+Py_RunMain() multiple times must
not crash.

Cleanup also test_get_argc_argv().
2021-09-20 10:30:02 +02:00
Steve Dower 09b4ad11f3
bpo-45188: Windows now regenerates frozen modules at the start of build instead of late (GH-28322)
This will enable us to drop the frozen module header files from the repository.

It does currently cause many source files to be built twice, which just takes more time. For whoever comes to fix this in the future, the files shared between freeze_module and pythoncore should be put into a static library that is consumed by both.
2021-09-15 18:11:12 +01:00
Eric Snow 044e8d866f
bpo-45019: Add a tool to generate list of modules to include for frozen modules (gh-27980)
Frozen modules must be added to several files in order to work properly. Before this change this had to be done manually. Here we add a tool to generate the relevant lines in those files instead. This helps us avoid mistakes and omissions.

https://bugs.python.org/issue45019
2021-08-30 17:25:11 -06:00
Ammar Askar 4823d9a512
bpo-43950: Add option to opt-out of PEP-657 (GH-27023)
Co-authored-by: Pablo Galindo <Pablogsal@gmail.com>
Co-authored-by: Batuhan Taskaya <batuhanosmantaskaya@gmail.com>
Co-authored-by: Ammar Askar <ammar@ammaraskar.com>
2021-07-07 20:07:12 +01:00
Gabriele N. Tornetta 2f180ce2cb
bpo-44530: Add co_qualname field to PyCodeObject (GH-26941) 2021-07-07 12:21:51 +01:00
Pablo Galindo 98eee94421
bpo-43950: Add code.co_positions (PEP 657) (GH-26955)
This PR is part of PEP 657 and augments the compiler to emit ending
line numbers as well as starting and ending columns from the AST
into compiled code objects. This allows bytecodes to be correlated
to the exact source code ranges that generated them.

This information is made available through the following public APIs:

* The `co_positions` method on code objects.
* The C API function `PyCode_Addr2Location`.

Co-authored-by: Batuhan Taskaya <isidentical@gmail.com>
Co-authored-by: Ammar Askar <ammar@ammaraskar.com>
2021-07-02 15:10:11 +01:00
Batuhan Taskaya 1b28187a0e
bpo-44313: generate LOAD_ATTR/CALL_FUNCTION for top-level imported objects (GH-26677) 2021-06-30 23:53:36 +01:00