Commit Graph

1039 Commits

Author SHA1 Message Date
Kai Zhang 228c92eb5c
gh-99582: freeze `zipimport` into `_bootstrap_python` (#99583)
Co-authored-by: Kumar Aditya <59607654+kumaraditya303@users.noreply.github.com>
2022-12-10 15:05:56 +05:30
Eric Snow 1160001b34
gh-81057: Move Threading-Related Globals to _PyRuntimeState (#100084)
https://github.com/python/cpython/issues/81057
2022-12-08 17:50:58 -07:00
Eric Snow bc8cdf8c3d
gh-81057: Move Ceval Trampoline Globals to _PyRuntimeState (gh-100083)
https://github.com/python/cpython/issues/81057
2022-12-08 17:17:20 -07:00
Eric Snow 8a3f06c54b
gh-81057: Move time Globals to _PyRuntimeState (gh-100122)
https://github.com/python/cpython/issues/81057
2022-12-08 16:46:09 -07:00
Eric Snow cda9f0236f
gh-81057: Move OS-Related Globals to _PyRuntimeState (gh-100082)
https://github.com/python/cpython/issues/81057
2022-12-08 15:38:06 -07:00
Jason R. Coombs 003f341e99
Fix zipfile packaging after GH-98103 (GH-99797)
* Add zipfile and test_zipfile to list of packages. Fixes regression introduced in #98103.

* Restore support for py -m test.test_zipfile
2022-11-26 13:00:05 -05:00
Victor Stinner 55bad199cf
gh-79315: Add Include/cpython/memoryobject.h header (#99723)
Move non-limited C API from Include/memoryobject.h to a new
Include/cpython/memoryobject.h header file.
2022-11-23 15:44:42 +01:00
Erlend E. Aasland 12b5a3c5f5
gh-93649: Split float/long tests from _testcapimodule.c (GH-99549)
Automerge-Triggered-By: GH:erlend-aasland
2022-11-17 00:56:56 -08:00
Eric Snow 9db1e17c80
gh-81057: Move the global Dict-Related Versions to _PyRuntimeState (gh-99497)
We also move the global func version.

https://github.com/python/cpython/issues/81057
2022-11-16 10:37:29 -07:00
Christian Heimes 33f42c269f
gh-96269: static and shared ext need different deps (#96316) 2022-11-16 14:03:35 +01:00
Victor Stinner 9a7e9f9921
gh-99289: Add COMPILEALL_OPTS to Makefile (#99291)
Add COMPILEALL_OPTS variable in Makefile to override compileall
options (default: -j0) in "make install". Also merge the compileall
commands into a single command building PYC files for the all
optimization levels (0, 1, 2) at once.

Co-authored-by: Gregory P. Smith <greg@krypto.org>
2022-11-14 13:43:45 +01:00
Eric Snow 67807cfc87
gh-81057: Move the Allocators to _PyRuntimeState (gh-99217)
The global allocators were stored in 3 static global variables: _PyMem_Raw, _PyMem, and _PyObject.  State for the "small block" allocator was stored in another 13.  That makes a total of 16 global variables. We are moving all 16 to the _PyRuntimeState struct as part of the work for gh-81057.  (If PEP 684 is accepted then we will follow up by moving them all to PyInterpreterState.)

https://github.com/python/cpython/issues/81057
2022-11-11 16:30:46 -07:00
Miro Hrončok 6abec1caff
gh-98707: configure --with-system-libmpdec and --with-system-expat no longer include vendored headers (#98711) 2022-11-11 19:59:30 +01:00
Eric Snow 52f91c642b
gh-90868: Adjust the Generated Objects (gh-99223)
We do the following:

* move the generated _PyUnicode_InitStaticStrings() to its own file
* move the generated _PyStaticObjects_CheckRefcnt() to its own file
* include pycore_global_objects.h in extension modules instead of pycore_runtime_init.h

These changes help us avoid including things that aren't needed.

https://github.com/python/cpython/issues/90868
2022-11-08 10:03:03 -07:00
Brandt Bucher e99380cc39
GH-98831: Add `regen-cases` to `regen-all` (#99107) 2022-11-04 15:00:51 -07:00
Victor Stinner 0faa0ba240
gh-92584: Remove the distutils package (#99061)
Remove the distutils package. It was deprecated in Python 3.10 by PEP
632 "Deprecate distutils module". For projects still using distutils
and cannot be updated to something else, the setuptools project can
be installed: it still provides distutils.

* Remove Lib/distutils/ directory
* Remove test_distutils
* Remove references to distutils
* Skip test_check_c_globals and test_peg_generator since they use
  distutils
2022-11-03 19:27:27 +01:00
serge-sans-paille 8c4de57de9
gh-98948: Remove obsolete readelf dependency (#98949)
This got introduced in commit 5884449539
to determine if readline is already linked against curses or tinfo in
the setup.py, which is no longer present.
2022-11-03 14:31:37 +01:00
Guido van Rossum 41bc101dd6
GH-98831: "Generate" the interpreter (#98830)
The switch cases (really TARGET(opcode) macros) have been moved from ceval.c to generated_cases.c.h. That file is generated from instruction definitions in bytecodes.c (which impersonates a C file so the C code it contains can be edited without custom support in e.g. VS Code).

The code generator lives in Tools/cases_generator (it has a README.md explaining how it works). The DSL used to describe the instructions is a work in progress, described in https://github.com/faster-cpython/ideas/blob/main/3.12/interpreter_definition.md.

This is surely a work-in-progress. An easy next step could be auto-generating super-instructions.

**IMPORTANT: Merge Conflicts**

If you get a merge conflict for instruction implementations in ceval.c, your best bet is to port your changes to bytecodes.c. That file looks almost the same as the original cases, except instead of `TARGET(NAME)` it uses `inst(NAME)`, and the trailing `DISPATCH()` call is omitted (the code generator adds it automatically).
2022-11-02 21:31:26 -07:00
Brett Cannon 22bab74c8f
gh-97731: Specify the full path to the docs for `make docclean` (GH-98982)
Specify the full path to the docs for `make docclean`
This is to have `make clean` not error out on cross-builds.
2022-11-02 12:34:22 -07:00
Miro Hrončok 5a8c4b9464
gh-98776: Fix make regen-test-levenshtein for out-of-tree builds (GH-98779)
Fixes https://github.com/python/cpython/issues/98776

Automerge-Triggered-By: GH:erlend-aasland
2022-10-28 04:31:33 -07:00
Pablo Galindo Salgado 1f737edb67
gh-96143: Move the perf trampoline files to the Python directory (#98675) 2022-10-25 23:34:22 +01:00
Victor Stinner 1863302d61
gh-97669: Create Tools/build/ directory (#97963)
Create Tools/build/ directory. Move the following scripts from
Tools/scripts/ to Tools/build/:

* check_extension_modules.py
* deepfreeze.py
* freeze_modules.py
* generate_global_objects.py
* generate_levenshtein_examples.py
* generate_opcode_h.py
* generate_re_casefix.py
* generate_sre_constants.py
* generate_stdlib_module_names.py
* generate_token.py
* parse_html5_entities.py
* smelly.py
* stable_abi.py
* umarshal.py
* update_file.py
* verify_ensurepip_wheels.py

Update references to these scripts.
2022-10-17 12:01:00 +02:00
Victor Stinner 0895c2a066
gh-97669: Create Tools/patchcheck/ directory (#98186)
Move patchcheck.py, reindent.py and untabify.py scripts to a new
Tools/patchcheck/ directory.
2022-10-12 10:09:21 +02:00
Jelle Zijlstra a54a69989e
gh-94808: Fix regex on exotic platforms (#98036)
The test failed on a buildbot because the pointer was only 7 hex characters. To be safe,
I bumped it down to 3: 4 in case we have 32-bit platforms, and 3 in case the pointer is very small.
2022-10-07 15:39:53 -07:00
Łukasz Langa bbc7cd649a
gh-97008: Add a Python implementation of AttributeError and NameError suggestions (#97022)
Relevant tests moved from test_exceptions to test_traceback to be able to
compare both implementations.

Co-authored-by: Carl Friedrich Bolz-Tereick <cfbolz@gmx.de>
2022-10-04 15:31:16 -07:00
Skip Montanaro 46eecc3b33
gh-97731: fix distclean target to clean docs (#97732)
fix distclean target to clean docs
2022-10-04 07:38:19 -07:00
Christian Heimes 1fc8bd3710
gh-95853: Multiple ops and debug for wasm_build.py (#96744) 2022-09-11 09:51:23 +02:00
Pablo Galindo Salgado 6d791a9736
gh-96143: Allow Linux perf profiler to see Python calls (GH-96123)
⚠️  ⚠️ Note for reviewers, hackers and fellow systems/low-level/compiler engineers ⚠️ ⚠️ 

If you have a lot of experience with this kind of shenanigans and want to improve the **first** version, **please make a PR against my branch** or **reach out by email** or **suggest code changes directly on GitHub**. 

If you have any **refinements or optimizations** please, wait until the first version is merged before starting hacking or proposing those so we can keep this PR productive.
2022-08-30 10:11:18 -07:00
Pablo Galindo Salgado 5b070c0d40
gh-95973: Add a new --with-dsymutil option to link debug information in macOS (GH-95974)
Automerge-Triggered-By: GH:pablogsal
2022-08-26 17:49:41 -07:00
Christian Heimes 8c1dbad36f
gh-96269: Fix build dependency on AIX (GH-96304) 2022-08-26 12:31:58 +02:00
Mark Shannon a4a9f2e879
GH-96177: Move GIL and eval breaker code out of ceval.c into ceval_gil.c. (GH-96204) 2022-08-24 14:21:01 +01:00
Christian Heimes 6087f491ea
gh-95853: WASM: better version and asset handling in scripts (GH-96045)
- support EMSDK tot-upstream and git releases
- allow WASM assents for wasm64-emscripten and WASI. This makes single file distributions on WASI easier.
- decouple WASM assets from browser builds
2022-08-19 08:36:12 +02:00
Kevin Modzelewski 214eb2cce5
gh-90536: Add support for the BOLT post-link binary optimizer (gh-95908)
* Add support for the BOLT post-link binary optimizer

Using [bolt](https://github.com/llvm/llvm-project/tree/main/bolt)
provides a fairly large speedup without any code or functionality
changes. It provides roughly a 1% speedup on pyperformance, and a
4% improvement on the Pyston web macrobenchmarks.

It is gated behind an `--enable-bolt` configure arg because not all
toolchains and environments are supported. It has been tested on a
Linux x86_64 toolchain, using llvm-bolt built from the LLVM 14.0.6
sources (their binary distribution of this version did not include bolt).

Compared to [a previous attempt](https://github.com/faster-cpython/ideas/issues/224),
this commit uses bolt's preferred "instrumentation" approach, as well as adds some non-PIE
flags which enable much better optimizations from bolt.

The effects of this change are a bit more dependent on CPU microarchitecture
than other changes, since it optimizes i-cache behavior which seems
to be a bit more variable between architectures. The 1%/4% numbers
were collected on an Intel Skylake CPU, and on an AMD Zen 3 CPU I
got a slightly larger speedup (2%/4%), and on a c6i.xlarge EC2 instance
I got a slightly lower speedup (1%/3%).

The low speedup on pyperformance is not entirely unexpected, because
BOLT improves i-cache behavior, and the benchmarks in the pyperformance
suite are small and tend to fit in i-cache.

This change uses the existing pgo profiling task (`python -m test --pgo`),
though I was able to measure about a 1% macrobenchmark improvement by
using the macrobenchmarks as the training task. I personally think that
both the PGO and BOLT tasks should be updated to use macrobenchmarks,
but for the sake of splitting up the work this PR uses the existing pgo task.

* Simplify the build flags

* Add a NEWS entry

* Update Makefile.pre.in

Co-authored-by: Dong-hee Na <donghee.na92@gmail.com>

* Update configure.ac

Co-authored-by: Dong-hee Na <donghee.na92@gmail.com>

* Add myself to ACKS

* Add docs

* Other review comments

* fix tab/space issue

* Make it more clear that --enable-bolt is experimental

* Add link to bolt's github page

Co-authored-by: Dong-hee Na <donghee.na92@gmail.com>
2022-08-19 06:33:54 +09:00
Eric Snow 6f6a4e6cc5
gh-90928: Statically Initialize the Keywords Tuple in Clinic-Generated Code (gh-95860)
We only statically initialize for core code and builtin modules.  Extension modules still create
the tuple at runtime.  We'll solve that part of interpreter isolation separately.

This change includes generated code. The non-generated changes are in:

* Tools/clinic/clinic.py
* Python/getargs.c
* Include/cpython/modsupport.h
* Makefile.pre.in (re-generate global strings after running clinic)
* very minor tweaks to Modules/_codecsmodule.c and Python/Python-tokenize.c

All other changes are generated code (clinic, global strings).
2022-08-11 15:25:49 -06:00
Victor Stinner 398ed84dc4
gh-93744: Remove configure --with-cxx-main option (#95651)
Remove the "configure --with-cxx-main" build option: it didn't work
for many years. Remove the MAINCC variable from configure and
Makefile.

The MAINCC variable was added by the issue gh-42471: commit
0f48d98b74. Previously, --with-cxx-main
was named --with-cxx.

Keep CXX and LDCXXSHARED variables, even if they are no longer used
by Python build system.
2022-08-05 13:26:58 +02:00
Christian Heimes 0fe645d6fd
gh-95174: Add pthread stubs for WASI (GH-95234)
Co-authored-by: Brett Cannon <brett@python.org>
2022-07-27 20:28:06 +02:00
Aivars Kalvāns 8c88e360e7
gh-95005: Replace PyAccu with PyUnicodeWriter (gh-95006) 2022-07-27 17:43:34 +09:00
Jason R. Coombs 3e718cf880
gh-95218: Move tests for importlib.resources into test_importlib.resources. (#95219)
* gh-95218: Move tests for importlib.resources into test_importlib.resources.

* Also update makefile

* Include test_importlib/resources in code ownership rule.
2022-07-24 20:53:10 -04:00
Christian Heimes 41a5b84eaf
gh-89758: Track all socket module dependencies (#95157) 2022-07-23 11:34:50 +02:00
Christian Heimes 81dca70d70
gh-93939: Build C extensions without setup.py (GH-94474)
Combines GH-93940, GH-94452, and GH-94433
2022-07-14 00:51:49 -07:00
Christian Heimes f5c02afaff
gh-94759: Create LCOV report with branch coverage (GH-94793) 2022-07-13 16:35:28 +02:00
Christian Heimes 18ef240a2c
gh-93649: Fix dependencies of _testcapi (#94695)
- header files are located in $(srcdir)
- dependencies must not list C files that are also in a makesetup Setup file
- generate SRCDIRS for OOT builds
2022-07-09 13:01:23 +02:00
Petr Viktorin be862b4e55
gh-93649: Split vectorcall testing from _testcapimodule.c (GH-94549)
The `_testcapimodule.c` file is getting too large to work with effectively.
This PR lays out a general structure of how tests can be split up, with more splitting to come later if the structure is OK.

Vectorcall tests aren't the biggest issue -- it's just an area I want to work on next, so I'm starting here.
An issue specific to vectorcall tests is that it wasn't clear that e.g. `MethodDescriptor2` is related to testing vectorcall: the `/* Test PEP 590 */` section had an ambiguous end. Separate file should make things like this much clearer.
OTOH, for some pieces it might not be clear where they should be -- I left `meth_fastcall` with tests of the other calling conventions. IMO, even with the ambiguity it's still worth it to split the huge file up.

I'm not sure about the buildsystem changes, hopefully CI will tell me what's wrong.

@vstinner, @markshannon: Do you think this is a good idea?

Automerge-Triggered-By: GH:encukou
2022-07-08 08:56:26 -07:00
Kumar Aditya 71697664d7
GH-90699: Move generated static initializer to pycore_runtime_generated.h (GH-94051) 2022-07-07 13:04:05 -07:00
Christian Heimes 7bd67d1d88
gh-93939: Add script to check extension modules (#94545)
Add script ``Tools/scripts/check_modules.py`` to check and validate builtin
and shared extension modules. The script also handles ``Modules/Setup`` and
will eventually replace ``setup.py``.

Co-authored-by: Victor Stinner <vstinner@python.org>
Co-authored-by: Erlend Egeberg Aasland <erlend.aasland@protonmail.com>
2022-07-05 09:25:45 +02:00
Christian Heimes 6485c3c726
gh-94404: Use module CFLAGS before PY_STDMODULE_CFLAGS (GH-94413)
``PY_STDMODULE_CFLAGS`` may contain include directories with system
headers. This can break compiling with built-in libmpdec.
2022-06-29 15:44:14 +02:00
Christian Heimes 44fa03d748
gh-93939: Create and install scripts in Makefile (GH-94324)
Scripts for 2to3, idle, and pydoc are now created and installed by make.
2022-06-28 14:56:06 +02:00
Christian Heimes 71868a0066
gh-90005: Rename MODULE_EGG variable to MODULE_EGG_STATE (GH-94301)
It makes it easier to look for module states in sysconfig without
special casing suffixes "_CFLAGS", "_DEPS", "_LDFLAGS", "_OBJS",
and "CTYPES_MALLOC_CLOSURE".
2022-06-26 22:52:06 -07:00
Christian Heimes bb8b931385
gh-90005: Port _ctypes to PY_STDLIB_MOD (GH-32229)
Co-authored-by: Erlend Egeberg Aasland <erlend.aasland@innova.no>

Automerge-Triggered-By: GH:tiran
2022-06-26 04:04:43 -07:00
Pablo Galindo Salgado 8e6eccabe8
Add ABI dump script (#94135) 2022-06-23 22:52:43 +01:00