Commit Graph

1018 Commits

Author SHA1 Message Date
Victor Stinner 1863302d61
gh-97669: Create Tools/build/ directory (#97963)
Create Tools/build/ directory. Move the following scripts from
Tools/scripts/ to Tools/build/:

* check_extension_modules.py
* deepfreeze.py
* freeze_modules.py
* generate_global_objects.py
* generate_levenshtein_examples.py
* generate_opcode_h.py
* generate_re_casefix.py
* generate_sre_constants.py
* generate_stdlib_module_names.py
* generate_token.py
* parse_html5_entities.py
* smelly.py
* stable_abi.py
* umarshal.py
* update_file.py
* verify_ensurepip_wheels.py

Update references to these scripts.
2022-10-17 12:01:00 +02:00
Victor Stinner 0895c2a066
gh-97669: Create Tools/patchcheck/ directory (#98186)
Move patchcheck.py, reindent.py and untabify.py scripts to a new
Tools/patchcheck/ directory.
2022-10-12 10:09:21 +02:00
Jelle Zijlstra a54a69989e
gh-94808: Fix regex on exotic platforms (#98036)
The test failed on a buildbot because the pointer was only 7 hex characters. To be safe,
I bumped it down to 3: 4 in case we have 32-bit platforms, and 3 in case the pointer is very small.
2022-10-07 15:39:53 -07:00
Łukasz Langa bbc7cd649a
gh-97008: Add a Python implementation of AttributeError and NameError suggestions (#97022)
Relevant tests moved from test_exceptions to test_traceback to be able to
compare both implementations.

Co-authored-by: Carl Friedrich Bolz-Tereick <cfbolz@gmx.de>
2022-10-04 15:31:16 -07:00
Skip Montanaro 46eecc3b33
gh-97731: fix distclean target to clean docs (#97732)
fix distclean target to clean docs
2022-10-04 07:38:19 -07:00
Christian Heimes 1fc8bd3710
gh-95853: Multiple ops and debug for wasm_build.py (#96744) 2022-09-11 09:51:23 +02:00
Pablo Galindo Salgado 6d791a9736
gh-96143: Allow Linux perf profiler to see Python calls (GH-96123)
⚠️  ⚠️ Note for reviewers, hackers and fellow systems/low-level/compiler engineers ⚠️ ⚠️ 

If you have a lot of experience with this kind of shenanigans and want to improve the **first** version, **please make a PR against my branch** or **reach out by email** or **suggest code changes directly on GitHub**. 

If you have any **refinements or optimizations** please, wait until the first version is merged before starting hacking or proposing those so we can keep this PR productive.
2022-08-30 10:11:18 -07:00
Pablo Galindo Salgado 5b070c0d40
gh-95973: Add a new --with-dsymutil option to link debug information in macOS (GH-95974)
Automerge-Triggered-By: GH:pablogsal
2022-08-26 17:49:41 -07:00
Christian Heimes 8c1dbad36f
gh-96269: Fix build dependency on AIX (GH-96304) 2022-08-26 12:31:58 +02:00
Mark Shannon a4a9f2e879
GH-96177: Move GIL and eval breaker code out of ceval.c into ceval_gil.c. (GH-96204) 2022-08-24 14:21:01 +01:00
Christian Heimes 6087f491ea
gh-95853: WASM: better version and asset handling in scripts (GH-96045)
- support EMSDK tot-upstream and git releases
- allow WASM assents for wasm64-emscripten and WASI. This makes single file distributions on WASI easier.
- decouple WASM assets from browser builds
2022-08-19 08:36:12 +02:00
Kevin Modzelewski 214eb2cce5
gh-90536: Add support for the BOLT post-link binary optimizer (gh-95908)
* Add support for the BOLT post-link binary optimizer

Using [bolt](https://github.com/llvm/llvm-project/tree/main/bolt)
provides a fairly large speedup without any code or functionality
changes. It provides roughly a 1% speedup on pyperformance, and a
4% improvement on the Pyston web macrobenchmarks.

It is gated behind an `--enable-bolt` configure arg because not all
toolchains and environments are supported. It has been tested on a
Linux x86_64 toolchain, using llvm-bolt built from the LLVM 14.0.6
sources (their binary distribution of this version did not include bolt).

Compared to [a previous attempt](https://github.com/faster-cpython/ideas/issues/224),
this commit uses bolt's preferred "instrumentation" approach, as well as adds some non-PIE
flags which enable much better optimizations from bolt.

The effects of this change are a bit more dependent on CPU microarchitecture
than other changes, since it optimizes i-cache behavior which seems
to be a bit more variable between architectures. The 1%/4% numbers
were collected on an Intel Skylake CPU, and on an AMD Zen 3 CPU I
got a slightly larger speedup (2%/4%), and on a c6i.xlarge EC2 instance
I got a slightly lower speedup (1%/3%).

The low speedup on pyperformance is not entirely unexpected, because
BOLT improves i-cache behavior, and the benchmarks in the pyperformance
suite are small and tend to fit in i-cache.

This change uses the existing pgo profiling task (`python -m test --pgo`),
though I was able to measure about a 1% macrobenchmark improvement by
using the macrobenchmarks as the training task. I personally think that
both the PGO and BOLT tasks should be updated to use macrobenchmarks,
but for the sake of splitting up the work this PR uses the existing pgo task.

* Simplify the build flags

* Add a NEWS entry

* Update Makefile.pre.in

Co-authored-by: Dong-hee Na <donghee.na92@gmail.com>

* Update configure.ac

Co-authored-by: Dong-hee Na <donghee.na92@gmail.com>

* Add myself to ACKS

* Add docs

* Other review comments

* fix tab/space issue

* Make it more clear that --enable-bolt is experimental

* Add link to bolt's github page

Co-authored-by: Dong-hee Na <donghee.na92@gmail.com>
2022-08-19 06:33:54 +09:00
Eric Snow 6f6a4e6cc5
gh-90928: Statically Initialize the Keywords Tuple in Clinic-Generated Code (gh-95860)
We only statically initialize for core code and builtin modules.  Extension modules still create
the tuple at runtime.  We'll solve that part of interpreter isolation separately.

This change includes generated code. The non-generated changes are in:

* Tools/clinic/clinic.py
* Python/getargs.c
* Include/cpython/modsupport.h
* Makefile.pre.in (re-generate global strings after running clinic)
* very minor tweaks to Modules/_codecsmodule.c and Python/Python-tokenize.c

All other changes are generated code (clinic, global strings).
2022-08-11 15:25:49 -06:00
Victor Stinner 398ed84dc4
gh-93744: Remove configure --with-cxx-main option (#95651)
Remove the "configure --with-cxx-main" build option: it didn't work
for many years. Remove the MAINCC variable from configure and
Makefile.

The MAINCC variable was added by the issue gh-42471: commit
0f48d98b74. Previously, --with-cxx-main
was named --with-cxx.

Keep CXX and LDCXXSHARED variables, even if they are no longer used
by Python build system.
2022-08-05 13:26:58 +02:00
Christian Heimes 0fe645d6fd
gh-95174: Add pthread stubs for WASI (GH-95234)
Co-authored-by: Brett Cannon <brett@python.org>
2022-07-27 20:28:06 +02:00
Aivars Kalvāns 8c88e360e7
gh-95005: Replace PyAccu with PyUnicodeWriter (gh-95006) 2022-07-27 17:43:34 +09:00
Jason R. Coombs 3e718cf880
gh-95218: Move tests for importlib.resources into test_importlib.resources. (#95219)
* gh-95218: Move tests for importlib.resources into test_importlib.resources.

* Also update makefile

* Include test_importlib/resources in code ownership rule.
2022-07-24 20:53:10 -04:00
Christian Heimes 41a5b84eaf
gh-89758: Track all socket module dependencies (#95157) 2022-07-23 11:34:50 +02:00
Christian Heimes 81dca70d70
gh-93939: Build C extensions without setup.py (GH-94474)
Combines GH-93940, GH-94452, and GH-94433
2022-07-14 00:51:49 -07:00
Christian Heimes f5c02afaff
gh-94759: Create LCOV report with branch coverage (GH-94793) 2022-07-13 16:35:28 +02:00
Christian Heimes 18ef240a2c
gh-93649: Fix dependencies of _testcapi (#94695)
- header files are located in $(srcdir)
- dependencies must not list C files that are also in a makesetup Setup file
- generate SRCDIRS for OOT builds
2022-07-09 13:01:23 +02:00
Petr Viktorin be862b4e55
gh-93649: Split vectorcall testing from _testcapimodule.c (GH-94549)
The `_testcapimodule.c` file is getting too large to work with effectively.
This PR lays out a general structure of how tests can be split up, with more splitting to come later if the structure is OK.

Vectorcall tests aren't the biggest issue -- it's just an area I want to work on next, so I'm starting here.
An issue specific to vectorcall tests is that it wasn't clear that e.g. `MethodDescriptor2` is related to testing vectorcall: the `/* Test PEP 590 */` section had an ambiguous end. Separate file should make things like this much clearer.
OTOH, for some pieces it might not be clear where they should be -- I left `meth_fastcall` with tests of the other calling conventions. IMO, even with the ambiguity it's still worth it to split the huge file up.

I'm not sure about the buildsystem changes, hopefully CI will tell me what's wrong.

@vstinner, @markshannon: Do you think this is a good idea?

Automerge-Triggered-By: GH:encukou
2022-07-08 08:56:26 -07:00
Kumar Aditya 71697664d7
GH-90699: Move generated static initializer to pycore_runtime_generated.h (GH-94051) 2022-07-07 13:04:05 -07:00
Christian Heimes 7bd67d1d88
gh-93939: Add script to check extension modules (#94545)
Add script ``Tools/scripts/check_modules.py`` to check and validate builtin
and shared extension modules. The script also handles ``Modules/Setup`` and
will eventually replace ``setup.py``.

Co-authored-by: Victor Stinner <vstinner@python.org>
Co-authored-by: Erlend Egeberg Aasland <erlend.aasland@protonmail.com>
2022-07-05 09:25:45 +02:00
Christian Heimes 6485c3c726
gh-94404: Use module CFLAGS before PY_STDMODULE_CFLAGS (GH-94413)
``PY_STDMODULE_CFLAGS`` may contain include directories with system
headers. This can break compiling with built-in libmpdec.
2022-06-29 15:44:14 +02:00
Christian Heimes 44fa03d748
gh-93939: Create and install scripts in Makefile (GH-94324)
Scripts for 2to3, idle, and pydoc are now created and installed by make.
2022-06-28 14:56:06 +02:00
Christian Heimes 71868a0066
gh-90005: Rename MODULE_EGG variable to MODULE_EGG_STATE (GH-94301)
It makes it easier to look for module states in sysconfig without
special casing suffixes "_CFLAGS", "_DEPS", "_LDFLAGS", "_OBJS",
and "CTYPES_MALLOC_CLOSURE".
2022-06-26 22:52:06 -07:00
Christian Heimes bb8b931385
gh-90005: Port _ctypes to PY_STDLIB_MOD (GH-32229)
Co-authored-by: Erlend Egeberg Aasland <erlend.aasland@innova.no>

Automerge-Triggered-By: GH:tiran
2022-06-26 04:04:43 -07:00
Pablo Galindo Salgado 8e6eccabe8
Add ABI dump script (#94135) 2022-06-23 22:52:43 +01:00
Pablo Galindo Salgado dd39e29ccb
Add ABI dump Makefile target (#94136) 2022-06-22 22:22:27 +01:00
Victor Stinner c1fb12e5af
gh-54781: Move Lib/tkinter/test/test_ttk/ to Lib/test/test_ttk/ (#94070)
* Move Lib/tkinter/test/test_tkinter/ to Lib/test/test_tkinter/.
* Move Lib/tkinter/test/test_ttk/ to Lib/test/test_ttk/.
* Add Lib/test/test_ttk/__init__.py based on test_ttk_guionly.py.
* Add Lib/test/test_tkinter/__init__.py
* Remove old Lib/test/test_tk.py.
* Remove old Lib/test/test_ttk_guionly.py.
* Add __main__ sub-modules.
* Update imports and update references to rename files.
2022-06-22 22:23:37 +02:00
Victor Stinner 616fa3465d
gh-54781: Move Lib/lib2to3/tests/ to Lib/test/test_lib2to3/ (#94049)
* Move Lib/lib2to3/tests/ to Lib/test/test_lib2to3/.
* Remove Lib/test/test_lib2to3.py.
* Update imports.
* all_project_files(): use different paths and sort files
  to make the tests more reproducible.
* Update references to tests.
2022-06-21 15:21:22 +02:00
Dennis Sweeney 5fcfdd87c9
GH-91432: Specialize FOR_ITER (GH-91713)
* Adds FOR_ITER_LIST and FOR_ITER_RANGE specializations.

* Adds _PyLong_AssignValue() internal function to avoid temporary boxing of ints.
2022-06-21 11:19:26 +01:00
Victor Stinner c735d54534
gh-93839: Move Lib/unttest/test/ to Lib/test/test_unittest/ (#94043)
* Move Lib/unittest/test/ to Lib/test/test_unittest/
* Remove Lib/test/test_unittest.py
* Replace unittest.test with test.test_unittest
* Remove unittest.load_tests()
* Rewrite unittest __init__.py and __main__.py
* Update build system, CODEOWNERS, and wasm_assets.py
2022-06-21 10:27:59 +02:00
Victor Stinner d82e0bfe8b
gh-93839: Move Lib/ctypes/test/ to Lib/test/test_ctypes/ (#94041)
* Move Lib/ctypes/test/ to Lib/test/test_ctypes/
* Remove Lib/test/test_ctypes.py
* Update imports and build system.
2022-06-21 10:24:33 +02:00
Victor Stinner 27b9894033
gh-93937, C API: Move PyFrame_GetBack() to Python.h (#93938)
Move the follow functions and type from frameobject.h to pyframe.h,
so the standard <Python.h> provide frame getter functions:

* PyFrame_Check()
* PyFrame_GetBack()
* PyFrame_GetBuiltins()
* PyFrame_GetGenerator()
* PyFrame_GetGlobals()
* PyFrame_GetLasti()
* PyFrame_GetLocals()
* PyFrame_Type

Remove #include "frameobject.h" from many C files. It's no longer
needed.
2022-06-19 12:02:33 +02:00
Christian Heimes 7a2cc35e1c
gh-84461: Fix pydebug Emscripten browser builds (GH-93982)
wasm_assets script did not take the ABIFLAG flag of sysconfigdata into
account.
2022-06-18 14:51:50 +02:00
Christian Heimes 084023ccbe
gh-84461: Fix circulare dependency on BUILDPYTHON (GH-93977) 2022-06-18 10:19:48 +02:00
Ken Jin a51742ab82
gh-93911: Specialize `LOAD_ATTR_PROPERTY` (GH-93912) 2022-06-17 23:13:17 +08:00
Christian Heimes 243ed5439c
gh-93584: Make all install+tests targets depends on all (GH-93589)
All install targets use the "all" target as synchronization point to
prevent race conditions with PGO builds. PGO builds use recursive make,
which can lead to two parallel `./python setup.py build` processes that
step on each others toes.

"test" targets now correctly compile PGO build in a clean repo.
2022-06-08 15:16:20 +02:00
Mark Shannon fa2b8b75eb
Improve object stats (#92845)
* Add incref/decref stats

* Show ratios for allocation in summary
2022-05-16 14:35:11 +01:00
Inada Naoki f9c9354a7a
gh-92536: PEP 623: Remove wstr and legacy APIs from Unicode (GH-92537) 2022-05-12 14:48:38 +09:00
Victor Stinner da5727a120
gh-92651: Remove the Include/token.h header file (#92652)
Remove the token.h header file. There was never any public tokenizer
C API. The token.h header file was only designed to be used by Python
internals.

Move Include/token.h to Include/internal/pycore_token.h. Including
this header file now requires that the Py_BUILD_CORE macro is
defined. It no longer checks for the Py_LIMITED_API macro.

Rename functions:

* PyToken_OneChar() => _PyToken_OneChar()
* PyToken_TwoChars() => _PyToken_TwoChars()
* PyToken_ThreeChars() => _PyToken_ThreeChars()
2022-05-11 23:22:50 +02:00
Petr Viktorin 83bce8ef14
gh-91324: Convert the stable ABI manifest to TOML (GH-92026) 2022-04-29 16:18:08 +02:00
Ethan Smith d1de10784d
gh-84461: Add HOSTRUNNER for program to run Python executable (GH-91931)
`HOSTRUNNER` is a program which can be used to run `BUILDPYTHON` for the host platform (for example, `python.js` requires `node`).

Also change depedencies from `build_all` to `all` so that targets which can't build everything (e.g. WASM) can still run `buildbottest` and `pythoninfo`.

cc @tiran
2022-04-28 05:14:19 -07:00
Victor Stinner 64a54e511d
gh-91719: Add pycore_opcode.h internal header file (#91906)
Move the following API from Include/opcode.h (public C API) to a new
Include/internal/pycore_opcode.h header file (internal C API):

* EXTRA_CASES
* _PyOpcode_Caches
* _PyOpcode_Deopt
* _PyOpcode_Jump
* _PyOpcode_OpName
* _PyOpcode_RelativeJump
2022-04-26 00:14:30 +02:00
Victor Stinner 20cc695286
gh-64783: Fix signal.NSIG value on FreeBSD (#91929)
Fix signal.NSIG value on FreeBSD to accept signal numbers greater
than 32, like signal.SIGRTMIN and signal.SIGRTMAX.

* Add Py_NSIG constant.
* Add pycore_signal.h internal header file.
* _Py_Sigset_Converter() now includes the range of valid signals in
  the error message.
2022-04-26 00:13:31 +02:00
Serhiy Storchaka f912cc0e41
gh-91575: Add a script for generating data for case-insensitive matching in re (GH-91660)
Also test that all extra cases are in BMP.
2022-04-22 21:37:46 +03:00
Victor Stinner 8a4e519e78
gh-79315: Add Include/cpython/pythread.h header (#91798) 2022-04-21 23:00:42 +02:00
Victor Stinner 6f9addb5ba
gh-79315: Add Include/cpython/modsupport.h header (#91797) 2022-04-21 23:00:26 +02:00