Commit Graph

1430 Commits

Author SHA1 Message Date
Noam Cohen a371a7e03e
gh-95023: Added os.setns and os.unshare functions (#95046)
Added os.setns and os.unshare to easily switch between namespaces
on Linux.

Co-authored-by: Christian Heimes <christian@python.org>
Co-authored-by: CAM Gerlach <CAM.Gerlach@Gerlach.CAM>
Co-authored-by: Victor Stinner <vstinner@python.org>
2022-10-20 11:08:54 +02:00
Dong-hee Na 83d84e67cd
gh-96761: Fix build process of the clang compiler for _bootstrap_python (gh-96945)
Co-authored-by: Matthias Goergens <matthias.goergens@gmail.com>
2022-09-23 11:00:44 +09:00
Dong-hee Na e47b96c44f
gh-89536: Use ThinLTO policy if possible (gh-96766) 2022-09-16 19:40:05 +09:00
Christian Heimes a36235d5c7
gh-84461: Omit resource mod and getresuid funcs on Emscripten (GH-96303) 2022-09-11 10:10:00 +02:00
Pablo Galindo Salgado 6d791a9736
gh-96143: Allow Linux perf profiler to see Python calls (GH-96123)
⚠️  ⚠️ Note for reviewers, hackers and fellow systems/low-level/compiler engineers ⚠️ ⚠️ 

If you have a lot of experience with this kind of shenanigans and want to improve the **first** version, **please make a PR against my branch** or **reach out by email** or **suggest code changes directly on GitHub**. 

If you have any **refinements or optimizations** please, wait until the first version is merged before starting hacking or proposing those so we can keep this PR productive.
2022-08-30 10:11:18 -07:00
Dong-hee Na af368a7db4
gh-96191: Update the configure file to use GitHub issue (gh-96211) 2022-08-29 14:01:37 +09:00
Pablo Galindo Salgado 5b070c0d40
gh-95973: Add a new --with-dsymutil option to link debug information in macOS (GH-95974)
Automerge-Triggered-By: GH:pablogsal
2022-08-26 17:49:41 -07:00
Christian Heimes 6087f491ea
gh-95853: WASM: better version and asset handling in scripts (GH-96045)
- support EMSDK tot-upstream and git releases
- allow WASM assents for wasm64-emscripten and WASI. This makes single file distributions on WASI easier.
- decouple WASM assets from browser builds
2022-08-19 08:36:12 +02:00
Kevin Modzelewski 214eb2cce5
gh-90536: Add support for the BOLT post-link binary optimizer (gh-95908)
* Add support for the BOLT post-link binary optimizer

Using [bolt](https://github.com/llvm/llvm-project/tree/main/bolt)
provides a fairly large speedup without any code or functionality
changes. It provides roughly a 1% speedup on pyperformance, and a
4% improvement on the Pyston web macrobenchmarks.

It is gated behind an `--enable-bolt` configure arg because not all
toolchains and environments are supported. It has been tested on a
Linux x86_64 toolchain, using llvm-bolt built from the LLVM 14.0.6
sources (their binary distribution of this version did not include bolt).

Compared to [a previous attempt](https://github.com/faster-cpython/ideas/issues/224),
this commit uses bolt's preferred "instrumentation" approach, as well as adds some non-PIE
flags which enable much better optimizations from bolt.

The effects of this change are a bit more dependent on CPU microarchitecture
than other changes, since it optimizes i-cache behavior which seems
to be a bit more variable between architectures. The 1%/4% numbers
were collected on an Intel Skylake CPU, and on an AMD Zen 3 CPU I
got a slightly larger speedup (2%/4%), and on a c6i.xlarge EC2 instance
I got a slightly lower speedup (1%/3%).

The low speedup on pyperformance is not entirely unexpected, because
BOLT improves i-cache behavior, and the benchmarks in the pyperformance
suite are small and tend to fit in i-cache.

This change uses the existing pgo profiling task (`python -m test --pgo`),
though I was able to measure about a 1% macrobenchmark improvement by
using the macrobenchmarks as the training task. I personally think that
both the PGO and BOLT tasks should be updated to use macrobenchmarks,
but for the sake of splitting up the work this PR uses the existing pgo task.

* Simplify the build flags

* Add a NEWS entry

* Update Makefile.pre.in

Co-authored-by: Dong-hee Na <donghee.na92@gmail.com>

* Update configure.ac

Co-authored-by: Dong-hee Na <donghee.na92@gmail.com>

* Add myself to ACKS

* Add docs

* Other review comments

* fix tab/space issue

* Make it more clear that --enable-bolt is experimental

* Add link to bolt's github page

Co-authored-by: Dong-hee Na <donghee.na92@gmail.com>
2022-08-19 06:33:54 +09:00
Christian Heimes 32ac98e899
gh-95853: Add script to automate WASM build (GH-95828)
Automate WASM build with a new Python script. The script provides
several build profiles with configure flags for Emscripten flavors
and WASI. The script can detect and use Emscripten SDK and WASI SDK from
default locations or env vars.

``configure`` now detects Node arguments and creates HOSTRUNNER
arguments for Node 16. It also sets correct arguments for
``wasm64-emscripten``.

Co-authored-by: Brett Cannon <brett@python.org>
2022-08-13 21:56:08 +02:00
Victor Stinner 398ed84dc4
gh-93744: Remove configure --with-cxx-main option (#95651)
Remove the "configure --with-cxx-main" build option: it didn't work
for many years. Remove the MAINCC variable from configure and
Makefile.

The MAINCC variable was added by the issue gh-42471: commit
0f48d98b74. Previously, --with-cxx-main
was named --with-cxx.

Keep CXX and LDCXXSHARED variables, even if they are no longer used
by Python build system.
2022-08-05 13:26:58 +02:00
Christian Heimes 0fe645d6fd
gh-95174: Add pthread stubs for WASI (GH-95234)
Co-authored-by: Brett Cannon <brett@python.org>
2022-07-27 20:28:06 +02:00
Christian Heimes 8b24d60f1b
gh-95174: WASI: skip missing sockets functions (GH-95179) 2022-07-27 08:19:23 +02:00
Christian Heimes daa64d6a59
gh-95085: Promote Emscripten and WASI to PEP 11 tier 3 (GH-95086) 2022-07-27 08:18:34 +02:00
Christian Heimes 9af7f87d76
gh-94801: Fix regression in configure's CPPFLAGS handling (GH-95288) 2022-07-26 21:39:04 +02:00
Christian Heimes 0d35a59ce3
gh-95174: Handle missing dup() and constants in WASI (GH-95229)
- check for ``dup()`` libc function
- handle missing ``F_DUPFD`` in ``dup2()`` replacement function
- add workaround for WASI libc bug in MSG_TRUNC
- ESHUTDOWN is missing, use EPIPE instead
- POLLPRI is missing, define as 0 (no-op)
2022-07-26 11:16:51 +02:00
Christian Heimes 97b4121f93
gh-94847: Don't force inlining in debug builds of libmpdec (GH-94848) 2022-07-18 08:47:03 +02:00
Christian Heimes 944ff8c57c
gh-94801: Use custom flags to check for headers and libs (#94802) 2022-07-15 14:48:01 +02:00
Christian Heimes 81dca70d70
gh-93939: Build C extensions without setup.py (GH-94474)
Combines GH-93940, GH-94452, and GH-94433
2022-07-14 00:51:49 -07:00
Christian Heimes 18ef240a2c
gh-93649: Fix dependencies of _testcapi (#94695)
- header files are located in $(srcdir)
- dependencies must not list C files that are also in a makesetup Setup file
- generate SRCDIRS for OOT builds
2022-07-09 13:01:23 +02:00
Christian Heimes 29f86d6c28
gh-94642: Remove -D_XOPEN_SOURCE from more pkg-config CFLAGS (#94657)
Some pkg-config pc files define CFLAGS with -D_XOPEN_SOURCE=600. We always want _XOPEN_SOURCE=700.
2022-07-07 14:47:32 +02:00
Christian Heimes 9c60b25a0d
gh-94644: Define HAVE_NCURSESW in pkg-config branch (#94651) 2022-07-07 12:57:12 +02:00
Christian Heimes e925241d95
gh-90005: Port readline and curses to PY_STDLIB_MOD (GH-94452)
Co-authored-by: Erlend Egeberg Aasland <erlend.aasland@protonmail.com>
2022-07-06 11:56:25 +02:00
Christian Heimes ec5e253556
gh-90005: Port _dbm module to PY_STDLIB_MOD (GH-94433) 2022-07-01 21:48:38 +02:00
Christian Heimes d6acdc1b60
gh-90005-ffi: Fix building _ctypes without pkg-config (GH-94451)
The fallback path did not set LIBFFI_LIBS variable to link with ``-lffi``.
2022-07-01 09:23:41 +02:00
Christian Heimes 67d208fbee
gh-93491: Fix PEP 11 tier detection for FreeBSD (GH-94441) 2022-06-30 18:04:10 +02:00
Christian Heimes 71868a0066
gh-90005: Rename MODULE_EGG variable to MODULE_EGG_STATE (GH-94301)
It makes it easier to look for module states in sysconfig without
special casing suffixes "_CFLAGS", "_DEPS", "_LDFLAGS", "_OBJS",
and "CTYPES_MALLOC_CLOSURE".
2022-06-26 22:52:06 -07:00
Christian Heimes bb8b931385
gh-90005: Port _ctypes to PY_STDLIB_MOD (GH-32229)
Co-authored-by: Erlend Egeberg Aasland <erlend.aasland@innova.no>

Automerge-Triggered-By: GH:tiran
2022-06-26 04:04:43 -07:00
Christian Heimes e69306f08b
gh-84461: Build Emscripten with WASM BigInt support (#94219) 2022-06-24 17:03:42 +02:00
Christian Heimes 8625802d85
gh-84461: Fix ctypes and test_ctypes on Emscripten (#94142)
- c_longlong and c_longdouble need experimental WASM bigint.
- Skip tests that need threading
- Define ``CTYPES_MAX_ARGCOUNT`` for Emscripten. libffi-emscripten 2022-06-23 supports up to 1000 args.
2022-06-24 12:40:43 +02:00
Christian Heimes 774ef28814
gh-84461: Silence some compiler warnings on WASM (GH-93978) 2022-06-20 13:34:40 +02:00
Christian Heimes 084023ccbe
gh-84461: Fix circulare dependency on BUILDPYTHON (GH-93977) 2022-06-18 10:19:48 +02:00
Christian Heimes cd543d0bc9
gh-90473: Include stdlib dir in wasmtime PYTHONPATH (GH-93797) 2022-06-14 12:15:13 +02:00
Christian Heimes dc5e02b2f4
gh-84461: Use HOSTRUNNER to run regression tests (GH-93694)
Co-authored-by: Brett Cannon <brett@python.org>
2022-06-11 08:42:23 +02:00
Christian Heimes 3124d9a5aa
gh-93491: Add support tier detection to configure (GH-93492)
Co-authored-by: Adam Turner <9087854+AA-Turner@users.noreply.github.com>
Co-authored-by: Steve Dower <steve.dower@microsoft.com>
Co-authored-by: Erlend Egeberg Aasland <erlend.aasland@protonmail.com>
2022-06-10 15:25:33 +02:00
Christian Heimes 22df2e0322
gh-90473: Define HOSTRUNNER for WASI (GH-93606) 2022-06-08 20:18:46 +02:00
Illia Volochii d1e2e0e1b2
gh-93475: Expose FICLONE and FICLONERANGE constants in fcntl (#93478) 2022-06-06 23:21:58 +02:00
Christian Heimes 069c96f84c
gh-90473: Skip and document more failing tests on WASI (GH-93436)
- Mark more ``umask()`` cases
- ``dup()`` is not supported
- ``/dev/null`` is not available
- document missing features
- mark more modules as not available
2022-06-03 00:44:48 +02:00
Eric Snow caa279d6fd
bpo-40514: Drop EXPERIMENTAL_ISOLATED_SUBINTERPRETERS (gh-93185)
This was added for bpo-40514 (gh-84694) to test out a per-interpreter GIL. However, it has since proven unnecessary to keep the experiment in the repo. (It can be done as a branch in a fork like normal.) So here we are removing:

* the configure option
* the macro
* the code enabled by the macro
2022-05-27 17:38:01 -06:00
Kumar Aditya cb04a09d2d
GH-93207: Remove HAVE_STDARG_PROTOTYPES configure check for stdarg.h (#93215) 2022-05-27 13:30:45 +02:00
Victor Stinner 71d8775fee
gh-93202: Always use %zd printf formatter (#93201)
Python now always use the ``%zu`` and ``%zd`` printf formats to
format a size_t or Py_ssize_t number. Building Python 3.12 requires a
C11 compiler, so these printf formats are now always supported.

* PyObject_Print() and _PyObject_Dump() now use the printf %zd format
  to display an object reference count.
* Update PY_FORMAT_SIZE_T comment.
* Remove outdated notes about the %zd format in PyBytes_FromFormat()
  and PyUnicode_FromFormat() documentations.
* configure no longer checks for the %zd format and no longer defines
  PY_FORMAT_SIZE_T macro in pyconfig.h.
* pymacconfig.h no longer undefines PY_FORMAT_SIZE_T: macOS 10.4 is
  no longer supported. Python 3.12 now requires macOS 10.6 (Snow
  Leopard) or newer.
2022-05-25 14:21:36 +02:00
Christian Heimes 137fd3d88a
gh-90473: Decrease recursion limit and skip tests on WASI (GH-92803) 2022-05-19 12:43:16 +02:00
Christian Heimes d81d57e959
gh-90473: Increase stack size, disable obmalloc on WASI (GH-92732) 2022-05-13 08:36:01 +02:00
Erlend Egeberg Aasland 269e726723
gh-90005: Cleanup after GH-31698 (#91642) 2022-05-09 11:00:20 +02:00
Dong-hee Na d284e8b3e3
Update CPyhton configuration for 3.12 (#92451)
* Update CPyhton configuration for 3.12

* Fix PC/pyconfig.h

* Add expect failure
2022-05-08 13:44:12 +01:00
Soumendra Ganguly ae553b3561
bpo-41818: Add os.login_tty() for *nix. (#29658)
* Add `os.login_tty(fd)` for Unix.

Reviewed-by: Christian Heimes <christian@python.org>
Signed-off-by: Gregory P. Smith [Google LLC] <greg@krypto.org>
2022-05-05 09:04:44 -07:00
Łukasz Langa 6c25bf07e8
gh-89452: Prefer gdbm-compat over ndbm (#92208)
This makes macOS gdbm provided by Homebrew not segfault through correct
selection of the linked library (-lgdbm_compat) *AND* the correct ndbm-style
header (gdbm-ndbm.h instead of the invalid ndbm.h).
2022-05-03 18:27:38 +02:00
Ethan Smith d1de10784d
gh-84461: Add HOSTRUNNER for program to run Python executable (GH-91931)
`HOSTRUNNER` is a program which can be used to run `BUILDPYTHON` for the host platform (for example, `python.js` requires `node`).

Also change depedencies from `build_all` to `all` so that targets which can't build everything (e.g. WASM) can still run `buildbottest` and `pythoninfo`.

cc @tiran
2022-04-28 05:14:19 -07:00
Christian Heimes 92c1037afc
gh-84461: Add --enable-wasm-pthreads and more file systems (GH-91820) 2022-04-23 14:59:33 +02:00
Christian Heimes 9b5ca5405e
gh-84461: Add sys._emscripten_info, improve docs and build (gh-91781) 2022-04-23 09:52:16 +02:00