Commit Graph

658 Commits

Author SHA1 Message Date
Petr Viktorin cd9a56c2b0
gh-103509: PEP 697 -- Limited C API for Extending Opaque Types (GH-103511)
Co-authored-by: Oleg Iarygin <oleg@arhadthedev.net>
Co-authored-by: Erlend E. Aasland <erlend.aasland@protonmail.com>
2023-05-04 09:56:53 +02:00
Dong-hee Na 5d9762e7cf
gh-101525: Fix make test if the --enable-bolt enabled (gh-103574) 2023-04-17 23:14:18 +09:00
Nikita Sobolev 7f3c106503
gh-103326: Remove `Python/importlib.h` (GH-103331)
Co-authored-by: Brett Cannon <brett@python.org>
Co-authored-by: Hugo van Kemenade <hugovk@users.noreply.github.com>
2023-04-12 11:27:14 -07:00
Matthias Görgens eff9f43924
gh-96821: Add config option `--with-strict-overflow` (#96823)
Co-authored-by: C.A.M. Gerlach <CAM.Gerlach@Gerlach.CAM>
Co-authored-by: Erlend E. Aasland <erlend.aasland@protonmail.com>
Co-authored-by: Adam Turner <9087854+AA-Turner@users.noreply.github.com>
Co-authored-by: Shantanu <hauntsaninja@gmail.com>
2023-03-04 13:39:52 -08:00
Jonathan Protzenko fcadc7e405
gh-99108: Import MD5 and SHA1 from HACL* (#102089)
Replaces our fallback non-OpenSSL MD5 and SHA1 implementations with those from HACL* as we've already done with SHA2.
2023-02-22 13:18:43 -08:00
Eli Schwartz 226484e475
gh-99942: correct the pkg-config/python-config flags for cygwin/android 2023-02-16 17:57:59 +00:00
Gregory P. Smith 0b13575e74
gh-99108: Refactor _sha256 & _sha512 into _sha2. (#101924)
This merges their code. They're backed by the same single HACL* static library, having them be a single module simplifies maintenance.

This should unbreak the wasm enscripten builds that currently fail due to linking in --whole-archive mode and the HACL* library appearing twice.

Long unnoticed error fixed: _sha512.SHA384Type was doubly assigned and was actually SHA512Type. Nobody depends on those internal names.

Also rename LIBHACL_ make vars to LIBHACL_SHA2_ in preperation for other future HACL things.
2023-02-15 22:08:20 -08:00
Jonathan Protzenko e5da9ab2c8
gh-99108: Import SHA2-384/512 from HACL* (#101707)
Replace the builtin hashlib implementations of SHA2-384 and SHA2-512
originally from LibTomCrypt with formally verified, side-channel resistant
code from the [HACL*](https://github.com/hacl-star/hacl-star/) project.
The builtins remain a fallback only used when OpenSSL does not provide them.
2023-02-14 01:25:16 -08:00
Erlend E. Aasland 2db2c4b455
gh-92547: Purge sqlite3_enable_shared_cache() detection from configure (#101873) 2023-02-13 13:36:42 +01:00
Jonathan Protzenko 1fcc0efdaa
gh-99108: Replace SHA2-224 & 256 with verified code from HACL* (#99109)
replacing hashlib primitives (for the non-OpenSSL case) with verified implementations from HACL*. This is the first PR in the series, and focuses specifically on SHA2-256 and SHA2-224.

This PR imports Hacl_Streaming_SHA2 into the Python tree. This is the HACL* implementation of SHA2, which combines a core implementation of SHA2 along with a layer of buffer management that allows updating the digest with any number of bytes. This supersedes the previous implementation in the tree.

@franziskuskiefer was kind enough to benchmark the changes: in addition to being verified (thus providing significant safety and security improvements), this implementation also provides a sizeable performance boost!

```
---------------------------------------------------------------
Benchmark                     Time             CPU   Iterations
---------------------------------------------------------------
Sha2_256_Streaming            3163 ns      3160 ns       219353     // this PR
LibTomCrypt_Sha2_256          5057 ns      5056 ns       136234     // library used by Python currently
``` 

The changes in this PR are as follows:
- import the subset of HACL* that covers SHA2-256/224 into `Modules/_hacl`
- rewire sha256module.c to use the HACL* implementation

Co-authored-by: Gregory P. Smith [Google LLC] <greg@krypto.org>
Co-authored-by: Erlend E. Aasland <erlend.aasland@protonmail.com>
2023-02-06 18:11:01 -08:00
Eric Snow c67b00534a
gh-101524: Split Up the _xxsubinterpreters Module (gh-101526)
This is step 1 in potentially dropping all the "channel"-related code. Channels have already been removed from PEP 554.

https://github.com/python/cpython/issues/101524
2023-02-03 18:14:43 -07:00
Dong-hee Na a958e7d35a
gh-101278: Drop -gdwarf-4 flag from the BOLT build (gh-101280) 2023-01-24 19:22:00 +09:00
Gregory Szorc 7589d713a1
gh-101060: conditionally add `-fno-reorder-blocks-and-partition` (gh-101061) 2023-01-24 18:34:44 +09:00
Gregory Szorc 79af40a403
gh-101047: Remove vestigial references to macOS libtool options (gh-101048)
LIBTOOL_CRUFT and OTHER_LIBTOOL_OPT variables have been unused since commit
19199830f7 in 2011.
2023-01-22 17:26:42 -05:00
Kushal Das 75c8133efe
gh-100340: Allows -Wno-int-conversion for wasm (#100341)
Fixes #100340 allows -Wno-int-conversion for wasm
2023-01-18 19:43:49 +01:00
Michał Górny 02a72f080d
gh-98636: Fix detecting gdbm_compat for _dbm module (#98643)
Fix the gdbm_compat library detection logic to actually check for
-lgdbm_compat independently of the ndbm detection.
This fixes the build failure with `--with-dbmliborder=gdbm`,
and implicit fallback to ndbm with the default value.
2023-01-11 22:46:28 +01:00
Dustin Spicuzza cc8748712e
gh-86082: bpo-41916: allow cross-compiled python to have -pthread set for CXX (#22525)
When cross-compiling, the compile/run test for -pthread always fails so -pthread
will never be automatically set without an override from the cache. ac_cv_pthread
can already be overridden, so do the same thing for ac_cv_cxx_thread.
2023-01-05 14:57:31 -08:00
Zachary Ware 2df82db485
gh-100540: Remove obsolete Modules/_ctypes/darwin/ dlfcn shim (GH-100541)
As far as I can tell, this hasn't been actually used since Mac OS X 10.2.
2022-12-29 16:13:28 -06:00
Zachary Ware 25590eb5de
gh-100540: Remove obsolete '--with-system-ffi' configure option (GH-100544)
It has had no effect on non-macOS platforms for a long time, and has had
the non-obvious effect of invoking `pkg_config` and not setting
`-DUSING_APPLE_OS_LIBFFI` on macOS since GH-22855.
2022-12-28 13:36:03 -06:00
Zachary Ware fd4b55ff17
gh-100540: Remove unnecessary '-DMACOSX' for ctypes on macOS (GH-100542)
The define was only used to protect a `#pragma clang diagnostic`
setting, which is already better guarded by `__clang__` anwyay.
2022-12-28 12:06:58 -06:00
Sam James e35ca417fe
gh-99086: Fix -Wstrict-prototypes, -Wimplicit-function-declaration warnings in configure.ac (#99406)
Follow up to 12078e78f6.
2022-11-26 18:38:49 +05:30
colorfulappl c450c8c9ed
gh-96002: Add functional test for Argument Clinic (#96178)
Co-authored-by: Kumar Aditya <59607654+kumaraditya303@users.noreply.github.com>
Co-authored-by: Erlend E. Aasland <erlend.aasland@protonmail.com>
2022-11-21 15:08:45 +01:00
Miro Hrončok 6abec1caff
gh-98707: configure --with-system-libmpdec and --with-system-expat no longer include vendored headers (#98711) 2022-11-11 19:59:30 +01:00
Noam Cohen 80c08d1cd6
gh-95389: expose popular ETHERTYPE_* constants in the socket module (#95390)
Co-authored-by: Erlend E. Aasland <erlend.aasland@protonmail.com>
Co-authored-by: CAM Gerlach <CAM.Gerlach@Gerlach.CAM>
2022-11-07 15:27:37 +01:00
Sam James 12078e78f6
gh-99086: Fix implicit int compiler warning in configure check for PTHREAD_SCOPE_SYSTEM (#99085) 2022-11-05 09:30:31 +01:00
serge-sans-paille 8c4de57de9
gh-98948: Remove obsolete readelf dependency (#98949)
This got introduced in commit 5884449539
to determine if readline is already linked against curses or tinfo in
the setup.py, which is no longer present.
2022-11-03 14:31:37 +01:00
Jelle Zijlstra f7241aaf6f
gh-98989: configure: add 3.11 to list of Pythons (#98988)
Closes #98989

Co-authored-by: Erlend E. Aasland <erlend.aasland@protonmail.com>
2022-11-02 10:05:42 -07:00
Pablo Galindo Salgado 1f737edb67
gh-96143: Move the perf trampoline files to the Python directory (#98675) 2022-10-25 23:34:22 +01:00
Noam Cohen a371a7e03e
gh-95023: Added os.setns and os.unshare functions (#95046)
Added os.setns and os.unshare to easily switch between namespaces
on Linux.

Co-authored-by: Christian Heimes <christian@python.org>
Co-authored-by: CAM Gerlach <CAM.Gerlach@Gerlach.CAM>
Co-authored-by: Victor Stinner <vstinner@python.org>
2022-10-20 11:08:54 +02:00
Dong-hee Na 83d84e67cd
gh-96761: Fix build process of the clang compiler for _bootstrap_python (gh-96945)
Co-authored-by: Matthias Goergens <matthias.goergens@gmail.com>
2022-09-23 11:00:44 +09:00
Dong-hee Na e47b96c44f
gh-89536: Use ThinLTO policy if possible (gh-96766) 2022-09-16 19:40:05 +09:00
Christian Heimes a36235d5c7
gh-84461: Omit resource mod and getresuid funcs on Emscripten (GH-96303) 2022-09-11 10:10:00 +02:00
Pablo Galindo Salgado 6d791a9736
gh-96143: Allow Linux perf profiler to see Python calls (GH-96123)
⚠️  ⚠️ Note for reviewers, hackers and fellow systems/low-level/compiler engineers ⚠️ ⚠️ 

If you have a lot of experience with this kind of shenanigans and want to improve the **first** version, **please make a PR against my branch** or **reach out by email** or **suggest code changes directly on GitHub**. 

If you have any **refinements or optimizations** please, wait until the first version is merged before starting hacking or proposing those so we can keep this PR productive.
2022-08-30 10:11:18 -07:00
Dong-hee Na af368a7db4
gh-96191: Update the configure file to use GitHub issue (gh-96211) 2022-08-29 14:01:37 +09:00
Pablo Galindo Salgado 5b070c0d40
gh-95973: Add a new --with-dsymutil option to link debug information in macOS (GH-95974)
Automerge-Triggered-By: GH:pablogsal
2022-08-26 17:49:41 -07:00
Christian Heimes 6087f491ea
gh-95853: WASM: better version and asset handling in scripts (GH-96045)
- support EMSDK tot-upstream and git releases
- allow WASM assents for wasm64-emscripten and WASI. This makes single file distributions on WASI easier.
- decouple WASM assets from browser builds
2022-08-19 08:36:12 +02:00
Kevin Modzelewski 214eb2cce5
gh-90536: Add support for the BOLT post-link binary optimizer (gh-95908)
* Add support for the BOLT post-link binary optimizer

Using [bolt](https://github.com/llvm/llvm-project/tree/main/bolt)
provides a fairly large speedup without any code or functionality
changes. It provides roughly a 1% speedup on pyperformance, and a
4% improvement on the Pyston web macrobenchmarks.

It is gated behind an `--enable-bolt` configure arg because not all
toolchains and environments are supported. It has been tested on a
Linux x86_64 toolchain, using llvm-bolt built from the LLVM 14.0.6
sources (their binary distribution of this version did not include bolt).

Compared to [a previous attempt](https://github.com/faster-cpython/ideas/issues/224),
this commit uses bolt's preferred "instrumentation" approach, as well as adds some non-PIE
flags which enable much better optimizations from bolt.

The effects of this change are a bit more dependent on CPU microarchitecture
than other changes, since it optimizes i-cache behavior which seems
to be a bit more variable between architectures. The 1%/4% numbers
were collected on an Intel Skylake CPU, and on an AMD Zen 3 CPU I
got a slightly larger speedup (2%/4%), and on a c6i.xlarge EC2 instance
I got a slightly lower speedup (1%/3%).

The low speedup on pyperformance is not entirely unexpected, because
BOLT improves i-cache behavior, and the benchmarks in the pyperformance
suite are small and tend to fit in i-cache.

This change uses the existing pgo profiling task (`python -m test --pgo`),
though I was able to measure about a 1% macrobenchmark improvement by
using the macrobenchmarks as the training task. I personally think that
both the PGO and BOLT tasks should be updated to use macrobenchmarks,
but for the sake of splitting up the work this PR uses the existing pgo task.

* Simplify the build flags

* Add a NEWS entry

* Update Makefile.pre.in

Co-authored-by: Dong-hee Na <donghee.na92@gmail.com>

* Update configure.ac

Co-authored-by: Dong-hee Na <donghee.na92@gmail.com>

* Add myself to ACKS

* Add docs

* Other review comments

* fix tab/space issue

* Make it more clear that --enable-bolt is experimental

* Add link to bolt's github page

Co-authored-by: Dong-hee Na <donghee.na92@gmail.com>
2022-08-19 06:33:54 +09:00
Christian Heimes 32ac98e899
gh-95853: Add script to automate WASM build (GH-95828)
Automate WASM build with a new Python script. The script provides
several build profiles with configure flags for Emscripten flavors
and WASI. The script can detect and use Emscripten SDK and WASI SDK from
default locations or env vars.

``configure`` now detects Node arguments and creates HOSTRUNNER
arguments for Node 16. It also sets correct arguments for
``wasm64-emscripten``.

Co-authored-by: Brett Cannon <brett@python.org>
2022-08-13 21:56:08 +02:00
Victor Stinner 398ed84dc4
gh-93744: Remove configure --with-cxx-main option (#95651)
Remove the "configure --with-cxx-main" build option: it didn't work
for many years. Remove the MAINCC variable from configure and
Makefile.

The MAINCC variable was added by the issue gh-42471: commit
0f48d98b74. Previously, --with-cxx-main
was named --with-cxx.

Keep CXX and LDCXXSHARED variables, even if they are no longer used
by Python build system.
2022-08-05 13:26:58 +02:00
Christian Heimes 0fe645d6fd
gh-95174: Add pthread stubs for WASI (GH-95234)
Co-authored-by: Brett Cannon <brett@python.org>
2022-07-27 20:28:06 +02:00
Christian Heimes 8b24d60f1b
gh-95174: WASI: skip missing sockets functions (GH-95179) 2022-07-27 08:19:23 +02:00
Christian Heimes daa64d6a59
gh-95085: Promote Emscripten and WASI to PEP 11 tier 3 (GH-95086) 2022-07-27 08:18:34 +02:00
Christian Heimes 9af7f87d76
gh-94801: Fix regression in configure's CPPFLAGS handling (GH-95288) 2022-07-26 21:39:04 +02:00
Christian Heimes 0d35a59ce3
gh-95174: Handle missing dup() and constants in WASI (GH-95229)
- check for ``dup()`` libc function
- handle missing ``F_DUPFD`` in ``dup2()`` replacement function
- add workaround for WASI libc bug in MSG_TRUNC
- ESHUTDOWN is missing, use EPIPE instead
- POLLPRI is missing, define as 0 (no-op)
2022-07-26 11:16:51 +02:00
Christian Heimes 97b4121f93
gh-94847: Don't force inlining in debug builds of libmpdec (GH-94848) 2022-07-18 08:47:03 +02:00
Christian Heimes 944ff8c57c
gh-94801: Use custom flags to check for headers and libs (#94802) 2022-07-15 14:48:01 +02:00
Christian Heimes 81dca70d70
gh-93939: Build C extensions without setup.py (GH-94474)
Combines GH-93940, GH-94452, and GH-94433
2022-07-14 00:51:49 -07:00
Christian Heimes 18ef240a2c
gh-93649: Fix dependencies of _testcapi (#94695)
- header files are located in $(srcdir)
- dependencies must not list C files that are also in a makesetup Setup file
- generate SRCDIRS for OOT builds
2022-07-09 13:01:23 +02:00
Christian Heimes 29f86d6c28
gh-94642: Remove -D_XOPEN_SOURCE from more pkg-config CFLAGS (#94657)
Some pkg-config pc files define CFLAGS with -D_XOPEN_SOURCE=600. We always want _XOPEN_SOURCE=700.
2022-07-07 14:47:32 +02:00
Christian Heimes 9c60b25a0d
gh-94644: Define HAVE_NCURSESW in pkg-config branch (#94651) 2022-07-07 12:57:12 +02:00