2021-04-08 17:32:21 -03:00
***** ***** ***** *
Configure Python
***** ***** ***** *
2024-03-03 11:40:02 -04:00
.. highlight :: sh
2022-12-05 10:41:44 -04:00
Build Requirements
==================
2023-08-30 10:28:58 -03:00
Features and minimum versions required to build CPython:
2022-12-05 10:41:44 -04:00
* A `C11 <https://en.cppreference.com/w/c/11> `_ compiler. `Optional C11
features
<https://en.wikipedia.org/wiki/C11_(C_standard_revision)#Optional_features>`_
are not required.
2023-08-30 10:28:58 -03:00
* On Windows, Microsoft Visual Studio 2017 or later is required.
2024-07-19 05:06:02 -03:00
* Support for `IEEE 754 <https://en.wikipedia.org/wiki/IEEE_754> `_
floating-point numbers and `floating-point Not-a-Number (NaN)
2022-12-05 10:41:44 -04:00
<https://en.wikipedia.org/wiki/NaN#Floating_point>`_.
* Support for threads.
2023-08-30 10:28:58 -03:00
* OpenSSL 1.1.1 is the minimum version and OpenSSL 3.0.9 is the recommended
minimum version for the :mod: `ssl` and :mod: `hashlib` extension modules.
2022-12-05 10:41:44 -04:00
2023-08-30 10:28:58 -03:00
* SQLite 3.15.2 for the :mod: `sqlite3` extension module.
* Tcl/Tk 8.5.12 for the :mod: `tkinter` module.
* Autoconf 2.71 and aclocal 1.16.4 are required to regenerate the
:file: `configure` script.
2024-03-03 11:40:02 -04:00
.. versionchanged :: 3.1
Tcl/Tk version 8.3.1 is now required.
2022-12-05 10:41:44 -04:00
2024-03-03 11:40:02 -04:00
.. versionchanged :: 3.5
On Windows, Visual Studio 2015 or later is now required.
Tcl/Tk version 8.4 is now required.
2022-12-05 10:41:44 -04:00
2024-03-03 11:40:02 -04:00
.. versionchanged :: 3.6
Selected C99 features are now required, like `` <stdint.h> `` and `` static
inline`` functions.
2022-12-05 10:41:44 -04:00
.. versionchanged :: 3.7
Thread support and OpenSSL 1.0.2 are now required.
2024-03-03 11:40:02 -04:00
.. versionchanged :: 3.10
OpenSSL 1.1.1 is now required.
Require SQLite 3.7.15.
2022-12-05 10:41:44 -04:00
2024-03-03 11:40:02 -04:00
.. versionchanged :: 3.11
C11 compiler, IEEE 754 and NaN support are now required.
On Windows, Visual Studio 2017 or later is required.
Tcl/Tk version 8.5.12 is now required for the :mod: `tkinter` module.
2023-08-30 10:28:58 -03:00
2024-03-03 11:40:02 -04:00
.. versionchanged :: 3.13
Autoconf 2.71, aclocal 1.16.4 and SQLite 3.15.2 are now required.
2022-12-05 10:41:44 -04:00
See also :pep: `7` "Style Guide for C Code" and :pep: `11` "CPython platform
support".
2023-09-06 15:09:21 -03:00
Generated files
===============
To reduce build dependencies, Python source code contains multiple generated
files. Commands to regenerate all generated files::
make regen-all
make regen-stdlib-module-names
make regen-limited-abi
make regen-configure
The `` Makefile.pre.in `` file documents generated files, their inputs, and tools used
to regenerate them. Search for `` regen-* `` make targets.
2023-11-15 16:47:14 -04:00
configure script
----------------
The `` make regen-configure `` command regenerates the `` aclocal.m4 `` file and
the `` configure `` script using the `` Tools/build/regen-configure.sh `` shell
script which uses an Ubuntu container to get the same tools versions and have a
reproducible output.
The container is optional, the following command can be run locally::
2023-09-06 15:09:21 -03:00
autoreconf -ivf -Werror
2023-11-15 16:47:14 -04:00
The generated files can change depending on the exact `` autoconf-archive `` ,
`` aclocal `` and `` pkg-config `` versions.
2023-09-06 15:09:21 -03:00
2021-04-08 17:32:21 -03:00
.. _configure-options:
Configure Options
=================
2023-08-30 10:28:58 -03:00
List all :file: `configure` script options using::
2021-04-08 17:32:21 -03:00
./configure --help
See also the :file: `Misc/SpecialBuilds.txt` in the Python source distribution.
General Options
---------------
2023-10-03 14:38:12 -03:00
.. option :: --enable-loadable-sqlite-extensions
2021-04-08 17:32:21 -03:00
2023-09-26 21:01:48 -03:00
Support loadable extensions in the :mod: `!_sqlite` extension module (default
is no) of the :mod: `sqlite3` module.
2021-04-12 18:27:35 -03:00
See the :meth: `sqlite3.Connection.enable_load_extension` method of the
:mod: `sqlite3` module.
2021-04-08 17:32:21 -03:00
.. versionadded :: 3.6
2023-10-03 14:38:12 -03:00
.. option :: --disable-ipv6
2021-04-08 17:32:21 -03:00
Disable IPv6 support (enabled by default if supported), see the
:mod: `socket` module.
2023-10-03 14:38:12 -03:00
.. option :: --enable-big-digits=[15|30]
2021-04-08 17:32:21 -03:00
2021-04-08 20:58:20 -03:00
Define the size in bits of Python :class: `int` digits: 15 or 30 bits.
2022-01-14 14:54:56 -04:00
By default, the digit size is 30.
2021-04-08 20:58:20 -03:00
Define the `` PYLONG_BITS_IN_DIGIT `` to `` 15 `` or `` 30 `` .
2021-04-08 17:32:21 -03:00
See :data: `sys.int_info.bits_per_digit <sys.int_info>` .
2023-10-03 14:38:12 -03:00
.. option :: --with-suffix=SUFFIX
2021-04-08 17:32:21 -03:00
2021-04-08 20:58:20 -03:00
Set the Python executable suffix to *SUFFIX* .
The default suffix is `` .exe `` on Windows and macOS (`` python.exe ``
2022-04-04 14:31:31 -03:00
executable), `` .js `` on Emscripten node, `` .html `` on Emscripten browser,
`` .wasm `` on WASI, and an empty string on other platforms (`` python ``
executable).
2021-11-29 11:01:55 -04:00
.. versionchanged :: 3.11
2022-04-04 14:31:31 -03:00
The default suffix on WASM platform is one of `` .js `` , `` .html ``
or `` .wasm `` .
2021-04-08 17:32:21 -03:00
2023-10-03 14:38:12 -03:00
.. option :: --with-tzpath=<list of absolute paths separated by pathsep>
2021-04-08 17:32:21 -03:00
2023-07-21 06:40:37 -03:00
Select the default time zone search path for :const: `zoneinfo.TZPATH` .
2021-04-12 18:27:35 -03:00
See the :ref:`Compile-time configuration
<zoneinfo_data_compile_time_config>` of the :mod:` zoneinfo` module.
2021-04-08 17:32:21 -03:00
2021-04-08 20:58:20 -03:00
Default: `` /usr/share/zoneinfo:/usr/lib/zoneinfo:/usr/share/lib/zoneinfo:/etc/zoneinfo `` .
See :data: `os.pathsep` path separator.
2021-04-08 17:32:21 -03:00
.. versionadded :: 3.9
2023-10-03 14:38:12 -03:00
.. option :: --without-decimal-contextvar
2021-04-08 17:32:21 -03:00
Build the `` _decimal `` extension module using a thread-local context rather
than a coroutine-local context (default), see the :mod: `decimal` module.
2023-07-21 06:40:37 -03:00
See :const: `decimal.HAVE_CONTEXTVAR` and the :mod: `contextvars` module.
2021-04-08 17:32:21 -03:00
.. versionadded :: 3.9
2023-10-03 14:38:12 -03:00
.. option :: --with-dbmliborder=<list of backend names>
2021-04-08 17:32:21 -03:00
Override order to check db backends for the :mod: `dbm` module
2021-04-08 20:58:20 -03:00
A valid value is a colon (`` : `` ) separated string with the backend names:
2021-04-08 17:32:21 -03:00
* `` ndbm `` ;
* `` gdbm `` ;
* `` bdb `` .
2023-10-03 14:38:12 -03:00
.. option :: --without-c-locale-coercion
2021-04-08 20:58:20 -03:00
Disable C locale coercion to a UTF-8 based locale (enabled by default).
2021-04-08 17:32:21 -03:00
2021-04-08 20:58:20 -03:00
Don't define the `` PY_COERCE_C_LOCALE `` macro.
2021-04-08 17:32:21 -03:00
See :envvar: `PYTHONCOERCECLOCALE` and the :pep: `538` .
2023-10-03 14:38:12 -03:00
.. option :: --without-freelists
2022-12-05 10:41:44 -04:00
Disable all freelists except the empty tuple singleton.
.. versionadded :: 3.11
2023-10-03 14:38:12 -03:00
.. option :: --with-platlibdir=DIRNAME
2021-04-08 17:32:21 -03:00
Python library directory name (default is `` lib `` ).
Fedora and SuSE use `` lib64 `` on 64-bit platforms.
See :data: `sys.platlibdir` .
.. versionadded :: 3.9
2023-10-03 14:38:12 -03:00
.. option :: --with-wheel-pkg-dir=PATH
2021-04-08 17:32:21 -03:00
Directory of wheel packages used by the :mod: `ensurepip` module
(none by default).
Some Linux distribution packaging policies recommend against bundling
dependencies. For example, Fedora installs wheel packages in the
`` /usr/share/python-wheels/ `` directory and don't install the
2023-09-26 21:01:48 -03:00
:mod: `!ensurepip._bundled` package.
2021-04-08 17:32:21 -03:00
.. versionadded :: 3.10
2023-10-03 14:38:12 -03:00
.. option :: --with-pkg-config=[check|yes|no]
2021-11-10 18:02:19 -04:00
Whether configure should use :program: `pkg-config` to detect build
dependencies.
* `` check `` (default): :program: `pkg-config` is optional
* `` yes `` : :program: `pkg-config` is mandatory
* `` no `` : configure does not use :program: `pkg-config` even when present
.. versionadded :: 3.11
2023-10-03 14:38:12 -03:00
.. option :: --enable-pystats
2021-12-16 09:40:54 -04:00
2023-09-06 12:54:59 -03:00
Turn on internal Python performance statistics gathering.
By default, statistics gathering is off. Use `` python3 -X pystats `` command
or set `` PYTHONSTATS=1 `` environment variable to turn on statistics
gathering at Python startup.
At Python exit, dump statistics if statistics gathering was on and not
cleared.
Effects:
* Add :option: `-X pystats <-X>` command line option.
* Add :envvar: `!PYTHONSTATS` environment variable.
* Define the `` Py_STATS `` macro.
* Add functions to the :mod: `sys` module:
* :func: `!sys._stats_on` : Turns on statistics gathering.
* :func: `!sys._stats_off` : Turns off statistics gathering.
* :func: `!sys._stats_clear` : Clears the statistics.
* :func: `!sys._stats_dump` : Dump statistics to file, and clears the statistics.
2021-12-16 09:40:54 -04:00
The statistics will be dumped to a arbitrary (probably unique) file in
2023-09-06 12:54:59 -03:00
`` /tmp/py_stats/ `` (Unix) or `` C:\temp\py_stats\ `` (Windows). If that
directory does not exist, results will be printed on stderr.
2021-12-16 09:40:54 -04:00
2022-04-28 09:02:45 -03:00
Use `` Tools/scripts/summarize_stats.py `` to read the stats.
2021-12-16 09:40:54 -04:00
2023-09-06 12:54:59 -03:00
Statistics:
* Opcode:
* Specialization: success, failure, hit, deferred, miss, deopt, failures;
* Execution count;
* Pair count.
* Call:
* Inlined Python calls;
* PyEval calls;
* Frames pushed;
* Frame object created;
* Eval calls: vector, generator, legacy, function VECTORCALL, build class,
slot, function "ex", API, method.
* Object:
* incref and decref;
* interpreter incref and decref;
* allocations: all, 512 bytes, 4 kiB, big;
* free;
* to/from free lists;
* dictionary materialized/dematerialized;
* type cache;
2024-03-03 11:40:02 -04:00
* optimization attempts;
2023-09-06 12:54:59 -03:00
* optimization traces created/executed;
* uops executed.
* Garbage collector:
* Garbage collections;
* Objects visited;
* Objects collected.
2021-12-16 09:40:54 -04:00
.. versionadded :: 3.11
2023-12-01 10:50:16 -04:00
.. _free-threading-build:
2023-10-03 14:38:12 -03:00
.. option :: --disable-gil
2023-08-21 17:37:07 -03:00
Enables **experimental** support for running Python without the
2023-12-06 07:55:58 -04:00
:term:`global interpreter lock` (GIL): free threading build.
Defines the `` Py_GIL_DISABLED `` macro and adds `` "t" `` to
:data: `sys.abiflags` .
2023-08-21 17:37:07 -03:00
2024-08-13 11:30:59 -03:00
See :ref: `whatsnew313-free-threaded-cpython` for more detail.
2023-08-21 17:37:07 -03:00
.. versionadded :: 3.13
2023-10-03 14:38:12 -03:00
.. option :: PKG_CONFIG
2023-09-20 13:51:53 -03:00
Path to `` pkg-config `` utility.
2023-10-03 14:38:12 -03:00
.. option :: PKG_CONFIG_LIBDIR
.. option :: PKG_CONFIG_PATH
2023-09-20 13:51:53 -03:00
`` pkg-config `` options.
C compiler options
------------------
2023-10-03 14:38:12 -03:00
.. option :: CC
2023-09-20 13:51:53 -03:00
C compiler command.
2023-10-03 14:38:12 -03:00
.. option :: CFLAGS
2023-09-20 13:51:53 -03:00
C compiler flags.
2023-10-03 14:38:12 -03:00
.. option :: CPP
2023-09-20 13:51:53 -03:00
C preprocessor command.
2023-10-03 14:38:12 -03:00
.. option :: CPPFLAGS
2023-09-20 13:51:53 -03:00
C preprocessor flags, e.g. :samp: `-I{include_dir}` .
Linker options
--------------
2023-10-03 14:38:12 -03:00
.. option :: LDFLAGS
2023-09-20 13:51:53 -03:00
Linker flags, e.g. :samp: `-L{library_directory}` .
2023-10-03 14:38:12 -03:00
.. option :: LIBS
2023-09-20 13:51:53 -03:00
Libraries to pass to the linker, e.g. :samp: `-l{library}` .
2023-10-03 14:38:12 -03:00
.. option :: MACHDEP
2023-09-20 13:51:53 -03:00
Name for machine-dependent library files.
Options for third-party dependencies
------------------------------------
.. versionadded :: 3.11
2023-10-03 14:38:12 -03:00
.. option :: BZIP2_CFLAGS
.. option :: BZIP2_LIBS
2023-09-20 13:51:53 -03:00
C compiler and linker flags to link Python to `` libbz2 `` , used by :mod: `bz2`
module, overriding `` pkg-config `` .
2023-10-03 14:38:12 -03:00
.. option :: CURSES_CFLAGS
.. option :: CURSES_LIBS
2023-09-20 13:51:53 -03:00
C compiler and linker flags for `` libncurses `` or `` libncursesw `` , used by
:mod: `curses` module, overriding `` pkg-config `` .
2023-10-03 14:38:12 -03:00
.. option :: GDBM_CFLAGS
.. option :: GDBM_LIBS
2023-09-20 13:51:53 -03:00
C compiler and linker flags for `` gdbm `` .
2023-10-03 14:38:12 -03:00
.. option :: LIBB2_CFLAGS
.. option :: LIBB2_LIBS
2023-09-20 13:51:53 -03:00
C compiler and linker flags for `` libb2 `` (:ref: `BLAKE2 <hashlib-blake2>` ),
used by :mod: `hashlib` module, overriding `` pkg-config `` .
2023-10-03 14:38:12 -03:00
.. option :: LIBEDIT_CFLAGS
.. option :: LIBEDIT_LIBS
2023-09-20 13:51:53 -03:00
C compiler and linker flags for `` libedit `` , used by :mod: `readline` module,
overriding `` pkg-config `` .
2023-10-03 14:38:12 -03:00
.. option :: LIBFFI_CFLAGS
.. option :: LIBFFI_LIBS
2023-09-20 13:51:53 -03:00
C compiler and linker flags for `` libffi `` , used by :mod: `ctypes` module,
overriding `` pkg-config `` .
2024-04-29 03:58:57 -03:00
.. option :: LIBMPDEC_CFLAGS
.. option :: LIBMPDEC_LIBS
C compiler and linker flags for `` libmpdec `` , used by :mod: `decimal` module,
overriding `` pkg-config `` .
.. note ::
These environment variables have no effect unless
:option: `--with-system-libmpdec` is specified.
2023-10-03 14:38:12 -03:00
.. option :: LIBLZMA_CFLAGS
.. option :: LIBLZMA_LIBS
2023-09-20 13:51:53 -03:00
C compiler and linker flags for `` liblzma `` , used by :mod: `lzma` module,
overriding `` pkg-config `` .
2023-10-03 14:38:12 -03:00
.. option :: LIBREADLINE_CFLAGS
.. option :: LIBREADLINE_LIBS
2023-09-20 13:51:53 -03:00
C compiler and linker flags for `` libreadline `` , used by :mod: `readline`
module, overriding `` pkg-config `` .
2023-10-03 14:38:12 -03:00
.. option :: LIBSQLITE3_CFLAGS
.. option :: LIBSQLITE3_LIBS
2023-09-20 13:51:53 -03:00
C compiler and linker flags for `` libsqlite3 `` , used by :mod: `sqlite3`
module, overriding `` pkg-config `` .
2023-10-03 14:38:12 -03:00
.. option :: LIBUUID_CFLAGS
.. option :: LIBUUID_LIBS
2023-09-20 13:51:53 -03:00
C compiler and linker flags for `` libuuid `` , used by :mod: `uuid` module,
overriding `` pkg-config `` .
2023-10-03 14:38:12 -03:00
.. option :: PANEL_CFLAGS
.. option :: PANEL_LIBS
2023-09-20 13:51:53 -03:00
2024-07-09 09:24:37 -03:00
C compiler and linker flags for PANEL, overriding `` pkg-config `` .
2023-09-20 13:51:53 -03:00
C compiler and linker flags for `` libpanel `` or `` libpanelw `` , used by
:mod: `curses.panel` module, overriding `` pkg-config `` .
2023-10-03 14:38:12 -03:00
.. option :: TCLTK_CFLAGS
.. option :: TCLTK_LIBS
2023-09-20 13:51:53 -03:00
C compiler and linker flags for TCLTK, overriding `` pkg-config `` .
2023-10-03 14:38:12 -03:00
.. option :: ZLIB_CFLAGS
.. option :: ZLIB_LIBS
2023-09-20 13:51:53 -03:00
C compiler and linker flags for `` libzlib `` , used by :mod: `gzip` module,
overriding `` pkg-config `` .
2022-05-01 22:24:41 -03:00
WebAssembly Options
-------------------
2022-04-04 14:31:31 -03:00
2023-10-03 14:38:12 -03:00
.. option :: --with-emscripten-target=[browser|node]
2022-04-04 14:31:31 -03:00
Set build flavor for `` wasm32-emscripten `` .
* `` browser `` (default): preload minimal stdlib, default MEMFS.
* `` node `` : NODERAWFS and pthread support.
.. versionadded :: 3.11
2023-10-03 14:38:12 -03:00
.. option :: --enable-wasm-dynamic-linking
2022-04-04 14:31:31 -03:00
Turn on dynamic linking support for WASM.
Dynamic linking enables `` dlopen `` . File size of the executable
increases due to limited dead code elimination and additional features.
.. versionadded :: 3.11
2022-04-23 09:59:33 -03:00
2023-10-03 14:38:12 -03:00
.. option :: --enable-wasm-pthreads
2022-04-23 09:59:33 -03:00
Turn on pthreads support for WASM.
.. versionadded :: 3.11
2022-04-04 14:31:31 -03:00
2021-04-08 17:32:21 -03:00
Install Options
---------------
2023-10-03 14:38:12 -03:00
.. option :: --prefix=PREFIX
2023-03-15 12:10:03 -03:00
Install architecture-independent files in PREFIX. On Unix, it
defaults to :file: `/usr/local` .
2023-08-28 21:14:21 -03:00
This value can be retrieved at runtime using :data: `sys.prefix` .
2023-03-15 12:10:03 -03:00
As an example, one can use `` --prefix="$HOME/.local/" `` to install
a Python in its home directory.
2023-10-03 14:38:12 -03:00
.. option :: --exec-prefix=EPREFIX
2023-03-15 12:10:03 -03:00
Install architecture-dependent files in EPREFIX, defaults to :option: `--prefix` .
2023-08-28 21:14:21 -03:00
This value can be retrieved at runtime using :data: `sys.exec_prefix` .
2023-03-15 12:10:03 -03:00
2023-10-03 14:38:12 -03:00
.. option :: --disable-test-modules
2021-04-08 17:32:21 -03:00
Don't build nor install test modules, like the :mod: `test` package or the
2023-09-26 21:01:48 -03:00
:mod: `!_testcapi` extension module (built and installed by default).
2021-04-08 17:32:21 -03:00
.. versionadded :: 3.10
2023-10-03 14:38:12 -03:00
.. option :: --with-ensurepip=[upgrade|install|no]
2021-04-08 17:32:21 -03:00
2021-04-08 20:58:20 -03:00
Select the :mod: `ensurepip` command run on Python installation:
* `` upgrade `` (default): run `` python -m ensurepip --altinstall --upgrade ``
command.
* `` install `` : run `` python -m ensurepip --altinstall `` command;
* `` no `` : don't run ensurepip;
2021-04-08 17:32:21 -03:00
.. versionadded :: 3.6
Performance options
-------------------
Configuring Python using `` --enable-optimizations --with-lto `` (PGO + LTO) is
gh-90536: Add support for the BOLT post-link binary optimizer (gh-95908)
* Add support for the BOLT post-link binary optimizer
Using [bolt](https://github.com/llvm/llvm-project/tree/main/bolt)
provides a fairly large speedup without any code or functionality
changes. It provides roughly a 1% speedup on pyperformance, and a
4% improvement on the Pyston web macrobenchmarks.
It is gated behind an `--enable-bolt` configure arg because not all
toolchains and environments are supported. It has been tested on a
Linux x86_64 toolchain, using llvm-bolt built from the LLVM 14.0.6
sources (their binary distribution of this version did not include bolt).
Compared to [a previous attempt](https://github.com/faster-cpython/ideas/issues/224),
this commit uses bolt's preferred "instrumentation" approach, as well as adds some non-PIE
flags which enable much better optimizations from bolt.
The effects of this change are a bit more dependent on CPU microarchitecture
than other changes, since it optimizes i-cache behavior which seems
to be a bit more variable between architectures. The 1%/4% numbers
were collected on an Intel Skylake CPU, and on an AMD Zen 3 CPU I
got a slightly larger speedup (2%/4%), and on a c6i.xlarge EC2 instance
I got a slightly lower speedup (1%/3%).
The low speedup on pyperformance is not entirely unexpected, because
BOLT improves i-cache behavior, and the benchmarks in the pyperformance
suite are small and tend to fit in i-cache.
This change uses the existing pgo profiling task (`python -m test --pgo`),
though I was able to measure about a 1% macrobenchmark improvement by
using the macrobenchmarks as the training task. I personally think that
both the PGO and BOLT tasks should be updated to use macrobenchmarks,
but for the sake of splitting up the work this PR uses the existing pgo task.
* Simplify the build flags
* Add a NEWS entry
* Update Makefile.pre.in
Co-authored-by: Dong-hee Na <donghee.na92@gmail.com>
* Update configure.ac
Co-authored-by: Dong-hee Na <donghee.na92@gmail.com>
* Add myself to ACKS
* Add docs
* Other review comments
* fix tab/space issue
* Make it more clear that --enable-bolt is experimental
* Add link to bolt's github page
Co-authored-by: Dong-hee Na <donghee.na92@gmail.com>
2022-08-18 18:33:54 -03:00
recommended for best performance. The experimental `` --enable-bolt `` flag can
also be used to improve performance.
2021-04-08 17:32:21 -03:00
2023-10-03 14:38:12 -03:00
.. option :: --enable-optimizations
2021-04-08 17:32:21 -03:00
Enable Profile Guided Optimization (PGO) using :envvar: `PROFILE_TASK`
(disabled by default).
2021-04-09 09:35:10 -03:00
The C compiler Clang requires `` llvm-profdata `` program for PGO. On
macOS, GCC also requires it: GCC is just an alias to Clang on macOS.
2021-04-08 17:32:21 -03:00
Disable also semantic interposition in libpython if `` --enable-shared `` and
GCC is used: add `` -fno-semantic-interposition `` to the compiler and linker
flags.
2024-04-15 18:17:28 -03:00
.. note ::
During the build, you may encounter compiler warnings about
profile data not being available for some source files.
These warnings are harmless, as only a subset of the code is exercised
during profile data acquisition.
To disable these warnings on Clang, manually suppress them by adding
`` -Wno-profile-instr-unprofiled `` to :envvar: `CFLAGS` .
2021-04-08 17:32:21 -03:00
.. versionadded :: 3.6
.. versionchanged :: 3.10
Use `` -fno-semantic-interposition `` on GCC.
.. envvar :: PROFILE_TASK
Environment variable used in the Makefile: Python command line arguments for
the PGO generation task.
Default: `` -m test --pgo --timeout=$(TESTTIMEOUT) `` .
.. versionadded :: 3.8
2023-10-03 13:43:23 -03:00
.. versionchanged :: 3.13
Task failure is no longer ignored silently.
2023-10-03 14:38:12 -03:00
.. option :: --with-lto=[full|thin|no|yes]
2021-04-08 17:32:21 -03:00
Enable Link Time Optimization (LTO) in any build (disabled by default).
2021-05-25 07:25:22 -03:00
The C compiler Clang requires `` llvm-ar `` for LTO (`` ar `` on macOS), as well
as an LTO-aware linker (`` ld.gold `` or `` lld `` ).
2021-04-09 09:35:10 -03:00
2021-04-08 17:32:21 -03:00
.. versionadded :: 3.6
2021-07-19 07:52:56 -03:00
.. versionadded :: 3.11
To use ThinLTO feature, use `` --with-lto=thin `` on Clang.
2022-09-16 07:40:05 -03:00
.. versionchanged :: 3.12
Use ThinLTO as the default optimization policy on Clang if the compiler accepts the flag.
2023-10-03 14:38:12 -03:00
.. option :: --enable-bolt
gh-90536: Add support for the BOLT post-link binary optimizer (gh-95908)
* Add support for the BOLT post-link binary optimizer
Using [bolt](https://github.com/llvm/llvm-project/tree/main/bolt)
provides a fairly large speedup without any code or functionality
changes. It provides roughly a 1% speedup on pyperformance, and a
4% improvement on the Pyston web macrobenchmarks.
It is gated behind an `--enable-bolt` configure arg because not all
toolchains and environments are supported. It has been tested on a
Linux x86_64 toolchain, using llvm-bolt built from the LLVM 14.0.6
sources (their binary distribution of this version did not include bolt).
Compared to [a previous attempt](https://github.com/faster-cpython/ideas/issues/224),
this commit uses bolt's preferred "instrumentation" approach, as well as adds some non-PIE
flags which enable much better optimizations from bolt.
The effects of this change are a bit more dependent on CPU microarchitecture
than other changes, since it optimizes i-cache behavior which seems
to be a bit more variable between architectures. The 1%/4% numbers
were collected on an Intel Skylake CPU, and on an AMD Zen 3 CPU I
got a slightly larger speedup (2%/4%), and on a c6i.xlarge EC2 instance
I got a slightly lower speedup (1%/3%).
The low speedup on pyperformance is not entirely unexpected, because
BOLT improves i-cache behavior, and the benchmarks in the pyperformance
suite are small and tend to fit in i-cache.
This change uses the existing pgo profiling task (`python -m test --pgo`),
though I was able to measure about a 1% macrobenchmark improvement by
using the macrobenchmarks as the training task. I personally think that
both the PGO and BOLT tasks should be updated to use macrobenchmarks,
but for the sake of splitting up the work this PR uses the existing pgo task.
* Simplify the build flags
* Add a NEWS entry
* Update Makefile.pre.in
Co-authored-by: Dong-hee Na <donghee.na92@gmail.com>
* Update configure.ac
Co-authored-by: Dong-hee Na <donghee.na92@gmail.com>
* Add myself to ACKS
* Add docs
* Other review comments
* fix tab/space issue
* Make it more clear that --enable-bolt is experimental
* Add link to bolt's github page
Co-authored-by: Dong-hee Na <donghee.na92@gmail.com>
2022-08-18 18:33:54 -03:00
Enable usage of the `BOLT post-link binary optimizer
2022-08-20 12:11:35 -03:00
<https://github.com/llvm/llvm-project/tree/main/bolt>`_ (disabled by
gh-90536: Add support for the BOLT post-link binary optimizer (gh-95908)
* Add support for the BOLT post-link binary optimizer
Using [bolt](https://github.com/llvm/llvm-project/tree/main/bolt)
provides a fairly large speedup without any code or functionality
changes. It provides roughly a 1% speedup on pyperformance, and a
4% improvement on the Pyston web macrobenchmarks.
It is gated behind an `--enable-bolt` configure arg because not all
toolchains and environments are supported. It has been tested on a
Linux x86_64 toolchain, using llvm-bolt built from the LLVM 14.0.6
sources (their binary distribution of this version did not include bolt).
Compared to [a previous attempt](https://github.com/faster-cpython/ideas/issues/224),
this commit uses bolt's preferred "instrumentation" approach, as well as adds some non-PIE
flags which enable much better optimizations from bolt.
The effects of this change are a bit more dependent on CPU microarchitecture
than other changes, since it optimizes i-cache behavior which seems
to be a bit more variable between architectures. The 1%/4% numbers
were collected on an Intel Skylake CPU, and on an AMD Zen 3 CPU I
got a slightly larger speedup (2%/4%), and on a c6i.xlarge EC2 instance
I got a slightly lower speedup (1%/3%).
The low speedup on pyperformance is not entirely unexpected, because
BOLT improves i-cache behavior, and the benchmarks in the pyperformance
suite are small and tend to fit in i-cache.
This change uses the existing pgo profiling task (`python -m test --pgo`),
though I was able to measure about a 1% macrobenchmark improvement by
using the macrobenchmarks as the training task. I personally think that
both the PGO and BOLT tasks should be updated to use macrobenchmarks,
but for the sake of splitting up the work this PR uses the existing pgo task.
* Simplify the build flags
* Add a NEWS entry
* Update Makefile.pre.in
Co-authored-by: Dong-hee Na <donghee.na92@gmail.com>
* Update configure.ac
Co-authored-by: Dong-hee Na <donghee.na92@gmail.com>
* Add myself to ACKS
* Add docs
* Other review comments
* fix tab/space issue
* Make it more clear that --enable-bolt is experimental
* Add link to bolt's github page
Co-authored-by: Dong-hee Na <donghee.na92@gmail.com>
2022-08-18 18:33:54 -03:00
default).
BOLT is part of the LLVM project but is not always included in their binary
distributions. This flag requires that `` llvm-bolt `` and `` merge-fdata ``
are available.
BOLT is still a fairly new project so this flag should be considered
experimental for now. Because this tool operates on machine code its success
is dependent on a combination of the build environment + the other
optimization configure args + the CPU architecture, and not all combinations
are supported.
2023-05-22 08:45:20 -03:00
BOLT versions before LLVM 16 are known to crash BOLT under some scenarios.
Use of LLVM 16 or newer for BOLT optimization is strongly encouraged.
The :envvar: `!BOLT_INSTRUMENT_FLAGS` and :envvar: `!BOLT_APPLY_FLAGS`
:program: `configure` variables can be defined to override the default set of
arguments for :program: `llvm-bolt` to instrument and apply BOLT data to
binaries, respectively.
gh-90536: Add support for the BOLT post-link binary optimizer (gh-95908)
* Add support for the BOLT post-link binary optimizer
Using [bolt](https://github.com/llvm/llvm-project/tree/main/bolt)
provides a fairly large speedup without any code or functionality
changes. It provides roughly a 1% speedup on pyperformance, and a
4% improvement on the Pyston web macrobenchmarks.
It is gated behind an `--enable-bolt` configure arg because not all
toolchains and environments are supported. It has been tested on a
Linux x86_64 toolchain, using llvm-bolt built from the LLVM 14.0.6
sources (their binary distribution of this version did not include bolt).
Compared to [a previous attempt](https://github.com/faster-cpython/ideas/issues/224),
this commit uses bolt's preferred "instrumentation" approach, as well as adds some non-PIE
flags which enable much better optimizations from bolt.
The effects of this change are a bit more dependent on CPU microarchitecture
than other changes, since it optimizes i-cache behavior which seems
to be a bit more variable between architectures. The 1%/4% numbers
were collected on an Intel Skylake CPU, and on an AMD Zen 3 CPU I
got a slightly larger speedup (2%/4%), and on a c6i.xlarge EC2 instance
I got a slightly lower speedup (1%/3%).
The low speedup on pyperformance is not entirely unexpected, because
BOLT improves i-cache behavior, and the benchmarks in the pyperformance
suite are small and tend to fit in i-cache.
This change uses the existing pgo profiling task (`python -m test --pgo`),
though I was able to measure about a 1% macrobenchmark improvement by
using the macrobenchmarks as the training task. I personally think that
both the PGO and BOLT tasks should be updated to use macrobenchmarks,
but for the sake of splitting up the work this PR uses the existing pgo task.
* Simplify the build flags
* Add a NEWS entry
* Update Makefile.pre.in
Co-authored-by: Dong-hee Na <donghee.na92@gmail.com>
* Update configure.ac
Co-authored-by: Dong-hee Na <donghee.na92@gmail.com>
* Add myself to ACKS
* Add docs
* Other review comments
* fix tab/space issue
* Make it more clear that --enable-bolt is experimental
* Add link to bolt's github page
Co-authored-by: Dong-hee Na <donghee.na92@gmail.com>
2022-08-18 18:33:54 -03:00
.. versionadded :: 3.12
2023-10-03 14:38:12 -03:00
.. option :: BOLT_APPLY_FLAGS
2023-09-20 13:51:53 -03:00
Arguments to `` llvm-bolt `` when creating a `BOLT optimized binary
<https://github.com/facebookarchive/BOLT>`_.
.. versionadded :: 3.12
2023-10-03 14:38:12 -03:00
.. option :: BOLT_INSTRUMENT_FLAGS
2023-09-20 13:51:53 -03:00
Arguments to `` llvm-bolt `` when instrumenting binaries.
.. versionadded :: 3.12
2023-10-03 14:38:12 -03:00
.. option :: --with-computed-gotos
2021-04-08 17:32:21 -03:00
Enable computed gotos in evaluation loop (enabled by default on supported
compilers).
2023-10-30 12:43:11 -03:00
.. option :: --without-mimalloc
2024-07-09 09:24:37 -03:00
Disable the fast :ref: `mimalloc <mimalloc>` allocator
2023-10-30 12:43:11 -03:00
(enabled by default).
See also :envvar: `PYTHONMALLOC` environment variable.
2023-10-03 14:38:12 -03:00
.. option :: --without-pymalloc
2021-04-08 17:32:21 -03:00
Disable the specialized Python memory allocator :ref: `pymalloc <pymalloc>`
(enabled by default).
See also :envvar: `PYTHONMALLOC` environment variable.
2023-10-03 14:38:12 -03:00
.. option :: --without-doc-strings
2021-04-08 17:32:21 -03:00
Disable static documentation strings to reduce the memory footprint (enabled
by default). Documentation strings defined in Python are not affected.
2021-04-08 20:58:20 -03:00
Don't define the `` WITH_DOC_STRINGS `` macro.
See the `` PyDoc_STRVAR() `` macro.
2021-04-08 17:32:21 -03:00
2023-10-03 14:38:12 -03:00
.. option :: --enable-profiling
2021-04-08 17:32:21 -03:00
Enable C-level code profiling with `` gprof `` (disabled by default).
2023-10-03 14:38:12 -03:00
.. option :: --with-strict-overflow
2023-03-04 17:39:52 -04:00
Add `` -fstrict-overflow `` to the C compiler flags (by default we add
`` -fno-strict-overflow `` instead).
2021-04-08 17:32:21 -03:00
.. _debug-build:
2021-04-29 08:06:59 -03:00
Python Debug Build
------------------
2021-04-08 17:32:21 -03:00
A debug build is Python built with the :option: `--with-pydebug` configure
option.
Effects of a debug build:
2021-04-29 08:06:59 -03:00
* Display all warnings by default: the list of default warning filters is empty
in the :mod: `warnings` module.
2021-04-08 17:32:21 -03:00
* Add `` d `` to :data: `sys.abiflags` .
2023-09-26 21:01:48 -03:00
* Add :func: `!sys.gettotalrefcount` function.
2021-04-08 17:32:21 -03:00
* Add :option: `-X showrefcount <-X>` command line option.
2022-05-24 18:23:01 -03:00
* Add :option: `-d` command line option and :envvar: `PYTHONDEBUG` environment
variable to debug the parser.
2022-04-16 19:57:00 -03:00
* Add support for the `` __lltrace__ `` variable: enable low-level tracing in the
2021-04-08 20:58:20 -03:00
bytecode evaluation loop if the variable is defined.
2021-04-29 08:06:59 -03:00
* Install :ref: `debug hooks on memory allocators <default-memory-allocators>`
to detect buffer overflow and other memory errors.
* Define `` Py_DEBUG `` and `` Py_REF_DEBUG `` macros.
2021-05-25 18:01:30 -03:00
* Add runtime checks: code surrounded by `` #ifdef Py_DEBUG `` and `` #endif `` .
2021-04-29 08:06:59 -03:00
Enable `` assert(...) `` and `` _PyObject_ASSERT(...) `` assertions: don't set
the `` NDEBUG `` macro (see also the :option: `--with-assertions` configure
option). Main runtime checks:
* Add sanity checks on the function arguments.
* Unicode and int objects are created with their memory filled with a pattern
to detect usage of uninitialized objects.
* Ensure that functions which can clear or replace the current exception are
not called with an exception raised.
2022-04-21 18:04:01 -03:00
* Check that deallocator functions don't change the current exception.
2021-04-29 08:06:59 -03:00
* The garbage collector (:func: `gc.collect` function) runs some basic checks
on objects consistency.
2023-09-26 21:01:48 -03:00
* The :c:macro: `!Py_SAFE_DOWNCAST()` macro checks for integer underflow and
2021-04-29 08:06:59 -03:00
overflow when downcasting from wide types to narrow types.
2021-04-08 17:32:21 -03:00
See also the :ref: `Python Development Mode <devmode>` and the
:option: `--with-trace-refs` configure option.
.. versionchanged :: 3.8
Release builds and debug builds are now ABI compatible: defining the
2021-04-29 08:06:59 -03:00
`` Py_DEBUG `` macro no longer implies the `` Py_TRACE_REFS `` macro (see the
2023-08-31 13:33:34 -03:00
:option: `--with-trace-refs` option).
2021-04-08 17:32:21 -03:00
Debug options
-------------
2023-10-03 14:38:12 -03:00
.. option :: --with-pydebug
2021-04-08 17:32:21 -03:00
2021-04-08 20:58:20 -03:00
:ref:`Build Python in debug mode <debug-build>`: define the ``Py_DEBUG``
macro (disabled by default).
2021-04-08 17:32:21 -03:00
2023-10-03 14:38:12 -03:00
.. option :: --with-trace-refs
2021-04-08 17:32:21 -03:00
Enable tracing references for debugging purpose (disabled by default).
Effects:
* Define the `` Py_TRACE_REFS `` macro.
2023-09-26 21:01:48 -03:00
* Add :func: `!sys.getobjects` function.
2021-04-08 17:32:21 -03:00
* Add :envvar: `PYTHONDUMPREFS` environment variable.
2023-08-31 13:33:34 -03:00
The :envvar: `PYTHONDUMPREFS` environment variable can be used to dump
objects and reference counts still alive at Python exit.
:ref: `Statically allocated objects <static-types>` are not traced.
2024-03-03 11:40:02 -04:00
.. versionadded :: 3.8
2023-08-31 13:33:34 -03:00
.. versionchanged :: 3.13
This build is now ABI compatible with release build and :ref:`debug build
<debug-build>`.
2021-04-08 17:32:21 -03:00
2023-10-03 14:38:12 -03:00
.. option :: --with-assertions
2021-04-08 17:32:21 -03:00
2021-04-08 20:58:20 -03:00
Build with C assertions enabled (default is no): `` assert(...); `` and
`` _PyObject_ASSERT(...); `` .
2021-04-08 17:32:21 -03:00
If set, the `` NDEBUG `` macro is not defined in the :envvar: `OPT` compiler
variable.
See also the :option: `--with-pydebug` option (:ref:`debug build
<debug-build>`) which also enables assertions.
.. versionadded :: 3.6
2023-10-03 14:38:12 -03:00
.. option :: --with-valgrind
2021-04-08 17:32:21 -03:00
Enable Valgrind support (default is no).
2023-10-03 14:38:12 -03:00
.. option :: --with-dtrace
2021-04-08 17:32:21 -03:00
Enable DTrace support (default is no).
2021-04-12 18:27:35 -03:00
See :ref:`Instrumenting CPython with DTrace and SystemTap
<instrumentation>`.
2021-04-08 17:32:21 -03:00
.. versionadded :: 3.6
2023-10-03 14:38:12 -03:00
.. option :: --with-address-sanitizer
2021-04-08 17:32:21 -03:00
2021-04-08 20:58:20 -03:00
Enable AddressSanitizer memory error detector, `` asan `` (default is no).
2021-04-08 17:32:21 -03:00
.. versionadded :: 3.6
2023-10-03 14:38:12 -03:00
.. option :: --with-memory-sanitizer
2021-04-08 17:32:21 -03:00
2021-04-08 20:58:20 -03:00
Enable MemorySanitizer allocation error detector, `` msan `` (default is no).
2021-04-08 17:32:21 -03:00
.. versionadded :: 3.6
2023-10-03 14:38:12 -03:00
.. option :: --with-undefined-behavior-sanitizer
2021-04-08 17:32:21 -03:00
2021-04-08 20:58:20 -03:00
Enable UndefinedBehaviorSanitizer undefined behaviour detector, `` ubsan ``
2021-04-08 17:32:21 -03:00
(default is no).
.. versionadded :: 3.6
2023-12-30 04:17:02 -04:00
.. option :: --with-thread-sanitizer
Enable ThreadSanitizer data race detector, `` tsan ``
(default is no).
.. versionadded :: 3.13
2021-04-08 17:32:21 -03:00
Linker options
--------------
2023-10-03 14:38:12 -03:00
.. option :: --enable-shared
2021-04-08 17:32:21 -03:00
2021-04-08 20:58:20 -03:00
Enable building a shared Python library: `` libpython `` (default is no).
2021-04-08 17:32:21 -03:00
2023-10-03 14:38:12 -03:00
.. option :: --without-static-libpython
2021-04-08 17:32:21 -03:00
Do not build `` libpythonMAJOR.MINOR.a `` and do not install `` python.o ``
(built and enabled by default).
.. versionadded :: 3.10
Libraries options
-----------------
2023-10-03 14:38:12 -03:00
.. option :: --with-libs='lib1 ...'
2021-04-08 17:32:21 -03:00
Link against additional libraries (default is no).
2023-10-03 14:38:12 -03:00
.. option :: --with-system-expat
2021-04-08 17:32:21 -03:00
2023-09-26 21:01:48 -03:00
Build the :mod: `!pyexpat` module using an installed `` expat `` library
2021-04-08 17:32:21 -03:00
(default is no).
2023-10-03 14:38:12 -03:00
.. option :: --with-system-libmpdec
2021-04-08 17:32:21 -03:00
2024-05-06 16:16:08 -03:00
Build the `` _decimal `` extension module using an installed `` mpdecimal ``
library, see the :mod: `decimal` module (default is yes).
2021-04-08 17:32:21 -03:00
.. versionadded :: 3.3
2024-05-06 16:16:08 -03:00
.. versionchanged :: 3.13
Default to using the installed `` mpdecimal `` library.
.. deprecated-removed :: 3.13 3.15
A copy of the `` mpdecimal `` library sources will no longer be distributed
with Python 3.15.
2024-04-29 03:58:57 -03:00
.. seealso :: :option: `LIBMPDEC_CFLAGS` and :option: `LIBMPDEC_LIBS` .
2023-10-03 14:38:12 -03:00
.. option :: --with-readline=readline|editline
2021-04-08 17:32:21 -03:00
2023-09-29 09:18:18 -03:00
Designate a backend library for the :mod: `readline` module.
2021-04-08 20:58:20 -03:00
2023-09-29 09:18:18 -03:00
* readline: Use readline as the backend.
* editline: Use editline as the backend.
2021-04-08 20:58:20 -03:00
.. versionadded :: 3.10
2023-10-03 14:38:12 -03:00
.. option :: --without-readline
2021-04-08 20:58:20 -03:00
Don't build the :mod: `readline` module (built by default).
Don't define the `` HAVE_LIBREADLINE `` macro.
2021-04-08 17:32:21 -03:00
.. versionadded :: 3.10
2023-10-03 14:38:12 -03:00
.. option :: --with-libm=STRING
2021-04-08 17:32:21 -03:00
Override `` libm `` math library to *STRING* (default is system-dependent).
2023-10-03 14:38:12 -03:00
.. option :: --with-libc=STRING
2021-04-08 17:32:21 -03:00
Override `` libc `` C library to *STRING* (default is system-dependent).
2023-10-03 14:38:12 -03:00
.. option :: --with-openssl=DIR
2021-04-08 17:32:21 -03:00
Root of the OpenSSL directory.
2021-04-08 20:58:20 -03:00
.. versionadded :: 3.7
2023-10-03 14:38:12 -03:00
.. option :: --with-openssl-rpath=[no|auto|DIR]
2021-04-08 17:32:21 -03:00
Set runtime library directory (rpath) for OpenSSL libraries:
* `` no `` (default): don't set rpath;
* `` auto `` : auto-detect rpath from :option: `--with-openssl` and
`` pkg-config `` ;
* *DIR* : set an explicit rpath.
.. versionadded :: 3.10
Security Options
----------------
2023-10-03 14:38:12 -03:00
.. option :: --with-hash-algorithm=[fnv|siphash13|siphash24]
2021-04-08 17:32:21 -03:00
Select hash algorithm for use in `` Python/pyhash.c `` :
2021-10-10 05:29:46 -03:00
* `` siphash13 `` (default);
* `` siphash24 `` ;
* `` fnv `` .
2021-04-08 17:32:21 -03:00
.. versionadded :: 3.4
2021-10-10 05:29:46 -03:00
.. versionadded :: 3.11
`` siphash13 `` is added and it is the new default.
2023-10-03 14:38:12 -03:00
.. option :: --with-builtin-hashlib-hashes=md5,sha1,sha256,sha512,sha3,blake2
2021-04-08 17:32:21 -03:00
Built-in hash modules:
* `` md5 `` ;
* `` sha1 `` ;
* `` sha256 `` ;
* `` sha512 `` ;
* `` sha3 `` (with shake);
* `` blake2 `` .
.. versionadded :: 3.9
2023-10-03 14:38:12 -03:00
.. option :: --with-ssl-default-suites=[python|openssl|STRING]
2021-04-08 17:32:21 -03:00
Override the OpenSSL default cipher suites string:
2021-04-08 20:58:20 -03:00
* `` python `` (default): use Python's preferred selection;
2021-04-08 17:32:21 -03:00
* `` openssl `` : leave OpenSSL's defaults untouched;
2021-05-01 15:53:10 -03:00
* *STRING* : use a custom string
2021-04-08 17:32:21 -03:00
See the :mod: `ssl` module.
.. versionadded :: 3.7
2021-05-01 15:53:10 -03:00
.. versionchanged :: 3.10
The settings `` python `` and *STRING* also set TLS 1.2 as minimum
protocol version.
2021-04-08 17:32:21 -03:00
2024-07-22 21:22:04 -03:00
.. option :: --disable-safety
2024-08-08 16:35:00 -03:00
Disable compiler options that are `recommended by OpenSSF`_ for security reasons with no performance overhead.
2024-07-22 21:22:04 -03:00
If this option is not enabled, CPython will be built based on safety compiler options with no slow down.
2024-08-08 16:35:00 -03:00
When this option is enabled, CPython will not be built with the compiler options listed below.
2024-07-22 21:22:04 -03:00
2024-08-08 16:35:00 -03:00
The following compiler options are disabled with :option: `!--disable-safety` :
* `-fstack-protector-strong`_ : Enable run-time checks for stack-based buffer overflows.
* `-Wtrampolines`_ : Enable warnings about trampolines that require executable stacks.
.. _recommended by OpenSSF: https://github.com/ossf/wg-best-practices-os-developers/blob/main/docs/Compiler-Hardening-Guides/Compiler-Options-Hardening-Guide-for-C-and-C++.md
.. _-fstack-protector-strong: https://github.com/ossf/wg-best-practices-os-developers/blob/main/docs/Compiler-Hardening-Guides/Compiler-Options-Hardening-Guide-for-C-and-C++.md#enable-run-time-checks-for-stack-based-buffer-overflows
.. _-Wtrampolines: https://github.com/ossf/wg-best-practices-os-developers/blob/main/docs/Compiler-Hardening-Guides/Compiler-Options-Hardening-Guide-for-C-and-C++.md#enable-warning-about-trampolines-that-require-executable-stacks
2024-07-22 21:22:04 -03:00
.. versionadded :: 3.14
.. option :: --enable-slower-safety
2024-08-08 16:35:00 -03:00
Enable compiler options that are `recommended by OpenSSF`_ for security reasons which require overhead.
2024-07-22 21:22:04 -03:00
If this option is not enabled, CPython will not be built based on safety compiler options which performance impact.
2024-08-08 16:35:00 -03:00
When this option is enabled, CPython will be built with the compiler options listed below.
The following compiler options are enabled with :option: `!--enable-slower-safety` :
* `-D_FORTIFY_SOURCE=3`_ : Fortify sources with compile- and run-time checks for unsafe libc usage and buffer overflows.
2024-07-22 21:22:04 -03:00
2024-08-08 16:35:00 -03:00
.. _-D_FORTIFY_SOURCE=3: https://github.com/ossf/wg-best-practices-os-developers/blob/main/docs/Compiler-Hardening-Guides/Compiler-Options-Hardening-Guide-for-C-and-C++.md#fortify-sources-for-unsafe-libc-usage-and-buffer-overflows
2024-07-22 21:22:04 -03:00
.. versionadded :: 3.14
2021-04-08 17:32:21 -03:00
macOS Options
-------------
2024-03-28 05:13:13 -03:00
See :source: `Mac/README.rst` .
2021-04-08 17:32:21 -03:00
2023-10-03 14:38:12 -03:00
.. option :: --enable-universalsdk
.. option :: --enable-universalsdk=SDKDIR
2021-04-08 17:32:21 -03:00
Create a universal binary build. *SDKDIR* specifies which macOS SDK should
be used to perform the build (default is no).
2023-10-03 14:38:12 -03:00
.. option :: --enable-framework
.. option :: --enable-framework=INSTALLDIR
2021-04-08 17:32:21 -03:00
Create a Python.framework rather than a traditional Unix install. Optional
*INSTALLDIR* specifies the installation path (default is no).
2023-10-03 14:38:12 -03:00
.. option :: --with-universal-archs=ARCH
2021-04-08 17:32:21 -03:00
2021-04-08 20:58:20 -03:00
Specify the kind of universal binary that should be created. This option is
2021-04-08 17:32:21 -03:00
only valid when :option: `--enable-universalsdk` is set.
2021-04-08 20:58:20 -03:00
Options:
2021-04-08 17:32:21 -03:00
* `` universal2 `` ;
* `` 32-bit `` ;
* `` 64-bit `` ;
* `` 3-way `` ;
* `` intel `` ;
* `` intel-32 `` ;
* `` intel-64 `` ;
* `` all `` .
2023-10-03 14:38:12 -03:00
.. option :: --with-framework-name=FRAMEWORK
2021-04-08 17:32:21 -03:00
Specify the name for the python framework on macOS only valid when
:option:`--enable-framework` is set (default: ``Python``).
2024-07-21 20:36:08 -03:00
.. option :: --with-app-store-compliance
.. option :: --with-app-store-compliance=PATCH-FILE
The Python standard library contains strings that are known to trigger
automated inspection tool errors when submitted for distribution by
the macOS and iOS App Stores. If enabled, this option will apply the list of
patches that are known to correct app store compliance. A custom patch
file can also be specified. This option is disabled by default.
.. versionadded :: 3.13
2024-03-28 05:13:13 -03:00
iOS Options
-----------
See :source: `iOS/README.rst` .
.. option :: --enable-framework=INSTALLDIR
Create a Python.framework. Unlike macOS, the *INSTALLDIR* argument
specifying the installation path is mandatory.
.. option :: --with-framework-name=FRAMEWORK
Specify the name for the framework (default: `` Python `` ).
2021-04-08 17:32:21 -03:00
2021-11-29 12:23:29 -04:00
Cross Compiling Options
-----------------------
Cross compiling, also known as cross building, can be used to build Python
for another CPU architecture or platform. Cross compiling requires a Python
2021-12-13 15:48:46 -04:00
interpreter for the build platform. The version of the build Python must match
the version of the cross compiled host Python.
2021-11-29 12:23:29 -04:00
2023-10-03 14:38:12 -03:00
.. option :: --build=BUILD
2021-11-29 12:23:29 -04:00
configure for building on BUILD, usually guessed by :program: `config.guess` .
2023-10-03 14:38:12 -03:00
.. option :: --host=HOST
2021-11-29 12:23:29 -04:00
cross-compile to build programs to run on HOST (target platform)
2023-10-03 14:38:12 -03:00
.. option :: --with-build-python=path/to/python
2021-11-29 12:23:29 -04:00
path to build `` python `` binary for cross compiling
.. versionadded :: 3.11
2023-10-03 14:38:12 -03:00
.. option :: CONFIG_SITE=file
2021-11-29 12:23:29 -04:00
An environment variable that points to a file with configure overrides.
2024-03-03 11:40:02 -04:00
Example *config.site* file:
.. code-block :: ini
2021-11-29 12:23:29 -04:00
# config.site-aarch64
ac_cv_buggy_getaddrinfo=no
ac_cv_file__dev_ptmx=yes
ac_cv_file__dev_ptc=no
2023-10-03 14:38:12 -03:00
.. option :: HOSTRUNNER
2023-09-20 13:51:53 -03:00
Program to run CPython for the host platform for cross-compilation.
.. versionadded :: 3.11
2021-11-29 12:23:29 -04:00
Cross compiling example::
CONFIG_SITE=config.site-aarch64 ../configure \
--build=x86_64-pc-linux-gnu \
--host=aarch64-unknown-linux-gnu \
--with-build-python=../x86_64/python
2021-04-09 09:35:10 -03:00
Python Build System
===================
Main files of the build system
------------------------------
* :file: `configure.ac` => :file: `configure` ;
* :file: `Makefile.pre.in` => :file: `Makefile` (created by :file: `configure` );
* :file: `pyconfig.h` (created by :file: `configure` );
* :file: `Modules/Setup` : C extensions built by the Makefile using
:file: `Module/makesetup` shell script;
Main build steps
----------------
* C files (`` .c `` ) are built as object files (`` .o `` ).
* A static `` libpython `` library (`` .a `` ) is created from objects files.
* `` python.o `` and the static `` libpython `` library are linked into the
final `` python `` program.
2023-08-16 18:13:05 -03:00
* C extensions are built by the Makefile (see :file: `Modules/Setup` ).
2021-04-09 09:35:10 -03:00
Main Makefile targets
---------------------
2024-03-07 13:21:28 -04:00
make
^^^^
For the most part, when rebuilding after editing some code or
refreshing your checkout from upstream, all you need to do is execute
`` make `` , which (per Make's semantics) builds the default target, the
first one defined in the Makefile. By tradition (including in the
CPython project) this is usually the `` all `` target. The
`` configure `` script expands an `` autoconf `` variable,
`` @DEF_MAKE_ALL_RULE@ `` to describe precisely which targets `` make
all`` will build. The three choices are:
* `` profile-opt `` (configured with `` --enable-optimizations `` )
* `` build_wasm `` (configured with `` --with-emscripten-target `` )
* `` build_all `` (configured without explicitly using either of the others)
Depending on the most recent source file changes, Make will rebuild
any targets (object files and executables) deemed out-of-date,
including running `` configure `` again if necessary. Source/target
dependencies are many and maintained manually however, so Make
sometimes doesn't have all the information necessary to correctly
detect all targets which need to be rebuilt. Depending on which
targets aren't rebuilt, you might experience a number of problems. If
you have build or test problems which you can't otherwise explain,
`` make clean && make `` should work around most dependency problems, at
the expense of longer build times.
make platform
^^^^^^^^^^^^^
Build the `` python `` program, but don't build the standard library
extension modules. This generates a file named `` platform `` which
contains a single line describing the details of the build platform,
e.g., `` macosx-14.3-arm64-3.12 `` or `` linux-x86_64-3.13 `` .
make profile-opt
^^^^^^^^^^^^^^^^
Build Python using profile-guided optimization (PGO). You can use the
configure :option: `--enable-optimizations` option to make this the
default target of the `` make `` command (`` make all `` or just
`` make `` ).
make clean
^^^^^^^^^^
Remove built files.
make distclean
^^^^^^^^^^^^^^
2024-07-31 21:26:09 -03:00
In addition to the work done by `` make clean `` , remove files
2024-03-07 13:21:28 -04:00
created by the configure script. `` configure `` will have to be run
before building again. [#]_
make install
^^^^^^^^^^^^
Build the `` all `` target and install Python.
make test
^^^^^^^^^
Build the `` all `` target and run the Python test suite with the
`` --fast-ci `` option. Variables:
* `` TESTOPTS `` : additional regrtest command-line options.
* `` TESTPYTHONOPTS `` : additional Python command-line options.
* `` TESTTIMEOUT `` : timeout in seconds (default: 10 minutes).
make buildbottest
^^^^^^^^^^^^^^^^^
This is similar to `` make test `` , but uses the `` --slow-ci ``
option and default timeout of 20 minutes, instead of `` --fast-ci `` option.
make regen-all
^^^^^^^^^^^^^^
Regenerate (almost) all generated files. These include (but are not
limited to) bytecode cases, and parser generator file.
`` make regen-stdlib-module-names `` and `` autoconf `` must be run
separately for the remaining `generated files <#generated-files> `_ .
2021-04-09 09:35:10 -03:00
C extensions
------------
Some C extensions are built as built-in modules, like the `` sys `` module.
They are built with the `` Py_BUILD_CORE_BUILTIN `` macro defined.
2024-03-03 11:40:02 -04:00
Built-in modules have no `` __file__ `` attribute:
.. code-block :: pycon
2021-04-09 09:35:10 -03:00
>>> import sys
>>> sys
<module 'sys' (built-in)>
>>> sys.__file__
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
AttributeError: module 'sys' has no attribute '__file__'
2021-10-05 18:48:44 -03:00
Other C extensions are built as dynamic libraries, like the `` _asyncio `` module.
2021-04-09 09:35:10 -03:00
They are built with the `` Py_BUILD_CORE_MODULE `` macro defined.
2024-03-03 11:40:02 -04:00
Example on Linux x86-64:
.. code-block :: pycon
2021-04-09 09:35:10 -03:00
>>> import _asyncio
>>> _asyncio
<module '_asyncio' from '/usr/lib64/python3.9/lib-dynload/_asyncio.cpython-39-x86_64-linux-gnu.so'>
>>> _asyncio.__file__
'/usr/lib64/python3.9/lib-dynload/_asyncio.cpython-39-x86_64-linux-gnu.so'
:file: `Modules/Setup` is used to generate Makefile targets to build C extensions.
At the beginning of the files, C extensions are built as built-in modules.
Extensions defined after the `` *shared* `` marker are built as dynamic libraries.
2023-09-14 09:56:43 -03:00
The :c:macro: `!PyAPI_FUNC()` , :c:macro: `!PyAPI_DATA()` and
:c:macro: `PyMODINIT_FUNC` macros of :file: `Include/exports.h` are defined
2021-04-09 09:35:10 -03:00
differently depending if the `` Py_BUILD_CORE_MODULE `` macro is defined:
* Use `` Py_EXPORTED_SYMBOL `` if the `` Py_BUILD_CORE_MODULE `` is defined
* Use `` Py_IMPORTED_SYMBOL `` otherwise.
If the `` Py_BUILD_CORE_BUILTIN `` macro is used by mistake on a C extension
2023-09-23 03:31:20 -03:00
built as a shared library, its :samp: `PyInit_{xxx}()` function is not exported,
2021-04-09 09:35:10 -03:00
causing an :exc: `ImportError` on import.
2021-04-08 17:32:21 -03:00
Compiler and linker flags
=========================
2021-04-08 20:58:20 -03:00
Options set by the `` ./configure `` script and environment variables and used by
`` Makefile `` .
2021-04-08 17:32:21 -03:00
2021-04-08 20:58:20 -03:00
Preprocessor flags
------------------
2021-04-08 17:32:21 -03:00
.. envvar :: CONFIGURE_CPPFLAGS
2021-04-08 20:58:20 -03:00
Value of :envvar: `CPPFLAGS` variable passed to the `` ./configure `` script.
2021-04-08 17:32:21 -03:00
.. versionadded :: 3.6
.. envvar :: CPPFLAGS
2023-09-23 03:31:20 -03:00
(Objective) C/C++ preprocessor flags, e.g. :samp: `-I{include_dir}` if you have
headers in a nonstandard directory *include_dir* .
2021-04-08 17:32:21 -03:00
Both :envvar: `CPPFLAGS` and :envvar: `LDFLAGS` need to contain the shell's
2023-08-16 18:13:05 -03:00
value to be able to build extension modules using the
2021-04-08 17:32:21 -03:00
directories specified in the environment variables.
.. envvar :: BASECPPFLAGS
.. versionadded :: 3.4
.. envvar :: PY_CPPFLAGS
Extra preprocessor flags added for building the interpreter object files.
Default: `` $(BASECPPFLAGS) -I. -I$(srcdir)/Include $(CONFIGURE_CPPFLAGS) $(CPPFLAGS) `` .
.. versionadded :: 3.2
Compiler flags
--------------
.. envvar :: CC
C compiler command.
2021-04-08 20:58:20 -03:00
Example: `` gcc -pthread `` .
.. envvar :: CXX
C++ compiler command.
Example: `` g++ -pthread `` .
2021-04-08 17:32:21 -03:00
.. envvar :: CFLAGS
C compiler flags.
.. envvar :: CFLAGS_NODIST
:envvar: `CFLAGS_NODIST` is used for building the interpreter and stdlib C
2023-08-16 18:13:05 -03:00
extensions. Use it when a compiler flag should *not* be part of
:envvar: `CFLAGS` once Python is installed (:gh: `65320` ).
2021-04-08 17:32:21 -03:00
2022-06-20 10:58:02 -03:00
In particular, :envvar: `CFLAGS` should not contain:
2022-10-06 22:01:30 -03:00
* the compiler flag `` -I `` (for setting the search path for include files).
The `` -I `` flags are processed from left to right, and any flags in
:envvar: `CFLAGS` would take precedence over user- and package-supplied `` -I ``
2022-06-20 10:58:02 -03:00
flags.
2022-10-06 22:01:30 -03:00
* hardening flags such as `` -Werror `` because distributions cannot control
2022-06-20 10:58:02 -03:00
whether packages installed by users conform to such heightened
standards.
2021-04-08 17:32:21 -03:00
.. versionadded :: 3.5
2022-11-14 08:43:45 -04:00
.. envvar :: COMPILEALL_OPTS
Options passed to the :mod: `compileall` command line when building PYC files
in `` make install `` . Default: `` -j0 `` .
.. versionadded :: 3.12
2021-04-08 17:32:21 -03:00
.. envvar :: EXTRA_CFLAGS
Extra C compiler flags.
.. envvar :: CONFIGURE_CFLAGS
2021-04-08 20:58:20 -03:00
Value of :envvar: `CFLAGS` variable passed to the `` ./configure ``
script.
2021-04-08 17:32:21 -03:00
.. versionadded :: 3.2
.. envvar :: CONFIGURE_CFLAGS_NODIST
2021-04-08 20:58:20 -03:00
Value of :envvar: `CFLAGS_NODIST` variable passed to the `` ./configure ``
script.
2021-04-08 17:32:21 -03:00
.. versionadded :: 3.5
.. envvar :: BASECFLAGS
2021-04-08 20:58:20 -03:00
Base compiler flags.
2021-04-08 17:32:21 -03:00
.. envvar :: OPT
Optimization flags.
.. envvar :: CFLAGS_ALIASING
Strict or non-strict aliasing flags used to compile `` Python/dtoa.c `` .
.. versionadded :: 3.7
2021-04-08 20:58:20 -03:00
.. envvar :: CCSHARED
Compiler flags used to build a shared library.
For example, `` -fPIC `` is used on Linux and on BSD.
2021-04-08 17:32:21 -03:00
.. envvar :: CFLAGSFORSHARED
Extra C flags added for building the interpreter object files.
2021-04-08 20:58:20 -03:00
Default: `` $(CCSHARED) `` when :option: `--enable-shared` is used, or an empty
string otherwise.
2021-04-08 17:32:21 -03:00
.. envvar :: PY_CFLAGS
Default: `` $(BASECFLAGS) $(OPT) $(CONFIGURE_CFLAGS) $(CFLAGS) $(EXTRA_CFLAGS) `` .
.. envvar :: PY_CFLAGS_NODIST
Default: `` $(CONFIGURE_CFLAGS_NODIST) $(CFLAGS_NODIST) -I$(srcdir)/Include/internal `` .
.. versionadded :: 3.5
.. envvar :: PY_STDMODULE_CFLAGS
C flags used for building the interpreter object files.
Default: `` $(PY_CFLAGS) $(PY_CFLAGS_NODIST) $(PY_CPPFLAGS) $(CFLAGSFORSHARED) `` .
.. versionadded :: 3.7
.. envvar :: PY_CORE_CFLAGS
Default: `` $(PY_STDMODULE_CFLAGS) -DPy_BUILD_CORE `` .
.. versionadded :: 3.2
.. envvar :: PY_BUILTIN_MODULE_CFLAGS
2021-04-08 20:58:20 -03:00
Compiler flags to build a standard library extension module as a built-in
module, like the :mod: `posix` module.
2021-04-08 17:32:21 -03:00
Default: `` $(PY_STDMODULE_CFLAGS) -DPy_BUILD_CORE_BUILTIN `` .
.. versionadded :: 3.8
2021-04-09 09:35:10 -03:00
.. envvar :: PURIFY
Purify command. Purify is a memory debugger program.
Default: empty string (not used).
2021-04-08 17:32:21 -03:00
Linker flags
------------
2021-04-09 09:35:10 -03:00
.. envvar :: LINKCC
Linker command used to build programs like `` python `` and `` _testembed `` .
2022-08-05 08:26:58 -03:00
Default: `` $(PURIFY) $(CC) `` .
2021-04-09 09:35:10 -03:00
2021-04-08 17:32:21 -03:00
.. envvar :: CONFIGURE_LDFLAGS
2021-04-08 20:58:20 -03:00
Value of :envvar: `LDFLAGS` variable passed to the `` ./configure `` script.
2021-04-08 17:32:21 -03:00
Avoid assigning :envvar: `CFLAGS` , :envvar: `LDFLAGS` , etc. so users can use
them on the command line to append to these values without stomping the
pre-set values.
.. versionadded :: 3.2
.. envvar :: LDFLAGS_NODIST
:envvar: `LDFLAGS_NODIST` is used in the same manner as
:envvar: `CFLAGS_NODIST` . Use it when a linker flag should *not* be part of
2023-08-16 18:13:05 -03:00
:envvar: `LDFLAGS` once Python is installed (:gh: `65320` ).
2021-04-08 17:32:21 -03:00
2022-06-20 10:58:02 -03:00
In particular, :envvar: `LDFLAGS` should not contain:
2022-10-06 22:01:30 -03:00
* the compiler flag `` -L `` (for setting the search path for libraries).
The `` -L `` flags are processed from left to right, and any flags in
:envvar: `LDFLAGS` would take precedence over user- and package-supplied `` -L ``
2022-06-20 10:58:02 -03:00
flags.
2021-04-08 17:32:21 -03:00
.. envvar :: CONFIGURE_LDFLAGS_NODIST
2021-04-08 20:58:20 -03:00
Value of :envvar: `LDFLAGS_NODIST` variable passed to the `` ./configure ``
script.
2021-04-08 17:32:21 -03:00
.. versionadded :: 3.8
.. envvar :: LDFLAGS
2023-09-23 03:31:20 -03:00
Linker flags, e.g. :samp: `-L{lib_dir}` if you have libraries in a nonstandard
directory *lib_dir* .
2021-04-08 17:32:21 -03:00
Both :envvar: `CPPFLAGS` and :envvar: `LDFLAGS` need to contain the shell's
2023-08-16 18:13:05 -03:00
value to be able to build extension modules using the
2021-04-08 17:32:21 -03:00
directories specified in the environment variables.
.. envvar :: LIBS
2021-04-08 20:58:20 -03:00
Linker flags to pass libraries to the linker when linking the Python
executable.
Example: `` -lrt `` .
2021-04-08 17:32:21 -03:00
.. envvar :: LDSHARED
Command to build a shared library.
Default: `` @LDSHARED@ $(PY_LDFLAGS) `` .
.. envvar :: BLDSHARED
2021-04-08 20:58:20 -03:00
Command to build `` libpython `` shared library.
2021-04-08 17:32:21 -03:00
Default: `` @BLDSHARED@ $(PY_CORE_LDFLAGS) `` .
.. envvar :: PY_LDFLAGS
Default: `` $(CONFIGURE_LDFLAGS) $(LDFLAGS) `` .
.. envvar :: PY_LDFLAGS_NODIST
Default: `` $(CONFIGURE_LDFLAGS_NODIST) $(LDFLAGS_NODIST) `` .
.. versionadded :: 3.8
.. envvar :: PY_CORE_LDFLAGS
Linker flags used for building the interpreter object files.
.. versionadded :: 3.8
2024-03-07 13:21:28 -04:00
.. rubric :: Footnotes
.. [#] `` git clean -fdx `` is an even more extreme way to "clean" your
checkout. It removes all files not known to Git.
When bug hunting using `` git bisect `` , this is
`recommended between probes <https://github.com/python/cpython/issues/114505#issuecomment-1907021718> `_
to guarantee a completely clean build. **Use with care** , as it
will delete all files not checked into Git, including your
new, uncommitted work.