This changeset increases the default size of the stack
for threads on macOS to the size of the stack
of the main thread and reenables the relevant
recursion test.
Reduce the number of unit tests run for the PGO generation task. This
speeds up the task by a factor of about 15x. Running the full unit test
suite is slow. This change may result in a slightly less optimized build
since not as many code branches will be executed. If you are willing to
wait for the much slower build, the old behavior can be restored using
'./configure [..] PROFILE_TASK="-m test --pgo-extended"'. We make no
guarantees as to which PGO task set produces a faster build. Users who
care should run their own relevant benchmarks as results can depend on
the environment, workload, and compiler tool chain.
Under some conditions the earlier fix for bpo-18075, "Infinite recursion
tests triggering a segfault on Mac OS X", now causes failures on macOS
when attempting to change stack limit with resource.setrlimit
resource.RLIMIT_STACK, like regrtest does when running the test suite.
The reverted change had specified a non-default stack size when linking
the python executable on macOS. As of macOS 10.14.4, the previous
code causes a hard failure when running tests, although similar
failures had been seen under some conditions under some earlier
systems. Reverting the change to the interpreter stack size at link
time helped for release builds but caused some tests to fail when
built --with-pydebug. Try the opposite approach: continue to build
the interpreter with an increased stack size on macOS and remove
the failing setrlimit call in regrtest initialization. This will
definitely avoid the resource.RLIMIT_STACK error and should have
no, or fewer, side effects.
* bpo-26836: Add os.memfd_create()
* Use the glibc wrapper for memfd_create()
Co-Authored-By: Christian Heimes <christian@python.org>
* Fix deletions caused by autoreconf.
* Use MFD_CLOEXEC as the default value for *flags*.
* Add memset_s to configure.ac.
* Revert memset_s changes.
* Apply the requested changes.
* Tweak the docs.
It is also possible to link against a library or executable with a
statically linked libpython, but not both with the same DLL. In fact
building a statically linked python is currently broken on Cygwin
for other (related) reasons.
The same problem applies to other POSIX-like layers over Windows
(MinGW, MSYS) but Python's build system does not seem to attempt
to support those platforms at the moment.
To embed Python into an application, a new --embed option must be
passed to "python3-config --libs --embed" to get "-lpython3.8" (link
the application to libpython). To support both 3.8 and older, try
"python3-config --libs --embed" first and fallback to "python3-config
--libs" (without --embed) if the previous command fails.
Add a pkg-config "python-3.8-embed" module to embed Python into an
application: "pkg-config python-3.8-embed --libs" includes
"-lpython3.8". To support both 3.8 and older, try "pkg-config
python-X.Y-embed --libs" first and fallback to "pkg-config python-X.Y
--libs" (without --embed) if the previous command fails (replace
"X.Y" with the Python version).
On the other hand, "pkg-config python3.8 --libs" no longer contains
"-lpython3.8". C extensions must not be linked to libpython (except
on Android, case handled by the script); this change is backward
incompatible on purpose.
"make install" now also installs "python-3.8-embed.pc".
Under some conditions the earlier fix for bpo-18075, "Infinite recursion
tests triggering a segfault on Mac OS X", now causes failures on macOS
when attempting to change stack limit with resource.setrlimit
resource.RLIMIT_STACK, like regrtest does when running the test suite.
The reverted change had specified a non-default stack size when linking
the python executable on macOS. As of macOS 10.14.4, the previous
code causes a hard failure when running tests, although similar
failures had been seen under some conditions under some earlier
systems. For now, revert the original change and resume using
the default stack size when linking the interpreter.
Release build and debug build are now ABI compatible: the Py_DEBUG
define no longer implies Py_TRACE_REFS define which introduces the
only ABI incompatibility.
A new "./configure --with-trace-refs" build option is now required to
get Py_TRACE_REFS define which adds sys.getobjects() function and
PYTHONDUMPREFS environment variable.
Changes:
* Add ./configure --with-trace-refs
* Py_DEBUG no longer implies Py_TRACE_REFS
"./configure --with-pymalloc" no longer adds the "m" flag to SOABI
(sys.implementation.cache_tag).
Enabling or disabling pymalloc has no impact on the ABI.
Add -fmax-type-align=8 to CFLAGS when clang compiler is detected.
The pymalloc memory allocator aligns memory on 8 bytes. On x86-64,
clang expects alignment on 16 bytes by default and so uses MOVAPS
instruction which can lead to segmentation fault. Instruct clang that
Python is limited to alignemnt on 8 bytes to use MOVUPS instruction
instead: slower but don't trigger a SIGSEGV if the memory is not
aligned on 16 bytes.
Sadly, the flag must be expected to CFLAGS and not just
CFLAGS_NODIST, since third party C extensions can have the same
issue.
On AIX, sys.platform doesn't contain the major version anymore.
Always return 'aix', instead of 'aix3' .. 'aix7'. Since
older Python versions include the version number, it is recommended to
always use sys.platform.startswith('aix').
Per POSIX, `nice(3)` requires `unistd.h` and `exit(3)` requires `stdlib.h`.
Fixing the test will prevent false positives with pedantic compilers like clang.
Use autoconfig to probe for shm_open() and shm_unlink(). Set SHM_NEEDS_LIBRT if we must
link with librt to get the shm_* functions. Change setup.py to use the autoconfig defines. These
changes should make it more likely that _multiprocessing/posixshmem.c gets built correctly on
different platforms.
Use crypt_r() when available instead of crypt() in the crypt module.
As a nice side effect: This also avoids a memory sanitizer flake as clang msan doesn't know about crypt's internal libc allocated buffer.
When compiling 3rd party C extensions, the linker flags used by the
compiler for the interpreter and the stdlib modules, will get
leaked into distutils. In order to avoid that, the PY_CORE_LDFLAGS
and PY_LDFLAGS_NODIST are introduced to keep those flags separated.
When using link time optimizations, the -flto flag is passed to
BASECFLAGS, which makes it propagate to distutils. Those flags
should be reserved for the interpreter and the stdlib extension
modules only, thus moving those flags to CFLAGS_NODIST.
Adds configure flags for msan and ubsan builds to make it easier to enable.
These also encode the detail that address sanitizer and memory sanitizer
should disable pymalloc.
Define MEMORY_SANITIZER when appropriate at build time and adds workarounds
to existing code to mark things as initialized where the sanitizer is otherwise unable to
determine that. This lets our build succeed under the memory sanitizer. not all tests
pass without sanitizer failures yet but we're in pretty good shape after this.
.o generated by clang in LTO mode actually are LLVM bitcode files, which
leads to a few errors during configure/build step:
- add lto flags to the BASECFLAGS instead of CFLAGS, as CFLAGS are used
to build autoconf test case, and some are not compatible with clang LTO
(they assume binary in the .o, not bitcode)
- force llvm-ar instead of ar, as ar is not aware of .o files generated
by clang -flto
Currently configure.ac uses AC_RUN_IFELSE to determine the byte order of doubles, but this silently fails under cross compilation and Python doesn't do floats properly.
Instead, steal a macro from autoconf-archive which compiles code using magic doubles (which encode to ASCII) and grep for the representation in the binary.
RFC because this doesn't yet handle the weird ancient ARMv4 OABI 'mixed-endian' encoding properly. This encoding is ancient and I don't believe the union of "Python 3.8 users" and "OABI users" has anything in. Should the support for this just be dropped too? Alternatively, someone will need to find an OABI toolchain to verify the encoding of the magic double.
It is unused.
<!--
Thanks for your contribution!
Please read this comment in its entirety. It's quite important.
# Pull Request title
It should be in the following format:
```
bpo-NNNN: Summary of the changes made
```
Where: bpo-NNNN refers to the issue number in the https://bugs.python.org.
Most PRs will require an issue number. Trivial changes, like fixing a typo, do not need an issue.
# Backport Pull Request title
If this is a backport PR (PR made against branches other than `master`),
please ensure that the PR title is in the following format:
```
[X.Y] <title from the original PR> (GH-NNNN)
```
Where: [X.Y] is the branch name, e.g. [3.6].
GH-NNNN refers to the PR number from `master`.
-->
Release GIL on grp.getgrnam(), grp.getgrgid(), pwd.getpwnam() and
pwd.getpwuid() if reentrant variants of these functions are available.
Patch by William Grzybowski.
Introduce a configure check for strsignal(3) which defines HAVE_STRSIGNAL for
signalmodule.c. Add some common signals on HP-UX. This change applies for
Windows and HP-UX.
bpo-32430: Rename Modules/Setup.dist to Modules/Setup
Remove the necessity to copy the former manually to the latter when updating the local source tree.
Instead of passing configure args such as --without-gcc or --with-icc,
instead prefer to rely on the native way of finding the compiler:
passing CC (or CPP or CXX depending).
This allows configure to find the correct compiler instead of having to
be explicitly told. It also more correctly builds on both macOS and
FreeBSD since the system compiler is used by default (cc)
This issue covers various changes for the macOS installers provided via python.org for 3.7.0.
- Provide a provisional new installer variant for macOS 10.9 and later systems with 64-bit (x86_64) architecture only. Apple has made it known that future versions of macOS will only fully support 64-bit executables and some other third-party software suppliers have chosen 10.9 as their oldest supported system.
- Support **Tcl/Tk 8.6** with the 10.9 installer variant.
- Upgrade **OpenSSL** to 1.1.0g and **SQLite** to 3.22.0.
- The compiler name used for the interpreter build and for modules built with **Distutils / pip** is now _gcc_ rather than _gcc-4.2_. And extension module builds will no longer try to force use of an old SDK if present.
Until now Python used a hard coded white list of default TLS cipher
suites. The old approach has multiple downsides. OpenSSL's default
selection was completely overruled. Python did neither benefit from new
cipher suites (ChaCha20, TLS 1.3 suites) nor blacklisted cipher suites.
For example we used to re-enable 3DES.
Python now defaults to OpenSSL DEFAULT cipher suite selection and black
lists all unwanted ciphers. Downstream vendors can override the default
cipher list with --with-ssl-default-suites.
Signed-off-by: Christian Heimes <christian@python.org>
glibc is deprecating libcrypt in favor of libxcrypt, however python assumes
that crypt.h will always be included. This change makes the header inclusion
explicit when libxcrypt is present on the system.
Add https://www.gnu.org/software/autoconf-archive/ax_check_openssl.html
to auto-detect compiler flags, linker flags and libraries to compile
OpenSSL extensions. The M4 macro uses pkg-config and falls back to
manual detection.
Add autoconf magic to detect usable X509_VERIFY_PARAM_set1_host()
and related functions.
Refactor setup.py to use new config vars to compile _ssl and _hashlib
modules.
Signed-off-by: Christian Heimes <christian@python.org>
This module has never been enabled by default, never worked correctly
on x86-64, and caused ABI problems that caused C extension
compatibility. See bpo-29137 for details/discussion.
Starting with AIX6.1 there is support in libc.a for uuid (RFC4122)
This patch provides the changes needed for this integration with the OS.
On AIX the base function is uuid_create() rather than uuid_generate_time()
The AIX uuid_t typedef is more aligned to the UUID field based definition
while the Linux typedef that is more aligned with UUID bytes
(or perhaps UUID bytes_le) definitions.
Modify the code to use ncurses is_pad() instead of checking WINDOW
_flags field. If your platform does not provide the is_pad(), the
existing way that checks the field will be enabled.
Note: This change does not drop support for platforms where do not
have both WINDOW _flags field and is_pad().
See PEP 539 for details.
Highlights of changes:
- Add Thread Specific Storage (TSS) API
- Document the Thread Local Storage (TLS) API as deprecated
- Update code that used TLS API to use TSS API
Python requires C implementations provide memmove, so we shouldn't need to check for it. The only place using this configure check was expat, where we can simply always define HAVE_MEMMOVE.
Allow configure --with-lto to apply to all builds, not just profile-opt builds.
Whether this is actually useful or not must be determined by the person
building CPython using their own toolchain.
My own quick test on x86_64 Debian 9 (gcc 6.3, binutils 2.28) seemed
to suggest that it wasn't, but I expect better toolchains can or will exist
at some point. The point is to allow it at all.
* bpo-27584: New addition of vSockets to the python socket module
Support for AF_VSOCK on Linux only
* bpo-27584: Fixes for V2
Fixed syntax and naming problems.
Fixed #ifdef AF_VSOCK checking
Restored original aclocal.m4
* bpo-27584: Fixes for V3
Added checking for fcntl and thread modules.
* bpo-27584: Fixes for V4
Fixed white space error
* bpo-27584: Fixes for V5
Added back comma in (CID, port).
* bpo-27584: Fixes for V6
Added news file.
socket.rst now reflects first Linux introduction of AF_VSOCK.
Fixed get_cid in test_socket.py.
Replaced PyLong_FromLong with PyLong_FromUnsignedLong in socketmodule.c
Got rid of extra AF_VSOCK #define.
Added sockaddr_vm to sock_addr.
* bpo-27584: Fixes for V7
Minor cleanup.
* bpo-27584: Fixes for V8
Put back #undef AF_VSOCK as it is necessary when vm_sockets.h is not installed.
Include sys/sysmacros.h for major(), minor(), and makedev(). GNU C libray
plans to remove the functions from sys/types.h.
Signed-off-by: Christian Heimes <christian@python.org>
`PYTHONFRAMEWORK` is defined in `Makefile` and it shoulnd't be used
in `pyconfig.h`.
`sysconfig.py --generate-posix-vars` reads config vars from Makefile
and `pyconfig.h`. Conflicting variables should be avoided.
Especially, string config variables in Makefile are unquoted, but
in `pyconfig.h` are keep quoted. So it should be private (starts with
underscore).
- new PYTHONCOERCECLOCALE config setting
- coerces legacy C locale to C.UTF-8, C.utf8 or UTF-8 by default
- always uses C.UTF-8 on Android
- uses `surrogateescape` on stdin and stdout in the coercion
target locales
- configure option to disable locale coercion at build time
- configure option to disable C locale warning at build time
Don't rebuild generated files based on file modification time
anymore, the action is now explicit. Replace "make touch"
with "make regen-all".
Changes:
* Remove "make touch", Tools/hg/hgtouch.py and .hgtouch
* Add a new "make regen-all" command to rebuild all generated files
* Add subcommands to only generate specific files:
- regen-ast: Include/Python-ast.h and Python/Python-ast.c
- regen-grammar: Include/graminit.h and Python/graminit.c
- regen-importlib: Python/importlib_external.h and Python/importlib.h
- regen-opcode: Include/opcode.h
- regen-opcode-targets: Python/opcode_targets.h
- regen-typeslots: Objects/typeslots.inc
* Rename PYTHON_FOR_GEN to PYTHON_FOR_REGEN
* pgen is now only built by by "make regen-grammar"
* Add $(srcdir)/ prefix to paths to source files to handle correctly
compilation outside the source directory
Note: $(PYTHON_FOR_REGEN) is no more used nor needed by "make"
default target building Python.
Detect when the "cc" compiler (and the $CC variable) is the Clang
compiler. The test is needed to add the -fno-strict-aliasing option
on FreeBSD where cc is clang.
Python/dtoa.c is not compiled correctly with clang 4.0 and
optimization level -O2 or higher, because of an aliasing issue on
the double/ULong[2] union.
LLVM bug report:
https://bugs.llvm.org//show_bug.cgi?id=31928
The presence of the ``--enable-optimizations`` flag is indicated by the
value of ``$enableval``, but the configure script was checking ``$withval``,
resulting in the ``--enable-optimizations`` flag being effectively ignored.
* Fix `make tags` warnings
`make tags` target tries to find C sources and headers in "Grammar" and
"Mac" folders and generates these warnings:
ctags: Warning: cannot open source file "Grammar/*.[ch]" : No such file or directory
ctags: Warning: cannot open source file "Mac/*.[ch]" : No such file or directory
This commit changes $SRCDIRS variable in configure.ac to remote these
directories. This variable is used only for tags generation.
Also, "configure" was regenerated with `autoreconf`.
* Fix `make tags` fail on non-default tag names
When ctags overrides default tags filename (e.g. `-f .tags`) `make tags`
is failed because it assumes to see default `tags` filename:
sort: cannot read: tags: No such file or directory
This commit explicitly specifies "tags" filename for tags generation.
Use --short form of git hash. Use output from "git describe" for tag.
Expected outputs:
1. previous hg
2. previous git
3. updated git
Release (tagged) build:
1. Python 3.7.0a0 (v3.7.0a0:4def2a2901a5, ...
2. Python 3.7.0a0 (v3.7.0a0^0:05f53735c8912f8df1077e897f052571e13c3496, ...
3. Python 3.7.0a0 (v3.7.0a0:05f53735c8, ...
Development build:
1. Python 3.7.0a0 (default:41df79263a11, ...
2. Python 3.7.0a0 (master:05f53735c8912f8df1077e897f052571e13c3496, ...
3. Python 3.7.0a0 (heads/master-dirty:05f53735c8, ...
"dirty" means the working tree has uncommitted changes.
See "git help describe" for more info.
sys.version and the platform module python_build(),
python_branch(), and python_revision() functions now use
git information rather than hg when building from a repo.
Based on original patches by Brett Cannon and Steve Dower.
Issue #27659: use -Werror=implicit-function-declaration when possible (GCC and
Clang, but it depends on the compiler version).
Patch written by Chi Hsuan Yen.
Patch by Masayuki Yamamoto, reviewed and rebased by Erik Bray.
This is a first step on the long road toward resupporting Cygwin, which does
not provide siginfo_t.si_band.
Tested on macOS 10.11 dtrace, Ubuntu 16.04 SystemTap, and libbcc.
Largely based by an initial patch by Jesús Cea Avión, with some
influence from Dave Malcolm's SystemTap patch and Nikhil Benesch's
unification patch.
Things deliberately left out for simplicity:
- ustack helpers, I have no way of testing them at this point since
they are Solaris-specific
- PyFrameObject * in function__entry/function__return, this is
SystemTap-specific
- SPARC support
- dynamic tracing
- sys module dtrace facility introspection
All of those might be added later.
Too many build tool chains do not properly support it, including building
and linking an executable fine that simply segfaults when you try to run
it (such as debian jessie 8.5's gcc 4.9). On others where it does appear
to build (ubuntu 16.04's gcc 5.4) there are still test_distutils and test_gdb
failures to deal with.
We're not going to spend time attempting to maintain a complicated white list
of what does and doesn't work in our configure.ac file.
Too many build tool chains do not properly support it, including building
and linking an executable fine that simply segfaults when you try to run
it (such as debian jessie 8.5's gcc 4.9). On others where it does appear
to build (ubuntu 16.04's gcc 5.4) there are still test_distutils and test_gdb
failures to deal with.
We're not going to spend time attempting to maintain a complicated white list
of what does and doesn't work in our configure.ac file.
required for PGO linking - to be a configure time error rather than
make time when --with-optimizations is enabled. Also improve our
ability to find the llvm-profdata tool on MacOS and some Linuxes.
required for PGO linking to be a configure time error rather than
make time when --with-optimizations is enabled. Also improve our
ability to find the llvm-profdata tool on MacOS and some Linuxes.
The dependendency on the $(PGEN) variable must only be
set when not cross-compiling. When cross-compiling,
$(PGEN) will not be used, so no need to build it.
Patch by Thomas Perl.
plat-$(PLATFORM_TRIPLET).
Rename the config directory (LIBPL) from config-$(LDVERSION) to
config-$(LDVERSION)-$(PLATFORM_TRIPLET).
Install the platform specifc _sysconfigdata module into the platform
directory and rename it to include the ABIFLAGS.
Issue #26839: On Linux, os.urandom() now calls getrandom() with GRND_NONBLOCK
to fall back on reading /dev/urandom if the urandom entropy pool is not
initialized yet. Patch written by Colm Buckley.
function instead of the getentropy() function. The getentropy() function is
blocking to generate very good quality entropy, os.urandom() doesn't need such
high-quality entropy.
function instead of the getentropy() function. The getentropy() function is
blocking to generate very good quality entropy, os.urandom() doesn't need such
high-quality entropy.
module binaries, with a platform triple of just "darwin", resulting in
file names like:
_ssl.cpython-35m-darwin.so
rather than just _ssl.so as previously.
Instead of attempting to encode differences in CPU architecture and OS X
deployment targets in the file name as is done on other platforms,
these continue to be managed by the use of Apple multi-architecture
("fat") files, by the system dynamic loader, and by logic in higher-levels
like sysconfig.get_platform() and pip.
Vinay Sajip
Jesús Cea
Doyle Rowland
Jakub Kulík
Benjamin Peterson
Sergey Fedoseev
Ronald Oussoren
Neil Schemenauer
Ned Deily
Łukasz Langa
Pablo Galindo
Zackery Spytz
David Carlier
E. M. Bray
Victor Stinner
pxinwr
xdegaye
Michael Felt
ngie-eign
Inada Naoki
Joannah Nanjekye
Gregory P. Smith
stratakis
serge-sans-paille
Bjorn Andersson
Ross Burton
Eitan Adler
William Grzybowski
Michael Osipov
Xiang Zhang
Antoine Pitrou
Serhiy Storchaka
Matthias Klose
Christian Heimes
Victor Stinner
Nathaniel J. Smith
Rob Boehne
Berker Peksag
Masayuki Yamamoto
pdox
octaviansoldea
caavery
Nick Coghlan
Brett Cannon
T. Wouters
Alex Wang
Alex Dzyoba
Xavier de Gaye
Martin Panter
Brett Cannon
Zachary Ware
doko@ubuntu.com
Raymond Hettinger
Stefan Krah
Yury Selivanov
Ned Deily
Larry Hastings