Commit Graph

367 Commits

Author SHA1 Message Date
Jessica Clarke dec0757549
bpo-43179: Generalise alignment for optimised string routines (GH-24624)
* Remove m68k-specific hack from ascii_decode

On m68k, alignments of primitives is more relaxed, with 4-byte and
8-byte types only requiring 2-byte alignment, thus using sizeof(size_t)
does not work. Instead, use the portable alternative.

Note that this is a minimal fix that only relaxes the assertion and the
condition for when to use the optimised version remains overly strict.
Such issues will be fixed tree-wide in the next commit.

NB: In C11 we could use _Alignof(size_t) instead, but for compatibility
we use autoconf.

* Optimise string routines for architectures with non-natural alignment

C only requires that sizeof(x) is a multiple of alignof(x), not that the
two are equal. Thus anywhere where we optimise based on alignment we
should be using alignof(x) not sizeof(x).

This is more annoying than it would be in C11 where we could just use
_Alignof(x) (and alignof(x) in C++11), but since we still require only
C99 we must plumb the information all the way from autoconf through the
various typedefs and defines.
2021-03-31 12:12:39 +02:00
Roland Hieber e1f7769513
bpo-13501: allow choosing between readline and libedit (GH-24189)
In contrast to macOS, libedit is available as its own include file and
library on Linux systems to prevent file name clashes. So if both
libraries are available on the system, readline is currently chosen by
default; and if only libedit is available, it is not found at all. This
patch adds a way to link against libedit by adding the following
arguments to configure:

  --with-readline           link against libreadline (the default)
  --with-readline=editline  link against libeditline
  --with-readline=no        disable building the readline module
  --without-readline        (same)

The runtime detection of libedit vs. readline was already done in commit
7105319ada (2019-12-04, serge-sans-paille: "bpo-38634: Allow
non-apple build to cope with libedit (GH-16986)").

Fixes: GH-12076 ("bpo-13501 Build or disable readline with Editline")
Fixes: bpo-13501 ("Make libedit support more generic; port readline / libedit to FreeBSD")
Co-authored-by: Enji Cooper (ngie-eign)
Co-authored-by: Martin Panter (vadmium)
Co-authored-by: Robert Marshall (kellinm)
2021-02-08 17:05:25 -08:00
Victor Stinner 52a327c1cb
bpo-39465: Add pycore_atomic_funcs.h header (GH-20766)
Add pycore_atomic_funcs.h internal header file: similar to
pycore_atomic.h but don't require to declare variables as atomic.

Add _Py_atomic_size_get() and _Py_atomic_size_set() functions.
2020-12-23 03:41:08 +01:00
Pablo Galindo a57b3d30f6
bpo-41625: Expose the splice() system call in the os module (GH-21947) 2020-11-17 00:00:38 +00:00
Christian Heimes cd9fed6afb
bpo-41001: Add os.eventfd() (#20930)
Co-authored-by: Kyle Stanley <aeros167@gmail.com>
2020-11-13 19:48:52 +01:00
Ronald Oussoren 41761933c1
bpo-41100: Support macOS 11 and Apple Silicon (GH-22855)
Co-authored-by:  Lawrence D’Anna <lawrence_danna@apple.com>

* Add support for macOS 11 and Apple Silicon (aka arm64)
   
  As a side effect of this work use the system copy of libffi on macOS, and remove the vendored copy

* Support building on recent versions of macOS while deploying to older versions

  This allows building installers on macOS 11 while still supporting macOS 10.9.
2020-11-08 10:05:27 +01:00
Alexey Izbyshev 976da903a7
bpo-35823: subprocess: Use vfork() instead of fork() on Linux when safe (GH-11671)
* bpo-35823: subprocess: Use vfork() instead of fork() on Linux when safe

When used to run a new executable image, fork() is not a good choice
for process creation, especially if the parent has a large working set:
fork() needs to copy page tables, which is slow, and may fail on systems
where overcommit is disabled, despite that the child is not going to
touch most of its address space.

Currently, subprocess is capable of using posix_spawn() instead, which
normally provides much better performance. However, posix_spawn() does not
support many of child setup operations exposed by subprocess.Popen().
Most notably, it's not possible to express `close_fds=True`, which
happens to be the default, via posix_spawn(). As a result, most users
can't benefit from faster process creation, at least not without
changing their code.

However, Linux provides vfork() system call, which creates a new process
without copying the address space of the parent, and which is actually
used by C libraries to efficiently implement posix_spawn(). Due to sharing
of the address space and even the stack with the parent, extreme care
is required to use vfork(). At least the following restrictions must hold:

* No signal handlers must execute in the child process. Otherwise, they
  might clobber memory shared with the parent, potentially confusing it.

* Any library function called after vfork() in the child must be
  async-signal-safe (as for fork()), but it must also not interact with any
  library state in a way that might break due to address space sharing
  and/or lack of any preparations performed by libraries on normal fork().
  POSIX.1 permits to call only execve() and _exit(), and later revisions
  remove vfork() specification entirely. In practice, however, almost all
  operations needed by subprocess.Popen() can be safely implemented on
  Linux.

* Due to sharing of the stack with the parent, the child must be careful
  not to clobber local variables that are alive across vfork() call.
  Compilers are normally aware of this and take extra care with vfork()
  (and setjmp(), which has a similar problem).

* In case the parent is privileged, special attention must be paid to vfork()
  use, because sharing an address space across different privilege domains
  is insecure[1].

This patch adds support for using vfork() instead of fork() on Linux
when it's possible to do safely given the above. In particular:

* vfork() is not used if credential switch is requested. The reverse case
  (simple subprocess.Popen() but another application thread switches
  credentials concurrently) is not possible for pure-Python apps because
  subprocess.Popen() and functions like os.setuid() are mutually excluded
  via GIL. We might also consider to add a way to opt-out of vfork() (and
  posix_spawn() on platforms where it might be implemented via vfork()) in
  a future PR.

* vfork() is not used if `preexec_fn != None`.

With this change, subprocess will still use posix_spawn() if possible, but
will fallback to vfork() on Linux in most cases, and, failing that,
to fork().

[1] https://ewontfix.com/7

Co-authored-by: Gregory P. Smith [Google LLC] <gps@google.com>
2020-10-23 17:47:01 -07:00
Kyle Evans 1800c60080
bpo-40423: Optimization: use close_range(2) if available (GH-22651)
close_range(2) should be preferred at all times if it's available, otherwise we'll use closefrom(2) if available with a fallback to fdwalk(3) or plain old loop over fd range in order of most efficient to least.

[note that this version does check for ENOSYS, but currently ignores all other errors]

Automerge-Triggered-By: @pablogsal
2020-10-11 13:18:53 -07:00
wasiher 5798f78777
bpo-41340: Removed fallback implementation for strdup (GH-21634) 2020-07-27 12:28:45 +09:00
Christian Heimes 9b60e55db2
bpo-40637: Add option to disable builtin hashes (GH-20121)
Signed-off-by: Christian Heimes <christian@python.org>

Automerge-Triggered-By: @tiran
2020-05-15 14:54:53 -07:00
Victor Stinner c5fa364f4e
bpo-40514: Add --with-experimental-isolated-subinterpreters (GH-19926)
Add --with-experimental-isolated-subinterpreters build option to
configure: better isolate subinterpreters, experimental build mode.

When used, force the usage of the libc malloc() memory allocator,
since pymalloc relies on the unique global interpreter lock (GIL).
2020-05-05 16:41:11 +02:00
karl ding 360371f79c
bpo-40291: Add support for CAN_J1939 sockets (GH-19538)
Add support for CAN_J1939 sockets that wrap SAE J1939 protocol
functionality provided by Linux 5.4+ kernels.
2020-04-29 15:31:19 -07:00
Zackery Spytz 97e0de04b8
bpo-25780: Expose CAN_RAW_JOIN_FILTERS in the socket module (GH-19190)
Co-Authored-By: Stefan Tatschner <stefan@rumpelsepp.org>
2020-04-09 13:03:49 +01:00
Stefan Krah 815280eb16
bpo-39794: Add --without-decimal-contextvar (#18702) 2020-02-29 19:43:42 +01:00
Victor Stinner b8d1262e8a
bpo-39395: putenv() and unsetenv() always available (GH-18135)
The os.putenv() and os.unsetenv() functions are now always available.

On non-Windows platforms, Python now requires setenv() and unsetenv()
functions to build.

Remove putenv_dict from posixmodule.c: it's not longer needed.
2020-01-24 14:05:48 +01:00
Victor Stinner b477d19a6b
bpo-39406: Implement os.putenv() with setenv() if available (GH-18128)
If setenv() C function is available, os.putenv() is now implemented
with setenv() instead of putenv(), so Python doesn't have to handle
the environment variable memory.
2020-01-22 22:48:16 +01:00
Michael Felt 39afa2d314 bpo-38021: Modify AIX platform_tag so it covers PEP 425 needs (GH-17303)
Provides a richer platform tag for AIX that we expect to be sufficient for PEP 425
binary distribution identification. Any backports to earlier Python versions will be
handled via setuptools.

Patch by Michael Felt.
2019-12-16 00:17:53 +10:00
Benjamin Peterson 5c0c325453 closes bpo-38713: Expose P_PIDFD in os if it's defined. (GH-17071)
https://bugs.python.org/issue38713
2019-11-05 21:58:31 -08:00
Victor Stinner 028f7349a0
bpo-37415: Fix stdatomic.h header check for ICC compiler (GH-16717)
Fix stdatomic.h header check for ICC compiler: the ICC implementation
lacks atomic_uintptr_t type which is needed by Python.

Test:

* atomic_int and atomic_uintptr_t types
* atomic_load_explicit() and atomic_store_explicit()
* memory_order_relaxed and memory_order_seq_cst constants

But don't test ATOMIC_VAR_INIT(): it's not used in Python.
2019-10-22 21:53:50 +02:00
Jakub Kulík e20134f889 bpo-38110: Use fdwalk for os.closerange() when available. (GH-15224)
Use fdwalk() on platforms that support it to implement os.closerange().
2019-09-11 10:11:57 -05:00
Benjamin Peterson f1c19031fd bpo-38068: Clean up gettimeofday configure logic. (GH-15775)
Assume gettimeofday exists and takes two arguments.
2019-09-10 03:37:59 -07:00
Sergey Fedoseev 52c1a6a15a Remove leftovers from the times when long long wasn't required (GH-15501)
In a38e9d1399 pyconfig.h.in was
manually edited and that edit was overwritten when running autoreconf.
2019-08-26 16:12:47 +01:00
Sergey Fedoseev a38e9d1399 bpo-27961: Remove leftovers from the times when long long wasn't required (GH-15388) 2019-08-22 16:28:28 +01:00
Pablo Galindo aac4d0342c
bpo-26826: Expose copy_file_range in the os module (GH-7255) 2019-05-31 19:39:47 +01:00
Zackery Spytz 43fdbd2729 bpo-26836: Add os.memfd_create() (#13567)
* bpo-26836: Add os.memfd_create()

* Use the glibc wrapper for memfd_create()

Co-Authored-By: Christian Heimes <christian@python.org>

* Fix deletions caused by autoreconf.

* Use MFD_CLOEXEC as the default value for *flags*.

* Add memset_s to configure.ac.

* Revert memset_s changes.

* Apply the requested changes.

* Tweak the docs.
2019-05-29 21:57:03 +02:00
Zackery Spytz 02db696732 bpo-32941: Add madvise() for mmap objects (GH-6172)
Allow mmap objects to access the madvise() system call.
2019-05-27 18:48:16 +02:00
David Carlier 51aa35e9e1 bpo-33164: update blake2 implementation (GH-6286) 2019-05-23 13:32:44 +09:00
pxinwr f2d7ac7e5b bpo-31904: Add posix module support for VxWorks (GH-12118) 2019-05-21 12:46:37 +02:00
Victor Stinner 5422e3cfb7
bpo-36722: Debug build loads libraries built in release mode (GH-12952)
In debug build, import now also looks for C extensions compiled in
release mode and for C extensions compiled in the stable ABI.
2019-04-26 01:40:00 +02:00
Victor Stinner f4e4703e74
bpo-36465: Make release and debug ABI compatible (GH-12615)
Release build and debug build are now ABI compatible: the Py_DEBUG
define no longer implies Py_TRACE_REFS define which introduces the
only ABI incompatibility.

A new "./configure --with-trace-refs" build option is now required to
get Py_TRACE_REFS define which adds sys.getobjects() function and
PYTHONDUMPREFS environment variable.

Changes:

* Add ./configure --with-trace-refs
* Py_DEBUG no longer implies Py_TRACE_REFS
2019-04-25 00:56:28 +02:00
Benjamin Peterson b84df2d7cc Run autoreconf. 2019-02-25 21:37:59 -08:00
Inada Naoki 001fee14e0
bpo-12822: use monotonic clock for condvar if possible (GH-11723) 2019-02-20 10:00:09 +09:00
Neil Schemenauer 5741c45acf
bpo-35903: Use autoconfig to probe for shm_open() and shm_unlink(). (#11765)
Use autoconfig to probe for shm_open() and shm_unlink().  Set SHM_NEEDS_LIBRT if we must
link with librt to get the shm_* functions.  Change setup.py to use the autoconfig defines.  These
changes should make it more likely that _multiprocessing/posixshmem.c gets built correctly on
different platforms.
2019-02-08 10:48:46 -08:00
Joannah Nanjekye 92b8322e7e bpo-35674: Add os.posix_spawnp() (GH-11554)
Add a new os.posix_spawnp() function.
2019-01-16 14:29:26 +01:00
Gregory P. Smith 387512c7ec
bpo-28503: Use crypt_r() when available instead of crypt() (GH-11373)
Use crypt_r() when available instead of crypt() in the crypt module.

As a nice side effect: This also avoids a memory sanitizer flake as clang msan doesn't know about crypt's internal libc allocated buffer.
2018-12-30 15:42:32 -08:00
Bjorn Andersson bb8165172a bpo-31425: Expose AF_QIPCRTR in socket module (GH-3706)
The AF_QIPCRTR address family was introduced in Linux v4.7.

Co-authored-by: Bjorn Andersson <bjorn.andersson@linaro.org>
2018-09-26 16:47:52 +03:00
Benjamin Peterson b3b8cb419e
run autoconf (GH-9411)
Follow up to 2a9c3805dd (bpo-34585).
2018-09-18 23:49:05 -07:00
Benjamin Peterson 865c17fb28
closes bpo-34640: Remove the TANH_PRESERVES_ZERO_SIGN configure check. (GH-9206) 2018-09-12 06:51:18 -07:00
Benjamin Peterson 50c99d917c Remove configure check LOG1P_DROPS_ZERO_SIGN. (GH-9193)
It is unused.

<!--
Thanks for your contribution!
Please read this comment in its entirety. It's quite important.

# Pull Request title

It should be in the following format:

```
bpo-NNNN: Summary of the changes made
```

Where: bpo-NNNN refers to the issue number in the https://bugs.python.org.

Most PRs will require an issue number. Trivial changes, like fixing a typo, do not need an issue.

# Backport Pull Request title

If this is a backport PR (PR made against branches other than `master`),
please ensure that the PR title is in the following format:

```
[X.Y] <title from the original PR> (GH-NNNN)
```

Where: [X.Y] is the branch name, e.g. [3.6].

GH-NNNN refers to the PR number from `master`.

-->
2018-09-11 16:30:04 -07:00
William Grzybowski 23e65b2555 bpo-33625: Release GIL for grp.getgr{nam,gid} and pwd.getpw{nam,uid} (GH-7081)
Release GIL on grp.getgrnam(), grp.getgrgid(), pwd.getpwnam() and
pwd.getpwuid() if reentrant variants of these functions are available.

Patch by William Grzybowski.
2018-09-07 14:06:15 +02:00
Michael Osipov 48ce4897f8 bpo-34412: Make signal.strsignal() work on HP-UX (GH-8786)
Introduce a configure check for strsignal(3) which defines HAVE_STRSIGNAL for
signalmodule.c. Add some common signals on HP-UX. This change applies for
Windows and HP-UX.
2018-08-23 16:27:19 +03:00
Benjamin Peterson 15c7b2abdf
bpo-34121: Fix detection of C11 atomic support on clang. (GH-8288) 2018-07-15 17:01:42 -07:00
Eitan Adler b91a3a0d61 bpo-33648: Remove PY_WARN_ON_C_LOCALE (GH-7114)
This code does not appear to be used anywhere in the python code base.
The use was removed in eb81795d7d.
2018-07-11 20:01:27 +09:00
Serhiy Storchaka 17d8830312 bpo-32493: Fix uuid.uuid1() on FreeBSD. (GH-7099)
Use uuid_enc_be() if available to encode UUID to bytes as big endian.
2018-05-25 00:45:09 +02:00
Eitan Adler 3055c947f9 closes bpo-33512: use standard for detecting long double (GH-6847) 2018-05-15 22:58:09 -07:00
Antoine Pitrou 9d3627e311
bpo-33332: Add signal.valid_signals() (GH-6581) 2018-05-04 13:00:50 +02:00
Christian Heimes 892d66e422
bpo-31429: Define TLS cipher suite on build time (#3532)
Until now Python used a hard coded white list of default TLS cipher
suites. The old approach has multiple downsides. OpenSSL's default
selection was completely overruled. Python did neither benefit from new
cipher suites (ChaCha20, TLS 1.3 suites) nor blacklisted cipher suites.
For example we used to re-enable 3DES.

Python now defaults to OpenSSL DEFAULT cipher suite selection and black
lists all unwanted ciphers. Downstream vendors can override the default
cipher list with --with-ssl-default-suites.

Signed-off-by: Christian Heimes <christian@python.org>
2018-01-29 14:10:18 +01:00
Pablo Galindo 6c6ddf97c4 bpo-20104: Expose `posix_spawn` in the os module (GH-5109)
Add os.posix_spawn to wrap the low level POSIX API of the same name.

Contributed by Pablo Galindo.
2018-01-28 17:56:10 -08:00
Pablo Galindo 4defba3b95 bpo-31368: Expose preadv and pwritev in the os module (#5239) 2018-01-27 17:16:37 +01:00
stratakis e768c86ef4 bpo-32635: Fix a segfault when importing the crypt module with libxcrypt. (#5284)
glibc is deprecating libcrypt in favor of libxcrypt, however python assumes
that crypt.h will always be included. This change makes the header inclusion
explicit when libxcrypt is present on the system.
2018-01-23 16:11:24 +01:00