Commit Graph

109 Commits

Author SHA1 Message Date
Christian Heimes 035deee265
bpo-1635741: Port _posixsubprocess module to multiphase init (GH-23406) 2020-11-21 20:28:14 +01:00
David CARLIER 13b865f0e1
bpo-42375: subprocess DragonFlyBSD build update. (GH-23320)
Same as FreeBSD, file descriptors in /dev/fd id from 0 to 63.
2020-11-18 23:24:15 -08:00
Alexey Izbyshev d3b4e06807
bpo-42146: Unify cleanup in subprocess_fork_exec() (GH-22970)
* bpo-42146: Unify cleanup in subprocess_fork_exec()

Also ignore errors from _enable_gc():
* They are always suppressed by the current code due to a bug.
* _enable_gc() is only used if `preexec_fn != None`, which is unsafe.
* We don't have a good way to handle errors in case we successfully
  created a child process.

Co-authored-by: Gregory P. Smith <greg@krypto.org>
2020-10-31 22:33:08 -07:00
Alexey Izbyshev c0590c0033
bpo-42146: Fix memory leak in subprocess.Popen() in case of uid/gid overflow (GH-22966)
Fix memory leak in subprocess.Popen() in case of uid/gid overflow

Also add a test that would catch this leak with `--huntrleaks`.

Alas, the test for `extra_groups` also exposes an inconsistency
in our error reporting: we use a custom ValueError for `extra_groups`,
but propagate OverflowError for `user` and `group`.
2020-10-25 17:09:32 -07:00
Gregory P. Smith be3c3a0e46
bpo-35823: Allow setsid() after vfork() on Linux. (GH-22945)
It should just be a syscall updating a couple of fields in the kernel side
process info.  Confirming, in glibc is appears to be a shim for the setsid
syscall (based on not finding any code implementing anything special for it)
and in uclibc (*much* easier to read) it is clearly just a setsid syscall shim.

A breadcrumb _suggesting_ that it is not allowed on Darwin/macOS comes from
a commit in emacs: https://lists.gnu.org/archive/html/bug-gnu-emacs/2017-04/msg00297.html
but I don't have a way to verify if that is true or not.
As we are not supporting vfork on macOS today I just left a note in a comment.
2020-10-24 12:07:35 -07:00
Alexey Izbyshev 473db47747
bpo-35823: subprocess: Fix handling of pthread_sigmask() errors (GH-22944)
Using POSIX_CALL() is incorrect since pthread_sigmask() returns
the error number instead of setting errno.

Also handle failure of the first call to pthread_sigmask()
in the parent process, and explain why we don't handle failure
of the second call in a comment.
2020-10-24 10:47:38 -07:00
Alexey Izbyshev 976da903a7
bpo-35823: subprocess: Use vfork() instead of fork() on Linux when safe (GH-11671)
* bpo-35823: subprocess: Use vfork() instead of fork() on Linux when safe

When used to run a new executable image, fork() is not a good choice
for process creation, especially if the parent has a large working set:
fork() needs to copy page tables, which is slow, and may fail on systems
where overcommit is disabled, despite that the child is not going to
touch most of its address space.

Currently, subprocess is capable of using posix_spawn() instead, which
normally provides much better performance. However, posix_spawn() does not
support many of child setup operations exposed by subprocess.Popen().
Most notably, it's not possible to express `close_fds=True`, which
happens to be the default, via posix_spawn(). As a result, most users
can't benefit from faster process creation, at least not without
changing their code.

However, Linux provides vfork() system call, which creates a new process
without copying the address space of the parent, and which is actually
used by C libraries to efficiently implement posix_spawn(). Due to sharing
of the address space and even the stack with the parent, extreme care
is required to use vfork(). At least the following restrictions must hold:

* No signal handlers must execute in the child process. Otherwise, they
  might clobber memory shared with the parent, potentially confusing it.

* Any library function called after vfork() in the child must be
  async-signal-safe (as for fork()), but it must also not interact with any
  library state in a way that might break due to address space sharing
  and/or lack of any preparations performed by libraries on normal fork().
  POSIX.1 permits to call only execve() and _exit(), and later revisions
  remove vfork() specification entirely. In practice, however, almost all
  operations needed by subprocess.Popen() can be safely implemented on
  Linux.

* Due to sharing of the stack with the parent, the child must be careful
  not to clobber local variables that are alive across vfork() call.
  Compilers are normally aware of this and take extra care with vfork()
  (and setjmp(), which has a similar problem).

* In case the parent is privileged, special attention must be paid to vfork()
  use, because sharing an address space across different privilege domains
  is insecure[1].

This patch adds support for using vfork() instead of fork() on Linux
when it's possible to do safely given the above. In particular:

* vfork() is not used if credential switch is requested. The reverse case
  (simple subprocess.Popen() but another application thread switches
  credentials concurrently) is not possible for pure-Python apps because
  subprocess.Popen() and functions like os.setuid() are mutually excluded
  via GIL. We might also consider to add a way to opt-out of vfork() (and
  posix_spawn() on platforms where it might be implemented via vfork()) in
  a future PR.

* vfork() is not used if `preexec_fn != None`.

With this change, subprocess will still use posix_spawn() if possible, but
will fallback to vfork() on Linux in most cases, and, failing that,
to fork().

[1] https://ewontfix.com/7

Co-authored-by: Gregory P. Smith [Google LLC] <gps@google.com>
2020-10-23 17:47:01 -07:00
Kyle Evans 7992579cd2
bpo-40422: Move _Py_closerange to fileutils.c (GH-22680)
This API is relatively lightweight and organizationally, given that it's
used by multiple modules, it makes sense to move it to fileutils.

Requires making sure that _posixsubprocess is compiled with the appropriate
Py_BUIILD_CORE_BUILTIN macro.
2020-10-13 22:04:44 +02:00
Kyle Evans c230fde847
bpo-40422: create a common _Py_closerange API (GH-19754)
Such an API can be used both for os.closerange and subprocess. For the latter, this yields potential improvement for platforms that have fdwalk but wouldn't have used it there. This will prove even more beneficial later for platforms that have close_range(2), as the new API will prefer that over all else if it's available.

The new API is structured to look more like close_range(2), closing from [start, end] rather than the [low, high) of os.closerange().

Automerge-Triggered-By: @gpshead
2020-10-11 11:54:11 -07:00
Christian Heimes 0d3350daa8
bpo-40955: Fix memory leak in subprocess module (GH-20825)
```
Direct leak of 8 byte(s) in 1 object(s) allocated from:
    #0 0x7f008bf19667 in __interceptor_malloc (/lib64/libasan.so.6+0xb0667)
    #1 0x7f007a0bee4a in subprocess_fork_exec /home/heimes/dev/python/cpython/Modules/_posixsubprocess.c:774
    #2 0xe0305b in cfunction_call Objects/methodobject.c:546
```

Signed-off-by: Christian Heimes <christian@python.org>
2020-06-12 09:18:43 -07:00
Victor Stinner 252346acd9
bpo-40453: Add PyConfig._isolated_subinterpreter (GH-19820)
An isolated subinterpreter cannot spawn threads, spawn a child
process or call os.fork().

* Add private _Py_NewInterpreter(isolated_subinterpreter) function.
* Add isolated=True keyword-only parameter to
  _xxsubinterpreters.create().
* Allow again os.fork() in "non-isolated" subinterpreters.
2020-05-01 11:33:44 +02:00
Victor Stinner e6f8abd500
bpo-38061: subprocess uses closefrom() on FreeBSD (GH-19697)
Optimize the subprocess module on FreeBSD using closefrom().
A single close(fd) syscall is cheap, but when sysconf(_SC_OPEN_MAX)
is high, the loop calling close(fd) on each file descriptor can take
several milliseconds.

The workaround on FreeBSD to improve performance was to load and
mount the fdescfs kernel module, but this is not enabled by default.

Initial patch by Ed Maste (emaste), Conrad Meyer (cem), Kyle Evans
(kevans) and Kubilay Kocak (koobs):
https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=242274
2020-04-24 12:06:58 +02:00
Hai Shi f707d94af6
bpo-39968: Convert extension modules' macros of get_module_state() to inline functions (GH-19017) 2020-03-16 14:15:01 +01:00
Victor Stinner be79373a78
bpo-39947: Add PyInterpreterState_Get() function (GH-18979)
* Rename _PyInterpreterState_Get() to PyInterpreterState_Get() and
  move it the limited C API.
* Add _PyInterpreterState_Get() alias to PyInterpreterState_Get() for
  backward compatibility with Python 3.8.
2020-03-13 18:15:33 +01:00
Petr Viktorin ffd9753a94
bpo-39245: Switch to public API for Vectorcall (GH-18460)
The bulk of this patch was generated automatically with:

    for name in \
        PyObject_Vectorcall \
        Py_TPFLAGS_HAVE_VECTORCALL \
        PyObject_VectorcallMethod \
        PyVectorcall_Function \
        PyObject_CallOneArg \
        PyObject_CallMethodNoArgs \
        PyObject_CallMethodOneArg \
    ;
    do
        echo $name
        git grep -lwz _$name | xargs -0 sed -i "s/\b_$name\b/$name/g"
    done

    old=_PyObject_FastCallDict
    new=PyObject_VectorcallDict
    git grep -lwz $old | xargs -0 sed -i "s/\b$old\b/$new/g"

and then cleaned up:

- Revert changes to in docs & news
- Revert changes to backcompat defines in headers
- Nudge misaligned comments
2020-02-11 17:46:57 +01:00
Gregory P. Smith f3751efb5c
bpo-38417: Add umask support to subprocess (GH-16726)
On POSIX systems, allow the umask to be set in the child process before we exec.
2019-10-12 13:24:56 -07:00
Orivej Desh 77abf23c67 bpo-6559: Update _posixsubprocess.fork_exec doc (GH-16283)
It did not list the argument added in d4cc7bf993.


https://bugs.python.org/issue6559



Automerge-Triggered-By: @gpshead
2019-09-20 10:01:09 -07:00
Patrick McLean 2b2ead7438 bpo-36046: Add user and group parameters to subprocess (GH-11950)
* subprocess: Add user, group and extra_groups paremeters to subprocess.Popen

This adds a `user` parameter to the Popen constructor that will call
setreuid() in the child before calling exec(). This allows processes
running as root to safely drop privileges before running the subprocess
without having to use a preexec_fn.

This also adds a `group` parameter that will call setregid() in
the child process before calling exec().

Finally an `extra_groups` parameter was added that will call
setgroups() to set the supplimental groups.
2019-09-12 18:15:44 +01:00
Dino Viehland 5a7d2e11aa bpo-38069: Convert _posixsubprocess to PEP-384 (GH-15780)
Summary:
Eliminate uses of `_Py_IDENTIFIER` from `_posixsubprocess`, replacing them with interned strings.

Also tries to find an existing version of the module, which will allow subinterpreters.



https://bugs.python.org/issue38069
2019-09-10 04:01:20 -07:00
Christian Heimes 98d90f745d
bpo-37951: Lift subprocess's fork() restriction (GH-15544) 2019-08-27 23:36:56 +02:00
Jeroen Demeyer 762f93ff2e bpo-37337: Add _PyObject_CallMethodNoArgs() (GH-14267) 2019-07-08 17:19:25 +09:00
Jakub Kulík 6f9bc72c79 bpo-35550: Fix incorrect Solaris define guards (GH-11275)
Python source code uses on several places ifdef sun or defined(sun) without the underscores, which is not standard compliant and shouldn't be used.

Defines should check for __sun instead. Reference: http://nadeausoftware.com/articles/2012/01/c_c_tip_how_use_compiler_predefined_macros_detect_operating_system#Solaris

https://bugs.python.org/issue35550
2018-12-30 18:16:40 -08:00
Gregory P. Smith 3015fb8ce4
bpo-35214: Add _Py_ prefix to MEMORY_SANITIZER def. (GH-10503)
Rename our new MEMORY_SANITIZER define to _Py_MEMORY_SANITIZER.
Project based C Preprocessor namespacing at its finest. :P
2018-11-12 22:01:22 -08:00
Gregory P. Smith 1584a00815
bpo-35214: Initial clang MemorySanitizer support (GH-10479)
Adds configure flags for msan and ubsan builds to make it easier to enable.
These also encode the detail that address sanitizer and memory sanitizer
should disable pymalloc.

Define MEMORY_SANITIZER when appropriate at build time and adds workarounds
to existing code to mark things as initialized where the sanitizer is otherwise unable to
determine that.  This lets our build succeed under the memory sanitizer.  not all tests
pass without sanitizer failures yet but we're in pretty good shape after this.
2018-11-12 12:07:14 -08:00
Eric Snow 5903296045
bpo-34651: Only allow the main interpreter to fork. (gh-9279)
When os.fork() is called (on platforms that support it) all threads but the current one are destroyed in the child process. Consequently we must ensure that all but the associated interpreter are likewise destroyed. The main interpreter is critical for runtime operation, so we must ensure that fork only happens in the main interpreter.

https://bugs.python.org/issue34651
2018-09-14 14:17:20 -07:00
Gregory P. Smith a20b6adb5a bpo-34658: Fix rare subprocess prexec_fn fork error. (GH-9255)
[bpo-34658](https://www.bugs.python.org/issue34658): Fix a rare interpreter unhandled exception state SystemError only
seen when using subprocess with a preexec_fn while an after_parent handler has
been registered with os.register_at_fork and the fork system call fails.



https://bugs.python.org/issue34658
2018-09-13 04:30:10 -07:00
Gregory P. Smith ce34410b8b
bpo-32270: Don't close stdin/out/err in pass_fds (GH-6242)
When subprocess.Popen() stdin= stdout= or stderr= handles are specified
and appear in pass_fds=, don't close the original fds after dup'ing them.

This implementation and unittest primarily came from @izbyshev (see the PR)

See also b89b52f284

This also removes the old manual p2cread, c2pwrite, and errwrite closing logic
as inheritable flags and _close_open_fds takes care of that properly today without special treatment.

This code is within child_exec() where it is the only thread so there is no
race condition between the dup and _Py_set_inheritable_async_safe call.
2018-09-10 17:46:22 -07:00
Alexey Izbyshev 0e7144b064 bpo-32844: Fix a subprocess misredirection of a low fd (GH5689)
bpo-32844: subprocess: Fix a potential misredirection of a low fd to stderr.

When redirecting, subprocess attempts to achieve the following state:
each fd to be redirected to is less than or equal to the fd
it is redirected from, which is necessary because redirection
occurs in the ascending order of destination descriptors.
It fails to do so in a couple of corner cases,
for example, if 1 is redirected to 2 and 0 is closed in the parent.
2018-03-26 12:49:35 -07:00
Alexey Izbyshev c1e46e94de bpo-32777: Fix _Py_set_inheritable async-safety in subprocess (GH-5560)
Fix a rare but potential pre-exec child process deadlock in subprocess on POSIX systems when marking file descriptors inheritable on exec in the child process.  This bug appears to have been introduced in 3.4 with the inheritable file descriptors support.

This also changes Python/fileutils.c `set_inheritable` to use the "slow" two `fcntl` syscall path instead of the "fast" single `ioctl` syscall path when asked to be async signal safe (by way of being asked not to raise exceptions).  `ioctl` is not a POSIX async-signal-safe approved function.

ref: http://pubs.opengroup.org/onlinepubs/9699919799/functions/V2_chap02.html
2018-02-05 22:09:34 -08:00
luzpaz a5293b4ff2 Fix miscellaneous typos (#4275) 2017-11-05 15:37:50 +02:00
Gregory P. Smith 163468a766 bpo-16500: Don't use string constants for os.register_at_fork() behavior (#1834)
Instead use keyword only arguments to os.register_at_fork for each of the scenarios.
Updates the documentation for clarity.
2017-05-29 10:03:41 -07:00
Antoine Pitrou 346cbd351e bpo-16500: Allow registering at-fork handlers (#1715)
* bpo-16500: Allow registering at-fork handlers

* Address Serhiy's comments

* Add doc for new C API

* Add doc for new Python-facing function

* Add NEWS entry + doc nit
2017-05-27 17:50:54 +02:00
Serhiy Storchaka 66bffd1663 bpo-30065: Fixed arguments validation in _posixsubprocess.fork_exec(). (#1110) 2017-04-19 21:12:46 +03:00
Serhiy Storchaka 9574e0adf2 Issue #20947: Fixed a gcc warning with -Wstrict-overflow. 2016-09-27 22:04:45 +03:00
Serhiy Storchaka 5ae4f49f4a Issue #20947: Fixed a gcc warning with -Wstrict-overflow. 2016-09-27 22:03:51 +03:00
Xavier de Gaye c716f183f8 Issue #26862: SYS_getdents64 does not need to be defined on android API 21. 2016-06-15 11:35:29 +02:00
Serhiy Storchaka ef1585eb9a Issue #25923: Added more const qualifiers to signatures of static and private functions. 2015-12-25 20:01:53 +02:00
Martin Panter 6a77c2d978 Issue #25764: Merge subprocess fix from 3.4 into 3.5 2015-12-05 02:03:42 +00:00
Martin Panter afdd51343c Issue #25764: Preserve subprocess fork exception when preexec_fn used
Also fix handling of failure to release the import lock.
2015-11-30 02:21:41 +00:00
Gregory P. Smith d0a5b1c343 Fixes #23564: Fix a partially broken sanity check in the _posixsubprocess
internals regarding how fds_to_pass were passed to the child.  The bug
had no actual impact as subprocess.py already avoided it.
2015-11-15 21:15:26 -08:00
Victor Stinner bc5b80bac1 Close #24784: Fix compilation without thread support
Add "#ifdef WITH_THREAD" around cals to:

* PyGILState_Check()
* _PyImport_AcquireLock()
* _PyImport_ReleaseLock()
2015-10-11 09:54:42 +02:00
Gregory P. Smith 9afe8a3306 Fix computation of max_fd on OpenBSD. Issue #23852. 2015-04-25 23:44:02 -07:00
Gregory P. Smith f9681776c9 Fix computation of max_fd on OpenBSD. Issue #23852. 2015-04-25 23:43:34 -07:00
Gregory P. Smith 4dff6f6fa6 Issue9951: update _hashopenssl and md5module to use _Py_strhex().
Also update _posixsubprocess to use Py_hexdigits instead of its own constant.
2015-04-25 23:42:38 +00:00
Victor Stinner 33f96f1800 Merge 3.4 (_posixsubprocess) 2015-04-02 16:25:01 +02:00
Victor Stinner e7c749238e Issue #23851: close() must not be retried when it fails with EINTR
See the PEP 475 for the rationale.
2015-04-02 16:24:46 +02:00
Victor Stinner 185fd33a17 Issue #23836: Use _Py_write_noraise() to retry on EINTR in child_exec() of
_posixsubprocess
2015-04-01 18:35:53 +02:00
Victor Stinner 160e819a1d Issue #23694: Fix usage of _Py_open() in the _posixsubprocess module
Don't call _Py_open() from _close_open_fds_safe() because it is call just after
fork(). It's not good to play with locks (the GIL) between fork() and exec().

Use instead _Py_open_noraise() which doesn't touch to the GIL.
2015-03-30 02:18:31 +02:00
Victor Stinner a555cfcb73 Issue #23694: Enhance _Py_open(), it now raises exceptions
* _Py_open() now raises exceptions on error. If open() fails, it raises an
  OSError with the filename.
* _Py_open() now releases the GIL while calling open()
* Add _Py_open_noraise() when _Py_open() cannot be used because the GIL is not
  held
2015-03-18 00:22:14 +01:00
Victor Stinner 8f437aac06 Issue #22290: Fix error handling in the _posixsubprocess module.
* Don't call the garbage collector with an exception set: it causes an
  assertion to fail in debug mode.
* Enhance also error handling if allocating an array for the executable list
  failed.
* Add an unit test for 4 different errors in the _posixsubprocess module.
2014-10-05 17:25:19 +02:00