Commit Graph

364 Commits

Author SHA1 Message Date
cptpcrd 3a4c44bb1e
gh-87474: Fix file descriptor leaks in subprocess.Popen (#96351)
This fixes several ways file descriptors could be leaked from `subprocess.Popen` constructor during error conditions by opening them later and using a context manager "fds to close" registration scheme to ensure they get closed before returning.

---------

Co-authored-by: Gregory P. Smith [Google] <greg@krypto.org>
2023-05-16 20:23:53 +00:00
Steve Dower 20cf32e761
gh-101283: Fix use of unbound variable (GH-101712) 2023-02-08 23:38:56 +00:00
Oleg Iarygin 23751ed826
gh-101283: Improved fallback logic for subprocess with shell=True on Windows (GH-101286) 2023-02-08 22:12:19 +00:00
andrei kulakov 80b3e32d62
Add note on capture_output arg to subprocess.run() docstring (#98012)
add note on capture_output arg to the docstring
2022-10-07 10:08:08 -07:00
andrei kulakov db64fb9bbe
gh-97825: fix AttributeError when calling subprocess.check_output(input=None) with encoding or errors args (#97826)
* fix AttributeError, add unit test
2022-10-04 17:47:49 -07:00
Christian Heimes 8184f0fce3
gh-95174: Handle missing waitpid and gethostbyname in WASI (GH-95181) 2022-07-24 08:04:06 +02:00
Christian Heimes db0b455ff4
gh-90473: Fail subprocess early on Emscripten/WASI (GH-92802) 2022-05-14 20:23:19 +02:00
Victor Stinner ada8b6d1b1
gh-57684: Add -P cmdline option and PYTHONSAFEPATH env var (#31542)
Add the -P command line option and the PYTHONSAFEPATH environment
variable to not prepend a potentially unsafe path to sys.path.

* Add sys.flags.safe_path flag.
* Add PyConfig.safe_path member.
* Programs/_bootstrap_python.c uses config.safe_path=0.
* Update subprocess._optim_args_from_interpreter_flags() to handle
  the -P command line option.
* Modules/getpath.py sets safe_path to 1 if a "._pth" file is
  present.
2022-05-06 01:34:11 +02:00
Gregory P. Smith f6dd14c653
gh-82616: Add process_group support to subprocess.Popen (#23930)
One more thing that can help prevent people from using `preexec_fn`.

Also adds conditional skips to two tests exposing ASAN flakiness on the Ubuntu 20.04 Address Sanitizer Github CI system. When that build is run on more modern systems the "problem" does not show up. It seems ASAN implementation related.

Co-authored-by: Zackery Spytz <zspytz@gmail.com>
Co-authored-by: Serhiy Storchaka <storchaka@gmail.com>
2022-05-05 16:22:32 -07:00
Inada Naoki 354ace8b07
gh-91954: Emit EncodingWarning from locale and subprocess (GH-91977)
locale.getpreferredencoding() and subprocess.Popen() emit EncodingWarning
2022-04-30 15:53:29 +09:00
Gregory P. Smith cd5726fe67
gh-91401: Add a failsafe way to disable vfork. (#91490)
Just in case there is ever an issue with _posixsubprocess's use of
vfork() due to the complexity of using it properly and potential
directions that Linux platforms where it defaults to on could take, this
adds a failsafe so that users can disable its use entirely by setting
a global flag.

No known reason to disable it exists. But it'd be a shame to encounter
one and not be able to use CPython without patching and rebuilding it.

See the linked issue for some discussion on reasoning.

Also documents the existing way to disable posix_spawn.
2022-04-25 16:19:39 -07:00
Christian Heimes 082d3495d0
bpo-40280: Emscripten fork_exec now fails early (GH-32224) 2022-04-01 21:20:56 +02:00
Christian Heimes 606e496dd6
bpo-40280: Use presence of msvcrt module to detect Windows (GH-30930) 2022-01-27 01:57:43 -08:00
Christian Heimes 7f4b69b907
bpo-40280: Change subprocess imports for cleaner error on wasm32 (GH-30620) 2022-01-17 07:23:36 +01:00
Rémi Lapeyre 4d2957c1b9
bpo-40497: Fix handling of check in subprocess.check_output() (GH-19897)
Co-authored-by: Tal Einat <taleinat@gmail.com>
Co-authored-by: Łukasz Langa <lukasz@langa.pl>
2021-09-20 17:09:05 +02:00
Jakub Kulík b1930bf75f
bpo-44935: enable posix_spawn() on Solaris (GH-27795)
Enable posix_spawn() on Solaris
2021-08-17 11:09:48 -07:00
Binbin 17b16e13bb
Fix typos in multiple files (GH-26689)
Co-authored-by: Terry Jan Reedy <tjreedy@udel.edu>
2021-06-12 22:47:44 -04:00
M. Kocher db0c5b786d
bpo-43776: Remove list call from args in Popen repr (GH-25338)
Removes the `list` call in the Popen `repr`.

Current implementation:

For cmd = `python --version`,  with `shell=True`.

```bash
<Popen: returncode: None args: ['p', 'y', 't', 'h', 'o', 'n', ' ', '-', '-',...>
```

For `shell=False` and args=`['python', '--version']`, the output is correct:

```bash
<Popen: returncode: None args: ['python', '--version']>
```

With the new changes the `repr`  yields:

For cmd = `python --version`,  with `shell=True`:

```bash
<Popen: returncode: None args: 'python --version'>
```

For `shell=False` and args=`['python', '--version']`, the output:

```bash
<Popen: returncode: None args: ['python', '--version']>
```

Automerge-Triggered-By: GH:gpshead
2021-04-28 01:16:38 -07:00
Inada Naoki 4827483f47
bpo-43510: Implement PEP 597 opt-in EncodingWarning. (GH-19481)
See [PEP 597](https://www.python.org/dev/peps/pep-0597/).

* Add `-X warn_default_encoding` and `PYTHONWARNDEFAULTENCODING`.
* Add EncodingWarning
* Add io.text_encoding()
* open(), TextIOWrapper() emits EncodingWarning when encoding is omitted and warn_default_encoding is enabled.
* _pyio.TextIOWrapper() uses UTF-8 as fallback default encoding used when failed to import locale module. (used during building Python)
* bz2, configparser, gzip, lzma, pathlib, tempfile modules use io.text_encoding().
* What's new entry
2021-03-29 12:28:14 +09:00
Victor Stinner d72e8d4875
bpo-41718: subprocess imports grp and pwd on demand (GH-24987)
The shutil and subprocess modules now only import the grp and pwd
modules when they are needed, which is not the case by default in
subprocess.
2021-03-23 17:42:51 +01:00
Chris Griffith b4fc44bb2d
bpo-43423 Fix IndexError in subprocess _communicate function (GH-24777)
Check to make sure stdout and stderr are not empty before selecting an item from them in Windows subprocess._communicate.

Co-authored-by: Gregory P. Smith <greg@krypto.org>
2021-03-11 11:43:29 -08:00
Kamil Turek b225d91f0a
bpo-43391: Remove the broken Python 2.4 link from the comment (GH-24736) 2021-03-04 10:24:42 -08:00
Gregory P. Smith 64abf37344
bpo-42388: Fix subprocess.check_output input=None when text=True (GH-23467)
When the modern text= spelling of the universal_newlines= parameter was added
for Python 3.7, check_output's special case around input=None was overlooked.
So it behaved differently with universal_newlines=True vs text=True.  This
reconciles the behavior to be consistent and adds a test to guarantee it.

Also clarifies the existing check_output documentation.

Co-authored-by: Alexey Izbyshev <izbyshev@ispras.ru>
2020-12-24 20:57:21 -08:00
Filipe Laíns 01a202ab6b
bpo-40550: Fix time-of-check/time-of-action issue in subprocess.Popen.send_signal. (GH-20010)
send_signal() now swallows the exception if the process it thought was still alive winds up not to exist anymore (always a plausible race condition despite the checks).

Co-authored-by: Gregory P. Smith <greg@krypto.org>
2020-11-21 01:22:08 -08:00
Ruben Vorderman 23c0fb8edd
bpo-41586: Add pipesize parameter to subprocess & F_GETPIPE_SZ and F_SETPIPE_SZ to fcntl. (GH-21921)
* Add F_SETPIPE_SZ and F_GETPIPE_SZ to fcntl module
* Add pipesize parameter for subprocess.Popen class

This will allow the user to control the size of the pipes.
On linux the default is 64K. When a pipe is full it blocks for writing.
When a pipe is empty it blocks for reading. On processes that are
very fast this can lead to a lot of wasted CPU cycles. On a typical
Linux system the max pipe size is 1024K which is much better.
For high performance-oriented libraries such as xopen it is nice to
be able to set the pipe size.

The workaround without this feature is to use my_popen_process.stdout.fileno() in
conjuction with fcntl and 1031 (value of F_SETPIPE_SZ) to acquire this behavior.
2020-10-19 16:30:02 -07:00
Pablo Galindo 1ed83adb0e
bpo-40939: Remove the old parser (GH-20768)
This commit removes the old parser, the deprecated parser module, the old parser compatibility flags and environment variables and all associated support code and documentation.
2020-06-11 17:30:46 +01:00
Victor Stinner 1def7754b7
bpo-40334: Rename PyConfig.use_peg to _use_peg_parser (GH-19670)
* Rename PyConfig.use_peg to _use_peg_parser
* Document PyConfig._use_peg_parser and mark it a deprecated
* Mark -X oldparser option and PYTHONOLDPARSER env var as deprecated
  in the documentation.
* Add use_old_parser() and skip_if_new_parser() to test.support
* Remove sys.flags.use_peg: use_old_parser() uses
  _testinternalcapi.get_configs() instead.
* Enhance test_embed tests
* subprocess._args_from_interpreter_flags() copies -X oldparser
2020-04-23 03:03:24 +02:00
Guido van Rossum 48b069a003
bpo-39481: Implementation for PEP 585 (#18239)
This implements things like `list[int]`,
which returns an object of type `types.GenericAlias`.
This object mostly acts as a proxy for `list`,
but has attributes `__origin__` and `__args__`
that allow recovering the parts (with values `list` and `(int,)`.

There is also an approximate notion of type variables;
e.g. `list[T]` has a `__parameters__` attribute equal to `(T,)`.
Type variables are objects of type `typing.TypeVar`.
2020-04-07 09:50:06 -07:00
Victor Stinner 65a796e527
bpo-40094: Add os.waitstatus_to_exitcode() (GH-19201)
Add os.waitstatus_to_exitcode() function to convert a wait status to an
exitcode.

Suggest waitstatus_to_exitcode() usage in the documentation when
appropriate.

Use waitstatus_to_exitcode() in:

* multiprocessing, os, subprocess and _bootsubprocess modules;
* test.support.wait_process();
* setup.py: run_command();
* and many tests.
2020-04-01 18:49:29 +02:00
Victor Stinner c6e5c1123b
bpo-39489: Remove COUNT_ALLOCS special build (GH-18259)
Remove:

* COUNT_ALLOCS macro
* sys.getcounts() function
* SHOW_ALLOC_COUNT code in listobject.c
* SHOW_TRACK_COUNT code in tupleobject.c
* PyConfig.show_alloc_count field
* -X showalloccount command line option
* @test.support.requires_type_collecting decorator
2020-02-03 15:17:15 +01:00
Alex Rebert d3ae95e1e9 bpo-35182: fix communicate() crash after child closes its pipes (GH-17020) (GH-18117)
When communicate() is called in a loop, it crashes when the child process
has already closed any piped standard stream, but still continues to be running

Co-authored-by: Andriy Maletsky <andriy.maletsky@gmail.com>
2020-01-22 15:28:31 -08:00
Victor Stinner e85a305503
bpo-38630: Fix subprocess.Popen.send_signal() race condition (GH-16984)
On Unix, subprocess.Popen.send_signal() now polls the process status.
Polling reduces the risk of sending a signal to the wrong process if
the process completed, the Popen.returncode attribute is still None,
and the pid has been reassigned (recycled) to a new different
process.
2020-01-15 17:38:55 +01:00
Batuhan Taşkaya 4dc5a9df59 bpo-39019: Implement missing __class_getitem__ for subprocess classes (GH-17558) 2019-12-30 16:02:04 +00:00
Andrey Doroschenko 645005e947 bpo-38724: Implement subprocess.Popen.__repr__ (GH-17151) 2019-11-17 16:08:31 +02:00
Gregory P. Smith f3751efb5c
bpo-38417: Add umask support to subprocess (GH-16726)
On POSIX systems, allow the umask to be set in the child process before we exec.
2019-10-12 13:24:56 -07:00
Victor Stinner faca855342 bpo-36046: posix_spawn() doesn't support uid/gid (GH-16384)
* subprocess.Popen now longer uses posix_spawn() if uid, gid or gids are set.
* test_subprocess: add "nobody" and "nfsnobody" group names for test_group().
* test_subprocess: test_user() and test_group() are now also tested with close_fds=False.
2019-09-25 15:52:49 +02:00
Patrick McLean 2b2ead7438 bpo-36046: Add user and group parameters to subprocess (GH-11950)
* subprocess: Add user, group and extra_groups paremeters to subprocess.Popen

This adds a `user` parameter to the Popen constructor that will call
setreuid() in the child before calling exec(). This allows processes
running as root to safely drop privileges before running the subprocess
without having to use a preexec_fn.

This also adds a `group` parameter that will call setregid() in
the child process before calling exec().

Finally an `extra_groups` parameter was added that will call
setgroups() to set the supplimental groups.
2019-09-12 18:15:44 +01:00
Gregory P. Smith 580d2782f7 bpo-37424: Avoid a hang in subprocess.run timeout output capture (GH-14490)
Fixes a possible hang when using a timeout on subprocess.run() while
capturing output. If the child process spawned its own children or otherwise
connected its stdout or stderr handles with another process, we could hang
after the timeout was reached and our child was killed when attempting to read
final output from the pipes.
2019-09-11 02:23:05 -07:00
Matthias 182e1d1f84 Fix subprocess docstring typo (GH-15812) 2019-09-10 14:51:09 +01:00
Ruslan Kuprieiev 042821ae3c bpo-37380: subprocess: don't use _active on win (GH-14360)
As noted by @eryksun in [1] and [2], using _cleanup and _active(in
__del__) is not necessary on Windows, since:

> Unlike Unix, a process in Windows doesn't have to be waited on by
> its parent to avoid a zombie. Keeping the handle open will actually
> create a zombie until the next _cleanup() call, which may be never
> if Popen() isn't called again.

This patch simply defines `subprocess._active` as `None`, for which we already
have the proper logic in place in `subprocess.Popen.__del__`, that prevents it
from trying to append the process to the `_active`. This patch also defines
`subprocess._cleanup` as a noop for Windows.

[1] https://bugs.python.org/issue37380#msg346333
[2] https://bugs.python.org/issue36067#msg336262

Signed-off-by: Ruslan Kuprieiev <ruslan@iterative.ai>
2019-06-28 18:12:16 +02:00
Steve Dower 60419a7e96
bpo-37363: Add audit events for a range of modules (GH-14301) 2019-06-24 08:42:54 -07:00
Rémi Lapeyre 8cc605acdd bpo-34886: Fix subprocess.run handling of exclusive arguments (GH-11727)
Fix an unintended ValueError from :func:`subprocess.run` when checking for
conflicting `input` and `stdin` or `capture_output` and `stdout` or `stderr` args
when they were explicitly provided but with `None` values within a passed in
`**kwargs` dict rather than as passed directly by name.
2019-06-08 07:56:24 -07:00
Serhiy Storchaka 9e3c452639
bpo-31961: Fix support of path-like executables in subprocess. (GH-5914) 2019-05-28 22:49:35 +03:00
Serhiy Storchaka 96aeaec647
bpo-36793: Remove unneeded __str__ definitions. (GH-13081)
Classes that define __str__ the same as __repr__ can
just inherit it from object.
2019-05-06 22:29:40 +03:00
Giampaolo Rodola bafa8487f7 subprocess: close pipes/fds by using ExitStack (GH-11686)
Close pipes/fds in subprocess by using ExitStack.

"In case of premature failure on X.Close() or os.close(X) the remaining pipes/fds will remain "open". Perhaps it makes sense to use contextlib.ExitStack."
- Rationale: https://github.com/python/cpython/pull/11575#discussion_r250288394
2019-01-29 13:14:24 -08:00
Gregory P. Smith 81d04bcf21
Fix docstr/comment typos in _use_posix_spawn(). (GH-11684) 2019-01-26 15:19:11 -08:00
Victor Stinner f6243ac1e4
bpo-35537: subprocess can use posix_spawn with pipes (GH-11575)
* subprocess.Popen can now also use os.posix_spawn() with pipes,
  but only if pipe file descriptors are greater than 2.
* Fix Popen._posix_spawn(): set '_child_created' attribute to True.
* Add Popen._close_pipe_fds() helper function to factorize the code.
2019-01-23 19:00:39 +01:00
Victor Stinner 8c349565e8
Revert "bpo-35537: subprocess can now use os.posix_spawnp (GH-11579)" (GH-11582)
This reverts commit 0785889468.
2019-01-16 23:38:06 +01:00
Victor Stinner 0785889468
bpo-35537: subprocess can now use os.posix_spawnp (GH-11579)
The subprocess module can now use the os.posix_spawnp() function,
if it is available, to locate the program in the PATH.
2019-01-16 15:26:20 +01:00
Victor Stinner 9daecf37a5
bpo-35537: subprocess uses os.posix_spawn in some cases (GH-11452)
The subprocess module can now use the os.posix_spawn() function
in some cases for better performance. Currently, it is only used on macOS
and Linux (using glibc 2.24 or newer) if all these conditions are met:

* executable path contains a directory
* close_fds=False
* preexec_fn, pass_fds, cwd, stdin, stdout, stderr
  and start_new_session parameters are not set

Co-authored-by: Joannah Nanjekye <nanjekyejoannah@gmail.com>
2019-01-16 00:02:35 +01:00