Commit Graph

376 Commits

Author SHA1 Message Date
Cody Maloney a9344cdffa
gh-121381 Remove subprocess._USE_VFORK escape hatch (#121383)
This flag was added as an escape hatch in gh-91401 and backported to
Python 3.10. The flag broke at some point between its addition and now.
As there is currently no publicly known environments that require this,
remove it rather than work on fixing it.

This leaves the flag in the subprocess module to not break code which
may have used / checked the flag itself.

discussion: https://discuss.python.org/t/subprocess-use-vfork-escape-hatch-broken-fix-or-remove/56915/2
2024-07-30 18:39:54 -07:00
Victor Stinner 05df063ad8
gh-120417: Fix "imported but unused" linter warnings (#120461)
Add __all__ to the following modules:
importlib.machinery, importlib.util and xml.sax.

Add also "# noqa: F401" in collections.abc,
subprocess and xml.sax.

* Sort __all__; remove collections.abc.__all__; remove private names

* Add tests
2024-06-14 20:39:50 +02:00
Savannah Ostrowski 5b941e57c7
GH-118844: Fix build failures when combining --disable-gil with --enable-experimental-jit (GH-118935) 2024-05-11 16:19:31 -07:00
Henrik Tunedal 133c1a7cdb
gh-118293: Suppress mouse cursor feedback when launching Windows processes with multiprocessing (GH-118315) 2024-04-28 21:10:44 +00:00
Paulo Neves 4abca7e1e7
gh-98966: Handle stdout=subprocess.STDOUT (GH-98967)
Explicitly handle the case where stdout=STDOUT
as otherwise the existing error handling gets
confused and reports hard to understand errors.

Signed-off-by: Paulo Neves <ptsneves@gmail.com>
2024-03-26 13:37:50 +01:00
Victor Stinner 27cf3ed00c
gh-90872: Fix subprocess.Popen.wait() for negative timeout (#116989)
On Windows, subprocess.Popen.wait() no longer calls
WaitForSingleObject() with a negative timeout: pass 0 ms if the
timeout is negative.
2024-03-19 14:42:44 +01:00
Brett Simmers 2731913dd5
gh-116167: Allow disabling the GIL with `PYTHON_GIL=0` or `-X gil=0` (#116338)
In free-threaded builds, running with `PYTHON_GIL=0` will now disable the
GIL. Follow-up issues track work to re-enable the GIL when loading an
incompatible extension, and to disable the GIL by default.

In order to support re-enabling the GIL at runtime, all GIL-related data
structures are initialized as usual, and disabling the GIL simply sets a flag
that causes `take_gil()` and `drop_gil()` to return early.
2024-03-11 11:02:58 -04:00
Serhiy Storchaka e2c097ebde
gh-104522: Fix OSError raised when run a subprocess (#114195)
Only set filename to cwd if it was caused by failed chdir(cwd).

_fork_exec() now returns "noexec:chdir" for failed chdir(cwd).

Co-authored-by: Robert O'Shea <PurityLake@users.noreply.github.com>
2024-01-17 16:52:42 -08:00
Jakub Kulík 2b93f52242
gh-113117: Support posix_spawn in subprocess.Popen with close_fds=True (#113118)
Add support for `os.POSIX_SPAWN_CLOSEFROM` and
`posix_spawn_file_actions_addclosefrom_np` and have the `subprocess` module use
them when available.  This means `posix_spawn` can now be used in the default
`close_fds=True` situation on many platforms.

Co-authored-by: Gregory P. Smith [Google LLC] <greg@krypto.org>
2023-12-17 21:34:57 +00:00
Jakub Kulík 48c907a15c
gh-113119 fix environment handling in subprocess.Popen when posix_spawn is used (#113120)
* Allow posix_spawn to inherit environment form parent environ variable.

With this change, posix_spawn call can behave similarly to execv with regards to environments when used in subprocess functions.
2023-12-17 05:19:05 +00:00
Russell Keith-Magee dc824c5dc1
gh-112736: Refactor del-safe symbol handling in subprocess (#112738)
Refactor delete-safe symbol handling in subprocess.

Only module globals are force-cleared during interpreter finalization, using a class reference instead of individually listing the constants everywhere is simpler.
2023-12-04 20:23:17 -08:00
Felipe A. Hernandez 3dcac78581
gh-106723: forward -Xfrozen_modules option to spawned process interpreters (#106724)
Co-authored-by: Kumar Aditya <kumaraditya@python.org>
Co-authored-by: Gregory P. Smith <greg@krypto.org>
2023-07-28 05:04:26 +00:00
cptpcrd 3a4c44bb1e
gh-87474: Fix file descriptor leaks in subprocess.Popen (#96351)
This fixes several ways file descriptors could be leaked from `subprocess.Popen` constructor during error conditions by opening them later and using a context manager "fds to close" registration scheme to ensure they get closed before returning.

---------

Co-authored-by: Gregory P. Smith [Google] <greg@krypto.org>
2023-05-16 20:23:53 +00:00
Steve Dower 20cf32e761
gh-101283: Fix use of unbound variable (GH-101712) 2023-02-08 23:38:56 +00:00
Oleg Iarygin 23751ed826
gh-101283: Improved fallback logic for subprocess with shell=True on Windows (GH-101286) 2023-02-08 22:12:19 +00:00
andrei kulakov 80b3e32d62
Add note on capture_output arg to subprocess.run() docstring (#98012)
add note on capture_output arg to the docstring
2022-10-07 10:08:08 -07:00
andrei kulakov db64fb9bbe
gh-97825: fix AttributeError when calling subprocess.check_output(input=None) with encoding or errors args (#97826)
* fix AttributeError, add unit test
2022-10-04 17:47:49 -07:00
Christian Heimes 8184f0fce3
gh-95174: Handle missing waitpid and gethostbyname in WASI (GH-95181) 2022-07-24 08:04:06 +02:00
Christian Heimes db0b455ff4
gh-90473: Fail subprocess early on Emscripten/WASI (GH-92802) 2022-05-14 20:23:19 +02:00
Victor Stinner ada8b6d1b1
gh-57684: Add -P cmdline option and PYTHONSAFEPATH env var (#31542)
Add the -P command line option and the PYTHONSAFEPATH environment
variable to not prepend a potentially unsafe path to sys.path.

* Add sys.flags.safe_path flag.
* Add PyConfig.safe_path member.
* Programs/_bootstrap_python.c uses config.safe_path=0.
* Update subprocess._optim_args_from_interpreter_flags() to handle
  the -P command line option.
* Modules/getpath.py sets safe_path to 1 if a "._pth" file is
  present.
2022-05-06 01:34:11 +02:00
Gregory P. Smith f6dd14c653
gh-82616: Add process_group support to subprocess.Popen (#23930)
One more thing that can help prevent people from using `preexec_fn`.

Also adds conditional skips to two tests exposing ASAN flakiness on the Ubuntu 20.04 Address Sanitizer Github CI system. When that build is run on more modern systems the "problem" does not show up. It seems ASAN implementation related.

Co-authored-by: Zackery Spytz <zspytz@gmail.com>
Co-authored-by: Serhiy Storchaka <storchaka@gmail.com>
2022-05-05 16:22:32 -07:00
Inada Naoki 354ace8b07
gh-91954: Emit EncodingWarning from locale and subprocess (GH-91977)
locale.getpreferredencoding() and subprocess.Popen() emit EncodingWarning
2022-04-30 15:53:29 +09:00
Gregory P. Smith cd5726fe67
gh-91401: Add a failsafe way to disable vfork. (#91490)
Just in case there is ever an issue with _posixsubprocess's use of
vfork() due to the complexity of using it properly and potential
directions that Linux platforms where it defaults to on could take, this
adds a failsafe so that users can disable its use entirely by setting
a global flag.

No known reason to disable it exists. But it'd be a shame to encounter
one and not be able to use CPython without patching and rebuilding it.

See the linked issue for some discussion on reasoning.

Also documents the existing way to disable posix_spawn.
2022-04-25 16:19:39 -07:00
Christian Heimes 082d3495d0
bpo-40280: Emscripten fork_exec now fails early (GH-32224) 2022-04-01 21:20:56 +02:00
Christian Heimes 606e496dd6
bpo-40280: Use presence of msvcrt module to detect Windows (GH-30930) 2022-01-27 01:57:43 -08:00
Christian Heimes 7f4b69b907
bpo-40280: Change subprocess imports for cleaner error on wasm32 (GH-30620) 2022-01-17 07:23:36 +01:00
Rémi Lapeyre 4d2957c1b9
bpo-40497: Fix handling of check in subprocess.check_output() (GH-19897)
Co-authored-by: Tal Einat <taleinat@gmail.com>
Co-authored-by: Łukasz Langa <lukasz@langa.pl>
2021-09-20 17:09:05 +02:00
Jakub Kulík b1930bf75f
bpo-44935: enable posix_spawn() on Solaris (GH-27795)
Enable posix_spawn() on Solaris
2021-08-17 11:09:48 -07:00
Binbin 17b16e13bb
Fix typos in multiple files (GH-26689)
Co-authored-by: Terry Jan Reedy <tjreedy@udel.edu>
2021-06-12 22:47:44 -04:00
M. Kocher db0c5b786d
bpo-43776: Remove list call from args in Popen repr (GH-25338)
Removes the `list` call in the Popen `repr`.

Current implementation:

For cmd = `python --version`,  with `shell=True`.

```bash
<Popen: returncode: None args: ['p', 'y', 't', 'h', 'o', 'n', ' ', '-', '-',...>
```

For `shell=False` and args=`['python', '--version']`, the output is correct:

```bash
<Popen: returncode: None args: ['python', '--version']>
```

With the new changes the `repr`  yields:

For cmd = `python --version`,  with `shell=True`:

```bash
<Popen: returncode: None args: 'python --version'>
```

For `shell=False` and args=`['python', '--version']`, the output:

```bash
<Popen: returncode: None args: ['python', '--version']>
```

Automerge-Triggered-By: GH:gpshead
2021-04-28 01:16:38 -07:00
Inada Naoki 4827483f47
bpo-43510: Implement PEP 597 opt-in EncodingWarning. (GH-19481)
See [PEP 597](https://www.python.org/dev/peps/pep-0597/).

* Add `-X warn_default_encoding` and `PYTHONWARNDEFAULTENCODING`.
* Add EncodingWarning
* Add io.text_encoding()
* open(), TextIOWrapper() emits EncodingWarning when encoding is omitted and warn_default_encoding is enabled.
* _pyio.TextIOWrapper() uses UTF-8 as fallback default encoding used when failed to import locale module. (used during building Python)
* bz2, configparser, gzip, lzma, pathlib, tempfile modules use io.text_encoding().
* What's new entry
2021-03-29 12:28:14 +09:00
Victor Stinner d72e8d4875
bpo-41718: subprocess imports grp and pwd on demand (GH-24987)
The shutil and subprocess modules now only import the grp and pwd
modules when they are needed, which is not the case by default in
subprocess.
2021-03-23 17:42:51 +01:00
Chris Griffith b4fc44bb2d
bpo-43423 Fix IndexError in subprocess _communicate function (GH-24777)
Check to make sure stdout and stderr are not empty before selecting an item from them in Windows subprocess._communicate.

Co-authored-by: Gregory P. Smith <greg@krypto.org>
2021-03-11 11:43:29 -08:00
Kamil Turek b225d91f0a
bpo-43391: Remove the broken Python 2.4 link from the comment (GH-24736) 2021-03-04 10:24:42 -08:00
Gregory P. Smith 64abf37344
bpo-42388: Fix subprocess.check_output input=None when text=True (GH-23467)
When the modern text= spelling of the universal_newlines= parameter was added
for Python 3.7, check_output's special case around input=None was overlooked.
So it behaved differently with universal_newlines=True vs text=True.  This
reconciles the behavior to be consistent and adds a test to guarantee it.

Also clarifies the existing check_output documentation.

Co-authored-by: Alexey Izbyshev <izbyshev@ispras.ru>
2020-12-24 20:57:21 -08:00
Filipe Laíns 01a202ab6b
bpo-40550: Fix time-of-check/time-of-action issue in subprocess.Popen.send_signal. (GH-20010)
send_signal() now swallows the exception if the process it thought was still alive winds up not to exist anymore (always a plausible race condition despite the checks).

Co-authored-by: Gregory P. Smith <greg@krypto.org>
2020-11-21 01:22:08 -08:00
Ruben Vorderman 23c0fb8edd
bpo-41586: Add pipesize parameter to subprocess & F_GETPIPE_SZ and F_SETPIPE_SZ to fcntl. (GH-21921)
* Add F_SETPIPE_SZ and F_GETPIPE_SZ to fcntl module
* Add pipesize parameter for subprocess.Popen class

This will allow the user to control the size of the pipes.
On linux the default is 64K. When a pipe is full it blocks for writing.
When a pipe is empty it blocks for reading. On processes that are
very fast this can lead to a lot of wasted CPU cycles. On a typical
Linux system the max pipe size is 1024K which is much better.
For high performance-oriented libraries such as xopen it is nice to
be able to set the pipe size.

The workaround without this feature is to use my_popen_process.stdout.fileno() in
conjuction with fcntl and 1031 (value of F_SETPIPE_SZ) to acquire this behavior.
2020-10-19 16:30:02 -07:00
Pablo Galindo 1ed83adb0e
bpo-40939: Remove the old parser (GH-20768)
This commit removes the old parser, the deprecated parser module, the old parser compatibility flags and environment variables and all associated support code and documentation.
2020-06-11 17:30:46 +01:00
Victor Stinner 1def7754b7
bpo-40334: Rename PyConfig.use_peg to _use_peg_parser (GH-19670)
* Rename PyConfig.use_peg to _use_peg_parser
* Document PyConfig._use_peg_parser and mark it a deprecated
* Mark -X oldparser option and PYTHONOLDPARSER env var as deprecated
  in the documentation.
* Add use_old_parser() and skip_if_new_parser() to test.support
* Remove sys.flags.use_peg: use_old_parser() uses
  _testinternalcapi.get_configs() instead.
* Enhance test_embed tests
* subprocess._args_from_interpreter_flags() copies -X oldparser
2020-04-23 03:03:24 +02:00
Guido van Rossum 48b069a003
bpo-39481: Implementation for PEP 585 (#18239)
This implements things like `list[int]`,
which returns an object of type `types.GenericAlias`.
This object mostly acts as a proxy for `list`,
but has attributes `__origin__` and `__args__`
that allow recovering the parts (with values `list` and `(int,)`.

There is also an approximate notion of type variables;
e.g. `list[T]` has a `__parameters__` attribute equal to `(T,)`.
Type variables are objects of type `typing.TypeVar`.
2020-04-07 09:50:06 -07:00
Victor Stinner 65a796e527
bpo-40094: Add os.waitstatus_to_exitcode() (GH-19201)
Add os.waitstatus_to_exitcode() function to convert a wait status to an
exitcode.

Suggest waitstatus_to_exitcode() usage in the documentation when
appropriate.

Use waitstatus_to_exitcode() in:

* multiprocessing, os, subprocess and _bootsubprocess modules;
* test.support.wait_process();
* setup.py: run_command();
* and many tests.
2020-04-01 18:49:29 +02:00
Victor Stinner c6e5c1123b
bpo-39489: Remove COUNT_ALLOCS special build (GH-18259)
Remove:

* COUNT_ALLOCS macro
* sys.getcounts() function
* SHOW_ALLOC_COUNT code in listobject.c
* SHOW_TRACK_COUNT code in tupleobject.c
* PyConfig.show_alloc_count field
* -X showalloccount command line option
* @test.support.requires_type_collecting decorator
2020-02-03 15:17:15 +01:00
Alex Rebert d3ae95e1e9 bpo-35182: fix communicate() crash after child closes its pipes (GH-17020) (GH-18117)
When communicate() is called in a loop, it crashes when the child process
has already closed any piped standard stream, but still continues to be running

Co-authored-by: Andriy Maletsky <andriy.maletsky@gmail.com>
2020-01-22 15:28:31 -08:00
Victor Stinner e85a305503
bpo-38630: Fix subprocess.Popen.send_signal() race condition (GH-16984)
On Unix, subprocess.Popen.send_signal() now polls the process status.
Polling reduces the risk of sending a signal to the wrong process if
the process completed, the Popen.returncode attribute is still None,
and the pid has been reassigned (recycled) to a new different
process.
2020-01-15 17:38:55 +01:00
Batuhan Taşkaya 4dc5a9df59 bpo-39019: Implement missing __class_getitem__ for subprocess classes (GH-17558) 2019-12-30 16:02:04 +00:00
Andrey Doroschenko 645005e947 bpo-38724: Implement subprocess.Popen.__repr__ (GH-17151) 2019-11-17 16:08:31 +02:00
Gregory P. Smith f3751efb5c
bpo-38417: Add umask support to subprocess (GH-16726)
On POSIX systems, allow the umask to be set in the child process before we exec.
2019-10-12 13:24:56 -07:00
Victor Stinner faca855342 bpo-36046: posix_spawn() doesn't support uid/gid (GH-16384)
* subprocess.Popen now longer uses posix_spawn() if uid, gid or gids are set.
* test_subprocess: add "nobody" and "nfsnobody" group names for test_group().
* test_subprocess: test_user() and test_group() are now also tested with close_fds=False.
2019-09-25 15:52:49 +02:00
Patrick McLean 2b2ead7438 bpo-36046: Add user and group parameters to subprocess (GH-11950)
* subprocess: Add user, group and extra_groups paremeters to subprocess.Popen

This adds a `user` parameter to the Popen constructor that will call
setreuid() in the child before calling exec(). This allows processes
running as root to safely drop privileges before running the subprocess
without having to use a preexec_fn.

This also adds a `group` parameter that will call setregid() in
the child process before calling exec().

Finally an `extra_groups` parameter was added that will call
setgroups() to set the supplimental groups.
2019-09-12 18:15:44 +01:00
Gregory P. Smith 580d2782f7 bpo-37424: Avoid a hang in subprocess.run timeout output capture (GH-14490)
Fixes a possible hang when using a timeout on subprocess.run() while
capturing output. If the child process spawned its own children or otherwise
connected its stdout or stderr handles with another process, we could hang
after the timeout was reached and our child was killed when attempting to read
final output from the pipes.
2019-09-11 02:23:05 -07:00