Commit Graph

408 Commits

Author SHA1 Message Date
Alexey Izbyshev a2670565d8 bpo-32236: open() emits RuntimeWarning if buffering=1 for binary mode (GH-4842)
If buffering=1 is specified for open() in binary mode, it is silently
treated as buffering=-1 (i.e., the default buffer size).
Coupled with the fact that line buffering is always supported in Python 2,
such behavior caused several issues (e.g., bpo-10344, bpo-21332).

Warn that line buffering is not supported if open() is called with
binary mode and buffering=1.
2018-10-20 02:22:31 +02:00
Zackery Spytz 73870bfeb9 closes bpo-32490: Fix filename duplication in subprocess exception message. (GH-9163)
8621bb5d93 sets the filename in directly in the FileNotFoundError, so we may revert the earlier fix 5f78040057.
2018-09-11 08:54:07 -07:00
Gregory P. Smith ce34410b8b
bpo-32270: Don't close stdin/out/err in pass_fds (GH-6242)
When subprocess.Popen() stdin= stdout= or stderr= handles are specified
and appear in pass_fds=, don't close the original fds after dup'ing them.

This implementation and unittest primarily came from @izbyshev (see the PR)

See also b89b52f284

This also removes the old manual p2cread, c2pwrite, and errwrite closing logic
as inheritable flags and _close_open_fds takes care of that properly today without special treatment.

This code is within child_exec() where it is the only thread so there is no
race condition between the dup and _Py_set_inheritable_async_safe call.
2018-09-10 17:46:22 -07:00
Michael Felt 89d79b1449 Remove AIX workaround test_subprocess (GH-8939)
An old apparent AIX behavior workaround in test_subprocess's
test_undecodable_env is no longer needed.
2018-08-26 10:29:36 -07:00
Victor Stinner 483422f57e
bpo-34044: subprocess.Popen copies startupinfo (GH-8090)
subprocess.Popen now copies the startupinfo argument to leave it
unchanged: it will modify the copy, so that the same STARTUPINFO
object can be used multiple times.

Add subprocess.STARTUPINFO.copy() method.
2018-07-05 22:54:17 +02:00
Victor Stinner 937ee9e745
Revert "bpo-33671: Add support.MS_WINDOWS and support.MACOS (GH-7800)" (GH-7919)
This reverts commit 8fbbdf0c31.
2018-06-26 02:11:06 +02:00
Victor Stinner 8fbbdf0c31
bpo-33671: Add support.MS_WINDOWS and support.MACOS (GH-7800)
* Add support.MS_WINDOWS: True if Python is running on Microsoft Windows.
* Add support.MACOS: True if Python is running on Apple macOS.
* Replace support.is_android with support.ANDROID
* Replace support.is_jython with support.JYTHON
* Cleanup code to initialize unix_shell
2018-06-22 19:25:44 +02:00
Gregory P. Smith 5f3d04fa4e
Improve the subprocess restore_signals=True test. (GH-7414)
It wasn't testing functionality.  Now it is (on Linux anyways).
2018-06-05 12:00:57 -07:00
Alexey Izbyshev 0e7144b064 bpo-32844: Fix a subprocess misredirection of a low fd (GH5689)
bpo-32844: subprocess: Fix a potential misredirection of a low fd to stderr.

When redirecting, subprocess attempts to achieve the following state:
each fd to be redirected to is less than or equal to the fd
it is redirected from, which is necessary because redirection
occurs in the ascending order of destination descriptors.
It fails to do so in a couple of corner cases,
for example, if 1 is redirected to 2 and 0 is closed in the parent.
2018-03-26 12:49:35 -07:00
Serhiy Storchaka b21d155f57
bpo-32964: Reuse a testing implementation of the path protocol in tests. (#5930) 2018-03-02 11:53:51 +02:00
Serhiy Storchaka be50a7b627 Revert "bpo-31961: subprocess now accepts path-like args (GH-4329)" (#5912)
* Revert "bpo-31961: subprocess now accepts path-like args (GH-4329)"

This reverts commit dd42cb71f2.
2018-02-27 18:03:46 -05:00
Zachary Ware 5537646bfa
bpo-30121: Fix test_subprocess for Windows Debug builds (GH-5758) 2018-02-19 14:02:38 -06:00
Anders Lorentsen dd42cb71f2 bpo-31961: subprocess now accepts path-like args (GH-4329)
Allow os.PathLike args in subprocess APIs.
2018-01-29 23:27:28 -08:00
Bo Bayles ce0f33d045 bpo-32102 Add "capture_output=True" to subprocess.run (GH-5149)
Add "capture_output=True" option to subprocess.run, this is equivalent to
setting stdout=PIPE, stderr=PIPE but is much more readable.
2018-01-29 22:40:39 -08:00
Gregory P. Smith f4d644f36f
bpo-25942: make subprocess more graceful on ^C (GH-5026)
Do not allow receiving a SIGINT to cause the subprocess module to trigger an
immediate SIGKILL of the child process.  SIGINT is normally sent to all child
processes by the OS at the same time already as was the established normal
behavior in 2.7 and 3.2.  This behavior change was introduced during the fix to https://bugs.python.org/issue12494 and is generally surprising to command line
tool users who expect other tools launched in child processes to get their own
SIGINT and do their own cleanup.

In Python 3.3-3.6 subprocess.call and subprocess.run would immediately
SIGKILL the child process upon receiving a SIGINT (which raises a
KeyboardInterrupt).  We now give the child a small amount of time to
exit gracefully before resorting to a SIGKILL.

This is also the case for subprocess.Popen.__exit__ which would
previously block indefinitely waiting for the child to die.  This was
hidden from many users by virtue of subprocess.call and subprocess.run
sending the signal immediately.

Behavior change: subprocess.Popen.__exit__ will not block indefinitely
when the exiting exception is a KeyboardInterrupt.  This is done for
user friendliness as people expect their ^C to actually happen.  This
could cause occasional orphaned Popen objects when not using `call` or
`run` with a child process that hasn't exited.

Refactoring involved: The Popen.wait method deals with the
KeyboardInterrupt second chance, existing platform specific internals
have been renamed to _wait().
Also fixes comment typos.
2018-01-29 21:27:39 -08:00
Victor Stinner b31206a223
bpo-32667: Fix tests when $PATH contains a file (#5322)
Some tests failed when the PATH environment variable contained a path
to an existing file. Fix tests to ignore also NotADirectoryError, not
only FileNotFoundError and PermissionError.
2018-01-25 19:06:05 +01:00
izbyshev 2d8f06382e bpo-32369: test_subprocess: Fix pass_fds check in test_close_fds() (#4920)
The last part of test_close_fds() doesn't match its own comment.
The following assertion always holds because fds_to_keep and open_fds
are disjoint by construction.

self.assertFalse(remaining_fds & fds_to_keep & open_fds,
                 "Some fds not in pass_fds were left open")

Fix the code to match the message in the assertion.
2017-12-18 12:26:49 -08:00
Segev Finer b2a6083eb0 bpo-19764: Implemented support for subprocess.Popen(close_fds=True) on Windows (#1218)
Even though Python marks any handles it opens as non-inheritable there
is still a race when using `subprocess.Popen` since creating a process
with redirected stdio requires temporarily creating inheritable handles.
By implementing support for `subprocess.Popen(close_fds=True)` we fix
this race.

In order to implement this we use PROC_THREAD_ATTRIBUTE_HANDLE_LIST
which is available since Windows Vista. Which allows to pass an explicit
list of handles to inherit when creating a process.

This commit also adds `STARTUPINFO.lpAttributeList["handle_list"]`
which can be used to control PROC_THREAD_ATTRIBUTE_HANDLE_LIST
directly.
2017-12-18 10:28:19 +01:00
andyclegg 7fed7bd8bb bpo-31756: subprocess.run should alias universal_newlines to text (#4049)
Improve human friendliness of the Popen API: Add text=False as a
keyword-only argument to subprocess.Popen along with a Popen
attribute .text_mode and set this based on the
encoding/errors/universal_newlines/text arguments.

The universal_newlines parameter and attribute are maintained for
backwards compatibility.
2017-10-22 19:01:19 -07:00
Victor Stinner 11045c9d8a bpo-31178: Mock os.waitpid() in test_subprocess (#3896)
Fix test_exception_errpipe_bad_data() and
test_exception_errpipe_normal() of test_subprocess: mock os.waitpid()
to avoid calling the real os.waitpid(0, 0) which is an unexpected
side effect of the test.
2017-10-05 06:32:53 -07:00
Oren Milman 0b3a87ef54 bpo-31471: Fix assertion failure in subprocess.Popen() on Windows, in case env has a bad keys() method. (#3580) 2017-09-14 22:30:27 +03:00
Antoine Pitrou a6a4dc816d bpo-31370: Remove support for threads-less builds (#3385)
* Remove Setup.config
* Always define WITH_THREAD for compatibility.
2017-09-07 18:56:24 +02:00
Ammar Askar 3fc499bca1 bpo-31178: Avoid concatenating bytes with str in subprocess error (#3066)
Avoid concatenating bytes with str in the typically rare subprocess error path (exec failed). Includes a mock based unittest to exercise the codepath.
2017-09-05 23:41:30 -07:00
Ned Deily 918edc0edb bpo-12383: Also ignore __PYVENV_LAUNCHER__ (#3278)
Used in macOS framework builds.
2017-09-04 00:00:21 -04:00
Gregory P. Smith a3a6df36b6 Skip two tests not intended to pass on Windows. (#3202) 2017-08-24 18:15:02 -07:00
Gregory P. Smith 8621bb5d93 bpo-22536: Set the filename in FileNotFoundError. (#3194)
Have the subprocess module set the filename in the FileNotFoundError
exception raised on POSIX systems when the executable or cwd are missing.
2017-08-24 14:58:25 -07:00
Victor Stinner 9a83f651f3 Add test_subprocess.test_nonexisting_with_pipes() (#3133)
bpo-30121: Test the Popen failure when Popen was created with pipes.
Create also NONEXISTING_CMD variable in test_subprocess.py.
2017-08-21 23:51:31 +02:00
Segev Finer 4d3851727f bpo-30121: Fix debug assert in subprocess on Windows (#1224)
* bpo-30121: Fix debug assert in subprocess on Windows

This is caused by closing HANDLEs using os.close which is for CRT file
descriptors and not for HANDLEs.

* bpo-30121: Suppress debug assertion in test_subprocess when ran directly
2017-08-18 15:18:13 +02:00
Victor Stinner 7b7c6dcfff bpo-31173: Rewrite WSTOPSIG test of test_subprocess (#3055)
The current test_child_terminated_in_stopped_state() function test
creates a child process which calls ptrace(PTRACE_TRACEME, 0, 0) and
then crash (SIGSEGV). The problem is that calling os.waitpid() in the
parent process is not enough to close the process: the child process
remains alive and so the unit test leaks a child process in a
strange state. Closing the child process requires non-trivial code,
maybe platform specific.

Remove the functional test and replaces it with an unit test which
mocks os.waitpid() using a new _testcapi.W_STOPCODE() function to
test the WIFSTOPPED() path.
2017-08-10 12:37:39 +02:00
Ville Skyttä 49b2734bf1 Spelling fixes (#2902) 2017-08-03 09:00:59 +03:00
Victor Stinner cc42c121eb bpo-31067: test_subprocess calls reap_children() (#2931)
test_subprocess now also calls reap_children() in tearDown(), not
only on setUp().
2017-07-28 18:00:22 +02:00
Victor Stinner cdee3f14f7 bpo-30764: test_subprocess uses SuppressCrashReport (#2405)
bpo-30764, bpo-29335: test_child_terminated_in_stopped_state() of
test_subprocess now uses support.SuppressCrashReport() to prevent the
creation of a core dump on FreeBSD.
2017-06-26 17:23:03 +02:00
Serhiy Storchaka d174d24a5d bpo-30730: Prevent environment variables injection in subprocess on Windows. (#2325)
Prevent passing other invalid environment variables and command arguments.
2017-06-23 19:39:27 +03:00
Nick Coghlan 6ea4186de3 bpo-28180: Implementation for PEP 538 (#659)
- new PYTHONCOERCECLOCALE config setting
- coerces legacy C locale to C.UTF-8, C.utf8 or UTF-8 by default
- always uses C.UTF-8 on Android
- uses `surrogateescape` on stdin and stdout in the coercion
  target locales
- configure option to disable locale coercion at build time
- configure option to disable C locale warning at build time
2017-06-11 13:16:15 +10:00
Gregory P. Smith 85aba238e4 subprocess test_empty_env typo fix. (#1877) 2017-05-30 16:21:47 -07:00
Gregory P. Smith b351248c1f bpo-12383: Refactor subprocess test_empty_env (#1874)
Bugfix: This test wasn't being run because it was skipping based on the
presence of Py_ENABLE_SHARED rather than its value.  It is always present
on POSIX systems but defaults to 0.

Refactoring: Move the environment variables that can be ignored into a
function.  Parse the list from the child process and filter out the ones
to exclude in the parent before checking that the rest is empty.

Feature: Adds always present environment variables to ignore when
running in a Gentoo sandbox so that the test can pass there.
2017-05-30 14:40:37 -07:00
Gregory P. Smith 56bc3b768c bpo-29335 - apply suggested test_subprocess simplifications from haypo and Zach: (#1757)
use faulthandler._sigsegv() and ctypes.util.find_library('c')
2017-05-23 07:49:13 -07:00
Serhiy Storchaka 66bffd1663 bpo-30065: Fixed arguments validation in _posixsubprocess.fork_exec(). (#1110) 2017-04-19 21:12:46 +03:00
Sayan Chowdhury d5c11f7ace bpo-28624: Add a test that checks that cwd parameter of Popen() accepts PathLike objects (#157) 2017-02-26 20:06:10 +03:00
Subhendu Ghosh ae160bba20 bpo-26128: Added __init__to subprocess.STARTUPINFO (#171)
The Windows-specific subprocess.STARTUPINFO class now accepts
keyword-only arguments to its constructor to set the various
data attributes.

Patch by Subhendu Ghosh.
2017-02-26 00:59:05 +10:00
Gregory P. Smith 60e6e962ba Skip the test requiring ctypes if ctypes is unavailable.
prevents http://buildbot.python.org/all/builders/x86%20Ubuntu%20Shared%203.x/builds/240/steps/test/logs/stdio
2017-01-22 22:20:04 -08:00
Gregory P. Smith 9358a6e62b Skip the test requiring ctypes if ctypes is unavailable.
prevents http://buildbot.python.org/all/builders/x86%20Ubuntu%20Shared%203.x/builds/240/steps/test/logs/stdio
2017-01-22 22:19:51 -08:00
Gregory P. Smith 1fa08bcbbb Skip the test requiring ctypes if ctypes is unavailable.
prevents http://buildbot.python.org/all/builders/x86%20Ubuntu%20Shared%203.x/builds/240/steps/test/logs/stdio
2017-01-22 22:19:38 -08:00
Gregory P. Smith 773a8c4f8d typo fix, extra '.' on MacOS :) 2017-01-22 20:55:20 -08:00
Gregory P. Smith 21a9b1dfc6 typo fix, extra '.' on MacOS :) 2017-01-22 20:55:02 -08:00
Gregory P. Smith 21d333b703 typo fix, extra '.' :) 2017-01-22 20:54:42 -08:00
Gregory P. Smith 5c8706c04a Issue #29335: Fix subprocess.Popen.wait() when the child process has
exited to a stopped instead of terminated state (ex: when under ptrace).
2017-01-22 17:30:28 -08:00
Gregory P. Smith 78034c81fb Issue #29335: Fix subprocess.Popen.wait() when the child process has
exited to a stopped instead of terminated state (ex: when under ptrace).
2017-01-22 17:29:44 -08:00
Gregory P. Smith 50e16e33af Issue #29335: Fix subprocess.Popen.wait() when the child process has
exited to a stopped instead of terminated state (ex: when under ptrace).
2017-01-22 17:28:38 -08:00
Gregory P. Smith 82604e03dc Issue #20572: Remove the subprocess.Popen.wait endtime parameter.
It was deprecated in 3.4 and undocumented prior to that.
2016-11-20 16:31:07 -08:00