Commit Graph

309 Commits

Author SHA1 Message Date
Serhiy Storchaka be50a7b627 Revert "bpo-31961: subprocess now accepts path-like args (GH-4329)" (#5912)
* Revert "bpo-31961: subprocess now accepts path-like args (GH-4329)"

This reverts commit dd42cb71f2.
2018-02-27 18:03:46 -05:00
Anders Lorentsen dd42cb71f2 bpo-31961: subprocess now accepts path-like args (GH-4329)
Allow os.PathLike args in subprocess APIs.
2018-01-29 23:27:28 -08:00
Bo Bayles ce0f33d045 bpo-32102 Add "capture_output=True" to subprocess.run (GH-5149)
Add "capture_output=True" option to subprocess.run, this is equivalent to
setting stdout=PIPE, stderr=PIPE but is much more readable.
2018-01-29 22:40:39 -08:00
Gregory P. Smith f4d644f36f
bpo-25942: make subprocess more graceful on ^C (GH-5026)
Do not allow receiving a SIGINT to cause the subprocess module to trigger an
immediate SIGKILL of the child process.  SIGINT is normally sent to all child
processes by the OS at the same time already as was the established normal
behavior in 2.7 and 3.2.  This behavior change was introduced during the fix to https://bugs.python.org/issue12494 and is generally surprising to command line
tool users who expect other tools launched in child processes to get their own
SIGINT and do their own cleanup.

In Python 3.3-3.6 subprocess.call and subprocess.run would immediately
SIGKILL the child process upon receiving a SIGINT (which raises a
KeyboardInterrupt).  We now give the child a small amount of time to
exit gracefully before resorting to a SIGKILL.

This is also the case for subprocess.Popen.__exit__ which would
previously block indefinitely waiting for the child to die.  This was
hidden from many users by virtue of subprocess.call and subprocess.run
sending the signal immediately.

Behavior change: subprocess.Popen.__exit__ will not block indefinitely
when the exiting exception is a KeyboardInterrupt.  This is done for
user friendliness as people expect their ^C to actually happen.  This
could cause occasional orphaned Popen objects when not using `call` or
`run` with a child process that hasn't exited.

Refactoring involved: The Popen.wait method deals with the
KeyboardInterrupt second chance, existing platform specific internals
have been renamed to _wait().
Also fixes comment typos.
2018-01-29 21:27:39 -08:00
Segev Finer b2a6083eb0 bpo-19764: Implemented support for subprocess.Popen(close_fds=True) on Windows (#1218)
Even though Python marks any handles it opens as non-inheritable there
is still a race when using `subprocess.Popen` since creating a process
with redirected stdio requires temporarily creating inheritable handles.
By implementing support for `subprocess.Popen(close_fds=True)` we fix
this race.

In order to implement this we use PROC_THREAD_ATTRIBUTE_HANDLE_LIST
which is available since Windows Vista. Which allows to pass an explicit
list of handles to inherit when creating a process.

This commit also adds `STARTUPINFO.lpAttributeList["handle_list"]`
which can be used to control PROC_THREAD_ATTRIBUTE_HANDLE_LIST
directly.
2017-12-18 10:28:19 +01:00
Victor Stinner 91106cd9ff
bpo-29240: PEP 540: Add a new UTF-8 Mode (#855)
* Add -X utf8 command line option, PYTHONUTF8 environment variable
  and a new sys.flags.utf8_mode flag.
* If the LC_CTYPE locale is "C" at startup: enable automatically the
  UTF-8 mode.
* Add _winapi.GetACP(). encodings._alias_mbcs() now calls
  _winapi.GetACP() to get the ANSI code page
* locale.getpreferredencoding() now returns 'UTF-8' in the UTF-8
  mode. As a side effect, open() now uses the UTF-8 encoding by
  default in this mode.
* Py_DecodeLocale() and Py_EncodeLocale() now use the UTF-8 encoding
  in the UTF-8 Mode.
* Update subprocess._args_from_interpreter_flags() to handle -X utf8
* Skip some tests relying on the current locale if the UTF-8 mode is
  enabled.
* Add test_utf8mode.py.
* _Py_DecodeUTF8_surrogateescape() gets a new optional parameter to
  return also the length (number of wide characters).
* pymain_get_global_config() and pymain_set_global_config() now
  always copy flag values, rather than only copying if the new value
  is greater than the old value.
2017-12-13 12:29:09 +01:00
Victor Stinner 747f48e2e9
bpo-32230: Set sys.warnoptions with -X dev (#4820)
Rather than supporting dev mode directly in the warnings module, this
instead adjusts the initialisation code to add an extra 'default'
entry to sys.warnoptions when dev mode is enabled.

This ensures that dev mode behaves *exactly* as if `-Wdefault` had
been passed on the command line, including in the way it interacts
with `sys.warnoptions`, and with other command line flags like `-bb`.

Fix also bpo-20361: have -b & -bb options take precedence over any
other warnings options.

Patch written by Nick Coghlan, with minor modifications of Victor Stinner.
2017-12-12 22:59:48 +01:00
Victor Stinner 09f3a8a124
bpo-32089: Fix warnings filters in dev mode (#4482)
The developer mode (-X dev) now creates all default warnings filters
to order filters in the correct order to always show ResourceWarning
and make BytesWarning depend on the -b option.

Write a functional test to make sure that ResourceWarning is logged
twice at the same location in the developer mode.

Add a new 'dev_mode' field to _PyCoreConfig.
2017-11-20 17:32:40 -08:00
Victor Stinner f39b674876
bpo-32094: Update subprocess for -X dev (#4480)
Modify subprocess._args_from_interpreter_flags() to handle -X dev
option.

Add also unit tests for test.support.args_from_interpreter_flags()
and test.support.optim_args_from_interpreter_flags().
2017-11-20 15:24:56 -08:00
James b5d9e08114 bpo-31884 subprocess: add Windows constants for process priority (#4150) 2017-11-08 06:18:59 -08:00
luzpaz a5293b4ff2 Fix miscellaneous typos (#4275) 2017-11-05 15:37:50 +02:00
andyclegg 7fed7bd8bb bpo-31756: subprocess.run should alias universal_newlines to text (#4049)
Improve human friendliness of the Popen API: Add text=False as a
keyword-only argument to subprocess.Popen along with a Popen
attribute .text_mode and set this based on the
encoding/errors/universal_newlines/text arguments.

The universal_newlines parameter and attribute are maintained for
backwards compatibility.
2017-10-22 19:01:19 -07:00
Joel Schaerer 88031a9ade Update subprocess.communicate() docstring.
Explicitly state that communicate() closes stdin after writing input to it.
2017-09-13 12:11:20 -07:00
Gregory P. Smith 2eb0cb4787 bpo-22635: Update the getstatusoutput docstring. (#3435)
To match the documentation updates already made.
Also renames the local variable used within to match
what it actually holds.
2017-09-07 16:11:02 -07:00
Antoine Pitrou a6a4dc816d bpo-31370: Remove support for threads-less builds (#3385)
* Remove Setup.config
* Always define WITH_THREAD for compatibility.
2017-09-07 18:56:24 +02:00
Ammar Askar 3fc499bca1 bpo-31178: Avoid concatenating bytes with str in subprocess error (#3066)
Avoid concatenating bytes with str in the typically rare subprocess error path (exec failed). Includes a mock based unittest to exercise the codepath.
2017-09-05 23:41:30 -07:00
Gregory P. Smith 8621bb5d93 bpo-22536: Set the filename in FileNotFoundError. (#3194)
Have the subprocess module set the filename in the FileNotFoundError
exception raised on POSIX systems when the executable or cwd are missing.
2017-08-24 14:58:25 -07:00
Segev Finer 4d3851727f bpo-30121: Fix debug assert in subprocess on Windows (#1224)
* bpo-30121: Fix debug assert in subprocess on Windows

This is caused by closing HANDLEs using os.close which is for CRT file
descriptors and not for HANDLEs.

* bpo-30121: Suppress debug assertion in test_subprocess when ran directly
2017-08-18 15:18:13 +02:00
Serhiy Storchaka d174d24a5d bpo-30730: Prevent environment variables injection in subprocess on Windows. (#2325)
Prevent passing other invalid environment variables and command arguments.
2017-06-23 19:39:27 +03:00
Victor Stinner d52aa31378 bpo-30418: Popen.communicate() always ignore EINVAL (#2002)
On Windows, subprocess.Popen.communicate() now also ignore EINVAL
on stdin.write() if the child process is still running but closed the
pipe.
2017-06-08 17:30:39 +02:00
Serhiy Storchaka 66bffd1663 bpo-30065: Fixed arguments validation in _posixsubprocess.fork_exec(). (#1110) 2017-04-19 21:12:46 +03:00
Benjamin Peterson cb90f26122 fix subprocess on Windows (#391) 2017-03-02 00:03:41 -08:00
Benjamin Peterson f52279257e allow path-like objects to be cwd on windows (#389)
#157 added the test, but it's currently (correctly) broken on windows.
2017-03-01 23:04:03 -08:00
Subhendu Ghosh ae160bba20 bpo-26128: Added __init__to subprocess.STARTUPINFO (#171)
The Windows-specific subprocess.STARTUPINFO class now accepts
keyword-only arguments to its constructor to set the various
data attributes.

Patch by Subhendu Ghosh.
2017-02-26 00:59:05 +10:00
Gregory P. Smith 5c8706c04a Issue #29335: Fix subprocess.Popen.wait() when the child process has
exited to a stopped instead of terminated state (ex: when under ptrace).
2017-01-22 17:30:28 -08:00
Gregory P. Smith 78034c81fb Issue #29335: Fix subprocess.Popen.wait() when the child process has
exited to a stopped instead of terminated state (ex: when under ptrace).
2017-01-22 17:29:44 -08:00
Gregory P. Smith 50e16e33af Issue #29335: Fix subprocess.Popen.wait() when the child process has
exited to a stopped instead of terminated state (ex: when under ptrace).
2017-01-22 17:28:38 -08:00
Victor Stinner 1459ca64fc Merge 3.6 2017-01-06 10:49:32 +01:00
Victor Stinner 9505b03bb0 Fix subprocess.Popen.__del__() fox Python shutdown
Issue #29174, #26741: subprocess.Popen.__del__() now keeps a strong reference
to warnings.warn() function.
2017-01-06 10:44:44 +01:00
Xavier de Gaye b35fc626c1 Issue #16255: subrocess.Popen uses /system/bin/sh on Android as the shell,
instead of /bin/sh.
2016-12-13 16:32:21 +01:00
Gregory P. Smith 82604e03dc Issue #20572: Remove the subprocess.Popen.wait endtime parameter.
It was deprecated in 3.4 and undocumented prior to that.
2016-11-20 16:31:07 -08:00
Gregory P. Smith f0e98c510d Issue #20572: The subprocess.Popen.wait method's undocumented endtime
parameter now raises a DeprecationWarning.  It was deprecated in 3.4.
It was never documented prior to that.
2016-11-20 16:25:14 -08:00
Martin Panter 3dca62440c Issue #26240: Merge subprocess doc string from 3.5 into 3.6 2016-10-25 23:41:42 +00:00
Martin Panter 4afdca056b Issue #26240: Clean up the subprocess module doc string
Patch by Tim Mitchell.
2016-10-25 22:20:48 +00:00
Steve Dower 050acaed99 Issue #6135: Adds encoding and errors parameters to subprocess 2016-09-06 20:16:17 -07:00
Victor Stinner c206f1eb1c subprocess: enhance ResourceWarning message
* Add the process identifier to the warning message
* Add also a comment to explain the issue
2016-06-14 16:42:59 +02:00
Gregory P. Smith ext:(%20%5BGoogle%20Inc.%5D) fcbf1ca4f9 merge from 3.5 - Fixes Issue #26373: subprocess.Popen.communicate
now correctly ignores BrokenPipeError when the child process dies
before .communicate() is called in more (all?) circumstances.
2016-06-04 00:34:15 +00:00
Gregory P. Smith ext:(%20%5BGoogle%20Inc.%5D) 1ef8c7e886 Fixes Issue #26373: subprocess.Popen.communicate now correctly ignores
BrokenPipeError when the child process dies before .communicate()
is called in more (all?) circumstances.
2016-06-04 00:22:17 +00:00
Gregory P. Smith ext:(%20%5BGoogle%20Inc.%5D) d6da7604d3 Issue #27167: Clarify the subprocess.CalledProcessError error message text
when the child process died due to a signal.
2016-06-03 06:14:06 +00:00
Victor Stinner 5a48e21ff1 subprocess now emits a ResourceWarning warning
Issue #26741: subprocess.Popen destructor now emits a ResourceWarning warning
if the child process is still running.
2016-05-20 12:11:15 +02:00
Victor Stinner a58e2c5c49 Issue #26741: POSIX implementation of subprocess.Popen._execute_child() now
sets the returncode attribute using the child process exit status when exec
failed.
2016-05-20 12:08:12 +02:00
Martin Panter f94ec1bd83 Issue #22274: Merge stderr=STDOUT fix from 3.5 2016-05-13 07:45:21 +00:00
Martin Panter c76358924f Issue #22274: Redirect stderr=STDOUT when stdout not redirected, by Akira Li 2016-05-13 01:54:44 +00:00
Martin Panter 528619b6c3 Issue #26782: Add STARTUPINFO to subprocess.__all__ on Windows 2016-04-16 23:42:37 +00:00
Victor Stinner 9def284387 subprocess._optim_args_from_interpreter_flags()
Issue #26100:

* Add subprocess._optim_args_from_interpreter_flags()
* Add test.support.optim_args_from_interpreter_flags()
* Use new functions in distutils, test_cmd_line_script, test_compileall and
  test_inspect

The change enables test_details() test of test_inspect when -O or -OO command
line option is used.
2016-01-18 12:15:08 +01:00
Gregory P. Smith 6edadfc9ca * Re-fix issue #19284: Don't generate the no-op -R command line
parameter to "enable" the always on sys.flags.hash_randomization
  in _args_from_interpreter_flags() used by multiprocessing and
  some unittests.  This simplifies the code.
* assert_python_ok docstring typo fix.
* Fix test_cmd_line not to fail if PYTHONHASHSEED is set to a fixed seed.
2015-12-13 20:09:42 -08:00
Gregory P. Smith 0212c4dc6b Re-fix issue #19284: Don't generate the no-op -R command line
parameter to "enable" the always on sys.flags.hash_randomization
in _args_from_interpreter_flags() used by multiprocessing and
some unittests.  This simplifies the code.
2015-12-13 20:05:55 -08:00
Gregory P. Smith 2cd1b3b089 Fix issue #6973: When we know a subprocess.Popen process has died, do
not allow the send_signal(), terminate(), or kill() methods to do
anything as they could potentially signal a different process.
2015-11-15 18:26:11 -08:00
Gregory P. Smith a0c9caad66 Fix issue #6973: When we know a subprocess.Popen process has died, do
not allow the send_signal(), terminate(), or kill() methods to do
anything as they could potentially signal a different process.
2015-11-15 18:19:10 -08:00
Gregory P. Smith 6e73000723 Add a subprocess.run() function than returns a CalledProcess instance for a
more consistent API than the existing call* functions.
(enhancement from issue 23342)
2015-04-14 16:14:25 -07:00