mirror of https://github.com/python/cpython
356 lines
11 KiB
ReStructuredText
356 lines
11 KiB
ReStructuredText
.. currentmodule:: asyncio
|
|
|
|
.. _asyncio-subprocess:
|
|
|
|
============
|
|
Subprocesses
|
|
============
|
|
|
|
This section describes high-level async/await asyncio APIs to
|
|
create and manage subprocesses.
|
|
|
|
.. _asyncio_example_subprocess_shell:
|
|
|
|
Here's an example of how asyncio can run a shell command and
|
|
communicate its result back::
|
|
|
|
import asyncio
|
|
|
|
async def run(cmd):
|
|
proc = await asyncio.create_subprocess_shell(
|
|
cmd,
|
|
stdout=asyncio.subprocess.PIPE,
|
|
stderr=asyncio.subprocess.PIPE)
|
|
|
|
stdout, stderr = await proc.communicate()
|
|
|
|
print(f'[{cmd!r} exited with {proc.returncode}]')
|
|
if stdout:
|
|
print(f'[stdout]\n{stdout.decode()}')
|
|
if stderr:
|
|
print(f'[stderr]\n{stderr.decode()}')
|
|
|
|
asyncio.run(run('ls /zzz'))
|
|
|
|
will print::
|
|
|
|
['ls /zzz' exited with 1]
|
|
[stderr]
|
|
ls: /zzz: No such file or directory
|
|
|
|
Because all asyncio subprocess functions are asynchronous and asyncio
|
|
provides many tools to work with such functions, it is easy to execute
|
|
and monitor multiple subprocesses in parallel. It is indeed trivial
|
|
to modify the above example to run a few commands at once::
|
|
|
|
async def main():
|
|
await asyncio.gather(
|
|
run('ls /zzz'),
|
|
run('sleep 1; echo "hello"'))
|
|
|
|
asyncio.run(main())
|
|
|
|
See also the `Examples`_ subsection.
|
|
|
|
|
|
Creating Subprocesses
|
|
=====================
|
|
|
|
.. coroutinefunction:: create_subprocess_exec(\*args, stdin=None, \
|
|
stdout=None, stderr=None, loop=None, \
|
|
limit=None, \*\*kwds)
|
|
|
|
Create a subprocess.
|
|
|
|
The *limit* argument sets the buffer limit for :class:`StreamReader`
|
|
wrappers for :attr:`Process.stdout` and :attr:`Process.stderr`
|
|
(if :attr:`subprocess.PIPE` is passed to *stdout* and *stderr* arguments).
|
|
|
|
Return a :class:`~asyncio.subprocess.Process` instance.
|
|
|
|
See the documentation of :meth:`loop.subprocess_exec` for other
|
|
parameters.
|
|
|
|
.. coroutinefunction:: create_subprocess_shell(cmd, stdin=None, \
|
|
stdout=None, stderr=None, loop=None, \
|
|
limit=None, \*\*kwds)
|
|
|
|
Run the shell command *cmd*.
|
|
|
|
The *limit* argument sets the buffer limit for :class:`StreamReader`
|
|
wrappers for :attr:`Process.stdout` and :attr:`Process.stderr`
|
|
(if :attr:`subprocess.PIPE` is passed to *stdout* and *stderr* arguments).
|
|
|
|
Return a :class:`~asyncio.subprocess.Process` instance.
|
|
|
|
See the documentation of :meth:`loop.subprocess_shell` for other
|
|
parameters.
|
|
|
|
.. important::
|
|
|
|
It is the application's responsibility to ensure that all whitespace and
|
|
metacharacters are quoted appropriately to avoid `shell injection
|
|
<https://en.wikipedia.org/wiki/Shell_injection#Shell_injection>`_
|
|
vulnerabilities. The :func:`shlex.quote` function can be used to properly
|
|
escape whitespace and shell metacharacters in strings that are going to be
|
|
used to construct shell commands.
|
|
|
|
.. note::
|
|
|
|
The default event loop that asyncio is pre-configured
|
|
to use on **Windows** does not support subprocesses. Subprocesses are
|
|
available for Windows if the :class:`ProactorEventLoop` is used.
|
|
See :ref:`Subprocess Support on Windows <asyncio-windows-subprocess>`
|
|
for details.
|
|
|
|
.. seealso::
|
|
|
|
asyncio has also *low-level* APIs to work with subprocesses:
|
|
:meth:`loop.subprocess_exec`, :meth:`loop.subprocess_shell`,
|
|
:meth:`loop.connect_read_pipe`, :meth:`loop.connect_write_pipe`,
|
|
as well as the :ref:`Subprocess Transports <asyncio-subprocess-transports>`
|
|
and :ref:`Subprocess Protocols <asyncio-subprocess-protocols>`.
|
|
|
|
|
|
Constants
|
|
=========
|
|
|
|
.. data:: asyncio.subprocess.PIPE
|
|
|
|
Can be passed to the *stdin*, *stdout* or *stderr* parameters.
|
|
|
|
If *PIPE* is passed to *stdin* argument, the
|
|
:attr:`Process.stdin <asyncio.subprocess.Process.stdin>` attribute
|
|
will point to a :class:`StreamWriter` instance.
|
|
|
|
If *PIPE* is passed to *stdout* or *stderr* arguments, the
|
|
:attr:`Process.stdout <asyncio.subprocess.Process.stdout>` and
|
|
:attr:`Process.stderr <asyncio.subprocess.Process.stderr>`
|
|
attributes will point to :class:`StreamReader` instances.
|
|
|
|
.. data:: asyncio.subprocess.STDOUT
|
|
|
|
Can be passed to the *stderr* parameter to redirect process'
|
|
*stderr* to *stdout*.
|
|
|
|
.. data:: asyncio.subprocess.DEVNULL
|
|
|
|
Can be passed as the *stdin*, *stdout* or *stderr* parameters
|
|
to redirect the corresponding subprocess' IO to :data:`os.devnull`.
|
|
|
|
|
|
Interacting with Subprocesses
|
|
=============================
|
|
|
|
Both :func:`create_subprocess_exec` and :func:`create_subprocess_shell`
|
|
functions return instances of the *Process* class. It is a high-level
|
|
wrapper that allows to watch for subprocesses completion and
|
|
communicate with them.
|
|
|
|
.. class:: asyncio.subprocess.Process
|
|
|
|
An object that wraps OS processes created by the
|
|
:func:`create_subprocess_exec` and :func:`create_subprocess_shell`
|
|
functions.
|
|
|
|
This class is designed to have a similar API to the
|
|
:class:`subprocess.Popen` class, but there are some
|
|
notable differences:
|
|
|
|
* unlike Popen, Process instances do not have an equivalent to
|
|
the :meth:`~subprocess.Popen.poll` method;
|
|
|
|
* the :meth:`~asyncio.subprocess.Process.communicate` and
|
|
:meth:`~asyncio.subprocess.Process.wait` methods don't take a
|
|
*timeout* parameter: use the :func:`wait_for` function;
|
|
|
|
* the :meth:`Process.wait() <asyncio.subprocess.Process.wait>` method
|
|
is asynchronous, whereas :meth:`subprocess.Popen.wait` method
|
|
is implemented as a blocking busy loop;
|
|
|
|
* the *universal_newlines* parameter is not supported.
|
|
|
|
This class is :ref:`not thread safe <asyncio-multithreading>`.
|
|
|
|
See also the :ref:`Subprocess and Threads <asyncio-subprocess-threads>`
|
|
section.
|
|
|
|
.. coroutinemethod:: wait()
|
|
|
|
Wait for child process to terminate.
|
|
|
|
Set and return the :attr:`returncode` attribute.
|
|
|
|
.. note::
|
|
|
|
This method can deadlock when using ``stdout=PIPE`` or
|
|
``stderr=PIPE`` and the child process generates so much output
|
|
that it blocks waiting for the OS pipe buffer to accept
|
|
more data. Use the :meth:`communicate` method when using pipes
|
|
to avoid this condition.
|
|
|
|
.. coroutinemethod:: communicate(input=None)
|
|
|
|
Interact with process:
|
|
|
|
1. send data to *stdin* (if *input* is not ``None``);
|
|
2. read data from *stdout* and *stderr*, until EOF is reached;
|
|
3. wait for process to terminate.
|
|
|
|
The optional *input* argument is the data (:class:`bytes` object)
|
|
that will be sent to the child process.
|
|
|
|
Return a tuple ``(stdout_data, stderr_data)``.
|
|
|
|
If either :exc:`BrokenPipeError` or :exc:`ConnectionResetError`
|
|
exception is raised when writing *input* into *stdin*, the
|
|
exception is ignored. This condition occurs when the process
|
|
exits before all data are written into *stdin*.
|
|
|
|
If it is desired to send data to the process' *stdin*,
|
|
the process needs to be created with ``stdin=PIPE``. Similarly,
|
|
to get anything other than ``None`` in the result tuple, the
|
|
process has to be created with ``stdout=PIPE`` and/or
|
|
``stderr=PIPE`` arguments.
|
|
|
|
Note, that the data read is buffered in memory, so do not use
|
|
this method if the data size is large or unlimited.
|
|
|
|
.. method:: send_signal(signal)
|
|
|
|
Sends the signal *signal* to the child process.
|
|
|
|
.. note::
|
|
|
|
On Windows, :py:data:`SIGTERM` is an alias for :meth:`terminate`.
|
|
``CTRL_C_EVENT`` and ``CTRL_BREAK_EVENT`` can be sent to processes
|
|
started with a *creationflags* parameter which includes
|
|
``CREATE_NEW_PROCESS_GROUP``.
|
|
|
|
.. method:: terminate()
|
|
|
|
Stop the child.
|
|
|
|
On Posix OSs the method sends :py:data:`signal.SIGTERM` to the
|
|
child process.
|
|
|
|
On Windows the Win32 API function :c:func:`TerminateProcess` is
|
|
called to stop the child process.
|
|
|
|
.. method:: kill()
|
|
|
|
Kill the child.
|
|
|
|
On Posix OSs the function sends :py:data:`SIGKILL` to the child
|
|
process.
|
|
|
|
On Windows this method is an alias for :meth:`terminate`.
|
|
|
|
.. attribute:: stdin
|
|
|
|
Standard input stream (:class:`StreamWriter`) or ``None``
|
|
if the process was created with ``stdin=None``.
|
|
|
|
.. attribute:: stdout
|
|
|
|
Standard output stream (:class:`StreamReader`) or ``None``
|
|
if the process was created with ``stdout=None``.
|
|
|
|
.. attribute:: stderr
|
|
|
|
Standard error stream (:class:`StreamReader`) or ``None``
|
|
if the process was created with ``stderr=None``.
|
|
|
|
.. warning::
|
|
|
|
Use the :meth:`communicate` method rather than
|
|
:attr:`process.stdin.write() <stdin>`,
|
|
:attr:`await process.stdout.read() <stdout>` or
|
|
:attr:`await process.stderr.read <stderr>`.
|
|
This avoids deadlocks due to streams pausing reading or writing
|
|
and blocking the child process.
|
|
|
|
.. attribute:: pid
|
|
|
|
Process identification number (PID).
|
|
|
|
Note that for processes created by the :func:`create_subprocess_shell`
|
|
function, this attribute is the PID of the spawned shell.
|
|
|
|
.. attribute:: returncode
|
|
|
|
Return code of the process when it exits.
|
|
|
|
A ``None`` value indicates that the process has not terminated yet.
|
|
|
|
A negative value ``-N`` indicates that the child was terminated
|
|
by signal ``N`` (Unix only).
|
|
|
|
|
|
.. _asyncio-subprocess-threads:
|
|
|
|
Subprocess and Threads
|
|
----------------------
|
|
|
|
asyncio built-in event loops support running subprocesses from
|
|
different threads, but there are the following limitations:
|
|
|
|
* An event loop must run in the main thread.
|
|
|
|
* The child watcher must be instantiated in the main thread,
|
|
before executing subprocesses from other threads. Call the
|
|
:func:`get_child_watcher` function in the main thread to instantiate
|
|
the child watcher.
|
|
|
|
Note, that alternative event loop implementations might not share
|
|
the above limitations; please refer to their documentation.
|
|
|
|
.. seealso::
|
|
|
|
The :ref:`Concurrency and multithreading in asyncio
|
|
<asyncio-multithreading>` section.
|
|
|
|
|
|
Examples
|
|
--------
|
|
|
|
An example using the :class:`~asyncio.subprocess.Process` class to
|
|
control a subprocess and the :class:`StreamReader` class to read from
|
|
the *stdout*.
|
|
|
|
.. _asyncio_example_create_subprocess_exec:
|
|
|
|
The subprocess is created by the :func:`create_subprocess_exec`
|
|
function::
|
|
|
|
import asyncio
|
|
import sys
|
|
|
|
async def get_date():
|
|
code = 'import datetime; print(datetime.datetime.now())'
|
|
|
|
# Create the subprocess; redirect the standard output
|
|
# into a pipe.
|
|
proc = await asyncio.create_subprocess_exec(
|
|
sys.executable, '-c', code,
|
|
stdout=asyncio.subprocess.PIPE)
|
|
|
|
# Read one line of output.
|
|
data = await proc.stdout.readline()
|
|
line = data.decode('ascii').rstrip()
|
|
|
|
# Wait for the subprocess exit.
|
|
await proc.wait()
|
|
return line
|
|
|
|
if sys.platform == "win32":
|
|
asyncio.set_event_loop_policy(
|
|
asyncio.WindowsProactorEventLoopPolicy())
|
|
|
|
date = asyncio.run(get_date())
|
|
print(f"Current date: {date}")
|
|
|
|
|
|
See also the :ref:`same example <asyncio_example_subprocess_proto>`
|
|
written using low-level APIs.
|