.. currentmodule:: asyncio .. _asyncio-subprocess: ============ Subprocesses ============ This section describes high-level async/await asyncio APIs to create and manage subprocesses. Here's an example of how asyncio can run a shell command and communicate its result back:: import asyncio async def run(cmd): proc = await asyncio.create_subprocess_shell( cmd, stdout=asyncio.subprocess.PIPE, stderr=asyncio.subprocess.PIPE) stdout, stderr = await proc.communicate() print(f'[{cmd!r} exited with {proc.returncode}]') if stdout: print(f'[stdout]\n{stdout.decode()}') if stderr: print(f'[stderr]\n{stderr.decode()}') asyncio.run(run('ls /zzz')) will print:: ['ls /zzz' exited with 1] [stderr] ls: /zzz: No such file or directory Because all asyncio subprocess functions are asynchronous and asyncio provides many tools to work with such functions, it is easy to execute and monitor multiple subprocesses in parallel. It is indeed trivial to modify the above example to run a few commands at once:: async def main(): await asyncio.gather( run('ls /zzz'), run('sleep 1; echo "hello"')) asyncio.run(main()) See also the `Examples`_ subsection. Creating Subprocesses ===================== .. coroutinefunction:: create_subprocess_exec(\*args, stdin=None, \ stdout=None, stderr=None, loop=None, \ limit=None, \*\*kwds) Create a subprocess. The *limit* argument sets the buffer limit for :class:`StreamReader` wrappers for :attr:`Process.stdout` and :attr:`Process.stderr` (if :attr:`subprocess.PIPE` is passed to *stdout* and *stderr* arguments). Return a :class:`~asyncio.subprocess.Process` instance. See the documentation of :meth:`loop.subprocess_exec` for other parameters. .. coroutinefunction:: create_subprocess_shell(cmd, stdin=None, \ stdout=None, stderr=None, loop=None, \ limit=None, \*\*kwds) Run the shell command *cmd*. The *limit* argument sets the buffer limit for :class:`StreamReader` wrappers for :attr:`Process.stdout` and :attr:`Process.stderr` (if :attr:`subprocess.PIPE` is passed to *stdout* and *stderr* arguments). Return a :class:`~asyncio.subprocess.Process` instance. See the documentation of :meth:`loop.subprocess_shell` for other parameters. .. important:: It is the application's responsibility to ensure that all whitespace and metacharacters are quoted appropriately to avoid `shell injection `_ vulnerabilities. The :func:`shlex.quote` function can be used to properly escape whitespace and shell metacharacters in strings that are going to be used to construct shell commands. .. note:: The default event loop that asyncio is pre-configured to use on **Windows** does not support subprocesses. Subprocesses are available for Windows if the :class:`ProactorEventLoop` is used. See :ref:`Subprocess Support on Windows ` for details. .. seealso:: asyncio has also *low-level* APIs to work with subprocesses: :meth:`loop.subprocess_exec`, :meth:`loop.subprocess_shell`, :meth:`loop.connect_read_pipe`, :meth:`loop.connect_write_pipe`, as well as the :ref:`Subprocess Transports ` and :ref:`Subprocess Protocols `. Constants ========= .. data:: asyncio.subprocess.PIPE Can be passed to the *stdin*, *stdout* or *stderr* parameters. If *PIPE* is passed to *stdin* argument, the :attr:`Process.stdin ` attribute will point to a :class:`StreamWriter` instance. If *PIPE* is passed to *stdout* or *stderr* arguments, the :attr:`Process.stdout ` and :attr:`Process.stderr ` attributes will point to :class:`StreamReader` instances. .. data:: asyncio.subprocess.STDOUT Can be passed to the *stderr* parameter to redirect process' *stderr* to *stdout*. .. data:: asyncio.subprocess.DEVNULL Can be passed as the *stdin*, *stdout* or *stderr* parameters to redirect the corresponding subprocess' IO to :data:`os.devnull`. Interacting with Subprocesses ============================= Both :func:`create_subprocess_exec` and :func:`create_subprocess_shell` functions return instances of the *Process* class. It is a high-level wrapper that allows to watch for subprocesses completion and communicate with them. .. class:: asyncio.subprocess.Process An object that wraps OS processes created by the :func:`create_subprocess_exec` and :func:`create_subprocess_shell` functions. This class is designed to have a similar API to the :class:`subprocess.Popen` class, but there are some notable differences: * unlike Popen, Process instances do not have an equivalent to the :meth:`~subprocess.Popen.poll` method; * the :meth:`~asyncio.subprocess.Process.communicate` and :meth:`~asyncio.subprocess.Process.wait` methods don't take a *timeout* parameter: use the :func:`wait_for` function; * the :meth:`Process.wait() ` method is asynchronous, whereas :meth:`subprocess.Popen.wait` method is implemented as a blocking busy loop; * the *universal_newlines* parameter is not supported. This class is :ref:`not thread safe `. See also the :ref:`Subprocess and Threads ` section. .. coroutinemethod:: wait() Wait for child process to terminate. Set and return the :attr:`returncode` attribute. .. note:: This method can deadlock when using ``stdout=PIPE`` or ``stderr=PIPE`` and the child process generates so much output that it blocks waiting for the OS pipe buffer to accept more data. Use the :meth:`communicate` method when using pipes to avoid this condition. .. coroutinemethod:: communicate(input=None) Interact with process: 1. send data to *stdin* (if *input* is not ``None``); 2. read data from *stdout* and *stderr*, until EOF is reached; 3. wait for process to terminate. The optional *input* argument is the data (:class:`bytes` object) that will be sent to the child process. Return a tuple ``(stdout_data, stderr_data)``. If either :exc:`BrokenPipeError` or :exc:`ConnectionResetError` exception is raised when writing *input* into *stdin*, the exception is ignored. This condition occurs when the process exits before all data are written into *stdin*. If it is desired to send data to the process' *stdin*, the process needs to be created with ``stdin=PIPE``. Similarly, to get anything other than ``None`` in the result tuple, the process has to be created with ``stdout=PIPE`` and/or ``stderr=PIPE`` arguments. Note, that the data read is buffered in memory, so do not use this method if the data size is large or unlimited. .. method:: send_signal(signal) Sends the signal *signal* to the child process. .. note:: On Windows, :py:data:`SIGTERM` is an alias for :meth:`terminate`. ``CTRL_C_EVENT`` and ``CTRL_BREAK_EVENT`` can be sent to processes started with a *creationflags* parameter which includes ``CREATE_NEW_PROCESS_GROUP``. .. method:: terminate() Stop the child. On Posix OSs the method sends :py:data:`signal.SIGTERM` to the child process. On Windows the Win32 API function :c:func:`TerminateProcess` is called to stop the child process. .. method:: kill() Kill the child. On Posix OSs the function sends :py:data:`SIGKILL` to the child process. On Windows this method is an alias for :meth:`terminate`. .. attribute:: stdin Standard input stream (:class:`StreamWriter`) or ``None`` if the process was created with ``stdin=None``. .. attribute:: stdout Standard output stream (:class:`StreamReader`) or ``None`` if the process was created with ``stdout=None``. .. attribute:: stderr Standard error stream (:class:`StreamReader`) or ``None`` if the process was created with ``stderr=None``. .. warning:: Use the :meth:`communicate` method rather than :attr:`process.stdin.write() `, :attr:`await process.stdout.read() ` or :attr:`await process.stderr.read `. This avoids deadlocks due to streams pausing reading or writing and blocking the child process. .. attribute:: pid Process identification number (PID). Note that for processes created by the :func:`create_subprocess_shell` function, this attribute is the PID of the spawned shell. .. attribute:: returncode Return code of the process when it exits. A ``None`` value indicates that the process has not terminated yet. A negative value ``-N`` indicates that the child was terminated by signal ``N`` (Unix only). .. _asyncio-subprocess-threads: Subprocess and Threads ---------------------- asyncio built-in event loops support running subprocesses from different threads, but there are the following limitations: * An event loop must run in the main thread. * The child watcher must be instantiated in the main thread, before executing subprocesses from other threads. Call the :func:`get_child_watcher` function in the main thread to instantiate the child watcher. Note, that alternative event loop implementations might not share the above limitations; please refer to their documentation. .. seealso:: The :ref:`Concurrency and multithreading in asyncio ` section. Examples -------- An example using the :class:`~asyncio.subprocess.Process` class to control a subprocess and the :class:`StreamReader` class to read from the *stdout*. The subprocess is created by the :func:`create_subprocess_exec` function:: import asyncio import sys async def get_date(): code = 'import datetime; print(datetime.datetime.now())' # Create the subprocess; redirect the standard output # into a pipe. proc = await asyncio.create_subprocess_exec( sys.executable, '-c', code, stdout=asyncio.subprocess.PIPE) # Read one line of output. data = await proc.stdout.readline() line = data.decode('ascii').rstrip() # Wait for the subprocess exit. await proc.wait() return line if sys.platform == "win32": asyncio.set_event_loop_policy( asyncio.WindowsProactorEventLoopPolicy()) date = asyncio.run(get_date()) print(f"Current date: {date}") See also the :ref:`same example ` written using low-level APIs.