asyncio earlier relied on subprocess module to send signals to the process, this has some drawbacks one being that subprocess module unnecessarily calls waitpid on child processes and hence it races with asyncio implementation which internally uses child watchers. To mitigate this, now asyncio sends signals directly to the process without going through the subprocess on non windows systems. On Windows it fallbacks to subprocess module handling but on windows there are no child watchers so this issue doesn't exists altogether.
In case the spawned process is setuid, we may not be able to send
signals to it, in which case our .kill() call will raise
PermissionError.
Ignore that in order to avoid .close() raising an exception. Hopefully
the process will exit as a result of receiving EOF on its stdin.
This will address the common mistake many asyncio users make:
an "except Exception" clause breaking Tasks cancellation.
In addition to this change, we stop inheriting asyncio.TimeoutError
and asyncio.InvalidStateError from their concurrent.futures.*
counterparts. There's no point for these exceptions to share the
inheritance chain.
In 3.9 we'll focus on implementing supervisors and cancel scopes,
which should allow better handling of all exceptions, including
SystemExit and KeyboardInterrupt
* asyncio: __del__() keep reference to warnings.warn
The __del__() methods of asyncio classes now keep a strong reference
to the warnings.warn() to be able to display the ResourceWarning
warning in more cases. Ensure that the function remains available if
instances are destroyed late during Python shutdown (while module
symbols are cleared).
* Rename warn parameter to _warn
"_warn" name is a hint that it's not the regular warnings.warn()
function.
Fix typo from commit 6370f345e1
Signed-off-by: Christian Heimes <christian@python.org>
<!--
Thanks for your contribution!
Please read this comment in its entirety. It's quite important.
# Pull Request title
It should be in the following format:
```
bpo-NNNN: Summary of the changes made
```
Where: bpo-NNNN refers to the issue number in the https://bugs.python.org.
Most PRs will require an issue number. Trivial changes, like fixing a typo, do not need an issue.
# Backport Pull Request title
If this is a backport PR (PR made against branches other than `master`),
please ensure that the PR title is in the following format:
```
[X.Y] <title from the original PR> (GH-NNNN)
```
Where: [X.Y] is the branch name, e.g. [3.6].
GH-NNNN refers to the PR number from `master`.
-->
<!-- issue-number: bpo-32262 -->
https://bugs.python.org/issue32262
<!-- /issue-number -->
* Convert asyncio/tasks.py to async/await
* Convert asyncio/queues.py to async/await
* Convert asyncio/test_utils.py to async/await
* Convert asyncio/base_subprocess.py to async/await
* Convert asyncio/subprocess.py to async/await
* Convert asyncio/streams.py to async/await
* Fix comments
* Convert asyncio/locks.py to async/await
* Convert asyncio.sleep to async def
* Add a comment
* Add missing news
* Convert stubs from AbstrctEventLoop to async functions
* Convert subprocess_shell/subprocess_exec
* Convert connect_read_pipe/connect_write_pip to async/await syntax
* Convert create_datagram_endpoint
* Convert create_unix_server/create_unix_connection
* Get rid of old style coroutines in unix_events.py
* Convert selector_events.py to async/await
* Convert wait_closed and create_connection
* Drop redundant line
* Convert base_events.py
* Code cleanup
* Drop redundant comments
* Fix indentation
* Add explicit tests for compatibility between old and new coroutines
* Convert windows event loop to use async/await
* Fix double awaiting of async function
* Convert asyncio/locks.py
* Improve docstring
* Convert tests to async/await
* Convert more tests
* Convert more tests
* Convert more tests
* Convert tests
* Improve test
Issue #26741: asyncio: BaseSubprocessTransport._process_exited() now copies the
return code from the child watched to the returncode attribute of the Popen
object. On Python 3.6, it is required to avoid a ResourceWarning.
Issue #26604:
* Add a new optional source parameter to _warnings.warn() and warnings.warn()
* Modify asyncore, asyncio and _pyio modules to set the source parameter when
logging a ResourceWarning warning
Issue #24763: Fix resource warnings when asyncio.BaseSubprocessTransport
constructor fails, if subprocess.Popen raises an exception for example.
Patch written by Martin Richard, test written by me.
transport was closed. The check broken a Tulip example and this limitation is
arbitrary. Check if _proc is None should be enough.
Enhance also close(): do nothing when called the second time.
Issue #23347: send_signal(), kill() and terminate() methods of
BaseSubprocessTransport now check if the transport was closed and if the
process exited.
Issue #23347: Refactor creation of subprocess transports. Changes on
BaseSubprocessTransport:
* Add a wait() method to wait until the child process exit
* The constructor now accepts an optional waiter parameter. The _post_init()
coroutine must not be called explicitly anymore. It makes subprocess
transports closer to other transports, and it gives more freedom if we want
later to change completly how subprocess transports are created.
* close() now kills the process instead of kindly terminate it: the child
process may ignore SIGTERM and continue to run. Call explicitly terminate()
and wait() if you want to kindly terminate the child process.
* close() now logs a warning in debug mode if the process is still running and
needs to be killed
* _make_subprocess_transport() is now fully asynchronous again: if the creation
of the transport failed, wait asynchronously for the process eixt. Before the
wait was synchronous. This change requires close() to *kill*, and not
terminate, the child process.
* Remove the _kill_wait() method, replaced with a more agressive close()
method. It fixes _make_subprocess_transport() on error.
BaseSubprocessTransport.close() calls the close() method of pipe transports,
whereas _kill_wait() closed directly pipes of the subprocess.Popen object
without unregistering file descriptors from the selector (which caused severe
bugs).
These changes simplifies the code of subprocess.py.
* PipeHandle now uses None instead of -1 for a closed handle
* Sort imports in windows_utils.
* Fix test_events on Python older than 3.5. Skip SSL tests on the
ProactorEventLoop if ssl.MemoryIO is missing
* Fix BaseEventLoop._create_connection_transport(). Close the transport if the
creation of the transport (if the waiter) gets an exception.
* _ProactorBasePipeTransport now sets _sock to None when the transport is
closed.
* Fix BaseSubprocessTransport.close(). Ignore pipes for which the protocol is
not set yet (still equal to None).
* TestLoop.close() now calls the close() method of the parent class
(BaseEventLoop).
* Cleanup BaseSelectorEventLoop: create the protocol on a separated line for
readability and ease debugging.
* Fix BaseSubprocessTransport._kill_wait(). Set the _returncode attribute, so
close() doesn't try to terminate the process.
* Tests: explicitly close event loops and transports
* UNIX pipe transports: add closed/closing in repr(). Add "closed" or "closing"
state in the __repr__() method of _UnixReadPipeTransport and
_UnixWritePipeTransport classes.
* If an exception is raised during the creation of a subprocess, kill the
subprocess (close pipes, kill and read the return status). Log an error in
such case.
* Fix SubprocessStreamProtocol.connection_made() to handle cancelled waiter.
Add unit test cancelling subprocess methods.
* Fix a race condition in BaseSubprocessTransport._try_finish().
If the process exited before the _post_init() method was called, scheduling
the call to _call_connection_lost() with call_soon() is wrong:
connection_made() must be called before connection_lost().
Reuse the BaseSubprocessTransport._call() method to schedule the call to
_call_connection_lost() to ensure that connection_made() and
connection_lost() are called in the correct order.
* Add repr(PipeHandle)
* Fix typo
* Tulip issue #184: Log subprocess events in debug mode
- Log stdin, stdout and stderr transports and protocols
- Log process identifier (pid)
- Log connection of pipes
- Log process exit
- Log Process.communicate() tasks: feed stdin, read stdout and stderr
- Add __repr__() method to many classes related to subprocesses
* Add BaseSubprocessTransport._pid attribute. Store the pid so it is still
accessible after the process exited. It's more convinient for debug.
* create_connection(): add the socket in the "connected to" debug log
* Clean up some docstrings and comments. Remove unused unimplemented
_read_from_self().
* Add a new asyncio.subprocess module
* Add new create_subprocess_exec() and create_subprocess_shell() functions
* The new asyncio.subprocess.SubprocessStreamProtocol creates stream readers
for stdout and stderr and a stream writer for stdin.
* The new asyncio.subprocess.Process class offers an API close to the
subprocess.Popen class:
- pid, returncode, stdin, stdout and stderr attributes
- communicate(), wait(), send_signal(), terminate() and kill() methods
* Remove STDIN (0), STDOUT (1) and STDERR (2) constants from base_subprocess
and unix_events, to not be confused with the symbols with the same name of
subprocess and asyncio.subprocess modules
* _ProactorBasePipeTransport.get_write_buffer_size() now counts also the size
of the pending write
* _ProactorBaseWritePipeTransport._loop_writing() may now pause the protocol if
the write buffer size is greater than the high water mark (64 KB by default)
Kumar Aditya
Allison Karlitskaya
Kumar Aditya
Yury Selivanov
Victor Stinner
Christian Heimes
Andrew Svetlov
INADA Naoki
Guido van Rossum
Victor Stinner
Yury Selivanov
Guido van Rossum