Issue #23347: send_signal(), kill() and terminate() methods of
BaseSubprocessTransport now check if the transport was closed and if the
process exited.
Issue #23347: Refactor creation of subprocess transports. Changes on
BaseSubprocessTransport:
* Add a wait() method to wait until the child process exit
* The constructor now accepts an optional waiter parameter. The _post_init()
coroutine must not be called explicitly anymore. It makes subprocess
transports closer to other transports, and it gives more freedom if we want
later to change completly how subprocess transports are created.
* close() now kills the process instead of kindly terminate it: the child
process may ignore SIGTERM and continue to run. Call explicitly terminate()
and wait() if you want to kindly terminate the child process.
* close() now logs a warning in debug mode if the process is still running and
needs to be killed
* _make_subprocess_transport() is now fully asynchronous again: if the creation
of the transport failed, wait asynchronously for the process eixt. Before the
wait was synchronous. This change requires close() to *kill*, and not
terminate, the child process.
* Remove the _kill_wait() method, replaced with a more agressive close()
method. It fixes _make_subprocess_transport() on error.
BaseSubprocessTransport.close() calls the close() method of pipe transports,
whereas _kill_wait() closed directly pipes of the subprocess.Popen object
without unregistering file descriptors from the selector (which caused severe
bugs).
These changes simplifies the code of subprocess.py.
* Cleanup gather(): use cancelled() method instead of using private Future
attribute
* Fix _UnixReadPipeTransport and _UnixWritePipeTransport. Only start reading
when connection_made() has been called.
* Issue #23333: Fix BaseSelectorEventLoop._accept_connection(). Close the
transport on error. In debug mode, log errors using call_exception_handler()
* Rephrase also the comment explaining why the waiter is not awaken immediatly.
* SSLProtocol.eof_received() doesn't instanciate ConnectionResetError exception
directly, it will be done by Future.set_exception(). The exception is not
used if the waiter was cancelled or if there is no waiter.
* PipeHandle now uses None instead of -1 for a closed handle
* Sort imports in windows_utils.
* Fix test_events on Python older than 3.5. Skip SSL tests on the
ProactorEventLoop if ssl.MemoryIO is missing
* Fix BaseEventLoop._create_connection_transport(). Close the transport if the
creation of the transport (if the waiter) gets an exception.
* _ProactorBasePipeTransport now sets _sock to None when the transport is
closed.
* Fix BaseSubprocessTransport.close(). Ignore pipes for which the protocol is
not set yet (still equal to None).
* TestLoop.close() now calls the close() method of the parent class
(BaseEventLoop).
* Cleanup BaseSelectorEventLoop: create the protocol on a separated line for
readability and ease debugging.
* Fix BaseSubprocessTransport._kill_wait(). Set the _returncode attribute, so
close() doesn't try to terminate the process.
* Tests: explicitly close event loops and transports
* UNIX pipe transports: add closed/closing in repr(). Add "closed" or "closing"
state in the __repr__() method of _UnixReadPipeTransport and
_UnixWritePipeTransport classes.
* Tulip issue 184: FlowControlMixin constructor now get the event loop if the
loop parameter is not set. Add unit tests to ensure that constructor of
StreamReader and StreamReaderProtocol classes get the event loop.
* Remove outdated TODO/XXX
written by Torsten Landschoff.
create_task(), call_at(), call_soon(), call_soon_threadsafe() and
run_in_executor() now raise an error if the event loop is closed.
functions:
* add_signal_handler()
* call_at()
* call_later()
* call_soon()
* call_soon_threadsafe()
* run_in_executor()
Fix also the error message of add_signal_handler() (fix the name of the
function).
Move the _loop attribute from the constructor of _SelectorTransport,
_ProactorBasePipeTransport and _UnixWritePipeTransport classes to the
constructor of the _FlowControlMixin class.
Add also an assertion to explicit that the parent class must ensure that the
loop is defined (not None)
* PipeServer.close() now cancels the "accept pipe" future which cancels the
overlapped operation.
* Fix _SelectorTransport.__repr__() if the transport was closed
* Fix debug log in BaseEventLoop.create_connection(): get the socket object
from the transport because SSL transport closes the old socket and creates a
new SSL socket object. Remove also the _SelectorSslTransport._rawsock
attribute: it contained the closed socket (not very useful) and it was not
used.
* Issue #22063: socket operations (sock_recv, sock_sendall, sock_connect,
sock_accept) of the proactor event loop don't raise an exception in debug
mode if the socket are in blocking mode. Overlapped operations also work on
blocking sockets.
* Fix unit tests in debug mode: mock a non-blocking socket for socket
operations which now raise an exception if the socket is blocking.
* _fatal_error() method of _UnixReadPipeTransport and _UnixWritePipeTransport
now log all exceptions in debug mode
* Don't log expected errors in unit tests
* Tulip issue 200: _WaitHandleFuture._unregister_wait() now catchs and logs
exceptions.
* Tulip issue 200: Log errors in debug mode instead of simply ignoring them.
Since Python 3.3, the C signal handler writes the signal number into the wakeup
file descriptor and then schedules the Python call using Py_AddPendingCall().
asyncio uses the wakeup file descriptor to wake up the event loop, and relies
on Py_AddPendingCall() to schedule the final callback with call_soon().
If the C signal handler is called in a thread different than the thread of the
event loop, the loop is awaken but Py_AddPendingCall() was not called yet. In
this case, the event loop has nothing to do and go to sleep again.
Py_AddPendingCall() is called while the event loop is sleeping again and so the
final callback is not scheduled immediatly.
This patch changes how asyncio handles signals. Instead of relying on
Py_AddPendingCall() and the wakeup file descriptor, asyncio now only relies on
the wakeup file descriptor. asyncio reads signal numbers from the wakeup file
descriptor to call its signal handler.
* Tulip issue #184: Log subprocess events in debug mode
- Log stdin, stdout and stderr transports and protocols
- Log process identifier (pid)
- Log connection of pipes
- Log process exit
- Log Process.communicate() tasks: feed stdin, read stdout and stderr
- Add __repr__() method to many classes related to subprocesses
* Add BaseSubprocessTransport._pid attribute. Store the pid so it is still
accessible after the process exited. It's more convinient for debug.
* create_connection(): add the socket in the "connected to" debug log
* Clean up some docstrings and comments. Remove unused unimplemented
_read_from_self().
* Tulip issue #183: log socket events in debug mode
- Log most important socket events: socket connected, new client, connection
reset or closed by peer (EOF), etc.
- Log time elapsed in DNS resolution (getaddrinfo)
- Log pause/resume reading
- Log time of SSL handshake
- Log SSL handshake errors
- Add a __repr__() method to many classes
* Fix ProactorEventLoop() in debug mode. ProactorEventLoop._make_self_pipe()
doesn't call call_soon() directly because it checks for the current loop
which fails, because the method is called to build the event loop.
* Cleanup _ProactorReadPipeTransport constructor. Not need to set again
_read_fut attribute to None, it is already done in the base class.
* _UnixSubprocessTransport: fix file mode of stdin. Open stdin in write mode,
not in read mode
* Examples: close the event loop at exit
* More reliable CoroWrapper.__del__. If the constructor is interrupted by
KeyboardInterrupt or the coroutine objet is destroyed lately, some the
_source_traceback attribute doesn't exist anymore.
* repr(Task): include also the future the task is waiting for
Fix ResourceWarning: create_connection(), create_datagram_endpoint() and
create_unix_server() methods of event loop now close the newly created socket
on error.
* Add a new asyncio.subprocess module
* Add new create_subprocess_exec() and create_subprocess_shell() functions
* The new asyncio.subprocess.SubprocessStreamProtocol creates stream readers
for stdout and stderr and a stream writer for stdin.
* The new asyncio.subprocess.Process class offers an API close to the
subprocess.Popen class:
- pid, returncode, stdin, stdout and stderr attributes
- communicate(), wait(), send_signal(), terminate() and kill() methods
* Remove STDIN (0), STDOUT (1) and STDERR (2) constants from base_subprocess
and unix_events, to not be confused with the symbols with the same name of
subprocess and asyncio.subprocess modules
* _ProactorBasePipeTransport.get_write_buffer_size() now counts also the size
of the pending write
* _ProactorBaseWritePipeTransport._loop_writing() may now pause the protocol if
the write buffer size is greater than the high water mark (64 KB by default)
Major changes:
- StreamReader.readexactly() now raises an IncompleteReadError if the
end of stream is reached before we received enough bytes, instead of
returning less bytes than requested.
- Unit tests use the main asyncio module instead of submodules like events
- _UnixWritePipeTransport now also supports character devices, as
_UnixReadPipeTransport. Patch written by Jonathan Slenders.
- Export more symbols: BaseEventLoop, BaseProactorEventLoop,
BaseSelectorEventLoop, Queue and Queue sublasses, Empty, Full
Victor Stinner
Guido van Rossum
Yury Selivanov
Andrew Svetlov
Charles-François Natali