Issue #23347: send_signal(), kill() and terminate() methods of
BaseSubprocessTransport now check if the transport was closed and if the
process exited.
Issue #23347: Refactor creation of subprocess transports. Changes on
BaseSubprocessTransport:
* Add a wait() method to wait until the child process exit
* The constructor now accepts an optional waiter parameter. The _post_init()
coroutine must not be called explicitly anymore. It makes subprocess
transports closer to other transports, and it gives more freedom if we want
later to change completly how subprocess transports are created.
* close() now kills the process instead of kindly terminate it: the child
process may ignore SIGTERM and continue to run. Call explicitly terminate()
and wait() if you want to kindly terminate the child process.
* close() now logs a warning in debug mode if the process is still running and
needs to be killed
* _make_subprocess_transport() is now fully asynchronous again: if the creation
of the transport failed, wait asynchronously for the process eixt. Before the
wait was synchronous. This change requires close() to *kill*, and not
terminate, the child process.
* Remove the _kill_wait() method, replaced with a more agressive close()
method. It fixes _make_subprocess_transport() on error.
BaseSubprocessTransport.close() calls the close() method of pipe transports,
whereas _kill_wait() closed directly pipes of the subprocess.Popen object
without unregistering file descriptors from the selector (which caused severe
bugs).
These changes simplifies the code of subprocess.py.
If ReadFile() fails with ERROR_BROKEN_PIPE, the operation is not pending: don't
register the overlapped.
I don't know if WSARecv() can fail with ERROR_BROKEN_PIPE. Since
Overlapped.WSARecv() already handled ERROR_BROKEN_PIPE, let me guess that it
has the same behaviour than ReadFile().
If UnregisterWaitEx() fais with ERROR_IO_PENDING, it doesn't mean that the wait
is unregistered yet. We still have to wait until the wait is cancelled.
Use a coroutine with asyncio.sleep() instead of call_later() to ensure that the
schedule call is cancelled.
Add also a unit test cancelling connect_pipe().
Overlapped.ConnectNamedPipe() now returns a boolean: True if the pipe is
connected (if ConnectNamedPipe() failed with ERROR_PIPE_CONNECTED), False if
the connection is in progress.
This change removes multiple hacks in IocpProactor.
Add _overlapped.ConnectPipe() which tries to connect to the pipe for
asynchronous I/O (overlapped): call CreateFile() in a loop until it doesn't
fail with ERROR_PIPE_BUSY. Use an increasing delay between 1 ms and 100 ms.
Remove Overlapped.WaitNamedPipeAndConnect() which is no more used.
This change fixes a race conditon related to _WaitHandleFuture.cancel() leading
to Python crash or "GetQueuedCompletionStatus() returned an unexpected event"
logs. Before, the overlapped object was destroyed too early, it was possible
that the wait completed whereas the overlapped object was already destroyed.
Sometimes, a different overlapped was allocated at the same address, leading to
unexpected completition.
_WaitHandleFuture.cancel() now waits until the wait is cancelled to clear its
reference to the overlapped object. To wait until the cancellation is done,
UnregisterWaitEx() is used with an event instead of UnregisterWait().
To wait for this event, a new _WaitCancelFuture class was added. It's a
simplified version of _WaitCancelFuture. For example, its cancel() method calls
UnregisterWait(), not UnregisterWaitEx(). _WaitCancelFuture should not be
cancelled.
The overlapped object is kept alive in _WaitHandleFuture until the wait is
unregistered.
Other changes:
* Add _overlapped.UnregisterWaitEx()
* Remove fast-path in IocpProactor.wait_for_handle() to immediatly set the
result if the wait already completed. I'm not sure that it's safe to
call immediatly UnregisterWaitEx() before the completion was signaled.
* Add IocpProactor._unregistered() to forget an overlapped which may never be
signaled, but may be signaled for the next loop iteration. It avoids to
block forever IocpProactor.close() if a wait was cancelled, and it may also
avoid some "... unexpected event ..." warnings.
* Fix a race condition in BaseSubprocessTransport._try_finish().
If the process exited before the _post_init() method was called, scheduling
the call to _call_connection_lost() with call_soon() is wrong:
connection_made() must be called before connection_lost().
Reuse the BaseSubprocessTransport._call() method to schedule the call to
_call_connection_lost() to ensure that connection_made() and
connection_lost() are called in the correct order.
* Add repr(PipeHandle)
* Fix typo
* PipeServer.close() now cancels the "accept pipe" future which cancels the
overlapped operation.
* Fix _SelectorTransport.__repr__() if the transport was closed
* Fix debug log in BaseEventLoop.create_connection(): get the socket object
from the transport because SSL transport closes the old socket and creates a
new SSL socket object. Remove also the _SelectorSslTransport._rawsock
attribute: it contained the closed socket (not very useful) and it was not
used.
* Issue #22063: socket operations (sock_recv, sock_sendall, sock_connect,
sock_accept) of the proactor event loop don't raise an exception in debug
mode if the socket are in blocking mode. Overlapped operations also work on
blocking sockets.
* Fix unit tests in debug mode: mock a non-blocking socket for socket
operations which now raise an exception if the socket is blocking.
* _fatal_error() method of _UnixReadPipeTransport and _UnixWritePipeTransport
now log all exceptions in debug mode
* Don't log expected errors in unit tests
* Tulip issue 200: _WaitHandleFuture._unregister_wait() now catchs and logs
exceptions.
* Tulip issue 200: Log errors in debug mode instead of simply ignoring them.
* _WaitHandleFuture.cancel() now notify IocpProactor through the overlapped
object that the wait was cancelled.
* Optimize IocpProactor.wait_for_handle() gets the result if the wait is
signaled immediatly.
* Enhance representation of Future and Future subclasses
- Add "created at filename:lineno" in the representation
- Add Future._repr_info() method which can be more easily overriden than
Future.__repr__(). It should now be more easy to enhance Future
representation without having to modify each subclass. For example,
_OverlappedFuture and _WaitHandleFuture get the new "created at" information.
- Use reprlib to format Future result, and function arguments when formatting a
callback, to limit the length of the representation.
* Fix repr(_WaitHandleFuture)
* _WaitHandleFuture and _OverlappedFuture: hide frames of internal calls in the
source traceback.
* Cleanup ProactorIocp._poll(): set the timeout to 0 after the first call to
GetQueuedCompletionStatus()
* test_locks: close the temporary event loop and check the condition lock
* Remove workaround in test_futures, no more needed
in the _cache dictionary, even if we already got the result. We need to keep a
reference to the overlapped object, otherwise the memory may be reused and
GetQueuedCompletionStatus() may use random bytes and behaves badly.
There is still a hack for ConnectNamedPipe(): the overlapped object is not
register into _cache if the overlapped object completed directly.
Log also an error in debug mode in ProactorIocp._loop() if we get an unexpected
event.
Add a protection in ProactorIocp.close() to avoid blocking, even if it should
not happen. I still don't understand exactly why some the completion of some
overlapped objects are not notified.
* Tulip issue #196: IocpProactor._poll() clears the reference to the
overlapped operation when the operation is done. It would be better to clear
the reference in a new _OverlappedFuture.set_result() method, but it cannot
be done yet because of a weird bug.
* BaseSelectorEventLoop._write_to_self() now logs errors in debug mode.
* Fix _WaitHandleFuture.cancel(): return the result of the parent cancel()
method.
* _OverlappedFuture.cancel() now clears its reference to the overlapped object.
Make also the _OverlappedFuture.ov attribute private.
* Check if _WaitHandleFuture completed before unregistering it in the callback.
Add also _WaitHandleFuture._poll() and repr(_WaitHandleFuture).
* _WaitHandleFuture now unregisters its wait handler if WaitForSingleObject()
raises an exception.
* _OverlappedFuture.set_exception() now cancels the overlapped operation.
Improve stability of the proactor event loop, especially operations on
overlapped objects:
* Tulip issue 195: Don't call UnregisterWait() twice if a _WaitHandleFuture is
cancelled twice to fix a crash.
* IocpProactor.close(): cancel futures to cancel overlapped operations, instead
of cancelling directly overlapped operations. Future objects may not call
ov.cancel() if the future was cancelled or if the overlapped was already
cancelled. The cancel() method of the future may also catch exceptions. Log
also errors on cancellation.
* tests: rename "f" to "fut"
* Add a __repr__() method to IocpProactor
* Add a destructor to IocpProactor which closes it
* _OverlappedFuture.cancel() doesn't cancel the overlapped anymore if it is
done: if it is already cancelled or completed. Log also an error if the
cancellation failed.
* Add the address of the overlapped object in repr(_OverlappedFuture)
* _OverlappedFuture truncates the source traceback to hide the call to the
parent constructor (useless in debug).
* Tulip issue #183: log socket events in debug mode
- Log most important socket events: socket connected, new client, connection
reset or closed by peer (EOF), etc.
- Log time elapsed in DNS resolution (getaddrinfo)
- Log pause/resume reading
- Log time of SSL handshake
- Log SSL handshake errors
- Add a __repr__() method to many classes
* Fix ProactorEventLoop() in debug mode. ProactorEventLoop._make_self_pipe()
doesn't call call_soon() directly because it checks for the current loop
which fails, because the method is called to build the event loop.
* Cleanup _ProactorReadPipeTransport constructor. Not need to set again
_read_fut attribute to None, it is already done in the base class.
* store the "self reading" future when the "self pipe" is closed (when the
event loop is closed)
* store "accept" futures to cancel them when we stop serving
* close the "accept socket" if the "accept future" is cancelled
Fix many warnings which can be seen when unit tests are run in verbose mode.
Yury Selivanov
Martin Panter
Victor Stinner
Guido van Rossum
Andrew Svetlov
Richard Oudkerk
Guido van Rossum