There is a race condition in create_connection() used with wait_for() to have a
timeout. sock_connect() registers the file descriptor of the socket to be
notified of write event (if connect() raises BlockingIOError). When
create_connection() is cancelled with a TimeoutError, sock_connect() coroutine
gets the exception, but it doesn't unregister the file descriptor for write
event. create_connection() gets the TimeoutError and closes the socket.
If you call again create_connection(), the new socket will likely gets the same
file descriptor, which is still registered in the selector. When sock_connect()
calls add_writer(), it tries to modify the entry instead of creating a new one.
This issue was originally reported in the Trollius project, but the bug comes
from Tulip in fact (Trollius is based on Tulip):
https://bitbucket.org/enovance/trollius/issue/15/after-timeouterror-on-wait_for
This change fixes the race condition. It also makes sock_connect() more
reliable (and portable) is sock.connect() raises an InterruptedError.
Don't raise a TimeoutError if we reached the timeout and the future completed
in the same iteration of the event loop. A side effect of the bug is that
Queue.get() looses items.
* PipeServer.close() now cancels the "accept pipe" future which cancels the
overlapped operation.
* Fix _SelectorTransport.__repr__() if the transport was closed
* Fix debug log in BaseEventLoop.create_connection(): get the socket object
from the transport because SSL transport closes the old socket and creates a
new SSL socket object. Remove also the _SelectorSslTransport._rawsock
attribute: it contained the closed socket (not very useful) and it was not
used.
* Issue #22063: socket operations (sock_recv, sock_sendall, sock_connect,
sock_accept) of the proactor event loop don't raise an exception in debug
mode if the socket are in blocking mode. Overlapped operations also work on
blocking sockets.
* Fix unit tests in debug mode: mock a non-blocking socket for socket
operations which now raise an exception if the socket is blocking.
* _fatal_error() method of _UnixReadPipeTransport and _UnixWritePipeTransport
now log all exceptions in debug mode
* Don't log expected errors in unit tests
* Tulip issue 200: _WaitHandleFuture._unregister_wait() now catchs and logs
exceptions.
* Tulip issue 200: Log errors in debug mode instead of simply ignoring them.
* _WaitHandleFuture.cancel() now notify IocpProactor through the overlapped
object that the wait was cancelled.
* Optimize IocpProactor.wait_for_handle() gets the result if the wait is
signaled immediatly.
* Enhance representation of Future and Future subclasses
- Add "created at filename:lineno" in the representation
- Add Future._repr_info() method which can be more easily overriden than
Future.__repr__(). It should now be more easy to enhance Future
representation without having to modify each subclass. For example,
_OverlappedFuture and _WaitHandleFuture get the new "created at" information.
- Use reprlib to format Future result, and function arguments when formatting a
callback, to limit the length of the representation.
* Fix repr(_WaitHandleFuture)
* _WaitHandleFuture and _OverlappedFuture: hide frames of internal calls in the
source traceback.
* Cleanup ProactorIocp._poll(): set the timeout to 0 after the first call to
GetQueuedCompletionStatus()
* test_locks: close the temporary event loop and check the condition lock
* Remove workaround in test_futures, no more needed
in the _cache dictionary, even if we already got the result. We need to keep a
reference to the overlapped object, otherwise the memory may be reused and
GetQueuedCompletionStatus() may use random bytes and behaves badly.
There is still a hack for ConnectNamedPipe(): the overlapped object is not
register into _cache if the overlapped object completed directly.
Log also an error in debug mode in ProactorIocp._loop() if we get an unexpected
event.
Add a protection in ProactorIocp.close() to avoid blocking, even if it should
not happen. I still don't understand exactly why some the completion of some
overlapped objects are not notified.
* Tulip issue #196: IocpProactor._poll() clears the reference to the
overlapped operation when the operation is done. It would be better to clear
the reference in a new _OverlappedFuture.set_result() method, but it cannot
be done yet because of a weird bug.
* BaseSelectorEventLoop._write_to_self() now logs errors in debug mode.
* Fix _WaitHandleFuture.cancel(): return the result of the parent cancel()
method.
* _OverlappedFuture.cancel() now clears its reference to the overlapped object.
Make also the _OverlappedFuture.ov attribute private.
* Check if _WaitHandleFuture completed before unregistering it in the callback.
Add also _WaitHandleFuture._poll() and repr(_WaitHandleFuture).
* _WaitHandleFuture now unregisters its wait handler if WaitForSingleObject()
raises an exception.
* _OverlappedFuture.set_exception() now cancels the overlapped operation.
Improve stability of the proactor event loop, especially operations on
overlapped objects:
* Tulip issue 195: Don't call UnregisterWait() twice if a _WaitHandleFuture is
cancelled twice to fix a crash.
* IocpProactor.close(): cancel futures to cancel overlapped operations, instead
of cancelling directly overlapped operations. Future objects may not call
ov.cancel() if the future was cancelled or if the overlapped was already
cancelled. The cancel() method of the future may also catch exceptions. Log
also errors on cancellation.
* tests: rename "f" to "fut"
* Add a __repr__() method to IocpProactor
* Add a destructor to IocpProactor which closes it
* _OverlappedFuture.cancel() doesn't cancel the overlapped anymore if it is
done: if it is already cancelled or completed. Log also an error if the
cancellation failed.
* Add the address of the overlapped object in repr(_OverlappedFuture)
* _OverlappedFuture truncates the source traceback to hide the call to the
parent constructor (useless in debug).
Since Python 3.3, the C signal handler writes the signal number into the wakeup
file descriptor and then schedules the Python call using Py_AddPendingCall().
asyncio uses the wakeup file descriptor to wake up the event loop, and relies
on Py_AddPendingCall() to schedule the final callback with call_soon().
If the C signal handler is called in a thread different than the thread of the
event loop, the loop is awaken but Py_AddPendingCall() was not called yet. In
this case, the event loop has nothing to do and go to sleep again.
Py_AddPendingCall() is called while the event loop is sleeping again and so the
final callback is not scheduled immediatly.
This patch changes how asyncio handles signals. Instead of relying on
Py_AddPendingCall() and the wakeup file descriptor, asyncio now only relies on
the wakeup file descriptor. asyncio reads signal numbers from the wakeup file
descriptor to call its signal handler.
If you want to handle the BrokenPipeError, you can easily reimplement
communicate().
Add also a unit test to ensure that stdin.write() + stdin.drain() raises
BrokenPipeError.
* Tulip issue #184: Log subprocess events in debug mode
- Log stdin, stdout and stderr transports and protocols
- Log process identifier (pid)
- Log connection of pipes
- Log process exit
- Log Process.communicate() tasks: feed stdin, read stdout and stderr
- Add __repr__() method to many classes related to subprocesses
* Add BaseSubprocessTransport._pid attribute. Store the pid so it is still
accessible after the process exited. It's more convinient for debug.
* create_connection(): add the socket in the "connected to" debug log
* Clean up some docstrings and comments. Remove unused unimplemented
_read_from_self().
* Tulip issue #183: log socket events in debug mode
- Log most important socket events: socket connected, new client, connection
reset or closed by peer (EOF), etc.
- Log time elapsed in DNS resolution (getaddrinfo)
- Log pause/resume reading
- Log time of SSL handshake
- Log SSL handshake errors
- Add a __repr__() method to many classes
* Fix ProactorEventLoop() in debug mode. ProactorEventLoop._make_self_pipe()
doesn't call call_soon() directly because it checks for the current loop
which fails, because the method is called to build the event loop.
* Cleanup _ProactorReadPipeTransport constructor. Not need to set again
_read_fut attribute to None, it is already done in the base class.
- loop, waiters and active_count attributes are now private
- attach(), detach() and wakeup() methods are now private
The sockets attribute remains public.
* Tulip issue #182: Improve logs of BaseEventLoop._run_once()
- Don't log non-blocking poll
- Only log polling with a timeout if it gets events or if it timed out after
more than 1 second.
* Fix some pyflakes warnings: remove unused imports
- CoroWrapper.__del__() now reuses repr(CoroWrapper) to log the "... was never
yielded from" warning
- Improve CoroWrapper: copy also the qualified name on Python 3.4, not only on
Python 3.5+
- repr(Task) and repr(CoroWrapper) now also includes where these objects were
created. If the coroutine is not a generator (don't use "yield from"), use
the location of the function, not the location of the coro() wrapper.
- Fix create_task(): truncate the traceback to hide the call to create_task().
- Issues #21936, #21163: Fix sporadic failures of
test_future_exception_never_retrieved()
- Handle.cancel() now clears references to callback and args
- In debug mode, repr(Handle) now contains the location where the Handle was
created.
- Tulip issue 185: Add a create_task() method to event loops. The create_task()
method can be overriden in custom event loop to implement their own task
class. For example, greenio and Pulsar projects use their own task class. The
create_task() method is now preferred over creating directly task using the
Task class.
- tests: fix a warning
- fix typo in the name of a test function
- Update AbstractEventLoop: add new event loop methods; update also the unit test
- Tulip issue #181: Faster create_connection(). Call directly
waiter.set_result() in the constructor of _ProactorBasePipeTransport and
_SelectorSocketTransport, instead of using of delaying the call with
call_soon().
- Cleanup iscoroutine()
* _UnixSubprocessTransport: fix file mode of stdin. Open stdin in write mode,
not in read mode
* Examples: close the event loop at exit
* More reliable CoroWrapper.__del__. If the constructor is interrupted by
KeyboardInterrupt or the coroutine objet is destroyed lately, some the
_source_traceback attribute doesn't exist anymore.
* repr(Task): include also the future the task is waiting for
- Sort imports
- Simplify/optimize iscoroutine(). Inline inspect.isgenerator(obj): replace it
with isinstance(obj, types.GeneratorType)
- CoroWrapper: check at runtime if Python has the yield-from bug #21209. If
Python has the bug, check if CoroWrapper.send() was called by yield-from to
decide if parameters must be unpacked or not.
- Fix "Task was destroyed but it is pending!" warning in
test_task_source_traceback()
Handle objects are created. Pass the traceback to call_exception_handler() in
the 'source_traceback' key.
The traceback is truncated to hide internal calls in asyncio, show only the
traceback from user code.
Add tests for the new source_traceback, and a test for the 'Future/Task
exception was never retrieved' log.
done: use the first line number of the code object instead of the current line
number of the generator frame.
The name of the coroutine is not enough because many coroutines may have the
same name. It's a common case in asyncio tests for example.
exception if the current loop is not None.
Guido van Rossum wrote:
"The behavior that you can set the loop to None (and keep track of it
explicitly) is part of the spec, and this should still be supported even in
debug mode. The behavior that we raise an error if you are caught having
multiple active loops per thread is just a debugging heuristic, and it
shouldn't break code that follows the spec."
available bytes from the "self pipe", not only a single byte. This change
reduces the risk of having the pipe full and so getting the innocuous
"BlockingIOError: [Errno 11] Resource temporarily unavailable" message.
Python 3.5
- Drop __slots__ optimization of CoroWrapper to be able to set the __qualname__
attribute.
- Add tests on __name__, __qualname__ and __module__ of a coroutine function
and coroutine object.
- Fix test_tasks when run in debug mode (PYTHONASYNCIODEBUG env var set) on
Python 3.3 or 3.4
repr(Handle) is shorter for function: "foo" instead of "<function foo at
0x...>". It now also includes the source of the callback, filename and line
number where it was defined, if available.
repr(Task) now also includes the current position in the code, filename and
line number, if available. If the coroutine (generator) is done, the line
number is omitted and "done" is added.
Add BaseEventLoop._closed attribute and use it to check if the event loop was
closed or not, instead of checking different attributes in each subclass of
BaseEventLoop.
run_forever() and run_until_complete() methods now raise a RuntimeError('Event loop is
closed') exception if the event loop was closed.
BaseProactorEventLoop.close() now also cancels "accept futures".
Fix ResourceWarning: create_connection(), create_datagram_endpoint() and
create_unix_server() methods of event loop now close the newly created socket
on error.
loop in debug mode. Raise a RuntimeError if the event loop of the current
thread is different. The check should help to debug thread-safetly issue.
Patch written by David Foster.
* Issue #159: Fix windows_utils.socketpair()
- Use "127.0.0.1" (IPv4) or "::1" (IPv6) host instead of "localhost", because
"localhost" may be a different IP address
- Reject also invalid arguments: only AF_INET/AF_INET6 with SOCK_STREAM (and
proto=0) are supported
* Reject add/remove reader/writer when event loop is closed.
* Fix ResourceWarning warnings
Add also a PYTHONASYNCIODEBUG environment variable to debug coroutines since
Python startup, to be able to debug coroutines defined directly in the asyncio
module.
* Add a new asyncio.subprocess module
* Add new create_subprocess_exec() and create_subprocess_shell() functions
* The new asyncio.subprocess.SubprocessStreamProtocol creates stream readers
for stdout and stderr and a stream writer for stdin.
* The new asyncio.subprocess.Process class offers an API close to the
subprocess.Popen class:
- pid, returncode, stdin, stdout and stderr attributes
- communicate(), wait(), send_signal(), terminate() and kill() methods
* Remove STDIN (0), STDOUT (1) and STDERR (2) constants from base_subprocess
and unix_events, to not be confused with the symbols with the same name of
subprocess and asyncio.subprocess modules
* _ProactorBasePipeTransport.get_write_buffer_size() now counts also the size
of the pending write
* _ProactorBaseWritePipeTransport._loop_writing() may now pause the protocol if
the write buffer size is greater than the high water mark (64 KB by default)
Windows) instead of using the "duplex" pipe transport. The new class uses a
simpler overlapped read to be notified when the pipe is closed. So the protocol
doesn't need to implement eof_received(): connection_lost() is called instead.
_UnixWritePipeTransport has the same approach.
Major changes:
- StreamReader.readexactly() now raises an IncompleteReadError if the
end of stream is reached before we received enough bytes, instead of
returning less bytes than requested.
- Unit tests use the main asyncio module instead of submodules like events
- _UnixWritePipeTransport now also supports character devices, as
_UnixReadPipeTransport. Patch written by Jonathan Slenders.
- Export more symbols: BaseEventLoop, BaseProactorEventLoop,
BaseSelectorEventLoop, Queue and Queue sublasses, Empty, Full
between the resolution of the BaseEventLoop.time() method and the resolution of
the selector. The granuarility is used in the scheduler to round time and
deadline.
logger.log() is now responsible to format the timeout. It might be faster if
the log is disabled for DEBUG level, but it's also more readable and fix
an issue with Python 2.6 in the Trollius project.
Logger.log() is "slow", logger.isEnabledFor() is faster and the logger is
disabled in most cases. A microbenchmark executing 100,000 dummy tasks is 22%
faster with this change.
* store the "self reading" future when the "self pipe" is closed (when the
event loop is closed)
* store "accept" futures to cancel them when we stop serving
* close the "accept socket" if the "accept future" is cancelled
Fix many warnings which can be seen when unit tests are run in verbose mode.
Victor Stinner
Yury Selivanov
R David Murray
Serhiy Storchaka
Andrew Svetlov
Guido van Rossum
Christian Heimes
Charles-François Natali