Commit Graph

69 Commits

Author SHA1 Message Date
Ben Darnell ea5a6363c3
bpo-39010: Fix errors logged on proactor loop restart (#22017)
Stopping and restarting a proactor event loop on windows can lead to
spurious errors logged (ConnectionResetError while reading from the
self pipe). This fixes the issue by ensuring that we don't attempt
to start multiple copies of the self-pipe reading loop.
2020-08-31 12:57:52 -07:00
Victor Stinner 602a971a2a
bpo-41467: Fix asyncio recv_into() on Windows (GH-21720)
On Windows, fix asyncio recv_into() return value when the socket/pipe
is closed (BrokenPipeError): return 0 rather than an empty byte
string (b'').
2020-08-04 02:40:10 +02:00
Chris Jerdonek 1ce5841eca
bpo-31033: Add a msg argument to Future.cancel() and Task.cancel() (GH-19979) 2020-05-15 16:55:50 -07:00
Andrew Svetlov bafd4b5ac8 bpo-29883: Asyncio proactor udp (GH-13440)
Follow-up for #1067


https://bugs.python.org/issue29883
2019-05-28 02:52:15 -07:00
Andrew Svetlov 23b4b697e5 bpo-36889: Merge asyncio streams (GH-13251)
https://bugs.python.org/issue36889
2019-05-27 12:56:22 -07:00
Yury Selivanov 431b540bf7
bpo-32528: Make asyncio.CancelledError a BaseException. (GH-13528)
This will address the common mistake many asyncio users make:
an "except Exception" clause breaking Tasks cancellation.

In addition to this change, we stop inheriting asyncio.TimeoutError
and asyncio.InvalidStateError from their concurrent.futures.*
counterparts.  There's no point for these exceptions to share the
inheritance chain.

In 3.9 we'll focus on implementing supervisors and cancel scopes,
which should allow better handling of all exceptions, including
SystemExit and KeyboardInterrupt
2019-05-27 14:45:12 +02:00
Victor Stinner b91140fdb1
bpo-11555: Enhance IocpProactor.close() log again (GH-11563)
Add repr(self) to the log to display the number of pending overlapped
in the log.
2019-01-15 12:13:48 +01:00
Victor Stinner b1e45739d8
bpo-34323: Enhance IocpProactor.close() log (GH-11555)
IocpProactor.close() now uses time to decide when to log: wait 1
second before the first log, then log every second. Log also the
number of seconds since close() was called.
2019-01-15 11:48:00 +01:00
Victor Stinner 9b07681c09
IocpProactor: prevent modification if closed (GH-11494)
* _wait_for_handle(), _register() and _unregister() methods of
  IocpProactor now raise an exception if closed
* Add "closed" to IocpProactor.__repr__()
* Simplify IocpProactor.close()
2019-01-10 11:23:26 +01:00
Vladimir Matveev 67ba547cf0 bpo-23057: Use 'raise' to emulate ctrl-c in proactor tests (#11274) 2019-01-05 22:44:59 +02:00
Vladimir Matveev b5c8cfa1da bpo-23057: add loop self socket as wakeup fd for signals (#11135) 2018-12-18 23:56:17 +02:00
Victor Stinner 6ea29c5e90 bpo-34687: Make asynico use ProactorEventLoop by default (GH-9538) 2018-09-25 11:27:08 -04:00
Andrew Svetlov 0baa72f4b2
bpo-34622: Extract asyncio exceptions into a separate module (GH-9141) 2018-09-11 10:13:04 -07:00
Yury Selivanov 8f4042964d
bpo-33792: Add selector and proactor windows policies (GH-7487) 2018-06-07 20:44:57 -04:00
Andrew Svetlov a19fb3c6aa
bpo-32622: Native sendfile on windows (#5565)
* Support sendfile on Windows Proactor event loop naively.
2018-02-25 19:32:14 +03:00
Andrew Svetlov 7c684073f9
bpo-32622: Implement loop.sendfile() (#5271) 2018-01-27 21:22:47 +02:00
Andrew Svetlov 7a6706bf48
Fix couple typos (#4839) 2017-12-13 17:50:16 +02:00
Yury Selivanov 6370f345e1
bpo-32262: Fix codestyle; use f-strings formatting where necessary. (#4775) 2017-12-10 18:36:12 -05:00
Andrew Svetlov 5f841b5538
bpo-32193: Convert asyncio to async/await usage (#4753)
* Convert asyncio/tasks.py to async/await

* Convert asyncio/queues.py to async/await

* Convert asyncio/test_utils.py to async/await

* Convert asyncio/base_subprocess.py to async/await

* Convert asyncio/subprocess.py to async/await

* Convert asyncio/streams.py to async/await

* Fix comments

* Convert asyncio/locks.py to async/await

* Convert asyncio.sleep to async def

* Add a comment

* Add missing news

* Convert stubs from AbstrctEventLoop to async functions

* Convert subprocess_shell/subprocess_exec

* Convert connect_read_pipe/connect_write_pip to async/await syntax

* Convert create_datagram_endpoint

* Convert create_unix_server/create_unix_connection

* Get rid of old style coroutines in unix_events.py

* Convert selector_events.py to async/await

* Convert wait_closed and create_connection

* Drop redundant line

* Convert base_events.py

* Code cleanup

* Drop redundant comments

* Fix indentation

* Add explicit tests for compatibility between old and new coroutines

* Convert windows event loop to use async/await

* Fix double awaiting of async function

* Convert asyncio/locks.py

* Improve docstring

* Convert tests to async/await

* Convert more tests

* Convert more tests

* Convert more tests

* Convert tests

* Improve test
2017-12-09 00:23:48 +02:00
Victor Stinner 4271dfd781
bpo-32154: Remove asyncio.selectors (#4605)
* Remove asyncio.selectors and asyncio._overlapped symbols from the
  namespace of the asyncio module
* Replace "from asyncio import selectors" with "import selectors"
* Replace "from asyncio import _overlapped" with "import _overlapped"

asyncio.selectors was added to support Python 3.3, which doesn't have
selectors in its standard library, and Python 3.4 in the same code
base. Same rationale for asyncio._overlapped. Python 3.3 reached its
end of life, and asyncio is no more maintained as a third party
module on PyPI.
2017-11-28 15:19:56 +01:00
Victor Stinner a10dc3efcb
asyncio: use directly socket.socketpair() (#4597)
Since Python 3.5, socket.socketpair() is also available on Windows,
and so can be used directly, rather than using
asyncio.windows_utils.socketpair().
2017-11-28 11:15:26 +01:00
Antoine Pitrou 525f40d231 bpo-31819: Add AbstractEventLoop.sock_recv_into() (#4051)
* bpo-31819: Add AbstractEventLoop.sock_recv_into()

* Add NEWS

* Add doc
2017-10-19 15:46:40 -04:00
INADA Naoki a83636247e Issue #28448: Fix C implemented asyncio.Future didn't work on Windows 2016-10-21 12:30:15 +09:00
Yury Selivanov 7661db6228 Issue #27041: asyncio: Add loop.create_future method 2016-05-16 15:38:39 -04:00
Martin Panter 6245cb3c01 Correct “an” → “a” with “Unicode”, “user”, “UTF”, etc
This affects documentation, code comments, and a debugging messages.
2016-04-15 02:14:19 +00:00
Yury Selivanov 59eb9a4da5 asyncio: async() function is deprecated in favour of ensure_future(). 2015-05-11 14:48:38 -04:00
Victor Stinner 5d44c08f1c Issue #23353, asyncio: Workaround CPython bug #23353
Don't use yield/yield-from in an except block of a generator. Store the
exception and handle it outside the except block.
2015-02-02 18:36:31 +01:00
Victor Stinner 1241ecc21b Issue #23347, asyncio: Make BaseSubprocessTransport.wait() private 2015-01-30 00:16:14 +01:00
Victor Stinner 47cd10d7a9 asyncio: sync with Tulip
Issue #23347: send_signal(), kill() and terminate() methods of
BaseSubprocessTransport now check if the transport was closed and if the
process exited.

Issue #23347: Refactor creation of subprocess transports. Changes on
BaseSubprocessTransport:

* Add a wait() method to wait until the child process exit
* The constructor now accepts an optional waiter parameter. The _post_init()
  coroutine must not be called explicitly anymore. It makes subprocess
  transports closer to other transports, and it gives more freedom if we want
  later to change completly how subprocess transports are created.
* close() now kills the process instead of kindly terminate it: the child
  process may ignore SIGTERM and continue to run. Call explicitly terminate()
  and wait() if you want to kindly terminate the child process.
* close() now logs a warning in debug mode if the process is still running and
  needs to be killed
* _make_subprocess_transport() is now fully asynchronous again: if the creation
  of the transport failed, wait asynchronously for the process eixt. Before the
  wait was synchronous. This change requires close() to *kill*, and not
  terminate, the child process.
* Remove the _kill_wait() method, replaced with a more agressive close()
  method. It fixes _make_subprocess_transport() on error.
  BaseSubprocessTransport.close() calls the close() method of pipe transports,
  whereas _kill_wait() closed directly pipes of the subprocess.Popen object
  without unregistering file descriptors from the selector (which caused severe
  bugs).

These changes simplifies the code of subprocess.py.
2015-01-30 00:05:19 +01:00
Victor Stinner 41063d2a59 asyncio, Tulip issue 204: Fix IocpProactor.recv()
If ReadFile() fails with ERROR_BROKEN_PIPE, the operation is not pending: don't
register the overlapped.

I don't know if WSARecv() can fail with ERROR_BROKEN_PIPE. Since
Overlapped.WSARecv() already handled ERROR_BROKEN_PIPE, let me guess that it
has the same behaviour than ReadFile().
2015-01-26 22:30:49 +01:00
Victor Stinner 24dfa3c1d6 Issue #23095, asyncio: Fix _WaitHandleFuture.cancel()
If UnregisterWaitEx() fais with ERROR_IO_PENDING, it doesn't mean that the wait
is unregistered yet. We still have to wait until the wait is cancelled.
2015-01-26 22:30:28 +01:00
Victor Stinner e0fd157ba0 Issue #23293, asyncio: Rewrite IocpProactor.connect_pipe() as a coroutine
Use a coroutine with asyncio.sleep() instead of call_later() to ensure that the
schedule call is cancelled.

Add also a unit test cancelling connect_pipe().
2015-01-26 15:04:03 +01:00
Victor Stinner a19b7b3fca asyncio: Fix ProactorEventLoop.start_serving_pipe()
If a client connected before the server was closed: drop the client (close the
pipe) and exit.
2015-01-26 15:03:20 +01:00
Victor Stinner 3d2256f671 Issue #23293, asyncio: Cleanup IocpProactor.close()
The special case for connect_pipe() is not more needed. connect_pipe() doesn't
use overlapped operations anymore.
2015-01-26 11:02:59 +01:00
Victor Stinner 2b77c5467f asyncio, Tulip issue 204: Fix IocpProactor.accept_pipe()
Overlapped.ConnectNamedPipe() now returns a boolean: True if the pipe is
connected (if ConnectNamedPipe() failed with ERROR_PIPE_CONNECTED), False if
the connection is in progress.

This change removes multiple hacks in IocpProactor.
2015-01-22 23:50:03 +01:00
Victor Stinner 7ffa2c5fdd Issue #23293, asyncio: Rewrite IocpProactor.connect_pipe()
Add _overlapped.ConnectPipe() which tries to connect to the pipe for
asynchronous I/O (overlapped): call CreateFile() in a loop until it doesn't
fail with ERROR_PIPE_BUSY. Use an increasing delay between 1 ms and 100 ms.

Remove Overlapped.WaitNamedPipeAndConnect() which is no more used.
2015-01-22 22:55:08 +01:00
Victor Stinner 752aba7f99 asyncio: IocpProactor.close() doesn't cancel anymore futures which are already
cancelled
2015-01-22 22:47:13 +01:00
Victor Stinner 1ca9392c70 Issue #23095, asyncio: IocpProactor.close() must not cancel pending
_WaitCancelFuture futures
2015-01-22 00:17:54 +01:00
Victor Stinner d0a28dee78 Issue #23095, asyncio: Rewrite _WaitHandleFuture.cancel()
This change fixes a race conditon related to _WaitHandleFuture.cancel() leading
to Python crash or "GetQueuedCompletionStatus() returned an unexpected event"
logs. Before, the overlapped object was destroyed too early, it was possible
that the wait completed whereas the overlapped object was already destroyed.
Sometimes, a different overlapped was allocated at the same address, leading to
unexpected completition.

_WaitHandleFuture.cancel() now waits until the wait is cancelled to clear its
reference to the overlapped object. To wait until the cancellation is done,
UnregisterWaitEx() is used with an event instead of UnregisterWait().

To wait for this event, a new _WaitCancelFuture class was added. It's a
simplified version of _WaitCancelFuture. For example, its cancel() method calls
UnregisterWait(), not UnregisterWaitEx(). _WaitCancelFuture should not be
cancelled.

The overlapped object is kept alive in _WaitHandleFuture until the wait is
unregistered.

Other changes:

* Add _overlapped.UnregisterWaitEx()
* Remove fast-path in IocpProactor.wait_for_handle() to immediatly set the
  result if the wait already completed. I'm not sure that it's safe to
  call immediatly UnregisterWaitEx() before the completion was signaled.
* Add IocpProactor._unregistered() to forget an overlapped which may never be
  signaled, but may be signaled for the next loop iteration. It avoids to
  block forever IocpProactor.close() if a wait was cancelled, and it may also
  avoid some "... unexpected event ..." warnings.
2015-01-21 23:39:51 +01:00
Victor Stinner 4bf22e033e asyncio: Close the transport on subprocess creation failure 2015-01-15 14:24:22 +01:00
Victor Stinner 15cc678d89 asyncio: Truncate to 80 columns 2015-01-09 00:09:10 +01:00
Victor Stinner 4d825b45a0 asyncio: IocpProactor.wait_for_handle() test now also checks the result of the
future
2014-12-19 17:10:44 +01:00
Victor Stinner 1b9763d0a9 asyncio: sync with Tulip
* Fix a race condition in BaseSubprocessTransport._try_finish().

  If the process exited before the _post_init() method was called, scheduling
  the call to _call_connection_lost() with call_soon() is wrong:
  connection_made() must be called before connection_lost().

  Reuse the BaseSubprocessTransport._call() method to schedule the call to
  _call_connection_lost() to ensure that connection_made() and
  connection_lost() are called in the correct order.

* Add repr(PipeHandle)

* Fix typo
2014-12-18 23:47:27 +01:00
Victor Stinner b261475a48 asyncio: sync with Tulip
* PipeServer.close() now cancels the "accept pipe" future which cancels the
  overlapped operation.
* Fix _SelectorTransport.__repr__() if the transport was closed
* Fix debug log in BaseEventLoop.create_connection(): get the socket object
  from the transport because SSL transport closes the old socket and creates a
  new SSL socket object. Remove also the _SelectorSslTransport._rawsock
  attribute: it contained the closed socket (not very useful) and it was not
  used.
* Issue #22063: socket operations (sock_recv, sock_sendall, sock_connect,
  sock_accept) of the proactor event loop don't raise an exception in debug
  mode if the socket are in blocking mode. Overlapped operations also work on
  blocking sockets.
* Fix unit tests in debug mode: mock a non-blocking socket for socket
  operations which now raise an exception if the socket is blocking.
* _fatal_error() method of _UnixReadPipeTransport and _UnixWritePipeTransport
  now log all exceptions in debug mode
* Don't log expected errors in unit tests
* Tulip issue 200: _WaitHandleFuture._unregister_wait() now catchs and logs
  exceptions.
* Tulip issue 200: Log errors in debug mode instead of simply ignoring them.
2014-08-25 23:20:52 +02:00
Victor Stinner 313a980904 asyncio: sync with Tulip
* _WaitHandleFuture.cancel() now notify IocpProactor through the overlapped
  object that the wait was cancelled.
* Optimize IocpProactor.wait_for_handle() gets the result if the wait is
  signaled immediatly.
* Enhance representation of Future and Future subclasses

  - Add "created at filename:lineno" in the representation
  - Add Future._repr_info() method which can be more easily overriden than
    Future.__repr__(). It should now be more easy to enhance Future
    representation without having to modify each subclass. For example,
    _OverlappedFuture and _WaitHandleFuture get the new "created at" information.
  - Use reprlib to format Future result, and function arguments when formatting a
    callback, to limit the length of the representation.

* Fix repr(_WaitHandleFuture)
* _WaitHandleFuture and _OverlappedFuture: hide frames of internal calls in the
  source traceback.
* Cleanup ProactorIocp._poll(): set the timeout to 0 after the first call to
  GetQueuedCompletionStatus()
* test_locks: close the temporary event loop and check the condition lock
* Remove workaround in test_futures, no more needed
2014-07-29 12:58:23 +02:00
Victor Stinner 42d3bdeed6 asyncio, tulip issue 196: ProactorIocp._register() now registers the overlapped
in the _cache dictionary, even if we already got the result. We need to keep a
reference to the overlapped object, otherwise the memory may be reused and
GetQueuedCompletionStatus() may use random bytes and behaves badly.

There is still a hack for ConnectNamedPipe(): the overlapped object is not
register into _cache if the overlapped object completed directly.

Log also an error in debug mode in ProactorIocp._loop() if we get an unexpected
event.

Add a protection in ProactorIocp.close() to avoid blocking, even if it should
not happen. I still don't understand exactly why some the completion of some
overlapped objects are not notified.
2014-07-28 00:18:43 +02:00
Victor Stinner 51e44ea66a Tulip issue 196: _OverlappedFuture.set_result() now clears its reference to the
overlapped object. IocpProactor._poll() now also ignores false alarms:
GetQueuedCompletionStatus() returns the overlapped but it is still pending.
2014-07-26 00:58:34 +02:00
Victor Stinner 65dd69a3da asyncio: sync with Tulip
* Tulip issue #196: IocpProactor._poll() clears the reference to the
  overlapped operation when the operation is done. It would be better to clear
  the reference in a new _OverlappedFuture.set_result() method, but it cannot
  be done yet because of a weird bug.
* BaseSelectorEventLoop._write_to_self() now logs errors in debug mode.
2014-07-25 22:36:05 +02:00
Victor Stinner 18a28dc5c2 asyncio: sync with Tulip
* Fix _WaitHandleFuture.cancel(): return the result of the parent cancel()
  method.
* _OverlappedFuture.cancel() now clears its reference to the overlapped object.
  Make also the _OverlappedFuture.ov attribute private.
* Check if _WaitHandleFuture completed before unregistering it in the callback.
  Add also _WaitHandleFuture._poll() and repr(_WaitHandleFuture).
* _WaitHandleFuture now unregisters its wait handler if WaitForSingleObject()
  raises an exception.
* _OverlappedFuture.set_exception() now cancels the overlapped operation.
2014-07-25 13:05:20 +02:00
Victor Stinner fea6a100dc asyncio: sync with Tulip
Improve stability of the proactor event loop, especially operations on
overlapped objects:

* Tulip issue 195: Don't call UnregisterWait() twice if a _WaitHandleFuture is
  cancelled twice to fix a crash.
* IocpProactor.close(): cancel futures to cancel overlapped operations, instead
  of cancelling directly overlapped operations. Future objects may not call
  ov.cancel() if the future was cancelled or if the overlapped was already
  cancelled. The cancel() method of the future may also catch exceptions. Log
  also errors on cancellation.
* tests: rename "f" to "fut"
* Add a __repr__() method to IocpProactor
* Add a destructor to IocpProactor which closes it
* _OverlappedFuture.cancel() doesn't cancel the overlapped anymore if it is
  done: if it is already cancelled or completed. Log also an error if the
  cancellation failed.
* Add the address of the overlapped object in repr(_OverlappedFuture)
* _OverlappedFuture truncates the source traceback to hide the call to the
  parent constructor (useless in debug).
2014-07-25 00:54:53 +02:00