When an `StopIteration` raises into `asyncio.Future`, this will cause
a thread to hang. This commit address this by not raising an exception
and silently transforming the `StopIteration` with a `RuntimeError`,
which the caller can reconstruct from `fut.exception().__cause__`
When wrapped, `_SSLProtocolTransport._force_close(exc)` is called just like in the unwrapped scenario `_SelectorTransport._force_close(exc)` or `_ProactorBasePipeTransport._force_close(exc)` would be called, except here the exception needs to be passed through the `SSLProtocol._abort()` method, which didn't accept an exception object.
This commit ensures that this path works, in the same way that the uvloop implementation of SSLProto passes on the exception (on which the current implementation of SSLProto is based).
Fix test_unhandled_exceptions() of test_asyncio.test_streams: break
explicitly a reference cycle.
Fix also StreamTests.tearDown(): the loop must not be closed
explicitly, but using set_event_loop() which takes care of shutting
down the executor with executor.shutdown(wait=True).
BaseEventLoop.close() calls executor.shutdown(wait=False).
* Try to fix asyncio.Server.wait_closed() again
I identified the condition that `wait_closed()` is intended
to wait for: the server is closed *and* there are no more
active connections.
When this condition first becomes true, `_wakeup()` is called
(either from `close()` or from `_detach()`) and it sets `_waiters`
to `None`. So we just check for `self._waiters is None`; if it's
not `None`, we know we have to wait, and do so.
A problem was that the new test introduced in 3.12 explicitly
tested that `wait_closed()` returns immediately when the server
is *not* closed but there are currently no active connections.
This was a mistake (probably a misunderstanding of the intended
semantics). I've fixed the test, and added a separate test that
checks exactly for this scenario.
I also fixed an oddity where in `_wakeup()` the result of the
waiter was set to the waiter itself. This result is not used
anywhere and I changed this to `None`, to avoid a GC cycle.
* Update Lib/asyncio/base_events.py
---------
Co-authored-by: Carol Willing <carolcode@willingconsulting.com>
- `ThreadedChildWatcher.close()` is now *officially* a no-op; `_join_threads()` never did anything.
- Threads created by that class are now named `asyncio-waitpid-NNN`.
- `test.test_asyncio.utils.TestCase.close_loop()` now waits for the child watcher's threads, but not forever; if a thread hangs, it raises `RuntimeError`.
asyncio.TaskGroup and asyncio.Timeout classes now raise proper RuntimeError
if they are improperly used.
* When they are used without entering the context manager.
* When they are used after finishing.
* When the context manager is entered more than once (simultaneously or
sequentially).
* If there is no current task when entering the context manager.
They now remain in a consistent state after an exception is thrown,
so subsequent operations can be performed correctly (if they are allowed).
Co-authored-by: James Hilton-Balfe <gobot1234yt@gmail.com>
Fix test_asyncio timeouts: don't measure the maximum duration, a test
should not measure a CI performance. Only measure the minimum
duration when a task has a timeout or delay. Add CLOCK_RES to
test_asyncio.utils.
Expect the test to be "short" but don't measure the exact performance
of the CI. SHORT_TIMEOUT is about 30 seconds whereas the cancelled
coroutine takes around 1 hour.
Replace harcoded sleep of 500 ms with synchronization using a pipe.
Fix also Process._feed_stdin(): catch also BrokenPipeError on
stdin.write(input), not only on stdin.drain().
Sometimes the child_handled event was missing because either
the child quits before it gets a chance to handle the signal,
or the parent asserts before the event notification is
delivered via IPC. Synchronize explicitly to avoid this.
SubprocessProtocol process_exited() method can be called before
pipe_data_received() and pipe_connection_lost() methods. Document it
and adapt the test for that.
Revert commit 282edd7b2a.
_child_watcher_callback() calls immediately _process_exited(): don't
add an additional delay with call_soon(). The reverted change didn't
make _process_exited() more determistic: it can still be called
before pipe_connection_lost() for example.
Co-authored-by: Davide Rizzo <sorcio@gmail.com>
test_asyncio and test_compileall now clean up multiprocessing by
calling multiprocessing _cleanup_tests(): explicitly clean up
resources and stop background processes like the resource tracker.
The default task name is "Task-<counter>" (if no name is passed in during Task creation).
This is initialized in `Task.__init__` (C impl) using string formatting, which can be quite slow.
Actually using the task name in real world code is not very common, so this is wasted init.
Let's defer this string formatting to the first time the name is read (in `get_name` impl),
so we don't need to pay the string formatting cost if the task name is never read.
We don't change the order in which tasks are assigned numbers (if they are) --
the number is set on task creation, as a PyLong instead of a formatted string.
Co-authored-by: Łukasz Langa <lukasz@langa.pl>
subprocess's communicate(None) closes stdin of the child process, after
sending no (extra) data. Make asyncio variant do the same.
This fixes issues with processes that waits for EOF on stdin before
continuing.
Also use `raise TimeOut from <CancelledError instance>` so that the CancelledError is set
in the `__cause__` field rather than in the `__context__` field.
Co-authored-by: Guido van Rossum <gvanrossum@gmail.com>
Co-authored-by: Alex Waygood <Alex.Waygood@Gmail.com>
This starts the process. Users who don't specify their own start method
and use the default on platforms where it is 'fork' will see a
DeprecationWarning upon multiprocessing.Pool() construction or upon
multiprocessing.Process.start() or concurrent.futures.ProcessPool use.
See the related issue and documentation within this change for details.
Partially revert changes made in GH-93453.
asyncio.DefaultEventLoopPolicy.get_event_loop() now emits a
DeprecationWarning and creates and sets a new event loop instead of
raising a RuntimeError if there is no current event loop set.
Co-authored-by: Guido van Rossum <gvanrossum@gmail.com>
asyncio.get_event_loop() now always return either running event loop or
the result of get_event_loop_policy().get_event_loop() call. The latter
should now raise an RuntimeError if no current event loop was set
instead of creating and setting a new event loop.
It affects also a number of asyncio functions and constructors which
call get_event_loop() implicitly: ensure_future(), shield(), gather(),
etc.
DeprecationWarning is no longer emitted if there is no running event loop but
the current event loop was set.
Co-authored-by: Łukasz Langa <lukasz@langa.pl>
Such buildbots (at the time of writing, only "AMD64 RHEL8 FIPS Only Blake2 Builtin Hash 3.x") cannot use multiprocessing with a fork server, so just skip the test there.
It was a no-op when used as recommended (after close()).
I had to debug one test (test__sock_sendfile_native_failure) --
the cleanup sequence for the test fixture was botched.
Hopefully that's not a portend of problems in user code --
this has never worked so people may well be doing this wrong. :-(
Co-authored-by: kumar aditya
Alas, warnings.catch_warnings() has global scope, not thread scope, so this is still not perfect, but it reduces the time during which warnings are ignored. Better solution welcome.
This is the next step for deprecating child watchers.
Until we've removed the API completely we have to use it, so this PR is mostly suppressing a lot of warnings when using the API internally.
Once the child watcher API is totally removed, the two child watcher implementations we actually use and need (Pidfd and Thread) will be turned into internal helpers.
There is no reason for this watcher to be attached to any particular loop.
This should make it safe to use regardless of the lifetime of the event loop running in the main thread
(relative to other loops).
Co-authored-by: Yury Selivanov <yury@edgedb.com>
Co-authored-by: Jelle Zijlstra <jelle.zijlstra@gmail.com>
This PR reverts gh-93369 and gh-97896 because they've made asyncio tests unstable. After these PRs were merged, random GitHub action jobs of random commits started to fail unrelated tests and test framework methods.
The reverting is necessary because such shrapnel failures are a symptom of some underlying bug that must be found and fixed first.
I had a hope that it's a server overload because we already have extremely rare disc access errors. However, one and a half day passed, and the failures continue to emerge both in PRs and commits.
Affected issue: gh-93357.
First reported in https://github.com/python/cpython/pull/97940#issuecomment-1270004134.
* Revert "gh-93357: Port test cases to IsolatedAsyncioTestCase, part 2 (#97896)"
This reverts commit 09aea94d29.
* Revert "gh-93357: Start porting asyncio server test cases to IsolatedAsyncioTestCase (#93369)"
This reverts commit ce8fc186ac.
Jamie Phan
Kristján Valur Jónsson
Guido van Rossum
Yilei Yang
Martijn Pieters
DPR
Pierre Ossman (ThinLinc team)
Victor Stinner
Kumar Aditya
zcxsythenew
Serhiy Storchaka
Russell Keith-Magee
Thomas Grainger
Alex Waygood
Nikita Sobolev
Davide Rizzo
Joon Hwan 김준환
J. Nick Koston
chgnrdv
Kumar Aditya
Jay
Itamar Ostricher
Marek Marczykowski-Górecki
Itayazolay
Ali-Akber Saifee
Irit Katriel
Gregory P. Smith
Shantanu
DarioDaF
Fantix King
Xiao Chen
Charlie Zhao
fancidev
Christoph Hamsen
Oleg Iarygin
Łukasz Langa