After a long deliberation we ended up feeling that the message argument for Future.cancel(), added in 3.9, was a bad idea, so we're deprecating it in 3.11 and plan to remove it in 3.13.
- Add requires_fork and requires_subprocess to more tests
- Skip extension import tests if dlopen is not available
- Don't assume that _testcapi is a shared extension
- Skip a lot of socket tests that don't work on Emscripten
- Skip mmap tests, mmap emulation is incomplete
- venv does not work yet
- Cannot get libc from executable
The "entire" test suite is now passing on Emscripten with EMSDK from git head (91 suites are skipped).
GH-30297 removed a duplicate `from test import support` statement from `test_asyncio.test_sslproto`. However, in between that PR being filed and it being merged, GH-31275 removed the _other_ `from test import support` statement. This means that `support` is now undefined in `test_asyncio.test_sslproto`, causing the CI to fail on all platforms for all PRS.
Example:
async with asyncio.timeout(5):
await some_task()
Will interrupt the await and raise TimeoutError if some_task() takes longer than 5 seconds.
Co-authored-by: Guido van Rossum <guido@python.org>
Also from the _asyncio C accelerator module,
and adjust one test that the change caused to fail.
For more discussion see the discussion starting here:
https://github.com/python/cpython/pull/31394#issuecomment-1053545331
(Basically, @asvetlov proposed to return False from cancel()
when there is already a pending cancellation, and I went along,
even though it wasn't necessary for the task group implementation,
and @agronholm has come up with a counterexample that fails
because of this change. So now I'm changing it back to the old
semantics (but still bumping the counter) until we can have a
proper discussion about this.)
* Remove task group names (for now)
We're not sure that they are needed, and once in the code
we would never be able to get rid of them.
Yury wrote:
> Ideally, there should be a way for someone to build a "trace"
> of taskgroups/task leading to the current running task.
> We could do that using contextvars, but I'm not sure we should
> do that in 3.11.
* Pass optional name on to task in create_task()
* Remove a bunch of unused stuff
asyncio/taskgroups.py is an adaptation of taskgroup.py from EdgeDb, with the following key changes:
- Allow creating new tasks as long as the last task hasn't finished
- Raise [Base]ExceptionGroup (directly) rather than TaskGroupError deriving from MultiError
- Instead of monkey-patching the parent task's cancel() method,
add a new public API to Task
The Task class has a new internal flag, `_cancel_requested`, which is set when `.cancel()` is called successfully. The `.cancelling()` method returns the value of this flag. Further `.cancel()` calls while this flag is set return False. To reset this flag, call `.uncancel()`.
Thus, a Task that catches and ignores `CancelledError` should call `.uncancel()` if it wants to be cancellable again; until it does so, it is deemed to be busy with uninterruptible cleanup.
This new Task API helps solve the problem where TaskGroup needs to distinguish between whether the parent task being cancelled "from the outside" vs. "from inside".
Co-authored-by: Yury Selivanov <yury@edgedb.com>
Co-authored-by: Andrew Svetlov <andrew.svetlov@gmail.com>
Due to significant security concerns, the reuse_address parameter of
asyncio.loop.create_datagram_endpoint, deprecated in Python 3.9, is
now removed. This is because of the behavior of the socket option
SO_REUSEADDR in UDP.
Co-authored-by: Łukasz Langa <lukasz@langa.pl>
Remove the @asyncio.coroutine decorator
enabling legacy generator-based coroutines to be compatible with async/await
code; remove asyncio.coroutines.CoroWrapper used for wrapping
legacy coroutine objects in the debug mode.
The decorator has been deprecated
since Python 3.8 and the removal was initially scheduled for Python 3.10.
Fix asyncio test_popen() of test_windows_utils by using a longer
timeout. Use military grade battle-tested test.support.SHORT_TIMEOUT
timeout rather than a hardcoded timeout of 10 seconds: it's 30
seconds by default, but it is made longer on slow buildbots.
WaitForMultipleObjects() timeout argument is in milliseconds.
asyncio.get_event_loop() emits now a deprecation warning when it creates a new event loop.
In future releases it will became an alias of asyncio.get_running_loop().
Fix problem with ssl.SSLContext.hostname_checks_common_name. OpenSSL does not
copy hostflags from *struct SSL_CTX* to *struct SSL*.
Signed-off-by: Christian Heimes <christian@python.org>
test_unix_events.py no longer checks if waitstatus_to_exitcode() mock
has been called or not to make the test more functional, rather than
checking the exact implementation.
Fix test_asyncio.test_call_later() race condition: don't measure
asyncio performance in the call_later() unit test. The test failed
randomly on the CI.
* Update code after merge review from 1st1
* Use a sentinel approach for loop parameter
Remove unnecessary _get_running_loop patching
* Use more clear function name (_verify_parameter_is_marker -> _verify_no_loop)
* Add init method to _LoopBoundMixin to check that loop param wasn't used
# Improve asyncio.wait function
The original code creates the futures set two times.
We can create this set before, avoiding the second creation.
This new behaviour [breaks the aiokafka library](https://github.com/aio-libs/aiokafka/pull/672), because it gives an iterator to that function, so the second iteration become empty.
Automerge-Triggered-By: GH:1st1
Simply closing the event loop isn't enough to avoid warnings. If we
don't also shut down the event loop's default executor, it sometimes
logs a "dangling thread" warning.
Follow-up to GH-22017
* bpo-41696: Fix handling of debug mode in asyncio.run
This allows PYTHONASYNCIODEBUG or -X dev to enable asyncio debug mode
when using asyncio.run
* 📜🤖 Added by blurb_it.
Co-authored-by: hauntsaninja <>
Co-authored-by: blurb-it[bot] <43283697+blurb-it[bot]@users.noreply.github.com>
Stopping and restarting a proactor event loop on windows can lead to
spurious errors logged (ConnectionResetError while reading from the
self pipe). This fixes the issue by ensuring that we don't attempt
to start multiple copies of the self-pipe reading loop.
Currently, if `asyncio.wait_for()` itself is cancelled it will always
raise `CancelledError` regardless if the underlying task is still
running. This is similar to a race with the timeout, which is handled
already.
When I was fixing bpo-32751 back in GH-7216 I missed the case when
*timeout* is zero or negative. This takes care of that.
Props to @aaliddell for noticing the inconsistency.
* bpo-41273: Proactor transport read loop to use recv_into
By using recv_into instead of recv we do not allocate a new buffer each
time _loop_reading calls recv.
This betters performance for any stream using proactor (basically any
asyncio stream on windows).
* bpo-41273: Double proactor read transport buffer size
By doubling the read buffer size we get better performance.
On Linux, skip tests using multiprocessing if the current user cannot
create a file in /dev/shm/ directory. Add the
skip_if_broken_multiprocessing_synchronize() function to the
test.support module.
Using a log2n way to fill a much smaller buffer, and receiving in a cleaner way with EOF.
The failing test was reproducible using the following command thanks to @aeros :
```bash
./python -m test test_asyncio.test_sock_lowlevel --match test_sock_client_racing -j100 -F -v
```
According to test results, we may still need to bump the timeout:
5aad027db9/Lib/test/test_asyncio/test_sock_lowlevel.py (L256-L257)
Skip new "racing" socket tests which fail randomly until someone fix
them, to ease analysis of buildbot failures (skip tests which are
known to be broken/unstable).
This updates _PyErr_ChainStackItem() to use _PyErr_SetObject()
instead of _PyErr_ChainExceptions(). This prevents a hang in
certain circumstances because _PyErr_SetObject() performs checks
to prevent cycles in the exception context chain while
_PyErr_ChainExceptions() doesn't.
Implements `asyncio.to_thread`, a coroutine for asynchronously running IO-bound functions in a separate thread without blocking the event loop. See the discussion starting from [here](https://github.com/python/cpython/pull/18410#issuecomment-628930973) in GH-18410 for context.
Automerge-Triggered-By: @aeros
Ifdef is not necessary, as AF_INET6 is supported from Windows Vista, and other code in overlapped.c uses AF_INET6 and is not ifdef'd.
Change the raised exception so users are not fooled to think it comes from Windows API.
Automerge-Triggered-By: @njsmith
When an asyncio.Task is cancelled, the exception traceback now
starts with where the task was first interrupted. Previously,
the traceback only had "depth one."
Currently, if asyncio.wait_for() timeout expires, it cancels
inner future and then always raises TimeoutError. In case
those future is task, it can handle cancelation mannually,
and those process can lead to some other exception. Current
implementation silently loses thoses exception.
To resolve this, wait_for will check was the cancelation
successfull or not. In case there was exception, wait_for
will reraise it.
Co-authored-by: Roman Skurikhin <roman.skurikhin@cruxlab.com>
The previous commits on bpo-29587 got exception chaining working
with gen.throw() in the `yield` case. This patch also gets the
`yield from` case working.
As a consequence, implicit exception chaining now also works in
the asyncio scenario of awaiting on a task when an exception is
already active.
Tests are included for both the asyncio case and the pure
generator-only case.
* Move socket related functions from test.support to socket_helper.
* Import socket, nntplib and urllib.error lazily in transient_internet().
* Remove importing multiprocess.
Replace hardcoded timeout constants in tests with SHORT_TIMEOUT of
test.support, so it's easier to ajdust this timeout for all tests at
once.
SHORT_TIMEOUT is 30 seconds by default, but it can be longer
depending on --timeout command line option.
The change makes almost all timeouts longer, except
test_reap_children() of test_support which is made 2x shorter:
SHORT_TIMEOUT should be enough. If this test starts to fail,
LONG_TIMEOUT should be used instead.
Uniformize also "from test import support" import in some test files.
Replace hardcoded timeout constants in tests with LONG_TIMEOUT of
test.support, so it's easier to ajdust this timeout for all tests at
once.
LONG_TIMEOUT is 5 minutes by default, but it can be longer depending
on --timeout command line option.
Tests no longer pass a timeout value to join_thread() of
test.support: use the default join_thread() timeout instead
(SHORT_TIMEOUT constant of test.support).
main() is now responsible to send the ANSWER, rather than
ServerProto. main() now waits until it got the HELLO before sending
the ANSWER over the new transport.
Previously, there was a race condition between main() replacing the
protocol and the protocol sending the ANSWER once it gets the HELLO.
TLSv1.3 was disabled for the test: reenable it.
This PR deprecate explicit loop parameters in all public asyncio APIs
This issues is split to be easier to review.
Third step: locks.py
https://bugs.python.org/issue36373
When the test is ran with `PYTHONWARNINGS=error` the environment variable is passed to the python interpreter used in `assert_python_ok` where `DeprecationWarning` from `@asyncio.coroutine` is converted into an error. Ignore the `DeprecationWarning` in `assert_python_ok`.
https://bugs.python.org/issue37323
bpo-35031, bpo-35998: Reintroduce workaround on Windows and FreeBSD
in test_start_tls_server_1() of test_asyncio: disable TLS v1.3 on the
client context.
Join the thread to prevent leaking a running thread and leaking a
reference.
Cleanup also the test:
* asyncioWindowsProactorEventLoopPolicy became the default policy,
there is no need to set it manually.
* Only start the thread once the loop is running.
* Use a shorter sleep in the thread (100 ms rather than 1 sec).
* Use close_loop(loop) rather than loop.close().
* Use longer variable names.
Replace asyncio.set_event_loop() with TestCase.set_event_loop() of
test_asyncio.utils: this method calls TestCase.close_loop() which
waits until the executor completes, to avoid leaking dangling
threads.
Inherit from test_asyncio.utils.TestCase rather than
unittest.TestCase.
From 3.8 async functions used with mock.patch return an `AsyncMock`. `_accept_connection2` is an async function where create_task is also mocked. Don't mock `create_task` so that tasks are created out of coroutine returned by `AsyncMock` and the tasks are completed.
https://bugs.python.org/issue37015
This will address the common mistake many asyncio users make:
an "except Exception" clause breaking Tasks cancellation.
In addition to this change, we stop inheriting asyncio.TimeoutError
and asyncio.InvalidStateError from their concurrent.futures.*
counterparts. There's no point for these exceptions to share the
inheritance chain.
In 3.9 we'll focus on implementing supervisors and cancel scopes,
which should allow better handling of all exceptions, including
SystemExit and KeyboardInterrupt
This PR proposes a solution to [bpo-35545](https://bugs.python.org/issue35545) by adding an optional `flowinfo` and `scopeid` to `asyncio.base_events._ipaddr_info` to carry the full address information into `_ipaddr_info` and avoid discarding IPv6 specific information.
Changelog entry & regression tests to come.
https://bugs.python.org/issue35545
When the future returned by shield is cancelled, its completion callback of the
inner future is not removed. This makes the callback list of inner inner future
grow each time a shield is created and cancelled.
This change unregisters the callback from the inner future when the outer
future is cancelled.
https://bugs.python.org/issue35125
*Moved from python/asyncio#493.*
This PR fixes issue python/asyncio#480, as explained in [this comment](https://github.com/python/asyncio/issues/480#issuecomment-278703828).
The `_SelectorDatagramTransport.sendto` method has to be modified ~~so `_sock.sendto` is used in all cases (because it is tricky to reliably tell if the socket is connected or not). Could that be an issue for connected sockets?~~ *EDIT* ... so `_sock.send` is used only if `_sock` is connected.
It also protects `socket.getsockname` against `OSError` in `_SelectorTransport`. This might happen on Windows if the socket is not connected (e.g. for UDP broadcasting).
https://bugs.python.org/issue31922
As in title, expose C `raise` function as `raise_function` in `signal` module. Also drop existing `raise_signal` in `_testcapi` module and replace all usages with new function.
https://bugs.python.org/issue35568
test_asyncio/test_sendfile.py now resets the event loop policy using
tearDownModule() as done in other tests, to prevent a warning when
running tests on Windows.
There is a race condition in SelectorEventLoopUnixSockSendfileTests that causes the prepare() method return a non connected server protocol, making the cleanup() method skips the correct handling of the transport. This commit makes prepare() always return a connected server protocol that can always be cleaned up correctly.
There is a race condition regarding signal delivery in test_signal_handling_args for
test_asyncio.test_events.KqueueEventLoopTests. The signal can be received at any moment outside the time window provided in the test. The fix is to wait for the signal to be received instead with a bigger timeout.
Replace time.time() with time.monotonic() in tests to measure time
delta.
test_zipfile64: display progress every minute (60 secs) rather than
every 5 minutes (5*60 seconds).
Modify asyncio tests to utilize the certificates from the test directory
instead of its own set, as they are the same and with each update they had
to be updated as well.
The call to `_untrack_reader` is performed too soon, causing the protocol
to forget about the reader before `connection_lost` can run and feed the
EOF to the reader. See bpo-35065.
Some FreeBSD buildbots fail to run this test as the eof was not being received by the server if the size is not big enough. This behaviour only appears if the client is using TLS1.3.
inspect.isfunction() processes both inspect.isfunction(func) and
inspect.isfunction(partial(func, arg)) correctly but some other functions in the
inspect module (iscoroutinefunction, isgeneratorfunction and isasyncgenfunction)
lack this functionality. This commits adds a new check in the mentioned functions
in the inspect module so they can work correctly with arbitrarily nested partial
functions.
The C implementation of asyncio.Task currently fails to perform the
cancellation cleanup correctly in the following scenario.
async def task1():
async def task2():
await task3 # task3 is never cancelled
asyncio.current_task().cancel()
await asyncio.create_task(task2())
The actuall error is a hardcoded call to `future_cancel()` instead of
calling the `cancel()` method of a future-like object.
Thanks to Vladimir Matveev for noticing the code discrepancy and to
Yury Selivanov for coming up with a pathological scenario.
The waiting is pretty normal for any asyncio program, logging its time just adds
a noise to logs without any useful information provided.
https://bugs.python.org/issue34849
* Insert the warn in the asyncio.sleep when the loop argument is used
* Insert the warn in the asyncio.wait and asyncio.wait_for when the loop argument is used
* Better format of the code
* Add news file
* change calls for get_event_loop() to calls for get_running_loop()
* Change message to be more clear in News
* Improve the comments in test_tasks
Store a weak reference to stream readerfor breaking strong references
It breaks the strong reference loop between reader and protocol and allows to detect and close the socket if the stream is deleted (garbage collected)
Update all test certs and keys to use future proof crypto settings:
* 3072 bit RSA keys
* SHA-256 signature
Signed-off-by: Christian Heimes <christian@python.org>
Various asyncio internals expect that the default executor is a
`ThreadPoolExecutor`, so deprecate passing anything else to
`loop.set_default_executor()`.
The cancellation of an overlapped WSARecv() has a race condition
which causes data loss because of the current implementation of
proactor in asyncio.
No longer cancel overlapped WSARecv() in _ProactorReadPipeTransport
to work around the race condition.
Remove the optimized recv_into() implementation to get simple
implementation of pause_reading() using the single _pending_data
attribute.
Move _feed_data_to_bufferred_proto() to protocols.py.
Remove set_protocol() method which became useless.
Fix "<CoroWrapper ...> was never yielded from" warning in
PyTask_PyFuture_Tests.test_error_in_call_soon() of
test_asyncio.test_tasks.
Close manually the coroutine on error.
* Fix AttributeError (not all SSL exceptions have 'errno' attribute)
* Increase default handshake timeout from 10 to 60 seconds
* Make sure start_tls can be cancelled correctly
* Make sure any error in SSLProtocol gets propagated (instead of just being logged)
Currently, asyncio.wait_for(fut), upon reaching the timeout deadline,
cancels the future and returns immediately. This is problematic for
when *fut* is a Task, because it will be left running for an arbitrary
amount of time. This behavior is iself surprising and may lead to
related bugs such as the one described in bpo-33638:
condition = asyncio.Condition()
async with condition:
await asyncio.wait_for(condition.wait(), timeout=0.5)
Currently, instead of raising a TimeoutError, the above code will fail
with `RuntimeError: cannot wait on un-acquired lock`, because
`__aexit__` is reached _before_ `condition.wait()` finishes its
cancellation and re-acquires the condition lock.
To resolve this, make `wait_for` await for the task cancellation.
The tradeoff here is that the `timeout` promise may be broken if the
task decides to handle its cancellation in a slow way. This represents
a behavior change and should probably not be back-patched to 3.6 and
earlier.
Use transport.set_write_buffer_limits() in sendfile tests of
test_asyncio to make sure that the protocol is paused after sending
4 KiB. Previously,
test_sendfile_fallback_close_peer_in_the_middle_of_receiving() failed
on FreeBSD if the DATA was smaller than the default limit of 64 KiB.
In this commit:
* Support BufferedProtocol in set_protocol() and start_tls()
* Fix proactor to cancel readers reliably
* Update tests to be compatible with OpenSSL 1.1.1
* Clarify BufferedProtocol docs
* Bump TLS tests timeouts to 60 seconds; eliminate possible race from start_serving
* Rewrite test_start_tls_server_1
bpo-32622, bpo-33353: On macOS, sock.connect() changes the
SO_SNDBUF value. Only set SO_SNDBUF and SO_RCVBUF buffer sizes
once a socket is connected or binded, not before.
bpo-32622, bpo-33353: sendfile() tests of test_asyncio use socket
buffers of 1 kB "to test on relative small data sets". Send only
160 KiB rather 10 MB to make the test much faster.
Shrink also SendfileBase.DATA from 1600 KiB to 160 KiB.
On Linux, 3 test_sock_sendfile_mix_with_regular_send() runs now take
less than 1 second, instead of 18 seconds.
On FreeBSD, the 3 tests didn't hang, but took 3 minutes. Now
the 3 tests pass in less than 1 seconds.
TLS 1.3 behaves slightly different than TLS 1.2. Session tickets and TLS
client cert auth are now handled after the initialy handshake. Tests now
either send/recv data to trigger session and client certs. Or tests
ignore ConnectionResetError / BrokenPipeError on the server side to
handle clients that force-close the socket fd.
To test TLS 1.3, OpenSSL 1.1.1-pre7-dev (git master + OpenSSL PR
https://github.com/openssl/openssl/pull/6340) is required.
Signed-off-by: Christian Heimes <christian@python.org>
Andrew Svetlov
Christian Heimes
Maximilian Hils
Alex Grönholm
Alex Waygood
Nikita Sobolev
Serhiy Storchaka
Guido van Rossum
Tin Tvrtković
Thomas Grainger
Steve Dower
Kumar Aditya
Kumar Aditya
Sam Bull
Joongi Kim
Christian Clauss
Hugo van Kemenade
Illia Volochii
Victor Stinner
Pablo Galindo
Inada Naoki
Richard Kojedzinszky
pxinwr
Zackery Spytz
Yurii Karabas
Diogo Dutra
Justin Turner Arthur
Jakub Kulík
Ben Darnell
Shantanu
Elvis Pranskevichus
Hai Shi
Tony Solomonik
Rémi Lapeyre
Fantix King
Chris Jerdonek
Kyle Stanley
Nathaniel J. Smith
Kjell Braden
romasku
Stefan Krah
xdegaye
Benjamin Peterson
Yury Selivanov
Victor Stinner
Emmanuel Arias
Xtreak
Ned Deily
Michael Felt
依云
sbstp
Yury Selivanov
Niklas Fiekas
Erwan Le Pape
Paul Monson
Romain Picard
Vincent Michel
Quentin Dawans
Giampaolo Rodola
Vladimir Matveev
stratakis
Srinivas Thatiparthy (శ్రీనివాస్ తాటిపర్తి)
twisteroid ambassador
João Júnior
Brett Cannon