Also from the _asyncio C accelerator module,
and adjust one test that the change caused to fail.
For more discussion see the discussion starting here:
https://github.com/python/cpython/pull/31394#issuecomment-1053545331
(Basically, @asvetlov proposed to return False from cancel()
when there is already a pending cancellation, and I went along,
even though it wasn't necessary for the task group implementation,
and @agronholm has come up with a counterexample that fails
because of this change. So now I'm changing it back to the old
semantics (but still bumping the counter) until we can have a
proper discussion about this.)
This changes cancelling() and uncancel() to return the count of pending cancellations.
This can be used to avoid bugs in certain edge cases (e.g. two timeouts going off at the same time).
* Remove task group names (for now)
We're not sure that they are needed, and once in the code
we would never be able to get rid of them.
Yury wrote:
> Ideally, there should be a way for someone to build a "trace"
> of taskgroups/task leading to the current running task.
> We could do that using contextvars, but I'm not sure we should
> do that in 3.11.
* Pass optional name on to task in create_task()
* Remove a bunch of unused stuff
asyncio/taskgroups.py is an adaptation of taskgroup.py from EdgeDb, with the following key changes:
- Allow creating new tasks as long as the last task hasn't finished
- Raise [Base]ExceptionGroup (directly) rather than TaskGroupError deriving from MultiError
- Instead of monkey-patching the parent task's cancel() method,
add a new public API to Task
The Task class has a new internal flag, `_cancel_requested`, which is set when `.cancel()` is called successfully. The `.cancelling()` method returns the value of this flag. Further `.cancel()` calls while this flag is set return False. To reset this flag, call `.uncancel()`.
Thus, a Task that catches and ignores `CancelledError` should call `.uncancel()` if it wants to be cancellable again; until it does so, it is deemed to be busy with uninterruptible cleanup.
This new Task API helps solve the problem where TaskGroup needs to distinguish between whether the parent task being cancelled "from the outside" vs. "from inside".
Co-authored-by: Yury Selivanov <yury@edgedb.com>
Co-authored-by: Andrew Svetlov <andrew.svetlov@gmail.com>
This parameter was accidentally removed when fixing
https://bugs.python.org/issue45129, this reverts the unnecessary changes
there.
Co-authored-by: Andrew Svetlov <andrew.svetlov@gmail.com>
Due to significant security concerns, the reuse_address parameter of
asyncio.loop.create_datagram_endpoint, deprecated in Python 3.9, is
now removed. This is because of the behavior of the socket option
SO_REUSEADDR in UDP.
Co-authored-by: Łukasz Langa <lukasz@langa.pl>
Remove the @asyncio.coroutine decorator
enabling legacy generator-based coroutines to be compatible with async/await
code; remove asyncio.coroutines.CoroWrapper used for wrapping
legacy coroutine objects in the debug mode.
The decorator has been deprecated
since Python 3.8 and the removal was initially scheduled for Python 3.10.
* [Enum] reduce scope of new format behavior
Instead of treating all Enums the same for format(), only user mixed-in
enums will be affected. In other words, IntEnum and IntFlag will not be
changing the format() behavior, due to the requirement that they be
drop-in replacements of existing integer constants.
If a user creates their own integer-based enum, then the new behavior
will apply:
class Grades(int, Enum):
A = 5
B = 4
C = 3
D = 2
F = 0
Now: format(Grades.B) -> DeprecationWarning and '4'
3.12: -> no warning, and 'B'
asyncio.get_event_loop() emits now a deprecation warning when it creates a new event loop.
In future releases it will became an alias of asyncio.get_running_loop().
test_unix_events.py no longer checks if waitstatus_to_exitcode() mock
has been called or not to make the test more functional, rather than
checking the exact implementation.
This is my first issue!
So, if there's anything wrong, please tell me!
Also, thank you always for all the contributors!
Automerge-Triggered-By: GH:asvetlov
* Update code after merge review from 1st1
* Use a sentinel approach for loop parameter
Remove unnecessary _get_running_loop patching
* Use more clear function name (_verify_parameter_is_marker -> _verify_no_loop)
* Add init method to _LoopBoundMixin to check that loop param wasn't used
# Improve asyncio.wait function
The original code creates the futures set two times.
We can create this set before, avoiding the second creation.
This new behaviour [breaks the aiokafka library](https://github.com/aio-libs/aiokafka/pull/672), because it gives an iterator to that function, so the second iteration become empty.
Automerge-Triggered-By: GH:1st1
* bpo-41696: Fix handling of debug mode in asyncio.run
This allows PYTHONASYNCIODEBUG or -X dev to enable asyncio debug mode
when using asyncio.run
* 📜🤖 Added by blurb_it.
Co-authored-by: hauntsaninja <>
Co-authored-by: blurb-it[bot] <43283697+blurb-it[bot]@users.noreply.github.com>
Stopping and restarting a proactor event loop on windows can lead to
spurious errors logged (ConnectionResetError while reading from the
self pipe). This fixes the issue by ensuring that we don't attempt
to start multiple copies of the self-pipe reading loop.
Currently, if `asyncio.wait_for()` itself is cancelled it will always
raise `CancelledError` regardless if the underlying task is still
running. This is similar to a race with the timeout, which is handled
already.
When I was fixing bpo-32751 back in GH-7216 I missed the case when
*timeout* is zero or negative. This takes care of that.
Props to @aaliddell for noticing the inconsistency.
asyncio.AbstractEventLoop.run_in_executor should be a method that returns an asyncio Future, not an async method.
This matches the concrete implementations, and the documentation better.
* bpo-41273: Proactor transport read loop to use recv_into
By using recv_into instead of recv we do not allocate a new buffer each
time _loop_reading calls recv.
This betters performance for any stream using proactor (basically any
asyncio stream on windows).
* bpo-41273: Double proactor read transport buffer size
By doubling the read buffer size we get better performance.
When I wrote the documentation for `asyncio.to_thread()`, I mistakenly assumed that `return await loop.run_in_executor(...)` within an async def function would return a Future. In reality, it returns a coroutine.
This likely won't affect typical usage of `asyncio.to_thread()`, but it's important for the documentation to be correct here. In general, we also tend to avoid returning futures from high-level APIs in asyncio.
Implements `asyncio.to_thread`, a coroutine for asynchronously running IO-bound functions in a separate thread without blocking the event loop. See the discussion starting from [here](https://github.com/python/cpython/pull/18410#issuecomment-628930973) in GH-18410 for context.
Automerge-Triggered-By: @aeros
When an asyncio.Task is cancelled, the exception traceback now
starts with where the task was first interrupted. Previously,
the traceback only had "depth one."
Currently, if asyncio.wait_for() timeout expires, it cancels
inner future and then always raises TimeoutError. In case
those future is task, it can handle cancelation mannually,
and those process can lead to some other exception. Current
implementation silently loses thoses exception.
To resolve this, wait_for will check was the cancelation
successfull or not. In case there was exception, wait_for
will reraise it.
Co-authored-by: Roman Skurikhin <roman.skurikhin@cruxlab.com>
Motivation for this PR (comment from @vstinner in bpo issue):
```
Warning seen o AMD64 Ubuntu Shared 3.x buildbot:
https://buildbot.python.org/all/#/builders/141/builds/2593
test_devnull_output (test.test_a=syncio.test_subprocess.SubprocessThreadedWatcherTests) ...
Warning -- threading_cleanup() failed to cleanup 1 threads (count: 1, dangling: 2)
```
The following implementation details for the new method are TBD:
1) Public vs private
2) Inclusion in `close()`
3) Name
4) Coroutine vs subroutine method
5) *timeout* parameter
If it's a private method, 3, 4, and 5 are significantly less important.
I started with the most minimal implementation that fixes the dangling threads without modifying the regression tests, which I think is particularly important. I typically try to avoid directly modifying existing tests as much as possible unless it's necessary to do so. However, I am open to changing any part of this.
https://bugs.python.org/issue38356
If waitpid() is called elsewhere, waitpid() call fails with
ChildProcessError: use return code 255 in this case, and log a
warning. It ensure that the pidfd file descriptor is closed if this
error occurs.
Capturing exceptions into names can lead to reference cycles though the __traceback__ attribute of the exceptions in some obscure cases that have been reported previously and fixed individually. As these variables are not used anyway, we can remove the binding to reduce the chances of creating reference cycles.
See for example GH-13135
* This just copies the docs from `StreamWriter` and `StreamReader`.
* Add docstring for asyncio functions.
https://bugs.python.org/issue36889
Automerge-Triggered-By: @asvetlov
This PR deprecate explicit loop parameters in all public asyncio APIs
This issues is split to be easier to review.
fourth step: queue.py
https://bugs.python.org/issue36373
This PR deprecate explicit loop parameters in all public asyncio APIs
This issues is split to be easier to review.
Third step: locks.py
https://bugs.python.org/issue36373
This PR deprecate explicit loop parameters in all public asyncio APIs
This issues is split to be easier to review.
Second step: streams.py
https://bugs.python.org/issue36373
Add BaseEventLoop.wait_executor_on_close attribute: true by default.
loop.close() now waits for the default executor to finish by default.
Set loop.wait_executor_on_close attribute to False to not wait for
the executor.
This will address the common mistake many asyncio users make:
an "except Exception" clause breaking Tasks cancellation.
In addition to this change, we stop inheriting asyncio.TimeoutError
and asyncio.InvalidStateError from their concurrent.futures.*
counterparts. There's no point for these exceptions to share the
inheritance chain.
In 3.9 we'll focus on implementing supervisors and cancel scopes,
which should allow better handling of all exceptions, including
SystemExit and KeyboardInterrupt
This PR proposes a solution to [bpo-35545](https://bugs.python.org/issue35545) by adding an optional `flowinfo` and `scopeid` to `asyncio.base_events._ipaddr_info` to carry the full address information into `_ipaddr_info` and avoid discarding IPv6 specific information.
Changelog entry & regression tests to come.
https://bugs.python.org/issue35545
When the future returned by shield is cancelled, its completion callback of the
inner future is not removed. This makes the callback list of inner inner future
grow each time a shield is created and cancelled.
This change unregisters the callback from the inner future when the outer
future is cancelled.
https://bugs.python.org/issue35125
*Moved from python/asyncio#493.*
This PR fixes issue python/asyncio#480, as explained in [this comment](https://github.com/python/asyncio/issues/480#issuecomment-278703828).
The `_SelectorDatagramTransport.sendto` method has to be modified ~~so `_sock.sendto` is used in all cases (because it is tricky to reliably tell if the socket is connected or not). Could that be an issue for connected sockets?~~ *EDIT* ... so `_sock.send` is used only if `_sock` is connected.
It also protects `socket.getsockname` against `OSError` in `_SelectorTransport`. This might happen on Windows if the socket is not connected (e.g. for UDP broadcasting).
https://bugs.python.org/issue31922
Added two keyword arguments, `delay` and `interleave`, to
`BaseEventLoop.create_connection`. Happy eyeballs is activated if
`delay` is specified.
We now have documentation for the new arguments. `staggered_race()` is in its own module, but not exported to the main asyncio package.
https://bugs.python.org/issue33530
asyncio.ProactorEventLoop now catchs and logs send errors when the
self-pipe is full: BaseProactorEventLoop._write_to_self() now catchs
and logs OSError exceptions, as done by
BaseSelectorEventLoop._write_to_self().
IocpProactor.close() now uses time to decide when to log: wait 1
second before the first log, then log every second. Log also the
number of seconds since close() was called.
* asyncio: __del__() keep reference to warnings.warn
The __del__() methods of asyncio classes now keep a strong reference
to the warnings.warn() to be able to display the ResourceWarning
warning in more cases. Ensure that the function remains available if
instances are destroyed late during Python shutdown (while module
symbols are cleared).
* Rename warn parameter to _warn
"_warn" name is a hint that it's not the regular warnings.warn()
function.
* _wait_for_handle(), _register() and _unregister() methods of
IocpProactor now raise an exception if closed
* Add "closed" to IocpProactor.__repr__()
* Simplify IocpProactor.close()
bpo-32622, bpo-35682: Fix asyncio.ProactorEventLoop.sendfile(): don't
attempt to set the result of an internal future if it's already done.
Fix asyncio _ProactorBasePipeTransport._force_close(): don't set the
result of _empty_waiter if it's already done.
The call to `_untrack_reader` is performed too soon, causing the protocol
to forget about the reader before `connection_lost` can run and feed the
EOF to the reader. See bpo-35065.
The waiting is pretty normal for any asyncio program, logging its time just adds
a noise to logs without any useful information provided.
https://bugs.python.org/issue34849
* Insert the warn in the asyncio.sleep when the loop argument is used
* Insert the warn in the asyncio.wait and asyncio.wait_for when the loop argument is used
* Better format of the code
* Add news file
* change calls for get_event_loop() to calls for get_running_loop()
* Change message to be more clear in News
* Improve the comments in test_tasks
Store a weak reference to stream readerfor breaking strong references
It breaks the strong reference loop between reader and protocol and allows to detect and close the socket if the stream is deleted (garbage collected)
Various asyncio internals expect that the default executor is a
`ThreadPoolExecutor`, so deprecate passing anything else to
`loop.set_default_executor()`.
The cancellation of an overlapped WSARecv() has a race condition
which causes data loss because of the current implementation of
proactor in asyncio.
No longer cancel overlapped WSARecv() in _ProactorReadPipeTransport
to work around the race condition.
Remove the optimized recv_into() implementation to get simple
implementation of pause_reading() using the single _pending_data
attribute.
Move _feed_data_to_bufferred_proto() to protocols.py.
Remove set_protocol() method which became useless.
* Fix AttributeError (not all SSL exceptions have 'errno' attribute)
* Increase default handshake timeout from 10 to 60 seconds
* Make sure start_tls can be cancelled correctly
* Make sure any error in SSLProtocol gets propagated (instead of just being logged)
Future.set_result and Future.set_exception now raise InvalidStateError
if the futures are not pending or running. This mirrors the behavior
of asyncio.Future, and prevents AssertionErrors in asyncio.wrap_future
when set_result is called multiple times.
Currently, asyncio.wait_for(fut), upon reaching the timeout deadline,
cancels the future and returns immediately. This is problematic for
when *fut* is a Task, because it will be left running for an arbitrary
amount of time. This behavior is iself surprising and may lead to
related bugs such as the one described in bpo-33638:
condition = asyncio.Condition()
async with condition:
await asyncio.wait_for(condition.wait(), timeout=0.5)
Currently, instead of raising a TimeoutError, the above code will fail
with `RuntimeError: cannot wait on un-acquired lock`, because
`__aexit__` is reached _before_ `condition.wait()` finishes its
cancellation and re-acquires the condition lock.
To resolve this, make `wait_for` await for the task cancellation.
The tradeoff here is that the `timeout` promise may be broken if the
task decides to handle its cancellation in a slow way. This represents
a behavior change and should probably not be back-patched to 3.6 and
earlier.
Use transport.set_write_buffer_limits() in sendfile tests of
test_asyncio to make sure that the protocol is paused after sending
4 KiB. Previously,
test_sendfile_fallback_close_peer_in_the_middle_of_receiving() failed
on FreeBSD if the DATA was smaller than the default limit of 64 KiB.
Stefan Zabka
Serhiy Storchaka
Guido van Rossum
Tin Tvrtković
Andrew Svetlov
Thomas Grainger
Nikita Sobolev
Alex Waygood
Kumar Aditya
Emiya
Jim Crist-Harif
Kumar Aditya
Sam Bull
Gregory P. Smith
Joongi Kim
Christian Clauss
Hugo van Kemenade
Jesús Cea
Mariusz Felisiak
Illia Volochii
Ethan Furman
Binbin
Pablo Galindo
Christian Heimes
Roberto Hueso
Richard Kojedzinszky
Chris Jerdonek
Victor Stinner
Zackery Spytz
Wansoo Kim
Yurii Karabas
Alex Grönholm
Diogo Dutra
Jakub Stasiak
Shantanu
Ben Darnell
Elvis Pranskevichus
Cleber Rosa
James Weaver
Vinay Sharma
Tony Solomonik
Rémi Lapeyre
Kyle Stanley
Fantix King
Kunal Bhalla
romasku
Galden
Lidi Zheng
Markus Mohrhard
idomic
Batuhan Taşkaya
Hill Ma
Benjamin Peterson
Yury Selivanov
Emmanuel Arias
Xtreak
Min ho Kim
Łukasz Langa
Victor Stinner
依云
sbstp
Yury Selivanov
Matthias Bussonnier
Niklas Fiekas
Erwan Le Pape
Romain Picard
Vincent Michel
twisteroid ambassador
gescheit
Quentin Dawans
Inada Naoki
Inada Naoki
Vladimir Matveev
Srinivas Thatiparthy (శ్రీనివాస్ తాటిపర్తి)
João Júnior
MartinAltmayer
jhaydaman