Commit Graph

687 Commits

Author SHA1 Message Date
Andrew Svetlov e5d67f1e31
[3.7] bpo-37279: Fix asyncio sendfile support when extra data are sent in fallback mode. (GH-14075). (GH-14103)
(cherry picked from commit ef2152354f)

Co-authored-by: Andrew Svetlov <andrew.svetlov@gmail.com>
2019-06-15 14:56:27 +03:00
Miss Islington (bot) 5e97450d83
Use threadpool for reading from file in sendfile fallback mode (GH-14076)
(cherry picked from commit 0237265e82)

Co-authored-by: Andrew Svetlov <andrew.svetlov@gmail.com>
2019-06-15 04:27:10 -07:00
Miss Islington (bot) 5d1d4e3179
bpo-36607: Eliminate RuntimeError raised by asyncio.all_tasks() (GH-13971)
If internal tasks weak set is changed by another thread during iteration.

https://bugs.python.org/issue36607
(cherry picked from commit 65aa64fae8)

Co-authored-by: Andrew Svetlov <andrew.svetlov@gmail.com>
2019-06-11 09:01:20 -07:00
Andrew Svetlov a79b6c578f
[3.7] bpo-37035: Don't log OSError (GH-13548) (#13594)
https://bugs.python.org/issue37035.
(cherry picked from commit 1f39c28e48)

Co-authored-by: Andrew Svetlov <andrew.svetlov@gmail.com>
2019-05-27 18:52:05 +03:00
Miss Islington (bot) 3887932e10
bpo-35721: Close socket pair if Popen in _UnixSubprocessTransport fails (GH-11553)
This slightly expands an existing test case `test_popen_error` to trigger a `ResourceWarning` and fixes it.

https://bugs.python.org/issue35721
(cherry picked from commit 9932fd91e8)

Co-authored-by: Niklas Fiekas <niklas.fiekas@backscattering.de>
2019-05-20 05:35:56 -07:00
Miss Islington (bot) 94704048e2
bpo-35545: Fix asyncio discarding IPv6 scopes (GH-11271)
This PR proposes a solution to [bpo-35545](https://bugs.python.org/issue35545) by adding an optional `flowinfo` and `scopeid` to `asyncio.base_events._ipaddr_info` to carry the full address information into `_ipaddr_info` and avoid discarding IPv6 specific information.

Changelog entry & regression tests to come.

https://bugs.python.org/issue35545
(cherry picked from commit ac8eb8f36b)

Co-authored-by: Erwan Le Pape <lepaperwan@users.noreply.github.com>
2019-05-17 02:05:19 -07:00
Victor Stinner c647ad9b51
Revert "bpo-36801: Fix waiting in StreamWriter.drain for closing SSL transport (GH-13098)" (GH-13328)
This reverts commit 93aa57ac65.
2019-05-14 22:29:54 +02:00
Miss Islington (bot) 93aa57ac65
bpo-36801: Fix waiting in StreamWriter.drain for closing SSL transport (GH-13098)
https://bugs.python.org/issue36801
(cherry picked from commit 1cc0ee7d9f)

Co-authored-by: Andrew Svetlov <andrew.svetlov@gmail.com>
2019-05-07 14:48:35 -07:00
Miss Islington (bot) 299f69c24c
bpo-35125: remove inner callback on outer cancellation in asyncio shield (GH-10340)
When the future returned by shield is cancelled, its completion callback of the
inner future is not removed. This makes the callback list of inner inner future
grow each time a shield is created and cancelled.

This change unregisters the callback from the inner future when the outer
future is cancelled.

https://bugs.python.org/issue35125
(cherry picked from commit b35acc5b3a)

Co-authored-by: Romain Picard <romain.picard@oakbits.com>
2019-05-07 12:38:00 -07:00
Miss Islington (bot) 19ca5b500a
bpo-31922: Do not connect UDP sockets when broadcast is allowed (GH-423)
*Moved from python/asyncioGH-493.*

This PR fixes issue python/asyncioGH-480, as explained in [this comment](https://github.com/python/asyncio/issues/480GH-issuecomment-278703828).

The `_SelectorDatagramTransport.sendto` method has to be modified ~~so `_sock.sendto` is used in all cases (because it is tricky to reliably tell if the socket is connected or not). Could that be an issue for connected sockets?~~ *EDIT* ... so `_sock.send` is used only if `_sock` is connected.

It also protects `socket.getsockname` against `OSError` in `_SelectorTransport`. This might happen on Windows if the socket is not connected (e.g. for UDP broadcasting).

https://bugs.python.org/issue31922
(cherry picked from commit 63deaa5b70)

Co-authored-by: Vincent Michel <vxgmichel@gmail.com>
2019-05-07 10:45:53 -07:00
Miss Islington (bot) 769ac7e7b8
bpo-36613: call remove_done_callback if exception (GH-12800)
Call remove_done_callback() in finally block.

https://bugs.python.org/issue36613
(cherry picked from commit c1964e9e21)

Co-authored-by: gescheit <gescheit12@gmail.com>
2019-05-03 08:35:52 -07:00
Miss Islington (bot) 7f7485c060
bpo-34745: Fix asyncio sslproto memory issues (GH-12386)
* Fix handshake timeout leak in asyncio/sslproto

Refs MagicStack/uvloopGH-222

* Break circular ref _SSLPipe <-> SSLProtocol

* bpo-34745: Fix asyncio ssl memory leak

* Break circular ref SSLProtocol <-> UserProtocol

* Add NEWS entry
(cherry picked from commit f683f46425)

Co-authored-by: Fantix King <fantix.king@gmail.com>
2019-03-17 16:09:14 -07:00
Miss Islington (bot) c9f26714d5
bpo-23846: Fix ProactorEventLoop._write_to_self() (GH-11566)
asyncio.ProactorEventLoop now catchs and logs send errors when the
self-pipe is full: BaseProactorEventLoop._write_to_self() now catchs
and logs OSError exceptions, as done by
BaseSelectorEventLoop._write_to_self().
(cherry picked from commit c9f872b0bd)

Co-authored-by: Victor Stinner <vstinner@redhat.com>
2019-01-15 05:17:05 -08:00
Victor Stinner d5a6adf628
[3.7] bpo-34323: Enhance IocpProactor.close() log (GH-11565)
* IocpProactor: prevent modification if closed (GH-11494)

* _wait_for_handle(), _register() and _unregister() methods of
  IocpProactor now raise an exception if closed
* Add "closed" to IocpProactor.__repr__()
* Simplify IocpProactor.close()

(cherry picked from commit 9b07681c09)

* bpo-34323: Enhance IocpProactor.close() log (GH-11555)

IocpProactor.close() now uses time to decide when to log: wait 1
second before the first log, then log every second. Log also the
number of seconds since close() was called.

(cherry picked from commit b1e45739d8)

* bpo-34323: Enhance IocpProactor.close() log again (GH-11563)

Add repr(self) to the log to display the number of pending overlapped
in the log.

(cherry picked from commit b91140fdb1)
2019-01-15 13:05:28 +01:00
Miss Islington (bot) 88bd26a72e
bpo-35682: Fix _ProactorBasePipeTransport._force_close() (GH-11462)
bpo-32622, bpo-35682: Fix asyncio.ProactorEventLoop.sendfile(): don't
attempt to set the result of an internal future if it's already done.

Fix asyncio _ProactorBasePipeTransport._force_close(): don't set the
result of _empty_waiter if it's already done.
(cherry picked from commit 80fda712c8)

Co-authored-by: Victor Stinner <vstinner@redhat.com>
2019-01-07 18:15:25 -08:00
Miss Islington (bot) fe91e9ba08 [3.7] bpo-35380: Enable TCP_NODELAY for proactor event loop (GH-10867) (GH-10872)
* bpo-35380: Enable TCP_NODELAY for proactor event loop (GH-10867)
(cherry picked from commit 3bc0ebab17)

Co-authored-by: Andrew Svetlov <andrew.svetlov@gmail.com>
2018-12-03 23:11:41 +02:00
Miss Islington (bot) 5dbb1b7df1 bpo-34970: Protect tasks weak set manipulation in asyncio.all_tasks() (GH-9837) (GH-9849)
https://bugs.python.org/issue34970
(cherry picked from commit 97cf082872)

Co-authored-by: Andrew Svetlov <andrew.svetlov@gmail.com>
2018-10-13 22:26:47 +03:00
Miss Islington (bot) 41e5ec377b
bpo-34769: Thread safety for _asyncgen_finalizer_hook(). (GH-9716)
(cherry picked from commit c880ffe7d2)

Co-authored-by: twisteroid ambassador <twisteroidambassador@users.noreply.github.com>
2018-10-09 09:03:35 -07:00
Miss Islington (bot) 172a81e42b [3.7] bpo-34263 Cap timeout submitted to epoll/select etc. to one day. (GH-8532) (GH-8586) 2018-07-31 11:29:07 -04:00
Miss Islington (bot) d5c75be555
bpo-33833: Fix ProactorSocketTransport AssertionError (GH-7893)
(cherry picked from commit 9045199c5a)

Co-authored-by: twisteroid ambassador <twisteroidambassador@users.noreply.github.com>
2018-07-30 13:04:30 -07:00
Miss Islington (bot) 3ed4414121
bpo-27500: Fix static version of getaddrinfo to resolve IPv6 (GH-7993)
(cherry picked from commit d904c238ca)

Co-authored-by: Yury Selivanov <yury@magic.io>
2018-06-28 19:16:48 -07:00
Miss Islington (bot) 21f4c780a1
bpo-30805: Avoid race condition with debug logging (GH-7545)
Supersedes https://github.com/python/cpython/pull/2490
(cherry picked from commit 12f482e0ae)

Co-authored-by: Yury Selivanov <yury@magic.io>
2018-06-08 15:42:07 -07:00
Miss Islington (bot) 17beebcc8b
bpo-33694: Fix typo in helper function name (GH-7522)
_feed_data_to_bufferred_proto() renamed to
_feed_data_to_buffered_proto() ("bufferred" => "buffered").

Typo spotted by Nathaniel J. Smith.
(cherry picked from commit ff6c077292)

Co-authored-by: Victor Stinner <vstinner@redhat.com>
2018-06-08 01:48:50 -07:00
Miss Islington (bot) 0738443a5b
bpo-33792: Add selector and proactor windows policies (GH-7487)
(cherry picked from commit 8f4042964d)

Co-authored-by: Yury Selivanov <yury@magic.io>
2018-06-07 18:31:50 -07:00
Miss Islington (bot) 8fa398d5ce bpo-33694: Fix race condition in asyncio proactor (GH-7498) (GH-7499)
The cancellation of an overlapped WSARecv() has a race condition
which causes data loss because of the current implementation of
proactor in asyncio.

No longer cancel overlapped WSARecv() in _ProactorReadPipeTransport
to work around the race condition.

Remove the optimized recv_into() implementation to get simple
implementation of pause_reading() using the single _pending_data
attribute.

Move _feed_data_to_bufferred_proto() to protocols.py.

Remove set_protocol() method which became useless.
(cherry picked from commit 79790bc35f)

Co-authored-by: Victor Stinner <vstinner@redhat.com>
2018-06-08 00:49:34 +02:00
Miss Islington (bot) 79c7e57c46 bpo-33769: start_tls: Fix error message; cancel callbacks on error (GH-7403) (GH-7428)
In addition to that, mark SSLTransport as "closed" in its "abort()" method to prevent bogus warnings.
(cherry picked from commit 415bc46a78)

Co-authored-by: Yury Selivanov <yury@magic.io>
2018-06-05 10:18:20 -04:00
Miss Islington (bot) 87936d03cb bpo-33734: asyncio/ssl: a bunch of bugfixes (GH-7321) (GH-7396)
* Fix AttributeError (not all SSL exceptions have 'errno' attribute)

* Increase default handshake timeout from 10 to 60 seconds
* Make sure start_tls can be cancelled correctly
* Make sure any error in SSLProtocol gets propagated (instead of just being logged)
(cherry picked from commit 9602643120)

Co-authored-by: Yury Selivanov <yury@magic.io>
2018-06-04 12:05:46 -04:00
Yury Selivanov af9cda9845
Revert "bpo-22087: Fix Policy.get_event_loop() to detect fork (GH-7208)" (GH-7233)
This reverts commit 2a7eb0b531.
2018-05-29 20:56:33 -04:00
Miss Islington (bot) d8948c5e09
bpo-32751: Wait for task cancellation in asyncio.wait_for() (GH-7216)
Currently, asyncio.wait_for(fut), upon reaching the timeout deadline,
cancels the future and returns immediately.  This is problematic for
when *fut* is a Task, because it will be left running for an arbitrary
amount of time.  This behavior is iself surprising and may lead to
related bugs such as the one described in bpo-33638:

    condition = asyncio.Condition()
    async with condition:
        await asyncio.wait_for(condition.wait(), timeout=0.5)

Currently, instead of raising a TimeoutError, the above code will fail
with `RuntimeError: cannot wait on un-acquired lock`, because
`__aexit__` is reached _before_ `condition.wait()` finishes its
cancellation and re-acquires the condition lock.

To resolve this, make `wait_for` await for the task cancellation.
The tradeoff here is that the `timeout` promise may be broken if the
task decides to handle its cancellation in a slow way.  This represents
a behavior change and should probably not be back-patched to 3.6 and
earlier.
(cherry picked from commit e2b340ab41)

Co-authored-by: Elvis Pranskevichus <elvis@magic.io>
2018-05-29 15:37:06 -07:00
Miss Islington (bot) 036434273e bpo-32684: Fix gather to propagate cancel of itself with return_exceptions (GH-7209) (#7222)
(cherry picked from commit 863b674909)

Co-authored-by: Yury Selivanov <yury@magic.io>
2018-05-29 18:29:12 -04:00
Miss Islington (bot) 2a7eb0b531
bpo-22087: Fix Policy.get_event_loop() to detect fork (GH-7208)
Original patch by Dan O'Reilly.
(cherry picked from commit 5d97b7bcc1)

Co-authored-by: Yury Selivanov <yury@magic.io>
2018-05-29 14:09:17 -07:00
Miss Islington (bot) 441afbd9c3
bpo-33353: test_asyncio use set_write_buffer_limits() (GH-7200)
Use transport.set_write_buffer_limits() in sendfile tests of
test_asyncio to make sure that the protocol is paused after sending
4 KiB. Previously,
test_sendfile_fallback_close_peer_in_the_middle_of_receiving() failed
on FreeBSD if the DATA was smaller than the default limit of 64 KiB.
(cherry picked from commit 9551f77192)

Co-authored-by: Victor Stinner <vstinner@redhat.com>
2018-05-29 07:21:46 -07:00
Miss Islington (bot) be5d616e55
bpo-33654: Support protocol type switching in SSLTransport.set_protocol() (GH-7194)
(cherry picked from commit 2179022d94)

Co-authored-by: Yury Selivanov <yury@magic.io>
2018-05-29 02:19:09 -07:00
Miss Islington (bot) eca085993c
bpo-33674: Pause the transport as early as possible (GH-7192)
(cherry picked from commit f295587c45)

Co-authored-by: Yury Selivanov <yury@magic.io>
2018-05-28 22:59:03 -07:00
Miss Islington (bot) 0dd8fd0358 bpo-33674: asyncio: Fix SSLProtocol race (GH-7175) (#7187)
Fix a race condition in SSLProtocol.connection_made() of
asyncio.sslproto: start immediately the handshake instead of using
call_soon(). Previously, data_received() could be called before the
handshake started, causing the handshake to hang or fail.
(cherry picked from commit be00a5583a)

Co-authored-by: Victor Stinner <vstinner@redhat.com>
2018-05-28 21:44:16 -04:00
Miss Islington (bot) 420092e255 bpo-32410: Avoid blocking on file IO in sendfile fallback code (GH-7172) (#7182)
(cherry picked from commit 7165754b6b)

Co-authored-by: Yury Selivanov <yury@magic.io>
2018-05-28 21:42:45 -04:00
Miss Islington (bot) ddc613f203
bpo-32610: Fix asyncio.all_tasks() to return only pending tasks. (GH-7174)
(cherry picked from commit 416c1ebd98)

Co-authored-by: Yury Selivanov <yury@magic.io>
2018-05-28 17:16:43 -07:00
Miss Islington (bot) 8d8b86116f
bpo-33469: RuntimeError after closing loop that used run_in_executor (GH-7171)
(cherry picked from commit fdccfe09f0)

Co-authored-by: Yury Selivanov <yury@magic.io>
2018-05-28 15:50:02 -07:00
Miss Islington (bot) 075c662086 bpo-33672: Fix Task.__repr__ crash with Cython's bogus coroutines (GH-7161) (GH-7173)
(cherry picked from commit 989b9e0e6d)

Co-authored-by: Yury Selivanov <yury@magic.io>
2018-05-28 18:31:07 -04:00
Miss Islington (bot) bc3a002e7d
bpo-33654: Support BufferedProtocol in set_protocol() and start_tls() (GH-7130)
In this commit:

* Support BufferedProtocol in set_protocol() and start_tls()
* Fix proactor to cancel readers reliably
* Update tests to be compatible with OpenSSL 1.1.1
* Clarify BufferedProtocol docs
* Bump TLS tests timeouts to 60 seconds; eliminate possible race from start_serving
* Rewrite test_start_tls_server_1
(cherry picked from commit dbf102271f)

Co-authored-by: Yury Selivanov <yury@magic.io>
2018-05-28 11:50:45 -07:00
Miss Islington (bot) f8fdb368e3 bpo-33505: Optimize asyncio.ensure_future by reordering if conditions (GH-6836) (#7162)
(cherry picked from commit e549c4be5f)

Co-authored-by: jimmylai <albert_chs@yahoo.com.tw>
2018-05-28 14:42:50 -04:00
Miss Islington (bot) 1f21ae710d bpo-31647: Fix write_eof() after close() for SelectorSocketTransport (GH-7149) (GH-7153)
Fixed bug where calling write_eof() on a _SelectorSocketTransport after
it's already closed raises AttributeError.
(cherry picked from commit 23f587e395)

Co-authored-by: twisteroid ambassador <twisteroidambassador@users.noreply.github.com>
2018-05-28 12:04:09 -04:00
Miss Islington (bot) b8b800090f bpo-33263: Fix FD leak in _SelectorSocketTransport (GH-6450) (#7022)
* bpo-33263 Fix FD leak in _SelectorSocketTransport. (GH-6450)

Under particular circumstances _SelectorSocketTransport can try to add a reader
even the transport is already being closed. This can lead to FD leak and
invalid stated of the following connections. Fixed the SelectorSocketTransport
to add the reader only if the trasport is still active.
(cherry picked from commit a84d0b361a)

Co-authored-by: Vlad Starostin <drtyrsa@yandex.ru>
2018-05-21 11:35:25 +03:00
Miss Islington (bot) b85115ed4c [3.7] bpo-32262: Fix typo in f-string (GH-7016)
Fix typo from commit 6370f345e1

(cherry picked from commit d361e99868)
Co-authored-by: Christian Heimes <christian@python.org>
2018-05-20 20:22:50 +02:00
Miss Islington (bot) 52d1741c95 bpo-33584: Fix several minor bugs in asyncio. (GH-7003) (GH-7005)
Fix the following bugs in the C implementation:

* get_future_loop() silenced all exceptions raised when look up the get_loop
  attribute, not just an AttributeError.
* enter_task() silenced all exceptions raised when look up the current task,
  not just a KeyError.
* repr() was called for a borrowed link in enter_task() and task_step_impl().
* str() was used instead of repr() in formatting one error message (in
  Python implementation too).
* There where few reference leaks in error cases.
(cherry picked from commit 6655354afc)

Co-authored-by: Serhiy Storchaka <storchaka@gmail.com>
2018-05-20 17:34:28 +03:00
Miss Islington (bot) 28ea38b97b
bpo-26819: Prevent proactor double read on resume (GH-6921)
The proactor event loop has a race condition when reading with
pausing/resuming. `resume_reading()` unconditionally schedules the read
function to read from the current future. If `resume_reading()` was
called before the previously scheduled done callback fires, this results
in two attempts to get the data from the most recent read and an
assertion failure. This commit tracks whether or not `resume_reading`
needs to reschedule the callback to restart the loop, preventing a
second attempt to read the data.
(cherry picked from commit 4151061855)

Co-authored-by: CtrlZvi <viz+github@flippedperspective.com>
2018-05-20 03:57:32 -07:00
Serhiy Storchaka fd93666c8a
[3.7] Fix duplicating words words. (GH-6296) (GH-6297)
Most of them have been added in 3.7.
(cherry picked from commit bac2d5ba30)
2018-03-28 23:05:24 +03:00
Miss Islington (bot) 211c0dbc6a Fix senfile typo (GH-6265) (#6274)
* Also in docs
(cherry picked from commit 65a34709f6)

Co-authored-by: Sam Dunster <me@sdunster.com>
2018-03-27 18:34:15 -07:00
Miss Islington (bot) bf0d116517 bpo-33037: Skip sending/receiving after SSL transport closing (GH-6044) (GH-6057)
* Skip write()/data_received() if sslpipe is destroyed
(cherry picked from commit 5e80a71ab6)

Co-authored-by: Andrew Svetlov <andrew.svetlov@gmail.com>
2018-03-10 18:27:01 +02:00
Miss Islington (bot) 632c1cb571
bpo-32622: Native sendfile on windows (GH-5565)
* Support sendfile on Windows Proactor event loop naively.
(cherry picked from commit a19fb3c6aa)

Co-authored-by: Andrew Svetlov <andrew.svetlov@gmail.com>
2018-02-25 09:10:58 -08:00