This runs test_asyncio sub-tests in parallel using sharding from Cinder. This suite is typically the longest-pole in runs because it is a test package with a lot of further sub-tests otherwise run serially. By breaking out the sub-tests as independent modules we can run a lot more in parallel.
After porting we can see the direct impact on a multicore system.
Without this change:
Running make test is 5 min 26 seconds
With this change:
Running make test takes 3 min 39 seconds
That'll vary based on system and parallelism. On a `-j 4` run similar to what CI and buildbot systems often do, it reduced the overall test suite completion latency by 10%.
The drawbacks are that this implementation is hacky and due to the sorting of the tests it obscures when the asyncio tests occur and involves changing CPython test infrastructure but, the wall time saved it is worth it, especially in low-core count CI runs as it pulls a long tail. The win for productivity and reserved CI resource usage is significant.
Future tests that deserve to be refactored into split up suites to benefit from are test_concurrent_futures and the way the _test_multiprocessing suite gets run for all start methods. As exposed by passing the -o flag to python -m test to get a list of the 10 longest running tests.
---------
Co-authored-by: Carl Meyer <carl@oddbird.net>
Co-authored-by: Gregory P. Smith <greg@krypto.org> [Google, LLC]
The default task name is "Task-<counter>" (if no name is passed in during Task creation).
This is initialized in `Task.__init__` (C impl) using string formatting, which can be quite slow.
Actually using the task name in real world code is not very common, so this is wasted init.
Let's defer this string formatting to the first time the name is read (in `get_name` impl),
so we don't need to pay the string formatting cost if the task name is never read.
We don't change the order in which tasks are assigned numbers (if they are) --
the number is set on task creation, as a PyLong instead of a formatted string.
Co-authored-by: Łukasz Langa <lukasz@langa.pl>
- Fix description of MAKE_CELL, which appeared to be inverted from the
actual behavior
- Fix stray ".:" (sphinx-contrib/sphinx-lint#63)
- Fix inconsistent indentation
- Add some missing code blocks
- Slight style improvements
subprocess's communicate(None) closes stdin of the child process, after
sending no (extra) data. Make asyncio variant do the same.
This fixes issues with processes that waits for EOF on stdin before
continuing.
This change has two small parts:
1. a follow-up to gh-103940 with one case I missed
2. adding a missing return that I noticed while working on related code
Using `datetime.datetime.utcnow()` and `datetime.datetime.utcfromtimestamp()` will now raise a `DeprecationWarning`.
We also have removed our internal uses of these functions and documented the change.
Clarify the docs of asyncio.loop.subprocess_exec()
Clarify the documentation of stdin, stdout and stderr arguments of
asyncio.loop.subprocess_exec().
Co-authored-by: Carol Willing <carolcode@willingconsulting.com>
Joshua Herman
Oleg Iarygin
Itamar Ostricher
Irit Katriel
Prince Roshan
Jelle Zijlstra
Mark Shannon
Barry Warsaw
Tian Gao
Paul Ganssle
Carl Meyer
Olga Matoula
Raymond Hettinger
Juhi Chandalia
Erlend E. Aasland
Wes Turner
Marek Marczykowski-Górecki
Eric Snow
Adam Turner
Finn Womack
Itayazolay
Nikita Sobolev
Inada Naoki
Reese Hyde
Lysandros Nikolaou
Stefano Rivera
Franek Magiera
Aidan Melen
Michael Blahay
180909
Hugo van Kemenade
Owain Davies
July Tikhonov
Carol Willing
Alex Waygood