Remove daemon threads from :mod:`concurrent.futures` by adding
an internal `threading._register_atexit()`, which calls registered functions
prior to joining all non-daemon threads. This allows for compatibility
with subinterpreters, which don't support daemon threads.
As reported initially by @rad-pat in #6084, the following script causes a deadlock.
```
from concurrent.futures import ProcessPoolExecutor
class ObjectWithPickleError():
"""Triggers a RuntimeError when sending job to the workers"""
def __reduce__(self):
raise RuntimeError()
if __name__ == "__main__":
e = ProcessPoolExecutor()
f = e.submit(id, ObjectWithPickleError())
e.shutdown(wait=False)
f.result() # Deadlock on get
```
This is caused by the fact that the main process is closing communication channels that might be necessary to the `queue_management_thread` later. To avoid this, this PR let the `queue_management_thread` manage all the closing.
https://bugs.python.org/issue39104
Automerge-Triggered-By: @pitrou
* Fixes issue 24882
* Add news file entry for change.
* Change test_concurrent_futures.ThreadPoolShutdownTest
Adjust the shutdown test so that, after submitting three jobs
to the executor, the test checks for less than three threads,
instead of looking for exactly three threads.
If idle threads are being recycled properly, then we should have
less than three threads.
* Switched idle count to semaphor, Updated tests
As suggested by reviewer tomMoral, swapped lock-protected counter
with a semaphore to track the number of unused threads.
Adjusted test_threads_terminate to wait for completiton of the
previous future before submitting a new one (and checking the
number of threads used).
Also added a new test to confirm the thread pool can be saturated.
* Updates tests as requested by pitrou.
* Correct minor whitespace error.
* Make test_saturation faster
Wrap the callback call within the `add_done_callback` function within concurrent.futures, in order to behave in an identical manner to callbacks added to a running future are triggered once it has completed.
Deprecated passing the following arguments as keyword arguments:
- "func" in functools.partialmethod(), weakref.finalize(),
profile.Profile.runcall(), cProfile.Profile.runcall(),
bdb.Bdb.runcall(), trace.Trace.runfunc() and
curses.wrapper().
- "function" in unittest.addModuleCleanup() and
unittest.TestCase.addCleanup().
- "fn" in the submit() method of concurrent.futures.ThreadPoolExecutor
and concurrent.futures.ProcessPoolExecutor.
- "callback" in contextlib.ExitStack.callback(),
contextlib.AsyncExitStack.callback() and
contextlib.AsyncExitStack.push_async_callback().
- "c" and "typeid" in the create() method of multiprocessing.managers.Server
and multiprocessing.managers.SharedMemoryServer.
- "obj" in weakref.finalize().
Also allowed to pass arbitrary keyword arguments (even "self" and "func")
if the above arguments are passed as positional argument.
ProcessPoolExecutor workers will hold the return value of their last task in memory until the next task is received. Since the return value has already been propagated to the parent process's Future (or has been discarded by this point), the object can be safely released.
Two kind of mistakes:
1. Missed space. After concatenating there is no space between words.
2. Missed comma. Causes unintentional concatenating in a list of strings.
Use a monotonic clock to compute timeouts in :meth:`Executor.map` and :func:`as_completed`, in order to prevent timeouts from deviating when the system clock is adjusted.
This may not be sufficient on all systems. On POSIX for example, the actual waiting (e.g. in ``sem_timedwait``) is specified to rely on the CLOCK_REALTIME clock.
Future.set_result and Future.set_exception now raise InvalidStateError
if the futures are not pending or running. This mirrors the behavior
of asyncio.Future, and prevents AssertionErrors in asyncio.wrap_future
when set_result is called multiple times.
Executors in concurrent.futures accepted tasks after executor was shutdown by interpreter exit. Tasks were left in PENDING state forever. This fix changes submit to instead raise a RuntimeError.
Fix deadlocks in :class:`concurrent.futures.ProcessPoolExecutor` when task arguments or results cause pickling or unpickling errors.
This should make sure that calls to the :class:`ProcessPoolExecutor` API always eventually return.
This was possible before. GH-1560 introduced a regression after 3.6.2 got
released where only sequences were accepted now. This commit addresses this
problem.
* bpo-31326: ProcessPoolExecutor waits for the call queue thread
concurrent.futures.ProcessPoolExecutor.shutdown() now explicitly
closes the call queue. Moreover, shutdown(wait=True) now also join
the call queue thread, to prevent leaking a dangling thread.
* Fix for shutdown() being called twice.
* bpo-27144: concurrent.futures as_complie and map iterators do not keep
reference to returned object
* Some nits. Improve wordings in docstrings and comments, and avoid relying on
sys.getrefcount() in tests.
* bpo-31249: Fix ref cycle in ThreadPoolExecutor
concurrent.futures: WorkItem.run() used by ThreadPoolExecutor now
breaks a reference cycle between an exception object and the WorkItem
object. ThreadPoolExecutor.shutdown() now also clears its threads
set.
* shutdown() now only clears threads if wait is true.
* Revert changes on shutdown()
bpo-29212: Fix the ugly ThreadPoolExecutor thread name.
Fixes the newly introduced ugly default thread name for concurrent.futures
thread.ThreadPoolExecutor threads. They'll now resemble the old <=3.5
threading default Thread-x names by being named ThreadPoolExecutor-y_n.
This changes the main documentation, doc strings, source code comments, and a
couple error messages in the test suite. In some cases the word was removed
or edited some other way to fix the grammar.
Victor Stinner
Kyle Stanley
Batuhan Taşkaya
Thomas Moreau
Brian Quinlan
ubordignon
Serhiy Storchaka
Inada Naoki
Sean
Sam Martin
Thomas Grainger
Dave Chevell
orlnub123
jhaydaman
Mark Nemec
INADA Naoki
Antoine Pitrou
Łukasz Langa
Victor Stinner
Grzegorz Grzywacz
Gregory P. Smith
Raymond Hettinger
Martin Panter
Antoine Pitrou
Guido van Rossum