Commit Graph

103 Commits

Author SHA1 Message Date
Gregory P. Smith ebb37fc3fd
gh-90622: Do not spawn ProcessPool workers on demand via fork method. (#91598)
Do not spawn ProcessPool workers on demand when they spawn via fork.

This avoids potential deadlocks in the child processes due to forking from
a multithreaded process.
2022-05-08 09:20:34 -07:00
Gregory P. Smith fa4f0a134e
gh-90622: Prevent max_tasks_per_child use with a fork mp_context. (#91587)
Prevent `max_tasks_per_child` use with a "fork" mp_context to avoid deadlocks.

Also defaults to "spawn" when no mp_context is supplied for safe convenience.
2022-05-06 00:04:53 -07:00
themylogin 9c204b148f
bpo-46787: Fix `ProcessPoolExecutor exception` memory leak (GH-31408) (#31408)
Do not store `ProcessPoolExecutor` work item exception traceback that prevents
exception frame locals from being garbage collected.
2022-05-02 14:24:39 -06:00
Yiannis Hadjicharalambous b11243e85e
concurrent.futures: Fix typo in docstring (#92121) 2022-05-02 09:05:26 -06:00
Serhiy Storchaka 6927632492
Remove trailing spaces (GH-31695) 2022-03-05 17:47:00 +02:00
Thomas Grainger 9b12b1b803
bpo-46522: fix concurrent.futures and io AttributeError messages (GH-30887)
Co-authored-by: blurb-it[bot] <43283697+blurb-it[bot]@users.noreply.github.com>
Co-authored-by: Andrew Svetlov <andrew.svetlov@gmail.com>
2022-02-23 02:25:00 +02:00
Kumar Aditya 7d7817cf0f
bpo-20369: concurrent.futures.wait() now deduplicates futures given a… (GH-30168)
* bpo-20369: concurrent.futures.wait() now deduplicates futures given as arg.

* 📜🤖 Added by blurb_it.

Co-authored-by: blurb-it[bot] <43283697+blurb-it[bot]@users.noreply.github.com>
2022-01-04 13:36:13 +00:00
Kumar Aditya da4b214304
bpo-42413: Replace `concurrent.futures.TimeoutError` and `asyncio.TimeoutError` with builtin `TimeoutError` (GH-30197)
Co-authored-by: Andrew Svetlov <andrew.svetlov@gmail.com>
2021-12-19 13:22:40 +02:00
Jakub Kulík 7431448b81
bpo-43498: Fix dictionary iteration error in _ExecutorManagerThread (GH-24868) 2021-11-29 14:02:56 +02:00
Logan Jones fdc0e09c33
bpo-44733: Add max_tasks_per_child to ProcessPoolExecutor (GH-27373)
Co-authored-by: Antoine Pitrou <antoine@python.org>
2021-11-20 21:19:41 +01:00
nullptr 0bfa1106ac
bpo-45021: Fix a hang in forked children (GH-28007)
_global_shutdown_lock should be reinitialized in forked children
2021-09-20 11:30:19 -07:00
Machinexa2 d56e700d6c
Use `from` imports (GH-26594)
from imports
2021-06-08 06:47:15 -03:00
Jesús Cea 32430aadad
bpo-35930: Raising an exception raised in a "future" instance will create reference cycles (#24995)
Before: https://lists.es.python.org/pipermail/general/attachments/20201229/0c14bc58/attachment-0002.png

After: https://lists.es.python.org/pipermail/general/attachments/20201229/0c14bc58/attachment-0003.png
2021-03-29 19:22:13 +02:00
Asheesh Laroia bf2e7e55d7
bpo-40692: Run more test_concurrent_futures tests (GH-20239)
In the case of multiprocessing.synchronize() being missing, the
test_concurrent_futures test suite now skips only the tests that
require multiprocessing.synchronize().

Validate that multiprocessing.synchronize exists as part of
_check_system_limits(), allowing ProcessPoolExecutor to raise
NotImplementedError during __init__, rather than crashing with
ImportError during __init__ when creating a lock imported from
multiprocessing.synchronize.

Use _check_system_limits() to disable tests of
ProcessPoolExecutor on systems without multiprocessing.synchronize.

Running the test suite without multiprocessing.synchronize reveals
that Lib/compileall.py crashes when it uses a ProcessPoolExecutor.
Therefore, change Lib/compileall.py to call _check_system_limits()
before creating the ProcessPoolExecutor.

Note that both Lib/compileall.py and Lib/test/test_compileall.py
were attempting to sanity-check ProcessPoolExecutor by expecting
ImportError. In multiprocessing.resource_tracker, sem_unlink() is also absent
on platforms where POSIX semaphores aren't available. Avoid using
sem_unlink() if it, too, does not exist.

Co-authored-by: Pablo Galindo <Pablogsal@gmail.com>
2021-02-08 03:15:51 +00:00
Shantanu 17dc1b789e
bpo-39349: Add cancel_futures to Executor.shutdown base class (GH-22023)
* Add cancel_futures parameter to the Executor base class, since it was missed in the original PR (https://github.com/python/cpython/pull/18057) that added cancel_futures.
2020-09-01 17:18:07 -04:00
Victor Stinner 90549676e0
bpo-40443: Remove unused imports in the stdlib (GH-19803) 2020-04-30 11:26:33 +02:00
Victor Stinner a4dfe8ede5
bpo-39995: Fix concurrent.futures _ThreadWakeup (GH-19760)
Fix a race condition in concurrent.futures._ThreadWakeup: access to
_ThreadWakeup is now protected with the shutdown lock.
2020-04-29 03:32:06 +02:00
Victor Stinner 1a275013d1
bpo-30966: concurrent.futures.Process.shutdown() closes queue (GH-19738)
Process.shutdown(wait=True) of concurrent.futures now closes
explicitly the result queue.
2020-04-27 20:53:37 +02:00
Kyle Stanley 1ac6e37929
bpo-39207: Spawn workers on demand in ProcessPoolExecutor (GH-19453)
Roughly based on 904e34d4e6, but with a few substantial differences.

/cc @pitrou @brianquinlan
2020-04-19 07:00:59 -07:00
Batuhan Taşkaya 0361556537
bpo-39481: PEP 585 for a variety of modules (GH-19423)
- concurrent.futures
- ctypes
- http.cookies
- multiprocessing
- queue
- tempfile
- unittest.case
- urllib.parse
2020-04-10 07:46:36 -07:00
Kyle Stanley b61b818d91
bpo-39812: Remove daemon threads in concurrent.futures (GH-19149)
Remove daemon threads from :mod:`concurrent.futures` by adding
an internal `threading._register_atexit()`, which calls registered functions
prior to joining all non-daemon threads. This allows for compatibility
with subinterpreters, which don't support daemon threads.
2020-03-27 20:31:22 +01:00
Thomas Moreau 0e89076247
bpo-39678: refactor queue manager thread (GH-18551) 2020-03-01 21:49:14 +01:00
Thomas Moreau a5cbab552d
bpo-39104: Fix hanging ProcessPoolExecutor on shutdown nowait with pickling failure (GH-17670)
As reported initially by @rad-pat in #6084, the following script causes a deadlock.

```
from concurrent.futures import ProcessPoolExecutor


class ObjectWithPickleError():
    """Triggers a RuntimeError when sending job to the workers"""

    def __reduce__(self):
        raise RuntimeError()


if __name__ == "__main__":
    e = ProcessPoolExecutor()
    f = e.submit(id, ObjectWithPickleError())
    e.shutdown(wait=False)
    f.result()  # Deadlock on get
```

This is caused by the fact that the main process is closing communication channels that might be necessary to the `queue_management_thread` later. To avoid this, this PR let the `queue_management_thread` manage all the closing.



https://bugs.python.org/issue39104



Automerge-Triggered-By: @pitrou
2020-02-16 10:09:26 -08:00
Kyle Stanley 339fd46cb7
bpo-39349: Add *cancel_futures* to Executor.shutdown() (GH-18057) 2020-02-02 13:49:00 +01:00
Brian Quinlan 242c26f53e
bpo-31783: Fix a race condition creating workers during shutdown (#13171)
* bpo-31783: Fix a race condition while creating workers during interpreter shutdown

* 📜🤖 Added by blurb_it.
2019-06-28 11:54:52 -07:00
ubordignon 552ace7498 Fix typo in Lib/concurrent/futures/thread.py (GH-13953) 2019-06-15 07:43:10 -04:00
Serhiy Storchaka 142566c028
[3.9] bpo-37116: Use PEP 570 syntax for positional-only parameters. (GH-12620)
Turn deprecation warnings added in 3.8 into TypeError.
2019-06-05 18:22:31 +03:00
Inada Naoki 9a7e5b1b42
bpo-35279: reduce default max_workers of ThreadPoolExecutor (GH-13618) 2019-05-28 21:02:52 +09:00
Sean 904e34d4e6 bpo-24882: Let ThreadPoolExecutor reuse idle threads before creating new thread (#6375)
* Fixes issue 24882

* Add news file entry for change.

* Change test_concurrent_futures.ThreadPoolShutdownTest

Adjust the shutdown test so that, after submitting three jobs
to the executor, the test checks for less than three threads,
instead of looking for exactly three threads.

If idle threads are being recycled properly, then we should have
less than three threads.

* Switched idle count to semaphor, Updated tests

As suggested by reviewer tomMoral, swapped lock-protected counter
with a semaphore to track the number of unused threads.

Adjusted test_threads_terminate to wait for completiton of the
previous future before submitting a new one (and checking the
number of threads used).

Also added a new test to confirm the thread pool can be saturated.

* Updates tests as requested by pitrou.

* Correct minor whitespace error.

* Make test_saturation faster
2019-05-22 23:29:58 +02:00
Sam Martin 2a3a2ece50 bpo-33110: Catch errors raised when running add_done_callback on already completed futures (GH-13141)
Wrap the callback call within the `add_done_callback` function within concurrent.futures, in order to behave in an identical manner to callbacks added to a running future are triggered once it has completed.
2019-05-22 23:29:02 +02:00
Brian Quinlan 39889864c0 bpo-26903: Limit ProcessPoolExecutor to 61 workers on Windows (GH-13132)
Co-Authored-By: brianquinlan <brian@sweetapp.com>
2019-05-08 14:04:53 -04:00
Brian Quinlan f7bda5c572 Don't import wait from connection, it shadows a name (GH-13112)
(lint cleanup) This import causes an argument parameter to shadow the global import name.
2019-05-07 13:31:11 -04:00
Serhiy Storchaka d53cf99dca
bpo-36542: Allow to overwrite the signature for Python functions. (GH-12705) 2019-05-06 22:40:27 +03:00
Thomas Grainger f938d8be4c Correct grammar in concurrent/futures/process.py docstring (GH-12761) 2019-04-12 09:17:17 -07:00
Serhiy Storchaka 42a139ed88
bpo-36492: Deprecate passing some arguments as keyword arguments. (GH-12637)
Deprecated passing the following arguments as keyword arguments:

- "func" in functools.partialmethod(), weakref.finalize(),
  profile.Profile.runcall(), cProfile.Profile.runcall(),
  bdb.Bdb.runcall(), trace.Trace.runfunc() and
  curses.wrapper().
- "function" in unittest.addModuleCleanup() and
  unittest.TestCase.addCleanup().
- "fn" in the submit() method of concurrent.futures.ThreadPoolExecutor
  and concurrent.futures.ProcessPoolExecutor.
- "callback" in contextlib.ExitStack.callback(),
  contextlib.AsyncExitStack.callback() and
  contextlib.AsyncExitStack.push_async_callback().
- "c" and "typeid" in the create() method of multiprocessing.managers.Server
  and multiprocessing.managers.SharedMemoryServer.
- "obj" in weakref.finalize().

Also allowed to pass arbitrary keyword arguments (even "self" and "func")
if the above arguments are passed as positional argument.
2019-04-01 09:16:35 +03:00
Dave Chevell 962bdeab19 bpo-35715: Liberate return value of _process_worker (GH-11514)
ProcessPoolExecutor workers will hold the return value of their last task in memory until the next task is received. Since the return value has already been propagated to the parent process's Future (or has been discarded by this point), the object can be safely released.
2019-03-16 22:28:51 +00:00
Serhiy Storchaka 34fd4c2019
bpo-35133: Fix mistakes when concatenate string literals on different lines. (GH-10284)
Two kind of mistakes:

1. Missed space. After concatenating there is no space between words.

2. Missed comma. Causes unintentional concatenating in a list of strings.
2018-11-05 16:20:25 +02:00
orlnub123 a94ee12c26 bpo-34819: Use a monotonic clock to compute timeouts in concurrent.futures (GH-9599)
Use a monotonic clock to compute timeouts in :meth:`Executor.map` and :func:`as_completed`, in order to prevent timeouts from deviating when the system clock is adjusted.

This may not be sufficient on all systems. On POSIX for example, the actual waiting (e.g. in ``sem_timedwait``) is specified to rely on the CLOCK_REALTIME clock.
2018-09-27 13:16:26 +02:00
jhaydaman 0a28c0d12e bpo-33238: Add InvalidStateError to concurrent.futures. (GH-7056)
Future.set_result and Future.set_exception now raise InvalidStateError
if the futures are not pending or running. This mirrors the behavior
of asyncio.Future, and prevents AssertionErrors in asyncio.wrap_future
when set_result is called multiple times.
2018-05-30 10:15:06 +03:00
Mark Nemec c4b695f85e bpo-33097: Fix submit accepting callable after executor shutdown by interpreter exit (GH-6144)
Executors in concurrent.futures accepted tasks after executor was shutdown by interpreter exit. Tasks were left in PENDING state forever. This fix changes submit to instead raise a RuntimeError.
2018-04-10 19:23:14 +02:00
Thomas Moreau 095ee415ce bpo-33056 FIX leaking fd in concurrent.futures.ProcessPoolExecutor (#6084) 2018-03-12 19:18:41 +02:00
INADA Naoki 4666ec597c
bpo-32596: Make lazy-load portable (GH-5316)
Global variables should not used as import target.
Use temporary variable instead.
2018-01-26 10:53:31 +09:00
INADA Naoki 6690bb9f17
bpo-32596: Lazy import concurrent.futures.process and thread (GH-5241) 2018-01-20 09:54:42 +09:00
Antoine Pitrou ab74504346
bpo-32576: use queue.SimpleQueue in critical places (#5216)
Where a queue may be invoked from a weakref callback, we need
to use the reentrant SimpleQueue.
2018-01-18 10:38:03 +01:00
Thomas Moreau 94459fd7dc bpo-31699 Deadlocks in `concurrent.futures.ProcessPoolExecutor` with pickling error (#3895)
Fix deadlocks in :class:`concurrent.futures.ProcessPoolExecutor` when task arguments or results cause pickling or unpickling errors.
This should make sure that calls to the :class:`ProcessPoolExecutor` API always eventually return.
2018-01-05 11:15:54 +01:00
Antoine Pitrou 63ff4131af
bpo-21423: Add an initializer argument to {Process,Thread}PoolExecutor (#4241)
* bpo-21423: Add an initializer argument to {Process,Thread}PoolExecutor

* Fix docstring
2017-11-04 11:05:49 +01:00
Thomas Moreau e8c368df22 bpo-31540: Allow passing multiprocessing context to ProcessPoolExecutor (#3682) 2017-10-03 11:53:17 +02:00
Łukasz Langa 574562c5dd bpo-31641: Allow arbitrary iterables in `concurrent.futures.as_completed()` (#3830)
This was possible before.  GH-1560 introduced a regression after 3.6.2 got
released where only sequences were accepted now.  This commit addresses this
problem.
2017-09-29 14:33:34 -07:00
Antoine Pitrou 2ef37607b7 Fix a c.f.as_completed() refleak previously introduced in bpo-27144 (#3270) 2017-09-03 15:09:23 +02:00
Victor Stinner b713adf27a bpo-31326: ProcessPoolExecutor waits for the call queue thread (#3265)
* bpo-31326: ProcessPoolExecutor waits for the call queue thread

concurrent.futures.ProcessPoolExecutor.shutdown() now explicitly
closes the call queue. Moreover, shutdown(wait=True) now also join
the call queue thread, to prevent leaking a dangling thread.

* Fix for shutdown() being called twice.
2017-09-02 00:25:11 +02:00