Commit Graph

92 Commits

Author SHA1 Message Date
Thomas Grainger e16d4ed590
gh-95166: cancel map waited on future on timeout (GH-95169)
Co-authored-by: Kumar Aditya <59607654+kumaraditya303@users.noreply.github.com>
2022-07-28 11:20:10 +02:00
Florian Bruhin 3f2dd0a7c0
test_concurrent_futures: Fix unneeded/confusing format call (#93119)
Added in 339fd46cb7 - but as noted in a comment, the test only tests ThreadPoolExecutor.
2022-07-18 19:12:40 -07:00
Victor Stinner 50e0866f87
Tests call sleeping_retry() with SHORT_TIMEOUT (#93870)
Tests now call busy_retry() and sleeping_retry() with SHORT_TIMEOUT
or LONG_TIMEOUT (of test.support), rather than hardcoded constants.

Add also WAIT_ACTIVE_CHILDREN_TIMEOUT constant to
_test_multiprocessing.
2022-06-15 18:49:14 +02:00
Victor Stinner 7e9eaad864
Add test.support.busy_retry() (#93770)
Add busy_retry() and sleeping_retry() functions to test.support.
2022-06-15 11:42:10 +02:00
Gregory P. Smith ebb37fc3fd
gh-90622: Do not spawn ProcessPool workers on demand via fork method. (#91598)
Do not spawn ProcessPool workers on demand when they spawn via fork.

This avoids potential deadlocks in the child processes due to forking from
a multithreaded process.
2022-05-08 09:20:34 -07:00
Gregory P. Smith fa4f0a134e
gh-90622: Prevent max_tasks_per_child use with a fork mp_context. (#91587)
Prevent `max_tasks_per_child` use with a "fork" mp_context to avoid deadlocks.

Also defaults to "spawn" when no mp_context is supplied for safe convenience.
2022-05-06 00:04:53 -07:00
Gregory P. Smith 7fa3a5a219
gh-91607: Fix several test_concurrent_futures tests to actually test what they claim (#91600)
* Fix test_concurrent_futures to actually test what it says.

Many ProcessPoolExecutor based tests were ignoring the mp_context
and using the default instead.  This meant we lacked proper test
coverage of all of them.

Also removes the old _prime_executor() worker delay seeding code
as it appears to have no point and causes 20-30 seconds extra
latency on this already long test.  It also interfered with some
of the refactoring to fix the above to not needlessly create their
own executor when setUp has already created an appropriate one.

* Don't import the name from multiprocessing directly to avoid confusion.

* 📜🤖 Added by blurb_it.

Co-authored-by: blurb-it[bot] <43283697+blurb-it[bot]@users.noreply.github.com>
2022-04-16 11:46:33 -07:00
Victor Stinner 9204bb72a2
bpo-46633: Skip tests on ASAN and/or MSAN builds (GH-31632)
Skip tests on ASAN and/or MSAN builds:

* multiprocessing tests
* test___all__
* test_concurrent_futures
* test_decimal
* test_peg_generator
* test_tools

The ASAN job of GitHub Actions no longer excludes these tests.
2022-03-01 15:44:08 +01:00
Kumar Aditya 7d7817cf0f
bpo-20369: concurrent.futures.wait() now deduplicates futures given a… (GH-30168)
* bpo-20369: concurrent.futures.wait() now deduplicates futures given as arg.

* 📜🤖 Added by blurb_it.

Co-authored-by: blurb-it[bot] <43283697+blurb-it[bot]@users.noreply.github.com>
2022-01-04 13:36:13 +00:00
Logan Jones fdc0e09c33
bpo-44733: Add max_tasks_per_child to ProcessPoolExecutor (GH-27373)
Co-authored-by: Antoine Pitrou <antoine@python.org>
2021-11-20 21:19:41 +01:00
nullptr 0bfa1106ac
bpo-45021: Fix a hang in forked children (GH-28007)
_global_shutdown_lock should be reinitialized in forked children
2021-09-20 11:30:19 -07:00
Serhiy Storchaka 40348acc18
bpo-45229: Remove test_main in many tests (GH-28405)
Instead of explicitly enumerate test classes for run_unittest()
use the unittest ability to discover tests. This also makes these
tests discoverable and runnable with unittest.

load_tests() can be used for dynamic generating tests and adding
doctests. setUpModule(), tearDownModule() and addModuleCleanup()
can be used for running code before and after all module tests.
2021-09-19 15:27:33 +03:00
Serhiy Storchaka 2a8127cafe
bpo-25130: Add calls of gc.collect() in tests to support PyPy (GH-28005) 2021-08-29 14:04:40 +03:00
Asheesh Laroia bf2e7e55d7
bpo-40692: Run more test_concurrent_futures tests (GH-20239)
In the case of multiprocessing.synchronize() being missing, the
test_concurrent_futures test suite now skips only the tests that
require multiprocessing.synchronize().

Validate that multiprocessing.synchronize exists as part of
_check_system_limits(), allowing ProcessPoolExecutor to raise
NotImplementedError during __init__, rather than crashing with
ImportError during __init__ when creating a lock imported from
multiprocessing.synchronize.

Use _check_system_limits() to disable tests of
ProcessPoolExecutor on systems without multiprocessing.synchronize.

Running the test suite without multiprocessing.synchronize reveals
that Lib/compileall.py crashes when it uses a ProcessPoolExecutor.
Therefore, change Lib/compileall.py to call _check_system_limits()
before creating the ProcessPoolExecutor.

Note that both Lib/compileall.py and Lib/test/test_compileall.py
were attempting to sanity-check ProcessPoolExecutor by expecting
ImportError. In multiprocessing.resource_tracker, sem_unlink() is also absent
on platforms where POSIX semaphores aren't available. Avoid using
sem_unlink() if it, too, does not exist.

Co-authored-by: Pablo Galindo <Pablogsal@gmail.com>
2021-02-08 03:15:51 +00:00
Hai Shi a7f5d93bb6
bpo-40275: Use new test.support helper submodules in tests (GH-21449) 2020-08-03 18:41:24 +02:00
Victor Stinner ddbeb2f3e0
bpo-38377: Add support.skip_if_broken_multiprocessing_synchronize() (GH-20944)
On Linux, skip tests using multiprocessing if the current user cannot
create a file in /dev/shm/ directory. Add the
skip_if_broken_multiprocessing_synchronize() function to the
test.support module.
2020-06-18 14:53:19 +02:00
Christian Heimes b022e5cffb
bpo-17258: Add requires_hashdigest to multiprocessing tests (GH-20412)
Skip some :mod:`multiprocessing` tests when MD5 hash digest is blocked.

Signed-off-by: Christian Heimes <christian@python.org>
2020-06-04 05:48:17 -07:00
Hai Shi e80697d687
bpo-40275: Adding threading_helper submodule in test.support (GH-20263) 2020-05-28 00:10:27 +02:00
Victor Stinner 5d1f32d33b
bpo-39995: Split test_concurrent_futures.test_crash() into sub-tests (GH-19739)
Now only test_error_during_result_unpickle_in_result_handler()
captures and ignores sys.stderr in the test process.

Tools like test.bisect_cmd don't support subTest() but only
work with the granularity of one method.

Remove unused ExecutorDeadlockTest._sleep_id() method.
2020-04-27 21:36:51 +02:00
Kyle Stanley 1ac6e37929
bpo-39207: Spawn workers on demand in ProcessPoolExecutor (GH-19453)
Roughly based on 904e34d4e6, but with a few substantial differences.

/cc @pitrou @brianquinlan
2020-04-19 07:00:59 -07:00
Thomas Moreau 0e89076247
bpo-39678: refactor queue manager thread (GH-18551) 2020-03-01 21:49:14 +01:00
Thomas Moreau a5cbab552d
bpo-39104: Fix hanging ProcessPoolExecutor on shutdown nowait with pickling failure (GH-17670)
As reported initially by @rad-pat in #6084, the following script causes a deadlock.

```
from concurrent.futures import ProcessPoolExecutor


class ObjectWithPickleError():
    """Triggers a RuntimeError when sending job to the workers"""

    def __reduce__(self):
        raise RuntimeError()


if __name__ == "__main__":
    e = ProcessPoolExecutor()
    f = e.submit(id, ObjectWithPickleError())
    e.shutdown(wait=False)
    f.result()  # Deadlock on get
```

This is caused by the fact that the main process is closing communication channels that might be necessary to the `queue_management_thread` later. To avoid this, this PR let the `queue_management_thread` manage all the closing.



https://bugs.python.org/issue39104



Automerge-Triggered-By: @pitrou
2020-02-16 10:09:26 -08:00
Kyle Stanley 339fd46cb7
bpo-39349: Add *cancel_futures* to Executor.shutdown() (GH-18057) 2020-02-02 13:49:00 +01:00
Brian Quinlan 884eb89d4a
bpo-39205: Tests that highlight a hang on ProcessPoolExecutor shutdown (#18221) 2020-01-27 16:50:37 -08:00
Victor Stinner 673c39331f
bpo-38546: Fix concurrent.futures test_ressources_gced_in_workers() (GH-17652)
Fix test_ressources_gced_in_workers() of test_concurrent_futures:
explicitly stop the manager to prevent leaking a child process
running in the background after the test completes.
2019-12-18 15:50:04 +01:00
Victor Stinner 9707e8e22d
bpo-38546: multiprocessing tests stop the resource tracker (GH-17641)
Multiprocessing and concurrent.futures tests now stop the resource
tracker process when tests complete.

Add ResourceTracker._stop() method to
multiprocessing.resource_tracker.

Add _cleanup_tests() helper function to multiprocessing.util: share
code between multiprocessing and concurrent.futures tests.
2019-12-17 18:37:26 +01:00
Victor Stinner 0d63bacefd
bpo-38614: Use test.support.SHORT_TIMEOUT constant (GH-17566)
Replace hardcoded timeout constants in tests with SHORT_TIMEOUT of
test.support, so it's easier to ajdust this timeout for all tests at
once.

SHORT_TIMEOUT is 30 seconds by default, but it can be longer
depending on --timeout command line option.

The change makes almost all timeouts longer, except
test_reap_children() of test_support which is made 2x shorter:
SHORT_TIMEOUT should be enough. If this test starts to fail,
LONG_TIMEOUT should be used instead.

Uniformize also "from test import support" import in some test files.
2019-12-11 11:30:03 +01:00
Victor Stinner e676244235
bpo-37421: test_concurrent_futures stops ForkServer (GH-14643)
test_concurrent_futures now explicitly stops the ForkServer instance
if it's running.
2019-07-08 10:49:11 +02:00
Victor Stinner 684cb47fff
bpo-37421: test_concurrent_futures cleans up multiprocessing (GH-14563)
test_concurrent_futures now cleans up multiprocessing to remove
immediately temporary directories created by
multiprocessing.util.get_temp_dir().

The test now uses setUpModule() and tearDownModule().
2019-07-03 11:10:30 +02:00
Serhiy Storchaka 142566c028
[3.9] bpo-37116: Use PEP 570 syntax for positional-only parameters. (GH-12620)
Turn deprecation warnings added in 3.8 into TypeError.
2019-06-05 18:22:31 +03:00
Inada Naoki 9a7e5b1b42
bpo-35279: reduce default max_workers of ThreadPoolExecutor (GH-13618) 2019-05-28 21:02:52 +09:00
Sean 904e34d4e6 bpo-24882: Let ThreadPoolExecutor reuse idle threads before creating new thread (#6375)
* Fixes issue 24882

* Add news file entry for change.

* Change test_concurrent_futures.ThreadPoolShutdownTest

Adjust the shutdown test so that, after submitting three jobs
to the executor, the test checks for less than three threads,
instead of looking for exactly three threads.

If idle threads are being recycled properly, then we should have
less than three threads.

* Switched idle count to semaphor, Updated tests

As suggested by reviewer tomMoral, swapped lock-protected counter
with a semaphore to track the number of unused threads.

Adjusted test_threads_terminate to wait for completiton of the
previous future before submitting a new one (and checking the
number of threads used).

Also added a new test to confirm the thread pool can be saturated.

* Updates tests as requested by pitrou.

* Correct minor whitespace error.

* Make test_saturation faster
2019-05-22 23:29:58 +02:00
Sam Martin 2a3a2ece50 bpo-33110: Catch errors raised when running add_done_callback on already completed futures (GH-13141)
Wrap the callback call within the `add_done_callback` function within concurrent.futures, in order to behave in an identical manner to callbacks added to a running future are triggered once it has completed.
2019-05-22 23:29:02 +02:00
Brian Quinlan 39889864c0 bpo-26903: Limit ProcessPoolExecutor to 61 workers on Windows (GH-13132)
Co-Authored-By: brianquinlan <brian@sweetapp.com>
2019-05-08 14:04:53 -04:00
Serhiy Storchaka 42a139ed88
bpo-36492: Deprecate passing some arguments as keyword arguments. (GH-12637)
Deprecated passing the following arguments as keyword arguments:

- "func" in functools.partialmethod(), weakref.finalize(),
  profile.Profile.runcall(), cProfile.Profile.runcall(),
  bdb.Bdb.runcall(), trace.Trace.runfunc() and
  curses.wrapper().
- "function" in unittest.addModuleCleanup() and
  unittest.TestCase.addCleanup().
- "fn" in the submit() method of concurrent.futures.ThreadPoolExecutor
  and concurrent.futures.ProcessPoolExecutor.
- "callback" in contextlib.ExitStack.callback(),
  contextlib.AsyncExitStack.callback() and
  contextlib.AsyncExitStack.push_async_callback().
- "c" and "typeid" in the create() method of multiprocessing.managers.Server
  and multiprocessing.managers.SharedMemoryServer.
- "obj" in weakref.finalize().

Also allowed to pass arbitrary keyword arguments (even "self" and "func")
if the above arguments are passed as positional argument.
2019-04-01 09:16:35 +03:00
Pablo Galindo 3ad8decd76
bpo-33716, test_concurrent_futures: increase timeout (GH-7828)
Increase the timeout from 1 min to 5 min.

Replace also time.time() with time.monotonic() for timeouts.
2018-06-21 12:30:37 +01:00
jhaydaman 0a28c0d12e bpo-33238: Add InvalidStateError to concurrent.futures. (GH-7056)
Future.set_result and Future.set_exception now raise InvalidStateError
if the futures are not pending or running. This mirrors the behavior
of asyncio.Future, and prevents AssertionErrors in asyncio.wrap_future
when set_result is called multiple times.
2018-05-30 10:15:06 +03:00
Mark Nemec c4b695f85e bpo-33097: Fix submit accepting callable after executor shutdown by interpreter exit (GH-6144)
Executors in concurrent.futures accepted tasks after executor was shutdown by interpreter exit. Tasks were left in PENDING state forever. This fix changes submit to instead raise a RuntimeError.
2018-04-10 19:23:14 +02:00
Leo Arias c3d9508ff2 bpo-32746: Fix multiple typos (GH-5144)
Fix typos found by codespell in docs, docstrings, and comments.
2018-02-03 19:36:10 -05:00
Thomas Moreau 94459fd7dc bpo-31699 Deadlocks in `concurrent.futures.ProcessPoolExecutor` with pickling error (#3895)
Fix deadlocks in :class:`concurrent.futures.ProcessPoolExecutor` when task arguments or results cause pickling or unpickling errors.
This should make sure that calls to the :class:`ProcessPoolExecutor` API always eventually return.
2018-01-05 11:15:54 +01:00
Antoine Pitrou 0a2ff23fe6
Silence error output in test_concurrent_futures (bpo-21423) (#4347)
* Silence error output in test_concurrent_futures (bpo-21423)
2017-11-09 15:33:43 +01:00
Antoine Pitrou 63ff4131af
bpo-21423: Add an initializer argument to {Process,Thread}PoolExecutor (#4241)
* bpo-21423: Add an initializer argument to {Process,Thread}PoolExecutor

* Fix docstring
2017-11-04 11:05:49 +01:00
Thomas Moreau e8c368df22 bpo-31540: Allow passing multiprocessing context to ProcessPoolExecutor (#3682) 2017-10-03 11:53:17 +02:00
Łukasz Langa 574562c5dd bpo-31641: Allow arbitrary iterables in `concurrent.futures.as_completed()` (#3830)
This was possible before.  GH-1560 introduced a regression after 3.6.2 got
released where only sequences were accepted now.  This commit addresses this
problem.
2017-09-29 14:33:34 -07:00
Victor Stinner 18e95b4176 bpo-31234: Join threads in tests (#3572)
Call thread.join() on threads to prevent the "dangling threads"
warning.
2017-09-14 08:43:04 -07:00
Victor Stinner 3bcf157c11 bpo-31249: Fix test_concurrent_futures dangling thread (#3521)
ProcessPoolShutdownTest.test_del_shutdown() now closes the call queue
and joins its thread, to prevent leaking a dangling thread.
2017-09-12 17:05:53 -07:00
Antoine Pitrou a6a4dc816d bpo-31370: Remove support for threads-less builds (#3385)
* Remove Setup.config
* Always define WITH_THREAD for compatibility.
2017-09-07 18:56:24 +02:00
Antoine Pitrou 2ef37607b7 Fix a c.f.as_completed() refleak previously introduced in bpo-27144 (#3270) 2017-09-03 15:09:23 +02:00
Grzegorz Grzywacz 97e1b1c814 bpo-27144: concurrent.futures as_complete and map iterators do not keep reference to returned object (#1560)
* bpo-27144: concurrent.futures as_complie and map iterators do not keep
reference to returned object

* Some nits.  Improve wordings in docstrings and comments, and avoid relying on
sys.getrefcount() in tests.
2017-09-01 18:54:00 +02:00
Victor Stinner 489d91c61d bpo-31249: test_concurrent_futures checks dangling threads (#3167)
Add a BaseTestCase class to test_concurrent_futures to check for
dangling threads and processes on all tests, not only tests using
ExecutorMixin.
2017-08-21 23:24:24 +02:00