Commit Graph

62 Commits

Author SHA1 Message Date
Inada Naoki 9a7e5b1b42
bpo-35279: reduce default max_workers of ThreadPoolExecutor (GH-13618) 2019-05-28 21:02:52 +09:00
Sean 904e34d4e6 bpo-24882: Let ThreadPoolExecutor reuse idle threads before creating new thread (#6375)
* Fixes issue 24882

* Add news file entry for change.

* Change test_concurrent_futures.ThreadPoolShutdownTest

Adjust the shutdown test so that, after submitting three jobs
to the executor, the test checks for less than three threads,
instead of looking for exactly three threads.

If idle threads are being recycled properly, then we should have
less than three threads.

* Switched idle count to semaphor, Updated tests

As suggested by reviewer tomMoral, swapped lock-protected counter
with a semaphore to track the number of unused threads.

Adjusted test_threads_terminate to wait for completiton of the
previous future before submitting a new one (and checking the
number of threads used).

Also added a new test to confirm the thread pool can be saturated.

* Updates tests as requested by pitrou.

* Correct minor whitespace error.

* Make test_saturation faster
2019-05-22 23:29:58 +02:00
Sam Martin 2a3a2ece50 bpo-33110: Catch errors raised when running add_done_callback on already completed futures (GH-13141)
Wrap the callback call within the `add_done_callback` function within concurrent.futures, in order to behave in an identical manner to callbacks added to a running future are triggered once it has completed.
2019-05-22 23:29:02 +02:00
Brian Quinlan 39889864c0 bpo-26903: Limit ProcessPoolExecutor to 61 workers on Windows (GH-13132)
Co-Authored-By: brianquinlan <brian@sweetapp.com>
2019-05-08 14:04:53 -04:00
Serhiy Storchaka 42a139ed88
bpo-36492: Deprecate passing some arguments as keyword arguments. (GH-12637)
Deprecated passing the following arguments as keyword arguments:

- "func" in functools.partialmethod(), weakref.finalize(),
  profile.Profile.runcall(), cProfile.Profile.runcall(),
  bdb.Bdb.runcall(), trace.Trace.runfunc() and
  curses.wrapper().
- "function" in unittest.addModuleCleanup() and
  unittest.TestCase.addCleanup().
- "fn" in the submit() method of concurrent.futures.ThreadPoolExecutor
  and concurrent.futures.ProcessPoolExecutor.
- "callback" in contextlib.ExitStack.callback(),
  contextlib.AsyncExitStack.callback() and
  contextlib.AsyncExitStack.push_async_callback().
- "c" and "typeid" in the create() method of multiprocessing.managers.Server
  and multiprocessing.managers.SharedMemoryServer.
- "obj" in weakref.finalize().

Also allowed to pass arbitrary keyword arguments (even "self" and "func")
if the above arguments are passed as positional argument.
2019-04-01 09:16:35 +03:00
Pablo Galindo 3ad8decd76
bpo-33716, test_concurrent_futures: increase timeout (GH-7828)
Increase the timeout from 1 min to 5 min.

Replace also time.time() with time.monotonic() for timeouts.
2018-06-21 12:30:37 +01:00
jhaydaman 0a28c0d12e bpo-33238: Add InvalidStateError to concurrent.futures. (GH-7056)
Future.set_result and Future.set_exception now raise InvalidStateError
if the futures are not pending or running. This mirrors the behavior
of asyncio.Future, and prevents AssertionErrors in asyncio.wrap_future
when set_result is called multiple times.
2018-05-30 10:15:06 +03:00
Mark Nemec c4b695f85e bpo-33097: Fix submit accepting callable after executor shutdown by interpreter exit (GH-6144)
Executors in concurrent.futures accepted tasks after executor was shutdown by interpreter exit. Tasks were left in PENDING state forever. This fix changes submit to instead raise a RuntimeError.
2018-04-10 19:23:14 +02:00
Leo Arias c3d9508ff2 bpo-32746: Fix multiple typos (GH-5144)
Fix typos found by codespell in docs, docstrings, and comments.
2018-02-03 19:36:10 -05:00
Thomas Moreau 94459fd7dc bpo-31699 Deadlocks in `concurrent.futures.ProcessPoolExecutor` with pickling error (#3895)
Fix deadlocks in :class:`concurrent.futures.ProcessPoolExecutor` when task arguments or results cause pickling or unpickling errors.
This should make sure that calls to the :class:`ProcessPoolExecutor` API always eventually return.
2018-01-05 11:15:54 +01:00
Antoine Pitrou 0a2ff23fe6
Silence error output in test_concurrent_futures (bpo-21423) (#4347)
* Silence error output in test_concurrent_futures (bpo-21423)
2017-11-09 15:33:43 +01:00
Antoine Pitrou 63ff4131af
bpo-21423: Add an initializer argument to {Process,Thread}PoolExecutor (#4241)
* bpo-21423: Add an initializer argument to {Process,Thread}PoolExecutor

* Fix docstring
2017-11-04 11:05:49 +01:00
Thomas Moreau e8c368df22 bpo-31540: Allow passing multiprocessing context to ProcessPoolExecutor (#3682) 2017-10-03 11:53:17 +02:00
Łukasz Langa 574562c5dd bpo-31641: Allow arbitrary iterables in `concurrent.futures.as_completed()` (#3830)
This was possible before.  GH-1560 introduced a regression after 3.6.2 got
released where only sequences were accepted now.  This commit addresses this
problem.
2017-09-29 14:33:34 -07:00
Victor Stinner 18e95b4176 bpo-31234: Join threads in tests (#3572)
Call thread.join() on threads to prevent the "dangling threads"
warning.
2017-09-14 08:43:04 -07:00
Victor Stinner 3bcf157c11 bpo-31249: Fix test_concurrent_futures dangling thread (#3521)
ProcessPoolShutdownTest.test_del_shutdown() now closes the call queue
and joins its thread, to prevent leaking a dangling thread.
2017-09-12 17:05:53 -07:00
Antoine Pitrou a6a4dc816d bpo-31370: Remove support for threads-less builds (#3385)
* Remove Setup.config
* Always define WITH_THREAD for compatibility.
2017-09-07 18:56:24 +02:00
Antoine Pitrou 2ef37607b7 Fix a c.f.as_completed() refleak previously introduced in bpo-27144 (#3270) 2017-09-03 15:09:23 +02:00
Grzegorz Grzywacz 97e1b1c814 bpo-27144: concurrent.futures as_complete and map iterators do not keep reference to returned object (#1560)
* bpo-27144: concurrent.futures as_complie and map iterators do not keep
reference to returned object

* Some nits.  Improve wordings in docstrings and comments, and avoid relying on
sys.getrefcount() in tests.
2017-09-01 18:54:00 +02:00
Victor Stinner 489d91c61d bpo-31249: test_concurrent_futures checks dangling threads (#3167)
Add a BaseTestCase class to test_concurrent_futures to check for
dangling threads and processes on all tests, not only tests using
ExecutorMixin.
2017-08-21 23:24:24 +02:00
Victor Stinner 3df9dec425 bpo-30845: Enhance test_concurrent_futures cleanup (#2564)
* bpo-30845: reap_children() now logs warnings

* bpo-30845: Enhance test_concurrent_futures cleanup

In setUp() and tearDown() methods of test_concurrent_futures tests,
make sure that tests don't leak threads nor processes. Clear
explicitly the reference to the executor to make it that it's
destroyed (to prevent "dangling threads" warning).
2017-07-04 13:14:04 +02:00
Gregory P. Smith a3d91b43c2 bpo-29212: Fix the ugly repr() ThreadPoolExecutor thread name. (#2315)
bpo-29212: Fix the ugly ThreadPoolExecutor thread name.

Fixes the newly introduced ugly default thread name for concurrent.futures
thread.ThreadPoolExecutor threads.  They'll now resemble the old <=3.5
threading default Thread-x names by being named ThreadPoolExecutor-y_n.
2017-06-21 23:41:13 -07:00
Raymond Hettinger 15f44ab043 Issue #27895: Spelling fixes (Contributed by Ville Skyttä). 2016-08-30 10:47:49 -07:00
Gregory P. Smith 50abe877ee Issue #27664: Add to concurrent.futures.thread.ThreadPoolExecutor()
the ability to specify a thread name prefix.
2016-08-07 10:19:20 -07:00
Martin Panter 46f50726a0 Issue #27076: Doc, comment and tests spelling fixes
Most fixes to Doc/ and Lib/ directories by Ville Skyttä.
2016-05-26 05:35:26 +00:00
Berker Peksag ce643913a9 Issue #9517: Move script_helper to the support package.
Patch by Christie Wilson.
2015-05-06 06:33:17 +03:00
Antoine Pitrou 1285c9b782 Issue #21817: When an exception is raised in a task submitted to a ProcessPoolExecutor, the remote traceback is now displayed in the parent process.
Patch by Claudiu Popa.
2015-01-17 20:02:14 +01:00
Antoine Pitrou 4aae276eca Issue #11271: concurrent.futures.Executor.map() now takes a *chunksize*
argument to allow batching of tasks in child processes and improve
performance of ProcessPoolExecutor.  Patch by Dan O'Reilly.
2014-10-04 20:20:10 +02:00
Guido van Rossum cfd4661e78 Closes #21527: Add default number of workers to ThreadPoolExecutor. (Claudiu Popa.) 2014-09-02 10:39:18 -07:00
R David Murray 475a476237 Eliminate DeprecationWarning in test_concurrent_futures. 2014-06-11 16:25:05 -04:00
Brian Quinlan 20efceb757 Issue #21362: concurrent.futures does not validate that max_workers is proper 2014-05-17 13:51:10 -07:00
Guido van Rossum e6994ff6e3 Fix issue #20367: concurrent.futures.as_completed() for duplicate arguments.
Patch by Glenn Langford.
2014-01-26 09:57:51 -08:00
Antoine Pitrou 355dda8d17 Issue #14407: Fix unittest test discovery in test_concurrent_futures. 2013-10-15 23:24:44 +02:00
Antoine Pitrou 9816a1e643 Issue #14407: Fix unittest test discovery in test_concurrent_futures. 2013-10-15 23:23:32 +02:00
Andrew Svetlov 6b973747f3 Issue #16284: Prevent keeping unnecessary references to worker functions in concurrent.futures ThreadPoolExecutor. 2012-11-03 15:36:01 +02:00
Antoine Pitrou 8b34b53c52 Issue #14406: Fix a race condition when using `concurrent.futures.wait(return_when=ALL_COMPLETED)`.
Patch by Matt Joiner.
2012-03-31 20:25:22 +02:00
Antoine Pitrou f70401e842 Issue #14406: Fix a race condition when using `concurrent.futures.wait(return_when=ALL_COMPLETED)`.
Patch by Matt Joiner.
2012-03-31 20:23:30 +02:00
Ross Lagerwall 66e2fb68ac Issue #12364: Fix a hang in concurrent.futures.ProcessPoolExecutor. 2012-01-08 08:29:40 +02:00
Ross Lagerwall 69f39a53f6 Merge with 3.2 for #12364. 2012-01-08 08:42:03 +02:00
Antoine Pitrou 6b4883dec0 PEP 3151 / issue #12555: reworking the OS and IO exception hierarchy. 2011-10-12 02:54:14 +02:00
Antoine Pitrou d06a065a44 Fix potential resource leaks in concurrent.futures.ProcessPoolExecutor
by joining all queues and processes when shutdown() is called.
2011-07-16 01:13:34 +02:00
Antoine Pitrou 8f629059db Make sure to reap worker threads and processes at the end of test_concurrent_futures 2011-07-15 20:26:35 +02:00
Antoine Pitrou 9470ab43a9 Make sure to reap worker threads and processes at the end of test_concurrent_futures 2011-07-15 20:25:20 +02:00
Antoine Pitrou 020436b0d4 Issue #12456: fix a possible hang on shutdown of a concurrent.futures.ProcessPoolExecutor. 2011-07-02 21:20:25 +02:00
Antoine Pitrou dd69649660 Issue #9205: concurrent.futures.ProcessPoolExecutor now detects killed
children and raises BrokenProcessPool in such a situation.  Previously it
would reliably freeze/deadlock.
2011-06-08 17:21:55 +02:00
Brian Quinlan 1ae2998f3e Increases some test timeouts to fix issue 11864. 2011-05-30 21:52:24 +10:00
Brian Quinlan f007876bd6 Issue #11777: Executor.map does not submit futures until iter.next() is called 2011-04-08 08:19:33 +10:00
Antoine Pitrou aebac0b55a Add tests for the atexit hook in concurrent.futures (part of #11635) 2011-03-24 15:47:39 +01:00
Antoine Pitrou 8e5e942c59 Use shorter timings in test_concurrent_futures to make it a bit faster
(it's still very slow)
2011-03-22 18:30:30 +01:00
Brian Quinlan 1d1df8257f Removes the 'Call' class which is used to control execution order and is unreliable on Windows 2011-01-03 02:56:39 +00:00