Commit Graph

56 Commits

Author SHA1 Message Date
Jakub Kulík 7431448b81
bpo-43498: Fix dictionary iteration error in _ExecutorManagerThread (GH-24868) 2021-11-29 14:02:56 +02:00
Logan Jones fdc0e09c33
bpo-44733: Add max_tasks_per_child to ProcessPoolExecutor (GH-27373)
Co-authored-by: Antoine Pitrou <antoine@python.org>
2021-11-20 21:19:41 +01:00
Machinexa2 d56e700d6c
Use `from` imports (GH-26594)
from imports
2021-06-08 06:47:15 -03:00
Asheesh Laroia bf2e7e55d7
bpo-40692: Run more test_concurrent_futures tests (GH-20239)
In the case of multiprocessing.synchronize() being missing, the
test_concurrent_futures test suite now skips only the tests that
require multiprocessing.synchronize().

Validate that multiprocessing.synchronize exists as part of
_check_system_limits(), allowing ProcessPoolExecutor to raise
NotImplementedError during __init__, rather than crashing with
ImportError during __init__ when creating a lock imported from
multiprocessing.synchronize.

Use _check_system_limits() to disable tests of
ProcessPoolExecutor on systems without multiprocessing.synchronize.

Running the test suite without multiprocessing.synchronize reveals
that Lib/compileall.py crashes when it uses a ProcessPoolExecutor.
Therefore, change Lib/compileall.py to call _check_system_limits()
before creating the ProcessPoolExecutor.

Note that both Lib/compileall.py and Lib/test/test_compileall.py
were attempting to sanity-check ProcessPoolExecutor by expecting
ImportError. In multiprocessing.resource_tracker, sem_unlink() is also absent
on platforms where POSIX semaphores aren't available. Avoid using
sem_unlink() if it, too, does not exist.

Co-authored-by: Pablo Galindo <Pablogsal@gmail.com>
2021-02-08 03:15:51 +00:00
Victor Stinner 90549676e0
bpo-40443: Remove unused imports in the stdlib (GH-19803) 2020-04-30 11:26:33 +02:00
Victor Stinner a4dfe8ede5
bpo-39995: Fix concurrent.futures _ThreadWakeup (GH-19760)
Fix a race condition in concurrent.futures._ThreadWakeup: access to
_ThreadWakeup is now protected with the shutdown lock.
2020-04-29 03:32:06 +02:00
Victor Stinner 1a275013d1
bpo-30966: concurrent.futures.Process.shutdown() closes queue (GH-19738)
Process.shutdown(wait=True) of concurrent.futures now closes
explicitly the result queue.
2020-04-27 20:53:37 +02:00
Kyle Stanley 1ac6e37929
bpo-39207: Spawn workers on demand in ProcessPoolExecutor (GH-19453)
Roughly based on 904e34d4e6, but with a few substantial differences.

/cc @pitrou @brianquinlan
2020-04-19 07:00:59 -07:00
Kyle Stanley b61b818d91
bpo-39812: Remove daemon threads in concurrent.futures (GH-19149)
Remove daemon threads from :mod:`concurrent.futures` by adding
an internal `threading._register_atexit()`, which calls registered functions
prior to joining all non-daemon threads. This allows for compatibility
with subinterpreters, which don't support daemon threads.
2020-03-27 20:31:22 +01:00
Thomas Moreau 0e89076247
bpo-39678: refactor queue manager thread (GH-18551) 2020-03-01 21:49:14 +01:00
Thomas Moreau a5cbab552d
bpo-39104: Fix hanging ProcessPoolExecutor on shutdown nowait with pickling failure (GH-17670)
As reported initially by @rad-pat in #6084, the following script causes a deadlock.

```
from concurrent.futures import ProcessPoolExecutor


class ObjectWithPickleError():
    """Triggers a RuntimeError when sending job to the workers"""

    def __reduce__(self):
        raise RuntimeError()


if __name__ == "__main__":
    e = ProcessPoolExecutor()
    f = e.submit(id, ObjectWithPickleError())
    e.shutdown(wait=False)
    f.result()  # Deadlock on get
```

This is caused by the fact that the main process is closing communication channels that might be necessary to the `queue_management_thread` later. To avoid this, this PR let the `queue_management_thread` manage all the closing.



https://bugs.python.org/issue39104



Automerge-Triggered-By: @pitrou
2020-02-16 10:09:26 -08:00
Kyle Stanley 339fd46cb7
bpo-39349: Add *cancel_futures* to Executor.shutdown() (GH-18057) 2020-02-02 13:49:00 +01:00
ubordignon 552ace7498 Fix typo in Lib/concurrent/futures/thread.py (GH-13953) 2019-06-15 07:43:10 -04:00
Serhiy Storchaka 142566c028
[3.9] bpo-37116: Use PEP 570 syntax for positional-only parameters. (GH-12620)
Turn deprecation warnings added in 3.8 into TypeError.
2019-06-05 18:22:31 +03:00
Brian Quinlan 39889864c0 bpo-26903: Limit ProcessPoolExecutor to 61 workers on Windows (GH-13132)
Co-Authored-By: brianquinlan <brian@sweetapp.com>
2019-05-08 14:04:53 -04:00
Brian Quinlan f7bda5c572 Don't import wait from connection, it shadows a name (GH-13112)
(lint cleanup) This import causes an argument parameter to shadow the global import name.
2019-05-07 13:31:11 -04:00
Serhiy Storchaka d53cf99dca
bpo-36542: Allow to overwrite the signature for Python functions. (GH-12705) 2019-05-06 22:40:27 +03:00
Thomas Grainger f938d8be4c Correct grammar in concurrent/futures/process.py docstring (GH-12761) 2019-04-12 09:17:17 -07:00
Serhiy Storchaka 42a139ed88
bpo-36492: Deprecate passing some arguments as keyword arguments. (GH-12637)
Deprecated passing the following arguments as keyword arguments:

- "func" in functools.partialmethod(), weakref.finalize(),
  profile.Profile.runcall(), cProfile.Profile.runcall(),
  bdb.Bdb.runcall(), trace.Trace.runfunc() and
  curses.wrapper().
- "function" in unittest.addModuleCleanup() and
  unittest.TestCase.addCleanup().
- "fn" in the submit() method of concurrent.futures.ThreadPoolExecutor
  and concurrent.futures.ProcessPoolExecutor.
- "callback" in contextlib.ExitStack.callback(),
  contextlib.AsyncExitStack.callback() and
  contextlib.AsyncExitStack.push_async_callback().
- "c" and "typeid" in the create() method of multiprocessing.managers.Server
  and multiprocessing.managers.SharedMemoryServer.
- "obj" in weakref.finalize().

Also allowed to pass arbitrary keyword arguments (even "self" and "func")
if the above arguments are passed as positional argument.
2019-04-01 09:16:35 +03:00
Dave Chevell 962bdeab19 bpo-35715: Liberate return value of _process_worker (GH-11514)
ProcessPoolExecutor workers will hold the return value of their last task in memory until the next task is received. Since the return value has already been propagated to the parent process's Future (or has been discarded by this point), the object can be safely released.
2019-03-16 22:28:51 +00:00
Mark Nemec c4b695f85e bpo-33097: Fix submit accepting callable after executor shutdown by interpreter exit (GH-6144)
Executors in concurrent.futures accepted tasks after executor was shutdown by interpreter exit. Tasks were left in PENDING state forever. This fix changes submit to instead raise a RuntimeError.
2018-04-10 19:23:14 +02:00
Thomas Moreau 095ee415ce bpo-33056 FIX leaking fd in concurrent.futures.ProcessPoolExecutor (#6084) 2018-03-12 19:18:41 +02:00
Thomas Moreau 94459fd7dc bpo-31699 Deadlocks in `concurrent.futures.ProcessPoolExecutor` with pickling error (#3895)
Fix deadlocks in :class:`concurrent.futures.ProcessPoolExecutor` when task arguments or results cause pickling or unpickling errors.
This should make sure that calls to the :class:`ProcessPoolExecutor` API always eventually return.
2018-01-05 11:15:54 +01:00
Antoine Pitrou 63ff4131af
bpo-21423: Add an initializer argument to {Process,Thread}PoolExecutor (#4241)
* bpo-21423: Add an initializer argument to {Process,Thread}PoolExecutor

* Fix docstring
2017-11-04 11:05:49 +01:00
Thomas Moreau e8c368df22 bpo-31540: Allow passing multiprocessing context to ProcessPoolExecutor (#3682) 2017-10-03 11:53:17 +02:00
Victor Stinner b713adf27a bpo-31326: ProcessPoolExecutor waits for the call queue thread (#3265)
* bpo-31326: ProcessPoolExecutor waits for the call queue thread

concurrent.futures.ProcessPoolExecutor.shutdown() now explicitly
closes the call queue. Moreover, shutdown(wait=True) now also join
the call queue thread, to prevent leaking a dangling thread.

* Fix for shutdown() being called twice.
2017-09-02 00:25:11 +02:00
Grzegorz Grzywacz 97e1b1c814 bpo-27144: concurrent.futures as_complete and map iterators do not keep reference to returned object (#1560)
* bpo-27144: concurrent.futures as_complie and map iterators do not keep
reference to returned object

* Some nits.  Improve wordings in docstrings and comments, and avoid relying on
sys.getrefcount() in tests.
2017-09-01 18:54:00 +02:00
Raymond Hettinger 15f44ab043 Issue #27895: Spelling fixes (Contributed by Ville Skyttä). 2016-08-30 10:47:49 -07:00
Martin Panter d2ad5718ad Issue #25523: Further a-to-an corrections new in 3.5 2015-11-02 04:20:33 +00:00
Antoine Pitrou 1285c9b782 Issue #21817: When an exception is raised in a task submitted to a ProcessPoolExecutor, the remote traceback is now displayed in the parent process.
Patch by Claudiu Popa.
2015-01-17 20:02:14 +01:00
Antoine Pitrou 4aae276eca Issue #11271: concurrent.futures.Executor.map() now takes a *chunksize*
argument to allow batching of tasks in child processes and improve
performance of ProcessPoolExecutor.  Patch by Dan O'Reilly.
2014-10-04 20:20:10 +02:00
Brian Quinlan 20efceb757 Issue #21362: concurrent.futures does not validate that max_workers is proper 2014-05-17 13:51:10 -07:00
Richard Oudkerk 1f2eaa97eb Fix import of SimpleQueue. 2013-10-16 17:06:22 +01:00
Ezio Melotti e0c69161bc #18741: merge with 3.3. 2013-08-17 16:13:22 +03:00
Ezio Melotti b5bc353b88 #18741: fix more typos. Patch by Févry Thibault. 2013-08-17 16:11:40 +03:00
Charles-François Natali 37cfb0a920 Issue #17914: Use os.cpu_count() instead of multiprocessing.cpu_count() where
applicable.
2013-06-28 19:25:45 +02:00
Andrew Svetlov 6b973747f3 Issue #16284: Prevent keeping unnecessary references to worker functions in concurrent.futures ThreadPoolExecutor. 2012-11-03 15:36:01 +02:00
Mark Dickinson 5ee2404d1e Fix concurrent.futures docstring typo: "Request Q" -> "Result Q". 2012-10-20 13:16:49 +01:00
Antoine Pitrou bdb1cf1ca5 Issue #12328: Fix multiprocessing's use of overlapped I/O on Windows.
Also, add a multiprocessing.connection.wait(rlist, timeout=None) function
for polling multiple objects at once.  Patch by sbt.

Complete changelist from sbt's patch:

* Adds a wait(rlist, timeout=None) function for polling multiple
  objects at once.  On Unix this is just a wrapper for
  select(rlist, [], [], timeout=None).

* Removes use of the SentinelReady exception and the sentinels argument
  to certain methods.  concurrent.futures.process has been changed to
  use wait() instead of SentinelReady.

* Fixes bugs concerning PipeConnection.poll() and messages of zero
  length.

* Fixes PipeListener.accept() to call ConnectNamedPipe() with
  overlapped=True.

* Fixes Queue.empty() and SimpleQueue.empty() so that they are
  threadsafe on Windows.

* Now PipeConnection.poll() and wait() will not modify the pipe except
  possibly by consuming a zero length message.  (Previously poll()
  could consume a partial message.)

* All of multiprocesing's pipe related blocking functions/methods are
  now interruptible by SIGINT on Windows.
2012-03-05 19:28:37 +01:00
Ross Lagerwall 69f39a53f6 Merge with 3.2 for #12364. 2012-01-08 08:42:03 +02:00
Ross Lagerwall 66e2fb68ac Issue #12364: Fix a hang in concurrent.futures.ProcessPoolExecutor. 2012-01-08 08:29:40 +02:00
Florent Xicluna 04842a84c3 Remove unused or redundant imports in concurrent.futures and multiprocessing. 2011-11-11 20:05:50 +01:00
Antoine Pitrou 0b447957c7 Close the call queue in concurrent.futures.ProcessPoolExecutor when
shutdown() is called, without waiting for the garbage collector to kick in.
2011-07-16 23:52:02 +02:00
Antoine Pitrou dc19c24832 Silence spurious "broken pipe" tracebacks when shutting down a ProcessPoolExecutor. 2011-07-16 01:51:58 +02:00
Antoine Pitrou d06a065a44 Fix potential resource leaks in concurrent.futures.ProcessPoolExecutor
by joining all queues and processes when shutdown() is called.
2011-07-16 01:13:34 +02:00
Antoine Pitrou 1c405b3e60 Followup to 51c1f2cedb96 (and issue #12456):
qsize() raises NotImplementedError on OS X, don't use it.
2011-07-03 13:17:06 +02:00
Antoine Pitrou 020436b0d4 Issue #12456: fix a possible hang on shutdown of a concurrent.futures.ProcessPoolExecutor. 2011-07-02 21:20:25 +02:00
Antoine Pitrou dd69649660 Issue #9205: concurrent.futures.ProcessPoolExecutor now detects killed
children and raises BrokenProcessPool in such a situation.  Previously it
would reliably freeze/deadlock.
2011-06-08 17:21:55 +02:00
Antoine Pitrou 441f935228 Fix typo in name of private function 2011-05-03 16:35:43 +02:00
Antoine Pitrou b87a56a519 Fix typo in name of private function 2011-05-03 16:34:42 +02:00