Commit Graph

62 Commits

Author SHA1 Message Date
Victor Stinner eb0d359b4b
bpo-40443: Remove unused imports in stdlib (GH-19815) 2020-05-01 02:38:00 +02:00
Batuhan Taşkaya 0361556537
bpo-39481: PEP 585 for a variety of modules (GH-19423)
- concurrent.futures
- ctypes
- http.cookies
- multiprocessing
- queue
- tempfile
- unittest.case
- urllib.parse
2020-04-10 07:46:36 -07:00
Batuhan Taşkaya ac10e0c932
bpo-39360: Ensure all workers exit when finalizing a multiprocessing Pool (GH-19009)
When the pull is not used via the context manager or terminate() is called, there is a system in multiprocessing.util that handles finalization of all pools via an atexit handler (the Finalize) class. This class registers the _terminate_pool handler in the registry of finalizers of the module, and that registry is called on interpreter exit via _exit_function. The problem is that the "happy" path with the context manager or manual call to finalize() does some extra steps that _terminate_pool does not. The step that is not executed when the atexit() handler calls _terminate_pool is pinging the _change_notifier queue to unblock the maintenance threads.

This commit moves the notification to the _terminate_pool function so is called from both code paths.

Co-authored-by: Pablo Galindo <Pablogsal@gmail.com>
2020-03-15 19:45:56 +00:00
Serhiy Storchaka 2085bd0877
bpo-37116: Use PEP 570 syntax for positional-only parameters. (GH-13700) 2019-06-01 11:00:15 +03:00
Pablo Galindo 7c994549dc
bpo-35493: Use Process.sentinel instead of sleeping for polling worker status in multiprocessing.Pool (#11488)
* bpo-35493: Use Process.sentinel instead of sleeping for polling worker status in multiprocessing.Pool

* Use self-pipe pattern to avoid polling for changes

* Refactor some variable names and add comments

* Restore timeout and poll

* Use reader object only on wait()

* Recompute worker sentinels every time

* Remove timeout and use change notifier

* Refactor some methods to be overloaded by the ThreadPool, document the cache class and fix typos
2019-03-16 22:34:24 +00:00
Pablo Galindo 3766f18c52
bpo-35378: Fix multiprocessing.Pool references (GH-11627)
Changes in this commit:

1. Use a _strong_ reference between the Pool and associated iterators
2. Rework PR #8450 to eliminate a cycle in the Pool.

There is no test in this commit because any test that automatically tests this behaviour needs to eliminate the pool before joining the pool to check that the pool object is garbaged collected/does not hang. But doing this will potentially leak threads and processes (see https://bugs.python.org/issue35413).
2019-02-11 17:29:00 +00:00
Victor Stinner 9a8d1d7562
bpo-35424: emit ResourceWarning at multiprocessing.Pool destruction (GH-10974)
multiprocessing.Pool destructor now emits ResourceWarning
if the pool is still running.
2018-12-20 20:33:51 +01:00
Victor Stinner 2dfe3511fe
bpo-35491, multiprocessing: replace "RUN" with RUN (GH-11178) 2018-12-16 23:40:49 +01:00
Victor Stinner 2b417fba25
Add multiprocessing.Pool.__repr__() (GH-11137)
* Add multiprocessing.Pool.__repr__() to ease debug
* RUN, CLOSE and TERMINATE constants values are now strings rather
  than integer to ease debug
2018-12-14 11:13:18 +01:00
Victor Stinner 08c2ba0717
bpo-35477: multiprocessing.Pool.__enter__() fails if called twice (GH-11134)
multiprocessing.Pool.__enter__() now fails if the pool is not
running: "with pool:" fails if used more than once.
2018-12-13 02:15:30 +01:00
Victor Stinner 9dfc754d61
Revert "bpo-34172: multiprocessing.Pool leaks resources after being deleted (GH-8450)" (GH-10971)
This reverts commit 97bfe8d3eb.
2018-12-06 08:51:47 +01:00
Julien Palard 5d236cafd7
bpo-19675: Terminate processes if construction of a pool is failing. (GH-5614) 2018-11-04 23:40:32 +01:00
tzickel 97bfe8d3eb bpo-34172: multiprocessing.Pool leaks resources after being deleted (GH-8450)
Fix a reference issue inside multiprocessing.Pool that caused the pool to remain alive if it was deleted without being closed or terminated explicitly.
2018-10-02 23:01:23 +02:00
Antoine Pitrou ab74504346
bpo-32576: use queue.SimpleQueue in critical places (#5216)
Where a queue may be invoked from a weakref callback, we need
to use the reentrant SimpleQueue.
2018-01-18 10:38:03 +01:00
Antoine Pitrou e91c2a5086 [Trivial] Remove now redundant assert (#3245) 2017-08-30 16:57:12 +02:00
Allen W. Smith, Ph.D bd73e72b4a bpo-5001: More-informative multiprocessing error messages (#3079)
* Make error message more informative

Replace assertions in error-reporting code with more-informative version that doesn't cause confusion over where and what the error is.

* Additional clarification + get travis to check

* Change from SystemError to TypeError

As suggested in PR comment by @pitrou, changing from SystemError; TypeError appears appropriate.

* NEWS file installation; ACKS addition (will do my best to justify it by additional work)

* Making current AssertionErrors in multiprocessing more informative

* Blurb added re multiprocessing managers.py, queues.py cleanup

* Further multiprocessing cleanup - went through pool.py

* Fix two asserts in multiprocessing/util.py

* Most asserts in multiprocessing more informative

* Didn't save right version

* Further work on multiprocessing error messages

* Correct typo

* Correct typo v2

* Blasted colon... serves me right for trying to work on two things at once

* Simplify NEWS entry

* Update 2017-08-18-17-16-38.bpo-5001.gwnthq.rst

* Update 2017-08-18-17-16-38.bpo-5001.gwnthq.rst

OK, never mind.

* Corrected (thanks to pitrou) error messages for notify

* Remove extraneous backslash in docstring.
2017-08-30 00:52:18 +02:00
Serhiy Storchaka 5affd23e6f bpo-29762: More use "raise from None". (#569)
This hides unwanted implementation details from tracebacks.
2017-04-05 09:37:24 +03:00
Xiang Zhang 794623bdb2 bpo-28699: fix abnormal behaviour of pools in multiprocessing.pool (GH-693)
an exception raised at the very first of an iterable would cause pools behave abnormally
(swallow the exception or hang)
2017-03-29 11:58:54 +08:00
Antoine Pitrou 8988945cdc bpo-29861: release references to multiprocessing Pool tasks (#743)
* bpo-29861: release references to multiprocessing Pool tasks

Release references to tasks, their arguments and their results as soon
as they are finished, instead of keeping them alive until another task
arrives.

* Comments in test
2017-03-24 13:52:11 +01:00
Charles-François Natali 78f55ffc63 Issue #23992: multiprocessing: make MapResult not fail-fast upon exception. 2016-02-10 22:58:18 +00:00
Serhiy Storchaka 63623ac252 Issue #23051: multiprocessing.Pool methods imap() and imap_unordered() now
handle exceptions raised by an iterator.  Patch by Alon Diamant and Davin
Potts.
2015-03-13 08:30:33 +02:00
Serhiy Storchaka 79fbeee237 Issue #23051: multiprocessing.Pool methods imap() and imap_unordered() now
handle exceptions raised by an iterator.  Patch by Alon Diamant and Davin
Potts.
2015-03-13 08:25:26 +02:00
Serhiy Storchaka 465e60e654 Issue #22033: Reprs of most Python implemened classes now contain actual
class name instead of hardcoded one.
2014-07-25 23:36:00 +03:00
Charles-François Natali a924fc7abc Issue #21565: multiprocessing: use contex-manager protocol for synchronization
primitives.
2014-05-25 14:12:12 +01:00
Richard Oudkerk 80a5be1d84 Issue #20980: Stop wrapping exception when using ThreadPool. 2014-03-23 12:30:54 +00:00
Victor Stinner 7fa767e517 Issue #20976: pyflakes: Remove unused imports 2014-03-20 09:16:38 +01:00
Richard Oudkerk 33aaa73cad Merge. 2013-10-28 23:23:04 +00:00
Richard Oudkerk e90cedb711 Issue #19425 -- a pickling error should not cause pool to hang. 2013-10-28 23:11:58 +00:00
Richard Oudkerk b1694cf588 Issue #18999: Make multiprocessing use context objects.
This allows different parts of a program to use different methods for
starting processes without interfering with each other.
2013-10-16 16:41:56 +01:00
Richard Oudkerk 84ed9a68bd Issue #8713: Support alternative start methods in multiprocessing on Unix.
See http://hg.python.org/sandbox/sbt#spawn
2013-08-14 15:35:41 +01:00
Charles-François Natali 37cfb0a920 Issue #17914: Use os.cpu_count() instead of multiprocessing.cpu_count() where
applicable.
2013-06-28 19:25:45 +02:00
Richard Oudkerk f3a0da24f3 Merge 2013-05-06 12:13:50 +01:00
Richard Oudkerk def51ca8ea Issue #17805: Add AsyncResult alias for ApplyResult 2013-05-06 12:10:04 +01:00
Richard Oudkerk 8575783a00 Issue #13813: Embed stringification of remote traceback in local
traceback raised when pool task raises an exception.
2013-05-06 11:38:25 +01:00
Andrew Svetlov f7a17b48d7 Replace IOError with OSError (#16715) 2012-12-25 16:47:37 +02:00
Hynek Schlawack 254af2644a #16307: Fix multiprocessing.Pool.map_async not calling its callbacks
Patch by Janne Karila.
2012-10-27 12:53:02 +02:00
Benjamin Peterson 3095f4724e raise a ValueError instead of an AssertionError when pool is an invalid state 2012-09-25 12:45:42 -04:00
Richard Oudkerk d69cfe88ea Issue #15064: Implement context manager protocol for multiprocessing types 2012-06-18 17:47:52 +01:00
Richard Oudkerk 0f884273b0 Issue #15101: Make pool finalizer avoid joining current thread. 2012-06-18 16:02:49 +01:00
Richard Oudkerk f29ec4b0c8 Issue #15101: Make pool finalizer avoid joining current thread. 2012-06-18 15:54:57 +01:00
Richard Oudkerk 58ba47f97b Merge fixes for #13854 and #12157. 2012-06-07 20:38:11 +01:00
Richard Oudkerk e41682b994 Issue #12157: pool.map() does not handle empty iterable correctly
Initial patch by mouad
2012-06-06 19:04:57 +01:00
Richard Oudkerk 692130a231 Issue #12091: simplify ApplyResult and MapResult with threading.Event
Patch by Charles-François Natali
2012-05-25 13:26:53 +01:00
Richard Oudkerk 3e268aac3b Mark multiprocessing files with "Licensed to PSF under a Contributor Agreement"
instead of BSD licence.
2012-04-30 12:13:55 +01:00
Antoine Pitrou de911b2915 Issue #12708: Add starmap() and starmap_async() methods (similar to itertools.starmap()) to multiprocessing.Pool.
Patch by Hynek Schlawack.
2011-12-21 11:03:24 +01:00
Florent Xicluna 5d1155c08e Closes #13258: Use callable() built-in in the standard library. 2011-10-28 14:45:05 +02:00
Charles-François Natali f8859e1808 Issue #10332: multiprocessing: fix a race condition when a Pool is closed
before all tasks have completed.
2011-10-24 18:45:29 +02:00
Victor Stinner 2fae27b735 Issue #12285: multiprocessing.Pool() raises a ValueError if the number of
processes if negative or null.
2011-06-20 17:53:35 +02:00
Antoine Pitrou bed9a5b6b3 Issue #11814: Fix likely typo in multiprocessing.Pool._terminate(). 2011-04-11 00:20:23 +02:00
Antoine Pitrou 81dee6b4d4 Issue #8428: Fix a race condition in multiprocessing.Pool when terminating
worker processes: new processes would be spawned while the pool is being
shut down.  Patch by Charles-François Natali.
2011-04-11 00:18:59 +02:00