Commit Graph

289 Commits

Author SHA1 Message Date
Victor Stinner e42339d3a0 [3.6] bpo-30595: Fix multiprocessing.Queue.get(timeout) (#2027) (#2881)
* bpo-30595: Fix multiprocessing.Queue.get(timeout) (#2027)

multiprocessing.Queue.get() with a timeout now polls its reader in
non-blocking mode if it succeeded to aquire the lock but the acquire
took longer than the timeout.

Co-Authored-By: Grzegorz Grzywacz <grzgrzgrz3@gmail.com>
(cherry picked from commit 1b7863c3b6)

* bpo-30595: Increase test_queue_feeder_donot_stop_onexc() timeout (#2148)

_test_multiprocessing.test_queue_feeder_donot_stop_onexc() now uses a
timeout of 1 second on Queue.get(), instead of 0.1 second, for slow
buildbots.
(cherry picked from commit 8f6eeaf21c)
2017-07-26 05:38:22 +02:00
Victor Stinner d0adfb25c5 [3.6] bpo-26762, bpo-31019: Backport multiprocessing fixes from master to 3.6 (#2879)
* bpo-26762: Avoid daemon process in _test_multiprocessing (#2842)

test_level() of _test_multiprocessing._TestLogging now uses regular
processes rather than daemon processes to prevent zombi processes
(to not "leak" processes).
(cherry picked from commit 06634950c5)

* test_multiprocessing: Fix dangling process/thread (#2850)

bpo-26762: Fix more dangling processes and threads in
test_multiprocessing:

* Queue: call close() followed by join_thread()
* Process: call join() or self.addCleanup(p.join)
(cherry picked from commit d7e64d9934)

* test_multiprocessing detects dangling per test case (#2841)

bpo-26762: test_multiprocessing now detects dangling processes and
threads per test case classes:

* setUpClass()/tearDownClass() of mixin classes now check if
  multiprocessing.process._dangling or threading._dangling was
  modified to detect "dangling" processses and threads.
* ManagerMixin.tearDownClass() now also emits a warning if it still
  has more than one active child process after 5 seconds.
* tearDownModule() now checks for dangling processes and threads
  before sleep 500 ms. And it now only sleeps if there is a least one
  dangling process or thread.
(cherry picked from commit ffb49408f0)

* bpo-26762: test_multiprocessing close more queues (#2855)

* Close explicitly queues to make sure that we don't leave dangling
  threads
* test_queue_in_process(): remove unused queue
* test_access() joins also the process to fix a random warning
(cherry picked from commit b4c52966c8)

* bpo-31019: Fix multiprocessing.Process.is_alive() (#2875)

multiprocessing.Process.is_alive() now removes the process from the
_children set if the process completed.

The change prevents leaking "dangling" processes.
(cherry picked from commit 2db64823c2)
2017-07-26 04:48:56 +02:00
Victor Stinner 7f3d65d6e4 bpo-30886: Fix multiprocessing.Queue.join_thread() (#2642) (#2643)
multiprocessing.Queue.join_thread() now waits until the thread
completes, even if the thread was started by the same process which
created the queue.

Fix the following warning which occurs randomly when running
test_handle_called_with_mp_queue of test_logging.QueueListenerTest:

Warning -- threading_cleanup() failed to cleanup -1 threads after 4 sec (count: 0, dangling: 1)
(cherry picked from commit 3b69d911c5)
2017-07-10 13:43:20 +02:00
Antoine Pitrou 38d6a40898 [3.6] Clear potential ref cycle between Process and Process target (GH-2470) (#2471)
* Clear potential ref cycle between Process and Process target

Besides Process.join() not being called, this was an indirect cause of bpo-30775.
The threading module already does this.

* Add issue reference.
(cherry picked from commit 79d37ae979)
2017-06-28 13:08:20 +02:00
Antoine Pitrou 6fd0345995 [3.6] bpo-24484: Avoid race condition in multiprocessing cleanup (GH-2159) (#2166)
* bpo-24484: Avoid race condition in multiprocessing cleanup

The finalizer registry can be mutated while inspected by multiprocessing
at process exit.

* Use test.support.start_threads()

* Add Misc/NEWS.
(cherry picked from commit 1eb6c0074d)
2017-06-13 17:51:26 +02:00
Antoine Pitrou 2783cc4262 [3.6] bpo-30414: multiprocessing.Queue._feed do not break from main loop on exc (GH-1683) (#1815)
* bpo-30414: multiprocesing.Queue._feed do not break from main loop on exc

Queue background running thread was not handling exceptions correctly.
Any exception occurred inside thread (putting unpickable object) cause
feeder to finish running. After that every message put into queue is
silently ignored.

* bpo-30414: multiprocesing.Queue._feed do not break from main loop on exc

Queue background running thread was not handling exceptions correctly.
Any exception occurred inside thread (putting unpickable object) cause
feeder to finish running. After that every message put into queue is
silently ignored.
(cherry picked from commit bc50f03db4)
2017-05-25 16:57:46 +02:00
Xiang Zhang 43d4c0329e bpo-30301: Fix AttributeError when using SimpleQueue.empty() (#1601) (#1628)
Under *spawn* and *forkserver* start methods, SimpleQueue.empty() could
raise AttributeError due to not setting _poll in __setstate__.
2017-05-17 22:03:35 +08:00
Antoine Pitrou a7e48b544b [3.6] bpo-30185: avoid KeyboardInterrupt tracebacks in forkserver (GH-1319) (#1454)
* bpo-30185: avoid KeyboardInterrupt tracebacks in forkserver

* Tweak comment.
(cherry picked from commit 6dd4d734ed)
2017-05-04 17:12:35 +02:00
Serhiy Storchaka e254617262 [3.6] bpo-30065: Fixed arguments validation in _posixsubprocess.fork_exec(). (GH-1110) (#1186)
(cherry picked from commit 66bffd1)
2017-04-19 23:59:02 +03:00
Xiang Zhang 346dcd65e6 bpo-28699: fix abnormal behaviour of pools in multiprocessing.pool (GH-882)
an exception raised at the very first of an iterable would cause pools behave abnormally
(swallow the exception or hang)
2017-03-29 12:50:28 +08:00
Antoine Pitrou cc3331fec8 bpo-29861: release references to multiprocessing Pool tasks (#743) (#800)
* bpo-29861: release references to multiprocessing Pool tasks (#743)

* bpo-29861: release references to multiprocessing Pool tasks

Release references to tasks, their arguments and their results as soon
as they are finished, instead of keeping them alive until another task
arrives.

* Comments in test

(cherry picked from commit 8988945cdc)

* Fix Misc/NEWS (hopefully)
2017-03-24 14:45:34 +01:00
Antoine Pitrou ebb39bcc04 Issue #28779: multiprocessing.set_forkserver_preload() would crash the forkserver process if a preloaded module instantiated some multiprocessing objects such as locks. 2016-12-10 17:16:17 +01:00
Antoine Pitrou cd2a201e5b Issue #28779: multiprocessing.set_forkserver_preload() would crash the forkserver process if a preloaded module instantiated some multiprocessing objects such as locks. 2016-12-10 17:13:16 +01:00
Davin Potts 5458647bb8 Issue #28053: Applying refactorings, docs and other cleanup to follow. 2016-09-09 18:03:10 -05:00
Davin Potts 37156a70b9 Issue #21201: Improves readability of multiprocessing error message from server to client for certain exceptions 2016-09-08 14:40:36 -05:00
Davin Potts 70897ec54c Issue #21201: Improves readability of multiprocessing error message from server to client for certain exceptions 2016-09-08 14:47:23 -05:00
Davin Potts 86a7668426 Fixes issue #6766: Updated multiprocessing Proxy Objects to support nesting 2016-09-07 18:48:01 -05:00
Serhiy Storchaka b6a9c9761c Issue #26778: Fixed "a/an/and" typos in code comment, documentation and error
messages.
2016-04-17 09:39:28 +03:00
Serhiy Storchaka 6a7b3a77b4 Issue #26778: Fixed "a/an/and" typos in code comment and documentation. 2016-04-17 08:32:47 +03:00
Victor Stinner a6d865c128 Issue #25654:
* multiprocessing: open file with closefd=False to avoid ResourceWarning
* _test_multiprocessing: open file with O_EXCL to detect bugs in tests (if a
  previous test forgot to remove TESTFN)
* test_sys_exit(): remove TESTFN after each loop iteration

Initial patch written by Serhiy Storchaka.
2016-03-25 09:29:50 +01:00
Charles-François Natali 78f55ffc63 Issue #23992: multiprocessing: make MapResult not fail-fast upon exception. 2016-02-10 22:58:18 +00:00
Berker Peksag 16fb674882 Fix typo in docstring of multiprocessing.spawn.spawn_main() 2016-01-07 19:02:55 +02:00
Serhiy Storchaka 225821c653 Issue #25899: Converted non-ASCII characters in docstrings and manpage
to ASCII replacements.  Original patch by Chris Angelico.
2015-12-18 13:05:04 +02:00
Antoine Pitrou d5aec7ba48 Issue #21116: Avoid blowing memory when allocating a multiprocessing shared
array that's larger than 50% of the available RAM.

Patch by Médéric Boquien.
2015-04-13 20:53:43 +02:00
Serhiy Storchaka 2116b12da5 Issue #23865: close() methods in multiple modules now are idempotent and more
robust at shutdown. If needs to release multiple resources, they are released
even if errors are occured.
2015-04-10 13:29:28 +03:00
Serhiy Storchaka 7e7a3dba5f Issue #23865: close() methods in multiple modules now are idempotent and more
robust at shutdown. If needs to release multiple resources, they are released
even if errors are occured.
2015-04-10 13:24:41 +03:00
Berker Peksag 291d7b0284 Issue #23400: Raise same exception on both Python 2 and 3 if sem_open is not available.
Patch by Davin Potts.
2015-04-08 17:57:44 +03:00
Berker Peksag 7ecfc82edb Issue #23400: Raise same exception on both Python 2 and 3 if sem_open is not available.
Patch by Davin Potts.
2015-04-08 17:56:30 +03:00
Serhiy Storchaka 79fbeee237 Issue #23051: multiprocessing.Pool methods imap() and imap_unordered() now
handle exceptions raised by an iterator.  Patch by Alon Diamant and Davin
Potts.
2015-03-13 08:25:26 +02:00
Serhiy Storchaka 63623ac252 Issue #23051: multiprocessing.Pool methods imap() and imap_unordered() now
handle exceptions raised by an iterator.  Patch by Alon Diamant and Davin
Potts.
2015-03-13 08:30:33 +02:00
Steve Dower 4e02f8f4ae Issue #18382: Zero-length messages are consumed by ReadFile on Windows 8 and later 2015-03-02 08:06:30 -08:00
Steve Dower 3f9e381030 Issue #18382: Zero-length messages are consumed by ReadFile on Windows 8 and later 2015-03-02 08:05:27 -08:00
Charles-François Natali 6e6c59b508 Issue #23285: PEP 475 -- Retry system calls failing with EINTR. 2015-02-07 13:27:50 +00:00
Steve Dower 438f4abb50 Issue #23060: Suppresses a multiprocessing assert that fails incorrectly
This is to keep buildbots running while we figure out the best long-term fix.
2014-12-17 06:35:49 -08:00
Serhiy Storchaka b757c83ec6 Issue #22581: Use more "bytes-like object" throughout the docs and comments. 2014-12-05 22:25:22 +02:00
Serhiy Storchaka 92bf919ed0 Issue #22581: Use more "bytes-like object" throughout the docs and comments. 2014-12-05 22:26:10 +02:00
Antoine Pitrou 0b87831de9 Simplify code in multiprocessing.Connection.send_bytes().
Followup to issue #20540; patch by Serhiy.
2014-07-31 18:41:57 -04:00
Serhiy Storchaka 465e60e654 Issue #22033: Reprs of most Python implemened classes now contain actual
class name instead of hardcoded one.
2014-07-25 23:36:00 +03:00
Charles-François Natali 6e20460dc6 Issue #21566: Make use of socket.listen() default backlog. 2014-07-23 19:28:13 +01:00
Serhiy Storchaka fbc877b794 Fixed bugs in reprs of CookieJar and multiprocessing.dummy.Value. 2014-07-22 11:10:37 +03:00
Serhiy Storchaka 1392df96ef Fixed bugs in reprs of CookieJar and multiprocessing.dummy.Value. 2014-07-22 11:09:36 +03:00
Charles-François Natali a924fc7abc Issue #21565: multiprocessing: use contex-manager protocol for synchronization
primitives.
2014-05-25 14:12:12 +01:00
Benjamin Peterson 40470e00b5 do not generate pipe names in the temporary dir 2014-04-14 12:24:37 -04:00
Richard Oudkerk 99d8dd2489 Issue #20990: Correction for 619331c67638. 2014-03-23 18:44:11 +00:00
Richard Oudkerk c346060440 Merge 3.3. 2014-03-23 12:52:16 +00:00
Richard Oudkerk 3e952d56ea Issue #20633: Replace relative import by absolute import. 2014-03-23 12:42:28 +00:00
Richard Oudkerk 80a5be1d84 Issue #20980: Stop wrapping exception when using ThreadPool. 2014-03-23 12:30:54 +00:00
Richard Oudkerk a40675a1a2 Issue #20990: Fix issues found by pyflakes for multiprocessing. 2014-03-23 11:54:15 +00:00
Victor Stinner 7fa767e517 Issue #20976: pyflakes: Remove unused imports 2014-03-20 09:16:38 +01:00
Victor Stinner 69b1e261fc Issue #20978: pyflakes: fix undefined names 2014-03-20 08:50:52 +01:00