Commit Graph

323 Commits

Author SHA1 Message Date
Steve Dower 44467e8ea4
bpo-35872 and bpo-35873: Clears __PYVENV_LAUNCHER__ variable (GH-11745)
After reading __PYVENV_LAUNCHER__ we now set sys._base_executable value for later use.
Make the same changes for macOS to avoid extra platform checks.
2019-02-04 07:20:19 -08:00
Miss Islington (bot) 6a9c0fca3f
bpo-35797: Fix default executable used by the multiprocessing module (GH-11676)
(cherry picked from commit 4e02f8f8b4)

Co-authored-by: Steve Dower <steve.dower@microsoft.com>
2019-01-25 15:14:41 -08:00
Victor Stinner 3c6b0d967e
[3.7] Revert "bpo-34172: multiprocessing.Pool leaks resources after being deleted (GH-8450) (GH-9676)" (#10968)
This reverts commit 97f998a4df.
2018-12-06 01:49:05 +01:00
Miss Islington (bot) 97f998a4df bpo-34172: multiprocessing.Pool leaks resources after being deleted (GH-8450) (GH-9676)
Fix a reference issue inside multiprocessing.Pool that caused the pool to remain alive if it was deleted without being closed or terminated explicitly.
(cherry picked from commit 97bfe8d3eb)

Co-authored-by: tzickel <tzickel@users.noreply.github.com>
2018-10-02 23:17:04 +02:00
Miss Islington (bot) 007fda436f bpo-34521: Fix tests in test_multiprocessing_spawn to use correctly CMSG_SPACE (GH-9613) (GH-9619)
After some failures in AMD64 FreeBSD CURRENT Debug 3.x buildbots
regarding tests in test_multiprocessing_spawn and after examining
similar failures in test_socket, some errors in the calculation of
ancillary data buffers were found in multiprocessing.reduction.

CMSG_LEN() can often be used as the buffer size for recvmsg() to
receive a single item of ancillary data, but RFC 3542 requires portable
applications to use CMSG_SPACE() and thus include space for padding,
even when the item will be the last in the buffer.

The failures we experience are due to the usage of CMSG_LEN() instead of
CMSG_SPACE().
(cherry picked from commit 077061a7b2)

Co-authored-by: Pablo Galindo <Pablogsal@gmail.com>
2018-09-28 06:07:56 -07:00
Miss Islington (bot) 1d30788493
bpo-34610: Fixed iterator of multiprocessing.managers.DictProxy. (GH-9113)
(cherry picked from commit e0e5065dae)

Co-authored-by: Serhiy Storchaka <storchaka@gmail.com>
2018-09-17 05:10:56 -07:00
Miss Islington (bot) 4bd5fce27d
bpo-34054: multiprocessing uses time.monotonic() (GH-8118)
The multiprocessing module now uses the monotonic clock
time.monotonic() instead of the system clock time.time() to implement
timeouts.
(cherry picked from commit c2368cbc83)

Co-authored-by: Victor Stinner <vstinner@redhat.com>
2018-07-06 05:11:21 -07:00
Victor Stinner 7b5856e7f0
Revert "bpo-33929: multiprocessing: fix handle leak on race condition (GH-7921)" (GH-7963)
This reverts commit 8b1ebcd7cb.
2018-06-27 15:17:34 +02:00
Miss Islington (bot) 8b1ebcd7cb
bpo-33929: multiprocessing: fix handle leak on race condition (GH-7921)
Fix a race condition in Popen of
multiprocessing.popen_spawn_win32. The child process now duplicates
the read end of pipe instead of "stealing" it.

Previously, the read end of pipe was "stolen" by the child process,
but it leaked a handle if the child process had been terminated
before it could steal the handle from the parent process.
(cherry picked from commit 2cc9d21fff)

Co-authored-by: Victor Stinner <vstinner@redhat.com>
2018-06-27 02:59:44 -07:00
Miss Islington (bot) bb5b529197 bpo-33078 - Fix queue size on pickling error (GH-6119) (GH-6178)
(cherry picked from commit e2f33add63)

Co-authored-by: Thomas Moreau <thomas.moreau.2010@gmail.com>
2018-03-21 17:21:15 +01:00
Miss Islington (bot) ff5d21331e bpo-31804: Fix multiprocessing.Process with broken standard streams (GH-6079) (GH-6080)
In some conditions the standard streams will be None or closed in the child process (for example if using "pythonw" instead of "python" on Windows).  Avoid failing with a non-0 exit code in those conditions.

Report and initial patch by poxthegreat.
(cherry picked from commit e756f66c83)

Co-authored-by: Antoine Pitrou <pitrou@free.fr>
2018-03-11 19:42:37 +01:00
Antoine Pitrou ab74504346
bpo-32576: use queue.SimpleQueue in critical places (#5216)
Where a queue may be invoked from a weakref callback, we need
to use the reentrant SimpleQueue.
2018-01-18 10:38:03 +01:00
Thomas Moreau 94459fd7dc bpo-31699 Deadlocks in `concurrent.futures.ProcessPoolExecutor` with pickling error (#3895)
Fix deadlocks in :class:`concurrent.futures.ProcessPoolExecutor` when task arguments or results cause pickling or unpickling errors.
This should make sure that calls to the :class:`ProcessPoolExecutor` API always eventually return.
2018-01-05 11:15:54 +01:00
Serhiy Storchaka a4a3020abc
bpo-32157: Removed explicit quotes around %r and {!r}. (#4582) 2017-11-28 22:54:42 +02:00
Antoine Pitrou cbe1756e3e
bpo-31310: multiprocessing's semaphore tracker should be launched again if crashed (#3247)
* bpo-31310: multiprocessing's semaphore tracker should be launched again if crashed

* Avoid mucking with process state in test.
Add a warning if the semaphore process died, as semaphores may then be leaked.

* Add NEWS entry
2017-11-03 14:31:38 +01:00
Antoine Pitrou fc6b348b12
bpo-31308: If multiprocessing's forkserver dies, launch it again when necessary (#3246)
* bpo-31308: If multiprocessing's forkserver dies, launch it again when necessary.

* Fix test on Windows

* Add NEWS entry

* Adopt a different approach: ignore SIGINT and SIGTERM, as in semaphore tracker.

* Fix comment

* Make sure the test doesn't muck with process state

* Also test previously-started processes

* Update 2017-08-30-17-59-36.bpo-31308.KbexyC.rst

* Avoid masking SIGTERM in forkserver.  It's not necessary and causes a race condition in test_many_processes.
2017-11-03 13:34:22 +01:00
Antoine Pitrou daeefd2e04 bpo-28326: Fix multiprocessing.Process when stdout and/or stderr is closed or None. (#4073) 2017-10-22 11:40:31 +02:00
Antoine Pitrou e91c2a5086 [Trivial] Remove now redundant assert (#3245) 2017-08-30 16:57:12 +02:00
Allen W. Smith, Ph.D bd73e72b4a bpo-5001: More-informative multiprocessing error messages (#3079)
* Make error message more informative

Replace assertions in error-reporting code with more-informative version that doesn't cause confusion over where and what the error is.

* Additional clarification + get travis to check

* Change from SystemError to TypeError

As suggested in PR comment by @pitrou, changing from SystemError; TypeError appears appropriate.

* NEWS file installation; ACKS addition (will do my best to justify it by additional work)

* Making current AssertionErrors in multiprocessing more informative

* Blurb added re multiprocessing managers.py, queues.py cleanup

* Further multiprocessing cleanup - went through pool.py

* Fix two asserts in multiprocessing/util.py

* Most asserts in multiprocessing more informative

* Didn't save right version

* Further work on multiprocessing error messages

* Correct typo

* Correct typo v2

* Blasted colon... serves me right for trying to work on two things at once

* Simplify NEWS entry

* Update 2017-08-18-17-16-38.bpo-5001.gwnthq.rst

* Update 2017-08-18-17-16-38.bpo-5001.gwnthq.rst

OK, never mind.

* Corrected (thanks to pitrou) error messages for notify

* Remove extraneous backslash in docstring.
2017-08-30 00:52:18 +02:00
Antoine Pitrou ee84a60858 bpo-18966: non-daemonic threads created by a multiprocessing.Process should be joined on exit (#3111)
* bpo-18966: non-daemonic threads created by a multiprocessing.Process should be joined on exit

* Add NEWS blurb
2017-08-16 20:53:28 +02:00
Allen W. Smith, Ph.D 48d9823a0e bpo-5001, bpo-31169: Fix two uninformative asserts in multiprocessing/managers.py (#3078)
* Make error message more informative

Replace assertions in error-reporting code with more-informative version that doesn't cause confusion over where and what the error is.

* Additional clarification + get travis to check

* Change from SystemError to TypeError

As suggested in PR comment by @pitrou, changing from SystemError; TypeError appears appropriate.

* NEWS file installation; ACKS addition (will do my best to justify it by additional work)
2017-08-12 17:37:09 +02:00
Victor Stinner 2db64823c2 bpo-31019: Fix multiprocessing.Process.is_alive() (#2875)
multiprocessing.Process.is_alive() now removes the process from the
_children set if the process completed.

The change prevents leaking "dangling" processes.
2017-07-26 02:32:42 +02:00
Antoine Pitrou 3051f0b78e bpo-30919: shared memory allocation performance regression in multiprocessing (#2708)
* Fix #30919: shared memory allocation performance regression in multiprocessing

* Change strategy for Arena directory choice

* Add blurb
2017-07-23 13:05:26 +02:00
Antoine Pitrou 896145d9d2 bpo-26732: fix too many fds in processes started with the "forkserver" method (#2813)
* bpo-26732: fix too many fds in processes started with the "forkserver" method

A child process would inherit as many fds as the number of still-running children.

* Add blurb and test comment
2017-07-22 13:22:54 +02:00
Gareth Rees 3913bad495 bpo-19896: Add typecodes 'q' and 'Q' to multiprocessing.sharedctypes (#2741)
* bpo-19896: Add typcodes 'q' and 'Q' to multiprocessing.sharedctypes. Patch by Antony Lee.

* Add NEWS entry.

* Slightly tweak NEWS entry

Make it clear this is more of a fix rather than a new feature.
2017-07-21 12:35:33 +02:00
Vitor Pereira ba75af7130 bpo-30794: added kill() method to multiprocessing.Process (#2528)
* bpo-30794: added kill() method to multiprocessing.Process

* Added entries to documentation and NEWS

* Refactored test_terminate and test_kill

* Fix SIGTERM and SIGKILL being used on Windows for the tests

* Added "versionadded" marker to the documentation

* Fix trailing whitespace in doc
2017-07-18 17:34:23 +02:00
Victor Stinner 3b69d911c5 bpo-30886: Fix multiprocessing.Queue.join_thread() (#2642)
multiprocessing.Queue.join_thread() now waits until the thread
completes, even if the thread was started by the same process which
created the queue.

Fix the following warning which occurs randomly when running
test_handle_called_with_mp_queue of test_logging.QueueListenerTest:

Warning -- threading_cleanup() failed to cleanup -1 threads after 4 sec (count: 0, dangling: 1)
2017-07-10 12:45:21 +02:00
Antoine Pitrou 48350412b7 bpo-29293: multiprocessing.Condition.notify() lacks parameter `n` (#2480)
* bpo-29293: multiprocessing.Condition.notify() lacks parameter `n`

* Add NEWS blurb
2017-07-04 08:59:22 +02:00
Antoine Pitrou 79d37ae979 Clear potential ref cycle between Process and Process target (#2470)
* Clear potential ref cycle between Process and Process target

Besides Process.join() not being called, this was an indirect cause of bpo-30775.
The threading module already does this.

* Add issue reference
2017-06-28 12:29:08 +02:00
Antoine Pitrou 13e96cc596 Fix bpo-30596: Add close() method to multiprocessing.Process (#2010)
* Fix bpo-30596: Add close() method to multiprocessing.Process

* Raise ValueError if close() is called before the Process is finished running

* Add docs

* Add NEWS blurb
2017-06-24 19:22:23 +02:00
Antoine Pitrou 1eb6c0074d bpo-24484: Avoid race condition in multiprocessing cleanup (#2159)
* bpo-24484: Avoid race condition in multiprocessing cleanup

The finalizer registry can be mutated while inspected by multiprocessing
at process exit.

* Use test.support.start_threads()

* Add Misc/NEWS
2017-06-13 17:10:39 +02:00
Antoine Pitrou 2b5cc5ebaf bpo-30643: Fix race condition in signal wakeup in forkserver (followup to PR #1989) (#2139)
* Fix race condition in signal wakeup in forkserver (followup to PR #1989)

There's an admittedly well-known race condition where ECHILD can arrive
just before the C function epoll_wait() and the latter wouldn't therefore
return EINTR.  The solution is to use set_wakeup_fd(), which was designed
to avoid such race conditions.

* Reset wakeup fd in child
2017-06-13 09:46:06 +02:00
Antoine Pitrou dfd5f34634 Fix bpo-30589: improve Process.exitcode with forkserver (#1989)
* Fix bpo-30589: improve Process.exitcode with forkserver

When the child is killed, Process.exitcode should return -signum, not 255.

* Add Misc/NEWS
2017-06-12 15:28:19 +02:00
Victor Stinner 1b7863c3b6 bpo-30595: Fix multiprocessing.Queue.get(timeout) (#2027)
multiprocessing.Queue.get() with a timeout now polls its reader in
non-blocking mode if it succeeded to aquire the lock but the acquire
took longer than the timeout.

Co-Authored-By: Grzegorz Grzywacz <grzgrzgrz3@gmail.com>
2017-06-10 13:10:13 +02:00
Antoine Pitrou 346cbd351e bpo-16500: Allow registering at-fork handlers (#1715)
* bpo-16500: Allow registering at-fork handlers

* Address Serhiy's comments

* Add doc for new C API

* Add doc for new Python-facing function

* Add NEWS entry + doc nit
2017-05-27 17:50:54 +02:00
grzgrzgrz3 bc50f03db4 bpo-30414: multiprocessing.Queue._feed do not break from main loop on exc (#1683)
* bpo-30414: multiprocesing.Queue._feed do not break from main loop on exc

Queue background running thread was not handling exceptions correctly.
Any exception occurred inside thread (putting unpickable object) cause
feeder to finish running. After that every message put into queue is
silently ignored.

* bpo-30414: multiprocesing.Queue._feed do not break from main loop on exc

Queue background running thread was not handling exceptions correctly.
Any exception occurred inside thread (putting unpickable object) cause
feeder to finish running. After that every message put into queue is
silently ignored.
2017-05-25 16:22:57 +02:00
Jon Dufresne 3972628de3 bpo-30296 Remove unnecessary tuples, lists, sets, and dicts (#1489)
* Replaced list(<generator expression>) with list comprehension
* Replaced dict(<generator expression>) with dict comprehension
* Replaced set(<list literal>) with set literal
* Replaced builtin func(<list comprehension>) with func(<generator
  expression>) when supported (e.g. any(), all(), tuple(), min(), &
  max())
2017-05-18 07:35:54 -07:00
Xiang Zhang 6f75bc003a bpo-30301: Fix AttributeError when using SimpleQueue.empty() (#1601)
Under  *spawn* and *forkserver* start methods, SimpleQueue.empty() could
raise AttributeError due to not setting _poll in __setstate__.
2017-05-17 21:04:00 +08:00
Antoine Pitrou 6dd4d734ed bpo-30185: avoid KeyboardInterrupt tracebacks in forkserver (#1319)
* bpo-30185: avoid KeyboardInterrupt tracebacks in forkserver

* Tweak comment
2017-05-04 16:44:53 +02:00
Serhiy Storchaka 66bffd1663 bpo-30065: Fixed arguments validation in _posixsubprocess.fork_exec(). (#1110) 2017-04-19 21:12:46 +03:00
Serhiy Storchaka 5affd23e6f bpo-29762: More use "raise from None". (#569)
This hides unwanted implementation details from tracebacks.
2017-04-05 09:37:24 +03:00
Xiang Zhang 794623bdb2 bpo-28699: fix abnormal behaviour of pools in multiprocessing.pool (GH-693)
an exception raised at the very first of an iterable would cause pools behave abnormally
(swallow the exception or hang)
2017-03-29 11:58:54 +08:00
Victor Stinner d6debb24e0 bpo-29919: Remove unused imports found by pyflakes (#137)
Make also minor PEP8 coding style fixes on modified imports.
2017-03-27 16:05:26 +02:00
Antoine Pitrou 8988945cdc bpo-29861: release references to multiprocessing Pool tasks (#743)
* bpo-29861: release references to multiprocessing Pool tasks

Release references to tasks, their arguments and their results as soon
as they are finished, instead of keeping them alive until another task
arrives.

* Comments in test
2017-03-24 13:52:11 +01:00
Serhiy Storchaka bdf6b910f9 bpo-29776: Use decorator syntax for properties. (#585) 2017-03-19 08:40:32 +02:00
Antoine Pitrou ebb39bcc04 Issue #28779: multiprocessing.set_forkserver_preload() would crash the forkserver process if a preloaded module instantiated some multiprocessing objects such as locks. 2016-12-10 17:16:17 +01:00
Antoine Pitrou cd2a201e5b Issue #28779: multiprocessing.set_forkserver_preload() would crash the forkserver process if a preloaded module instantiated some multiprocessing objects such as locks. 2016-12-10 17:13:16 +01:00
Davin Potts 5458647bb8 Issue #28053: Applying refactorings, docs and other cleanup to follow. 2016-09-09 18:03:10 -05:00
Davin Potts 37156a70b9 Issue #21201: Improves readability of multiprocessing error message from server to client for certain exceptions 2016-09-08 14:40:36 -05:00
Davin Potts 70897ec54c Issue #21201: Improves readability of multiprocessing error message from server to client for certain exceptions 2016-09-08 14:47:23 -05:00