Commit Graph

378 Commits

Author SHA1 Message Date
idomic db098bc1f0
bpo-39244: multiprocessing return default start method first on macOS (GH-18625) 2020-05-26 17:54:21 +03:00
Victor Stinner eb0d359b4b
bpo-40443: Remove unused imports in stdlib (GH-19815) 2020-05-01 02:38:00 +02:00
Victor Stinner 9adccc1384
bpo-30966: Add multiprocessing.SimpleQueue.close() (GH-19735)
Add a new close() method to multiprocessing.SimpleQueue to explicitly
close the queue.

Automerge-Triggered-By: @pitrou
2020-04-27 09:11:10 -07:00
Antoine Pitrou eba9f6155d
bpo-40330: Fix utf-8 size check in ShareableList (GH-19606)
The item size must be checked after encoding to bytes, not before.

Automerge-Triggered-By: @pitrou
2020-04-20 11:54:55 -07:00
Thomas Krennwallner c8f1715283
bpo-38891: avoid quadratic item access performance of ShareableList (GH-18996)
Avoid linear runtime of ShareableList.__getitem__ and
ShareableList.__setitem__ by storing running allocated bytes in
ShareableList._allocated_bytes instead of the number of bytes for
a particular stored item.

Co-authored-by: Antoine Pitrou <antoine@python.org>
2020-04-19 17:19:24 +02:00
Galden c606624af8
Fix two typos in multiprocessing (GH-19571) 2020-04-18 08:58:29 +02:00
Dong-hee Na a5900ecf9f
bpo-40221: Update multiprocessing to use _at_fork_reinit (GH-19511) 2020-04-15 01:35:36 +09:00
Dong-hee Na e1945307d3
bpo-40221: Update multiprocessing to use _at_fork_reinit (GH-19477) 2020-04-14 22:15:52 +09:00
Batuhan Taşkaya 0361556537
bpo-39481: PEP 585 for a variety of modules (GH-19423)
- concurrent.futures
- ctypes
- http.cookies
- multiprocessing
- queue
- tempfile
- unittest.case
- urllib.parse
2020-04-10 07:46:36 -07:00
Victor Stinner 65a796e527
bpo-40094: Add os.waitstatus_to_exitcode() (GH-19201)
Add os.waitstatus_to_exitcode() function to convert a wait status to an
exitcode.

Suggest waitstatus_to_exitcode() usage in the documentation when
appropriate.

Use waitstatus_to_exitcode() in:

* multiprocessing, os, subprocess and _bootsubprocess modules;
* test.support.wait_process();
* setup.py: run_command();
* and many tests.
2020-04-01 18:49:29 +02:00
Batuhan Taşkaya ac10e0c932
bpo-39360: Ensure all workers exit when finalizing a multiprocessing Pool (GH-19009)
When the pull is not used via the context manager or terminate() is called, there is a system in multiprocessing.util that handles finalization of all pools via an atexit handler (the Finalize) class. This class registers the _terminate_pool handler in the registry of finalizers of the module, and that registry is called on interpreter exit via _exit_function. The problem is that the "happy" path with the context manager or manual call to finalize() does some extra steps that _terminate_pool does not. The step that is not executed when the atexit() handler calls _terminate_pool is pinging the _change_notifier queue to unblock the maintenance threads.

This commit moves the notification to the _terminate_pool function so is called from both code paths.

Co-authored-by: Pablo Galindo <Pablogsal@gmail.com>
2020-03-15 19:45:56 +00:00
Pablo Galindo 6012f30bef
bpo-39850: Add support for abstract sockets in multiprocessing (GH-18866) 2020-03-09 13:48:01 +00:00
Christopher Hunt c2ac4cf040
bpo-35727: Use exit code 0 on sys.exit() in multiprocessing.Process. (GH-11538) 2020-02-21 10:33:04 +01:00
Adam Meily 0be3246d4f bpo-39439: Fix multiprocessing spawn path in a venv on Windows (GH-18158) 2020-01-28 21:34:23 +11:00
Victor Stinner 9707e8e22d
bpo-38546: multiprocessing tests stop the resource tracker (GH-17641)
Multiprocessing and concurrent.futures tests now stop the resource
tracker process when tests complete.

Add ResourceTracker._stop() method to
multiprocessing.resource_tracker.

Add _cleanup_tests() helper function to multiprocessing.util: share
code between multiprocessing and concurrent.futures tests.
2019-12-17 18:37:26 +01:00
Pablo Galindo 293dd23477
Remove binding of captured exceptions when not used to reduce the chances of creating cycles (GH-17246)
Capturing exceptions into names can lead to reference cycles though the __traceback__ attribute of the exceptions in some obscure cases that have been reported previously and fixed individually. As these variables are not used anyway, we can remove the binding to reduce the chances of creating reference cycles.

See for example GH-13135
2019-11-19 21:34:03 +00:00
Jake Tesler c6b20be85c bpo-38707: Fix for multiprocessing.Process MainThread.native_id (GH-17088)
This PR implements a fix for `multiprocessing.Process` objects; the error occurs when Processes are created using either `fork` or `forkserver` as the `start_method`.

In these instances, the `MainThread` of the newly created `Process` object retains all attributes from its parent's `MainThread` object, including the `native_id` attribute. The resulting behavior is such that the new process' `MainThread` captures an incorrect/outdated `native_id` (the parent's instead of its own). 

This change forces the Process object to update its `native_id` attribute during the bootstrap process.

cc @vstinner





https://bugs.python.org/issue38707



Automerge-Triggered-By: @pitrou
2019-11-19 11:50:12 -08:00
Gregory P. Smith f3751efb5c
bpo-38417: Add umask support to subprocess (GH-16726)
On POSIX systems, allow the umask to be set in the child process before we exec.
2019-10-12 13:24:56 -07:00
Steve Dower f2b7556ef8 bpo-38092: Reduce overhead when using multiprocessing in a Windows virtual environment (GH-16098)
https://bugs.python.org/issue38092
2019-09-13 09:40:19 -07:00
Patrick McLean 2b2ead7438 bpo-36046: Add user and group parameters to subprocess (GH-11950)
* subprocess: Add user, group and extra_groups paremeters to subprocess.Popen

This adds a `user` parameter to the Popen constructor that will call
setreuid() in the child before calling exec(). This allows processes
running as root to safely drop privileges before running the subprocess
without having to use a preexec_fn.

This also adds a `group` parameter that will call setregid() in
the child process before calling exec().

Finally an `extra_groups` parameter was added that will call
setgroups() to set the supplimental groups.
2019-09-12 18:15:44 +01:00
Min ho Kim 39d87b5471 Fix typos mostly in comments, docs and test names (GH-15209) 2019-08-30 16:21:19 -04:00
Victor Stinner 8fbeb14312
bpo-37421: multiprocessing tests now stop ForkServer (GH-14601)
multiprocessing tests now stop the ForkServer instance if it's
running: close the "alive" file descriptor to ask the server to stop
and then remove its UNIX address.
2019-07-05 16:15:39 +02:00
Victor Stinner 9d40554e0d
bpo-37421: Fix multiprocessing get_temp_dir() finalizer (GH-14572)
Fix multiprocessing.util.get_temp_dir() finalizer: clear also the
'tempdir' configuration of the current process, so next call to
get_temp_dir() will create a new temporary directory, rather than
reusing the removed temporary directory.
2019-07-04 12:28:55 +02:00
Steve Dower 9048c49322
bpo-37369: Fix initialization of sys members when launched via an app container (GH-14428)
sys._base_executable is now always defined on all platforms, and can be overridden through configuration.
Also adds test.support.PythonSymlink to encapsulate platform-specific logic for symlinking sys.executable
2019-06-29 10:34:11 -07:00
Serhiy Storchaka 142566c028
[3.9] bpo-37116: Use PEP 570 syntax for positional-only parameters. (GH-12620)
Turn deprecation warnings added in 3.8 into TypeError.
2019-06-05 18:22:31 +03:00
Serhiy Storchaka 2085bd0877
bpo-37116: Use PEP 570 syntax for positional-only parameters. (GH-13700) 2019-06-01 11:00:15 +03:00
Victor Stinner 17a5588740
bpo-33725: multiprocessing uses spawn by default on macOS (GH-13603)
On macOS, the multiprocessing module now uses the "spawn" start
method by default.
2019-05-28 16:02:50 +02:00
Thomas Moreau c09a9f56c0 bpo-36888: Add multiprocessing.parent_process() (GH-13247) 2019-05-20 21:37:05 +02:00
Pierre Glaser b1dfcad6f0 bpo-36867: Create the resource_tracker before launching SharedMemoryManagers (GH-13276) 2019-05-13 21:15:32 +02:00
Antoine Pitrou 95da83d9ba
bpo-36894: Fix regression in test_multiprocessing_spawn (no tests run on Windows) (GH-13290) 2019-05-13 20:02:46 +02:00
Pierre Glaser f22cc69b01 bpo-36867: Make semaphore_tracker track other system resources (GH-13222)
The multiprocessing.resource_tracker replaces the multiprocessing.semaphore_tracker module. Other than semaphores, resource_tracker also tracks shared_memory segments. Patch by Pierre Glaser.
2019-05-10 22:59:08 +02:00
Pierre Glaser d0d64ad1f5 bpo-36368: Ignore SIGINT in SharedMemoryManager servers. (GH-12483)
Fix a bug crashing SharedMemoryManager instances in interactive sessions after
a Ctrl-C (KeyboardInterrupt) was sent.
2019-05-10 20:42:35 +02:00
Serhiy Storchaka d53cf99dca
bpo-36542: Allow to overwrite the signature for Python functions. (GH-12705) 2019-05-06 22:40:27 +03:00
penguindustin 9646630895 bpo-36766: Typos in docs and code comments (GH-13116) 2019-05-06 14:57:17 -04:00
Thomas Moreau 004b93ea89 bpo-36668: FIX reuse semaphore tracker for child processes (#5172)
Fix the multiprocessing.semaphore_tracker so it is reused by child processes.
2019-04-24 21:45:52 +02:00
Serhiy Storchaka 42a139ed88
bpo-36492: Deprecate passing some arguments as keyword arguments. (GH-12637)
Deprecated passing the following arguments as keyword arguments:

- "func" in functools.partialmethod(), weakref.finalize(),
  profile.Profile.runcall(), cProfile.Profile.runcall(),
  bdb.Bdb.runcall(), trace.Trace.runfunc() and
  curses.wrapper().
- "function" in unittest.addModuleCleanup() and
  unittest.TestCase.addCleanup().
- "fn" in the submit() method of concurrent.futures.ThreadPoolExecutor
  and concurrent.futures.ProcessPoolExecutor.
- "callback" in contextlib.ExitStack.callback(),
  contextlib.AsyncExitStack.callback() and
  contextlib.AsyncExitStack.push_async_callback().
- "c" and "typeid" in the create() method of multiprocessing.managers.Server
  and multiprocessing.managers.SharedMemoryServer.
- "obj" in weakref.finalize().

Also allowed to pass arbitrary keyword arguments (even "self" and "func")
if the above arguments are passed as positional argument.
2019-04-01 09:16:35 +03:00
Pablo Galindo 7c994549dc
bpo-35493: Use Process.sentinel instead of sleeping for polling worker status in multiprocessing.Pool (#11488)
* bpo-35493: Use Process.sentinel instead of sleeping for polling worker status in multiprocessing.Pool

* Use self-pipe pattern to avoid polling for changes

* Refactor some variable names and add comments

* Restore timeout and poll

* Use reader object only on wait()

* Recompute worker sentinels every time

* Remove timeout and use change notifier

* Refactor some methods to be overloaded by the ThreadPool, document the cache class and fix typos
2019-03-16 22:34:24 +00:00
Davin Potts aadef2b416
bpo-36102: Prepend slash to all POSIX shared memory block names (#12036) 2019-02-25 16:41:52 -06:00
Davin Potts e895de3e7f
bpo-35813: Tests and docs for shared_memory (#11816)
* Added tests for shared_memory submodule.

* Added tests for ShareableList.

* Fix bug in allocationn size during creation of empty ShareableList illuminated by existing test run on Linux.

* Initial set of docs for shared_memory module.

* Added docs for ShareableList, added doctree entry for shared_memory submodule, name refactoring for greater clarity.

* Added examples to SharedMemoryManager docs, for ease of documentation switched away from exclusively registered functions to some explicit methods on SharedMemoryManager.

* Wording tweaks to docs.

* Fix test failures on Windows.

* Added tests around SharedMemoryManager.

* Documentation tweaks.

* Fix inappropriate test on Windows.

* Further documentation tweaks.

* Fix bare exception.

* Removed __copyright__.

* Fixed typo in doc, removed comment.

* Updated SharedMemoryManager preliminary tests to reflect change of not supporting all registered functions on SyncManager.

* Added Sphinx doctest run controls.

* CloseHandle should be in a finally block in case MapViewOfFile fails.

* Missed opportunity to use with statement.

* Switch to self.addCleanup to spare long try/finally blocks and save one indentation, change to use decorator to skip test instead.

* Simplify the posixshmem extension module.

Provide shm_open() and shm_unlink() functions.  Move other
functionality into the shared_memory.py module.

* Added to doc around size parameter of SharedMemory.

* Changed PosixSharedMemory.size to use os.fstat.

* Change SharedMemory.buf to a read-only property as well as NamedSharedMemory.size.

* Marked as provisional per PEP411 in docstring.

* Changed SharedMemoryTracker to be private.

* Removed registered Proxy Objects from SharedMemoryManager.

* Removed shareable_wrap().

* Removed shareable_wrap() and dangling references to it.

* For consistency added __reduce__ to key classes.

* Fix for potential race condition on Windows for O_CREX.

* Remove unused imports.

* Update access to kernel32 on Windows per feedback from eryksun.

* Moved kernel32 calls to _winapi.

* Removed ShareableList.copy as redundant.

* Changes to _winapi use from eryksun feedback.

* Adopt simpler SharedMemory API, collapsing PosixSharedMemory and WindowsNamedSharedMemory into one.

* Fix missing docstring on class, add test for ignoring size when attaching.

* Moved SharedMemoryManager to managers module, tweak to fragile test.

* Tweak to exception in OpenFileMapping suggested by eryksun.

* Mark a few dangling bits as private as suggested by Giampaolo.
2019-02-23 22:08:16 -06:00
Rémi Lapeyre a31f4cc881 bpo-35918: Remove broken has_key method and add test (#11819) 2019-02-11 16:37:24 -08:00
Pablo Galindo 3766f18c52
bpo-35378: Fix multiprocessing.Pool references (GH-11627)
Changes in this commit:

1. Use a _strong_ reference between the Pool and associated iterators
2. Rework PR #8450 to eliminate a cycle in the Pool.

There is no test in this commit because any test that automatically tests this behaviour needs to eliminate the pool before joining the pool to check that the pool object is garbaged collected/does not hang. But doing this will potentially leak threads and processes (see https://bugs.python.org/issue35413).
2019-02-11 17:29:00 +00:00
Steve Dower a8474d025c
bpo-35872 and bpo-35873: Clears __PYVENV_LAUNCHER__ variable (GH-11745)
After reading __PYVENV_LAUNCHER__ we now set sys._base_executable value for later use.
Make the same changes for macOS to avoid extra platform checks.
2019-02-03 23:19:38 -08:00
Davin Potts e5ef45b8f5
bpo-35813: Added shared_memory submodule of multiprocessing. (#11664)
Added shared_memory submodule to multiprocessing in time for first alpha with cross-platform tests soon to follow.
2019-02-01 22:52:23 -06:00
Steve Dower 4e02f8f8b4
bpo-35797: Fix default executable used by the multiprocessing module (GH-11676) 2019-01-25 14:59:12 -08:00
Victor Stinner 9a8d1d7562
bpo-35424: emit ResourceWarning at multiprocessing.Pool destruction (GH-10974)
multiprocessing.Pool destructor now emits ResourceWarning
if the pool is still running.
2018-12-20 20:33:51 +01:00
Victor Stinner 2dfe3511fe
bpo-35491, multiprocessing: replace "RUN" with RUN (GH-11178) 2018-12-16 23:40:49 +01:00
Victor Stinner 7acd50ad8b
bpo-35491: Enhance multiprocessing.BaseProcess.__repr__() (GH-11138)
* Add the pid and parent pid to multiprocessing.BaseProcess.__repr__().
* Add negative sign (ex: "-SIGTERM") to exitcode (process killed
  by a signal)
* Only call _popen.poll() once.

Example:
  <ForkProcess(ForkPoolWorker-1, started daemon)>
becomes:
  <ForkProcess name='ForkPoolWorker-1' pid=12449 parent=12448 started daemon>

Example:
  <ForkProcess(ForkPoolWorker-1, stopped[SIGTERM] daemon)>
becomes:
  <ForkProcess name='ForkPoolWorker-1' pid=12960 parent=12959 stopped exitcode=-SIGTERM daemon>
2018-12-14 12:58:52 +01:00
Victor Stinner 2b417fba25
Add multiprocessing.Pool.__repr__() (GH-11137)
* Add multiprocessing.Pool.__repr__() to ease debug
* RUN, CLOSE and TERMINATE constants values are now strings rather
  than integer to ease debug
2018-12-14 11:13:18 +01:00
Victor Stinner 08c2ba0717
bpo-35477: multiprocessing.Pool.__enter__() fails if called twice (GH-11134)
multiprocessing.Pool.__enter__() now fails if the pool is not
running: "with pool:" fails if used more than once.
2018-12-13 02:15:30 +01:00
Victor Stinner 9dfc754d61
Revert "bpo-34172: multiprocessing.Pool leaks resources after being deleted (GH-8450)" (GH-10971)
This reverts commit 97bfe8d3eb.
2018-12-06 08:51:47 +01:00