Commit Graph

439 Commits

Author SHA1 Message Date
Roy Hyunjin Han bbb49888a7
gh-103134: Update multiprocessing.managers.ListProxy and DictProxy (GH-103133) 2024-05-20 14:28:36 +00:00
Tian Gao 998c3856c1
gh-83856: Honor atexit for all multiprocessing start methods (GH-114279)
Use atexit for all multiprocessing start methods to cleanup.
See the GH-114279 PR discussion and related issue for details as to why.
2024-05-03 11:45:46 -07:00
Henrik Tunedal 133c1a7cdb
gh-118293: Suppress mouse cursor feedback when launching Windows processes with multiprocessing (GH-118315) 2024-04-28 21:10:44 +00:00
Malcolm Smith 872c0714fc
gh-71052: Change Android's `sys.platform` from "linux" to "android"
Co-authored-by: Erlend E. Aasland <erlend.aasland@protonmail.com>
2024-03-11 19:25:39 +00:00
Miguel Brito 686ec17f50
bpo-43952: Fix multiprocessing Listener authkey bug (GH-25845)
Listener.accept() no longer hangs when authkey is an empty bytes object.
2024-02-27 14:57:59 +00:00
Malcolm Smith 4827968af8
gh-71052: Enable test_concurrent_futures on platforms that lack multiprocessing (gh-115917)
Enable test_concurrent_futures on platforms that support threading but not multiprocessing.
2024-02-25 11:38:18 -08:00
Petr Viktorin 4a9e6497c2
gh-104090: Add exit code to multiprocessing ResourceTracker (GH-115410)
This builds on https://github.com/python/cpython/pull/106807, which adds
a return code to ResourceTracker, to make future debugging easier.
Testing this “in situ” proved difficult, since the global ResourceTracker is
involved in test infrastructure. So, the tests here create a new instance and
feed it fake data.

---------

Co-authored-by: Yonatan Bitton <yonatan.bitton@perception-point.io>
Co-authored-by: Yonatan Bitton <bityob@gmail.com>
Co-authored-by: Antoine Pitrou <antoine@python.org>
2024-02-21 13:54:57 +01:00
Steve Dower ea25f32d5f
gh-89240: Enable multiprocessing on Windows to use large process pools (GH-107873)
We add _winapi.BatchedWaitForMultipleObjects to wait for larger numbers of handles.
This is an internal module, hence undocumented, and should be used with caution.
Check the docstring for info before using BatchedWaitForMultipleObjects.
2024-02-13 00:28:35 +00:00
Hood Chatham 4b75032c88
gh-114807: multiprocessing: don't raise ImportError if _multiprocessing is missing (#114808)
`_multiprocessing` is only used under the `if _winapi:` block, this moves the import to be within the `_winapi` ImportError handling try/except for equivalent treatment.
2024-02-11 01:59:50 -08:00
Ronald Oussoren c7d59bd8cf
gh-101225: Increase the socket backlog when creating a multiprocessing.connection.Listener (#113567)
Increase the backlog for multiprocessing.connection.Listener` objects created
 by `multiprocessing.manager` and `multiprocessing.resource_sharer` to
 significantly reduce the risk of getting a connection refused error when creating
 a `multiprocessing.connection.Connection` to them.
2024-01-13 10:48:33 +01:00
Xu Song ce77ee5035
gh-113421: Fix multiprocessing logger for "%(filename)s" (GH-113423) 2023-12-24 12:04:12 +02:00
Victor Stinner 4026ad5b2c
gh-113009: Fix multiprocessing Process.terminate() on Windows (#113128)
On Windows, Process.terminate() no longer sets the returncode
attribute to always call WaitForSingleObject() in Process.wait().
Previously, sometimes the process was still running after
TerminateProcess() even if GetExitCodeProcess() is not STILL_ACTIVE.
2023-12-15 15:57:49 +01:00
pan324 81ee026091
gh-82300: Add track parameter to multiprocessing.shared_memory (#110778)
Add a track parameter to shared memory to allow resource tracking via the side-launched resource tracker process to be disabled on platforms that use it (POSIX).

This allows people who do not want automated cleanup at process exit because they are using the shared memory with processes not participating in Python's resource tracking to use the shared_memory API.

Co-authored-by: Łukasz Langa <lukasz@langa.pl>
Co-authored-by: Guido van Rossum <gvanrossum@gmail.com>
Co-authored-by: Antoine Pitrou <pitrou@free.fr>
Co-authored-by: Gregory P. Smith <greg@krypto.org>
2023-12-05 00:11:44 -08:00
Nikita Sobolev ae8116cfa9
gh-107431: Make `multiprocessing.managers.{DictProxy,ListProxy}` generic (#107433)
Make `multiprocessing.managers.{DictProxy,ListProxy}` generic for type annotation use.  `ListProxy[str]` for example.

Co-authored-by: Alex Waygood <Alex.Waygood@Gmail.com>
Co-authored-by: Gregory P. Smith <greg@krypto.org>
2023-11-10 23:23:27 +00:00
Victor Stinner a46e960768
gh-109649: Use os.process_cpu_count() (#110165)
Replace os.cpu_count() with os.process_cpu_count() in modules:

* compileall
* concurrent.futures
* multiprocessing

Replace os.cpu_count() with os.process_cpu_count() in programs:

* _decimal deccheck.py test
* freeze.py
* multissltests.py
* python -m test (regrtest)
* wasm_build.py

Other changes:

* test.pythoninfo logs os.process_cpu_count().
* regrtest gets os.process_cpu_count() / os.cpu_count() in headers.
2023-10-01 03:14:57 +02:00
Victor Stinner 6351842121
gh-109047: concurrent.futures catches PythonFinalizationError (#109810)
concurrent.futures: The *executor manager thread* now catches
exceptions when adding an item to the *call queue*. During Python
finalization, creating a new thread can now raise RuntimeError. Catch
the exception and call terminate_broken() in this case.

Add test_python_finalization_error() to test_concurrent_futures.

concurrent.futures._ExecutorManagerThread changes:

* terminate_broken() no longer calls shutdown_workers() since the
  call queue is no longer working anymore (read and write ends of
  the queue pipe are closed).
* terminate_broken() now terminates child processes, not only
  wait until they complete.
* _ExecutorManagerThread.terminate_broken() now holds shutdown_lock
  to prevent race conditons with ProcessPoolExecutor.submit().

multiprocessing.Queue changes:

* Add _terminate_broken() method.
* _start_thread() sets _thread to None on exception to prevent
  leaking "dangling threads" even if the thread was not started
  yet.
2023-09-29 19:31:19 +00:00
Victor Stinner bd4518c60c
gh-110036: multiprocessing Popen.terminate() catches PermissionError (#110037)
On Windows, multiprocessing Popen.terminate() now catchs
PermissionError and get the process exit code. If the process is
still running, raise again the PermissionError. Otherwise, the
process terminated as expected: store its exit code.
2023-09-29 02:41:12 +02:00
Dale Collison 74723e1110
gh-109461: Update logging module lock to use context manager (#109462)
Co-authored-by: Victor Stinner <vstinner@python.org>
2023-09-27 18:26:41 +02:00
Antoine Pitrou 0eb98837b6
gh-109593: Fix reentrancy issue in multiprocessing resource_tracker (#109629)
---------

Co-authored-by: blurb-it[bot] <43283697+blurb-it[bot]@users.noreply.github.com>
2023-09-26 13:57:25 +02:00
Serhiy Storchaka 0b4e090422
gh-109370: Fix unexpected traceback output in test_concurrent_futures (GH-109780)
Follow-up of gh-107219.

* Only close the connection writer on Windows.
* Also use existing constant _winapi.ERROR_OPERATION_ABORTED instead of
  WSA_OPERATION_ABORTED.
2023-09-26 10:06:07 +03:00
Victor Stinner a9b1f84790
gh-107219: Fix concurrent.futures terminate_broken() (#109244)
Fix a race condition in concurrent.futures. When a process in the
process pool was terminated abruptly (while the future was running or
pending), close the connection write end. If the call queue is
blocked on sending bytes to a worker process, closing the connection
write end interrupts the send, so the queue can be closed.

Changes:

* _ExecutorManagerThread.terminate_broken() now closes
  call_queue._writer.
* multiprocessing PipeConnection.close() now interrupts
  WaitForMultipleObjects() in _send_bytes() by cancelling the
  overlapped operation.
2023-09-11 08:11:31 +00:00
albanD add8d45cbe
gh-108520: Fix bad fork detection in nested multiprocessing use case (#108568)
gh-107275 introduced a regression where a SemLock would fail being passed along nested child processes, as the `is_fork_ctx` attribute would be left missing after the first deserialization.

---------

Co-authored-by: blurb-it[bot] <43283697+blurb-it[bot]@users.noreply.github.com>
Co-authored-by: Antoine Pitrou <pitrou@free.fr>
2023-08-30 17:07:41 +00:00
albanD 1700d34d31
gh-77377: Ensure multiprocessing SemLock is valid for spawn-based Process before serializing it (#107275)
Ensure multiprocessing SemLock is valid for spawn Process before serializing it.

Creating a multiprocessing SemLock with a fork context, and then trying to pass it to a spawn-created Process, would segfault if not detected early.

---------

Co-authored-by: blurb-it[bot] <43283697+blurb-it[bot]@users.noreply.github.com>
Co-authored-by: Antoine Pitrou <pitrou@free.fr>
2023-08-23 20:27:35 +00:00
Yuxin Wu a794ebeb02
More actionable error message when spawn is incorrectly used. (#102203)
Co-authored-by: Yuxin Wu <ppwwyyxx@users.noreply.github.com>
Co-authored-by: Oleg Iarygin <oleg@arhadthedev.net>
2023-08-15 18:03:45 -07:00
Dong-hee Na 6515ec3d3d
gh-107963: Fix set_forkserver_preload to check the type of given list (#107965)
gh-107963: Fix set_forkserver_preload to check the type of given list
2023-08-15 15:58:12 +02:00
Andrew Geng 5f7d4ecf30
gh-106558: break ref cycles through exceptions in multiprocessing manager (#106559) 2023-08-11 17:44:18 +00:00
shailshouryya fabcbe9c12
gh-106739: Add `rtype_cache` to `warnings.warn` message when leaked objects found (#106740)
Adding the `rtype_cache` to the `warnings.warn` message improves the
previous, somewhat vague message from

```
/Users/username/cpython/Lib/multiprocessing/resource_tracker.py:224: UserWarning: resource_tracker: There appear to be 6 leaked semaphore objects to clean up at shutdown
```

to

```
/Users/username/cpython/Lib/multiprocessing/resource_tracker.py:224: UserWarning: resource_tracker: There appear to be 6 leaked semaphore objects to clean up at shutdown: {'/mp-yor5cvj8', '/mp-10jx8eqr', '/mp-eobsx9tt', '/mp-0lml23vl', '/mp-9dgtsa_m', '/mp-frntyv4s'}
```

---------

Co-authored-by: blurb-it[bot] <43283697+blurb-it[bot]@users.noreply.github.com>
2023-07-25 17:27:36 +00:00
Gregory P. Smith c60df361ce
gh-90876: Restore the ability to import multiprocessing when `sys.executable` is `None` (#106464)
Prevent `multiprocessing.spawn` from failing to *import* in environments
where `sys.executable` is `None`.  This regressed in 3.11 with the addition
of support for path-like objects in multiprocessing.

Adds a test decorator to have tests only run when part of test_multiprocessing_spawn to `_test_multiprocessing.py` so we can start to avoid re-running the same not-global-state specific test in all 3 modes when there is no need.
2023-07-06 22:46:50 +00:00
Luccccifer ef5d00a592
gh-104536: Improve `multiprocessing.process._cleanup` logic (#104537)
Fix a race condition in the internal `multiprocessing.process` cleanup
logic that could manifest as an unintended `AttributeError` when calling
`BaseProcess.close()`.

---------

Co-authored-by: Oleg Iarygin <oleg@arhadthedev.net>
Co-authored-by: Gregory P. Smith <greg@krypto.org>
2023-05-22 03:48:57 +00:00
Christian Heimes 3ed57e4995
gh-61460: Stronger HMAC in multiprocessing (#20380)
bpo-17258:  `multiprocessing` now supports stronger HMAC algorithms for inter-process connection authentication rather than only HMAC-MD5.

Signed-off-by: Christian Heimes <christian@python.org>

gpshead: I Reworked to be more robust while keeping the idea.

The protocol modification idea remains, but we now take advantage of the
message length as an indicator of legacy vs modern protocol version.  No
more regular expression usage.  We now default to HMAC-SHA256, but do so
in a way that will be compatible when communicating with older clients
or older servers. No protocol transition period is needed.

More integration tests to verify these claims remain true are required. I'm
unaware of anyone depending on multiprocessing connections between
different Python versions.

---------

Signed-off-by: Christian Heimes <christian@python.org>
Co-authored-by: Gregory P. Smith [Google] <greg@krypto.org>
2023-05-20 23:33:09 +00:00
Gregory P. Smith d4c410f0f9
gh-84559: Remove the new multiprocessing warning, too disruptive. (#101551)
This reverts the core of #100618 while leaving relevant documentation
improvements and minor refactorings in place.
2023-02-03 15:20:46 -08:00
Gregory P. Smith 0ca67e6313
GH-84559: Deprecate fork being the multiprocessing default. (#100618)
This starts the process. Users who don't specify their own start method
and use the default on platforms where it is 'fork' will see a
DeprecationWarning upon multiprocessing.Pool() construction or upon
multiprocessing.Process.start() or concurrent.futures.ProcessPool use.

See the related issue and documentation within this change for details.
2023-02-02 15:50:35 -08:00
Nikita Sobolev ce39aaffee
gh-99509: Add `__class_getitem__` to `multiprocessing.queues.Queue` (#99511) 2022-12-26 20:50:55 -08:00
Arne de Laat a694b8222e
Fix typo in exception message in `multiprocessing.pool` (#99900) 2022-11-30 20:57:28 +05:30
Zackery Spytz 85c128e34d
bpo-40882: Fix a memory leak in SharedMemory on Windows (GH-20684)
In multiprocessing.shared_memory.SharedMemory(), the temporary view
returned by MapViewOfFile() should be unmapped when it is no longer
needed.
2022-11-25 17:39:48 +00:00
Gregory P. Smith abf5b6ff43
gh-61460: Add a comment describing the multiprocessing.connection protocol (gh-99623)
Describe the multiprocessing connection protocol.

It isn't a good protocol, but it is what it is.  This way we can more
easily reason about making changes to it in a backwards compatible way.
2022-11-20 10:20:04 -08:00
Gregory P. Smith 49f61068f4
gh-97514: Don't use Linux abstract sockets for multiprocessing (#98501)
Linux abstract sockets are insecure as they lack any form of filesystem
permissions so their use allows anyone on the system to inject code into
the process.

This removes the default preference for abstract sockets in
multiprocessing introduced in Python 3.9+ via
https://github.com/python/cpython/pull/18866 while fixing
https://github.com/python/cpython/issues/84031.

Explicit use of an abstract socket by a user now generates a
RuntimeWarning.  If we choose to keep this warning, it should be
backported to the 3.7 and 3.8 branches.
2022-10-20 15:30:09 -07:00
Steve Dower e48f9b2b7e
gh-98360: multiprocessing now spawns children on Windows with correct argv[0] in virtual environments (GH-98462) 2022-10-20 14:53:38 +01:00
Koki Saito 27e59afa2a
gh-97816: Remove unused variables in `mutliprocessing.managers.Server` (#97817)
Remove unused local variables.
2022-10-03 22:29:17 -07:00
Koki Saito 19ca114645
gh-96819: multiprocessing.resource_tracker: check if length of pipe write <= 512 (#96890)
Co-authored-by: Kumar Aditya <59607654+kumaraditya303@users.noreply.github.com>
2022-10-02 17:41:01 -07:00
Irit Katriel e37a158725
GH-83658: make multiprocessing.Pool raise an exception if maxtasksperchild is not None or a positive int (GH-93364)
Closes #83658.
2022-06-17 00:14:26 -07:00
samtygier 9a458befdd
gh-91577: SharedMemory move imports out of methods (#91579)
SharedMemory.unlink() uses the unregister() function from resource_tracker. Previously it was imported in the method, but this can fail if the method is called during interpreter shutdown, for example when unlink is part of a __del__() method.

Moving the import to the top of the file, means that the unregister() method is available during interpreter shutdown.

The register call in SharedMemory.__init__() can also use this imported resource_tracker.
2022-06-16 15:41:51 +02:00
Leo Trol 30610d2837
gh-90549: Fix leak of global named resources using multiprocessing spawn (#30617)
Co-authored-by: XD Trol <milestonejxd@gmail.com>
Co-authored-by: Antoine Pitrou <pitrou@free.fr>
2022-06-09 18:55:12 +02:00
Gregory P. Smith f6dd14c653
gh-82616: Add process_group support to subprocess.Popen (#23930)
One more thing that can help prevent people from using `preexec_fn`.

Also adds conditional skips to two tests exposing ASAN flakiness on the Ubuntu 20.04 Address Sanitizer Github CI system. When that build is run on more modern systems the "problem" does not show up. It seems ASAN implementation related.

Co-authored-by: Zackery Spytz <zspytz@gmail.com>
Co-authored-by: Serhiy Storchaka <storchaka@gmail.com>
2022-05-05 16:22:32 -07:00
Géry Ogam dfb1b9da8a
bpo-47029: Fix BrokenPipeError in multiprocessing.Queue at garbage collection and explicit close (#31913) 2022-05-03 17:49:57 -06:00
Ma Lin 1699128c48
GH-91355: micro-optimize Connection.send_bytes() method (gh-32247) 2022-05-03 23:41:45 +09:00
Alexander Shadchin f882d33778
Fix missing `f` prefix on f-strings (GH-91910) 2022-04-27 09:30:54 +03:00
Gregory P. Smith cd5726fe67
gh-91401: Add a failsafe way to disable vfork. (#91490)
Just in case there is ever an issue with _posixsubprocess's use of
vfork() due to the complexity of using it properly and potential
directions that Linux platforms where it defaults to on could take, this
adds a failsafe so that users can disable its use entirely by setting
a global flag.

No known reason to disable it exists. But it'd be a shame to encounter
one and not be able to use CPython without patching and rebuilding it.

See the linked issue for some discussion on reasoning.

Also documents the existing way to disable posix_spawn.
2022-04-25 16:19:39 -07:00
Géry Ogam 5576ddbbbc
bpo-46720: Add support for path-like objects to multiprocessing.set_executable for Windows (GH-31279)
This bring the API to be on a par with Unix-like systems.
2022-04-22 15:47:09 -07:00
Victor Stinner 061a8bf77c
gh-91231: Add shutdown_timeout to multiprocessing BaseManager (#32112)
Add an optional keyword 'shutdown_timeout' parameter to the
multiprocessing.BaseManager constructor. Kill the process if
terminate() takes longer than the timeout.

Multiprocessing tests pass test.support.SHORT_TIMEOUT
to BaseManager.shutdown_timeout.
2022-04-19 16:27:00 +02:00