Split test_multiprocessing_fork, test_multiprocessing_forkserver and
test_multiprocessing_spawn into test packages. Each package is made
of 4 sub-tests: processes, threads, manager and misc. It allows
running more tests in parallel and so reduce the total test duration.
Ensure multiprocessing SemLock is valid for spawn Process before serializing it.
Creating a multiprocessing SemLock with a fork context, and then trying to pass it to a spawn-created Process, would segfault if not detected early.
---------
Co-authored-by: blurb-it[bot] <43283697+blurb-it[bot]@users.noreply.github.com>
Co-authored-by: Antoine Pitrou <pitrou@free.fr>
Prevent `multiprocessing.spawn` from failing to *import* in environments
where `sys.executable` is `None`. This regressed in 3.11 with the addition
of support for path-like objects in multiprocessing.
Adds a test decorator to have tests only run when part of test_multiprocessing_spawn to `_test_multiprocessing.py` so we can start to avoid re-running the same not-global-state specific test in all 3 modes when there is no need.
bpo-17258: `multiprocessing` now supports stronger HMAC algorithms for inter-process connection authentication rather than only HMAC-MD5.
Signed-off-by: Christian Heimes <christian@python.org>
gpshead: I Reworked to be more robust while keeping the idea.
The protocol modification idea remains, but we now take advantage of the
message length as an indicator of legacy vs modern protocol version. No
more regular expression usage. We now default to HMAC-SHA256, but do so
in a way that will be compatible when communicating with older clients
or older servers. No protocol transition period is needed.
More integration tests to verify these claims remain true are required. I'm
unaware of anyone depending on multiprocessing connections between
different Python versions.
---------
Signed-off-by: Christian Heimes <christian@python.org>
Co-authored-by: Gregory P. Smith [Google] <greg@krypto.org>
This starts the process. Users who don't specify their own start method
and use the default on platforms where it is 'fork' will see a
DeprecationWarning upon multiprocessing.Pool() construction or upon
multiprocessing.Process.start() or concurrent.futures.ProcessPool use.
See the related issue and documentation within this change for details.
Tests now call busy_retry() and sleeping_retry() with SHORT_TIMEOUT
or LONG_TIMEOUT (of test.support), rather than hardcoded constants.
Add also WAIT_ACTIVE_CHILDREN_TIMEOUT constant to
_test_multiprocessing.
Add an optional keyword 'shutdown_timeout' parameter to the
multiprocessing.BaseManager constructor. Kill the process if
terminate() takes longer than the timeout.
Multiprocessing tests pass test.support.SHORT_TIMEOUT
to BaseManager.shutdown_timeout.
Skip tests on ASAN and/or MSAN builds:
* multiprocessing tests
* test___all__
* test_concurrent_futures
* test_decimal
* test_peg_generator
* test_tools
The ASAN job of GitHub Actions no longer excludes these tests.
For threads, and for multiprocessing, it's always been the case that ``args=list`` works fine when passed to ``Process()`` or ``Thread()``, and such code is common in the wild. But, according to the docs, only a tuple can be used. This brings the docs into synch with reality.
Doc changes by Charlie Zhao.
Co-authored-by: Tim Peters <tim.peters@gmail.com>
Add a PID to names of POSIX shared memory objects to allow
running multiprocessing tests (test_multiprocessing_fork,
test_multiprocessing_spawn, etc) in parallel.
On Linux, skip tests using multiprocessing if the current user cannot
create a file in /dev/shm/ directory. Add the
skip_if_broken_multiprocessing_synchronize() function to the
test.support module.
* Move socket related functions from test.support to socket_helper.
* Import socket, nntplib and urllib.error lazily in transient_internet().
* Remove importing multiprocess.
Log "Warning -- ..." test warnings into sys.__stderr__ rather than
sys.stderr, to ensure to display them even if sys.stderr is captured.
test.libregrtest.utils.print_warning() now calls
test.support.print_warning().
Moreover, the following tests now check the child process exit code:
* test_os.PtyTests
* test_mailbox.test_lock_conflict()
* test_tempfile.test_process_awareness()
* test_uuid.testIssue8621()
* multiprocessing resource tracker tests
When the pull is not used via the context manager or terminate() is called, there is a system in multiprocessing.util that handles finalization of all pools via an atexit handler (the Finalize) class. This class registers the _terminate_pool handler in the registry of finalizers of the module, and that registry is called on interpreter exit via _exit_function. The problem is that the "happy" path with the context manager or manual call to finalize() does some extra steps that _terminate_pool does not. The step that is not executed when the atexit() handler calls _terminate_pool is pinging the _change_notifier queue to unblock the maintenance threads.
This commit moves the notification to the _terminate_pool function so is called from both code paths.
Co-authored-by: Pablo Galindo <Pablogsal@gmail.com>
Victor Stinner
albanD
Dong-hee Na
Andrew Geng
Gregory P. Smith
Christian Heimes
Serhiy Storchaka
Jack Hindmarch
Koki Saito
Kumar Aditya
Irit Katriel
Leo Trol
Géry Ogam
Charlie Zhao
Kumar Aditya
Nikita Sobolev
finefoot
Inada Naoki
Vinay Sharma
Hai Shi
idomic
Antoine Pitrou
Batuhan Taşkaya