Commit Graph

189 Commits

Author SHA1 Message Date
Gregory P. Smith d4c410f0f9
gh-84559: Remove the new multiprocessing warning, too disruptive. (#101551)
This reverts the core of #100618 while leaving relevant documentation
improvements and minor refactorings in place.
2023-02-03 15:20:46 -08:00
Gregory P. Smith 0ca67e6313
GH-84559: Deprecate fork being the multiprocessing default. (#100618)
This starts the process. Users who don't specify their own start method
and use the default on platforms where it is 'fork' will see a
DeprecationWarning upon multiprocessing.Pool() construction or upon
multiprocessing.Process.start() or concurrent.futures.ProcessPool use.

See the related issue and documentation within this change for details.
2023-02-02 15:50:35 -08:00
Serhiy Storchaka 76f43fc09a
gh-60203: Always pass True/False as boolean arguments in tests (GH-99983)
Unless we explicitly test non-bool values.
2022-12-04 14:28:56 +02:00
Jack Hindmarch a3be874334
gh-92886: Fix tests that fail when running with optimizations (`-O`) in `_test_multiprocessing.py` (GH-93233) 2022-10-19 11:07:07 +01:00
Koki Saito 19ca114645
gh-96819: multiprocessing.resource_tracker: check if length of pipe write <= 512 (#96890)
Co-authored-by: Kumar Aditya <59607654+kumaraditya303@users.noreply.github.com>
2022-10-02 17:41:01 -07:00
Koki Saito 14d4f68ebb
gh-97706: multiprocessing tests: Delete unused variable `rand` (#97707) 2022-10-02 17:37:36 -07:00
Christian Heimes 5956de16cd
gh-95212: make multiprocessing test case parallel-safe (GH-95213) 2022-07-25 06:44:40 +02:00
Kumar Aditya cfafd3adf8
GH-94736: mark SemLock test as linux only (GH-94750)
See https://buildbot.python.org/all/#/builders/172/builds/2522
The PR skips the test on non-linux platforms.

Automerge-Triggered-By: GH:pablogsal
2022-07-11 07:35:47 -07:00
Kumar Aditya f5b76330cf
GH-94736: Fix _multiprocessing.SemLock subclassing (#94738)
* fix allocator and deallocator

* 📜🤖 Added by blurb_it.

* code review

Co-authored-by: blurb-it[bot] <43283697+blurb-it[bot]@users.noreply.github.com>
2022-07-11 13:12:36 +01:00
Irit Katriel e37a158725
GH-83658: make multiprocessing.Pool raise an exception if maxtasksperchild is not None or a positive int (GH-93364)
Closes #83658.
2022-06-17 00:14:26 -07:00
Victor Stinner 50e0866f87
Tests call sleeping_retry() with SHORT_TIMEOUT (#93870)
Tests now call busy_retry() and sleeping_retry() with SHORT_TIMEOUT
or LONG_TIMEOUT (of test.support), rather than hardcoded constants.

Add also WAIT_ACTIVE_CHILDREN_TIMEOUT constant to
_test_multiprocessing.
2022-06-15 18:49:14 +02:00
Victor Stinner 7e9eaad864
Add test.support.busy_retry() (#93770)
Add busy_retry() and sleeping_retry() functions to test.support.
2022-06-15 11:42:10 +02:00
Leo Trol 30610d2837
gh-90549: Fix leak of global named resources using multiprocessing spawn (#30617)
Co-authored-by: XD Trol <milestonejxd@gmail.com>
Co-authored-by: Antoine Pitrou <pitrou@free.fr>
2022-06-09 18:55:12 +02:00
Géry Ogam 5576ddbbbc
bpo-46720: Add support for path-like objects to multiprocessing.set_executable for Windows (GH-31279)
This bring the API to be on a par with Unix-like systems.
2022-04-22 15:47:09 -07:00
Victor Stinner 061a8bf77c
gh-91231: Add shutdown_timeout to multiprocessing BaseManager (#32112)
Add an optional keyword 'shutdown_timeout' parameter to the
multiprocessing.BaseManager constructor. Kill the process if
terminate() takes longer than the timeout.

Multiprocessing tests pass test.support.SHORT_TIMEOUT
to BaseManager.shutdown_timeout.
2022-04-19 16:27:00 +02:00
Victor Stinner 9204bb72a2
bpo-46633: Skip tests on ASAN and/or MSAN builds (GH-31632)
Skip tests on ASAN and/or MSAN builds:

* multiprocessing tests
* test___all__
* test_concurrent_futures
* test_decimal
* test_peg_generator
* test_tools

The ASAN job of GitHub Actions no longer excludes these tests.
2022-03-01 15:44:08 +01:00
Charlie Zhao e466faa9df
bpo-45735: Promise the long-time truth that `args=list` works (GH-30982)
For threads, and for multiprocessing, it's always been the case that ``args=list`` works fine when passed to ``Process()`` or ``Thread()``, and such code is common in the wild. But, according to the docs, only a tuple can be used. This brings the docs into synch with reality.

Doc changes by Charlie Zhao.
Co-authored-by: Tim Peters <tim.peters@gmail.com>
2022-02-25 22:17:13 -06:00
Kumar Aditya af6b406885
bpo-25066: Added repr for multiprocessing.Event (GH-29749)
Co-authored-by: Pablo Galindo Salgado <Pablogsal@gmail.com>
2021-12-09 13:16:45 +00:00
Nikita Sobolev 746d648d47
bpo-45125: Improves pickling docs and tests for `shared_memory` (GH-28294) 2021-10-01 13:45:59 +03:00
Serhiy Storchaka eb4495e8e2
bpo-45310: Fix parrallel shared memory tests (GH-28661)
Add a PID to names of POSIX shared memory objects to allow
running multiprocessing tests (test_multiprocessing_fork,
test_multiprocessing_spawn, etc) in parallel.
2021-10-01 10:56:32 +03:00
Nikita Sobolev f604cf1c37
bpo-45209: fix `UserWarning: resource_tracker` in test_multiprocessing (GH-28377) 2021-09-21 19:49:42 +02:00
Nikita Sobolev 19871fce3b
bpo-45052: Unskips a failing `test_shared_memory_basics` test (GH-28182) 2021-09-06 17:55:34 +01:00
Nikita Sobolev dd7b816ac8
bpo-45042: Now test classes decorated with `requires_hashdigest` are not skipped (GH-28060)
Co-authored-by: Serhiy Storchaka <storchaka@gmail.com>
2021-09-04 23:42:36 +03:00
Serhiy Storchaka 2a8127cafe
bpo-25130: Add calls of gc.collect() in tests to support PyPy (GH-28005) 2021-08-29 14:04:40 +03:00
finefoot 85b920498b
bpo-30256: Add manager_owned keyword arg to AutoProxy (GH-16341)
Co-authored-by: Jordan Speicher <jordan@jspeicher.com>
2021-07-01 20:45:02 -07:00
Inada Naoki 35715d1e72
bpo-43651: PEP 597: Fix EncodingWarning in some tests (GH-25145)
* test_asyncio
* test_bz2
* test_math
* test_cmath
* test_cmd_line
* test_cmd_line_script
* test_compile
* test_contextlib
* test_profile
* ctypes/test/test_find
* test_multiprocessing
* test_configparser
* test_csv
* test_dbm_dumb
* test_decimal
* test_difflib
* os.fdopen() calls io.text_encoding() to emit EncodingWarning for right place.
2021-04-04 09:01:23 +09:00
Vinay Sharma 475a5fbb56
bpo-41344: Raise ValueError when creating shared memory of size 0 (GH-21556) 2020-08-30 20:03:11 +01:00
Victor Stinner fbf43f051e
bpo-41521: Rename blacklist parameter to not_exported (GH-21824)
Rename "blacklist" parameter of test.support.check__all__() to
"not_exported".
2020-08-17 07:20:40 +02:00
Hai Shi 598a951844
bpo-40275: Use new test.support helper submodules in tests (GH-21764) 2020-08-07 17:18:38 +02:00
Vinay Sharma bfd0fbdc13
bpo-38169: Increase code coverage for SharedMemory and ShareableList (GH-16139) 2020-07-19 22:35:52 +09:00
Hai Shi 0c4f0f3b29
bpo-40275: Use new test.support helper submodules in tests (GH-21169) 2020-06-30 15:46:31 +02:00
Serhiy Storchaka 9355868458
bpo-41043: Escape literal part of the path for glob(). (GH-20994) 2020-06-20 11:10:31 +03:00
Victor Stinner ddbeb2f3e0
bpo-38377: Add support.skip_if_broken_multiprocessing_synchronize() (GH-20944)
On Linux, skip tests using multiprocessing if the current user cannot
create a file in /dev/shm/ directory. Add the
skip_if_broken_multiprocessing_synchronize() function to the
test.support module.
2020-06-18 14:53:19 +02:00
Christian Heimes b022e5cffb
bpo-17258: Add requires_hashdigest to multiprocessing tests (GH-20412)
Skip some :mod:`multiprocessing` tests when MD5 hash digest is blocked.

Signed-off-by: Christian Heimes <christian@python.org>
2020-06-04 05:48:17 -07:00
Hai Shi e80697d687
bpo-40275: Adding threading_helper submodule in test.support (GH-20263) 2020-05-28 00:10:27 +02:00
idomic db098bc1f0
bpo-39244: multiprocessing return default start method first on macOS (GH-18625) 2020-05-26 17:54:21 +03:00
Hai Shi caa3ef284a
bpo-40135: Fix multiprocessing test_shared_memory_across_processes() (GH-19892)
Don't define shared memory block's name in test_shared_memory_across_processes():
use SharedMemory(create=True) instead.
2020-05-04 17:05:54 +02:00
Victor Stinner 9adccc1384
bpo-30966: Add multiprocessing.SimpleQueue.close() (GH-19735)
Add a new close() method to multiprocessing.SimpleQueue to explicitly
close the queue.

Automerge-Triggered-By: @pitrou
2020-04-27 09:11:10 -07:00
Serhiy Storchaka 16994912c9
bpo-40275: Avoid importing socket in test.support (GH-19603)
* Move socket related functions from test.support to socket_helper.
* Import socket, nntplib and urllib.error lazily in transient_internet().
* Remove importing multiprocess.
2020-04-25 10:06:29 +03:00
Victor Stinner d663d34685
bpo-39983: Add test.support.print_warning() (GH-19683)
Log "Warning -- ..." test warnings into sys.__stderr__ rather than
sys.stderr, to ensure to display them even if sys.stderr is captured.

test.libregrtest.utils.print_warning() now calls
test.support.print_warning().
2020-04-23 19:03:52 +02:00
Antoine Pitrou eba9f6155d
bpo-40330: Fix utf-8 size check in ShareableList (GH-19606)
The item size must be checked after encoding to bytes, not before.

Automerge-Triggered-By: @pitrou
2020-04-20 11:54:55 -07:00
Victor Stinner 278c1e159c
bpo-40094: Add test.support.wait_process() (GH-19254)
Moreover, the following tests now check the child process exit code:

* test_os.PtyTests
* test_mailbox.test_lock_conflict()
* test_tempfile.test_process_awareness()
* test_uuid.testIssue8621()
* multiprocessing resource tracker tests
2020-03-31 20:08:12 +02:00
Batuhan Taşkaya ac10e0c932
bpo-39360: Ensure all workers exit when finalizing a multiprocessing Pool (GH-19009)
When the pull is not used via the context manager or terminate() is called, there is a system in multiprocessing.util that handles finalization of all pools via an atexit handler (the Finalize) class. This class registers the _terminate_pool handler in the registry of finalizers of the module, and that registry is called on interpreter exit via _exit_function. The problem is that the "happy" path with the context manager or manual call to finalize() does some extra steps that _terminate_pool does not. The step that is not executed when the atexit() handler calls _terminate_pool is pinging the _change_notifier queue to unblock the maintenance threads.

This commit moves the notification to the _terminate_pool function so is called from both code paths.

Co-authored-by: Pablo Galindo <Pablogsal@gmail.com>
2020-03-15 19:45:56 +00:00
Pablo Galindo 6012f30bef
bpo-39850: Add support for abstract sockets in multiprocessing (GH-18866) 2020-03-09 13:48:01 +00:00
Christopher Hunt c2ac4cf040
bpo-35727: Use exit code 0 on sys.exit() in multiprocessing.Process. (GH-11538) 2020-02-21 10:33:04 +01:00
Victor Stinner 9707e8e22d
bpo-38546: multiprocessing tests stop the resource tracker (GH-17641)
Multiprocessing and concurrent.futures tests now stop the resource
tracker process when tests complete.

Add ResourceTracker._stop() method to
multiprocessing.resource_tracker.

Add _cleanup_tests() helper function to multiprocessing.util: share
code between multiprocessing and concurrent.futures tests.
2019-12-17 18:37:26 +01:00
Victor Stinner 7772b1af5e
bpo-38614: Use support timeout constants (GH-17572) 2019-12-11 22:17:04 +01:00
Victor Stinner 0d63bacefd
bpo-38614: Use test.support.SHORT_TIMEOUT constant (GH-17566)
Replace hardcoded timeout constants in tests with SHORT_TIMEOUT of
test.support, so it's easier to ajdust this timeout for all tests at
once.

SHORT_TIMEOUT is 30 seconds by default, but it can be longer
depending on --timeout command line option.

The change makes almost all timeouts longer, except
test_reap_children() of test_support which is made 2x shorter:
SHORT_TIMEOUT should be enough. If this test starts to fail,
LONG_TIMEOUT should be used instead.

Uniformize also "from test import support" import in some test files.
2019-12-11 11:30:03 +01:00
Victor Stinner c98b0199a9
bpo-38614: Use test.support.LONG_TIMEOUT constant (GH-17562)
Replace hardcoded timeout constants in tests with LONG_TIMEOUT of
test.support, so it's easier to ajdust this timeout for all tests at
once.

LONG_TIMEOUT is 5 minutes by default, but it can be longer depending
on --timeout command line option.
2019-12-10 21:12:26 +01:00
Jake Tesler c6b20be85c bpo-38707: Fix for multiprocessing.Process MainThread.native_id (GH-17088)
This PR implements a fix for `multiprocessing.Process` objects; the error occurs when Processes are created using either `fork` or `forkserver` as the `start_method`.

In these instances, the `MainThread` of the newly created `Process` object retains all attributes from its parent's `MainThread` object, including the `native_id` attribute. The resulting behavior is such that the new process' `MainThread` captures an incorrect/outdated `native_id` (the parent's instead of its own). 

This change forces the Process object to update its `native_id` attribute during the bootstrap process.

cc @vstinner





https://bugs.python.org/issue38707



Automerge-Triggered-By: @pitrou
2019-11-19 11:50:12 -08:00