Commit Graph

27 Commits

Author SHA1 Message Date
Victor Stinner c608477532
gh-120417: Remove unused imports in tests (part 1) (#120629) 2024-06-17 21:05:56 +02:00
Kirill Podoprigora d687d3fcfa
gh-118140: Make the``test_concurrent_futures.test_init`` quiet. (GH-118141)
Add stream argument to unittest.TextTestRunner call
2024-04-23 14:36:06 +02:00
Sam Gross 520cf2170e
gh-117688: Fix deadlock in test_no_stale_references with GIL disabled (#117720)
Check `my_object_collected.wait()` in a loop to give the main thread a
chance to merge the reference count fields. Additionally, call
`my_object_collected.set()` in a background thread to avoid deadlocking
when the destructor is called asynchronously via the eval breaker
within the body of of `my_object_collected.wait()`.
2024-04-15 12:54:56 -04:00
Malcolm Smith 1d3225ae05
gh-116622: Test updates for Android (#117299)
- re-enable test_fcntl_64_bit on Linux aarch64, but disable it on all
  Android ABIs
- use support.setswitchinterval in all relevant tests
- skip test_fma_zero_result on Android x86_64
- accept EACCES when calling os.get_terminal_size on Android
2024-04-05 20:57:36 +00:00
Sam Gross 42205143f8
gh-117478: Add `@support.requires_gil_enabled` decorator (#117479)
Co-authored-by: Kirill Podoprigora <kirill.bast9@mail.ru>
2024-04-04 14:10:46 -04:00
Sam Gross 397d88db5e
gh-117344: Skip flaky tests in free-threaded build (#117355)
The tests are not reliable with the GIL disabled. In theory, they can
fail with the GIL enabled too, but the failures are much more likely
with the GIL disabled.
2024-03-29 13:34:04 -04:00
Russell Keith-Magee f006338017
gh-114099: Additions to standard library to support iOS (GH-117052)
Co-authored-by: Hugo van Kemenade <1324225+hugovk@users.noreply.github.com>
Co-authored-by: Malcolm Smith <smith@chaquo.com>
Co-authored-by: Ned Deily <nad@python.org>
2024-03-28 03:59:33 -04:00
mpage 33da0e844c
gh-114271: Fix race in `Thread.join()` (#114839)
There is a race between when `Thread._tstate_lock` is released[^1] in `Thread._wait_for_tstate_lock()`
and when `Thread._stop()` asserts[^2] that it is unlocked. Consider the following execution
involving threads A, B, and C:

1. A starts.
2. B joins A, blocking on its `_tstate_lock`.
3. C joins A, blocking on its `_tstate_lock`.
4. A finishes and releases its `_tstate_lock`.
5. B acquires A's `_tstate_lock` in `_wait_for_tstate_lock()`, releases it, but is swapped
   out before calling `_stop()`.
6. C is scheduled, acquires A's `_tstate_lock` in `_wait_for_tstate_lock()` but is swapped
   out before releasing it.
7. B is scheduled, calls `_stop()`, which asserts that A's `_tstate_lock` is not held.
   However, C holds it, so the assertion fails.

The race can be reproduced[^3] by inserting sleeps at the appropriate points in
the threading code. To do so, run the `repro_join_race.py` from the linked repo.

There are two main parts to this PR:

1. `_tstate_lock` is replaced with an event that is attached to `PyThreadState`.
   The event is set by the runtime prior to the thread being cleared (in the same
   place that `_tstate_lock` was released). `Thread.join()` blocks waiting for the
   event to be set.
2. `_PyInterpreterState_WaitForThreads()` provides the ability to wait for all
   non-daemon threads to exit. To do so, an `is_daemon` predicate was added to
   `PyThreadState`. This field is set each time a thread is created. `threading._shutdown()`
   now calls into `_PyInterpreterState_WaitForThreads()` instead of waiting on
   `_tstate_lock`s.

[^1]: 441affc9e7/Lib/threading.py (L1201)
[^2]: 441affc9e7/Lib/threading.py (L1115)
[^3]: 8194653279

---------

Co-authored-by: blurb-it[bot] <43283697+blurb-it[bot]@users.noreply.github.com>
Co-authored-by: Antoine Pitrou <antoine@python.org>
2024-03-16 13:56:30 +01:00
Donghee Na 20578a1f68
gh-112536: Add TSAN builds on Github Actions (#116872) 2024-03-16 11:10:37 +01:00
Sam Gross 7d1abe9502
gh-116682: stdout may be empty in test_cancel_futures_wait_false (#116683)
If the `shutdown()` call happens before the worker thread starts executing
the task, then nothing will be printed to stdout.
2024-03-12 20:11:58 -04:00
Gregory P. Smith 92ce41cce1
gh-71052: fix test_concurrent_futures wasi regression. (#115923)
Fix the WASI test_concurrent_futures regression from #115917.
2024-02-26 00:02:56 +00:00
Malcolm Smith 4827968af8
gh-71052: Enable test_concurrent_futures on platforms that lack multiprocessing (gh-115917)
Enable test_concurrent_futures on platforms that support threading but not multiprocessing.
2024-02-25 11:38:18 -08:00
Petr Viktorin 4a9e6497c2
gh-104090: Add exit code to multiprocessing ResourceTracker (GH-115410)
This builds on https://github.com/python/cpython/pull/106807, which adds
a return code to ResourceTracker, to make future debugging easier.
Testing this “in situ” proved difficult, since the global ResourceTracker is
involved in test infrastructure. So, the tests here create a new instance and
feed it fake data.

---------

Co-authored-by: Yonatan Bitton <yonatan.bitton@perception-point.io>
Co-authored-by: Yonatan Bitton <bityob@gmail.com>
Co-authored-by: Antoine Pitrou <antoine@python.org>
2024-02-21 13:54:57 +01:00
Sam Gross a3af3cb4f4
gh-110481: Implement inter-thread queue for biased reference counting (#114824)
Biased reference counting maintains two refcount fields in each object:
`ob_ref_local` and `ob_ref_shared`. The true refcount is the sum of these two
fields. In some cases, when refcounting operations are split across threads,
the ob_ref_shared field can be negative (although the total refcount must be
at least zero). In this case, the thread that decremented the refcount
requests that the owning thread give up ownership and merge the refcount
fields.
2024-02-09 17:08:32 -05:00
Antoine Pitrou 0e9c364f4a
GH-110829: Ensure Thread.join() joins the OS thread (#110848)
Joining a thread now ensures the underlying OS thread has exited. This is required for safer fork() in multi-threaded processes.

---------

Co-authored-by: blurb-it[bot] <43283697+blurb-it[bot]@users.noreply.github.com>
2023-11-04 13:59:24 +00:00
elfstrom a376a72bd9
gh-109917: Fix test instability in test_concurrent_futures (#110306)
The test had an instability issue due to the ordering of the dummy
queue operation and the real wakeup pipe operations. Both primitives
are thread safe but not done atomically as a single update and may
interleave arbitrarily. With the old order of operations this can lead
to an incorrect state where the dummy queue is full but the wakeup
pipe is empty. By swapping the order in clear() I think this can no
longer happen in any possible operation interleaving (famous last
words).
2023-10-03 22:59:49 +01:00
Victor Stinner a46e960768
gh-109649: Use os.process_cpu_count() (#110165)
Replace os.cpu_count() with os.process_cpu_count() in modules:

* compileall
* concurrent.futures
* multiprocessing

Replace os.cpu_count() with os.process_cpu_count() in programs:

* _decimal deccheck.py test
* freeze.py
* multissltests.py
* python -m test (regrtest)
* wasm_build.py

Other changes:

* test.pythoninfo logs os.process_cpu_count().
* regrtest gets os.process_cpu_count() / os.cpu_count() in headers.
2023-10-01 03:14:57 +02:00
Victor Stinner 6351842121
gh-109047: concurrent.futures catches PythonFinalizationError (#109810)
concurrent.futures: The *executor manager thread* now catches
exceptions when adding an item to the *call queue*. During Python
finalization, creating a new thread can now raise RuntimeError. Catch
the exception and call terminate_broken() in this case.

Add test_python_finalization_error() to test_concurrent_futures.

concurrent.futures._ExecutorManagerThread changes:

* terminate_broken() no longer calls shutdown_workers() since the
  call queue is no longer working anymore (read and write ends of
  the queue pipe are closed).
* terminate_broken() now terminates child processes, not only
  wait until they complete.
* _ExecutorManagerThread.terminate_broken() now holds shutdown_lock
  to prevent race conditons with ProcessPoolExecutor.submit().

multiprocessing.Queue changes:

* Add _terminate_broken() method.
* _start_thread() sets _thread to None on exception to prevent
  leaking "dangling threads" even if the thread was not started
  yet.
2023-09-29 19:31:19 +00:00
Victor Stinner 9be283e5e1
gh-109594: Fix concurrent.futures test_timeout() (#110018)
Fix test_timeout() of test_concurrent_futures.test_wait. Remove the
future which may or may not complete depending if it takes longer
than the timeout ot not. Keep the second future which does not
complete before wait(). Make also the test faster: 0.5 second instead
of 6 seconds, so remove @support.requires_resource('walltime')
decorator.
2023-09-28 15:21:15 +02:00
Victor Stinner b1aebf1e65
gh-109565: Fix concurrent.futures test_future_times_out() (#109949)
as_completed() uses a timeout of 100 ms instead of 10 ms. Windows
monotonic clock resolution is around 15.6 ms.
2023-09-27 08:51:44 +00:00
Victor Stinner 2897142d2e
gh-109832: concurrent.futures test_deadlock restores sys.stderr (#109887)
test_error_at_task_unpickle() and
test_error_during_result_unpickle_in_result_handler() now restore
sys.stderr which is overriden by _raise_error_ignore_stderr().
2023-09-26 12:43:45 +02:00
elfstrom 405b06375a
gh-105829: Fix concurrent.futures.ProcessPoolExecutor deadlock (#108513)
This fixes issue #105829, https://github.com/python/cpython/issues/105829

Co-authored-by: blurb-it[bot] <43283697+blurb-it[bot]@users.noreply.github.com>
Co-authored-by: Antoine Pitrou <antoine@python.org>
Co-authored-by: Chris Withers <chris@withers.org>
Co-authored-by: Thomas Moreau <thomas.moreau.2010@gmail.com>
2023-09-22 13:55:56 +01:00
Victor Stinner 1eb1b45183
gh-109702: Increase concurrent_futures deadlock timeout (#109703)
Replace SHORT_TIMEOUT with LONG_TIMEOUT in test_deadlock of
test_concurrent_futures.
2023-09-22 03:13:31 +02:00
Victor Stinner a8cae4071c
gh-107219: Fix concurrent.futures terminate_broken() (#108974)
Fix a race condition in _ExecutorManagerThread.terminate_broken():
ignore the InvalidStateError on future.set_exception(). It can happen
if the future is cancelled before the caller.

Moreover, test_crash_big_data() now waits explicitly until the
executor completes.
2023-09-06 15:57:01 +02:00
Serhiy Storchaka 1e0d62793a
gh-108416: Mark slow but not CPU bound test methods with requires_resource('walltime') (GH-108480) 2023-09-05 17:56:30 +03:00
Serhiy Storchaka f3ba0a74cd
gh-108416: Mark slow test methods with @requires_resource('cpu') (GH-108421)
Only mark tests which spend significant system or user time,
by itself or in subprocesses.
2023-09-02 07:45:34 +03:00
Victor Stinner aa6f787faa
gh-108388: Convert test_concurrent_futures to package (#108401)
Convert test_concurrent_futures to a package of sub-tests.
2023-08-24 19:21:44 +02:00