Commit Graph

493 Commits

Author SHA1 Message Date
Eric Snow 8b209fd4f8
gh-76785: Expand How Interpreter Channels Handle Interpreter Finalization (gh-121805)
See 6b98b274b6 for an explanation of the problem and solution.  Here I've applied the solution to channels.
2024-07-15 19:43:59 +00:00
Eric Snow 6b98b274b6
gh-76785: Expand How Interpreter Queues Handle Interpreter Finalization (gh-116431)
Any cross-interpreter mechanism for passing objects between interpreters must be very careful to respect isolation, even when the object is effectively immutable (e.g. int, str).  Here this especially relates to when an interpreter sends one of its objects, and then is destroyed while the inter-interpreter machinery (e.g. queue) still holds a reference to the object.

When I added interpreters.Queue, I dealt with that case (using an atexit hook) by silently removing all items from the queue that were added by the finalizing interpreter.

Later, while working on concurrent.futures.InterpreterPoolExecutor (gh-116430), I noticed it was somewhat surprising when items were silently removed from the queue when the originating interpreter was destroyed.  (See my comment on that PR.) 
 It took me a little while to realize what was going on.  I expect that users, which much less context than I have, would experience the same pain.

My approach, here, to improving the situation is to give users three options:

1. return a singleton (interpreters.queues.UNBOUND) from Queue.get() in place of each removed item
2. raise an exception (interpreters.queues.ItemInterpreterDestroyed) from Queue.get() in place of each removed item
3. existing behavior: silently remove each item (i.e. Queue.get() skips each one)

The default will now be (1), but users can still explicitly opt in any of them, including to the silent removal behavior.

The behavior for each item may be set with the corresponding Queue.put() call. and a queue-wide default may be set when the queue is created.  (This is the same as I did for "synconly".)
2024-07-15 12:49:23 -06:00
sobolevn e2822360da
gh-121571: Do not use `EnvironmentError` in tests, use `OSError` instead (#121572) 2024-07-10 13:11:46 +03:00
Victor Stinner 7435f053b4
Move get_signal_name() to test.support (#121251)
* Move get_signal_name() from test.libregrtest to test.support.
* Use get_signal_name() in support.script_helper.
* support.script_helper now decodes stdout and stderr from UTF-8,
  instead of ASCII, if a command failed.
2024-07-02 10:34:13 +02:00
devdanzin 9e45fd9858
gh-121016: Add test for `PYTHON_BASIC_REPL` envioronment variable (#121017) 2024-06-26 10:39:07 +00:00
Victor Stinner e9f4d80fa6
gh-120417: Add #noqa: F401 to tests (#120627)
Ignore linter "imported but unused" warnings in tests when the linter
doesn't understand how the import is used.
2024-06-18 15:51:47 +00:00
Victor Stinner c608477532
gh-120417: Remove unused imports in tests (part 1) (#120629) 2024-06-17 21:05:56 +02:00
neonene 50a389565a
gh-117398: Add datetime C-API type check test for subinterpreters (gh-119604)
Check if the DateTime C-API type matches the datetime.date type on main and shared/isolated subinterpreters.
2024-06-13 12:05:03 -06:00
Eric Snow e6076d1e13
gh-119659: Get the datetime CAPI Tests Running Again (gh-120180)
The tests were accidentally disabled by 2da0dc0, which didn't handle classes correctly.

I considered updating no_rerun() to support classes, but the way test_datetime.py works would have made things fairly messy.  Plus, it looks like the refleaks we had encountered before have been resolved.
2024-06-07 11:44:56 -06:00
Sam Gross 47fb4327b5
gh-117657: Fix race involving immortalizing objects (#119927)
The free-threaded build currently immortalizes objects that use deferred
reference counting (see gh-117783). This typically happens once the
first non-main thread is created, but the behavior can be suppressed for
tests, in subinterpreters, or during a compile() call.

This fixes a race condition involving the tracking of whether the
behavior is suppressed.
2024-06-03 20:58:41 +00:00
Zac Hatfield-Dodds 055c739536
CI: set correct working directory for Hypothesis cache (GH-119345)
Set cwd for Hypothesis database
2024-05-29 14:13:18 +02:00
Nikita Sobolev 2da0dc094f
gh-119659: Move `@no_rerun` to `test.support` (#119660) 2024-05-28 18:50:50 +03:00
Geoffrey Thomas ef172521a9
Remove almost all unpaired backticks in docstrings (#119231)
As reported in #117847 and #115366, an unpaired backtick in a docstring
tends to confuse e.g. Sphinx running on subclasses of standard library
objects, and the typographic style of using a backtick as an opening
quote is no longer in favor. Convert almost all uses of the form

    The variable `foo' should do xyz

to

    The variable 'foo' should do xyz

and also fix up miscellaneous other unpaired backticks (extraneous /
missing characters).

No functional change is intended here other than in human-readable
docstrings.
2024-05-22 12:35:18 -04:00
Sam Gross 723d4d2fe8
gh-118527: Intern code consts in free-threaded build (#118667)
We already intern and immortalize most string constants. In the
free-threaded build, other constants can be a source of reference count
contention because they are shared by all threads running the same code
objects.
2024-05-06 20:12:39 -04:00
Hugo van Kemenade 3b3f8dea57
gh-117225: Move colorize functionality to own internal module (#118283) 2024-05-01 12:27:06 -06:00
Russell Keith-Magee 21336aa127
gh-118201: Accomodate flaky behavior of `os.sysconf` on iOS (GH-118453) 2024-04-30 22:31:00 -04:00
Guido van Rossum 7d83f7bcc4
gh-118335: Configure Tier 2 interpreter at build time (#118339)
The code for Tier 2 is now only compiled when configured
with `--enable-experimental-jit[=yes|interpreter]`.

We drop support for `PYTHON_UOPS` and -`Xuops`,
but you can disable the interpreter or JIT
at runtime by setting `PYTHON_JIT=0`.
You can also build it without enabling it by default
using `--enable-experimental-jit=yes-off`;
enable with `PYTHON_JIT=1`.

On Windows, the `build.bat` script supports
`--experimental-jit`, `--experimental-jit-off`,
`--experimental-interpreter`.

In the C code, `_Py_JIT` is defined as before
when the JIT is enabled; the new variable
`_Py_TIER2` is defined when the JIT *or* the
interpreter is enabled. It is actually a bitmask:
1: JIT; 2: default-off; 4: interpreter.
2024-04-30 18:26:34 -07:00
Sam Gross 7ccacb220d
gh-117783: Immortalize objects that use deferred reference counting (#118112)
Deferred reference counting is not fully implemented yet. As a temporary
measure, we immortalize objects that would use deferred reference
counting to avoid multi-threaded scaling bottlenecks.

This is only performed in the free-threaded build once the first
non-main thread is started. Additionally, some tests, including refleak
tests, suppress this behavior.
2024-04-29 14:36:02 -04:00
Kirill Podoprigora 51c70de998
gh-118351: Adapt support.TEST_MODULES_ENABLED for builds without the config variable (GH-118354) 2024-04-29 16:50:11 +01:00
Pablo Galindo Salgado 345e1e04ec
gh-112730: Make the test suite resilient to color-activation environment variables (#117672) 2024-04-24 21:25:22 +01:00
Eric Snow 03e3e31723
gh-76785: Rename _xxsubinterpreters to _interpreters (gh-117791)
See https://discuss.python.org/t/pep-734-multiple-interpreters-in-the-stdlib/41147/26.
2024-04-24 16:18:24 +00:00
Hugo van Kemenade 975081b11e
gh-117225: Add color to doctest output (#117583)
Co-authored-by: Alex Waygood <Alex.Waygood@Gmail.com>
2024-04-24 14:27:40 +03:00
Irit Katriel c179c0e6cb
gh-117680: make _PyInstructionSequence a PyObject and use it in tests (#117629) 2024-04-17 16:42:04 +01:00
Eric Snow fd259fdabe
gh-76785: Handle Legacy Interpreters Properly (gh-117490)
This is similar to the situation with threading._DummyThread.  The methods (incl. __del__()) of interpreters.Interpreter objects must be careful with interpreters not created by interpreters.create().  The simplest thing to start with is to disable any method that modifies or runs in the interpreter.  As part of this, the runtime keeps track of where an interpreter was created.  We also handle interpreter "refcounts" properly.
2024-04-11 23:23:25 +00:00
Sam Gross 25f6ff5d3e
gh-117649: Raise ImportError for unsupported modules in free-threaded build (#117651)
The free-threaded build does not currently support the combination of
single-phase init modules and non-isolated subinterpreters. Ensure that
`check_multi_interp_extensions` is always `True` for subinterpreters in
the free-threaded build so that importing these modules raises an
`ImportError`.
2024-04-11 15:00:54 -04:00
Eric Snow 993c3cca16
gh-76785: Add More Tests to test_interpreters.test_api (gh-117662)
In addition to the increase test coverage, this is a precursor to sorting out how we handle interpreters created directly via the C-API.
2024-04-10 18:37:01 -06:00
Erlend E. Aasland ca62ffd1a5
gh-116303: Skip tests if C recursion limit is unavailable (GH-117368)
The test suite fetches the C recursion limit from the _testcapi
extension module. Test extension modules can be disabled using the
--disable-test-modules configure option.
2024-04-08 14:45:25 +02:00
Sam Gross 42205143f8
gh-117478: Add `@support.requires_gil_enabled` decorator (#117479)
Co-authored-by: Kirill Podoprigora <kirill.bast9@mail.ru>
2024-04-04 14:10:46 -04:00
Eric Snow 976bcb2379
gh-76785: Raise InterpreterError, Not RuntimeError (gh-117489)
I had meant to switch everything to InterpreterError when I added it a while back.  At the time I missed a few key spots.

As part of this, I've added print-the-exception to _PyXI_InitTypes() and fixed an error case in `_PyStaticType_InitBuiltin().
2024-04-03 10:58:39 -06:00
Erlend E. Aasland ea94b3b149
gh-116303: Skip test module dependent tests if test modules are unavailable (#117341) 2024-04-03 15:11:36 +02:00
Eric Snow 857d3151c9
gh-76785: Consolidate Some Interpreter-related Testing Helpers (gh-117485)
This eliminates the duplication of functionally identical helpers in the _testinternalcapi and _xxsubinterpreters modules.
2024-04-02 23:16:50 +00:00
Eric Snow f341d6017d
gh-76785: Add PyInterpreterConfig Helpers (gh-117170)
These helpers make it easier to customize and inspect the config used to initialize interpreters.  This is especially valuable in our tests.  I found inspiration from the PyConfig API for the PyInterpreterConfig dict conversion stuff.  As part of this PR I've also added a bunch of tests.
2024-04-02 20:35:52 +00:00
Erlend E. Aasland 6c8ac8a32f
gh-116303: Handle disabled test modules in test.support helpers (#116482)
Make sure test.support helpers skip iso. failing if test extension
modules are disabled. Also log TEST_MODULES in test.pythoninfo.
2024-03-28 08:40:37 +00:00
Malcolm Smith 74c8568d07
gh-71042: Add `platform.android_ver` (#116674) 2024-03-27 17:53:27 +01:00
Victor Stinner a114d08a89
gh-92906: Enable test_cext and test_cppext on Windows (#117000)
On Windows in release mode, the test_cext and test_cppext can now
build C and C++ extensions.

* test_cext now also builds the C extension without options.
* test_cppext now also builds the C++ extension without options.
* Add C++14 test to test_cppext; C++11 is not supported by MSVC.
* Make setup_venv_with_pip_setuptools_wheel() quiet when
  support.verbose is false. Only show stdout and stderr on failure.
2024-03-19 14:03:27 +00:00
Antoine Pitrou 86bc40dd41
gh-112536: Add test_threading to TSAN tests (#116898) 2024-03-16 11:55:46 +00:00
Sam Gross 8e2aab7ad5
gh-116604: Fix test_gc on free-threaded build (#116662)
The free-threaded GC only does full collections, so it uses a threshold that
is a maximum of a fixed value (default 2000) and proportional to the number of
live objects. If there were many live objects after the previous collection,
then the threshold may be larger than 10,000 causing
`test_indirect_calls_with_gc_disabled` to fail.

This manually sets the threshold to `(1000, 0, 0)` for the test. The `0`
disables the proportional scaling.
2024-03-13 13:27:36 +00:00
Jason R. Coombs a254807761
gh-116307: Proper fix for 'mod' leaking across importlib tests (#116680)
gh-116307: Create a new import helper 'isolated modules' and use that instead of 'Clean Import' to ensure that tests from importlib_resources don't leave modules in sys.modules.
2024-03-12 21:36:21 +00:00
Malcolm Smith 872c0714fc
gh-71052: Change Android's `sys.platform` from "linux" to "android"
Co-authored-by: Erlend E. Aasland <erlend.aasland@protonmail.com>
2024-03-11 19:25:39 +00:00
Victor Stinner d9bcdda39c
gh-116417: Add _testlimitedcapi C extension (#116419)
Add a new C extension "_testlimitedcapi" which is only built with the
limited C API.

Move heaptype_relative.c and vectorcall_limited.c from
Modules/_testcapi/ to Modules/_testlimitedcapi/.

* configure: add _testlimitedcapi test extension.
* Update generate_stdlib_module_names.py.
* Update make check-c-globals.

Co-authored-by: Erlend E. Aasland <erlend.aasland@protonmail.com>
2024-03-07 18:31:12 +00:00
Eric Snow 4402b3cbcf
gh-76785: Minor Improvements to "interpreters" Module (gh-116328)
This includes adding pickle support to various classes, and small changes to improve the maintainability of the low-level _xxinterpqueues module.
2024-03-05 08:54:46 -07:00
Eric Snow 936d4611d6
gh-76785: Fix interpreters.Queue.get_nowait() (gh-116166)
I missed this change in gh-115566.
2024-03-01 16:36:35 +00:00
Malcolm Smith 41d5391c55
gh-71052: Add test exclusions to support running the test suite on Android (#115918) 2024-02-29 22:32:50 +01:00
Eric Snow e80abd57a8
gh-76785: Update test.support.interpreters to Align With PEP 734 (gh-115566)
This brings the code under test.support.interpreters, and the corresponding extension modules, in line with recent updates to PEP 734.

(Note: PEP 734 has not been accepted at this time.  However, we are using an internal copy of the implementation in the test suite to exercise the existing subinterpreters feature.)
2024-02-28 16:08:08 -07:00
Petr Viktorin 7acf1fb5a7
gh-114911: Add CPUStopwatch test helper (GH-114912)
A few of our tests measure the time of CPU-bound operation, mainly
to avoid quadratic or worse behaviour.
Add a helper to ignore GC and time spent in other processes.
2024-02-28 12:53:48 +01:00
Donghee Na a2d4281415
gh-112087: Make __sizeof__ and listiter_{len, next} to be threadsafe (gh-114843) 2024-02-15 02:00:50 +09:00
Russell Keith-Magee 391659b3da
gh-114099: Add test exclusions to support running the test suite on iOS (#114889)
Add test annotations required to run the test suite on iOS (PEP 730).

The majority of the change involve annotating tests that use subprocess,
but are skipped on Emscripten/WASI for other reasons, and including
iOS/tvOS/watchOS under the same umbrella as macOS/darwin checks.

`is_apple` and `is_apple_mobile` test helpers have been added to
identify *any* Apple platform, and "any Apple platform except macOS",
respectively.
2024-02-05 01:04:57 +01:00
Nikita Sobolev ba253a4794
gh-108303: Move `.whl` test files to `Lib/test/wheeldata/` (#114343) 2024-01-23 16:33:12 +01:00
Nikita Sobolev 8f5e7d739f
gh-108303: Move `smtpd` to `test.support` (#114368)
Update test_logging.py and test_smtplib.py.
2024-01-22 14:36:29 +01:00
Ronald Oussoren 5adff3f86f
gh-80931: Skip some socket tests while hunting for refleaks on macOS (#114057)
Some socket tests related to sending file descriptors cause a file descriptor leak on macOS, all of them tests that send one or more descriptors than cannot be received on the read end.  This appears to be a platform bug.

This PR skips those tests when doing a refleak test run to avoid hiding other problems.
2024-01-21 11:25:15 +01:00