Commit Graph

142 Commits

Author SHA1 Message Date
Steve Dower f56d132deb
gh-112984 Update Windows build and installer for free-threaded builds (GH-113129) 2024-01-17 21:52:23 +00:00
Serhiy Storchaka e08b70fab1
gh-108927: Fix removing testing modules from sys.modules (GH-108952)
It breaks import machinery if the test module has submodules used in
other tests.
2023-12-04 15:43:27 +00:00
Łukasz Langa 3932b0f7b1
gh-110722: Make `-m test -T -j` use sys.monitoring (GH-111710)
Now all results from worker processes are aggregated and
displayed together as a summary at the end of a regrtest run.

The traditional trace is left in place for use with sequential
in-process test runs but now raises a warning that those
numbers are not precise.

`-T -j` requires `--with-pydebug` as it relies on `-Xpresite=`.
2023-11-10 18:17:45 +01:00
Victor Stinner d9a5530d23
gh-110367: Make regrtest --verbose3 compatible with --huntrleaks -jN (#111577)
"./python -m test -j1 -R 3:3 --verbose3" now works as expected, since
run_single_test() does not replace sys.stdout with StringIO in this
case.
2023-11-01 04:28:55 +01:00
Serhiy Storchaka f6a45a03d0
gh-111165: Move test running code from test.support to libregrtest (GH-111166)
Remove no longer used functions run_unittest() and run_doctest() from
the test.support module.
2023-10-25 12:41:21 +03:00
Serhiy Storchaka 9a1fe09622
gh-110918: regrtest: allow to intermix --match and --ignore options (GH-110919)
Test case matching patterns specified by options --match, --ignore,
--matchfile and --ignorefile are now tested in the order of
specification, and the last match determines whether the test case be run
or ignored.
2023-10-21 17:44:46 +03:00
Victor Stinner 7237fb578d
gh-110932: Fix regrtest for SOURCE_DATE_EPOCH (#111143)
If the SOURCE_DATE_EPOCH environment variable is defined, use its
value as the random seed.
2023-10-21 10:37:48 +02:00
Zachary Ware b75186f69e
regrtest: Prepend 'use' options in --{fast,slow}-ci (GH-110363)
This allows individual resources to be disabled without having to explicitly re-enable all others.
2023-10-15 20:34:28 +02:00
Victor Stinner ce298a1c15
regrtest: --fast-ci/slow-ci no longer imply --rerun-fail (#110849) 2023-10-14 00:25:38 +00:00
Victor Stinner e0c4437793
Add support.MS_WINDOWS constant (#110446) 2023-10-06 02:37:28 +02:00
Victor Stinner af29282fce
gh-110367: Fix regrtest test_worker_output_on_failure() on ASAN build (#110387)
Set ASAN_OPTIONS="handle_segv=0" env var to run the test.
2023-10-05 14:42:36 +02:00
Victor Stinner 6592976061
gh-110367: Enhance regrtest -jN --verbose3 (#110368)
When using worker processes (-jN) with --verbose3 option, regrtest
can now display the worker output even if a worker process does
crash.  Previously, sys.stdout and sys.stderr were replaced and so
the worker output was lost on a crash.
2023-10-05 08:24:43 +02:00
Victor Stinner efd8c7a7c9
gh-109276: regrtest: shorter list of resources (#110326) 2023-10-04 09:39:50 +00:00
Nikita Sobolev 1465386720
gh-110171: `libregrtest` always sets `random.seed` (#110172) 2023-10-04 06:42:12 +00:00
Victor Stinner 65c285062c
gh-110164: regrtest disables random if SOURCE_DATE_EPOCH (#110168)
If the SOURCE_DATE_EPOCH environment variable is defined, regrtest
now disables randomization of tests.
2023-10-01 22:41:03 +02:00
Victor Stinner 53eb9a676f
gh-110152: regrtest handles cross compilation and HOSTRUNNER (#110156)
* _add_python_opts() now handles cross compilation and HOSTRUNNER.
* display_header() now tells if Python is cross-compiled, display
  HOSTRUNNER, and get the host platform.
* Remove Tools/scripts/run_tests.py script.
* Remove "make hostrunnertest": use "make buildbottest"
  or "make test" instead.
2023-09-30 22:37:23 +00:00
Victor Stinner 2c234196ea
gh-109276: regrtest: add WORKER_FAILED state (#110148)
Rename WORKER_ERROR to WORKER_BUG. Add WORKER_FAILED state: it does
not stop the manager, whereas WORKER_BUG does.

Change also TestResults.display_result() order: display failed tests
at the end, the important important information.

WorkerThread now tries to get the signal name for negative exit code.
2023-09-30 22:48:26 +02:00
Victor Stinner 5ae6c6d053
gh-109566: regrtest --fast-ci no longer enables --nowindows (#110121)
The --nowindows option is deprecated and does nothing but logs a
warning.
2023-09-29 19:49:39 +00:00
Victor Stinner 62881a79a8
gh-109566: regrtest doesn't enable --rerun if --python is used (#109969)
regrtest: --fast-ci and --slow-ci options no longer enable --rerun if
the --python option is used.
2023-09-27 16:09:23 +02:00
Victor Stinner 91fb8daa24
gh-109566: Fix regrtest Python options for WASM/WASI (#109954)
WASM and WASI buildbots use multiple PYTHON environment variables
such as PYTHONPATH and _PYTHON_HOSTRUNNER. Don't use -E if the
--python=COMMAND option is used.
2023-09-27 12:01:16 +02:00
Victor Stinner 9abba715e3
gh-109566: Fix regrtest code adding Python options (#109926)
* On Windows, use subprocess.run() instead of os.execv().
* Only add needed options
* Rename reexec parameter to _add_python_opts.
* Rename --no-reexec option to --dont-add-python-opts.
2023-09-26 21:59:11 +00:00
Victor Stinner fbfec5642e
gh-109566: regrtest reexecutes the process (#109909)
When --fast-ci or --slow-ci option is used, regrtest now replaces the
current process with a new process to add "-u -W default -bb -E"
options to Python.

Changes:

* PCbuild/rt.bat and Tools/scripts/run_tests.py no longer need to add
  "-u -W default -bb -E" options to Python: it's now done by
  regrtest.
* Fix Tools/scripts/run_tests.py: flush stdout before replacing the
  process. Previously, buffered messages were lost.
2023-09-26 20:46:52 +02:00
Victor Stinner 859618c8cd
gh-109566, regrtest: Add --fast-ci and --slow-ci options (#109570)
* Add --fast-ci and --slow-ci options to libregrtest:

  * --fast-ci uses a default timeout of 10 minutes and "-u all,-cpu"
    (skip slowest tests).
  * --slow-ci uses a default timeout of 20 minues and "-u all" (run
    all tests).

* regrtest header now lists test resources.
* Makefile changes:

  * "make test", "make hostrunnertest" and "make coverage-report" now
    use --fast-ci option and TESTTIMEOUT variable.
  * "make buildbottest" now uses "--slow-ci". Remove options which
    became redundant with "--slow-ci".
  * "make testall" and "make testuniversal" now use --slow-ci option
    and TESTTIMEOUT variable.
  * "make testall" now uses "find -exec rm ..." instead of
    "find ... -print|xargs rm ...", same as "make clean".

* GitHub Actions workflow:

  * Ubuntu and Address Sanitizer jobs now use "make test". Remove
    options which became redundant with "--fast-ci".
  * Windows jobs now use --fast-ci option.
  * Use -j0 to detect the number of CPUs.

* Set Makefile TESTTIMEOUT default to an empty string, since
  --slow-ci and --fast-ci use different default timeout. It's now
  accepted to pass "--timeout=" to regrtest: treated as not timeout.
* Tools/scripts/run_tests.py now uses --fast-ci option.
* Tools/buildbot/test.bat now uses --slow-ci option. Remove
  --timeout=1200 option, redundant with --slow-ci.
2023-09-26 17:22:50 +02:00
Victor Stinner 72fb39c965
gh-109276: regrtest re-runs "env changed" tests (#109831)
When a test fails with "env changed" and --rerun option is used, the
test is now re-run in verbose mode in a fresh process.
2023-09-25 16:21:01 +02:00
Victor Stinner faebed4a67
gh-109276: Enhance libregrtest results (#109828)
* Factorize code listing "bad / env changed / ..." tests.
* Add TestResults.is_all_good() method.
* Move "All 400 tests OK." to the end
* Move "Test suite interrupted by signal SIGINT." to the end.
2023-09-25 13:50:15 +00:00
Victor Stinner 74c72a2fc7
gh-109425: regrtest decodes worker stdout with backslashreplace (#109428)
libregrtest now decodes stdout of test worker processes with the
"backslashreplace" error handler to log corrupted stdout, instead of
failing with an error and not logging the stdout.
2023-09-14 23:37:37 +00:00
Victor Stinner 75cdd9a904
gh-109276: libregrtest: WASM use filename for JSON (#109340)
On Emscripten and WASI platforms, or if --python command line option
is used, libregrtest now uses a filename for the JSON file.

Emscripten and WASI buildbot workers run the main test process with a
different Python (Linux) which spawns Emscripten/WASI processes using
the command specified in --python command line option. Passing a file
descriptor from the parent process to the child process doesn't work
in this case.

* Add JsonFile and JsonFileType classes
* Add RunTests.json_file_use_filename() method.
* Add a test in test_regrtest on the --python command line option.
* test_regrtest: add parallel=False parameter.
* Split long RunWorkers._runtest() function into sub-functions.
2023-09-13 00:41:25 +02:00
Victor Stinner a84cb74d42
gh-109276: libregrtest calls random.seed() before each test (#109279)
libregrtest now calls random.seed() before running each test file
when -r/--randomize command line option is used. Moreover, it's also
called in worker processes. It should help to make tests more
deterministic. Previously, it was only called once in the main
process before running all test files and it was not called in worker
processes.

* Convert some f-strings to regular strings in test_regrtest when
  f-string is not needed.
* Remove unused all_methods variable from test_regrtest.
* Add RunTests members are now mandatory.
2023-09-12 05:35:08 +02:00
Guido van Rossum bcce5e2718
gh-109039: Branch prediction for Tier 2 interpreter (#109038)
This adds a 16-bit inline cache entry to the conditional branch instructions POP_JUMP_IF_{FALSE,TRUE,NONE,NOT_NONE} and their instrumented variants, which is used to keep track of the branch direction.

Each time we encounter these instructions we shift the cache entry left by one and set the bottom bit to whether we jumped.

Then when it's time to translate such a branch to Tier 2 uops, we use the bit count from the cache entry to decided whether to continue translating the "didn't jump" branch or the "jumped" branch.

The counter is initialized to a pattern of alternating ones and zeros to avoid bias.

The .pyc file magic number is updated. There's a new test, some fixes for existing tests, and a few miscellaneous cleanups.
2023-09-11 18:20:24 +00:00
Victor Stinner 7aa8fcc8eb
gh-109162: libregrtest: use relative imports (#109250)
libregrtest.__init__ no longer exposes any symbol, so
"python -m test.libregrtest.worker" imports less modules.
2023-09-11 07:02:35 +00:00
Victor Stinner 1ec45378e9
gh-109162: libregrtest: add single.py and result.py (#109243)
* Add single.py and result.py files.
* Rename runtest.py to runtests.py.
* Move run_single_test() function and its helper functions to
  single.py.
* Move remove_testfn(), abs_module_name() and normalize_test_name()
  to utils.py.
* Move setup_support() to setup.py.
* Move type hints like TestName to utils.py.
* Rename runtest.py to runtests.py.
2023-09-11 00:07:18 +00:00
Victor Stinner a939b65aa6
gh-109162: libregrtest: add worker.py (#109229)
Add new worker.py file:

* Move create_worker_process() and worker_process() to this file.
* Add main() function to worker.py. create_worker_process() now
  runs the command: "python -m test.libregrtest.worker JSON".
* create_worker_process() now starts the worker process in the
  current working directory. Regrtest now gets the absolute path of
  the reflog.txt filename: -R command line option filename.
* Remove --worker-json command line option.
  Remove test_regrtest.test_worker_json().

Related changes:

* Add write_json() and from_json() methods to TestResult.
* Rename select_temp_dir() to get_temp_dir() and move it to utils.
* Rename make_temp_dir() to get_work_dir() and move it to utils.
  It no longer calls os.makedirs(): Regrtest.main() now calls it.
* Move fix_umask() to utils. The function is now called by
  setup_tests().
* Move StrPath to utils.
* Add exit_timeout() context manager to utils.
* RunTests: Replace junit_filename (StrPath) with use_junit (bool).
2023-09-11 01:11:22 +02:00
Victor Stinner 24fa8f2046
gh-109162: libregrtest: fix _decode_worker_job() (#109202)
Decode also HuntRefleak() object inside the RunTests object.

Add an unit test on huntrleaks with multiprocessing (-R -jN).
2023-09-10 00:51:24 +02:00
Victor Stinner a56c928756
gh-109162: Refactor libregrtest WorkerJob (#109171)
* Rename --worker-args command line option to --worker-json.
* Rename _parse_worker_args() to _parse_worker_json().
* WorkerJob:

  * Add runtests attribute
  * Remove test_name and rerun attribute

* Rename run_test_in_subprocess() to create_worker_process().
* Rename run_tests_worker() to worker_process().
* create_worker_process() uses json.dump(): write directly JSON to
  stdout.
* Convert MultiprocessResult to a frozen dataclass.
* Rename RunTests.match_tests to RunTests.match_tests_dict.
2023-09-09 01:03:39 +00:00
Victor Stinner ac8409b38b
gh-109162: Regrtest copies 'ns' attributes (#109168)
* Regrtest.__init__() now copies 'ns' namespace attributes to
  Regrtest attributes. Regrtest match_tests and ignore_tests
  attributes have type FilterTuple (tuple), instead of a list.
* Add RunTests.copy(). Regrtest._rerun_failed_tests() now uses
  RunTests.copy().
* Replace Regrtest.all_tests (list) with Regrtest.first_runtests
  (RunTests).
* Make random_seed maximum 10x larger (9 digits, instead of 8).
2023-09-08 23:48:54 +00:00
Serhiy Storchaka 04a0830b00
gh-89392: Remove support of test_main() in libregrtest (GH-108876) 2023-09-05 08:36:43 +03:00
Victor Stinner 1170d5a292
gh-108834: regrtest --fail-rerun exits with code 5 (#108896)
When the --fail-rerun option is used and a test fails and then pass,
regrtest now uses exit code 5 ("rerun) instead of 2 ("bad test").
2023-09-05 01:09:42 +00:00
Victor Stinner 31c2945f14
gh-108834: regrtest reruns failed tests in subprocesses (#108839)
When using --rerun option, regrtest now re-runs failed tests
in verbose mode in fresh worker processes to have more
deterministic behavior. So it can write its final report even
if a test killed a worker progress.

Add --fail-rerun option to regrtest: exit with non-zero exit code
if a test failed pass passed when re-run in verbose mode (in a
fresh process). That's now more useful since tests can pass
when re-run in a fresh worker progress, whereas they failed
when run after other tests when tests are run sequentially.

Rename --verbose2 option (-w) to --rerun. Keep --verbose2 as a
deprecated alias.

Changes:

* Fix and enhance statistics in regrtest summary. Add "(filtered)"
  when --match and/or --ignore options are used.
* Add RunTests class.
* Add TestResult.get_rerun_match_tests() method
* Rewrite code to serialize/deserialize worker arguments as JSON
  using a new WorkerJob class.
* Fix stats when a test is run with --forever --rerun.
* If failed test names cannot be parsed, log a warning and don't
  filter tests.
* test_regrtest.test_rerun_success() now uses a marker file, since
  the test is re-run in a separated process.
* Add tests on normalize_test_name() function.
* Add test_success() and test_skip() tests to test_regrtest.
2023-09-03 21:37:15 +00:00
Victor Stinner d4e534cbb3
regrtest computes statistics (#108793)
test_netrc, test_pep646_syntax and test_xml_etree now return results
in the test_main() function.

Changes:

* Rewrite TestResult as a dataclass with a new State class.
* Add test.support.TestStats class and Regrtest.stats_dict attribute.
* libregrtest.runtest functions now modify a TestResult instance
  in-place.
* libregrtest summary lists the number of run tests and skipped
  tests, and denied resources.
* Add TestResult.has_meaningful_duration() method.
* Compute TestResult duration in the upper function.
* Use time.perf_counter() instead of time.monotonic().
* Regrtest: rename 'resource_denieds' attribute to 'resource_denied'.
* Rename CHILD_ERROR to MULTIPROCESSING_ERROR.
* Use match/case syntadx to have different code depending on the
  test state.

Co-authored-by: Alex Waygood <Alex.Waygood@Gmail.com>
2023-09-02 18:09:36 +02:00
Guido van Rossum 76ce537fb1
Fix test_regrtest when run with uops always on (#108778)
The fix has two parts:
- When `-X uops` is detected, pass it on to the subprocess created to run the manufactured test.
  I need this so I can run `./python -Xuops -m test test_regrtest` and see it fail without the next fix.
- Use `-R 6:3:` in `ArgsTestCase.test_huntrleaks` instead of `-R 3:3:` -- it takes longer to settle with `-X uops`.
2023-09-01 16:27:09 -07:00
Victor Stinner 2ac3eec103
gh-101634: regrtest reports decoding error as failed test (#106169)
When running the Python test suite with -jN option, if a worker stdout
cannot be decoded from the locale encoding report a failed testn so the
exitcode is non-zero.
2023-06-28 04:26:52 +02:00
Nikita Sobolev 995386071f
bpo-46523: fix tests rerun when `setUp[Class|Module]` fails (#30895)
Co-authored-by: Jelle Zijlstra <jelle.zijlstra@gmail.com>
Co-authored-by: Łukasz Langa <lukasz@langa.pl>
2023-04-07 12:43:41 +02:00
Victor Stinner c76db37c0d
gh-98903: Test suite fails with exit code 4 if no tests ran (#98904)
The Python test suite now fails wit exit code 4 if no tests ran. It
should help detecting typos in test names and test methods.

* Add "EXITCODE_" constants to Lib/test/libregrtest/main.py.
* Fix a typo: "NO TEST RUN" becomes "NO TESTS RAN"
2022-11-02 15:37:40 +01:00
Victor Stinner 4f85cec9e2
gh-93353: regrtest supports checking tmp files with -j2 (#93909)
regrtest now also implements checking for leaked temporary files and
directories when using -jN for N >= 2. Use tempfile.mkdtemp() to
create the temporary directory. Skip this check on WASI.
2022-06-16 21:48:26 +02:00
Victor Stinner 36934a16e8
gh-93353: Fix regrtest for -jN with N >= 2 (GH-93813) 2022-06-14 18:04:53 +02:00
Victor Stinner e566ce5496
gh-93353: regrtest checks for leaked temporary files (#93776)
When running tests with -jN, create a temporary directory per process
and mark a test as "environment changed" if a test leaks a temporary
file or directory.
2022-06-14 13:43:02 +02:00
Victor Stinner 3ceb4b8d3a
gh-84623: Remove unused imports in tests (#93772) 2022-06-13 16:56:03 +02:00
Victor Stinner 5185956527
gh-69443: Add test.support.Py_DEBUG constant (#93226) 2022-05-26 00:12:54 +02:00
Serhiy Storchaka 54d068adfb
gh-91904: Fix setting envvar PYTHONREGRTEST_UNICODE_GUARD (GH-91905)
It always failed on non-UTF-8 locale and prevented running regrtests.
2022-04-25 17:35:14 +03:00
Christian Heimes 8464fbc42e
bpo-40280: Skip subprocess-based tests on wasm32-emscripten (GH-30615) 2022-01-25 08:09:06 +01:00