* test.bisect_cmd now exit with code 0 on success, and code 1 on
failure. Before, it was the opposite.
* test.bisect_cmd now runs the test worker process with
-X faulthandler.
* regrtest RunTests: Add create_python_cmd() and bisect_cmd()
methods.
Now all results from worker processes are aggregated and
displayed together as a summary at the end of a regrtest run.
The traditional trace is left in place for use with sequential
in-process test runs but now raises a warning that those
numbers are not precise.
`-T -j` requires `--with-pydebug` as it relies on `-Xpresite=`.
Rename WORKER_ERROR to WORKER_BUG. Add WORKER_FAILED state: it does
not stop the manager, whereas WORKER_BUG does.
Change also TestResults.display_result() order: display failed tests
at the end, the important important information.
WorkerThread now tries to get the signal name for negative exit code.
* Add --fast-ci and --slow-ci options to libregrtest:
* --fast-ci uses a default timeout of 10 minutes and "-u all,-cpu"
(skip slowest tests).
* --slow-ci uses a default timeout of 20 minues and "-u all" (run
all tests).
* regrtest header now lists test resources.
* Makefile changes:
* "make test", "make hostrunnertest" and "make coverage-report" now
use --fast-ci option and TESTTIMEOUT variable.
* "make buildbottest" now uses "--slow-ci". Remove options which
became redundant with "--slow-ci".
* "make testall" and "make testuniversal" now use --slow-ci option
and TESTTIMEOUT variable.
* "make testall" now uses "find -exec rm ..." instead of
"find ... -print|xargs rm ...", same as "make clean".
* GitHub Actions workflow:
* Ubuntu and Address Sanitizer jobs now use "make test". Remove
options which became redundant with "--fast-ci".
* Windows jobs now use --fast-ci option.
* Use -j0 to detect the number of CPUs.
* Set Makefile TESTTIMEOUT default to an empty string, since
--slow-ci and --fast-ci use different default timeout. It's now
accepted to pass "--timeout=" to regrtest: treated as not timeout.
* Tools/scripts/run_tests.py now uses --fast-ci option.
* Tools/buildbot/test.bat now uses --slow-ci option. Remove
--timeout=1200 option, redundant with --slow-ci.
* Factorize code listing "bad / env changed / ..." tests.
* Add TestResults.is_all_good() method.
* Move "All 400 tests OK." to the end
* Move "Test suite interrupted by signal SIGINT." to the end.
* Rename runtest_mp.py to run_workers.py
* Move exit_timeout() and temp_cwd() context managers from
Regrtest.main() to Regrtest.run_tests(). Actions like --list-tests
or --list-cases don't need these protections.
* Regrtest: remove selected and tests attributes. Pass 'selected' to
list_tests(), list_cases() and run_tests(). display_result() now
expects a TestTuple, instead of TestList.
* Rename setup_tests() to setup_process() and rename setup_support()
to setup_tests().
* Move _adjust_resource_limits() to utils and rename it to
adjust_rlimit_nofile().
* Move replace_stdout() to utils.
* Fix RunTests.verbose type: it's an int.
* Add single.py and result.py files.
* Rename runtest.py to runtests.py.
* Move run_single_test() function and its helper functions to
single.py.
* Move remove_testfn(), abs_module_name() and normalize_test_name()
to utils.py.
* Move setup_support() to setup.py.
* Move type hints like TestName to utils.py.
* Rename runtest.py to runtests.py.
Add new worker.py file:
* Move create_worker_process() and worker_process() to this file.
* Add main() function to worker.py. create_worker_process() now
runs the command: "python -m test.libregrtest.worker JSON".
* create_worker_process() now starts the worker process in the
current working directory. Regrtest now gets the absolute path of
the reflog.txt filename: -R command line option filename.
* Remove --worker-json command line option.
Remove test_regrtest.test_worker_json().
Related changes:
* Add write_json() and from_json() methods to TestResult.
* Rename select_temp_dir() to get_temp_dir() and move it to utils.
* Rename make_temp_dir() to get_work_dir() and move it to utils.
It no longer calls os.makedirs(): Regrtest.main() now calls it.
* Move fix_umask() to utils. The function is now called by
setup_tests().
* Move StrPath to utils.
* Add exit_timeout() context manager to utils.
* RunTests: Replace junit_filename (StrPath) with use_junit (bool).
Victor Stinner
Alex Waygood
Łukasz Langa
Serhiy Storchaka