Commit Graph

12 Commits

Author SHA1 Message Date
Serhiy Storchaka f6a45a03d0
gh-111165: Move test running code from test.support to libregrtest (GH-111166)
Remove no longer used functions run_unittest() and run_doctest() from
the test.support module.
2023-10-25 12:41:21 +03:00
Victor Stinner 2c234196ea
gh-109276: regrtest: add WORKER_FAILED state (#110148)
Rename WORKER_ERROR to WORKER_BUG. Add WORKER_FAILED state: it does
not stop the manager, whereas WORKER_BUG does.

Change also TestResults.display_result() order: display failed tests
at the end, the important important information.

WorkerThread now tries to get the signal name for negative exit code.
2023-09-30 22:48:26 +02:00
Victor Stinner 859618c8cd
gh-109566, regrtest: Add --fast-ci and --slow-ci options (#109570)
* Add --fast-ci and --slow-ci options to libregrtest:

  * --fast-ci uses a default timeout of 10 minutes and "-u all,-cpu"
    (skip slowest tests).
  * --slow-ci uses a default timeout of 20 minues and "-u all" (run
    all tests).

* regrtest header now lists test resources.
* Makefile changes:

  * "make test", "make hostrunnertest" and "make coverage-report" now
    use --fast-ci option and TESTTIMEOUT variable.
  * "make buildbottest" now uses "--slow-ci". Remove options which
    became redundant with "--slow-ci".
  * "make testall" and "make testuniversal" now use --slow-ci option
    and TESTTIMEOUT variable.
  * "make testall" now uses "find -exec rm ..." instead of
    "find ... -print|xargs rm ...", same as "make clean".

* GitHub Actions workflow:

  * Ubuntu and Address Sanitizer jobs now use "make test". Remove
    options which became redundant with "--fast-ci".
  * Windows jobs now use --fast-ci option.
  * Use -j0 to detect the number of CPUs.

* Set Makefile TESTTIMEOUT default to an empty string, since
  --slow-ci and --fast-ci use different default timeout. It's now
  accepted to pass "--timeout=" to regrtest: treated as not timeout.
* Tools/scripts/run_tests.py now uses --fast-ci option.
* Tools/buildbot/test.bat now uses --slow-ci option. Remove
  --timeout=1200 option, redundant with --slow-ci.
2023-09-26 17:22:50 +02:00
Victor Stinner 72fb39c965
gh-109276: regrtest re-runs "env changed" tests (#109831)
When a test fails with "env changed" and --rerun option is used, the
test is now re-run in verbose mode in a fresh process.
2023-09-25 16:21:01 +02:00
Victor Stinner faebed4a67
gh-109276: Enhance libregrtest results (#109828)
* Factorize code listing "bad / env changed / ..." tests.
* Add TestResults.is_all_good() method.
* Move "All 400 tests OK." to the end
* Move "Test suite interrupted by signal SIGINT." to the end.
2023-09-25 13:50:15 +00:00
Alex Waygood 92ed7e4df1
gh-109413: Fix some trivial mypy nitpicks in libregrtest (#109454) 2023-09-15 17:01:28 +00:00
Alex Waygood 3b9d10b031
gh-109413: libregrtest: Add and improve type annotations (#109405) 2023-09-14 18:33:18 +00:00
Victor Stinner 7aa8fcc8eb
gh-109162: libregrtest: use relative imports (#109250)
libregrtest.__init__ no longer exposes any symbol, so
"python -m test.libregrtest.worker" imports less modules.
2023-09-11 07:02:35 +00:00
Victor Stinner 0c139b5f2f
gh-109162: libregrtest: rename runtest_mp.py to run_workers.py (#109248)
* Rename runtest_mp.py to run_workers.py
* Move exit_timeout() and temp_cwd() context managers from
  Regrtest.main() to Regrtest.run_tests(). Actions like --list-tests
  or --list-cases don't need these protections.
* Regrtest: remove selected and tests attributes. Pass 'selected' to
  list_tests(), list_cases() and run_tests(). display_result() now
  expects a TestTuple, instead of TestList.
* Rename setup_tests() to setup_process() and rename setup_support()
  to setup_tests().
* Move _adjust_resource_limits() to utils and rename it to
  adjust_rlimit_nofile().
* Move replace_stdout() to utils.
* Fix RunTests.verbose type: it's an int.
2023-09-11 03:27:37 +00:00
Victor Stinner 1ec45378e9
gh-109162: libregrtest: add single.py and result.py (#109243)
* Add single.py and result.py files.
* Rename runtest.py to runtests.py.
* Move run_single_test() function and its helper functions to
  single.py.
* Move remove_testfn(), abs_module_name() and normalize_test_name()
  to utils.py.
* Move setup_support() to setup.py.
* Move type hints like TestName to utils.py.
* Rename runtest.py to runtests.py.
2023-09-11 00:07:18 +00:00
Victor Stinner a939b65aa6
gh-109162: libregrtest: add worker.py (#109229)
Add new worker.py file:

* Move create_worker_process() and worker_process() to this file.
* Add main() function to worker.py. create_worker_process() now
  runs the command: "python -m test.libregrtest.worker JSON".
* create_worker_process() now starts the worker process in the
  current working directory. Regrtest now gets the absolute path of
  the reflog.txt filename: -R command line option filename.
* Remove --worker-json command line option.
  Remove test_regrtest.test_worker_json().

Related changes:

* Add write_json() and from_json() methods to TestResult.
* Rename select_temp_dir() to get_temp_dir() and move it to utils.
* Rename make_temp_dir() to get_work_dir() and move it to utils.
  It no longer calls os.makedirs(): Regrtest.main() now calls it.
* Move fix_umask() to utils. The function is now called by
  setup_tests().
* Move StrPath to utils.
* Add exit_timeout() context manager to utils.
* RunTests: Replace junit_filename (StrPath) with use_junit (bool).
2023-09-11 01:11:22 +02:00
Victor Stinner db5bfe91f8
gh-109162: libregrtest: add TestResults class (#109208)
* Add TestResults class.
* Move Regrtest methods to TestResults:

  * accumulate_result(): now takes a RunTests parameter
  * get_executed()
  * no_tests_run()

* Add methods to TestResults:

  * add_junit()
  * display_result()
  * display_summary()
  * need_rerun()
  * prepare_rerun()
  * write_junit()

* Rename 'need_rerun' attribute to 'bad_results'.
* Rename 'total_stats' attribute to 'stats'.
2023-09-10 02:30:43 +00:00