* bpo-33718: regrtest: use format_duration() to display failed tests (GH-7686)
* Enhance also format_duration(): work on integers and rounds towards
+infinity (math.ceil).
* Write unit tests on format_duration()
(cherry picked from commit 4ffe9c2b25)
* bpo-33873: regrtest: Add warning on -R 1:3 (GH-7736)
regrtest: Add warning when using less than 3 warmup runs like -R 1:3.
(cherry picked from commit cac4fef886)
* bpo-33873: Fix bug in `runtest.py` and add checks for invalid `-R` parameters (GH-7735)
Fix bug in `Lib/test/libregrtest/runtest.py` that makes running tests an extra time than the specified number of runs.
Add check for invalid --huntrleaks/-R parameters.
(cherry picked from commit 58ed7307ea)
If tests are re-run, use "xxx then yyy" result format (ex: "FAILURE
then SUCCESS") to show that some failing tests have been re-run.
Add also test_regrtest.test_rerun_fail() test.
(cherry picked from commit c45fc7673e)
Co-authored-by: Victor Stinner <vstinner@redhat.com>
* bpo-31009: Fix support.fd_count() on Windows
On Windows, test.support.fd_count() now calls
msvcrt.CrtSetReportMode() to not kill the process nor log any error
on stderr on os.dup(fd) if the file descriptor is invalid.
* Fix for release mode
When running the test suite using --use=all / -u all, exclude tzdata
since it makes test_datetime too slow (15-20 min on some buildbots)
which then times out on some buildbots.
-u tzdata must now be enabled explicitly, -u tzdata or -u all,tzdata,
to run all test_datetime tests.
Fix also regrtest command line parser to allow passing -u
extralargefile to run test_zipfile64.
Travis CI: remove -tzdata. Replace -u all,-tzdata,-cpu with -u all,-cpu since tzdata is now excluded from -u all.
* Change the regrtest --huntrleaks checker to decide if a test file
leaks or not. Require that each run leaks at least 1 reference.
* Warmup runs are now completely ignored: ignored in the checker test
and not used anymore to compute the sum.
* Add an unit test for a reference leak.
Example of reference differences previously considered a failure
(leak) and now considered as success (success, no leak):
[3, 0, 0]
[0, 1, 0]
[8, -8, 1]
* bpo-30764: regrtest: change exit code on failure
* Exit code 2 if failed tests ("bad")
* Exit code 3 if interrupted
* bpo-30764: regrtest: add --fail-env-changed option
If the option is set, mark a test as failed if it alters the
environment, for example if it creates a file without removing it.
* regrtest --list-cases now supports --match and --match-file options.
Example: ./python -m test --list-cases -m FileTests test_os
* --list-cases now also sets support.verbose to False to prevent
messages to stdout when loading test modules.
* Add support._match_test() private function.
* Add a new option taking a filename to get a list of test names to
filter tests.
* support.match_tests becomes a list.
* Modify run_unittest() to accept to match the whole test identifier,
not just a part of a test identifier.
For example, the following command only runs test_default_timeout()
of the BarrierTests class of test_threading:
$ ./python -m test -v test_threading -m test.test_threading.BarrierTests.test_default_timeout
Remove also some empty lines from test_regrtest.py to make flake8
tool happy.
Issue #29362: Catch a crash of a worker process as a normal failure and
continue to run next tests. It allows to get the usual test summary: single
line result (OK/FAIL), total duration, etc.
It's sometimes hard to check quickly if tests succeeded, failed or something
bad happened. I added a final "Result: xxx" line which summarizes all outputs
into a single line, written at the end (it should always be the last line of
the output).
test_forever() stores its state into the builtins module since the test module
is reloaded at each run.
Remove also warning to detect leaked tests of a previous run.
* Fix accumulate_result(): don't use time on interrupted and failed test
* Add unit test for interrupted test
* Add unit test on --slow with interrupted test, with and without
multiprocessing
* Fix "-m test --forever": replace _test_forever() with self._test_forever()
* Add unit test for --forever
* Add unit test for a failing test
* Fix also some pyflakes warnings in libregrtest
Slaves (child processes running tests for regrtest -jN) now inherit
--memlimit/-M, --threshold/-t and --nowindows/-n options.
* -M, -t and -n are now supported with -jN
* Factorize code to run tests.
* run_test_in_subprocess() now pass the whole "ns" namespace to the child
process.
* test all available ways to run the Python test suite
* test many regrtest options: --slow, --coverage, -r, -u, etc.
Note: python -m test --coverage doesn't work on Windows.
Victor Stinner
Miss Islington (bot)
Antoine Pitrou
Victor Stinner
Louie Lu
Xiang Zhang
Martin Panter
R David Murray
Serhiy Storchaka