* Change the regrtest --huntrleaks checker to decide if a test file
leaks or not. Require that each run leaks at least 1 reference.
* Warmup runs are now completely ignored: ignored in the checker test
and not used anymore to compute the sum.
* Add an unit test for a reference leak.
Example of reference differences previously considered a failure
(leak) and now considered as success (success, no leak):
[3, 0, 0]
[0, 1, 0]
[8, -8, 1]
* bpo-30764: regrtest: change exit code on failure
* Exit code 2 if failed tests ("bad")
* Exit code 3 if interrupted
* bpo-30764: regrtest: add --fail-env-changed option
If the option is set, mark a test as failed if it alters the
environment, for example if it creates a file without removing it.
* regrtest --list-cases now supports --match and --match-file options.
Example: ./python -m test --list-cases -m FileTests test_os
* --list-cases now also sets support.verbose to False to prevent
messages to stdout when loading test modules.
* Add support._match_test() private function.
Use a build/ directory in the build directory, not in the source
directory, since the source directory may be read-only and must not
be modified.
Fallback on the source directory if the build directory is not
available (missing "abs_builddir" sysconfig variable).
* Add a new option taking a filename to get a list of test names to
filter tests.
* support.match_tests becomes a list.
* Modify run_unittest() to accept to match the whole test identifier,
not just a part of a test identifier.
For example, the following command only runs test_default_timeout()
of the BarrierTests class of test_threading:
$ ./python -m test -v test_threading -m test.test_threading.BarrierTests.test_default_timeout
Remove also some empty lines from test_regrtest.py to make flake8
tool happy.
Buildbots don't run tests with -vv and so only log "xxx was modified
by test_xxx" which is not enough to debug such random issue. In many
cases, I'm unable to reproduce the warning and so unable to fix it.
Always logging the value before and value after should help to debug
such warning on buildbots.
Issue #29362: Catch a crash of a worker process as a normal failure and
continue to run next tests. It allows to get the usual test summary: single
line result (OK/FAIL), total duration, etc.
It's sometimes hard to check quickly if tests succeeded, failed or something
bad happened. I added a final "Result: xxx" line which summarizes all outputs
into a single line, written at the end (it should always be the last line of
the output).
* regrtest now uses subprocesses when the -j1 command line option
is used: each test file runs in a fresh child process. Before, the -j1 option
was ignored.
* Tools/buildbot/test.bat script now uses -j1 by default to run
each test file in fresh child process.
* Replace get/restore methods with a Resource class and Resource subclasses
* Create ModuleAttr, ModuleAttrList and ModuleAttrDict helper classes
* Use __subclasses__() to get resource classes instead of using an hardcoded
list (2 shutil resources were missinged in the list!)
* Don't define MultiprocessingProcessDangling resource if the multiprocessing
module is missing
* Nicer diff for dictionaries. Useful for the big os.environ dict
* Reorder code to group resources
Victor Stinner
mlouielu
Ivan Levkivskyi
Xiang Zhang
Serhiy Storchaka
Eric V. Smith
Steve Dower
Martin Panter
Alexander Belopolsky