* Change the regrtest --huntrleaks checker to decide if a test file
leaks or not. Require that each run leaks at least 1 reference.
* Warmup runs are now completely ignored: ignored in the checker test
and not used anymore to compute the sum.
* Add an unit test for a reference leak.
Example of reference differences previously considered a failure
(leak) and now considered as success (success, no leak):
[3, 0, 0]
[0, 1, 0]
[8, -8, 1]
* bpo-30764: regrtest: change exit code on failure
* Exit code 2 if failed tests ("bad")
* Exit code 3 if interrupted
* bpo-30764: regrtest: add --fail-env-changed option
If the option is set, mark a test as failed if it alters the
environment, for example if it creates a file without removing it.
* regrtest --list-cases now supports --match and --match-file options.
Example: ./python -m test --list-cases -m FileTests test_os
* --list-cases now also sets support.verbose to False to prevent
messages to stdout when loading test modules.
* Add support._match_test() private function.
Use a build/ directory in the build directory, not in the source
directory, since the source directory may be read-only and must not
be modified.
Fallback on the source directory if the build directory is not
available (missing "abs_builddir" sysconfig variable).
* Add a new option taking a filename to get a list of test names to
filter tests.
* support.match_tests becomes a list.
* Modify run_unittest() to accept to match the whole test identifier,
not just a part of a test identifier.
For example, the following command only runs test_default_timeout()
of the BarrierTests class of test_threading:
$ ./python -m test -v test_threading -m test.test_threading.BarrierTests.test_default_timeout
Remove also some empty lines from test_regrtest.py to make flake8
tool happy.
Buildbots don't run tests with -vv and so only log "xxx was modified
by test_xxx" which is not enough to debug such random issue. In many
cases, I'm unable to reproduce the warning and so unable to fix it.
Always logging the value before and value after should help to debug
such warning on buildbots.
Issue #29362: Catch a crash of a worker process as a normal failure and
continue to run next tests. It allows to get the usual test summary: single
line result (OK/FAIL), total duration, etc.
It's sometimes hard to check quickly if tests succeeded, failed or something
bad happened. I added a final "Result: xxx" line which summarizes all outputs
into a single line, written at the end (it should always be the last line of
the output).
* regrtest now uses subprocesses when the -j1 command line option
is used: each test file runs in a fresh child process. Before, the -j1 option
was ignored.
* Tools/buildbot/test.bat script now uses -j1 by default to run
each test file in fresh child process.
* Replace get/restore methods with a Resource class and Resource subclasses
* Create ModuleAttr, ModuleAttrList and ModuleAttrDict helper classes
* Use __subclasses__() to get resource classes instead of using an hardcoded
list (2 shutil resources were missinged in the list!)
* Don't define MultiprocessingProcessDangling resource if the multiprocessing
module is missing
* Nicer diff for dictionaries. Useful for the big os.environ dict
* Reorder code to group resources
* Rename libregrtest.main_in_temp_cwd() to libregrtest.main()
* Add regrtest.main_in_temp_cwd() alias to libregrtest.main()
* Move old main_in_temp_cwd() code into libregrtest.Regrtest.main()
* Update multiple scripts to call libregrtest.main()
Issue #26538: libregrtest: Fix setup_tests() to keep module.__path__ type
(_NamespacePath), don't convert to a list.
Add _NamespacePath.__setitem__() method to importlib._bootstrap_external.
* Fix accumulate_result(): don't use time on interrupted and failed test
* Add unit test for interrupted test
* Add unit test on --slow with interrupted test, with and without
multiprocessing
* Fix "-m test --forever": replace _test_forever() with self._test_forever()
* Add unit test for --forever
* Add unit test for a failing test
* Fix also some pyflakes warnings in libregrtest
* Remove runtest_ns(): pass directly ns to runtest().
* Create also Regrtest.rerun_failed_tests() method.
* Inline again Regrtest.run_test(): it's no more justified to have a method
Slaves (child processes running tests for regrtest -jN) now inherit
--memlimit/-M, --threshold/-t and --nowindows/-n options.
* -M, -t and -n are now supported with -jN
* Factorize code to run tests.
* run_test_in_subprocess() now pass the whole "ns" namespace to the child
process.
Running the Python test suite with -jN now:
- Display the duration of tests which took longer than 30 seconds
- Display the tests currently running since at least 30 seconds
- Display the tests we are waiting for when the test suite is interrupted
Clenaup also run_test_in_subprocess() code.
Python doesn't display the refcount anymore by default. It only displays it
when -X showrefcount command line option is used, which is not the case here.
regrtest can be run with -X showrefcount, the option is not inherited by child
processes.
Move the code to run tests in multiple processes using threading and subprocess
to a new submodule.
Move also slave_runner() (renamed to run_tests_slave()) and
run_test_in_subprocess() (renamed to run_tests_in_subprocess()) there.
with attributes and methods.
The --threshold command line option is now ignored if the gc module is missing.
* Convert main() variables to Regrtest attributes, document some attributes
* Convert accumulate_result() function to a method
* Create setup_python() function and setup_regrtest() method.
* Import gc at top level
* Move resource.setrlimit() and the code to make the module paths absolute into
the new setup_python() function. So this code is no more executed when the
module is imported, only when main() is executed. We have a better control on
when the setup is done.
* Move textwrap import from printlist() to the top level.
* Some other minor cleanup.
Start to split regrtest.py into smaller parts with the creation of
Lib/test/libregrtest/cmdline.py: code to handle the command line, especially
parsing command line arguments. This part of the code is tested by
test_regrtest.
Victor Stinner
mlouielu
Ivan Levkivskyi
Xiang Zhang
Serhiy Storchaka
Eric V. Smith
Steve Dower
Martin Panter
Alexander Belopolsky
Brett Cannon