* regrtest: Add --cleanup option to remove "test_python_*" directories
of previous failed test jobs.
* Add "make cleantest" to run "python3 -m test --cleanup".
regrtest now uses sys.unraisablehook() to mark a test as "environment
altered" (ENV_CHANGED) if it emits an "unraisable exception".
Moreover, regrtest logs a warning in this case.
Use "python3 -m test --fail-env-changed" to catch unraisable
exceptions in tests.
regrtest now always detects uncollectable objects. Previously, the
check was only enabled by --findleaks. The check now also works with
-jN/--multiprocess N.
--findleaks becomes a deprecated alias to --fail-env-changed.
Rewrite run_tests_multiprocess() function as a new MultiprocessRunner
class with multiple methods to better report errors and stop
immediately when needed.
Changes:
* Worker processes are now killed immediately if tests are
interrupted or if a test does crash (CHILD_ERROR): worker
processes are killed.
* Rewrite how errors in a worker thread are reported to
the main thread. No longer ignore BaseException or parsing errors
silently.
* Remove 'finished' variable: use worker.is_alive() instead
* Always compute omitted tests. Add Regrtest.get_executed() method.
* Add TestResult and MultiprocessResult types to ensure that results
always have the same fields.
* runtest() now handles KeyboardInterrupt
* accumulate_result() and format_test_result() now takes a TestResult
* cleanup_test_droppings() is now called by runtest() and mark the
test as ENV_CHANGED if the test leaks support.TESTFN file.
* runtest() now includes code "around" the test in the test timing
* Add print_warning() in test.libregrtest.utils to standardize how
libregrtest logs warnings to ease parsing the test output.
* support.unload() is now called with abstest rather than test_name
* Rename 'test' variable/parameter to 'test_name'
* dash_R(): remove unused the_module parameter
* Remove unused imports
As in title, expose C `raise` function as `raise_function` in `signal` module. Also drop existing `raise_signal` in `_testcapi` module and replace all usages with new function.
https://bugs.python.org/issue35568
bpo-34279, bpo-35412: support.run_unittest() no longer raises
TestDidNotRun if a test result contains skipped tests. The
exception is now only raised if no test have been run and no test
have been skipped.
If tests are re-run, use "xxx then yyy" result format (ex: "FAILURE
then SUCCESS") to show that some failing tests have been re-run.
Add also test_regrtest.test_rerun_fail() test.
* bpo-31009: Fix support.fd_count() on Windows
On Windows, test.support.fd_count() now calls
msvcrt.CrtSetReportMode() to not kill the process nor log any error
on stderr on os.dup(fd) if the file descriptor is invalid.
* Fix for release mode
When running the test suite using --use=all / -u all, exclude tzdata
since it makes test_datetime too slow (15-20 min on some buildbots)
which then times out on some buildbots.
-u tzdata must now be enabled explicitly, -u tzdata or -u all,tzdata,
to run all test_datetime tests.
Fix also regrtest command line parser to allow passing -u
extralargefile to run test_zipfile64.
Travis CI: remove -tzdata. Replace -u all,-tzdata,-cpu with -u all,-cpu since tzdata is now excluded from -u all.
* Change the regrtest --huntrleaks checker to decide if a test file
leaks or not. Require that each run leaks at least 1 reference.
* Warmup runs are now completely ignored: ignored in the checker test
and not used anymore to compute the sum.
* Add an unit test for a reference leak.
Example of reference differences previously considered a failure
(leak) and now considered as success (success, no leak):
[3, 0, 0]
[0, 1, 0]
[8, -8, 1]
* bpo-30764: regrtest: change exit code on failure
* Exit code 2 if failed tests ("bad")
* Exit code 3 if interrupted
* bpo-30764: regrtest: add --fail-env-changed option
If the option is set, mark a test as failed if it alters the
environment, for example if it creates a file without removing it.
* regrtest --list-cases now supports --match and --match-file options.
Example: ./python -m test --list-cases -m FileTests test_os
* --list-cases now also sets support.verbose to False to prevent
messages to stdout when loading test modules.
* Add support._match_test() private function.
* Add a new option taking a filename to get a list of test names to
filter tests.
* support.match_tests becomes a list.
* Modify run_unittest() to accept to match the whole test identifier,
not just a part of a test identifier.
For example, the following command only runs test_default_timeout()
of the BarrierTests class of test_threading:
$ ./python -m test -v test_threading -m test.test_threading.BarrierTests.test_default_timeout
Remove also some empty lines from test_regrtest.py to make flake8
tool happy.
Issue #29362: Catch a crash of a worker process as a normal failure and
continue to run next tests. It allows to get the usual test summary: single
line result (OK/FAIL), total duration, etc.
It's sometimes hard to check quickly if tests succeeded, failed or something
bad happened. I added a final "Result: xxx" line which summarizes all outputs
into a single line, written at the end (it should always be the last line of
the output).
test_forever() stores its state into the builtins module since the test module
is reloaded at each run.
Remove also warning to detect leaked tests of a previous run.
Victor Stinner
Gregory P. Smith
Paul Monson
Vladimir Matveev
Pablo Galindo
Srinivas Thatiparthy (శ్రీనివాస్ తాటిపర్తి)
Andrés Delfino
Antoine Pitrou
Victor Stinner
Louie Lu
Xiang Zhang
Martin Panter
R David Murray
Serhiy Storchaka