TextTestRunner of unittest.runner now uses time.perf_counter() rather
than time.time() to measure the execution time of a test: time.time()
can go backwards, whereas time.perf_counter() is monotonic.
Similar change made in libregrtest, pprint and random.
Fix bug in `Lib/test/libregrtest/runtest.py` that makes running tests an extra time than the specified number of runs.
Add check for invalid --huntrleaks/-R parameters.
If tests are re-run, use "xxx then yyy" result format (ex: "FAILURE
then SUCCESS") to show that some failing tests have been re-run.
Add also test_regrtest.test_rerun_fail() test.
* "running:" progress: Format number of seconds as hours and minutes
* format_duration(): count also minutes as hours
* Create Lib/test/libregrtest/utils.py
* No longer clear filters, like --match, to re-run failed tests in
verbose mode (-w option).
* Tests result: always indicate if tests have been interrupted.
* Enhance tests summary
* Rename support._match_test() to support.match_test(): make it
public
* Remove support.match_tests global variable. It is replaced with a
new support.set_match_tests() function, so match_test() doesn't
have to check each time if patterns were modified.
* Rewrite match_test(): use different code paths depending on the
kind of patterns for best performances.
Co-Authored-By: Serhiy Storchaka <storchaka@gmail.com>
kB (*kilo* byte) unit means 1000 bytes, whereas KiB ("kibibyte")
means 1024 bytes. KB was misused: replace kB or KB with KiB when
appropriate.
Same change for MB and GB which become MiB and GiB.
Change the output of Tools/iobench/iobench.py.
Round also the size of the documentation from 5.5 MB to 5 MiB.
When regrtest in run inside IDLE, sys.stdout and sys.stderr are not
TextIOWrapper objects and have no file descriptor associated:
sys.stderr.fileno() raises io.UnsupportedOperation.
Disable faulthandler and don't replace sys.stdout in that case.
Use a pool of integer objects toprevent false alarm when checking for
memory block leaks. Fill the pool with values in -1000..1000 which
are the most common (reference, memory block, file descriptor)
differences.
Co-Authored-By: Antoine Pitrou <pitrou@free.fr>
* Add Lib/test/pythoninfo.py: script collecting various informations
about Python to help debugging test failures.
* regrtest: remove sys.hash_info and sys.flags from header.
* Travis CI, Appveyor: run pythoninfo before tests
* bpo-26732: fix too many fds in processes started with the "forkserver" method
A child process would inherit as many fds as the number of still-running children.
* Add blurb and test comment
When running the test suite using --use=all / -u all, exclude tzdata
since it makes test_datetime too slow (15-20 min on some buildbots)
which then times out on some buildbots.
-u tzdata must now be enabled explicitly, -u tzdata or -u all,tzdata,
to run all test_datetime tests.
Fix also regrtest command line parser to allow passing -u
extralargefile to run test_zipfile64.
Travis CI: remove -tzdata. Replace -u all,-tzdata,-cpu with -u all,-cpu since tzdata is now excluded from -u all.
If threading_cleanup() fails to cleanup threads, set a a new
support.environment_altered flag to true, flag uses by save_env which
is used by regrtest to check if a test altered the environment. At
the end, the test file fails with ENV_CHANGED instead of SUCCESS, to
report that it altered the environment.
* Change the regrtest --huntrleaks checker to decide if a test file
leaks or not. Require that each run leaks at least 1 reference.
* Warmup runs are now completely ignored: ignored in the checker test
and not used anymore to compute the sum.
* Add an unit test for a reference leak.
Example of reference differences previously considered a failure
(leak) and now considered as success (success, no leak):
[3, 0, 0]
[0, 1, 0]
[8, -8, 1]
* bpo-30764: regrtest: change exit code on failure
* Exit code 2 if failed tests ("bad")
* Exit code 3 if interrupted
* bpo-30764: regrtest: add --fail-env-changed option
If the option is set, mark a test as failed if it alters the
environment, for example if it creates a file without removing it.
* regrtest --list-cases now supports --match and --match-file options.
Example: ./python -m test --list-cases -m FileTests test_os
* --list-cases now also sets support.verbose to False to prevent
messages to stdout when loading test modules.
* Add support._match_test() private function.
Use a build/ directory in the build directory, not in the source
directory, since the source directory may be read-only and must not
be modified.
Fallback on the source directory if the build directory is not
available (missing "abs_builddir" sysconfig variable).
* Add a new option taking a filename to get a list of test names to
filter tests.
* support.match_tests becomes a list.
* Modify run_unittest() to accept to match the whole test identifier,
not just a part of a test identifier.
For example, the following command only runs test_default_timeout()
of the BarrierTests class of test_threading:
$ ./python -m test -v test_threading -m test.test_threading.BarrierTests.test_default_timeout
Remove also some empty lines from test_regrtest.py to make flake8
tool happy.
Buildbots don't run tests with -vv and so only log "xxx was modified
by test_xxx" which is not enough to debug such random issue. In many
cases, I'm unable to reproduce the warning and so unable to fix it.
Always logging the value before and value after should help to debug
such warning on buildbots.
Issue #29362: Catch a crash of a worker process as a normal failure and
continue to run next tests. It allows to get the usual test summary: single
line result (OK/FAIL), total duration, etc.
It's sometimes hard to check quickly if tests succeeded, failed or something
bad happened. I added a final "Result: xxx" line which summarizes all outputs
into a single line, written at the end (it should always be the last line of
the output).
* regrtest now uses subprocesses when the -j1 command line option
is used: each test file runs in a fresh child process. Before, the -j1 option
was ignored.
* Tools/buildbot/test.bat script now uses -j1 by default to run
each test file in fresh child process.
* Replace get/restore methods with a Resource class and Resource subclasses
* Create ModuleAttr, ModuleAttrList and ModuleAttrDict helper classes
* Use __subclasses__() to get resource classes instead of using an hardcoded
list (2 shutil resources were missinged in the list!)
* Don't define MultiprocessingProcessDangling resource if the multiprocessing
module is missing
* Nicer diff for dictionaries. Useful for the big os.environ dict
* Reorder code to group resources
* Rename libregrtest.main_in_temp_cwd() to libregrtest.main()
* Add regrtest.main_in_temp_cwd() alias to libregrtest.main()
* Move old main_in_temp_cwd() code into libregrtest.Regrtest.main()
* Update multiple scripts to call libregrtest.main()
Issue #26538: libregrtest: Fix setup_tests() to keep module.__path__ type
(_NamespacePath), don't convert to a list.
Add _NamespacePath.__setitem__() method to importlib._bootstrap_external.
* Fix accumulate_result(): don't use time on interrupted and failed test
* Add unit test for interrupted test
* Add unit test on --slow with interrupted test, with and without
multiprocessing
* Fix "-m test --forever": replace _test_forever() with self._test_forever()
* Add unit test for --forever
* Add unit test for a failing test
* Fix also some pyflakes warnings in libregrtest
* Remove runtest_ns(): pass directly ns to runtest().
* Create also Regrtest.rerun_failed_tests() method.
* Inline again Regrtest.run_test(): it's no more justified to have a method
Slaves (child processes running tests for regrtest -jN) now inherit
--memlimit/-M, --threshold/-t and --nowindows/-n options.
* -M, -t and -n are now supported with -jN
* Factorize code to run tests.
* run_test_in_subprocess() now pass the whole "ns" namespace to the child
process.
Running the Python test suite with -jN now:
- Display the duration of tests which took longer than 30 seconds
- Display the tests currently running since at least 30 seconds
- Display the tests we are waiting for when the test suite is interrupted
Clenaup also run_test_in_subprocess() code.
Python doesn't display the refcount anymore by default. It only displays it
when -X showrefcount command line option is used, which is not the case here.
regrtest can be run with -X showrefcount, the option is not inherited by child
processes.
Move the code to run tests in multiple processes using threading and subprocess
to a new submodule.
Move also slave_runner() (renamed to run_tests_slave()) and
run_test_in_subprocess() (renamed to run_tests_in_subprocess()) there.
with attributes and methods.
The --threshold command line option is now ignored if the gc module is missing.
* Convert main() variables to Regrtest attributes, document some attributes
* Convert accumulate_result() function to a method
* Create setup_python() function and setup_regrtest() method.
* Import gc at top level
* Move resource.setrlimit() and the code to make the module paths absolute into
the new setup_python() function. So this code is no more executed when the
module is imported, only when main() is executed. We have a better control on
when the setup is done.
* Move textwrap import from printlist() to the top level.
* Some other minor cleanup.
Start to split regrtest.py into smaller parts with the creation of
Lib/test/libregrtest/cmdline.py: code to handle the command line, especially
parsing command line arguments. This part of the code is tested by
test_regrtest.
sth
Victor Stinner
Pablo Galindo
Steve Dower
Brett Cannon
Ned Deily
Ivan Levkivskyi
Victor Stinner
Antoine Pitrou
mlouielu
Xiang Zhang
Serhiy Storchaka
Eric V. Smith
Martin Panter
Alexander Belopolsky
Brett Cannon