Test case matching patterns specified by options --match, --ignore,
--matchfile and --ignorefile are now tested in the order of
specification, and the last match determines whether the test case be run
or ignored.
Increase support.LOOPBACK_TIMEOUT from 5 to 10 seconds. Also increase
the timeout depending on the --timeout option. For example, for a
test timeout of 40 minutes (ARM Raspbian 3.x), use LOOPBACK_TIMEOUT
of 20 seconds instead of 5 seconds before.
Remove replace_stdout(): call sys.stdout.reconfigure() instead of set
the error handler to backslashreplace.
display_header() logs an empty line and flush stdout.
Remove encoding workaround in display_header() since stdout error
handler is now set to backslashreplace earlier.
libregrtest now calls random.seed() before running each test file
when -r/--randomize command line option is used. Moreover, it's also
called in worker processes. It should help to make tests more
deterministic. Previously, it was only called once in the main
process before running all test files and it was not called in worker
processes.
* Convert some f-strings to regular strings in test_regrtest when
f-string is not needed.
* Remove unused all_methods variable from test_regrtest.
* Add RunTests members are now mandatory.
* Rename runtest_mp.py to run_workers.py
* Move exit_timeout() and temp_cwd() context managers from
Regrtest.main() to Regrtest.run_tests(). Actions like --list-tests
or --list-cases don't need these protections.
* Regrtest: remove selected and tests attributes. Pass 'selected' to
list_tests(), list_cases() and run_tests(). display_result() now
expects a TestTuple, instead of TestList.
* Rename setup_tests() to setup_process() and rename setup_support()
to setup_tests().
* Move _adjust_resource_limits() to utils and rename it to
adjust_rlimit_nofile().
* Move replace_stdout() to utils.
* Fix RunTests.verbose type: it's an int.
* Add single.py and result.py files.
* Rename runtest.py to runtests.py.
* Move run_single_test() function and its helper functions to
single.py.
* Move remove_testfn(), abs_module_name() and normalize_test_name()
to utils.py.
* Move setup_support() to setup.py.
* Move type hints like TestName to utils.py.
* Rename runtest.py to runtests.py.
Add new worker.py file:
* Move create_worker_process() and worker_process() to this file.
* Add main() function to worker.py. create_worker_process() now
runs the command: "python -m test.libregrtest.worker JSON".
* create_worker_process() now starts the worker process in the
current working directory. Regrtest now gets the absolute path of
the reflog.txt filename: -R command line option filename.
* Remove --worker-json command line option.
Remove test_regrtest.test_worker_json().
Related changes:
* Add write_json() and from_json() methods to TestResult.
* Rename select_temp_dir() to get_temp_dir() and move it to utils.
* Rename make_temp_dir() to get_work_dir() and move it to utils.
It no longer calls os.makedirs(): Regrtest.main() now calls it.
* Move fix_umask() to utils. The function is now called by
setup_tests().
* Move StrPath to utils.
* Add exit_timeout() context manager to utils.
* RunTests: Replace junit_filename (StrPath) with use_junit (bool).
* Add attributes to Regrtest and RunTests:
* gc_threshold
* memory_limit
* python_cmd
* use_resources
* Remove WorkerJob class. Add as_json() and from_json() methods to
RunTests. A worker process now only uses RunTests for all
parameters.
* Add tests on support.set_memlimit() in test_support. Create
_parse_memlimit() and also adds tests on it.
* Remove 'ns' parameter from runtest.py.
* Rename dash_R() runtest_refleak(). The function now gets
huntrleaks and quiet arguments, instead of 'ns' argument.
* Add attributes to Regrtest and RunTests:
* verbose
* quiet
* huntrleaks
* test_dir
* Add HuntRefleak class.
test.libregrtest now marks a test as ENV_CHANGED (altered the
execution environment) if a thread raises an exception but does not
catch it. It sets a hook on threading.excepthook. Use
--fail-env-changed option to mark the test as failed.
libregrtest regrtest_unraisable_hook() explicitly flushs
sys.stdout, sys.stderr and sys.__stderr__.
RegressionTestResult.USE_XML must now be set to True to get the JUnit
XML output.
Reduce the number of imports when --junit-xml=FILE option is not
used: 153 => 144 (-9).
test_repl.test_close_stdin() now calls
support.suppress_msvcrt_asserts() to fix the test on Windows.
* Move suppress_msvcrt_asserts() from test.libregrtest.setup to
test.support. Make its verbose parameter optional: verbose=False by
default.
* Add msvcrt.GetErrorMode().
* SuppressCrashReport now uses GetErrorMode() and SetErrorMode() of
the msvcrt module, rather than using ctypes.
* Remove also an unused variable (deadline) in wait_process().
Under some conditions the earlier fix for bpo-18075, "Infinite recursion
tests triggering a segfault on Mac OS X", now causes failures on macOS
when attempting to change stack limit with resource.setrlimit
resource.RLIMIT_STACK, like regrtest does when running the test suite.
The reverted change had specified a non-default stack size when linking
the python executable on macOS. As of macOS 10.14.4, the previous
code causes a hard failure when running tests, although similar
failures had been seen under some conditions under some earlier
systems. Reverting the change to the interpreter stack size at link
time helped for release builds but caused some tests to fail when
built --with-pydebug. Try the opposite approach: continue to build
the interpreter with an increased stack size on macOS and remove
the failing setrlimit call in regrtest initialization. This will
definitely avoid the resource.RLIMIT_STACK error and should have
no, or fewer, side effects.
regrtest now uses sys.unraisablehook() to mark a test as "environment
altered" (ENV_CHANGED) if it emits an "unraisable exception".
Moreover, regrtest logs a warning in this case.
Use "python3 -m test --fail-env-changed" to catch unraisable
exceptions in tests.
Fix reference leak hunting in regrtest: compute also deltas (of
reference count, allocated memory blocks, file descriptor count)
during warmup, to ensure that everything is initialized before
starting to hunt reference leaks.
Other changes:
* Replace gc.collect() with support.gc_collect()
* Move calls to read memory statistics from dash_R_cleanup() to
dash_R()
* Pass regrtest 'ns' to dash_R()
* dash_R() is now more quiet with --quiet option (don't display
progress).
* Precompute the full range for "for it in range(repcount):" to
ensure that the iteration doesn't allocate anything new.
* dash_R() now is responsible to call warm_caches().
When regrtest in run inside IDLE, sys.stdout and sys.stderr are not
TextIOWrapper objects and have no file descriptor associated:
sys.stderr.fileno() raises io.UnsupportedOperation.
Disable faulthandler and don't replace sys.stdout in that case.
Issue #26538: libregrtest: Fix setup_tests() to keep module.__path__ type
(_NamespacePath), don't convert to a list.
Add _NamespacePath.__setitem__() method to importlib._bootstrap_external.
Alex Waygood
Serhiy Storchaka
Victor Stinner
Nikita Sobolev
Gregory P. Smith
Łukasz Langa
Pablo Galindo Salgado
Ned Deily
Victor Stinner
Steve Dower
Steve Dower
Victor Stinner