Test case matching patterns specified by options --match, --ignore,
--matchfile and --ignorefile are now tested in the order of
specification, and the last match determines whether the test case be run
or ignored.
Increase support.LOOPBACK_TIMEOUT from 5 to 10 seconds. Also increase
the timeout depending on the --timeout option. For example, for a
test timeout of 40 minutes (ARM Raspbian 3.x), use LOOPBACK_TIMEOUT
of 20 seconds instead of 5 seconds before.
When using worker processes (-jN) with --verbose3 option, regrtest
can now display the worker output even if a worker process does
crash. Previously, sys.stdout and sys.stderr were replaced and so
the worker output was lost on a crash.
* _add_python_opts() now handles cross compilation and HOSTRUNNER.
* display_header() now tells if Python is cross-compiled, display
HOSTRUNNER, and get the host platform.
* Remove Tools/scripts/run_tests.py script.
* Remove "make hostrunnertest": use "make buildbottest"
or "make test" instead.
Rename WORKER_ERROR to WORKER_BUG. Add WORKER_FAILED state: it does
not stop the manager, whereas WORKER_BUG does.
Change also TestResults.display_result() order: display failed tests
at the end, the important important information.
WorkerThread now tries to get the signal name for negative exit code.
In the subprocess code path, wait until the child process completes
with a timeout of EXIT_TIMEOUT seconds.
Fix create_worker_process() regression: use start_new_session=True if
USE_PROCESS_GROUP is true.
WorkerThread.wait_stopped() uses a timeout of 60 seconds, instead of
30 seconds.
Split test_gdb.py file into a test_gdb package made of multiple
tests, so tests can now be run in parallel.
* Create Lib/test/test_gdb/ directory.
* Split test_gdb.py into multiple files in Lib/test/test_gdb/
directory.
* Move Lib/test/gdb_sample.py to Lib/test/test_gdb/ directory.
Update get_sample_script(): use __file__ to locate gdb_sample.py.
* Move gdb_has_frame_select() and HAS_PYUP_PYDOWN to test_misc.py.
* Explicitly skip test_gdb on Windows. Previously, test_gdb was
skipped even if gdb was available because of
gdb_has_frame_select().
Fix the test when run on an installed Python: use "abs_srcdir" of
sysconfig, and skip the test if the Python source code cannot be
found.
* Tools/patchcheck/patchcheck.py, Tools/freeze/test/freeze.py and
Lib/test/libregrtest/utils.py now first try to get "abs_srcdir"
from sysconfig, before getting "srcdir" from sysconfig.
* test.pythoninfo logs sysconfig "abs_srcdir".
WASM and WASI buildbots use multiple PYTHON environment variables
such as PYTHONPATH and _PYTHON_HOSTRUNNER. Don't use -E if the
--python=COMMAND option is used.
Fix copy_source_tree() function of test_tools.test_freeze:
* Don't copy SRC_DIR/build/ anymore. This directory is modified by
other tests running in parallel.
* Add test.support.copy_python_src_ignore().
* Use sysconfig to get the source directory.
* Use sysconfig.get_config_var() to get CONFIG_ARGS variable.
* On Windows, use subprocess.run() instead of os.execv().
* Only add needed options
* Rename reexec parameter to _add_python_opts.
* Rename --no-reexec option to --dont-add-python-opts.
Remove replace_stdout(): call sys.stdout.reconfigure() instead of set
the error handler to backslashreplace.
display_header() logs an empty line and flush stdout.
Remove encoding workaround in display_header() since stdout error
handler is now set to backslashreplace earlier.
When --fast-ci or --slow-ci option is used, regrtest now replaces the
current process with a new process to add "-u -W default -bb -E"
options to Python.
Changes:
* PCbuild/rt.bat and Tools/scripts/run_tests.py no longer need to add
"-u -W default -bb -E" options to Python: it's now done by
regrtest.
* Fix Tools/scripts/run_tests.py: flush stdout before replacing the
process. Previously, buffered messages were lost.
* Add --fast-ci and --slow-ci options to libregrtest:
* --fast-ci uses a default timeout of 10 minutes and "-u all,-cpu"
(skip slowest tests).
* --slow-ci uses a default timeout of 20 minues and "-u all" (run
all tests).
* regrtest header now lists test resources.
* Makefile changes:
* "make test", "make hostrunnertest" and "make coverage-report" now
use --fast-ci option and TESTTIMEOUT variable.
* "make buildbottest" now uses "--slow-ci". Remove options which
became redundant with "--slow-ci".
* "make testall" and "make testuniversal" now use --slow-ci option
and TESTTIMEOUT variable.
* "make testall" now uses "find -exec rm ..." instead of
"find ... -print|xargs rm ...", same as "make clean".
* GitHub Actions workflow:
* Ubuntu and Address Sanitizer jobs now use "make test". Remove
options which became redundant with "--fast-ci".
* Windows jobs now use --fast-ci option.
* Use -j0 to detect the number of CPUs.
* Set Makefile TESTTIMEOUT default to an empty string, since
--slow-ci and --fast-ci use different default timeout. It's now
accepted to pass "--timeout=" to regrtest: treated as not timeout.
* Tools/scripts/run_tests.py now uses --fast-ci option.
* Tools/buildbot/test.bat now uses --slow-ci option. Remove
--timeout=1200 option, redundant with --slow-ci.
* Factorize code listing "bad / env changed / ..." tests.
* Add TestResults.is_all_good() method.
* Move "All 400 tests OK." to the end
* Move "Test suite interrupted by signal SIGINT." to the end.
Skip test_freeze_simple_script() of test_tools.test_freeze if Python
is built with "./configure --enable-optimizations", which means with
Profile Guided Optimization (PGO): it just makes the test too slow.
The freeze tool is tested by many other CIs with other (faster)
compiler flags.
test.pythoninfo now gets also get_build_info() of
test.libregrtests.utils.
libregrtest now decodes stdout of test worker processes with the
"backslashreplace" error handler to log corrupted stdout, instead of
failing with an error and not logging the stdout.
On WASI platform, get_temp_dir() should behave differently since the
parent process is a WASI process and uses a different get_temp_dir()
path.
Fix also WorkerThread._runtest(): don't read JSON file if the worker
process exit code is non-zero.
On Emscripten and WASI, or if --python command line is used,
libregrtest now writes JSON into stdout, instead of using a name
file.
* Add JsonFileType.STDOUT.
* Remove JsonFileType.FILENAME.
* test.pythoninfo logs environment variables related to
cross-compilation and running Python on Emscripten/WASI.
On Emscripten and WASI platforms, or if --python command line option
is used, libregrtest now uses a filename for the JSON file.
Emscripten and WASI buildbot workers run the main test process with a
different Python (Linux) which spawns Emscripten/WASI processes using
the command specified in --python command line option. Passing a file
descriptor from the parent process to the child process doesn't work
in this case.
* Add JsonFile and JsonFileType classes
* Add RunTests.json_file_use_filename() method.
* Add a test in test_regrtest on the --python command line option.
* test_regrtest: add parallel=False parameter.
* Split long RunWorkers._runtest() function into sub-functions.
Fix Emscripten and WASI: start the test worker process in the Python
source code directory, where 'python.js' and 'python.wasm' can be
found. Then worker_process() changes to a temporary directory created
to run tests.
* create_worker_process() uses os_helper.SAVEDCWD as cwd.
* worker_process() uses get_temp_dir() as the parent directory for
get_work_dir().
* Don't use plural but singual for "test" in "Run 1 test ..."
message.
* Remove unused imports.
* Add WORK_DIR_PREFIX and WORKER_WORK_DIR_PREFIX constants.
Don't spawn more threads than the number of jobs: these worker
threads would never get anything to do.
* Add the number of tests in "Run ... tests in ..." message.
* Add RunTests.get_jobs() method.
* Add plural() function.
* count() uses f-string.
libregrtest now calls random.seed() before running each test file
when -r/--randomize command line option is used. Moreover, it's also
called in worker processes. It should help to make tests more
deterministic. Previously, it was only called once in the main
process before running all test files and it was not called in worker
processes.
* Convert some f-strings to regular strings in test_regrtest when
f-string is not needed.
* Remove unused all_methods variable from test_regrtest.
* Add RunTests members are now mandatory.
libregrtest now uses a separated file descriptor to write test result
as JSON. Previously, if a test wrote debug messages late around the
JSON, the main test process failed to parse JSON.
Rename TestResult.write_json() to TestResult.write_json_into().
worker_process() no longer writes an empty line at the end. There is
no need to separate test process output from the JSON output anymore,
since JSON is now written into a separated file descriptor.
create_worker_process() now always spawn the process with
close_fds=True.
* Rename runtest_mp.py to run_workers.py
* Move exit_timeout() and temp_cwd() context managers from
Regrtest.main() to Regrtest.run_tests(). Actions like --list-tests
or --list-cases don't need these protections.
* Regrtest: remove selected and tests attributes. Pass 'selected' to
list_tests(), list_cases() and run_tests(). display_result() now
expects a TestTuple, instead of TestList.
* Rename setup_tests() to setup_process() and rename setup_support()
to setup_tests().
* Move _adjust_resource_limits() to utils and rename it to
adjust_rlimit_nofile().
* Move replace_stdout() to utils.
* Fix RunTests.verbose type: it's an int.
Serhiy Storchaka
Victor Stinner
Zachary Ware
Nikita Sobolev
Alex Waygood