Commit Graph

326 Commits

Author SHA1 Message Date
Ronald Oussoren 5adff3f86f
gh-80931: Skip some socket tests while hunting for refleaks on macOS (#114057)
Some socket tests related to sending file descriptors cause a file descriptor leak on macOS, all of them tests that send one or more descriptors than cannot be received on the read end.  This appears to be a platform bug.

This PR skips those tests when doing a refleak test run to avoid hiding other problems.
2024-01-21 11:25:15 +01:00
Nikita Sobolev 9c93350f58
gh-108303: Move all doctest related files and tests to `Lib/test/test_doctest/` (#112109)
Co-authored-by: Brett Cannon <brett@python.org>
2024-01-18 16:58:11 +01:00
Samet YASLAN 88cb972000
gh-112536: Add support for thread sanitizer (TSAN) (gh-112648) 2023-12-30 17:17:02 +09:00
Victor Stinner f8852634ed
gh-108223: Refer to PEP 703 as Free Threading (#112780) 2023-12-06 12:55:58 +01:00
Serhiy Storchaka e08b70fab1
gh-108927: Fix removing testing modules from sys.modules (GH-108952)
It breaks import machinery if the test module has submodules used in
other tests.
2023-12-04 15:43:27 +00:00
Alex Waygood 70a38ffb3d
gh-109413: libregrtest: enable mypy's `--strict-optional` check on most files (#112586)
Co-authored-by: Victor Stinner <vstinner@python.org>
2023-12-01 14:54:33 +00:00
Victor Stinner f8ff80f635
gh-109413: regrtest: add WorkerRunTests class (#112588) 2023-12-01 13:46:50 +00:00
Alex Waygood 674c288b1c
gh-109413: Run mypy on `libregrtest` in CI (#112558)
Co-authored-by: Hugo van Kemenade <hugovk@users.noreply.github.com>
2023-11-30 23:00:14 +00:00
Hugo van Kemenade 3b3ec0d77f
gh-111863: Rename `Py_NOGIL` to `Py_GIL_DISABLED` (#111864)
Rename Py_NOGIL to Py_GIL_DISABLED
2023-11-20 15:52:00 +02:00
Łukasz Langa 3932b0f7b1
gh-110722: Make `-m test -T -j` use sys.monitoring (GH-111710)
Now all results from worker processes are aggregated and
displayed together as a summary at the end of a regrtest run.

The traditional trace is left in place for use with sequential
in-process test runs but now raises a warning that those
numbers are not precise.

`-T -j` requires `--with-pydebug` as it relies on `-Xpresite=`.
2023-11-10 18:17:45 +01:00
Victor Stinner 6f09f69b7f
gh-111881: Import doctest lazily in libregrtest (#111884)
In most cases, doctest is not needed. So don't always import it at
startup. The change reduces the number of modules already
imported when a test is run.
2023-11-09 15:00:10 +00:00
Victor Stinner d9a5530d23
gh-110367: Make regrtest --verbose3 compatible with --huntrleaks -jN (#111577)
"./python -m test -j1 -R 3:3 --verbose3" now works as expected, since
run_single_test() does not replace sys.stdout with StringIO in this
case.
2023-11-01 04:28:55 +01:00
Serhiy Storchaka a4981921aa
gh-111165: Add missed "support." prefix for "verbose" (GH-111327) 2023-10-25 14:50:25 +00:00
Serhiy Storchaka f6a45a03d0
gh-111165: Move test running code from test.support to libregrtest (GH-111166)
Remove no longer used functions run_unittest() and run_doctest() from
the test.support module.
2023-10-25 12:41:21 +03:00
Serhiy Storchaka 9a1fe09622
gh-110918: regrtest: allow to intermix --match and --ignore options (GH-110919)
Test case matching patterns specified by options --match, --ignore,
--matchfile and --ignorefile are now tested in the order of
specification, and the last match determines whether the test case be run
or ignored.
2023-10-21 17:44:46 +03:00
Victor Stinner 7237fb578d
gh-110932: Fix regrtest for SOURCE_DATE_EPOCH (#111143)
If the SOURCE_DATE_EPOCH environment variable is defined, use its
value as the random seed.
2023-10-21 10:37:48 +02:00
Zachary Ware b75186f69e
regrtest: Prepend 'use' options in --{fast,slow}-ci (GH-110363)
This allows individual resources to be disabled without having to explicitly re-enable all others.
2023-10-15 20:34:28 +02:00
Victor Stinner ce298a1c15
regrtest: --fast-ci/slow-ci no longer imply --rerun-fail (#110849) 2023-10-14 00:25:38 +00:00
Nikita Sobolev 732532b0af
gh-108303: Move all inspect test files to `test_inspect/` (#109607) 2023-10-10 22:15:11 +02:00
Victor Stinner e0c4437793
Add support.MS_WINDOWS constant (#110446) 2023-10-06 02:37:28 +02:00
Victor Stinner 0db2f1475e
gh-110167: Increase support.LOOPBACK_TIMEOUT to 10 seconds (#110413)
Increase support.LOOPBACK_TIMEOUT from 5 to 10 seconds. Also increase
the timeout depending on the --timeout option. For example, for a
test timeout of 40 minutes (ARM Raspbian 3.x), use LOOPBACK_TIMEOUT
of 20 seconds instead of 5 seconds before.
2023-10-05 20:54:27 +02:00
Victor Stinner 6592976061
gh-110367: Enhance regrtest -jN --verbose3 (#110368)
When using worker processes (-jN) with --verbose3 option, regrtest
can now display the worker output even if a worker process does
crash.  Previously, sys.stdout and sys.stderr were replaced and so
the worker output was lost on a crash.
2023-10-05 08:24:43 +02:00
Victor Stinner efd8c7a7c9
gh-109276: regrtest: shorter list of resources (#110326) 2023-10-04 09:39:50 +00:00
Nikita Sobolev 1465386720
gh-110171: `libregrtest` always sets `random.seed` (#110172) 2023-10-04 06:42:12 +00:00
Alex Waygood dddc757303
gh-110276: Run `test_str`, not `test_unicode`, as part of the PGO build (#110277)
`test_unicode` does not exist
2023-10-03 13:54:21 +01:00
Victor Stinner 65c285062c
gh-110164: regrtest disables random if SOURCE_DATE_EPOCH (#110168)
If the SOURCE_DATE_EPOCH environment variable is defined, regrtest
now disables randomization of tests.
2023-10-01 22:41:03 +02:00
Victor Stinner a46e960768
gh-109649: Use os.process_cpu_count() (#110165)
Replace os.cpu_count() with os.process_cpu_count() in modules:

* compileall
* concurrent.futures
* multiprocessing

Replace os.cpu_count() with os.process_cpu_count() in programs:

* _decimal deccheck.py test
* freeze.py
* multissltests.py
* python -m test (regrtest)
* wasm_build.py

Other changes:

* test.pythoninfo logs os.process_cpu_count().
* regrtest gets os.process_cpu_count() / os.cpu_count() in headers.
2023-10-01 03:14:57 +02:00
Victor Stinner 53eb9a676f
gh-110152: regrtest handles cross compilation and HOSTRUNNER (#110156)
* _add_python_opts() now handles cross compilation and HOSTRUNNER.
* display_header() now tells if Python is cross-compiled, display
  HOSTRUNNER, and get the host platform.
* Remove Tools/scripts/run_tests.py script.
* Remove "make hostrunnertest": use "make buildbottest"
  or "make test" instead.
2023-09-30 22:37:23 +00:00
Victor Stinner 2c234196ea
gh-109276: regrtest: add WORKER_FAILED state (#110148)
Rename WORKER_ERROR to WORKER_BUG. Add WORKER_FAILED state: it does
not stop the manager, whereas WORKER_BUG does.

Change also TestResults.display_result() order: display failed tests
at the end, the important important information.

WorkerThread now tries to get the signal name for negative exit code.
2023-09-30 22:48:26 +02:00
Victor Stinner 5ae6c6d053
gh-109566: regrtest --fast-ci no longer enables --nowindows (#110121)
The --nowindows option is deprecated and does nothing but logs a
warning.
2023-09-29 19:49:39 +00:00
Victor Stinner 235aacdeed
gh-109566: regrtest _add_python_opts() handles KeyboardInterrupt (#110062)
In the subprocess code path, wait until the child process completes
with a timeout of EXIT_TIMEOUT seconds.

Fix create_worker_process() regression: use start_new_session=True if
USE_PROCESS_GROUP is true.

WorkerThread.wait_stopped() uses a timeout of 60 seconds, instead of
30 seconds.
2023-09-29 00:51:22 +00:00
Victor Stinner 8f324b7ecd
gh-109972: Split test_gdb.py into test_gdb package (#109977)
Split test_gdb.py file into a test_gdb package made of multiple
tests, so tests can now be run in parallel.

* Create Lib/test/test_gdb/ directory.
* Split test_gdb.py into multiple files in Lib/test/test_gdb/
  directory.
* Move Lib/test/gdb_sample.py to Lib/test/test_gdb/ directory.
  Update get_sample_script(): use __file__ to locate gdb_sample.py.
* Move gdb_has_frame_select() and HAS_PYUP_PYDOWN to test_misc.py.
* Explicitly skip test_gdb on Windows. Previously, test_gdb was
  skipped even if gdb was available because of
  gdb_has_frame_select().
2023-09-28 11:24:15 +00:00
Victor Stinner 62881a79a8
gh-109566: regrtest doesn't enable --rerun if --python is used (#109969)
regrtest: --fast-ci and --slow-ci options no longer enable --rerun if
the --python option is used.
2023-09-27 16:09:23 +02:00
Victor Stinner b89ed9df39
gh-109615: Fix support test_copy_python_src_ignore() (#109958)
Fix the test when run on an installed Python: use "abs_srcdir" of
sysconfig, and skip the test if the Python source code cannot be
found.

* Tools/patchcheck/patchcheck.py, Tools/freeze/test/freeze.py and
  Lib/test/libregrtest/utils.py now first try to get "abs_srcdir"
  from sysconfig, before getting "srcdir" from sysconfig.
* test.pythoninfo logs sysconfig "abs_srcdir".
2023-09-27 12:32:12 +02:00
Victor Stinner 91fb8daa24
gh-109566: Fix regrtest Python options for WASM/WASI (#109954)
WASM and WASI buildbots use multiple PYTHON environment variables
such as PYTHONPATH and _PYTHON_HOSTRUNNER. Don't use -E if the
--python=COMMAND option is used.
2023-09-27 12:01:16 +02:00
Victor Stinner 1512d6c6ee
gh-109615: Fix test_tools.test_freeze SRCDIR (#109935)
Fix copy_source_tree() function of test_tools.test_freeze:

* Don't copy SRC_DIR/build/ anymore. This directory is modified by
  other tests running in parallel.
* Add test.support.copy_python_src_ignore().
* Use sysconfig to get the source directory.
* Use sysconfig.get_config_var() to get CONFIG_ARGS variable.
2023-09-27 08:18:39 +00:00
Victor Stinner 9abba715e3
gh-109566: Fix regrtest code adding Python options (#109926)
* On Windows, use subprocess.run() instead of os.execv().
* Only add needed options
* Rename reexec parameter to _add_python_opts.
* Rename --no-reexec option to --dont-add-python-opts.
2023-09-26 21:59:11 +00:00
Victor Stinner b1e4f6e83e
gh-109276, gh-109508: Fix libregrtest stdout (#109903)
Remove replace_stdout(): call sys.stdout.reconfigure() instead of set
the error handler to backslashreplace.

display_header() logs an empty line and flush stdout.

Remove encoding workaround in display_header() since stdout error
handler is now set to backslashreplace earlier.
2023-09-26 21:34:50 +02:00
Victor Stinner fbfec5642e
gh-109566: regrtest reexecutes the process (#109909)
When --fast-ci or --slow-ci option is used, regrtest now replaces the
current process with a new process to add "-u -W default -bb -E"
options to Python.

Changes:

* PCbuild/rt.bat and Tools/scripts/run_tests.py no longer need to add
  "-u -W default -bb -E" options to Python: it's now done by
  regrtest.
* Fix Tools/scripts/run_tests.py: flush stdout before replacing the
  process. Previously, buffered messages were lost.
2023-09-26 20:46:52 +02:00
Victor Stinner 859618c8cd
gh-109566, regrtest: Add --fast-ci and --slow-ci options (#109570)
* Add --fast-ci and --slow-ci options to libregrtest:

  * --fast-ci uses a default timeout of 10 minutes and "-u all,-cpu"
    (skip slowest tests).
  * --slow-ci uses a default timeout of 20 minues and "-u all" (run
    all tests).

* regrtest header now lists test resources.
* Makefile changes:

  * "make test", "make hostrunnertest" and "make coverage-report" now
    use --fast-ci option and TESTTIMEOUT variable.
  * "make buildbottest" now uses "--slow-ci". Remove options which
    became redundant with "--slow-ci".
  * "make testall" and "make testuniversal" now use --slow-ci option
    and TESTTIMEOUT variable.
  * "make testall" now uses "find -exec rm ..." instead of
    "find ... -print|xargs rm ...", same as "make clean".

* GitHub Actions workflow:

  * Ubuntu and Address Sanitizer jobs now use "make test". Remove
    options which became redundant with "--fast-ci".
  * Windows jobs now use --fast-ci option.
  * Use -j0 to detect the number of CPUs.

* Set Makefile TESTTIMEOUT default to an empty string, since
  --slow-ci and --fast-ci use different default timeout. It's now
  accepted to pass "--timeout=" to regrtest: treated as not timeout.
* Tools/scripts/run_tests.py now uses --fast-ci option.
* Tools/buildbot/test.bat now uses --slow-ci option. Remove
  --timeout=1200 option, redundant with --slow-ci.
2023-09-26 17:22:50 +02:00
Victor Stinner 4091deba88
gh-109739: regrtest disables load tracker if refleak (#109871)
regrtest: Fix reference leak check on Windows. Disable the load
tracker on Windows in the reference leak check mode (-R option).
2023-09-26 01:05:07 +00:00
Victor Stinner 72fb39c965
gh-109276: regrtest re-runs "env changed" tests (#109831)
When a test fails with "env changed" and --rerun option is used, the
test is now re-run in verbose mode in a fresh process.
2023-09-25 16:21:01 +02:00
Victor Stinner faebed4a67
gh-109276: Enhance libregrtest results (#109828)
* Factorize code listing "bad / env changed / ..." tests.
* Add TestResults.is_all_good() method.
* Move "All 400 tests OK." to the end
* Move "Test suite interrupted by signal SIGINT." to the end.
2023-09-25 13:50:15 +00:00
Victor Stinner 81cd1bd713
gh-103053: Skip test_freeze_simple_script() on PGO build (#109591)
Skip test_freeze_simple_script() of test_tools.test_freeze if Python
is built with "./configure --enable-optimizations", which means with
Profile Guided Optimization (PGO): it just makes the test too slow.
The freeze tool is tested by many other CIs with other (faster)
compiler flags.

test.pythoninfo now gets also get_build_info() of
test.libregrtests.utils.
2023-09-20 01:58:34 +00:00
Victor Stinner ef659b9616
gh-109508: Fix libregrtest formatting of getcwd() (#109537) 2023-09-18 16:45:48 +02:00
Alex Waygood 54fbfa8d5e
gh-109413: Improve mypy config for libregrtest (#109518)
Improve the mypy config file for libregrtest
2023-09-17 17:35:51 -07:00
Alex Waygood 92ed7e4df1
gh-109413: Fix some trivial mypy nitpicks in libregrtest (#109454) 2023-09-15 17:01:28 +00:00
Nikita Sobolev 82505dc351
gh-108303: Move `test_future` into its own test_future_stmt subdir (#109368) 2023-09-15 09:52:24 +02:00
Victor Stinner 74c72a2fc7
gh-109425: regrtest decodes worker stdout with backslashreplace (#109428)
libregrtest now decodes stdout of test worker processes with the
"backslashreplace" error handler to log corrupted stdout, instead of
failing with an error and not logging the stdout.
2023-09-14 23:37:37 +00:00
Alex Waygood 3b9d10b031
gh-109413: libregrtest: Add and improve type annotations (#109405) 2023-09-14 18:33:18 +00:00