Commit Graph

101 Commits

Author SHA1 Message Date
Nikita Sobolev 995386071f
bpo-46523: fix tests rerun when `setUp[Class|Module]` fails (#30895)
Co-authored-by: Jelle Zijlstra <jelle.zijlstra@gmail.com>
Co-authored-by: Łukasz Langa <lukasz@langa.pl>
2023-04-07 12:43:41 +02:00
Victor Stinner c76db37c0d
gh-98903: Test suite fails with exit code 4 if no tests ran (#98904)
The Python test suite now fails wit exit code 4 if no tests ran. It
should help detecting typos in test names and test methods.

* Add "EXITCODE_" constants to Lib/test/libregrtest/main.py.
* Fix a typo: "NO TEST RUN" becomes "NO TESTS RAN"
2022-11-02 15:37:40 +01:00
Victor Stinner 4f85cec9e2
gh-93353: regrtest supports checking tmp files with -j2 (#93909)
regrtest now also implements checking for leaked temporary files and
directories when using -jN for N >= 2. Use tempfile.mkdtemp() to
create the temporary directory. Skip this check on WASI.
2022-06-16 21:48:26 +02:00
Victor Stinner 36934a16e8
gh-93353: Fix regrtest for -jN with N >= 2 (GH-93813) 2022-06-14 18:04:53 +02:00
Victor Stinner e566ce5496
gh-93353: regrtest checks for leaked temporary files (#93776)
When running tests with -jN, create a temporary directory per process
and mark a test as "environment changed" if a test leaks a temporary
file or directory.
2022-06-14 13:43:02 +02:00
Victor Stinner 3ceb4b8d3a
gh-84623: Remove unused imports in tests (#93772) 2022-06-13 16:56:03 +02:00
Victor Stinner 5185956527
gh-69443: Add test.support.Py_DEBUG constant (#93226) 2022-05-26 00:12:54 +02:00
Serhiy Storchaka 54d068adfb
gh-91904: Fix setting envvar PYTHONREGRTEST_UNICODE_GUARD (GH-91905)
It always failed on non-UTF-8 locale and prevented running regrtests.
2022-04-25 17:35:14 +03:00
Christian Heimes 8464fbc42e
bpo-40280: Skip subprocess-based tests on wasm32-emscripten (GH-30615) 2022-01-25 08:09:06 +01:00
Victor Stinner 9d32714384
bpo-45745: Remove regrtest --findleaks options (GH-29514)
Remove the --findleaks command line option of regrtest: use the
--fail-env-changed option instead. Since Python 3.7, it was a
deprecated alias to the --fail-env-changed option.
2021-11-12 16:19:09 +01:00
Victor Stinner 676201a59f
bpo-45410: regrtest replaces print_warning.orig_stderr (GH-28926)
When running Python tests with -W, runtest() now replaces
support.print_warning.orig_stderr to preserve the messages order.

Add an unit test.
2021-10-13 17:35:21 +02:00
Łukasz Langa f1afef5e0d
bpo-44708: Only re-run test methods that match names of previously failing test methods (GH-27287)
* Move to a static argparse.Namespace subclass
* Roughly annotate runtest.py
* Refactor libregrtest to use lossless test result objects
* Only re-run test methods that match names of previously failing test methods
* Adopt tests to cover test method name matching

Co-authored-by: Pablo Galindo Salgado <Pablogsal@gmail.com>
2021-07-22 20:25:58 +02:00
Łukasz Langa 7915c96ffd
bpo-44647: Add a permanent Unicode-valued env var to regrtest (#27187) 2021-07-16 15:24:02 +02:00
Victor Stinner b136b1aac4
bpo-43843: libregrtest uses threading.excepthook (GH-25400)
test.libregrtest now marks a test as ENV_CHANGED (altered the
execution environment) if a thread raises an exception but does not
catch it. It sets a hook on threading.excepthook. Use
--fail-env-changed option to mark the test as failed.

libregrtest regrtest_unraisable_hook() explicitly flushs
sys.stdout, sys.stderr and sys.__stderr__.
2021-04-16 14:33:10 +02:00
Victor Stinner 701b63894f
bpo-38912: regrtest logs unraisable exception into sys.__stderr__ (GH-21718)
regrtest_unraisable_hook() temporarily replaces sys.stderr with
sys.__stderr__ to help to display errors when a test captures stderr.
2020-08-03 22:51:23 +02:00
Hai Shi 3ddc634cd5
bpo-40275: Use new test.support helper submodules in tests (GH-21219) 2020-06-30 15:46:06 +02:00
Serhiy Storchaka 9355868458
bpo-41043: Escape literal part of the path for glob(). (GH-20994) 2020-06-20 11:10:31 +03:00
Victor Stinner 57572b103e
bpo-40443: Remove unused imports in tests (GH-19805) 2020-04-30 01:48:37 +02:00
Pablo Galindo e0cd8aa70a
bpo-37957: Allow regrtest to receive a file with test (and subtests) to ignore (GH-16989)
When building Python in some uncommon platforms there are some known tests that will fail. Right now, the test suite has the ability to ignore entire tests using the -x option and to receive a filter file using the --matchfile filter. The problem with the --matchfile option is that it receives a file with patterns to accept and when you want to ignore a couple of tests and subtests, is too cumbersome to lists ALL tests that are not the ones that you want to accept and he problem with -x is that is not easy to ignore just a subtests that fail and the whole test needs to be ignored.

For these reasons, add a new option to allow to ignore a list of test and subtests for these situations.
2019-11-19 23:46:49 +00:00
Victor Stinner 0ec618af98
bpo-37531: regrtest ignores output on timeout (GH-16659)
bpo-37531, bpo-38207: On timeout, regrtest no longer attempts to call
`popen.communicate() again: it can hang until all child processes
using stdout and stderr pipes completes. Kill the worker process and
ignores its output.

Reenable test_regrtest.test_multiprocessing_timeout().

bpo-37531: Change also the faulthandler timeout of the main process
from 1 minute to 5 minutes, for Python slowest buildbots.
2019-10-08 18:45:43 +02:00
Victor Stinner 098e25672f
bpo-36670: Enhance regrtest (GH-16556)
* Add log() method: add timestamp and load average prefixes
  to main messages.
* WindowsLoadTracker:

  * LOAD_FACTOR_1 is now computed using SAMPLING_INTERVAL
  * Initialize the load to the arithmetic mean of the first 5 values
    of the Processor Queue Length value (so over 5 seconds), rather
    than 0.0.
  * Handle BrokenPipeError and when typeperf exit.

* format_duration(1.5) now returns '1.5 sec', rather than
  '1 sec 500 ms'
2019-10-03 16:15:16 +02:00
Victor Stinner b2dd2dd6e9
bpo-37531: Skip test_regrtest.test_multiprocessing_timeout() on all platforms (GH-16282) 2019-09-19 16:38:13 +02:00
Victor Stinner 0a963fbc9c
bpo-38203: faulthandler.dump_traceback_later() is always available (GH-16249)
dump_traceback_later() and cancel_dump_traceback_later() functions of
the faulthandler module are always available since Python 3.7.
2019-09-18 14:15:10 +02:00
Victor Stinner b9877cd2cc
bpo-37531: Skip test_regrtest.test_multiprocessing_timeout() on Windows (GH-16247)
It is a known and tracked bug: disable the test until it's fixed.
2019-09-18 01:58:29 +02:00
Victor Stinner b0c8369c60
bpo-37531: Fix regrtest timeout for subprocesses (GH-15072)
Co-Authored-By: Joannah Nanjekye <joannah.nanjekye@ibm.com>
2019-08-14 14:18:51 +02:00
Gregory P. Smith e95ac20103
bpo-37667: Add regression test for regrtest. (GH-14929)
Verify that it appears to find roughly the right number of tests in the stdlib's testsuite.
2019-07-23 22:28:48 -07:00
Victor Stinner 47fbc4e45b
bpo-37359: Add --cleanup option to python3 -m test (GH-14332)
* regrtest: Add --cleanup option to remove "test_python_*" directories
  of previous failed test jobs.
* Add "make cleantest" to run "python3 -m test --cleanup".
2019-06-24 12:03:00 +02:00
Paul Monson f355069a33 bpo-36511: Add buildbot scripts and fix tests for Windows ARM32 buildbot (GH-13454) 2019-06-19 13:09:54 -07:00
Victor Stinner 95f61c8b16
bpo-37069: regrtest uses sys.unraisablehook (GH-13759)
regrtest now uses sys.unraisablehook() to mark a test as "environment
altered" (ENV_CHANGED) if it emits an "unraisable exception".
Moreover, regrtest logs a warning in this case.

Use "python3 -m test --fail-env-changed" to catch unraisable
exceptions in tests.
2019-06-13 01:09:04 +02:00
Paul Monson e7e5039d69 bpo-37181: Fix test_regrtest failures on Windows arm64 (GH-13872) 2019-06-07 10:58:41 -07:00
Victor Stinner 837acc1957
bpo-36719: Fix regrtest re-run (GH-12964)
Properly handle a test which fail but then pass.

Add test_rerun_success() unit test.
2019-04-26 09:56:37 +02:00
Victor Stinner 75120d2205
bpo-36719: regrtest always detect uncollectable objects (GH-12951)
regrtest now always detects uncollectable objects. Previously, the
check was only enabled by --findleaks. The check now also works with
-jN/--multiprocess N.

--findleaks becomes a deprecated alias to --fail-env-changed.
2019-04-26 09:28:53 +02:00
Victor Stinner 3cde440f20
bpo-36725: Refactor regrtest multiprocessing code (GH-12961)
Rewrite run_tests_multiprocess() function as a new MultiprocessRunner
class with multiple methods to better report errors and stop
immediately when needed.

Changes:

* Worker processes are now killed immediately if tests are
  interrupted or if a test does crash (CHILD_ERROR): worker
  processes are killed.
* Rewrite how errors in a worker thread are reported to
  the main thread. No longer ignore BaseException or parsing errors
  silently.
* Remove 'finished' variable: use worker.is_alive() instead
* Always compute omitted tests. Add Regrtest.get_executed() method.
2019-04-26 08:40:25 +02:00
Victor Stinner 4d29983185
bpo-36725: regrtest: add TestResult type (GH-12960)
* Add TestResult and MultiprocessResult types to ensure that results
  always have the same fields.
* runtest() now handles KeyboardInterrupt
* accumulate_result() and format_test_result() now takes a TestResult
* cleanup_test_droppings() is now called by runtest() and mark the
  test as ENV_CHANGED if the test leaks support.TESTFN file.
* runtest() now includes code "around" the test in the test timing
* Add print_warning() in test.libregrtest.utils to standardize how
  libregrtest logs warnings to ease parsing the test output.
* support.unload() is now called with abstest rather than test_name
* Rename 'test' variable/parameter to 'test_name'
* dash_R(): remove unused the_module parameter
* Remove unused imports
2019-04-26 04:08:53 +02:00
Victor Stinner 235e7b2b0d
bpo-36465: Fix test_regrtest on Windows (GH-12945)
Fix Py_DEBUG constant: check for sys.gettotalrefcount attribute
rather than sys.getobjects.

Update also SpecialBuilds.txt documentation.
2019-04-25 01:46:31 +02:00
Vladimir Matveev c24c6c2c93 bpo-35568: add 'raise_signal' function (GH-11335)
As in title, expose C `raise` function as `raise_function` in `signal` module. Also drop existing `raise_signal` in `_testcapi` module and replace all usages with new function.


https://bugs.python.org/issue35568
2019-01-08 01:58:25 -08:00
Victor Stinner 3a8f4fef4a
bpo-34279: regrtest consider that skipped tests are ran (GH-11132)
bpo-34279, bpo-35412: support.run_unittest() no longer raises
TestDidNotRun if a test result contains skipped tests. The
exception is now only raised if no test have been run and no test
have been skipped.
2018-12-14 13:06:50 +01:00
Pablo Galindo 9724348b43 bpo-34279, regrtest: Issue a warning if no tests have been executed (GH-10150) 2018-11-29 18:17:44 +01:00
Srinivas Thatiparthy (శ్రీనివాస్ తాటిపర్తి) 70f61b933f bpo-35202: Remove unused imports in Lib directory (GH-10450) 2018-11-15 21:59:19 +09:00
Victor Stinner 012f5b968a
bpo-34605, libregrtest: Rename --slaveargs to --worker-args (GH-9099)
Rename also run_tests_slave() to run_tests_worker().
2018-09-07 17:20:42 +02:00
Victor Stinner 4ffe9c2b25
bpo-33718: regrtest: use format_duration() to display failed tests (GH-7686)
* Enhance also format_duration(): work on integers and rounds towards
  +infinity (math.ceil).
* Write unit tests on format_duration()
2018-06-14 14:58:13 +02:00
Victor Stinner c45fc7673e
bpo-33718: regrtest: use "xxx then yyy" result if re-run (GH-7521)
If tests are re-run, use "xxx then yyy" result format (ex: "FAILURE
then SUCCESS") to show that some failing tests have been re-run.

Add also test_regrtest.test_rerun_fail() test.
2018-06-08 09:53:51 +02:00
Andrés Delfino c3b7a6dfb9 bpo-33352: Skip test_regrtest test if rt.bat does not exist (GH-6654) 2018-04-30 18:30:53 +02:00
Antoine Pitrou a6a4dc816d bpo-31370: Remove support for threads-less builds (#3385)
* Remove Setup.config
* Always define WITH_THREAD for compatibility.
2017-09-07 18:56:24 +02:00
Victor Stinner 302bbbe9ba bpo-31009: Fix support.fd_count() on Windows (#2862)
* bpo-31009: Fix support.fd_count() on Windows

On Windows, test.support.fd_count() now calls
msvcrt.CrtSetReportMode() to not kill the process nor log any error
on stderr on os.dup(fd) if the file descriptor is invalid.

* Fix for release mode
2017-07-25 17:45:53 +02:00
Victor Stinner 5b392bbaeb bpo-30822: Exclude tzdata from regrtest --all (#2775)
When running the test suite using --use=all / -u all, exclude tzdata
since it makes test_datetime too slow (15-20 min on some buildbots)
which then times out on some buildbots.

-u tzdata must now be enabled explicitly, -u tzdata or -u all,tzdata,
to run all test_datetime tests.

Fix also regrtest command line parser to allow passing -u
extralargefile to run test_zipfile64.

Travis CI: remove -tzdata. Replace -u all,-tzdata,-cpu with -u all,-cpu since tzdata is now excluded from -u all.
2017-07-20 15:46:32 +02:00
Victor Stinner 48b5c422ff bpo-30776: reduce regrtest -R false positives (#2422)
* Change the regrtest --huntrleaks checker to decide if a test file
  leaks or not. Require that each run leaks at least 1 reference.
* Warmup runs are now completely ignored: ignored in the checker test
  and not used anymore to compute the sum.
* Add an unit test for a reference leak.

Example of reference differences previously considered a failure
(leak) and now considered as success (success, no leak):

    [3, 0, 0]
    [0, 1, 0]
    [8, -8, 1]
2017-06-27 02:02:04 +02:00
Victor Stinner 63f54c6893 bpo-30764: regrtest: add --fail-env-changed option (#2402)
* bpo-30764: regrtest: change exit code on failure

* Exit code 2 if failed tests ("bad")
* Exit code 3 if interrupted

* bpo-30764: regrtest: add --fail-env-changed option

If the option is set, mark a test as failed if it alters the
environment, for example if it creates a file without removing it.
2017-06-26 18:33:19 +02:00
Victor Stinner ace56d5836 bpo-30523: regrtest --list-cases --match (#2401)
* regrtest --list-cases now supports --match and --match-file options.
  Example: ./python -m test --list-cases -m FileTests test_os
* --list-cases now also sets support.verbose to False to prevent
  messages to stdout when loading test modules.
* Add support._match_test() private function.
2017-06-26 14:18:51 +02:00
Louie Lu b0c58d3c56 bpo-30523: Add --list-cases unittest (#2243)
* bpo-30523: Add --list-cases unittest

* Addressed haypo's request

* Addressed haypo's request
2017-06-16 13:15:28 +02:00