Commit Graph

71 Commits

Author SHA1 Message Date
Victor Stinner 2ec4545ebe
bpo-37359: Add --cleanup option to python3 -m test (GH-14332) (GH-14334)
* regrtest: Add --cleanup option to remove "test_python_*" directories
  of previous failed test jobs.
* Add "make cleantest" to run "python3 -m test --cleanup".

(cherry picked from commit 47fbc4e45b)
2019-06-24 12:51:47 +02:00
Victor Stinner 3c93153f7d
bpo-36915: regrtest always remove tempdir of worker processes (GH-13312)
When using multiprocessing (-jN option), worker processes now create
their temporary directory inside the temporary directory of the
main process. So the main process is able to remove temporary
directories of worker processes even if they crash or when they are
killed by regrtest on KeyboardInterrupt (CTRL+c).

Rework also how multiprocessing arguments are parsed in main.py.
2019-05-14 15:49:16 +02:00
Victor Stinner b0917df329
bpo-36719: regrtest -jN no longer stops on crash (GH-13231)
"python3 -m test -jN ..." now continues the execution of next tests
when a worker process crash (CHILD_ERROR state). Previously, the test
suite stopped immediately. Use --failfast to stop at the first error.

Moreover, --forever now also implies --failfast.
2019-05-13 19:17:54 +02:00
Victor Stinner 00db7c73af
bpo-36719: regrtest closes explicitly WindowsLoadTracker (GH-12965)
Regrtest.finalize() now closes explicitly the WindowsLoadTracker
instance.
2019-04-26 11:12:26 +02:00
Victor Stinner 837acc1957
bpo-36719: Fix regrtest re-run (GH-12964)
Properly handle a test which fail but then pass.

Add test_rerun_success() unit test.
2019-04-26 09:56:37 +02:00
Victor Stinner 75120d2205
bpo-36719: regrtest always detect uncollectable objects (GH-12951)
regrtest now always detects uncollectable objects. Previously, the
check was only enabled by --findleaks. The check now also works with
-jN/--multiprocess N.

--findleaks becomes a deprecated alias to --fail-env-changed.
2019-04-26 09:28:53 +02:00
Victor Stinner 3cde440f20
bpo-36725: Refactor regrtest multiprocessing code (GH-12961)
Rewrite run_tests_multiprocess() function as a new MultiprocessRunner
class with multiple methods to better report errors and stop
immediately when needed.

Changes:

* Worker processes are now killed immediately if tests are
  interrupted or if a test does crash (CHILD_ERROR): worker
  processes are killed.
* Rewrite how errors in a worker thread are reported to
  the main thread. No longer ignore BaseException or parsing errors
  silently.
* Remove 'finished' variable: use worker.is_alive() instead
* Always compute omitted tests. Add Regrtest.get_executed() method.
2019-04-26 08:40:25 +02:00
Victor Stinner 4d29983185
bpo-36725: regrtest: add TestResult type (GH-12960)
* Add TestResult and MultiprocessResult types to ensure that results
  always have the same fields.
* runtest() now handles KeyboardInterrupt
* accumulate_result() and format_test_result() now takes a TestResult
* cleanup_test_droppings() is now called by runtest() and mark the
  test as ENV_CHANGED if the test leaks support.TESTFN file.
* runtest() now includes code "around" the test in the test timing
* Add print_warning() in test.libregrtest.utils to standardize how
  libregrtest logs warnings to ease parsing the test output.
* support.unload() is now called with abstest rather than test_name
* Rename 'test' variable/parameter to 'test_name'
* dash_R(): remove unused the_module parameter
* Remove unused imports
2019-04-26 04:08:53 +02:00
Paul Monson 264a0b40b0 bpo-36638: Fix WindowsLoadTracker exception on some Windows versions (GH-12849) 2019-04-17 18:06:06 -07:00
Ammar Askar e16467af0b bpo-34060: Report system load when running test suite for Windows (GH-8357)
While Windows exposes the system processor queue length, the raw value
used for load calculations on Unix systems, it does not provide an API
to access the averaged value. Hence to calculate the load we must track
and average it ourselves. We can't use multiprocessing or a thread to
read it in the background while the tests run since using those would
conflict with test_multiprocessing and test_xxsubprocess.

Thus, we use Window's asynchronous IO API to run the tracker in the
background with it sampling at the correct rate. When we wish to access
the load we check to see if there's new data on the stream, if there is,
we update our load values.
2019-04-09 08:20:41 -04:00
Steve Dower 28f6cb34f6
bpo-35683: Improve Azure Pipelines steps (GH-11493) 2019-01-22 10:49:52 -08:00
Pablo Galindo 9724348b43 bpo-34279, regrtest: Issue a warning if no tests have been executed (GH-10150) 2018-11-29 18:17:44 +01:00
Steve Dower 38df97a03c
Add --tempdir option for test run (GH-10322) 2018-11-17 04:14:36 -08:00
Steve Dower d0f49d2f50
bpo-34582: Adds JUnit XML output for regression tests (GH-9210) 2018-09-18 09:10:26 -07:00
Victor Stinner 012f5b968a
bpo-34605, libregrtest: Rename --slaveargs to --worker-args (GH-9099)
Rename also run_tests_slave() to run_tests_worker().
2018-09-07 17:20:42 +02:00
Pablo Galindo 58ed7307ea
bpo-33873: Fix bug in `runtest.py` and add checks for invalid `-R` parameters (GH-7735)
Fix bug in `Lib/test/libregrtest/runtest.py` that makes running tests an extra time than the specified number of runs.

Add check for invalid --huntrleaks/-R parameters.
2018-06-26 15:17:26 +01:00
Pablo Galindo cac4fef886 bpo-33873: regrtest: Add warning on -R 1:3 (GH-7736)
regrtest: Add warning when using less than 3 warmup runs like -R 1:3.
2018-06-16 02:20:56 +02:00
Victor Stinner c45fc7673e
bpo-33718: regrtest: use "xxx then yyy" result if re-run (GH-7521)
If tests are re-run, use "xxx then yyy" result format (ex: "FAILURE
then SUCCESS") to show that some failing tests have been re-run.

Add also test_regrtest.test_rerun_fail() test.
2018-06-08 09:53:51 +02:00
Victor Stinner 7ad16eb51c
bpo-33718: regrtest: enhance "running:" formatting (GH-7292)
* "running:" progress: Format number of seconds as hours and minutes
* format_duration(): count also minutes as hours
* Create Lib/test/libregrtest/utils.py
2018-06-01 11:04:45 +02:00
Victor Stinner 9e24930dfd
bpo-33718: regrtest keeps filters to re-run fails (GH-7291)
* No longer clear filters, like --match, to re-run failed tests in
  verbose mode (-w option).
* Tests result: always indicate if tests have been interrupted.
* Enhance tests summary
2018-06-01 00:48:57 +02:00
Victor Stinner c6c05d0e69
regrtest: repeat summary after re-run (GH-7159)
Using -w, when failing tests are re-run in verbose mode, display
again the tests results at the end.
2018-05-28 21:03:43 +02:00
Victor Stinner 803ddd8ce2
bpo-31324: Optimize support._match_test() (#4421)
* Rename support._match_test() to support.match_test(): make it
  public
* Remove support.match_tests global variable. It is replaced with a
  new support.set_match_tests() function, so match_test() doesn't
  have to check each time if patterns were modified.
* Rewrite match_test(): use different code paths depending on the
  kind of patterns for best performances.

Co-Authored-By: Serhiy Storchaka <storchaka@gmail.com>
2017-11-21 15:34:02 -08:00
Victor Stinner b907abc885 bpo-30871: Add test.pythoninfo (#3075)
* Add Lib/test/pythoninfo.py: script collecting various informations
  about Python to help debugging test failures.
* regrtest: remove sys.hash_info and sys.flags from header.
* Travis CI, Appveyor: run pythoninfo before tests
2017-08-17 16:40:51 +02:00
Victor Stinner b97d99cd7f regrtest: count also env changed as failures in progress (#3061) 2017-08-10 16:19:03 +02:00
Victor Stinner 5e87592fd1 bpo-30764: Fix regrtest --fail-env-changed --forever (#2536)
--forever now stops if a fail changes the environment.
2017-07-03 11:15:58 +02:00
Victor Stinner 63f54c6893 bpo-30764: regrtest: add --fail-env-changed option (#2402)
* bpo-30764: regrtest: change exit code on failure

* Exit code 2 if failed tests ("bad")
* Exit code 3 if interrupted

* bpo-30764: regrtest: add --fail-env-changed option

If the option is set, mark a test as failed if it alters the
environment, for example if it creates a file without removing it.
2017-06-26 18:33:19 +02:00
Victor Stinner ace56d5836 bpo-30523: regrtest --list-cases --match (#2401)
* regrtest --list-cases now supports --match and --match-file options.
  Example: ./python -m test --list-cases -m FileTests test_os
* --list-cases now also sets support.verbose to False to prevent
  messages to stdout when loading test modules.
* Add support._match_test() private function.
2017-06-26 14:18:51 +02:00
mlouielu a49c935cfd bpo-30523: regrtest: Add --list-cases option (#2238)
* bpo-30523: regrtest: Add --list-cases option

* bpo-30523: Enhance --list-cases

* Add get_abs_module() function, use it in list_cases()
* list_cases() now logs skipped tests into stderr

* Remove unused doctest
2017-06-16 11:36:19 +02:00
Victor Stinner f7e07840d7 bpo-30284: Fix regrtest for out of tree build (#1481)
Use a build/ directory in the build directory, not in the source
directory, since the source directory may be read-only and must not
be modified.

Fallback on the source directory if the build directory is not
available (missing "abs_builddir" sysconfig variable).
2017-06-15 00:44:05 +02:00
Victor Stinner 74683fc624 bpo-30258: regrtest: Fix run_tests_multiprocess() (#1479)
If the child process exited with a non-zero code, don't strip the
last line of stdout anymore.

Add also a sanity check in accumulate_result().
2017-05-09 11:34:01 +02:00
Victor Stinner 3d0056842c bpo-30263: regrtest: log system load (#1452)
* bpo-30263: regrtest: log system load

* regrtest: log the number of CPUs

Log the number of CPUs in the header. --verbose now imply --header.
2017-05-04 15:21:12 +02:00
Victor Stinner 5bad70def6 regrtest: don't fail immediately if a child does crash
Issue #29362: Catch a crash of a worker process as a normal failure and
continue to run next tests. It allows to get the usual test summary: single
line result (OK/FAIL), total duration, etc.
2017-02-06 12:42:00 +01:00
Victor Stinner e8cfec5abc Issue #29035: Simplify a regex in libregrtest
regrtest: simplify the regex used to match test names for the --fromfile
command line option.
2017-01-03 01:38:58 +01:00
Victor Stinner c24217e144 regrtest --fromfile now accepts a list of filenames 2016-12-09 16:05:51 +01:00
Victor Stinner b96ef55d49 regrtest: log FS and locale encodings 2016-09-08 21:46:56 -07:00
Steve Dower 22d0698d3b Adds test.support.PGO and skips tests that are not useful for PGO. 2016-09-06 19:38:15 -07:00
Victor Stinner ab98367747 Cleanup libregrtest
* main.py: remove unused import
* runtest: simplify runtest_inner() parameters, reuse ns parameter
2016-08-22 14:28:52 +02:00
Victor Stinner f2abf5c11a regrtest: replace "Result:" with "Tests result:" 2016-08-19 17:54:25 +02:00
Victor Stinner 636860354e regrtest: add a summary of the summary, "Result: xxx"
It's sometimes hard to check quickly if tests succeeded, failed or something
bad happened. I added a final "Result: xxx" line which summarizes all outputs
into a single line, written at the end (it should always be the last line of
the output).
2016-08-17 16:12:16 +02:00
Victor Stinner c5a01f8551 regrtest: set interrupted to True if re-run is interrupted 2016-08-17 16:00:12 +02:00
Victor Stinner 8f00319294 regrtest: add newlines in output for readability 2016-08-17 15:42:21 +02:00
Victor Stinner 435eaf4422 regrtest: nicer output for durations
Use milliseconds and minutes units, not only seconds.
2016-08-17 12:22:52 +02:00
Victor Stinner 1b8b42344e regrtest: display test result (passed, failed, ...)
* in multiprocessing mode: always display the result
* sequential mode: only display the result if the test did not pass
2016-05-20 13:37:40 +02:00
Victor Stinner 622583e9bf regrtest: round final timing towards +inf 2016-03-27 18:28:15 +02:00
Victor Stinner 3aac0adfe0 Cleanup regrtest "main()" function
* Rename libregrtest.main_in_temp_cwd() to libregrtest.main()
* Add regrtest.main_in_temp_cwd() alias to libregrtest.main()
* Move old main_in_temp_cwd() code into libregrtest.Regrtest.main()
* Update multiple scripts to call libregrtest.main()
2016-03-24 17:53:20 +01:00
Victor Stinner 2b60b7237e regrtest: mention in tests run sequentially or in parallel 2016-03-24 11:55:29 +01:00
Victor Stinner 5de16e80c1 regrtest: fix --fromfile feature
* Update code for the name regrtest output format.
* Enhance also test_regrtest test on --fromfile
2016-03-24 09:43:00 +01:00
Victor Stinner 69649f21f0 regrtest: display test duration in sequential mode
Only display duration if a test takes more than 30 seconds.
2016-03-23 12:14:10 +01:00
Victor Stinner 24f949e10c regrtest: add time to output
Timestamps should help to debug slow buildbots, and timeout and hang on
buildbots.
2016-03-22 15:14:09 +01:00
Victor Stinner 3909e58994 Close #25373: Fix regrtest --slow with interrupted test
* Fix accumulate_result(): don't use time on interrupted and failed test
* Add unit test for interrupted test
* Add unit test on --slow with interrupted test, with and without
  multiprocessing
2015-10-11 10:37:25 +02:00