Commit Graph

43 Commits

Author SHA1 Message Date
Victor Stinner e566ce5496
gh-93353: regrtest checks for leaked temporary files (#93776)
When running tests with -jN, create a temporary directory per process
and mark a test as "environment changed" if a test leaks a temporary
file or directory.
2022-06-14 13:43:02 +02:00
Christian Heimes 59be9cd748
bpo-40280: Detect if WASM platform supports threading (GH-32243)
Automerge-Triggered-By: GH:tiran
2022-04-02 01:12:44 -07:00
Victor Stinner 9d32714384
bpo-45745: Remove regrtest --findleaks options (GH-29514)
Remove the --findleaks command line option of regrtest: use the
--fail-env-changed option instead. Since Python 3.7, it was a
deprecated alias to the --fail-env-changed option.
2021-11-12 16:19:09 +01:00
Victor Stinner 676201a59f
bpo-45410: regrtest replaces print_warning.orig_stderr (GH-28926)
When running Python tests with -W, runtest() now replaces
support.print_warning.orig_stderr to preserve the messages order.

Add an unit test.
2021-10-13 17:35:21 +02:00
Victor Stinner 7733307739
bpo-45410: regrtest -W leaves stdout/err FD unchanged (GH-28915)
support.print_warning() now stores the original value of
sys.__stderr__ and uses it to log warnings. libregrtest uses the same
stream to log unraisable exceptions and uncaught threading
exceptions.

Partially revert commit dbe213de7ef28712bbfdb9d94a33abb9c33ef0c2:
libregrtest no longer replaces sys.__stdout__, sys.__stderr__, and
stdout and stderr file descriptors.

Remove also a few unused imports in libregrtest.
2021-10-13 14:08:18 +02:00
Victor Stinner dbe213de7e
bpo-45410: Enhance libregrtest -W/--verbose3 option (GH-28908)
libregrtest -W/--verbose3 now also replace sys.__stdout__,
sys.__stderr__, and stdout and stderr file descriptors (fd 1 and fd
2).

support.print_warning() messages are now logged in the expected
order.

The "./python -m test test_eintr -W" command no longer logs into
stdout if the test pass.
2021-10-13 01:52:22 +02:00
Victor Stinner 679cb4781e
bpo-44895: libregrtest: refleak check clears types later (GH-28113)
libregrtest now clears the type cache later to reduce the risk of
false alarm when checking for reference leaks. Previously, the type
cache was cleared too early and libregrtest raised a false alarm
about reference leaks under very specific conditions.

Move also support.gc_collect() outside clear/cleanup functions to
make the garbage collection more explicit.

Co-authored-by: Irit Katriel <1055913+iritkatriel@users.noreply.github.com>
2021-09-01 17:45:27 +02:00
Łukasz Langa f1afef5e0d
bpo-44708: Only re-run test methods that match names of previously failing test methods (GH-27287)
* Move to a static argparse.Namespace subclass
* Roughly annotate runtest.py
* Refactor libregrtest to use lossless test result objects
* Only re-run test methods that match names of previously failing test methods
* Adopt tests to cover test method name matching

Co-authored-by: Pablo Galindo Salgado <Pablogsal@gmail.com>
2021-07-22 20:25:58 +02:00
Victor Stinner 0473fb2229
bpo-41718: libregrtest runtest avoids import_helper (GH-24983)
Inline import_helper.unload() in libregrtest.runtest to avoid one
import.
2021-03-23 01:08:49 +01:00
Victor Stinner 10417dd15f
bpo-41718: Reduce libregrtest runtest imports (GH-24980)
Move clear_caches() from libregrtest.refleak to libregrtest.utils to
avoid importing libregrtest.refleak when it's not needed.

clear_caches() now only calls re.purge() if 're' is in sys.modules.
2021-03-23 00:17:05 +01:00
Victor Stinner 532e063fc2
bpo-41718: regrtest saved_test_environment avoids imports (GH-24934)
Reduce the number of modules imported by libregrtest.

saved_test_environment no longer imports modules at startup, but try
to get them from sys.modules. If an module is missing, skip the test.
It also sets directly support.environment_altered.

runtest() now now two saved_test_environment instances: one before
importing the test module, one after importing it.

Remove imports from test.libregrtest.save_env:

* asyncio
* logging
* multiprocessing
* shutil
* sysconfig
* urllib.request
* warnings

When a test method imports a module (ex: warnings) and the test
has a side effect (ex: add a warnings filter), the side effect is not
detected, because the module was not imported when Python
enters the saved_test_environment context manager.
2021-03-22 23:52:13 +01:00
Hai Shi f7ba40b505
bpo-40275: Use new test.support helper submodules in tests (GH-20849) 2020-06-25 12:38:51 +02:00
Victor Stinner d663d34685
bpo-39983: Add test.support.print_warning() (GH-19683)
Log "Warning -- ..." test warnings into sys.__stderr__ rather than
sys.stderr, to ensure to display them even if sys.stderr is captured.

test.libregrtest.utils.print_warning() now calls
test.support.print_warning().
2020-04-23 19:03:52 +02:00
Pablo Galindo e0cd8aa70a
bpo-37957: Allow regrtest to receive a file with test (and subtests) to ignore (GH-16989)
When building Python in some uncommon platforms there are some known tests that will fail. Right now, the test suite has the ability to ignore entire tests using the -x option and to receive a filter file using the --matchfile filter. The problem with the --matchfile option is that it receives a file with patterns to accept and when you want to ignore a couple of tests and subtests, is too cumbersome to lists ALL tests that are not the ones that you want to accept and he problem with -x is that is not easy to ignore just a subtests that fail and the whole test needs to be ignored.

For these reasons, add a new option to allow to ignore a list of test and subtests for these situations.
2019-11-19 23:46:49 +00:00
Greg Price 455122a009 bpo-37936: Systematically distinguish rooted vs. unrooted in .gitignore (GH-15823)
A root cause of bpo-37936 is that it's easy to write a .gitignore
rule that's intended to apply to a specific file (e.g., the
`pyconfig.h` generated by `./configure`) but actually applies to all
similarly-named files in the tree (e.g., `PC/pyconfig.h`.)

Specifically, any rule with no non-trailing slashes is applied in an
"unrooted" way, to files anywhere in the tree.  This means that if we
write the rules in the most obvious-looking way, then

 * for specific files we want to ignore that happen to be in
   subdirectories (like `Modules/config.c`), the rule will work
   as intended, staying "rooted" to the top of the tree; but

 * when a specific file we want to ignore happens to be at the root of
   the repo (like `platform`), then the obvious rule (`platform`) will
   apply much more broadly than intended: if someone tries to add a
   file or directory named `platform` somewhere else in the tree, it
   will unexpectedly get ignored.

That's surprising behavior that can make the .gitignore file's
behavior feel finicky and unpredictable.

To avoid it, we can simply always give a rule "rooted" behavior when
that's what's intended, by systematically using leading slashes.

Further, to help make the pattern obvious when looking at the file and
minimize any need for thinking about the syntax when adding new rules:
separate the rules into one group for each type, with brief comments
identifying them.

For most of these rules it's clear whether they're meant to be rooted
or unrooted, but in a handful of cases I've only guessed.  In that
case the safer default (the choice that won't hide information) is the
narrower, rooted meaning, with a leading slash.  If for some of these
the unrooted meaning is desired after all, it'll be easy to move them
to the unrooted section at the top.
2019-09-11 10:25:26 +01:00
Victor Stinner b0c8369c60
bpo-37531: Fix regrtest timeout for subprocesses (GH-15072)
Co-Authored-By: Joannah Nanjekye <joannah.nanjekye@ibm.com>
2019-08-14 14:18:51 +02:00
Victor Stinner b0917df329
bpo-36719: regrtest -jN no longer stops on crash (GH-13231)
"python3 -m test -jN ..." now continues the execution of next tests
when a worker process crash (CHILD_ERROR state). Previously, the test
suite stopped immediately. Use --failfast to stop at the first error.

Moreover, --forever now also implies --failfast.
2019-05-13 19:17:54 +02:00
Victor Stinner 75120d2205
bpo-36719: regrtest always detect uncollectable objects (GH-12951)
regrtest now always detects uncollectable objects. Previously, the
check was only enabled by --findleaks. The check now also works with
-jN/--multiprocess N.

--findleaks becomes a deprecated alias to --fail-env-changed.
2019-04-26 09:28:53 +02:00
Victor Stinner 3cde440f20
bpo-36725: Refactor regrtest multiprocessing code (GH-12961)
Rewrite run_tests_multiprocess() function as a new MultiprocessRunner
class with multiple methods to better report errors and stop
immediately when needed.

Changes:

* Worker processes are now killed immediately if tests are
  interrupted or if a test does crash (CHILD_ERROR): worker
  processes are killed.
* Rewrite how errors in a worker thread are reported to
  the main thread. No longer ignore BaseException or parsing errors
  silently.
* Remove 'finished' variable: use worker.is_alive() instead
* Always compute omitted tests. Add Regrtest.get_executed() method.
2019-04-26 08:40:25 +02:00
Victor Stinner 4d29983185
bpo-36725: regrtest: add TestResult type (GH-12960)
* Add TestResult and MultiprocessResult types to ensure that results
  always have the same fields.
* runtest() now handles KeyboardInterrupt
* accumulate_result() and format_test_result() now takes a TestResult
* cleanup_test_droppings() is now called by runtest() and mark the
  test as ENV_CHANGED if the test leaks support.TESTFN file.
* runtest() now includes code "around" the test in the test timing
* Add print_warning() in test.libregrtest.utils to standardize how
  libregrtest logs warnings to ease parsing the test output.
* support.unload() is now called with abstest rather than test_name
* Rename 'test' variable/parameter to 'test_name'
* dash_R(): remove unused the_module parameter
* Remove unused imports
2019-04-26 04:08:53 +02:00
Victor Stinner 5aaac94eeb
bpo-36560: Fix reference leak hunting in regrtest (GH-12744)
Fix reference leak hunting in regrtest: compute also deltas (of
reference count, allocated memory blocks, file descriptor count)
during warmup, to ensure that everything is initialized before
starting to hunt reference leaks.

Other changes:

* Replace gc.collect() with support.gc_collect()
* Move calls to read memory statistics from dash_R_cleanup() to
  dash_R()
* Pass regrtest 'ns' to dash_R()
* dash_R() is now more quiet with --quiet option (don't display
  progress).
* Precompute the full range for "for it in range(repcount):" to
  ensure that the iteration doesn't allocate anything new.
* dash_R() now is responsible to call warm_caches().
2019-04-09 14:23:47 +02:00
Anthony Sottile 8377cd4fcd Clean up code which checked presence of os.{stat,lstat,chmod} (#11643) 2019-02-25 23:32:27 +01:00
Victor Stinner 8db5b54463
bpo-35513, unittest: TextTestRunner uses time.perf_counter() (GH-11180)
TextTestRunner of unittest.runner now uses time.perf_counter() rather
than time.time() to measure the execution time of a test: time.time()
can go backwards, whereas time.perf_counter() is monotonic.

Similar change made in libregrtest, pprint and random.
2018-12-17 11:30:34 +01:00
Pablo Galindo 9724348b43 bpo-34279, regrtest: Issue a warning if no tests have been executed (GH-10150) 2018-11-29 18:17:44 +01:00
Steve Dower d0f49d2f50
bpo-34582: Adds JUnit XML output for regression tests (GH-9210) 2018-09-18 09:10:26 -07:00
Pablo Galindo 58ed7307ea
bpo-33873: Fix bug in `runtest.py` and add checks for invalid `-R` parameters (GH-7735)
Fix bug in `Lib/test/libregrtest/runtest.py` that makes running tests an extra time than the specified number of runs.

Add check for invalid --huntrleaks/-R parameters.
2018-06-26 15:17:26 +01:00
Victor Stinner 803ddd8ce2
bpo-31324: Optimize support._match_test() (#4421)
* Rename support._match_test() to support.match_test(): make it
  public
* Remove support.match_tests global variable. It is replaced with a
  new support.set_match_tests() function, so match_test() doesn't
  have to check each time if patterns were modified.
* Rewrite match_test(): use different code paths depending on the
  kind of patterns for best performances.

Co-Authored-By: Serhiy Storchaka <storchaka@gmail.com>
2017-11-21 15:34:02 -08:00
Victor Stinner e3510d74aa bpo-31160: regrtest now reaps child processes (#3044)
Add a post_test_cleanup() function which currently only calls
support.reap_children().
2017-08-09 17:44:33 +02:00
Victor Stinner 21a0a6c9f8 threading_cleanup() failure marks test as ENV_CHANGED (#2500)
If threading_cleanup() fails to cleanup threads, set a a new
support.environment_altered flag to true, flag uses by save_env which
is used by regrtest to check if a test altered the environment. At
the end, the test file fails with ENV_CHANGED instead of SUCCESS, to
report that it altered the environment.
2017-06-30 10:59:52 +02:00
mlouielu a49c935cfd bpo-30523: regrtest: Add --list-cases option (#2238)
* bpo-30523: regrtest: Add --list-cases option

* bpo-30523: Enhance --list-cases

* Add get_abs_module() function, use it in list_cases()
* list_cases() now logs skipped tests into stderr

* Remove unused doctest
2017-06-16 11:36:19 +02:00
Serhiy Storchaka 839102603c Issue #23839: Various caches now are cleared before running every test file. 2016-11-11 11:46:44 +02:00
Victor Stinner 21c8c92422 Merge 3.5 (issue #27829) 2016-09-23 11:15:50 +02:00
Victor Stinner ab98367747 Cleanup libregrtest
* main.py: remove unused import
* runtest: simplify runtest_inner() parameters, reuse ns parameter
2016-08-22 14:28:52 +02:00
Victor Stinner 1b8b42344e regrtest: display test result (passed, failed, ...)
* in multiprocessing mode: always display the result
* sequential mode: only display the result if the test did not pass
2016-05-20 13:37:40 +02:00
Victor Stinner 9759dd3343 Issue #26295: When using "python3 -m test --testdir=TESTDIR", regrtest doesn't
add "test." prefix to test module names.

regrtest also prepends testdir to sys.path.
2016-03-30 02:32:52 +02:00
Victor Stinner 69649f21f0 regrtest: display test duration in sequential mode
Only display duration if a test takes more than 30 seconds.
2016-03-23 12:14:10 +01:00
Brett Cannon 11faa21843 Merge from 3.5 for issue #25188. 2015-10-02 16:20:49 -07:00
Victor Stinner 6f20a2e01f Issue #25220, libregrtest: Pass directly ns to runtest()
* Remove runtest_ns(): pass directly ns to runtest().
* Create also Regrtest.rerun_failed_tests() method.
* Inline again Regrtest.run_test(): it's no more justified to have a method
2015-09-30 02:32:11 +02:00
Victor Stinner a204502dbf Issue #25220, libregrtest: Set support.use_resources in setup_tests()
* Rename setup_python() to setup_tests()
* Remove use_resources parameter of runtest()
2015-09-30 02:17:28 +02:00
Victor Stinner 8bb19f094b Issue #25220, libregrtest: Add runtest_ns() function
* Factorize code to run tests.
* run_test_in_subprocess() now pass the whole "ns" namespace to the child
  process.
2015-09-30 01:32:39 +02:00
Victor Stinner f33536c430 Issue #25220: Use print(flush=True) in libregrtest 2015-09-30 00:48:27 +02:00
Victor Stinner 56e05dd0b0 Issue #25220: Create libregrtest/runtest_mp.py
Move the code to run tests in multiple processes using threading and subprocess
to a new submodule.

Move also slave_runner() (renamed to run_tests_slave()) and
run_test_in_subprocess() (renamed to run_tests_in_subprocess()) there.
2015-09-29 23:15:38 +02:00
Victor Stinner 3844fe5ed8 Issue #25220: Move most regrtest.py code to libregrtest 2015-09-26 10:38:01 +02:00