Issue #18103: Update README.txt and test_idle to describe and run gui tests.

Lib/idlelib/idle_test/README.txt

@@ -1,14 +1,19 @@
 README FOR IDLE TESTS IN IDLELIB.IDLE_TEST
 
+
+1. Test Files
+
 The idle directory, idlelib, has over 60 xyz.py files. The idle_test
-subdirectory should contain a test_xyy.py for each one. (For test modules,
-make 'xyz' lower case.) Each should start with the following cut-paste
-template, with the blanks after after '.'. 'as', and '_' filled in.
+subdirectory should contain a test_xyy.py for each. (For test modules, make
+'xyz' lower case, and possibly shorten it.) Each file should start with the
+something like the following template, with the blanks after after '.' and 'as',
+and before and after '_' filled in.
 ---
 import unittest
-import idlelib. as 
+from test.support import requires
+import idlelib. as
 
-class Test_(unittest.TestCase):
+class _Test(unittest.TestCase):
 
     def test_(self):
 
@@ -21,43 +26,75 @@ Once test_xyy is written, the following should go at the end of xyy.py,
 with xyz (lowercased) added after 'test_'.
 ---
 if __name__ == "__main__":
+    from test import support; support.use_resources = ['gui']
     import unittest
     unittest.main('idlelib.idle_test.test_', verbosity=2, exit=False)
 ---
 
-In Idle, pressing F5 in an editor window with either xyz.py or test_xyz.py
-loaded will then run the test with the version of Python running Idle and
-tracebacks will appear in the Shell window. The options are appropriate for
-developers running (as opposed to importing) either type of file during
-development: verbosity=2 lists all test_y methods; exit=False avoids a
-spurious sys.exit traceback when running in Idle. The following command
-lines also run test_xyz.py
+
+2. Gui Tests
+
+Gui tests need 'requires' and 'use_resources' from test.support
+(test.test_support in 2.7). A test is a gui test if it creates a Tk root or
+master object either directly or indirectly by instantiating a tkinter or
+idle class. For the benefit of buildbot machines that do not have a graphics
+screen, gui tests must be 'guarded' by "requires('gui')" in a setUp
+function or method. This will typically be setUpClass.
+---
+    @classmethod
+    def setUpClass(cls):
+        requires('gui')
+---
+All gui objects must be destroyed by the end of the test, perhaps in a tearDown
+function.
+
+Support.requires('gui') returns true if it is either called in a main module
+(which never happens on buildbots) or if use_resources contains 'gui'.
+Use_resources is set by test.regrtest but not by unittest. So when running
+tests in another module with unittest, we set it ourselves, as in the xyz.py
+template above.
+
+Since non-gui tests always run, but gui tests only sometimes, tests of non-gui
+operations should best avoid needing a gui. Methods that make incidental use of
+tkinter variables and messageboxes can do this by using the mock classes in
+idle_test/mock_tk.py.
+
+
+3. Running Tests
+
+Assume that xyz.py and test_xyz.py end with the "if __name__" statements given
+above. In Idle, pressing F5 in an editor window with either loaded will run all
+tests in the test_xyz file with the version of Python running Idle.  The test
+report and any tracebacks will appear in the Shell window. The options in these
+"if __name__" statements are appropriate for developers running (as opposed to
+importing) either of the files during development: verbosity=2 lists all test
+methods in the file; exit=False avoids a spurious sys.exit traceback that would
+otherwise occur when running in Idle. The following command lines also run
+all test methods, including gui tests, in test_xyz.py. (The exceptions are that
+idlelib and idlelib.idle start Idle and idlelib.PyShell should (issue 18330).)
 
 python -m idlelib.xyz  # With the capitalization of the xyz module
-python -m unittest -v idlelib.idle_test.test_xyz
+python -m idlelib.idle_test.test_xyz
 
-To run all idle tests either interactively ('>>>', with unittest imported)
-or from a command line, use one of the following.
+To run all idle_test/test_*.py tests, either interactively
+('>>>', with unittest imported) or from a command line, use one of the
+following. (Notes: unittest does not run gui tests; in 2.7, 'test ' (with the
+space) is 'test.regrtest '; where present, -v and -ugui can be omitted.)
 
 >>> unittest.main('idlelib.idle_test', verbosity=2, exit=False)
 python -m unittest -v idlelib.idle_test
+python -m test -v -ugui test_idle
 python -m test.test_idle
-python -m test test_idle
 
-The idle tests are 'discovered' in idlelib.idle_test.__init__.load_tests,
+The idle tests are 'discovered' by idlelib.idle_test.__init__.load_tests,
 which is also imported into test.test_idle. Normally, neither file should be
-changed when working on individual test modules. The last command runs runs
+changed when working on individual test modules. The third command runs runs
 unittest indirectly through regrtest. The same happens when the entire test
-suite is run with 'python -m test'. So it must work for buildbots to stay green.
+suite is run with 'python -m test'. So that command must work for buildbots
+to stay green. Idle tests must not disturb the environment in a way that
+makes other tests fail (issue 18081).
 
-To run an individual Testcase or test method, extend the
-dotted name given to unittest on the command line.
+To run an individual Testcase or test method, extend the dotted name given to
+unittest on the command line. (But gui tests will not this way.)
 
 python -m unittest -v idlelib.idle_test.text_xyz.Test_case.test_meth
-
-To disable test/test_idle.py, there are at least two choices.
-a. Comment out 'load_tests' line, no no tests are discovered (simple and safe);
-Running no tests passes, so there is no indication that nothing was run.
-b.Before that line, make module an unexpected skip for regrtest with
-import unittest; raise unittest.SkipTest('skip for buildbots')
-When run directly with unittest, this causes a normal exit and traceback.

Lib/test/test_idle.py

@@ -10,5 +10,10 @@ idletest = import_module('idlelib.idle_test')
 load_tests = idletest.load_tests
 
 if __name__ == '__main__':
+    # Until unittest supports resources, we emulate regrtest's -ugui
+    # so loaded tests run the same as if textually present here.
+    # If any Idle test ever needs another resource, add it to the list.
+    from test import support
+    support.use_resources = ['gui']  # use_resources is initially None
     import unittest
     unittest.main(verbosity=2, exit=False)