diff --git a/Doc/library/unittest.rst b/Doc/library/unittest.rst index 69227cf8cba..a52677d5bbf 100644 --- a/Doc/library/unittest.rst +++ b/Doc/library/unittest.rst @@ -11,9 +11,9 @@ .. versionadded:: 2.1 -.. versionchanged:: 2.7 - Added :ref:`skipping and expected failures `. +.. versionchanged:: 2.7 + Added test :ref:`skipping and expected failures `. The Python unit testing framework, sometimes referred to as "PyUnit," is a Python language version of JUnit, by Kent Beck and Erich Gamma. JUnit is, in @@ -52,27 +52,28 @@ The test case and test fixture concepts are supported through the :class:`TestCase` and :class:`FunctionTestCase` classes; the former should be used when creating new tests, and the latter can be used when integrating existing test code with a :mod:`unittest`\ -driven framework. When building test -fixtures using :class:`TestCase`, the :meth:`setUp` and :meth:`tearDown` methods -can be overridden to provide initialization and cleanup for the fixture. With -:class:`FunctionTestCase`, existing functions can be passed to the constructor -for these purposes. When the test is run, the fixture initialization is run -first; if it succeeds, the cleanup method is run after the test has been -executed, regardless of the outcome of the test. Each instance of the -:class:`TestCase` will only be used to run a single test method, so a new -fixture is created for each test. +fixtures using :class:`TestCase`, the :meth:`~TestCase.setUp` and +:meth:`~TestCase.tearDown` methods can be overridden to provide initialization +and cleanup for the fixture. With :class:`FunctionTestCase`, existing functions +can be passed to the constructor for these purposes. When the test is run, the +fixture initialization is run first; if it succeeds, the cleanup method is run +after the test has been executed, regardless of the outcome of the test. Each +instance of the :class:`TestCase` will only be used to run a single test method, +so a new fixture is created for each test. Test suites are implemented by the :class:`TestSuite` class. This class allows individual tests and test suites to be aggregated; when the suite is executed, all tests added directly to the suite and in "child" test suites are run. A :class:`ClassTestSuite` contains the test cases of a class. -A test runner is an object that provides a single method, :meth:`run`, which -accepts a :class:`TestCase` or :class:`TestSuite` object as a parameter, and -returns a result object. The class :class:`TestResult` is provided for use as -the result object. :mod:`unittest` provides the :class:`TextTestRunner` as an -example test runner which reports test results on the standard error stream by -default. Alternate runners can be implemented for other environments (such as -graphical environments) without any need to derive from a specific class. +A test runner is an object that provides a single method, +:meth:`~TestRunner.run`, which accepts a :class:`TestCase` or :class:`TestSuite` +object as a parameter, and returns a result object. The class +:class:`TestResult` is provided for use as the result object. :mod:`unittest` +provides the :class:`TextTestRunner` as an example test runner which reports +test results on the standard error stream by default. Alternate runners can be +implemented for other environments (such as graphical environments) without any +need to derive from a specific class. .. seealso:: @@ -104,17 +105,17 @@ Here is a short script to test three functions from the :mod:`random` module:: def setUp(self): self.seq = range(10) - def testshuffle(self): + def test_shuffle(self): # make sure the shuffled sequence does not lose any elements random.shuffle(self.seq) self.seq.sort() self.assertEqual(self.seq, range(10)) - def testchoice(self): + def test_choice(self): element = random.choice(self.seq) self.assert_(element in self.seq) - def testsample(self): + def test_sample(self): self.assertRaises(ValueError, random.sample, self.seq, 20) for element in random.sample(self.seq, 5): self.assert_(element in self.seq) @@ -122,21 +123,22 @@ Here is a short script to test three functions from the :mod:`random` module:: if __name__ == '__main__': unittest.main() -A testcase is created by subclassing :class:`unittest.TestCase`. The three +A testcase is created by subclassing :class:`unittest.TestCase`. The three individual tests are defined with methods whose names start with the letters ``test``. This naming convention informs the test runner about which methods represent tests. -The crux of each test is a call to :meth:`assertEqual` to check for an expected -result; :meth:`assert_` to verify a condition; or :meth:`assertRaises` to verify -that an expected exception gets raised. These methods are used instead of the -:keyword:`assert` statement so the test runner can accumulate all test results -and produce a report. +The crux of each test is a call to :meth:`~TestCase.assertEqual` to check for an +expected result; :meth:`~TestCase.assert_` to verify a condition; or +:meth:`~TestCase.assertRaises` to verify that an expected exception gets raised. +These methods are used instead of the :keyword:`assert` statement so the test +runner can accumulate all test results and produce a report. -When a :meth:`setUp` method is defined, the test runner will run that method -prior to each test. Likewise, if a :meth:`tearDown` method is defined, the test -runner will invoke that method after each test. In the example, :meth:`setUp` -was used to create a fresh sequence for each test. +When a :meth:`~TestCase.setUp` method is defined, the test runner will run that +method prior to each test. Likewise, if a :meth:`~TestCase.tearDown` method is +defined, the test runner will invoke that method after each test. In the +example, :meth:`~TestCase.setUp` was used to create a fresh sequence for each +test. The final block shows a simple way to run the tests. :func:`unittest.main` provides a command line interface to the test script. When run from the command @@ -191,8 +193,8 @@ The testing code of a :class:`TestCase` instance should be entirely self contained, such that it can be run either in isolation or in arbitrary combination with any number of other test cases. -The simplest :class:`TestCase` subclass will simply override the :meth:`runTest` -method in order to perform specific testing code:: +The simplest :class:`TestCase` subclass will simply override the +:meth:`~TestCase.runTest` method in order to perform specific testing code:: import unittest @@ -221,8 +223,8 @@ the above case, constructing a :class:`Widget` in each of 100 Widget test case subclasses would mean unsightly duplication. Luckily, we can factor out such set-up code by implementing a method called -:meth:`setUp`, which the testing framework will automatically call for us when -we run the test:: +:meth:`~TestCase.setUp`, which the testing framework will automatically call for +us when we run the test:: import unittest @@ -241,12 +243,12 @@ we run the test:: self.failUnless(self.widget.size() == (100,150), 'wrong size after resize') -If the :meth:`setUp` method raises an exception while the test is running, the -framework will consider the test to have suffered an error, and the -:meth:`runTest` method will not be executed. +If the :meth:`~TestCase.setUp` method raises an exception while the test is +running, the framework will consider the test to have suffered an error, and the +:meth:`~TestCase.runTest` method will not be executed. -Similarly, we can provide a :meth:`tearDown` method that tidies up after the -:meth:`runTest` method has been run:: +Similarly, we can provide a :meth:`~TestCase.tearDown` method that tidies up +after the :meth:`~TestCase.runTest` method has been run:: import unittest @@ -258,8 +260,8 @@ Similarly, we can provide a :meth:`tearDown` method that tidies up after the self.widget.dispose() self.widget = None -If :meth:`setUp` succeeded, the :meth:`tearDown` method will be run whether -:meth:`runTest` succeeded or not. +If :meth:`~TestCase.setUp` succeeded, the :meth:`~TestCase.tearDown` method will +be run whether :meth:`~TestCase.runTest` succeeded or not. Such a working environment for the testing code is called a :dfn:`fixture`. @@ -288,11 +290,12 @@ mechanism:: self.failUnless(self.widget.size() == (100,150), 'wrong size after resize') -Here we have not provided a :meth:`runTest` method, but have instead provided -two different test methods. Class instances will now each run one of the -:meth:`test\*` methods, with ``self.widget`` created and destroyed separately -for each instance. When creating an instance we must specify the test method it -is to run. We do this by passing the method name in the constructor:: +Here we have not provided a :meth:`~TestCase.runTest` method, but have instead +provided two different test methods. Class instances will now each run one of +the :meth:`test\*` methods, with ``self.widget`` created and destroyed +separately for each instance. When creating an instance we must specify the +test method it is to run. We do this by passing the method name in the +constructor:: defaultSizeTestCase = WidgetTestCase('testDefaultSize') resizeTestCase = WidgetTestCase('testResize') @@ -411,6 +414,11 @@ may treat :exc:`AssertionError` differently. recommended. Taking the time to set up proper :class:`TestCase` subclasses will make future test refactorings infinitely easier. +In some cases, the existing tests may have been written using the :mod:`doctest` +module. If so, :mod:`doctest` provides a :class:`DocTestSuite` class that can +automatically build :class:`unittest.TestSuite` instances from the existing +:mod:`doctest`\ -based tests. + .. _unittest-skipping: @@ -489,6 +497,13 @@ The following decorators implement test skipping and expected failures: Classes and functions --------------------- +This section describes in depth the API of :mod:`unittest`. + + +.. _testcase-objects: + +Test cases +~~~~~~~~~~ .. class:: TestCase([methodName]) @@ -512,7 +527,191 @@ Classes and functions Here, we create two instances of :class:`WidgetTestCase`, each of which runs a single test. - *methodName* defaults to ``'runTest'``. + *methodName* defaults to :meth:`runTest`. + + :class:`TestCase` instances provide three groups of methods: one group used + to run the test, another used by the test implementation to check conditions + and report failures, and some inquiry methods allowing information about the + test itself to be gathered. + + Methods in the first group (running the test) are: + + + .. method:: setUp() + + Method called to prepare the test fixture. This is called immediately + before calling the test method; any exception raised by this method will + be considered an error rather than a test failure. The default + implementation does nothing. + + + .. method:: tearDown() + + Method called immediately after the test method has been called and the + result recorded. This is called even if the test method raised an + exception, so the implementation in subclasses may need to be particularly + careful about checking internal state. Any exception raised by this + method will be considered an error rather than a test failure. This + method will only be called if the :meth:`setUp` succeeds, regardless of + the outcome of the test method. The default implementation does nothing. + + + .. method:: run([result]) + + Run the test, collecting the result into the test result object passed as + *result*. If *result* is omitted or :const:`None`, a temporary result + object is created (by calling the :meth:`defaultTestCase` method) and + used; this result object is not returned to :meth:`run`'s caller. + + The same effect may be had by simply calling the :class:`TestCase` + instance. + + + .. method:: skip(reason) + + Skips the current test. See :ref:`unittest-skipping` for more + information. + + + .. method:: debug() + + Run the test without collecting the result. This allows exceptions raised + by the test to be propagated to the caller, and can be used to support + running tests under a debugger. + + The test code can use any of the following methods to check for and report + failures. + + + .. method:: assert_(expr[, msg]) + failUnless(expr[, msg]) + assertTrue(expr[, msg]) + + Signal a test failure if *expr* is false; the explanation for the error + will be *msg* if given, otherwise it will be :const:`None`. + + + .. method:: assertEqual(first, second[, msg]) + failUnlessEqual(first, second[, msg]) + + Test that *first* and *second* are equal. If the values do not compare + equal, the test will fail with the explanation given by *msg*, or + :const:`None`. Note that using :meth:`failUnlessEqual` improves upon + doing the comparison as the first parameter to :meth:`failUnless`: the + default value for *msg* can be computed to include representations of both + *first* and *second*. + + + .. method:: assertNotEqual(first, second[, msg]) + failIfEqual(first, second[, msg]) + + Test that *first* and *second* are not equal. If the values do compare + equal, the test will fail with the explanation given by *msg*, or + :const:`None`. Note that using :meth:`failIfEqual` improves upon doing + the comparison as the first parameter to :meth:`failUnless` is that the + default value for *msg* can be computed to include representations of both + *first* and *second*. + + + .. method:: assertAlmostEqual(first, second[, places[, msg]]) + failUnlessAlmostEqual(first, second[, places[, msg]]) + + Test that *first* and *second* are approximately equal by computing the + difference, rounding to the given number of decimal *places* (default 7), + and comparing to zero. + + Note that comparing a given number of decimal places is not the same as + comparing a given number of significant digits. If the values do not + compare equal, the test will fail with the explanation given by *msg*, or + :const:`None`. + + + .. method:: assertNotAlmostEqual(first, second[, places[, msg]]) + failIfAlmostEqual(first, second[, places[, msg]]) + + Test that *first* and *second* are not approximately equal by computing + the difference, rounding to the given number of decimal *places* (default + 7), and comparing to zero. + + Note that comparing a given number of decimal places is not the same as + comparing a given number of significant digits. If the values do not + compare equal, the test will fail with the explanation given by *msg*, or + :const:`None`. + + + .. method:: assertRaises(exception[, callable, ...]) + failUnlessRaises(exception[, callable, ...]) + + Test that an exception is raised when *callable* is called with any + positional or keyword arguments that are also passed to + :meth:`assertRaises`. The test passes if *exception* is raised, is an + error if another exception is raised, or fails if no exception is raised. + To catch any of a group of exceptions, a tuple containing the exception + classes may be passed as *exception*. + + .. versionchanged:: 2.7 + + If *callable* is omitted or None, returns a context manager so that the + code under test can be written inline rather than as a function:: + + with self.failUnlessRaises(some_error_class): + do_something() + + .. method:: failIf(expr[, msg]) + assertFalse(expr[, msg]) + + The inverse of the :meth:`failUnless` method is the :meth:`failIf` method. + This signals a test failure if *expr* is true, with *msg* or :const:`None` + for the error message. + + + .. method:: fail([msg]) + + Signals a test failure unconditionally, with *msg* or :const:`None` for + the error message. + + + .. attribute:: failureException + + This class attribute gives the exception raised by the test method. If a + test framework needs to use a specialized exception, possibly to carry + additional information, it must subclass this exception in order to "play + fair" with the framework. The initial value of this attribute is + :exc:`AssertionError`. + + Testing frameworks can use the following methods to collect information on + the test: + + + .. method:: countTestCases() + + Return the number of tests represented by this test object. For + :class:`TestCase` instances, this will always be ``1``. + + + .. method:: defaultTestResult() + + Return an instance of the test result class that should be used for this + test case class (if no other result instance is provided to the + :meth:`run` method). + + For :class:`TestCase` instances, this will always be an instance of + :class:`TestResult`; subclasses of :class:`TestCase` should override this + as necessary. + + + .. method:: id() + + Return a string identifying the specific test case. This is usually the + full name of the test method, including the module and class name. + + + .. method:: shortDescription() + + Returns a one-line description of the test, or :const:`None` if no + description has been provided. The default implementation of this method + returns the first line of the test method's docstring, if available, or + :const:`None`. .. class:: FunctionTestCase(testFunc[, setUp[, tearDown[, description]]]) @@ -524,6 +723,11 @@ Classes and functions -based test framework. +.. _testsuite-objects: + +Grouping tests +~~~~~~~~~~~~~~ + .. class:: TestSuite([tests]) This class represents an aggregation of individual tests cases and test suites. @@ -535,6 +739,53 @@ Classes and functions test suites that will be used to build the suite initially. Additional methods are provided to add test cases and suites to the collection later on. + :class:`TestSuite` (including :class:`ClassTestSuite`) objects behave much + like :class:`TestCase` objects, except they do not actually implement a test. + Instead, they are used to aggregate tests into groups of tests that should be + run together. Some additional methods are available to add tests to + :class:`TestSuite` instances: + + + .. method:: TestSuite.addTest(test) + + Add a :class:`TestCase` or :class:`TestSuite` to the suite. + + + .. method:: TestSuite.addTests(tests) + + Add all the tests from an iterable of :class:`TestCase` and :class:`TestSuite` + instances to this test suite. + + This is equivalent to iterating over *tests*, calling :meth:`addTest` for each + element. + + :class:`TestSuite` shares the following methods with :class:`TestCase`: + + + .. method:: run(result) + + Run the tests associated with this suite, collecting the result into the + test result object passed as *result*. Note that unlike + :meth:`TestCase.run`, :meth:`TestSuite.run` requires the result object to + be passed in. + + + .. method:: debug() + + Run the tests associated with this suite without collecting the + result. This allows exceptions raised by the test to be propagated to the + caller and can be used to support running tests under a debugger. + + + .. method:: countTestCases() + + Return the number of tests represented by this test object, including all + individual tests and sub-suites. + + In the typical usage of a :class:`TestSuite` object, the :meth:`run` method + is invoked by a :class:`TestRunner` rather than by the end-user test harness. + + .. class:: ClassTestSuite(tests, collected_from) This subclass of :class:`TestSuite` repesents an aggregation of individuals @@ -543,17 +794,278 @@ Classes and functions class they came from. +Loading and running tests +~~~~~~~~~~~~~~~~~~~~~~~~~ + .. class:: TestLoader() - This class is responsible for loading tests according to various criteria and - returning them wrapped in a :class:`TestSuite`. It can load all tests within a - given module or :class:`TestCase` subclass. + The :class:`TestLoader` class is used to create test suites from classes and + modules. Normally, there is no need to create an instance of this class; the + :mod:`unittest` module provides an instance that can be shared as + ``unittest.defaultTestLoader``. Using a subclass or instance, however, allows + customization of some configurable properties. + + :class:`TestLoader` objects have the following methods: -.. class:: TestResult() + .. method:: loadTestsFromTestCase(testCaseClass) - This class is used to compile information about which tests have succeeded and - which have failed. + Return a suite of all tests cases contained in the :class:`TestCase`\ -derived + :class:`testCaseClass`. + + + .. method:: loadTestsFromModule(module) + + Return a suite of all tests cases contained in the given module. This + method searches *module* for classes derived from :class:`TestCase` and + creates an instance of the class for each test method defined for the + class. + + .. warning:: + + While using a hierarchy of :class:`TestCase`\ -derived classes can be + convenient in sharing fixtures and helper functions, defining test + methods on base classes that are not intended to be instantiated + directly does not play well with this method. Doing so, however, can + be useful when the fixtures are different and defined in subclasses. + + + .. method:: loadTestsFromName(name[, module]) + + Return a suite of all tests cases given a string specifier. + + The specifier *name* is a "dotted name" that may resolve either to a + module, a test case class, a test method within a test case class, a + :class:`TestSuite` instance, or a callable object which returns a + :class:`TestCase` or :class:`TestSuite` instance. These checks are + applied in the order listed here; that is, a method on a possible test + case class will be picked up as "a test method within a test case class", + rather than "a callable object". + + For example, if you have a module :mod:`SampleTests` containing a + :class:`TestCase`\ -derived class :class:`SampleTestCase` with three test + methods (:meth:`test_one`, :meth:`test_two`, and :meth:`test_three`), the + specifier ``'SampleTests.SampleTestCase'`` would cause this method to return a + suite which will run all three test methods. Using the specifier + ``'SampleTests.SampleTestCase.test_two'`` would cause it to return a test suite + which will run only the :meth:`test_two` test method. The specifier can refer + to modules and packages which have not been imported; they will be imported as a + side-effect. + + The method optionally resolves *name* relative to the given *module*. + + + .. method:: loadTestsFromNames(names[, module]) + + Similar to :meth:`loadTestsFromName`, but takes a sequence of names rather + than a single name. The return value is a test suite which supports all + the tests defined for each name. + + + .. method:: getTestCaseNames(testCaseClass) + + Return a sorted sequence of method names found within *testCaseClass*; + this should be a subclass of :class:`TestCase`. + + The following attributes of a :class:`TestLoader` can be configured either by + subclassing or assignment on an instance: + + + .. attribute:: testMethodPrefix + + String giving the prefix of method names which will be interpreted as test + methods. The default value is ``'test'``. + + This affects :meth:`getTestCaseNames` and all the :meth:`loadTestsFrom\*` + methods. + + + .. attribute:: sortTestMethodsUsing + + Function to be used to compare method names when sorting them in + :meth:`getTestCaseNames` and all the :meth:`loadTestsFrom\*` methods. The + default value is the built-in :func:`cmp` function; the attribute can also + be set to :const:`None` to disable the sort. + + + .. attribute:: suiteClass + + Callable object that constructs a test suite from a list of tests. No + methods on the resulting object are needed. The default value is the + :class:`TestSuite` class. + + This affects all the :meth:`loadTestsFrom\*` methods. + + + .. attribute:: classSuiteClass + + Callable object that constructs a test suite for the tests cases from one + class. The default value is :class:`ClassTestSuite`. + + +.. class:: TestResult + + This class is used to compile information about which tests have succeeded + and which have failed. + + A :class:`TestResult` object stores the results of a set of tests. The + :class:`TestCase` and :class:`TestSuite` classes ensure that results are + properly recorded; test authors do not need to worry about recording the + outcome of tests. + + Testing frameworks built on top of :mod:`unittest` may want access to the + :class:`TestResult` object generated by running a set of tests for reporting + purposes; a :class:`TestResult` instance is returned by the + :meth:`TestRunner.run` method for this purpose. + + :class:`TestResult` instances have the following attributes that will be of + interest when inspecting the results of running a set of tests: + + + .. attribute:: errors + + A list containing 2-tuples of :class:`TestCase` instances and strings + holding formatted tracebacks. Each tuple represents a test which raised an + unexpected exception. + + .. versionchanged:: 2.2 + + Contains formatted tracebacks instead of :func:`sys.exc_info` results. + + + .. attribute:: failures + + A list containing 2-tuples of :class:`TestCase` instances and strings + holding formatted tracebacks. Each tuple represents a test where a failure + was explicitly signalled using the :meth:`TestCase.fail\*` or + :meth:`TestCase.assert\*` methods. + + .. versionchanged:: 2.2 + + Contains formatted tracebacks instead of :func:`sys.exc_info` results. + + .. attribute:: skipped + + A list containing 2-tuples of :class:`TestCase` instances and strings + holding the reason for skipping the test. + + .. versionadded:: 2.7 + + .. attribute:: expectedFailures + + A list contaning 2-tuples of :class:`TestCase` instances and strings + holding formatted tracebacks. Each tuple represents a expected failures + of the test case. + + .. attribute:: unexpectedSuccesses + + A list containing :class:`TestCase` instances that were marked as expected + failures, but succeeded. + + .. attribute:: shouldStop + + Set to ``True`` when the execution of tests should stop by :meth:`stop`. + + + .. attribute:: testsRun + + The total number of tests run so far. + + + .. method:: wasSuccessful() + + Return :const:`True` if all tests run so far have passed, otherwise returns + :const:`False`. + + + .. method:: stop() + + This method can be called to signal that the set of tests being run should + be aborted by setting the :attr:`shouldStop` attribute to :const:`True`. + :class:`TestRunner` objects should respect this flag and return without + running any additional tests. + + For example, this feature is used by the :class:`TextTestRunner` class to + stop the test framework when the user signals an interrupt from the + keyboard. Interactive tools which provide :class:`TestRunner` + implementations can use this in a similar manner. + + The following methods of the :class:`TestResult` class are used to maintain + the internal data structures, and may be extended in subclasses to support + additional reporting requirements. This is particularly useful in building + tools which support interactive reporting while tests are being run. + + + .. method:: startTest(test) + + Called when the test case *test* is about to be run. + + The default implementation simply increments the instance's :attr:`testsRun` + counter. + + + .. method:: stopTest(test) + + Called after the test case *test* has been executed, regardless of the + outcome. + + The default implementation does nothing. + + + .. method:: addError(test, err) + + Called when the test case *test* raises an unexpected exception *err* is a + tuple of the form returned by :func:`sys.exc_info`: ``(type, value, + traceback)``. + + The default implementation appends a tuple ``(test, formatted_err)`` to + the instance's :attr:`errors` attribute, where *formatted_err* is a + formatted traceback derived from *err*. + + + .. method:: addFailure(test, err) + + Called when the test case *test* signals a failure. *err* is a tuple of the form + returned by :func:`sys.exc_info`: ``(type, value, traceback)``. + + The default implementation appends a tuple ``(test, formatted_err)`` to + the instance's :attr:`failures` attribute, where *formatted_err* is a + formatted traceback derived from *err*. + + + .. method:: addSuccess(test) + + Called when the test case *test* succeeds. + + The default implementation does nothing. + + + .. method:: addSkip(test, reason) + + Called when the test case *test* is skipped. *reason* is the reason the + test gave for skipping. + + The default implementation appends a tuple ``(test, reason)`` to the + instance's :attr:`skipped` attribute. + + + .. method:: addExpectedFailure(test, err) + + Called when the test case *test* fails, but was marked with the + :func:`expectedFailure` decorator. + + The default implementation appends a tuple ``(test, formatted_err)`` to + the instance's :attr:`expectedFailures` attribute, where *formatted_err* + is a formatted traceback derived from *err*. + + + .. method:: addUnexpectedSuccess(test) + + Called when the test case *test* was marked with the + :func:`expectedFailure` decorator, but succeeded. + + The default implementation appends the test to the instance's + :attr:`unexpectedSuccesses` attribute. .. data:: defaultTestLoader @@ -581,491 +1093,3 @@ Classes and functions The *testRunner* argument can either be a test runner class or an already created instance of it. - -In some cases, the existing tests may have been written using the :mod:`doctest` -module. If so, that module provides a :class:`DocTestSuite` class that can -automatically build :class:`unittest.TestSuite` instances from the existing -:mod:`doctest`\ -based tests. - -.. versionadded:: 2.3 - - -.. _testcase-objects: - -TestCase Objects ----------------- - -Each :class:`TestCase` instance represents a single test, but each concrete -subclass may be used to define multiple tests --- the concrete class represents -a single test fixture. The fixture is created and cleaned up for each test -case. - -:class:`TestCase` instances provide three groups of methods: one group used to -run the test, another used by the test implementation to check conditions and -report failures, and some inquiry methods allowing information about the test -itself to be gathered. - -Methods in the first group (running the test) are: - - -.. method:: TestCase.setUp() - - Method called to prepare the test fixture. This is called immediately before - calling the test method; any exception raised by this method will be considered - an error rather than a test failure. The default implementation does nothing. - - -.. method:: TestCase.tearDown() - - Method called immediately after the test method has been called and the result - recorded. This is called even if the test method raised an exception, so the - implementation in subclasses may need to be particularly careful about checking - internal state. Any exception raised by this method will be considered an error - rather than a test failure. This method will only be called if the - :meth:`setUp` succeeds, regardless of the outcome of the test method. The - default implementation does nothing. - - -.. method:: TestCase.run([result]) - - Run the test, collecting the result into the test result object passed as - *result*. If *result* is omitted or :const:`None`, a temporary result object is - created (by calling the :meth:`defaultTestCase` method) and used; this result - object is not returned to :meth:`run`'s caller. - - The same effect may be had by simply calling the :class:`TestCase` instance. - - -.. method:: TestCase.skip(reason) - - Skips the current test. See :ref:`unittest-skipping`. - - -.. method:: TestCase.debug() - - Run the test without collecting the result. This allows exceptions raised by - the test to be propagated to the caller, and can be used to support running - tests under a debugger. - -The test code can use any of the following methods to check for and report -failures. - - -.. method:: TestCase.assert_(expr[, msg]) - TestCase.failUnless(expr[, msg]) - TestCase.assertTrue(expr[, msg]) - - Signal a test failure if *expr* is false; the explanation for the error will be - *msg* if given, otherwise it will be :const:`None`. - - -.. method:: TestCase.assertEqual(first, second[, msg]) - TestCase.failUnlessEqual(first, second[, msg]) - - Test that *first* and *second* are equal. If the values do not compare equal, - the test will fail with the explanation given by *msg*, or :const:`None`. Note - that using :meth:`failUnlessEqual` improves upon doing the comparison as the - first parameter to :meth:`failUnless`: the default value for *msg* can be - computed to include representations of both *first* and *second*. - - -.. method:: TestCase.assertNotEqual(first, second[, msg]) - TestCase.failIfEqual(first, second[, msg]) - - Test that *first* and *second* are not equal. If the values do compare equal, - the test will fail with the explanation given by *msg*, or :const:`None`. Note - that using :meth:`failIfEqual` improves upon doing the comparison as the first - parameter to :meth:`failUnless` is that the default value for *msg* can be - computed to include representations of both *first* and *second*. - - -.. method:: TestCase.assertAlmostEqual(first, second[, places[, msg]]) - TestCase.failUnlessAlmostEqual(first, second[, places[, msg]]) - - Test that *first* and *second* are approximately equal by computing the - difference, rounding to the given number of decimal *places* (default 7), - and comparing to zero. - Note that comparing a given number of decimal places is not the same as - comparing a given number of significant digits. If the values do not compare - equal, the test will fail with the explanation given by *msg*, or :const:`None`. - - -.. method:: TestCase.assertNotAlmostEqual(first, second[, places[, msg]]) - TestCase.failIfAlmostEqual(first, second[, places[, msg]]) - - Test that *first* and *second* are not approximately equal by computing the - difference, rounding to the given number of decimal *places* (default 7), - and comparing to zero. - Note that comparing a given number of decimal places is not the same as - comparing a given number of significant digits. If the values do not compare - equal, the test will fail with the explanation given by *msg*, or :const:`None`. - - -.. method:: TestCase.assertRaises(exception[, callable, ...]) - TestCase.failUnlessRaises(exception[, callable, ...]) - - Test that an exception is raised when *callable* is called with any positional - or keyword arguments that are also passed to :meth:`assertRaises`. The test - passes if *exception* is raised, is an error if another exception is raised, or - fails if no exception is raised. To catch any of a group of exceptions, a tuple - containing the exception classes may be passed as *exception*. - - .. versionchanged:: 2.7 - - If *callable* is omitted or None, returns a context manager so that the code - under test can be written inline rather than as a function:: - - with self.failUnlessRaises(some_error_class): - do_something() - -.. method:: TestCase.failIf(expr[, msg]) - TestCase.assertFalse(expr[, msg]) - - The inverse of the :meth:`failUnless` method is the :meth:`failIf` method. This - signals a test failure if *expr* is true, with *msg* or :const:`None` for the - error message. - - -.. method:: TestCase.fail([msg]) - - Signals a test failure unconditionally, with *msg* or :const:`None` for the - error message. - - -.. attribute:: TestCase.failureException - - This class attribute gives the exception raised by the :meth:`test` method. If - a test framework needs to use a specialized exception, possibly to carry - additional information, it must subclass this exception in order to "play fair" - with the framework. The initial value of this attribute is - :exc:`AssertionError`. - -Testing frameworks can use the following methods to collect information on the -test: - - -.. method:: TestCase.countTestCases() - - Return the number of tests represented by this test object. For - :class:`TestCase` instances, this will always be ``1``. - - -.. method:: TestCase.defaultTestResult() - - Return an instance of the test result class that should be used for this test - case class (if no other result instance is provided to the :meth:`run` method). - - For :class:`TestCase` instances, this will always be an instance of - :class:`TestResult`; subclasses of :class:`TestCase` should override this as - necessary. - - -.. method:: TestCase.id() - - Return a string identifying the specific test case. This is usually the full - name of the test method, including the module and class name. - - -.. method:: TestCase.shortDescription() - - Returns a one-line description of the test, or :const:`None` if no description - has been provided. The default implementation of this method returns the first - line of the test method's docstring, if available, or :const:`None`. - - -.. _testsuite-objects: - -TestSuite Objects ------------------ - -:class:`TestSuite` (including :class:`ClassTestSuite`) objects behave much like -:class:`TestCase` objects, except they do not actually implement a test. -Instead, they are used to aggregate tests into groups of tests that should be -run together. Some additional methods are available to add tests to -:class:`TestSuite` instances: - - -.. method:: TestSuite.addTest(test) - - Add a :class:`TestCase` or :class:`TestSuite` to the suite. - - -.. method:: TestSuite.addTests(tests) - - Add all the tests from an iterable of :class:`TestCase` and :class:`TestSuite` - instances to this test suite. - - This is equivalent to iterating over *tests*, calling :meth:`addTest` for each - element. - -:class:`TestSuite` shares the following methods with :class:`TestCase`: - - -.. method:: TestSuite.run(result) - - Run the tests associated with this suite, collecting the result into the test - result object passed as *result*. Note that unlike :meth:`TestCase.run`, - :meth:`TestSuite.run` requires the result object to be passed in. - - -.. method:: TestSuite.debug() - - Run the tests associated with this suite without collecting the result. This - allows exceptions raised by the test to be propagated to the caller and can be - used to support running tests under a debugger. - - -.. method:: TestSuite.countTestCases() - - Return the number of tests represented by this test object, including all - individual tests and sub-suites. - -In the typical usage of a :class:`TestSuite` object, the :meth:`run` method is -invoked by a :class:`TestRunner` rather than by the end-user test harness. - - -.. _testresult-objects: - -TestResult Objects ------------------- - -A :class:`TestResult` object stores the results of a set of tests. The -:class:`TestCase` and :class:`TestSuite` classes ensure that results are -properly recorded; test authors do not need to worry about recording the outcome -of tests. - -Testing frameworks built on top of :mod:`unittest` may want access to the -:class:`TestResult` object generated by running a set of tests for reporting -purposes; a :class:`TestResult` instance is returned by the -:meth:`TestRunner.run` method for this purpose. - -:class:`TestResult` instances have the following attributes that will be of -interest when inspecting the results of running a set of tests: - - -.. attribute:: TestResult.errors - - A list containing 2-tuples of :class:`TestCase` instances and strings holding - formatted tracebacks. Each tuple represents a test which raised an unexpected - exception. - - .. versionchanged:: 2.2 - Contains formatted tracebacks instead of :func:`sys.exc_info` results. - - -.. attribute:: TestResult.failures - - A list containing 2-tuples of :class:`TestCase` instances and strings holding - formatted tracebacks. Each tuple represents a test where a failure was - explicitly signalled using the :meth:`TestCase.fail\*` or - :meth:`TestCase.assert\*` methods. - - .. versionchanged:: 2.2 - Contains formatted tracebacks instead of :func:`sys.exc_info` results. - - -.. attribute:: TestResult.testsRun - - The total number of tests run so far. - - -.. method:: TestResult.wasSuccessful() - - Returns :const:`True` if all tests run so far have passed, otherwise returns - :const:`False`. - - -.. method:: TestResult.stop() - - This method can be called to signal that the set of tests being run should be - aborted by setting the :class:`TestResult`'s ``shouldStop`` attribute to - :const:`True`. :class:`TestRunner` objects should respect this flag and return - without running any additional tests. - - For example, this feature is used by the :class:`TextTestRunner` class to stop - the test framework when the user signals an interrupt from the keyboard. - Interactive tools which provide :class:`TestRunner` implementations can use this - in a similar manner. - -The following methods of the :class:`TestResult` class are used to maintain the -internal data structures, and may be extended in subclasses to support -additional reporting requirements. This is particularly useful in building -tools which support interactive reporting while tests are being run. - - -.. method:: TestResult.startTest(test) - - Called when the test case *test* is about to be run. - - The default implementation simply increments the instance's ``testsRun`` - counter. - - -.. method:: TestResult.stopTest(test) - - Called after the test case *test* has been executed, regardless of the outcome. - - The default implementation does nothing. - - -.. method:: TestResult.addError(test, err) - - Called when the test case *test* raises an unexpected exception *err* is a tuple - of the form returned by :func:`sys.exc_info`: ``(type, value, traceback)``. - - The default implementation appends a tuple ``(test, formatted_err)`` to the - instance's ``errors`` attribute, where *formatted_err* is a formatted - traceback derived from *err*. - - -.. method:: TestResult.addFailure(test, err) - - Called when the test case *test* signals a failure. *err* is a tuple of the form - returned by :func:`sys.exc_info`: ``(type, value, traceback)``. - - The default implementation appends a tuple ``(test, formatted_err)`` to the - instance's ``failures`` attribute, where *formatted_err* is a formatted - traceback derived from *err*. - - -.. method:: TestResult.addSuccess(test) - - Called when the test case *test* succeeds. - - The default implementation does nothing. - - -.. method:: TestResult.addSkip(test, reason) - - Called when the test case *test* is skipped. *reason* is the reason the test - gave for skipping. - - The default implementation appends a tuple ``(test, reason)`` to the - instance's ``skipped`` attribute. - - -.. method:: TestResult.addExpectedFailure(test, err) - - Called when the test case *test* fails, but was marked with the - :func:`expectedFailure` decorator. - - The default implementation appends a tuple ``(test, formatted_err)`` to the - instance's ``expectedFailures`` attribute, where *formatted_err* is a - formatted traceback derived from *err*. - - -.. method:: TestResult.addUnexpectedSuccess(test) - - Called when the test case *test* was marked with the :func:`expectedFailure` - decorator, but succeeded. - - The default implementation appends the test to the instance's - ``unexpectedSuccesses`` attribute. - - -.. _testloader-objects: - -TestLoader Objects ------------------- - -The :class:`TestLoader` class is used to create test suites from classes and -modules. Normally, there is no need to create an instance of this class; the -:mod:`unittest` module provides an instance that can be shared as -``unittest.defaultTestLoader``. Using a subclass or instance, however, allows -customization of some configurable properties. - -:class:`TestLoader` objects have the following methods: - - -.. method:: TestLoader.loadTestsFromTestCase(testCaseClass) - - Return a suite of all tests cases contained in the :class:`TestCase`\ -derived - :class:`testCaseClass`. - - -.. method:: TestLoader.loadTestsFromModule(module) - - Return a suite of all tests cases contained in the given module. This method - searches *module* for classes derived from :class:`TestCase` and creates an - instance of the class for each test method defined for the class. - - .. warning:: - - While using a hierarchy of :class:`TestCase`\ -derived classes can be convenient - in sharing fixtures and helper functions, defining test methods on base classes - that are not intended to be instantiated directly does not play well with this - method. Doing so, however, can be useful when the fixtures are different and - defined in subclasses. - - -.. method:: TestLoader.loadTestsFromName(name[, module]) - - Return a suite of all tests cases given a string specifier. - - The specifier *name* is a "dotted name" that may resolve either to a module, a - test case class, a test method within a test case class, a :class:`TestSuite` - instance, or a callable object which returns a :class:`TestCase` or - :class:`TestSuite` instance. These checks are applied in the order listed here; - that is, a method on a possible test case class will be picked up as "a test - method within a test case class", rather than "a callable object". - - For example, if you have a module :mod:`SampleTests` containing a - :class:`TestCase`\ -derived class :class:`SampleTestCase` with three test - methods (:meth:`test_one`, :meth:`test_two`, and :meth:`test_three`), the - specifier ``'SampleTests.SampleTestCase'`` would cause this method to return a - suite which will run all three test methods. Using the specifier - ``'SampleTests.SampleTestCase.test_two'`` would cause it to return a test suite - which will run only the :meth:`test_two` test method. The specifier can refer - to modules and packages which have not been imported; they will be imported as a - side-effect. - - The method optionally resolves *name* relative to the given *module*. - - -.. method:: TestLoader.loadTestsFromNames(names[, module]) - - Similar to :meth:`loadTestsFromName`, but takes a sequence of names rather than - a single name. The return value is a test suite which supports all the tests - defined for each name. - - -.. method:: TestLoader.getTestCaseNames(testCaseClass) - - Return a sorted sequence of method names found within *testCaseClass*; this - should be a subclass of :class:`TestCase`. - -The following attributes of a :class:`TestLoader` can be configured either by -subclassing or assignment on an instance: - - -.. attribute:: TestLoader.testMethodPrefix - - String giving the prefix of method names which will be interpreted as test - methods. The default value is ``'test'``. - - This affects :meth:`getTestCaseNames` and all the :meth:`loadTestsFrom\*` - methods. - - -.. attribute:: TestLoader.sortTestMethodsUsing - - Function to be used to compare method names when sorting them in - :meth:`getTestCaseNames` and all the :meth:`loadTestsFrom\*` methods. The - default value is the built-in :func:`cmp` function; the attribute can also be - set to :const:`None` to disable the sort. - - -.. attribute:: TestLoader.suiteClass - - Callable object that constructs a test suite from a list of tests. No methods on - the resulting object are needed. The default value is the :class:`TestSuite` - class. - - This affects all the :meth:`loadTestsFrom\*` methods. - - -.. attribute:: TestLoader.classSuiteClass - - Callable object that constructs a test suite for the tests cases from one - class. The default value is :class:`ClassTestSuite`. -