Writing Python Test Cases
-------------------------
Skip Montanaro
If you add a new module to Python or modify the functionality of an existing
module, it is your responsibility to write one or more test cases to test
that new functionality. The mechanics of the test system are fairly
straightforward. If you are writing test cases for module zyzzyx, you need
to create a file in .../Lib/test named test_zyzzyx.py and an expected output
file in .../Lib/test/output named test_zyzzyx ("..." represents the
top-level directory in the Python source tree, the directory containing the
configure script). Generate the initial version of the test output file by
executing:
cd .../Lib/test
python regrtest.py -g test_zyzzyx.py
Any time you modify test_zyzzyx.py you need to generate a new expected
output file. Don't forget to desk check the generated output to make sure
it's really what you expected to find! To run a single test after modifying
a module, simply run regrtest.py without the -g flag:
cd .../Lib/test
python regrtest.py test_zyzzyx.py
To run the entire test suite, make the "test" target at the top level:
cd ...
make test
Test cases generate output based upon computed values and branches taken in
the code. When executed, regrtest.py compares the actual output generated
by executing the test case with the expected output and reports success or
failure. It stands to reason that if the actual and expected outputs are to
match, they must not contain any machine dependencies. This means
your test cases should not print out absolute machine addresses or floating
point numbers with large numbers of significant digits.
Writing good test cases is a skilled task and is too complex to discuss in
detail in this short document. Many books have been written on the subject.
I'll show my age by suggesting that Glenford Myers' "The Art of Software
Testing", published in 1979, is still the best introduction to the subject
available. It is short (177 pages), easy to read, and discusses the major
elements of software testing, though its publication predates the
object-oriented software revolution, so doesn't cover that subject at all.
Unfortunately, it is very expensive (about $100 new). If you can borrow it
or find it used (around $20), I strongly urge you to pick up a copy.
As an author of at least part of a module, you will be writing unit tests
(isolated tests of functions and objects defined by the module) using white
box techniques. (Unlike black box testing, where you only have the external
interfaces to guide your test case writing, in white box testing you can see
the code being tested and tailor your test cases to exercise it more
completely).
The most important goal when writing test cases is to break things. A test
case that doesn't uncover a bug is less valuable than one that does. In
designing test cases you should pay attention to the following:
1. Your test cases should exercise all the functions and objects defined
in the module, not just the ones meant to be called by users of your
module. This may require you to write test code that uses the module
in ways you don't expect (explicitly calling internal functions, for
example - see test_atexit.py).
2. You should consider any boundary values that may tickle exceptional
conditions (e.g. if you were testing a division module you might well
want to generate tests with numerators and denominators at the limits
of floating point and integer numbers on the machine performing the
tests as well as a denominator of zero).
3. You should exercise as many paths through the code as possible. This
may not always be possible, but is a goal to strive for. In
particular, when considering if statements (or their equivalent), you
want to create test cases that exercise both the true and false
branches. For while and for statements, you should create test cases
that exercise the loop zero, one and multiple times.
d6d06ade26