mirror of https://github.com/python/cpython
restructured a bit and added some more content...
This commit is contained in:
Skip Montanaro
parent
4d06923122
commit
e9e5dcd4db
185
Lib/test/README
185
Lib/test/README
|
|
@ -1,40 +1,73 @@
|
|||
Writing Python Test Cases
|
||||
-------------------------
|
||||
Writing Python Regression Tests
|
||||
-------------------------------
|
||||
Skip Montanaro
|
||||
(skip@mojam.com)
|
||||
|
||||
|
||||
Introduction
|
||||
|
||||
If you add a new module to Python or modify the functionality of an existing
|
||||
module, it is your responsibility to write one or more test cases to test
|
||||
that new functionality. The mechanics of the test system are fairly
|
||||
straightforward. If you are writing test cases for module zyzzyx, you need
|
||||
to create a file in .../Lib/test named test_zyzzyx.py and an expected output
|
||||
file in .../Lib/test/output named test_zyzzyx ("..." represents the
|
||||
top-level directory in the Python source tree, the directory containing the
|
||||
configure script). Generate the initial version of the test output file by
|
||||
executing:
|
||||
module, you should write one or more test cases to exercise that new
|
||||
functionality. The mechanics of how the test system operates are fairly
|
||||
straightforward. When a test case is run, the output is compared with the
|
||||
expected output that is stored in .../Lib/test/output. If the test runs to
|
||||
completion and the actual and expected outputs match, the test succeeds, if
|
||||
not, it fails. If an ImportError is raised, the test is not run.
|
||||
|
||||
cd .../Lib/test
|
||||
python regrtest.py -g test_zyzzyx.py
|
||||
You will be writing unit tests (isolated tests of functions and objects
|
||||
defined by the module) using white box techniques. Unlike black box
|
||||
testing, where you only have the external interfaces to guide your test case
|
||||
writing, in white box testing you can see the code being tested and tailor
|
||||
your test cases to exercise it more completely. In particular, you will be
|
||||
able to refer to the C and Python code in the CVS repository when writing
|
||||
your regression test cases.
|
||||
|
||||
Any time you modify test_zyzzyx.py you need to generate a new expected
|
||||
|
||||
Executing Test Cases
|
||||
|
||||
If you are writing test cases for module spam, you need to create a file
|
||||
in .../Lib/test named test_spam.py and an expected output file in
|
||||
.../Lib/test/output named test_spam ("..." represents the top-level
|
||||
directory in the Python source tree, the directory containing the configure
|
||||
script). From the top-level directory, generate the initial version of the
|
||||
test output file by executing:
|
||||
|
||||
./python Lib/test/regrtest.py -g test_spam.py
|
||||
|
||||
Any time you modify test_spam.py you need to generate a new expected
|
||||
output file. Don't forget to desk check the generated output to make sure
|
||||
it's really what you expected to find! To run a single test after modifying
|
||||
a module, simply run regrtest.py without the -g flag:
|
||||
|
||||
cd .../Lib/test
|
||||
python regrtest.py test_zyzzyx.py
|
||||
./python Lib/test/regrtest.py test_spam.py
|
||||
|
||||
While debugging a regression test, you can of course execute it
|
||||
independently of the regression testing framework and see what it prints:
|
||||
|
||||
./python Lib/test/test_spam.py
|
||||
|
||||
To run the entire test suite, make the "test" target at the top level:
|
||||
|
||||
cd ...
|
||||
make test
|
||||
|
||||
Test cases generate output based upon computed values and branches taken in
|
||||
the code. When executed, regrtest.py compares the actual output generated
|
||||
by executing the test case with the expected output and reports success or
|
||||
failure. It stands to reason that if the actual and expected outputs are to
|
||||
match, they must not contain any machine dependencies. This means
|
||||
your test cases should not print out absolute machine addresses or floating
|
||||
point numbers with large numbers of significant digits.
|
||||
On non-Unix platforms where make may not be available, you can simply
|
||||
execute the two runs of regrtest (optimized and non-optimized) directly:
|
||||
|
||||
./python Lib/test/regrtest.py
|
||||
./python -O Lib/test/regrtest.py
|
||||
|
||||
|
||||
Test cases generate output based upon values computed by the test code.
|
||||
When executed, regrtest.py compares the actual output generated by executing
|
||||
the test case with the expected output and reports success or failure. It
|
||||
stands to reason that if the actual and expected outputs are to match, they
|
||||
must not contain any machine dependencies. This means your test cases
|
||||
should not print out absolute machine addresses (e.g. the return value of
|
||||
the id() builtin function) or floating point numbers with large numbers of
|
||||
significant digits (unless you understand what you are doing!).
|
||||
|
||||
|
||||
Test Case Writing Tips
|
||||
|
||||
Writing good test cases is a skilled task and is too complex to discuss in
|
||||
detail in this short document. Many books have been written on the subject.
|
||||
|
|
@ -46,32 +79,88 @@ object-oriented software revolution, so doesn't cover that subject at all.
|
|||
Unfortunately, it is very expensive (about $100 new). If you can borrow it
|
||||
or find it used (around $20), I strongly urge you to pick up a copy.
|
||||
|
||||
As an author of at least part of a module, you will be writing unit tests
|
||||
(isolated tests of functions and objects defined by the module) using white
|
||||
box techniques. (Unlike black box testing, where you only have the external
|
||||
interfaces to guide your test case writing, in white box testing you can see
|
||||
the code being tested and tailor your test cases to exercise it more
|
||||
completely).
|
||||
|
||||
The most important goal when writing test cases is to break things. A test
|
||||
case that doesn't uncover a bug is less valuable than one that does. In
|
||||
designing test cases you should pay attention to the following:
|
||||
case that doesn't uncover a bug is much less valuable than one that does.
|
||||
In designing test cases you should pay attention to the following:
|
||||
|
||||
1. Your test cases should exercise all the functions and objects defined
|
||||
in the module, not just the ones meant to be called by users of your
|
||||
module. This may require you to write test code that uses the module
|
||||
in ways you don't expect (explicitly calling internal functions, for
|
||||
example - see test_atexit.py).
|
||||
* Your test cases should exercise all the functions and objects defined
|
||||
in the module, not just the ones meant to be called by users of your
|
||||
module. This may require you to write test code that uses the module
|
||||
in ways you don't expect (explicitly calling internal functions, for
|
||||
example - see test_atexit.py).
|
||||
|
||||
2. You should consider any boundary values that may tickle exceptional
|
||||
conditions (e.g. if you were testing a division module you might well
|
||||
want to generate tests with numerators and denominators at the limits
|
||||
of floating point and integer numbers on the machine performing the
|
||||
tests as well as a denominator of zero).
|
||||
* You should consider any boundary values that may tickle exceptional
|
||||
conditions (e.g. if you were writing regression tests for division,
|
||||
you might well want to generate tests with numerators and denominators
|
||||
at the limits of floating point and integer numbers on the machine
|
||||
performing the tests as well as a denominator of zero).
|
||||
|
||||
3. You should exercise as many paths through the code as possible. This
|
||||
may not always be possible, but is a goal to strive for. In
|
||||
particular, when considering if statements (or their equivalent), you
|
||||
want to create test cases that exercise both the true and false
|
||||
branches. For while and for statements, you should create test cases
|
||||
that exercise the loop zero, one and multiple times.
|
||||
* You should exercise as many paths through the code as possible. This
|
||||
may not always be possible, but is a goal to strive for. In
|
||||
particular, when considering if statements (or their equivalent), you
|
||||
want to create test cases that exercise both the true and false
|
||||
branches. For loops, you should create test cases that exercise the
|
||||
loop zero, one and multiple times.
|
||||
|
||||
* You should test with obviously invalid input. If you know that a
|
||||
function requires an integer input, try calling it with other types of
|
||||
objects to see how it responds.
|
||||
|
||||
* You should test with obviously out-of-range input. If the domain of a
|
||||
function is only defined for positive integers, try calling it with a
|
||||
negative integer.
|
||||
|
||||
* If you are going to fix a bug that wasn't uncovered by an existing
|
||||
test, try to write a test case that exposes the bug (preferably before
|
||||
fixing it).
|
||||
|
||||
|
||||
Regression Test Writing Rules
|
||||
|
||||
Each test case is different. There is no "standard" form for a Python
|
||||
regression test case, though there are some general rules:
|
||||
|
||||
* If your test case detects a failure, raise TestFailed (found in
|
||||
test_support).
|
||||
|
||||
* Import everything you'll need as early as possible.
|
||||
|
||||
* If you'll be importing objects from a module that is at least
|
||||
partially platform-dependent, only import those objects you need for
|
||||
the current test case to avoid spurious ImportError exceptions that
|
||||
prevent the test from running to completion.
|
||||
|
||||
* Print all your test case results using the print statement. For
|
||||
non-fatal errors, print an error message (or omit a successful
|
||||
completion print) to indicate the failure, but proceed instead of
|
||||
raising TestFailed.
|
||||
|
||||
|
||||
Miscellaneous
|
||||
|
||||
There is a test_support module you can import from your test case. It
|
||||
provides the following useful objects:
|
||||
|
||||
* TestFailed - raise this exception when your regression test detects a
|
||||
failure.
|
||||
|
||||
* findfile(file) - you can call this function to locate a file somewhere
|
||||
along sys.path or in the Lib/test tree - see test_linuxaudiodev.py for
|
||||
an example of its use.
|
||||
|
||||
* verbose - you can use this variable to control print output. Many
|
||||
modules use it. Search for "verbose" in the test_*.py files to see
|
||||
lots of examples.
|
||||
|
||||
* fcmp(x,y) - you can call this function to compare two floating point
|
||||
numbers when you expect them to only be approximately equal withing a
|
||||
fuzz factor (test_support.FUZZ, which defaults to 1e-6).
|
||||
|
||||
Python and C statement coverage results are currently available at
|
||||
|
||||
http://www.musi-cal.com/~skip/python/Python/dist/src/
|
||||
|
||||
As of this writing (July, 2000) these results are being generated nightly.
|
||||
You can refer to the summaries and the test coverage output files to see
|
||||
where coverage is adequate or lacking and write test cases to beef up the
|
||||
coverage.
|
||||
|
|
|
|||
Loading…
Reference in New Issue