cpython/Lib/test
Andrew M. Kuchling cdec8c746b Test case for Cookie.py 2000-08-19 15:21:12 +00:00
..
output Test case for Cookie.py 2000-08-19 15:21:12 +00:00
README Raise 'TestSkipped' (from the test_support) module rather than 'ImportError' 2000-08-04 13:34:43 +00:00
__init__.py Add __init__.py to test package. 1997-11-06 15:41:23 +00:00
audiotest.au The encoding type was wrong, I think. 1999-02-05 20:57:44 +00:00
autotest.py Restore old behavior of autotest and testall, using regrtest. 1997-03-07 21:05:43 +00:00
greyrgb.uue Use uuencoded test images. 1997-04-16 00:30:45 +00:00
pystone.py Separate out a function pystones(loops=LOOPS) which runs the benchmark 1997-11-06 15:45:05 +00:00
re_tests.py -- enabled some temporarily disabled RE tests 2000-08-08 17:06:53 +00:00
regex_tests.py 1.5a3 prerelease 1 from AMK 1997-08-13 22:34:14 +00:00
regrtest.py When a KeyboardInterrupt is caught, just use the "raise" syntax to 2000-08-18 16:04:05 +00:00
reperf.py Measure performance of sub(), split(), findall(). 1998-07-17 21:10:42 +00:00
sortperf.py Add Tim's worst case scenario. 1998-05-26 15:05:12 +00:00
string_tests.py small updates to string_join: 2000-07-11 20:55:38 +00:00
test.xml Added minidom tests. 2000-07-01 14:54:16 +00:00
test_MimeWriter.py Untabified -- to satisfy the tab nanny. 1998-06-09 19:19:40 +00:00
test_al.py typos fixed by Rob Hooft 2000-06-28 14:48:01 +00:00
test_array.py added count, extend, index, pop and remove to arraymodule 2000-07-31 20:52:21 +00:00
test_atexit.py Added the atexit module and documentation from Skip Montanaro 2000-06-28 15:07:31 +00:00
test_audioop.py Mass check-in after untabifying all files that need it. 1998-03-26 19:42:58 +00:00
test_b1.py Break the cycles after testing cmp() on cyclic objects. 2000-07-14 04:32:09 +00:00
test_b2.py Added testsuite for new zip() builtin. 2000-08-03 15:48:07 +00:00
test_binascii.py tests for binascii.b2a_hex() and binascii.a2b_hex(). 2000-08-15 06:08:31 +00:00
test_binhex.py Raise 'TestSkipped' (from the test_support) module rather than 'ImportError' 2000-08-04 13:34:43 +00:00
test_bsddb.py Remove the temp file when we're done. 1999-03-24 19:03:01 +00:00
test_builtin.py Initial revision 1992-01-27 17:00:37 +00:00
test_cd.py typos fixed by Rob Hooft 2000-06-28 14:48:01 +00:00
test_cgi.py test the non-multipart sections of the cgi module 2000-08-03 22:11:43 +00:00
test_cl.py typos fixed by Rob Hooft 2000-06-28 14:48:01 +00:00
test_class.py Apply SF patch #101029: call __getitem__ with a proper slice object if there 2000-08-17 22:37:32 +00:00
test_cmath.py Mass check-in after untabifying all files that need it. 1998-03-26 19:42:58 +00:00
test_compile.py added test case for fixed duplicate arguments bug in Python/compile.c 2000-07-25 22:15:45 +00:00
test_contains.py Deviant1 didn't work as advertised 2000-04-27 21:42:48 +00:00
test_cookie.py Test case for Cookie.py 2000-08-19 15:21:12 +00:00
test_cpickle.py Charles Waldman writes: 2000-04-21 20:49:58 +00:00
test_crypt.py Many scripts, but small changes. Update the way the scripts obtain the 1996-12-20 22:36:52 +00:00
test_dbm.py Mass check-in after untabifying all files that need it. 1998-03-26 19:42:58 +00:00
test_dl.py Raise 'TestSkipped' (from the test_support) module rather than 'ImportError' 2000-08-04 13:34:43 +00:00
test_errno.py Mass check-in after untabifying all files that need it. 1998-03-26 19:42:58 +00:00
test_exceptions.py Create two new exceptions: IndentationError and TabError. These are 2000-07-11 17:53:00 +00:00
test_extcall.py Break a cycle created in the saboteur() function. 2000-07-15 00:42:09 +00:00
test_fcntl.py Daniel Neri: OpenBSD is just as BSD'ish as the other BSD's ;-) 1999-04-19 17:22:12 +00:00
test_fork1.py Chris Herborth <chrish@pobox.com>: 2000-08-15 18:52:33 +00:00
test_format.py Marc-Andre Lemburg <mal@lemburg.com>: 2000-06-30 10:26:29 +00:00
test_gc.py Neil Schemenauer: GC enable(), disable(), isenabled() interface. 2000-08-06 22:45:31 +00:00
test_gdbm.py Added test case for accessing gsbm database by key after it's closed; 2000-02-07 17:15:48 +00:00
test_gl.py Raise 'TestSkipped' (from the test_support) module rather than 'ImportError' 2000-08-04 13:34:43 +00:00
test_grammar.py list comprehensions. see 2000-08-12 18:09:51 +00:00
test_grp.py Mass check-in after untabifying all files that need it. 1998-03-26 19:42:58 +00:00
test_gzip.py Exercise .readline() and .readlines(). More data is written to the 2000-07-29 20:18:34 +00:00
test_hash.py This patch addresses two main issues: (1) There exist some non-fatal 2000-06-29 19:17:04 +00:00
test_imageop.py Spelling fixes supplied by Rob W. W. Hooft. All these are fixes in either 2000-07-16 12:04:32 +00:00
test_imgfile.py Move unified findfile() into test_support.py 1998-04-23 20:13:30 +00:00
test_largefile.py Add largefile support for Linux64 and WIn64. Add test_largefile and some minor 2000-08-11 19:02:59 +00:00
test_linuxaudiodev.py Oooopsss.....tab and space mismatch corrected. 2000-08-04 15:25:58 +00:00
test_long.py Christian Tismer: added test to ensure that multiplication commutes. 2000-04-10 17:41:37 +00:00
test_longexp.py verify that Python raises SyntaxError for long and deeply-nested expressions 2000-06-20 19:13:27 +00:00
test_math.py remove all occurence of math.rint() from the sources 2000-08-10 04:23:30 +00:00
test_md5.py Added a test of the md5.hexdigest() method. Funny enough, this test 2000-08-15 06:01:36 +00:00
test_minidom.py Compute the name of the test file so that it can be found both when this 2000-07-03 16:37:42 +00:00
test_mmap.py Fix inconsistent use of tab/space in indentation. 2000-07-30 15:38:35 +00:00
test_new.py Mass check-in after untabifying all files that need it. 1998-03-26 19:42:58 +00:00
test_nis.py Raise 'TestSkipped' (from the test_support) module rather than 'ImportError' 2000-08-04 13:34:43 +00:00
test_ntpath.py Remove the test for abspath with an empty path - too hard to do in a cross-platform manner. 2000-08-14 23:06:37 +00:00
test_opcodes.py Mass check-in after untabifying all files that need it. 1998-03-26 19:42:58 +00:00
test_openpty.py Raise 'TestSkipped' (from the test_support) module rather than 'ImportError' 2000-08-04 13:34:43 +00:00
test_operations.py Initial revision 1992-01-27 17:00:37 +00:00
test_operator.py Mass check-in after untabifying all files that need it. 1998-03-26 19:42:58 +00:00
test_pickle.py Basic regr tests for pickle/cPickle 1999-03-25 22:38:49 +00:00
test_pkg.py Apply SF patch #101135, adding 'import module as m' and 'from module import 2000-08-17 22:55:00 +00:00
test_popen2.py -- changed test to work on platforms which have os.popen 2000-07-27 07:42:43 +00:00
test_posixpath.py new test file for posixpath module 2000-07-12 00:20:08 +00:00
test_pow.py Revise tests to support str(<long int object>) not appending "L". 1999-12-23 15:36:42 +00:00
test_pty.py Raise 'TestSkipped' (from the test_support) module rather than 'ImportError' 2000-08-04 13:34:43 +00:00
test_pwd.py typos fixed by Rob Hooft 2000-06-28 14:48:01 +00:00
test_pyexpat.py Change pyexpat test suite to exercise the .returns_unicode attribute, 2000-06-27 00:37:25 +00:00
test_re.py Better conformance to the Python Style Guide: use spaces around operators. 2000-08-18 16:09:56 +00:00
test_regex.py Mass check-in after untabifying all files that need it. 1998-03-26 19:42:58 +00:00
test_rfc822.py added a test for "To: :" patch 1999-07-12 18:47:00 +00:00
test_rgbimg.py Convert some old-style string exceptions to class exceptions. 2000-08-18 14:50:20 +00:00
test_rotor.py Oops. Of course, print b should also be print `b`. :-( 1997-08-18 13:42:28 +00:00
test_select.py Made an error message slightly more useful if select() returns something goofy 2000-07-25 00:51:31 +00:00
test_sha.py Added Jeremy's test code for the sha module. 1999-03-24 19:04:32 +00:00
test_signal.py Raise 'TestSkipped' (from the test_support) module rather than 'ImportError' 2000-08-04 13:34:43 +00:00
test_socket.py Skip Montanaro: modify test_socket.py to listen/connect using loopback addr 2000-06-29 17:00:54 +00:00
test_sre.py -- changed findall to return empty strings instead of None 2000-08-09 09:14:35 +00:00
test_strftime.py # Bah. The same problem occurred a second time. 1999-04-08 20:22:46 +00:00
test_string.py Raise 'TestSkipped' (from the test_support) module rather than 'ImportError' 2000-08-04 13:34:43 +00:00
test_strop.py Mass check-in after untabifying all files that need it. 1998-03-26 19:42:58 +00:00
test_struct.py Mass check-in after untabifying all files that need it. 1998-03-26 19:42:58 +00:00
test_sunaudiodev.py Move unified findfile() into test_support.py 1998-04-23 20:13:30 +00:00
test_support.py Restore TestSkipped exception; appears to have disappeared in last checkin. 2000-07-24 06:55:00 +00:00
test_thread.py Use random instead of whrandom. 1998-05-20 17:05:52 +00:00
test_time.py Mass check-in after untabifying all files that need it. 1998-03-26 19:42:58 +00:00
test_timing.py Test of the timing module 1997-01-13 23:03:07 +00:00
test_tokenize.py Move unified findfile() into test_support.py 1998-04-23 20:13:30 +00:00
test_types.py Barry's patch to test the new setdefault() method. 2000-08-08 16:13:23 +00:00
test_ucn.py New test for the ucnhash module. 2000-06-30 09:45:20 +00:00
test_unicode.py Removing UTF-16 aware Unicode comparison code. This kind of compare 2000-08-08 08:04:29 +00:00
test_unicodedata.py Marc-Andre Lemburg: 2000-03-28 20:29:59 +00:00
test_unpack.py Convert some old-style string exceptions to class exceptions. 2000-08-18 14:50:20 +00:00
test_urlparse.py Checking in empty tests for urlparse, as future place holders 2000-08-03 17:28:50 +00:00
test_userdict.py Test suite for UserDict 1999-03-26 15:32:05 +00:00
test_userlist.py robustify UserList constructor -- will now accept any sequence 2000-03-31 00:17:46 +00:00
test_userstring.py factor out test definitions to string_tests module 2000-07-10 17:08:42 +00:00
test_winreg.py Spelling fixes supplied by Rob W. W. Hooft. All these are fixes in either 2000-07-16 12:04:32 +00:00
test_winreg2.py Was quite broken and incomplete. Now passes the test suite, but is still incomplete. 2000-07-28 03:45:32 +00:00
test_winsound.py Fix spelling error and remove Windows line endings. 2000-04-24 14:01:51 +00:00
test_xmllib.py New version of xmllib from Sjoerd. 1998-01-29 14:55:24 +00:00
test_zipfile.py Test for zipfile.py, by Jim Ahlstrom. 2000-04-10 13:23:04 +00:00
test_zlib.py typos fixed by Rob Hooft 2000-06-28 14:48:01 +00:00
testall.py Restore old behavior of autotest and testall, using regrtest. 1997-03-07 21:05:43 +00:00
testimg.uue Use uuencoded test images. 1997-04-09 21:25:01 +00:00
testimgr.uue Use uuencoded test images. 1997-04-09 21:25:01 +00:00
testrgb.uue Use uuencoded test images. 1997-04-09 21:25:01 +00:00
tokenize_tests.py Tests for tokenize.py (Ka-Ping Yee) 1997-10-27 22:15:06 +00:00

README

		      Writing Python Regression Tests
		      -------------------------------
			       Skip Montanaro
			      (skip@mojam.com)


Introduction

If you add a new module to Python or modify the functionality of an existing
module, you should write one or more test cases to exercise that new
functionality.  The mechanics of how the test system operates are fairly
straightforward.  When a test case is run, the output is compared with the
expected output that is stored in .../Lib/test/output.  If the test runs to
completion and the actual and expected outputs match, the test succeeds, if
not, it fails.  If an ImportError or test_support.TestSkipped error is
raised, the test is not run.

You will be writing unit tests (isolated tests of functions and objects
defined by the module) using white box techniques.  Unlike black box
testing, where you only have the external interfaces to guide your test case
writing, in white box testing you can see the code being tested and tailor
your test cases to exercise it more completely.  In particular, you will be
able to refer to the C and Python code in the CVS repository when writing
your regression test cases.


Executing Test Cases

If you are writing test cases for module spam, you need to create a file
in .../Lib/test named test_spam.py and an expected output file in
.../Lib/test/output named test_spam ("..."  represents the top-level
directory in the Python source tree, the directory containing the configure
script).  From the top-level directory, generate the initial version of the
test output file by executing:

    ./python Lib/test/regrtest.py -g test_spam.py

Any time you modify test_spam.py you need to generate a new expected
output file.  Don't forget to desk check the generated output to make sure
it's really what you expected to find!  To run a single test after modifying
a module, simply run regrtest.py without the -g flag:

    ./python Lib/test/regrtest.py test_spam.py

While debugging a regression test, you can of course execute it
independently of the regression testing framework and see what it prints:

    ./python Lib/test/test_spam.py

To run the entire test suite, make the "test" target at the top level:

    make test

On non-Unix platforms where make may not be available, you can simply
execute the two runs of regrtest (optimized and non-optimized) directly:

    ./python Lib/test/regrtest.py
    ./python -O Lib/test/regrtest.py


Test cases generate output based upon values computed by the test code.
When executed, regrtest.py compares the actual output generated by executing
the test case with the expected output and reports success or failure.  It
stands to reason that if the actual and expected outputs are to match, they
must not contain any machine dependencies.  This means your test cases
should not print out absolute machine addresses (e.g. the return value of
the id() builtin function) or floating point numbers with large numbers of
significant digits (unless you understand what you are doing!).


Test Case Writing Tips

Writing good test cases is a skilled task and is too complex to discuss in
detail in this short document.  Many books have been written on the subject.
I'll show my age by suggesting that Glenford Myers' "The Art of Software
Testing", published in 1979, is still the best introduction to the subject
available.  It is short (177 pages), easy to read, and discusses the major
elements of software testing, though its publication predates the
object-oriented software revolution, so doesn't cover that subject at all.
Unfortunately, it is very expensive (about $100 new).  If you can borrow it
or find it used (around $20), I strongly urge you to pick up a copy.

The most important goal when writing test cases is to break things.  A test
case that doesn't uncover a bug is much less valuable than one that does.
In designing test cases you should pay attention to the following:

    * Your test cases should exercise all the functions and objects defined
      in the module, not just the ones meant to be called by users of your
      module.  This may require you to write test code that uses the module
      in ways you don't expect (explicitly calling internal functions, for
      example - see test_atexit.py).

    * You should consider any boundary values that may tickle exceptional
      conditions (e.g. if you were writing regression tests for division,
      you might well want to generate tests with numerators and denominators
      at the limits of floating point and integer numbers on the machine
      performing the tests as well as a denominator of zero).

    * You should exercise as many paths through the code as possible.  This
      may not always be possible, but is a goal to strive for.  In
      particular, when considering if statements (or their equivalent), you
      want to create test cases that exercise both the true and false
      branches.  For loops, you should create test cases that exercise the
      loop zero, one and multiple times.

    * You should test with obviously invalid input.  If you know that a
      function requires an integer input, try calling it with other types of
      objects to see how it responds.

    * You should test with obviously out-of-range input.  If the domain of a
      function is only defined for positive integers, try calling it with a
      negative integer.

    * If you are going to fix a bug that wasn't uncovered by an existing
      test, try to write a test case that exposes the bug (preferably before
      fixing it).


Regression Test Writing Rules

Each test case is different.  There is no "standard" form for a Python
regression test case, though there are some general rules:

    * If your test case detects a failure, raise TestFailed (found in
      test_support).

    * Import everything you'll need as early as possible.

    * If you'll be importing objects from a module that is at least
      partially platform-dependent, only import those objects you need for
      the current test case to avoid spurious ImportError exceptions that
      prevent the test from running to completion.

    * Print all your test case results using the print statement.  For
      non-fatal errors, print an error message (or omit a successful
      completion print) to indicate the failure, but proceed instead of
      raising TestFailed.


Miscellaneous

There is a test_support module you can import from your test case.  It
provides the following useful objects:

    * TestFailed - raise this exception when your regression test detects a
      failure.

    * findfile(file) - you can call this function to locate a file somewhere
      along sys.path or in the Lib/test tree - see test_linuxaudiodev.py for
      an example of its use.

    * verbose - you can use this variable to control print output.  Many
      modules use it.  Search for "verbose" in the test_*.py files to see
      lots of examples.

    * fcmp(x,y) - you can call this function to compare two floating point
      numbers when you expect them to only be approximately equal withing a
      fuzz factor (test_support.FUZZ, which defaults to 1e-6).

Python and C statement coverage results are currently available at

    http://www.musi-cal.com/~skip/python/Python/dist/src/

As of this writing (July, 2000) these results are being generated nightly.
You can refer to the summaries and the test coverage output files to see
where coverage is adequate or lacking and write test cases to beef up the
coverage.