cpython/Lib/test
Fredrik Lundh 06d126803c Move uchhash functionality into unicodedata (after the recent
crop of changes, the files are small enough to do this).  Also
adds "name" and "lookup" functions to unicodedata.
2001-01-24 07:59:11 +00:00
..
output Add simple test of list comprehension that uses a name that isn't 2001-01-23 01:51:40 +00:00
README Added note saying to use test_support.TESTFN for a temporary filename, 2000-10-23 16:37:14 +00:00
__init__.py Add __init__.py to test package. 1997-11-06 15:41:23 +00:00
audiotest.au The encoding type was wrong, I think. 1999-02-05 20:57:44 +00:00
autotest.py Rehabilitate autotest.py. 2000-08-23 05:28:45 +00:00
greyrgb.uue Use uuencoded test images. 1997-04-16 00:30:45 +00:00
pickletester.py Typo repair. 2001-01-22 22:43:35 +00:00
pystone.py Make reindent.py happy (convert everything to 4-space indents!). 2000-10-23 17:22:08 +00:00
re_tests.py from the really-stupid-bug department: uppercase literals should match 2001-01-15 18:28:14 +00:00
regex_tests.py Make reindent.py happy (convert everything to 4-space indents!). 2000-10-23 17:22:08 +00:00
regrtest.py Christmas present to myself: changed regrtest in two ways: 2000-12-30 22:21:22 +00:00
reperf.py Measure performance of sub(), split(), findall(). 1998-07-17 21:10:42 +00:00
sortperf.py Add Tim's worst case scenario. 1998-05-26 15:05:12 +00:00
string_tests.py Whitespace normalization. Leaving tokenize_tests.py alone for now. 2001-01-18 02:22:22 +00:00
test.xml Added minidom tests. 2000-07-01 14:54:16 +00:00
test.xml.out Added necessary test input file for test_sax.py 2000-09-24 18:57:26 +00:00
test_MimeWriter.py Make reindent.py happy (convert everything to 4-space indents!). 2000-10-23 17:22:08 +00:00
test_StringIO.py Added a test for the StringIO write() error I just fixed. 2000-10-12 16:46:28 +00:00
test___all__.py a few more modules get __all__ 2001-01-24 06:27:27 +00:00
test_al.py a bold attempt to fix things broken by MAL's verify patch: import 2001-01-17 21:51:36 +00:00
test_array.py a bold attempt to fix things broken by MAL's verify patch: import 2001-01-17 21:51:36 +00:00
test_atexit.py a bold attempt to fix things broken by MAL's verify patch: import 2001-01-17 21:51:36 +00:00
test_audioop.py a bold attempt to fix things broken by MAL's verify patch: import 2001-01-17 21:51:36 +00:00
test_augassign.py Make reindent.py happy (convert everything to 4-space indents!). 2000-10-23 17:22:08 +00:00
test_b1.py OK, changed my mind once more on this. The comparison hierarchy is 2001-01-22 19:30:07 +00:00
test_b2.py Now that Marc-Andre has retracted unistr(), remove the tests. 2001-01-19 21:57:52 +00:00
test_bastion.py add test of bastion and rexec to std regression test suite 2001-01-19 03:22:48 +00:00
test_binascii.py This patch removes all uses of "assert" in the regression test suite 2001-01-17 19:11:13 +00:00
test_binhex.py a bold attempt to fix things broken by MAL's verify patch: import 2001-01-17 21:51:36 +00:00
test_bisect.py Fred, THIS NEEDS DOCS! The function docstrings tell the tale. 2000-12-29 02:06:45 +00:00
test_bsddb.py a bold attempt to fix things broken by MAL's verify patch: import 2001-01-17 21:51:36 +00:00
test_bufio.py a bold attempt to fix things broken by MAL's verify patch: import 2001-01-17 21:51:36 +00:00
test_builtin.py Initial revision 1992-01-27 17:00:37 +00:00
test_cd.py a bold attempt to fix things broken by MAL's verify patch: import 2001-01-17 21:51:36 +00:00
test_cfgparser.py Add test cases for ConfigParser.remove_option() behavior. This includes 2000-12-04 16:30:40 +00:00
test_cgi.py This patch removes all uses of "assert" in the regression test suite 2001-01-17 19:11:13 +00:00
test_charmapcodec.py Whitespace normalization. Leaving tokenize_tests.py alone for now. 2001-01-18 02:22:22 +00:00
test_cl.py a bold attempt to fix things broken by MAL's verify patch: import 2001-01-17 21:51:36 +00:00
test_class.py Finn Bock (SF patch #103345): Avoid outdated exec form in 2001-01-22 14:51:41 +00:00
test_cmath.py a bold attempt to fix things broken by MAL's verify patch: import 2001-01-17 21:51:36 +00:00
test_coercion.py Use numbers that can be accurately represented on binary machines. I hope 2001-01-03 01:52:11 +00:00
test_compare.py __rcmp__ no longer gets called on instances. Remove the test for it. 2001-01-04 01:34:52 +00:00
test_compile.py add test for SyntaxError on 2001-01-19 03:25:56 +00:00
test_contains.py Make reindent.py happy (convert everything to 4-space indents!). 2000-10-23 17:22:08 +00:00
test_cookie.py a bold attempt to fix things broken by MAL's verify patch: import 2001-01-17 21:51:36 +00:00
test_copy_reg.py Test the exception-raising for error cases in copy_reg. 2000-10-11 22:17:35 +00:00
test_cpickle.py Reorganize pickle/cPickle testing so the tests pass regardless of the order 2001-01-22 22:05:20 +00:00
test_crypt.py Whitespace normalization. Leaving tokenize_tests.py alone for now. 2001-01-18 02:22:22 +00:00
test_dbm.py a bold attempt to fix things broken by MAL's verify patch: import 2001-01-17 21:51:36 +00:00
test_dl.py a bold attempt to fix things broken by MAL's verify patch: import 2001-01-17 21:51:36 +00:00
test_dospath.py Make reindent.py happy (convert everything to 4-space indents!). 2000-10-23 17:22:08 +00:00
test_dumbdbm.py Use a saner test filename, to work on Windows. 2001-01-19 21:06:50 +00:00
test_errno.py a bold attempt to fix things broken by MAL's verify patch: import 2001-01-17 21:51:36 +00:00
test_exceptions.py Use 4-space indents. 2000-10-23 17:00:30 +00:00
test_extcall.py Patch #103344: Sort dicts from extcall for easier comparison with Jython. 2001-01-21 18:52:02 +00:00
test_fcntl.py a bold attempt to fix things broken by MAL's verify patch: import 2001-01-17 21:51:36 +00:00
test_file.py This patch removes all uses of "assert" in the regression test suite 2001-01-17 19:11:13 +00:00
test_fork1.py This patch removes all uses of "assert" in the regression test suite 2001-01-17 19:11:13 +00:00
test_format.py Whitespace normalization. Leaving tokenize_tests.py alone for now. 2001-01-18 02:22:22 +00:00
test_funcattrs.py Add some regression tests of coredump bugs in funcobject.c 2.31. Also 2001-01-19 19:55:12 +00:00
test_gc.py This patch removes all uses of "assert" in the regression test suite 2001-01-17 19:11:13 +00:00
test_gdbm.py a bold attempt to fix things broken by MAL's verify patch: import 2001-01-17 21:51:36 +00:00
test_getopt.py This patch removes all uses of "assert" in the regression test suite 2001-01-17 19:11:13 +00:00
test_gettext.py Make reindent.py happy (convert everything to 4-space indents!). 2000-10-23 17:22:08 +00:00
test_gl.py a bold attempt to fix things broken by MAL's verify patch: import 2001-01-17 21:51:36 +00:00
test_grammar.py Add simple test of list comprehension that uses a name that isn't 2001-01-23 01:51:40 +00:00
test_grp.py a bold attempt to fix things broken by MAL's verify patch: import 2001-01-17 21:51:36 +00:00
test_gzip.py Marc-Andre must not have run these tests -- they used verify() but 2001-01-17 21:43:06 +00:00
test_hash.py Make reindent.py happy (convert everything to 4-space indents!). 2000-10-23 17:22:08 +00:00
test_imageop.py a bold attempt to fix things broken by MAL's verify patch: import 2001-01-17 21:51:36 +00:00
test_imgfile.py a bold attempt to fix things broken by MAL's verify patch: import 2001-01-17 21:51:36 +00:00
test_import.py Insert the current directory to the front of sys.path -- and remove it 2000-10-24 17:16:32 +00:00
test_largefile.py Make reindent.py happy (convert everything to 4-space indents!). 2000-10-23 17:22:08 +00:00
test_linuxaudiodev.py It seems our sound cards can't play mulaw data. Use native-format 2000-10-08 00:20:20 +00:00
test_long.py This patch removes all uses of "assert" in the regression test suite 2001-01-17 19:11:13 +00:00
test_longexp.py Make reindent.py happy (convert everything to 4-space indents!). 2000-10-23 17:22:08 +00:00
test_mailbox.py Added test for regression on SourceForge bug #117490. 2000-10-23 13:39:15 +00:00
test_math.py Update the code to better reflect recommended style: 2000-12-12 23:11:42 +00:00
test_md5.py Update the code to better reflect recommended style: 2000-12-12 23:11:42 +00:00
test_mimetools.py Add mimetools testcase. 2000-09-30 17:03:19 +00:00
test_minidom.py a bold attempt to fix things broken by MAL's verify patch: import 2001-01-17 21:51:36 +00:00
test_mmap.py This patch removes all uses of "assert" in the regression test suite 2001-01-17 19:11:13 +00:00
test_new.py a bold attempt to fix things broken by MAL's verify patch: import 2001-01-17 21:51:36 +00:00
test_nis.py a bold attempt to fix things broken by MAL's verify patch: import 2001-01-17 21:51:36 +00:00
test_ntpath.py Make reindent.py happy (convert everything to 4-space indents!). 2000-10-23 17:22:08 +00:00
test_opcodes.py Update the code to better reflect recommended style: 2000-12-12 23:11:42 +00:00
test_openpty.py a bold attempt to fix things broken by MAL's verify patch: import 2001-01-17 21:51:36 +00:00
test_operations.py Make reindent.py happy (convert everything to 4-space indents!). 2000-10-23 17:22:08 +00:00
test_operator.py Update the code to better reflect recommended style: 2000-12-12 23:11:42 +00:00
test_parser.py a bold attempt to fix things broken by MAL's verify patch: import 2001-01-17 21:51:36 +00:00
test_pickle.py Reorganize pickle/cPickle testing so the tests pass regardless of the order 2001-01-22 22:05:20 +00:00
test_pkg.py Patch #103343: Allow the important test_pkg to succeed under Jython. 2001-01-21 19:51:53 +00:00
test_poll.py This patch removes all uses of "assert" in the regression test suite 2001-01-17 19:11:13 +00:00
test_popen2.py The "more" cmd varies across Windows flavors, sometimes adding stray 2000-09-01 20:38:55 +00:00
test_posixpath.py Make reindent.py happy (convert everything to 4-space indents!). 2000-10-23 17:22:08 +00:00
test_pow.py Update the code to better reflect recommended style: 2000-12-12 23:11:42 +00:00
test_pty.py a bold attempt to fix things broken by MAL's verify patch: import 2001-01-17 21:51:36 +00:00
test_pwd.py a bold attempt to fix things broken by MAL's verify patch: import 2001-01-17 21:51:36 +00:00
test_pyexpat.py The "context" parameter to the ExternalEntityRefParameter exposes internal 2000-12-23 22:12:07 +00:00
test_re.py This patch removes all uses of "assert" in the regression test suite 2001-01-17 19:11:13 +00:00
test_regex.py a bold attempt to fix things broken by MAL's verify patch: import 2001-01-17 21:51:36 +00:00
test_rfc822.py a bold attempt to fix things broken by MAL's verify patch: import 2001-01-17 21:51:36 +00:00
test_rgbimg.py a bold attempt to fix things broken by MAL's verify patch: import 2001-01-17 21:51:36 +00:00
test_richcmp.py Whitespace normalization. 2001-01-19 06:12:17 +00:00
test_rotor.py Update the code to better reflect recommended style: 2000-12-12 23:11:42 +00:00
test_sax.py Whitespace normalization. Leaving tokenize_tests.py alone for now. 2001-01-18 02:22:22 +00:00
test_select.py a bold attempt to fix things broken by MAL's verify patch: import 2001-01-17 21:51:36 +00:00
test_sha.py Added Jeremy's test code for the sha module. 1999-03-24 19:04:32 +00:00
test_signal.py a bold attempt to fix things broken by MAL's verify patch: import 2001-01-17 21:51:36 +00:00
test_socket.py a bold attempt to fix things broken by MAL's verify patch: import 2001-01-17 21:51:36 +00:00
test_sre.py a bold attempt to fix things broken by MAL's verify patch: import 2001-01-17 21:51:36 +00:00
test_strftime.py a bold attempt to fix things broken by MAL's verify patch: import 2001-01-17 21:51:36 +00:00
test_string.py a bold attempt to fix things broken by MAL's verify patch: import 2001-01-17 21:51:36 +00:00
test_strop.py a bold attempt to fix things broken by MAL's verify patch: import 2001-01-17 21:51:36 +00:00
test_struct.py Update the code to better reflect recommended style: 2000-12-12 23:11:42 +00:00
test_sunaudiodev.py a bold attempt to fix things broken by MAL's verify patch: import 2001-01-17 21:51:36 +00:00
test_sundry.py Restore alphabetic order. Also try to import rlcompleter and curses, but 2001-01-23 09:50:30 +00:00
test_support.py docstring typo 2001-01-20 19:12:54 +00:00
test_thread.py a bold attempt to fix things broken by MAL's verify patch: import 2001-01-17 21:51:36 +00:00
test_threadedtempfile.py Fix stupidity. 2001-01-13 03:45:59 +00:00
test_time.py Update the code to better reflect recommended style: 2000-12-12 23:11:42 +00:00
test_timing.py a bold attempt to fix things broken by MAL's verify patch: import 2001-01-17 21:51:36 +00:00
test_tokenize.py a bold attempt to fix things broken by MAL's verify patch: import 2001-01-17 21:51:36 +00:00
test_types.py Update the code to better reflect recommended style: 2000-12-12 23:11:42 +00:00
test_ucn.py Move uchhash functionality into unicodedata (after the recent 2001-01-24 07:59:11 +00:00
test_unicode.py Change verify() function to raise TestFailed, not AssertionError. 2001-01-19 19:01:56 +00:00
test_unicodedata.py This patch removes all uses of "assert" in the regression test suite 2001-01-17 19:11:13 +00:00
test_unpack.py Update the code to better reflect recommended style: 2000-12-12 23:11:42 +00:00
test_urllib.py added some tests for urlencode 2001-01-20 20:22:30 +00:00
test_urlparse.py Add test cases based on RFC 1808. So now we actually have a test suite 2001-01-05 05:57:04 +00:00
test_userdict.py This patch removes all uses of "assert" in the regression test suite 2001-01-17 19:11:13 +00:00
test_userlist.py a bold attempt to fix things broken by MAL's verify patch: import 2001-01-17 21:51:36 +00:00
test_userstring.py a bold attempt to fix things broken by MAL's verify patch: import 2001-01-17 21:51:36 +00:00
test_wave.py Use binary mode to open "wave" files. 2000-12-19 06:32:57 +00:00
test_winreg.py a bold attempt to fix things broken by MAL's verify patch: import 2001-01-17 21:51:36 +00:00
test_winsound.py Make reindent.py happy (convert everything to 4-space indents!). 2000-10-23 17:22:08 +00:00
test_xmllib.py a bold attempt to fix things broken by MAL's verify patch: import 2001-01-17 21:51:36 +00:00
test_xreadline.py Whitespace normalization. Leaving tokenize_tests.py alone for now. 2001-01-18 02:22:22 +00:00
test_zipfile.py Make reindent.py happy (convert everything to 4-space indents!). 2000-10-23 17:22:08 +00:00
test_zlib.py Make reindent.py happy (convert everything to 4-space indents!). 2000-10-23 17:22:08 +00:00
testall.py Restore old behavior of autotest and testall, using regrtest. 1997-03-07 21:05:43 +00:00
testcodec.py Whitespace normalization. Leaving tokenize_tests.py alone for now. 2001-01-18 02:22:22 +00:00
testimg.uue Use uuencoded test images. 1997-04-09 21:25:01 +00:00
testimgr.uue Use uuencoded test images. 1997-04-09 21:25:01 +00:00
testrgb.uue Use uuencoded test images. 1997-04-09 21:25:01 +00:00
tokenize_tests.py Add tokenizer support and tests for u'', U"", uR'', Ur"", etc. 2001-01-15 22:04:30 +00:00

README

		      Writing Python Regression Tests
		      -------------------------------
			       Skip Montanaro
			      (skip@mojam.com)


Introduction

If you add a new module to Python or modify the functionality of an existing
module, you should write one or more test cases to exercise that new
functionality.  The mechanics of how the test system operates are fairly
straightforward.  When a test case is run, the output is compared with the
expected output that is stored in .../Lib/test/output.  If the test runs to
completion and the actual and expected outputs match, the test succeeds, if
not, it fails.  If an ImportError or test_support.TestSkipped error is
raised, the test is not run.

You will be writing unit tests (isolated tests of functions and objects
defined by the module) using white box techniques.  Unlike black box
testing, where you only have the external interfaces to guide your test case
writing, in white box testing you can see the code being tested and tailor
your test cases to exercise it more completely.  In particular, you will be
able to refer to the C and Python code in the CVS repository when writing
your regression test cases.


Executing Test Cases

If you are writing test cases for module spam, you need to create a file
in .../Lib/test named test_spam.py and an expected output file in
.../Lib/test/output named test_spam ("..."  represents the top-level
directory in the Python source tree, the directory containing the configure
script).  From the top-level directory, generate the initial version of the
test output file by executing:

    ./python Lib/test/regrtest.py -g test_spam.py

Any time you modify test_spam.py you need to generate a new expected
output file.  Don't forget to desk check the generated output to make sure
it's really what you expected to find!  To run a single test after modifying
a module, simply run regrtest.py without the -g flag:

    ./python Lib/test/regrtest.py test_spam.py

While debugging a regression test, you can of course execute it
independently of the regression testing framework and see what it prints:

    ./python Lib/test/test_spam.py

To run the entire test suite, make the "test" target at the top level:

    make test

On non-Unix platforms where make may not be available, you can simply
execute the two runs of regrtest (optimized and non-optimized) directly:

    ./python Lib/test/regrtest.py
    ./python -O Lib/test/regrtest.py


Test cases generate output based upon values computed by the test code.
When executed, regrtest.py compares the actual output generated by executing
the test case with the expected output and reports success or failure.  It
stands to reason that if the actual and expected outputs are to match, they
must not contain any machine dependencies.  This means your test cases
should not print out absolute machine addresses (e.g. the return value of
the id() builtin function) or floating point numbers with large numbers of
significant digits (unless you understand what you are doing!).


Test Case Writing Tips

Writing good test cases is a skilled task and is too complex to discuss in
detail in this short document.  Many books have been written on the subject.
I'll show my age by suggesting that Glenford Myers' "The Art of Software
Testing", published in 1979, is still the best introduction to the subject
available.  It is short (177 pages), easy to read, and discusses the major
elements of software testing, though its publication predates the
object-oriented software revolution, so doesn't cover that subject at all.
Unfortunately, it is very expensive (about $100 new).  If you can borrow it
or find it used (around $20), I strongly urge you to pick up a copy.

The most important goal when writing test cases is to break things.  A test
case that doesn't uncover a bug is much less valuable than one that does.
In designing test cases you should pay attention to the following:

    * Your test cases should exercise all the functions and objects defined
      in the module, not just the ones meant to be called by users of your
      module.  This may require you to write test code that uses the module
      in ways you don't expect (explicitly calling internal functions, for
      example - see test_atexit.py).

    * You should consider any boundary values that may tickle exceptional
      conditions (e.g. if you were writing regression tests for division,
      you might well want to generate tests with numerators and denominators
      at the limits of floating point and integer numbers on the machine
      performing the tests as well as a denominator of zero).

    * You should exercise as many paths through the code as possible.  This
      may not always be possible, but is a goal to strive for.  In
      particular, when considering if statements (or their equivalent), you
      want to create test cases that exercise both the true and false
      branches.  For loops, you should create test cases that exercise the
      loop zero, one and multiple times.

    * You should test with obviously invalid input.  If you know that a
      function requires an integer input, try calling it with other types of
      objects to see how it responds.

    * You should test with obviously out-of-range input.  If the domain of a
      function is only defined for positive integers, try calling it with a
      negative integer.

    * If you are going to fix a bug that wasn't uncovered by an existing
      test, try to write a test case that exposes the bug (preferably before
      fixing it).

    * If you need to create a temporary file, you can use the filename in
      test_support.TESTFN to do so.  It is important to remove the file
      when done; other tests should be able to use the name without cleaning
      up after your test.


Regression Test Writing Rules

Each test case is different.  There is no "standard" form for a Python
regression test case, though there are some general rules:

    * If your test case detects a failure, raise TestFailed (found in
      test_support).

    * Import everything you'll need as early as possible.

    * If you'll be importing objects from a module that is at least
      partially platform-dependent, only import those objects you need for
      the current test case to avoid spurious ImportError exceptions that
      prevent the test from running to completion.

    * Print all your test case results using the print statement.  For
      non-fatal errors, print an error message (or omit a successful
      completion print) to indicate the failure, but proceed instead of
      raising TestFailed.

    * Use "assert" sparingly, if at all.  It's usually better to just print
      what you got, and rely on regrtest's got-vs-expected comparison to
      catch deviations from what you expect.  assert statements aren't
      executed at all when regrtest is run in -O mode; and, because they
      cause the test to stop immediately, can lead to a long & tedious
      test-fix, test-fix, test-fix, ... cycle when things are badly broken
      (and note that "badly broken" often includes running the test suite
      for the first time on new platforms or under new implementations of
      the language).


Miscellaneous

There is a test_support module you can import from your test case.  It
provides the following useful objects:

    * TestFailed - raise this exception when your regression test detects a
      failure.

    * TestSkipped - raise this if the test could not be run because the
      platform doesn't offer all the required facilities (like large
      file support), even if all the required modules are available.

    * findfile(file) - you can call this function to locate a file somewhere
      along sys.path or in the Lib/test tree - see test_linuxaudiodev.py for
      an example of its use.

    * verbose - you can use this variable to control print output.  Many
      modules use it.  Search for "verbose" in the test_*.py files to see
      lots of examples.

    * use_large_resources - true iff tests requiring large time or space
      should be run.

    * fcmp(x,y) - you can call this function to compare two floating point
      numbers when you expect them to only be approximately equal withing a
      fuzz factor (test_support.FUZZ, which defaults to 1e-6).

NOTE:  Always import something from test_support like so:

    from test_support import verbose

or like so:

    import test_support
    ... use test_support.verbose in the code ...

Never import anything from test_support like this:

    from test.test_support import verbose

"test" is a package already, so can refer to modules it contains without
"test." qualification.  If you do an explicit "test.xxx" qualification, that
can fool Python into believing test.xxx is a module distinct from the xxx
in the current package, and you can end up importing two distinct copies of
xxx.  This is especially bad if xxx=test_support, as regrtest.py can (and
routinely does) overwrite its "verbose" and "use_large_resources"
attributes:  if you get a second copy of test_support loaded, it may not
have the same values for those as regrtest intended.


Python and C statement coverage results are currently available at

    http://www.musi-cal.com/~skip/python/Python/dist/src/

As of this writing (July, 2000) these results are being generated nightly.
You can refer to the summaries and the test coverage output files to see
where coverage is adequate or lacking and write test cases to beef up the
coverage.