1955 lines
74 KiB
TeX
1955 lines
74 KiB
TeX
\section{\module{doctest} ---
|
|
Test interactive Python examples}
|
|
|
|
\declaremodule{standard}{doctest}
|
|
\moduleauthor{Tim Peters}{tim@python.org}
|
|
\sectionauthor{Tim Peters}{tim@python.org}
|
|
\sectionauthor{Moshe Zadka}{moshez@debian.org}
|
|
\sectionauthor{Edward Loper}{edloper@users.sourceforge.net}
|
|
|
|
\modulesynopsis{A framework for verifying interactive Python examples.}
|
|
|
|
The \refmodule{doctest} module searches for pieces of text that look like
|
|
interactive Python sessions, and then executes those sessions to
|
|
verify that they work exactly as shown. There are several common ways to
|
|
use doctest:
|
|
|
|
\begin{itemize}
|
|
\item To check that a module's docstrings are up-to-date by verifying
|
|
that all interactive examples still work as documented.
|
|
\item To perform regression testing by verifying that interactive
|
|
examples from a test file or a test object work as expected.
|
|
\item To write tutorial documentation for a package, liberally
|
|
illustrated with input-output examples. Depending on whether
|
|
the examples or the expository text are emphasized, this has
|
|
the flavor of "literate testing" or "executable documentation".
|
|
\end{itemize}
|
|
|
|
Here's a complete but small example module:
|
|
|
|
\begin{verbatim}
|
|
"""
|
|
This is the "example" module.
|
|
|
|
The example module supplies one function, factorial(). For example,
|
|
|
|
>>> factorial(5)
|
|
120
|
|
"""
|
|
|
|
def factorial(n):
|
|
"""Return the factorial of n, an exact integer >= 0.
|
|
|
|
If the result is small enough to fit in an int, return an int.
|
|
Else return a long.
|
|
|
|
>>> [factorial(n) for n in range(6)]
|
|
[1, 1, 2, 6, 24, 120]
|
|
>>> [factorial(long(n)) for n in range(6)]
|
|
[1, 1, 2, 6, 24, 120]
|
|
>>> factorial(30)
|
|
265252859812191058636308480000000L
|
|
>>> factorial(30L)
|
|
265252859812191058636308480000000L
|
|
>>> factorial(-1)
|
|
Traceback (most recent call last):
|
|
...
|
|
ValueError: n must be >= 0
|
|
|
|
Factorials of floats are OK, but the float must be an exact integer:
|
|
>>> factorial(30.1)
|
|
Traceback (most recent call last):
|
|
...
|
|
ValueError: n must be exact integer
|
|
>>> factorial(30.0)
|
|
265252859812191058636308480000000L
|
|
|
|
It must also not be ridiculously large:
|
|
>>> factorial(1e100)
|
|
Traceback (most recent call last):
|
|
...
|
|
OverflowError: n too large
|
|
"""
|
|
|
|
\end{verbatim}
|
|
% allow LaTeX to break here.
|
|
\begin{verbatim}
|
|
|
|
import math
|
|
if not n >= 0:
|
|
raise ValueError("n must be >= 0")
|
|
if math.floor(n) != n:
|
|
raise ValueError("n must be exact integer")
|
|
if n+1 == n: # catch a value like 1e300
|
|
raise OverflowError("n too large")
|
|
result = 1
|
|
factor = 2
|
|
while factor <= n:
|
|
result *= factor
|
|
factor += 1
|
|
return result
|
|
|
|
def _test():
|
|
import doctest
|
|
doctest.testmod()
|
|
|
|
if __name__ == "__main__":
|
|
_test()
|
|
\end{verbatim}
|
|
|
|
If you run \file{example.py} directly from the command line,
|
|
\refmodule{doctest} works its magic:
|
|
|
|
\begin{verbatim}
|
|
$ python example.py
|
|
$
|
|
\end{verbatim}
|
|
|
|
There's no output! That's normal, and it means all the examples
|
|
worked. Pass \programopt{-v} to the script, and \refmodule{doctest}
|
|
prints a detailed log of what it's trying, and prints a summary at the
|
|
end:
|
|
|
|
\begin{verbatim}
|
|
$ python example.py -v
|
|
Trying:
|
|
factorial(5)
|
|
Expecting:
|
|
120
|
|
ok
|
|
Trying:
|
|
[factorial(n) for n in range(6)]
|
|
Expecting:
|
|
[1, 1, 2, 6, 24, 120]
|
|
ok
|
|
Trying:
|
|
[factorial(long(n)) for n in range(6)]
|
|
Expecting:
|
|
[1, 1, 2, 6, 24, 120]
|
|
ok
|
|
\end{verbatim}
|
|
|
|
And so on, eventually ending with:
|
|
|
|
\begin{verbatim}
|
|
Trying:
|
|
factorial(1e100)
|
|
Expecting:
|
|
Traceback (most recent call last):
|
|
...
|
|
OverflowError: n too large
|
|
ok
|
|
1 items had no tests:
|
|
__main__._test
|
|
2 items passed all tests:
|
|
1 tests in __main__
|
|
8 tests in __main__.factorial
|
|
9 tests in 3 items.
|
|
9 passed and 0 failed.
|
|
Test passed.
|
|
$
|
|
\end{verbatim}
|
|
|
|
That's all you need to know to start making productive use of
|
|
\refmodule{doctest}! Jump in. The following sections provide full
|
|
details. Note that there are many examples of doctests in
|
|
the standard Python test suite and libraries. Especially useful examples
|
|
can be found in the standard test file \file{Lib/test/test_doctest.py}.
|
|
|
|
\subsection{Simple Usage: Checking Examples in
|
|
Docstrings\label{doctest-simple-testmod}}
|
|
|
|
The simplest way to start using doctest (but not necessarily the way
|
|
you'll continue to do it) is to end each module \module{M} with:
|
|
|
|
\begin{verbatim}
|
|
def _test():
|
|
import doctest
|
|
doctest.testmod()
|
|
|
|
if __name__ == "__main__":
|
|
_test()
|
|
\end{verbatim}
|
|
|
|
\refmodule{doctest} then examines docstrings in module \module{M}.
|
|
|
|
Running the module as a script causes the examples in the docstrings
|
|
to get executed and verified:
|
|
|
|
\begin{verbatim}
|
|
python M.py
|
|
\end{verbatim}
|
|
|
|
This won't display anything unless an example fails, in which case the
|
|
failing example(s) and the cause(s) of the failure(s) are printed to stdout,
|
|
and the final line of output is
|
|
\samp{***Test Failed*** \var{N} failures.}, where \var{N} is the
|
|
number of examples that failed.
|
|
|
|
Run it with the \programopt{-v} switch instead:
|
|
|
|
\begin{verbatim}
|
|
python M.py -v
|
|
\end{verbatim}
|
|
|
|
and a detailed report of all examples tried is printed to standard
|
|
output, along with assorted summaries at the end.
|
|
|
|
You can force verbose mode by passing \code{verbose=True} to
|
|
\function{testmod()}, or
|
|
prohibit it by passing \code{verbose=False}. In either of those cases,
|
|
\code{sys.argv} is not examined by \function{testmod()} (so passing
|
|
\programopt{-v} or not has no effect).
|
|
|
|
For more information on \function{testmod()}, see
|
|
section~\ref{doctest-basic-api}.
|
|
|
|
\subsection{Simple Usage: Checking Examples in a Text
|
|
File\label{doctest-simple-testfile}}
|
|
|
|
Another simple application of doctest is testing interactive examples
|
|
in a text file. This can be done with the \function{testfile()}
|
|
function:
|
|
|
|
\begin{verbatim}
|
|
import doctest
|
|
doctest.testfile("example.txt")
|
|
\end{verbatim}
|
|
|
|
That short script executes and verifies any interactive Python
|
|
examples contained in the file \file{example.txt}. The file content
|
|
is treated as if it were a single giant docstring; the file doesn't
|
|
need to contain a Python program! For example, perhaps \file{example.txt}
|
|
contains this:
|
|
|
|
\begin{verbatim}
|
|
The ``example`` module
|
|
======================
|
|
|
|
Using ``factorial``
|
|
-------------------
|
|
|
|
This is an example text file in reStructuredText format. First import
|
|
``factorial`` from the ``example`` module:
|
|
|
|
>>> from example import factorial
|
|
|
|
Now use it:
|
|
|
|
>>> factorial(6)
|
|
120
|
|
\end{verbatim}
|
|
|
|
Running \code{doctest.testfile("example.txt")} then finds the error
|
|
in this documentation:
|
|
|
|
\begin{verbatim}
|
|
File "./example.txt", line 14, in example.txt
|
|
Failed example:
|
|
factorial(6)
|
|
Expected:
|
|
120
|
|
Got:
|
|
720
|
|
\end{verbatim}
|
|
|
|
As with \function{testmod()}, \function{testfile()} won't display anything
|
|
unless an example fails. If an example does fail, then the failing
|
|
example(s) and the cause(s) of the failure(s) are printed to stdout, using
|
|
the same format as \function{testmod()}.
|
|
|
|
By default, \function{testfile()} looks for files in the calling
|
|
module's directory. See section~\ref{doctest-basic-api} for a
|
|
description of the optional arguments that can be used to tell it to
|
|
look for files in other locations.
|
|
|
|
Like \function{testmod()}, \function{testfile()}'s verbosity can be
|
|
set with the \programopt{-v} command-line switch or with the optional
|
|
keyword argument \var{verbose}.
|
|
|
|
For more information on \function{testfile()}, see
|
|
section~\ref{doctest-basic-api}.
|
|
|
|
\subsection{How It Works\label{doctest-how-it-works}}
|
|
|
|
This section examines in detail how doctest works: which docstrings it
|
|
looks at, how it finds interactive examples, what execution context it
|
|
uses, how it handles exceptions, and how option flags can be used to
|
|
control its behavior. This is the information that you need to know
|
|
to write doctest examples; for information about actually running
|
|
doctest on these examples, see the following sections.
|
|
|
|
\subsubsection{Which Docstrings Are Examined?\label{doctest-which-docstrings}}
|
|
|
|
The module docstring, and all function, class and method docstrings are
|
|
searched. Objects imported into the module are not searched.
|
|
|
|
In addition, if \code{M.__test__} exists and "is true", it must be a
|
|
dict, and each entry maps a (string) name to a function object, class
|
|
object, or string. Function and class object docstrings found from
|
|
\code{M.__test__} are searched, and strings are treated as if they
|
|
were docstrings. In output, a key \code{K} in \code{M.__test__} appears
|
|
with name
|
|
|
|
\begin{verbatim}
|
|
<name of M>.__test__.K
|
|
\end{verbatim}
|
|
|
|
Any classes found are recursively searched similarly, to test docstrings in
|
|
their contained methods and nested classes.
|
|
|
|
\versionchanged[A "private name" concept is deprecated and no longer
|
|
documented]{2.4}
|
|
|
|
\subsubsection{How are Docstring Examples
|
|
Recognized?\label{doctest-finding-examples}}
|
|
|
|
In most cases a copy-and-paste of an interactive console session works
|
|
fine, but doctest isn't trying to do an exact emulation of any specific
|
|
Python shell. All hard tab characters are expanded to spaces, using
|
|
8-column tab stops. If you don't believe tabs should mean that, too
|
|
bad: don't use hard tabs, or write your own \class{DocTestParser}
|
|
class.
|
|
|
|
\versionchanged[Expanding tabs to spaces is new; previous versions
|
|
tried to preserve hard tabs, with confusing results]{2.4}
|
|
|
|
\begin{verbatim}
|
|
>>> # comments are ignored
|
|
>>> x = 12
|
|
>>> x
|
|
12
|
|
>>> if x == 13:
|
|
... print "yes"
|
|
... else:
|
|
... print "no"
|
|
... print "NO"
|
|
... print "NO!!!"
|
|
...
|
|
no
|
|
NO
|
|
NO!!!
|
|
>>>
|
|
\end{verbatim}
|
|
|
|
Any expected output must immediately follow the final
|
|
\code{'>>>~'} or \code{'...~'} line containing the code, and
|
|
the expected output (if any) extends to the next \code{'>>>~'}
|
|
or all-whitespace line.
|
|
|
|
The fine print:
|
|
|
|
\begin{itemize}
|
|
|
|
\item Expected output cannot contain an all-whitespace line, since such a
|
|
line is taken to signal the end of expected output. If expected
|
|
output does contain a blank line, put \code{<BLANKLINE>} in your
|
|
doctest example each place a blank line is expected.
|
|
\versionchanged[\code{<BLANKLINE>} was added; there was no way to
|
|
use expected output containing empty lines in
|
|
previous versions]{2.4}
|
|
|
|
\item Output to stdout is captured, but not output to stderr (exception
|
|
tracebacks are captured via a different means).
|
|
|
|
\item If you continue a line via backslashing in an interactive session,
|
|
or for any other reason use a backslash, you should use a raw
|
|
docstring, which will preserve your backslashes exactly as you type
|
|
them:
|
|
|
|
\begin{verbatim}
|
|
>>> def f(x):
|
|
... r'''Backslashes in a raw docstring: m\n'''
|
|
>>> print f.__doc__
|
|
Backslashes in a raw docstring: m\n
|
|
\end{verbatim}
|
|
|
|
Otherwise, the backslash will be interpreted as part of the string.
|
|
For example, the "{\textbackslash}" above would be interpreted as a
|
|
newline character. Alternatively, you can double each backslash in the
|
|
doctest version (and not use a raw string):
|
|
|
|
\begin{verbatim}
|
|
>>> def f(x):
|
|
... '''Backslashes in a raw docstring: m\\n'''
|
|
>>> print f.__doc__
|
|
Backslashes in a raw docstring: m\n
|
|
\end{verbatim}
|
|
|
|
\item The starting column doesn't matter:
|
|
|
|
\begin{verbatim}
|
|
>>> assert "Easy!"
|
|
>>> import math
|
|
>>> math.floor(1.9)
|
|
1.0
|
|
\end{verbatim}
|
|
|
|
and as many leading whitespace characters are stripped from the
|
|
expected output as appeared in the initial \code{'>>>~'} line
|
|
that started the example.
|
|
\end{itemize}
|
|
|
|
\subsubsection{What's the Execution Context?\label{doctest-execution-context}}
|
|
|
|
By default, each time \refmodule{doctest} finds a docstring to test, it
|
|
uses a \emph{shallow copy} of \module{M}'s globals, so that running tests
|
|
doesn't change the module's real globals, and so that one test in
|
|
\module{M} can't leave behind crumbs that accidentally allow another test
|
|
to work. This means examples can freely use any names defined at top-level
|
|
in \module{M}, and names defined earlier in the docstring being run.
|
|
Examples cannot see names defined in other docstrings.
|
|
|
|
You can force use of your own dict as the execution context by passing
|
|
\code{globs=your_dict} to \function{testmod()} or
|
|
\function{testfile()} instead.
|
|
|
|
\subsubsection{What About Exceptions?\label{doctest-exceptions}}
|
|
|
|
No problem, provided that the traceback is the only output produced by
|
|
the example: just paste in the traceback.\footnote{Examples containing
|
|
both expected output and an exception are not supported. Trying
|
|
to guess where one ends and the other begins is too error-prone,
|
|
and that also makes for a confusing test.}
|
|
Since tracebacks contain details that are likely to change rapidly (for
|
|
example, exact file paths and line numbers), this is one case where doctest
|
|
works hard to be flexible in what it accepts.
|
|
|
|
Simple example:
|
|
|
|
\begin{verbatim}
|
|
>>> [1, 2, 3].remove(42)
|
|
Traceback (most recent call last):
|
|
File "<stdin>", line 1, in ?
|
|
ValueError: list.remove(x): x not in list
|
|
\end{verbatim}
|
|
|
|
That doctest succeeds if \exception{ValueError} is raised, with the
|
|
\samp{list.remove(x): x not in list} detail as shown.
|
|
|
|
The expected output for an exception must start with a traceback
|
|
header, which may be either of the following two lines, indented the
|
|
same as the first line of the example:
|
|
|
|
\begin{verbatim}
|
|
Traceback (most recent call last):
|
|
Traceback (innermost last):
|
|
\end{verbatim}
|
|
|
|
The traceback header is followed by an optional traceback stack, whose
|
|
contents are ignored by doctest. The traceback stack is typically
|
|
omitted, or copied verbatim from an interactive session.
|
|
|
|
The traceback stack is followed by the most interesting part: the
|
|
line(s) containing the exception type and detail. This is usually the
|
|
last line of a traceback, but can extend across multiple lines if the
|
|
exception has a multi-line detail:
|
|
|
|
\begin{verbatim}
|
|
>>> raise ValueError('multi\n line\ndetail')
|
|
Traceback (most recent call last):
|
|
File "<stdin>", line 1, in ?
|
|
ValueError: multi
|
|
line
|
|
detail
|
|
\end{verbatim}
|
|
|
|
The last three lines (starting with \exception{ValueError}) are
|
|
compared against the exception's type and detail, and the rest are
|
|
ignored.
|
|
|
|
Best practice is to omit the traceback stack, unless it adds
|
|
significant documentation value to the example. So the last example
|
|
is probably better as:
|
|
|
|
\begin{verbatim}
|
|
>>> raise ValueError('multi\n line\ndetail')
|
|
Traceback (most recent call last):
|
|
...
|
|
ValueError: multi
|
|
line
|
|
detail
|
|
\end{verbatim}
|
|
|
|
Note that tracebacks are treated very specially. In particular, in the
|
|
rewritten example, the use of \samp{...} is independent of doctest's
|
|
\constant{ELLIPSIS} option. The ellipsis in that example could be left
|
|
out, or could just as well be three (or three hundred) commas or digits,
|
|
or an indented transcript of a Monty Python skit.
|
|
|
|
Some details you should read once, but won't need to remember:
|
|
|
|
\begin{itemize}
|
|
|
|
\item Doctest can't guess whether your expected output came from an
|
|
exception traceback or from ordinary printing. So, e.g., an example
|
|
that expects \samp{ValueError: 42 is prime} will pass whether
|
|
\exception{ValueError} is actually raised or if the example merely
|
|
prints that traceback text. In practice, ordinary output rarely begins
|
|
with a traceback header line, so this doesn't create real problems.
|
|
|
|
\item Each line of the traceback stack (if present) must be indented
|
|
further than the first line of the example, \emph{or} start with a
|
|
non-alphanumeric character. The first line following the traceback
|
|
header indented the same and starting with an alphanumeric is taken
|
|
to be the start of the exception detail. Of course this does the
|
|
right thing for genuine tracebacks.
|
|
|
|
\item When the \constant{IGNORE_EXCEPTION_DETAIL} doctest option is
|
|
is specified, everything following the leftmost colon is ignored.
|
|
|
|
\item The interactive shell omits the traceback header line for some
|
|
\exception{SyntaxError}s. But doctest uses the traceback header
|
|
line to distinguish exceptions from non-exceptions. So in the rare
|
|
case where you need to test a \exception{SyntaxError} that omits the
|
|
traceback header, you will need to manually add the traceback header
|
|
line to your test example.
|
|
|
|
\item For some \exception{SyntaxError}s, Python displays the character
|
|
position of the syntax error, using a \code{\^} marker:
|
|
|
|
\begin{verbatim}
|
|
>>> 1 1
|
|
File "<stdin>", line 1
|
|
1 1
|
|
^
|
|
SyntaxError: invalid syntax
|
|
\end{verbatim}
|
|
|
|
Since the lines showing the position of the error come before the
|
|
exception type and detail, they are not checked by doctest. For
|
|
example, the following test would pass, even though it puts the
|
|
\code{\^} marker in the wrong location:
|
|
|
|
\begin{verbatim}
|
|
>>> 1 1
|
|
Traceback (most recent call last):
|
|
File "<stdin>", line 1
|
|
1 1
|
|
^
|
|
SyntaxError: invalid syntax
|
|
\end{verbatim}
|
|
|
|
\end{itemize}
|
|
|
|
\versionchanged[The ability to handle a multi-line exception detail,
|
|
and the \constant{IGNORE_EXCEPTION_DETAIL} doctest option,
|
|
were added]{2.4}
|
|
|
|
\subsubsection{Option Flags and Directives\label{doctest-options}}
|
|
|
|
A number of option flags control various aspects of doctest's
|
|
behavior. Symbolic names for the flags are supplied as module constants,
|
|
which can be or'ed together and passed to various functions. The names
|
|
can also be used in doctest directives (see below).
|
|
|
|
The first group of options define test semantics, controlling
|
|
aspects of how doctest decides whether actual output matches an
|
|
example's expected output:
|
|
|
|
\begin{datadesc}{DONT_ACCEPT_TRUE_FOR_1}
|
|
By default, if an expected output block contains just \code{1},
|
|
an actual output block containing just \code{1} or just
|
|
\code{True} is considered to be a match, and similarly for \code{0}
|
|
versus \code{False}. When \constant{DONT_ACCEPT_TRUE_FOR_1} is
|
|
specified, neither substitution is allowed. The default behavior
|
|
caters to that Python changed the return type of many functions
|
|
from integer to boolean; doctests expecting "little integer"
|
|
output still work in these cases. This option will probably go
|
|
away, but not for several years.
|
|
\end{datadesc}
|
|
|
|
\begin{datadesc}{DONT_ACCEPT_BLANKLINE}
|
|
By default, if an expected output block contains a line
|
|
containing only the string \code{<BLANKLINE>}, then that line
|
|
will match a blank line in the actual output. Because a
|
|
genuinely blank line delimits the expected output, this is
|
|
the only way to communicate that a blank line is expected. When
|
|
\constant{DONT_ACCEPT_BLANKLINE} is specified, this substitution
|
|
is not allowed.
|
|
\end{datadesc}
|
|
|
|
\begin{datadesc}{NORMALIZE_WHITESPACE}
|
|
When specified, all sequences of whitespace (blanks and newlines) are
|
|
treated as equal. Any sequence of whitespace within the expected
|
|
output will match any sequence of whitespace within the actual output.
|
|
By default, whitespace must match exactly.
|
|
\constant{NORMALIZE_WHITESPACE} is especially useful when a line
|
|
of expected output is very long, and you want to wrap it across
|
|
multiple lines in your source.
|
|
\end{datadesc}
|
|
|
|
\begin{datadesc}{ELLIPSIS}
|
|
When specified, an ellipsis marker (\code{...}) in the expected output
|
|
can match any substring in the actual output. This includes
|
|
substrings that span line boundaries, and empty substrings, so it's
|
|
best to keep usage of this simple. Complicated uses can lead to the
|
|
same kinds of "oops, it matched too much!" surprises that \regexp{.*}
|
|
is prone to in regular expressions.
|
|
\end{datadesc}
|
|
|
|
\begin{datadesc}{IGNORE_EXCEPTION_DETAIL}
|
|
When specified, an example that expects an exception passes if
|
|
an exception of the expected type is raised, even if the exception
|
|
detail does not match. For example, an example expecting
|
|
\samp{ValueError: 42} will pass if the actual exception raised is
|
|
\samp{ValueError: 3*14}, but will fail, e.g., if
|
|
\exception{TypeError} is raised.
|
|
|
|
Note that a similar effect can be obtained using \constant{ELLIPSIS},
|
|
and \constant{IGNORE_EXCEPTION_DETAIL} may go away when Python releases
|
|
prior to 2.4 become uninteresting. Until then,
|
|
\constant{IGNORE_EXCEPTION_DETAIL} is the only clear way to write a
|
|
doctest that doesn't care about the exception detail yet continues
|
|
to pass under Python releases prior to 2.4 (doctest directives
|
|
appear to be comments to them). For example,
|
|
|
|
\begin{verbatim}
|
|
>>> (1, 2)[3] = 'moo' #doctest: +IGNORE_EXCEPTION_DETAIL
|
|
Traceback (most recent call last):
|
|
File "<stdin>", line 1, in ?
|
|
TypeError: object doesn't support item assignment
|
|
\end{verbatim}
|
|
|
|
passes under Python 2.4 and Python 2.3. The detail changed in 2.4,
|
|
to say "does not" instead of "doesn't".
|
|
|
|
\end{datadesc}
|
|
|
|
\begin{datadesc}{SKIP}
|
|
|
|
When specified, do not run the example at all. This can be useful
|
|
in contexts where doctest examples serve as both documentation and
|
|
test cases, and an example should be included for documentation
|
|
purposes, but should not be checked. E.g., the example's output
|
|
might be random; or the example might depend on resources which
|
|
would be unavailable to the test driver.
|
|
|
|
The SKIP flag can also be used for temporarily "commenting out"
|
|
examples.
|
|
|
|
\end{datadesc}
|
|
|
|
\begin{datadesc}{COMPARISON_FLAGS}
|
|
A bitmask or'ing together all the comparison flags above.
|
|
\end{datadesc}
|
|
|
|
The second group of options controls how test failures are reported:
|
|
|
|
\begin{datadesc}{REPORT_UDIFF}
|
|
When specified, failures that involve multi-line expected and
|
|
actual outputs are displayed using a unified diff.
|
|
\end{datadesc}
|
|
|
|
\begin{datadesc}{REPORT_CDIFF}
|
|
When specified, failures that involve multi-line expected and
|
|
actual outputs will be displayed using a context diff.
|
|
\end{datadesc}
|
|
|
|
\begin{datadesc}{REPORT_NDIFF}
|
|
When specified, differences are computed by \code{difflib.Differ},
|
|
using the same algorithm as the popular \file{ndiff.py} utility.
|
|
This is the only method that marks differences within lines as
|
|
well as across lines. For example, if a line of expected output
|
|
contains digit \code{1} where actual output contains letter \code{l},
|
|
a line is inserted with a caret marking the mismatching column
|
|
positions.
|
|
\end{datadesc}
|
|
|
|
\begin{datadesc}{REPORT_ONLY_FIRST_FAILURE}
|
|
When specified, display the first failing example in each doctest,
|
|
but suppress output for all remaining examples. This will prevent
|
|
doctest from reporting correct examples that break because of
|
|
earlier failures; but it might also hide incorrect examples that
|
|
fail independently of the first failure. When
|
|
\constant{REPORT_ONLY_FIRST_FAILURE} is specified, the remaining
|
|
examples are still run, and still count towards the total number of
|
|
failures reported; only the output is suppressed.
|
|
\end{datadesc}
|
|
|
|
\begin{datadesc}{REPORTING_FLAGS}
|
|
A bitmask or'ing together all the reporting flags above.
|
|
\end{datadesc}
|
|
|
|
"Doctest directives" may be used to modify the option flags for
|
|
individual examples. Doctest directives are expressed as a special
|
|
Python comment following an example's source code:
|
|
|
|
\begin{productionlist}[doctest]
|
|
\production{directive}
|
|
{"\#" "doctest:" \token{directive_options}}
|
|
\production{directive_options}
|
|
{\token{directive_option} ("," \token{directive_option})*}
|
|
\production{directive_option}
|
|
{\token{on_or_off} \token{directive_option_name}}
|
|
\production{on_or_off}
|
|
{"+" | "-"}
|
|
\production{directive_option_name}
|
|
{"DONT_ACCEPT_BLANKLINE" | "NORMALIZE_WHITESPACE" | ...}
|
|
\end{productionlist}
|
|
|
|
Whitespace is not allowed between the \code{+} or \code{-} and the
|
|
directive option name. The directive option name can be any of the
|
|
option flag names explained above.
|
|
|
|
An example's doctest directives modify doctest's behavior for that
|
|
single example. Use \code{+} to enable the named behavior, or
|
|
\code{-} to disable it.
|
|
|
|
For example, this test passes:
|
|
|
|
\begin{verbatim}
|
|
>>> print range(20) #doctest: +NORMALIZE_WHITESPACE
|
|
[0, 1, 2, 3, 4, 5, 6, 7, 8, 9,
|
|
10, 11, 12, 13, 14, 15, 16, 17, 18, 19]
|
|
\end{verbatim}
|
|
|
|
Without the directive it would fail, both because the actual output
|
|
doesn't have two blanks before the single-digit list elements, and
|
|
because the actual output is on a single line. This test also passes,
|
|
and also requires a directive to do so:
|
|
|
|
\begin{verbatim}
|
|
>>> print range(20) # doctest:+ELLIPSIS
|
|
[0, 1, ..., 18, 19]
|
|
\end{verbatim}
|
|
|
|
Multiple directives can be used on a single physical line, separated
|
|
by commas:
|
|
|
|
\begin{verbatim}
|
|
>>> print range(20) # doctest: +ELLIPSIS, +NORMALIZE_WHITESPACE
|
|
[0, 1, ..., 18, 19]
|
|
\end{verbatim}
|
|
|
|
If multiple directive comments are used for a single example, then
|
|
they are combined:
|
|
|
|
\begin{verbatim}
|
|
>>> print range(20) # doctest: +ELLIPSIS
|
|
... # doctest: +NORMALIZE_WHITESPACE
|
|
[0, 1, ..., 18, 19]
|
|
\end{verbatim}
|
|
|
|
As the previous example shows, you can add \samp{...} lines to your
|
|
example containing only directives. This can be useful when an
|
|
example is too long for a directive to comfortably fit on the same
|
|
line:
|
|
|
|
\begin{verbatim}
|
|
>>> print range(5) + range(10,20) + range(30,40) + range(50,60)
|
|
... # doctest: +ELLIPSIS
|
|
[0, ..., 4, 10, ..., 19, 30, ..., 39, 50, ..., 59]
|
|
\end{verbatim}
|
|
|
|
Note that since all options are disabled by default, and directives apply
|
|
only to the example they appear in, enabling options (via \code{+} in a
|
|
directive) is usually the only meaningful choice. However, option flags
|
|
can also be passed to functions that run doctests, establishing different
|
|
defaults. In such cases, disabling an option via \code{-} in a directive
|
|
can be useful.
|
|
|
|
\versionchanged[Constants \constant{DONT_ACCEPT_BLANKLINE},
|
|
\constant{NORMALIZE_WHITESPACE}, \constant{ELLIPSIS},
|
|
\constant{IGNORE_EXCEPTION_DETAIL},
|
|
\constant{REPORT_UDIFF}, \constant{REPORT_CDIFF},
|
|
\constant{REPORT_NDIFF}, \constant{REPORT_ONLY_FIRST_FAILURE},
|
|
\constant{COMPARISON_FLAGS} and \constant{REPORTING_FLAGS}
|
|
were added; by default \code{<BLANKLINE>} in expected output
|
|
matches an empty line in actual output; and doctest directives
|
|
were added]{2.4}
|
|
\versionchanged[Constant \constant{SKIP} was added]{2.5}
|
|
|
|
There's also a way to register new option flag names, although this
|
|
isn't useful unless you intend to extend \refmodule{doctest} internals
|
|
via subclassing:
|
|
|
|
\begin{funcdesc}{register_optionflag}{name}
|
|
Create a new option flag with a given name, and return the new
|
|
flag's integer value. \function{register_optionflag()} can be
|
|
used when subclassing \class{OutputChecker} or
|
|
\class{DocTestRunner} to create new options that are supported by
|
|
your subclasses. \function{register_optionflag} should always be
|
|
called using the following idiom:
|
|
|
|
\begin{verbatim}
|
|
MY_FLAG = register_optionflag('MY_FLAG')
|
|
\end{verbatim}
|
|
|
|
\versionadded{2.4}
|
|
\end{funcdesc}
|
|
|
|
\subsubsection{Warnings\label{doctest-warnings}}
|
|
|
|
\refmodule{doctest} is serious about requiring exact matches in expected
|
|
output. If even a single character doesn't match, the test fails. This
|
|
will probably surprise you a few times, as you learn exactly what Python
|
|
does and doesn't guarantee about output. For example, when printing a
|
|
dict, Python doesn't guarantee that the key-value pairs will be printed
|
|
in any particular order, so a test like
|
|
|
|
% Hey! What happened to Monty Python examples?
|
|
% Tim: ask Guido -- it's his example!
|
|
\begin{verbatim}
|
|
>>> foo()
|
|
{"Hermione": "hippogryph", "Harry": "broomstick"}
|
|
\end{verbatim}
|
|
|
|
is vulnerable! One workaround is to do
|
|
|
|
\begin{verbatim}
|
|
>>> foo() == {"Hermione": "hippogryph", "Harry": "broomstick"}
|
|
True
|
|
\end{verbatim}
|
|
|
|
instead. Another is to do
|
|
|
|
\begin{verbatim}
|
|
>>> d = foo().items()
|
|
>>> d.sort()
|
|
>>> d
|
|
[('Harry', 'broomstick'), ('Hermione', 'hippogryph')]
|
|
\end{verbatim}
|
|
|
|
There are others, but you get the idea.
|
|
|
|
Another bad idea is to print things that embed an object address, like
|
|
|
|
\begin{verbatim}
|
|
>>> id(1.0) # certain to fail some of the time
|
|
7948648
|
|
>>> class C: pass
|
|
>>> C() # the default repr() for instances embeds an address
|
|
<__main__.C instance at 0x00AC18F0>
|
|
\end{verbatim}
|
|
|
|
The \constant{ELLIPSIS} directive gives a nice approach for the last
|
|
example:
|
|
|
|
\begin{verbatim}
|
|
>>> C() #doctest: +ELLIPSIS
|
|
<__main__.C instance at 0x...>
|
|
\end{verbatim}
|
|
|
|
Floating-point numbers are also subject to small output variations across
|
|
platforms, because Python defers to the platform C library for float
|
|
formatting, and C libraries vary widely in quality here.
|
|
|
|
\begin{verbatim}
|
|
>>> 1./7 # risky
|
|
0.14285714285714285
|
|
>>> print 1./7 # safer
|
|
0.142857142857
|
|
>>> print round(1./7, 6) # much safer
|
|
0.142857
|
|
\end{verbatim}
|
|
|
|
Numbers of the form \code{I/2.**J} are safe across all platforms, and I
|
|
often contrive doctest examples to produce numbers of that form:
|
|
|
|
\begin{verbatim}
|
|
>>> 3./4 # utterly safe
|
|
0.75
|
|
\end{verbatim}
|
|
|
|
Simple fractions are also easier for people to understand, and that makes
|
|
for better documentation.
|
|
|
|
\subsection{Basic API\label{doctest-basic-api}}
|
|
|
|
The functions \function{testmod()} and \function{testfile()} provide a
|
|
simple interface to doctest that should be sufficient for most basic
|
|
uses. For a less formal introduction to these two functions, see
|
|
sections \ref{doctest-simple-testmod} and
|
|
\ref{doctest-simple-testfile}.
|
|
|
|
\begin{funcdesc}{testfile}{filename\optional{, module_relative}\optional{,
|
|
name}\optional{, package}\optional{,
|
|
globs}\optional{, verbose}\optional{,
|
|
report}\optional{, optionflags}\optional{,
|
|
extraglobs}\optional{, raise_on_error}\optional{,
|
|
parser}\optional{, encoding}}
|
|
|
|
All arguments except \var{filename} are optional, and should be
|
|
specified in keyword form.
|
|
|
|
Test examples in the file named \var{filename}. Return
|
|
\samp{(\var{failure_count}, \var{test_count})}.
|
|
|
|
Optional argument \var{module_relative} specifies how the filename
|
|
should be interpreted:
|
|
|
|
\begin{itemize}
|
|
\item If \var{module_relative} is \code{True} (the default), then
|
|
\var{filename} specifies an OS-independent module-relative
|
|
path. By default, this path is relative to the calling
|
|
module's directory; but if the \var{package} argument is
|
|
specified, then it is relative to that package. To ensure
|
|
OS-independence, \var{filename} should use \code{/} characters
|
|
to separate path segments, and may not be an absolute path
|
|
(i.e., it may not begin with \code{/}).
|
|
\item If \var{module_relative} is \code{False}, then \var{filename}
|
|
specifies an OS-specific path. The path may be absolute or
|
|
relative; relative paths are resolved with respect to the
|
|
current working directory.
|
|
\end{itemize}
|
|
|
|
Optional argument \var{name} gives the name of the test; by default,
|
|
or if \code{None}, \code{os.path.basename(\var{filename})} is used.
|
|
|
|
Optional argument \var{package} is a Python package or the name of a
|
|
Python package whose directory should be used as the base directory
|
|
for a module-relative filename. If no package is specified, then
|
|
the calling module's directory is used as the base directory for
|
|
module-relative filenames. It is an error to specify \var{package}
|
|
if \var{module_relative} is \code{False}.
|
|
|
|
Optional argument \var{globs} gives a dict to be used as the globals
|
|
when executing examples. A new shallow copy of this dict is
|
|
created for the doctest, so its examples start with a clean slate.
|
|
By default, or if \code{None}, a new empty dict is used.
|
|
|
|
Optional argument \var{extraglobs} gives a dict merged into the
|
|
globals used to execute examples. This works like
|
|
\method{dict.update()}: if \var{globs} and \var{extraglobs} have a
|
|
common key, the associated value in \var{extraglobs} appears in the
|
|
combined dict. By default, or if \code{None}, no extra globals are
|
|
used. This is an advanced feature that allows parameterization of
|
|
doctests. For example, a doctest can be written for a base class, using
|
|
a generic name for the class, then reused to test any number of
|
|
subclasses by passing an \var{extraglobs} dict mapping the generic
|
|
name to the subclass to be tested.
|
|
|
|
Optional argument \var{verbose} prints lots of stuff if true, and prints
|
|
only failures if false; by default, or if \code{None}, it's true
|
|
if and only if \code{'-v'} is in \code{sys.argv}.
|
|
|
|
Optional argument \var{report} prints a summary at the end when true,
|
|
else prints nothing at the end. In verbose mode, the summary is
|
|
detailed, else the summary is very brief (in fact, empty if all tests
|
|
passed).
|
|
|
|
Optional argument \var{optionflags} or's together option flags. See
|
|
section~\ref{doctest-options}.
|
|
|
|
Optional argument \var{raise_on_error} defaults to false. If true,
|
|
an exception is raised upon the first failure or unexpected exception
|
|
in an example. This allows failures to be post-mortem debugged.
|
|
Default behavior is to continue running examples.
|
|
|
|
Optional argument \var{parser} specifies a \class{DocTestParser} (or
|
|
subclass) that should be used to extract tests from the files. It
|
|
defaults to a normal parser (i.e., \code{\class{DocTestParser}()}).
|
|
|
|
Optional argument \var{encoding} specifies an encoding that should
|
|
be used to convert the file to unicode.
|
|
|
|
\versionadded{2.4}
|
|
|
|
\versionchanged[The parameter \var{encoding} was added]{2.5}
|
|
|
|
\end{funcdesc}
|
|
|
|
\begin{funcdesc}{testmod}{\optional{m}\optional{, name}\optional{,
|
|
globs}\optional{, verbose}\optional{,
|
|
isprivate}\optional{, report}\optional{,
|
|
optionflags}\optional{, extraglobs}\optional{,
|
|
raise_on_error}\optional{, exclude_empty}}
|
|
|
|
All arguments are optional, and all except for \var{m} should be
|
|
specified in keyword form.
|
|
|
|
Test examples in docstrings in functions and classes reachable
|
|
from module \var{m} (or module \module{__main__} if \var{m} is not
|
|
supplied or is \code{None}), starting with \code{\var{m}.__doc__}.
|
|
|
|
Also test examples reachable from dict \code{\var{m}.__test__}, if it
|
|
exists and is not \code{None}. \code{\var{m}.__test__} maps
|
|
names (strings) to functions, classes and strings; function and class
|
|
docstrings are searched for examples; strings are searched directly,
|
|
as if they were docstrings.
|
|
|
|
Only docstrings attached to objects belonging to module \var{m} are
|
|
searched.
|
|
|
|
Return \samp{(\var{failure_count}, \var{test_count})}.
|
|
|
|
Optional argument \var{name} gives the name of the module; by default,
|
|
or if \code{None}, \code{\var{m}.__name__} is used.
|
|
|
|
Optional argument \var{exclude_empty} defaults to false. If true,
|
|
objects for which no doctests are found are excluded from consideration.
|
|
The default is a backward compatibility hack, so that code still
|
|
using \method{doctest.master.summarize()} in conjunction with
|
|
\function{testmod()} continues to get output for objects with no tests.
|
|
The \var{exclude_empty} argument to the newer \class{DocTestFinder}
|
|
constructor defaults to true.
|
|
|
|
Optional arguments \var{extraglobs}, \var{verbose}, \var{report},
|
|
\var{optionflags}, \var{raise_on_error}, and \var{globs} are the same as
|
|
for function \function{testfile()} above, except that \var{globs}
|
|
defaults to \code{\var{m}.__dict__}.
|
|
|
|
Optional argument \var{isprivate} specifies a function used to
|
|
determine whether a name is private. The default function treats
|
|
all names as public. \var{isprivate} can be set to
|
|
\code{doctest.is_private} to skip over names that are
|
|
private according to Python's underscore naming convention.
|
|
\deprecated{2.4}{\var{isprivate} was a stupid idea -- don't use it.
|
|
If you need to skip tests based on name, filter the list returned by
|
|
\code{DocTestFinder.find()} instead.}
|
|
|
|
\versionchanged[The parameter \var{optionflags} was added]{2.3}
|
|
|
|
\versionchanged[The parameters \var{extraglobs}, \var{raise_on_error}
|
|
and \var{exclude_empty} were added]{2.4}
|
|
\end{funcdesc}
|
|
|
|
There's also a function to run the doctests associated with a single object.
|
|
This function is provided for backward compatibility. There are no plans
|
|
to deprecate it, but it's rarely useful:
|
|
|
|
\begin{funcdesc}{run_docstring_examples}{f, globs\optional{,
|
|
verbose}\optional{, name}\optional{,
|
|
compileflags}\optional{, optionflags}}
|
|
|
|
Test examples associated with object \var{f}; for example, \var{f} may
|
|
be a module, function, or class object.
|
|
|
|
A shallow copy of dictionary argument \var{globs} is used for the
|
|
execution context.
|
|
|
|
Optional argument \var{name} is used in failure messages, and defaults
|
|
to \code{"NoName"}.
|
|
|
|
If optional argument \var{verbose} is true, output is generated even
|
|
if there are no failures. By default, output is generated only in case
|
|
of an example failure.
|
|
|
|
Optional argument \var{compileflags} gives the set of flags that should
|
|
be used by the Python compiler when running the examples. By default, or
|
|
if \code{None}, flags are deduced corresponding to the set of future
|
|
features found in \var{globs}.
|
|
|
|
Optional argument \var{optionflags} works as for function
|
|
\function{testfile()} above.
|
|
\end{funcdesc}
|
|
|
|
\subsection{Unittest API\label{doctest-unittest-api}}
|
|
|
|
As your collection of doctest'ed modules grows, you'll want a way to run
|
|
all their doctests systematically. Prior to Python 2.4, \refmodule{doctest}
|
|
had a barely documented \class{Tester} class that supplied a rudimentary
|
|
way to combine doctests from multiple modules. \class{Tester} was feeble,
|
|
and in practice most serious Python testing frameworks build on the
|
|
\refmodule{unittest} module, which supplies many flexible ways to combine
|
|
tests from multiple sources. So, in Python 2.4, \refmodule{doctest}'s
|
|
\class{Tester} class is deprecated, and \refmodule{doctest} provides two
|
|
functions that can be used to create \refmodule{unittest} test suites from
|
|
modules and text files containing doctests. These test suites can then be
|
|
run using \refmodule{unittest} test runners:
|
|
|
|
\begin{verbatim}
|
|
import unittest
|
|
import doctest
|
|
import my_module_with_doctests, and_another
|
|
|
|
suite = unittest.TestSuite()
|
|
for mod in my_module_with_doctests, and_another:
|
|
suite.addTest(doctest.DocTestSuite(mod))
|
|
runner = unittest.TextTestRunner()
|
|
runner.run(suite)
|
|
\end{verbatim}
|
|
|
|
There are two main functions for creating \class{\refmodule{unittest}.TestSuite}
|
|
instances from text files and modules with doctests:
|
|
|
|
\begin{funcdesc}{DocFileSuite}{\optional{module_relative}\optional{,
|
|
package}\optional{, setUp}\optional{,
|
|
tearDown}\optional{, globs}\optional{,
|
|
optionflags}\optional{, parser}\optional{,
|
|
encoding}}
|
|
|
|
Convert doctest tests from one or more text files to a
|
|
\class{\refmodule{unittest}.TestSuite}.
|
|
|
|
The returned \class{\refmodule{unittest}.TestSuite} is to be run by the
|
|
unittest framework and runs the interactive examples in each file. If an
|
|
example in any file fails, then the synthesized unit test fails, and a
|
|
\exception{failureException} exception is raised showing the name of the
|
|
file containing the test and a (sometimes approximate) line number.
|
|
|
|
Pass one or more paths (as strings) to text files to be examined.
|
|
|
|
Options may be provided as keyword arguments:
|
|
|
|
Optional argument \var{module_relative} specifies how
|
|
the filenames in \var{paths} should be interpreted:
|
|
|
|
\begin{itemize}
|
|
\item If \var{module_relative} is \code{True} (the default), then
|
|
each filename specifies an OS-independent module-relative
|
|
path. By default, this path is relative to the calling
|
|
module's directory; but if the \var{package} argument is
|
|
specified, then it is relative to that package. To ensure
|
|
OS-independence, each filename should use \code{/} characters
|
|
to separate path segments, and may not be an absolute path
|
|
(i.e., it may not begin with \code{/}).
|
|
\item If \var{module_relative} is \code{False}, then each filename
|
|
specifies an OS-specific path. The path may be absolute or
|
|
relative; relative paths are resolved with respect to the
|
|
current working directory.
|
|
\end{itemize}
|
|
|
|
Optional argument \var{package} is a Python package or the name
|
|
of a Python package whose directory should be used as the base
|
|
directory for module-relative filenames. If no package is
|
|
specified, then the calling module's directory is used as the base
|
|
directory for module-relative filenames. It is an error to specify
|
|
\var{package} if \var{module_relative} is \code{False}.
|
|
|
|
Optional argument \var{setUp} specifies a set-up function for
|
|
the test suite. This is called before running the tests in each
|
|
file. The \var{setUp} function will be passed a \class{DocTest}
|
|
object. The setUp function can access the test globals as the
|
|
\var{globs} attribute of the test passed.
|
|
|
|
Optional argument \var{tearDown} specifies a tear-down function
|
|
for the test suite. This is called after running the tests in each
|
|
file. The \var{tearDown} function will be passed a \class{DocTest}
|
|
object. The setUp function can access the test globals as the
|
|
\var{globs} attribute of the test passed.
|
|
|
|
Optional argument \var{globs} is a dictionary containing the
|
|
initial global variables for the tests. A new copy of this
|
|
dictionary is created for each test. By default, \var{globs} is
|
|
a new empty dictionary.
|
|
|
|
Optional argument \var{optionflags} specifies the default
|
|
doctest options for the tests, created by or-ing together
|
|
individual option flags. See section~\ref{doctest-options}.
|
|
See function \function{set_unittest_reportflags()} below for
|
|
a better way to set reporting options.
|
|
|
|
Optional argument \var{parser} specifies a \class{DocTestParser} (or
|
|
subclass) that should be used to extract tests from the files. It
|
|
defaults to a normal parser (i.e., \code{\class{DocTestParser}()}).
|
|
|
|
Optional argument \var{encoding} specifies an encoding that should
|
|
be used to convert the file to unicode.
|
|
|
|
\versionadded{2.4}
|
|
|
|
\versionchanged[The global \code{__file__} was added to the
|
|
globals provided to doctests loaded from a text file using
|
|
\function{DocFileSuite()}]{2.5}
|
|
|
|
\versionchanged[The parameter \var{encoding} was added]{2.5}
|
|
|
|
\end{funcdesc}
|
|
|
|
\begin{funcdesc}{DocTestSuite}{\optional{module}\optional{,
|
|
globs}\optional{, extraglobs}\optional{,
|
|
test_finder}\optional{, setUp}\optional{,
|
|
tearDown}\optional{, checker}}
|
|
Convert doctest tests for a module to a
|
|
\class{\refmodule{unittest}.TestSuite}.
|
|
|
|
The returned \class{\refmodule{unittest}.TestSuite} is to be run by the
|
|
unittest framework and runs each doctest in the module. If any of the
|
|
doctests fail, then the synthesized unit test fails, and a
|
|
\exception{failureException} exception is raised showing the name of the
|
|
file containing the test and a (sometimes approximate) line number.
|
|
|
|
Optional argument \var{module} provides the module to be tested. It
|
|
can be a module object or a (possibly dotted) module name. If not
|
|
specified, the module calling this function is used.
|
|
|
|
Optional argument \var{globs} is a dictionary containing the
|
|
initial global variables for the tests. A new copy of this
|
|
dictionary is created for each test. By default, \var{globs} is
|
|
a new empty dictionary.
|
|
|
|
Optional argument \var{extraglobs} specifies an extra set of
|
|
global variables, which is merged into \var{globs}. By default, no
|
|
extra globals are used.
|
|
|
|
Optional argument \var{test_finder} is the \class{DocTestFinder}
|
|
object (or a drop-in replacement) that is used to extract doctests
|
|
from the module.
|
|
|
|
Optional arguments \var{setUp}, \var{tearDown}, and \var{optionflags}
|
|
are the same as for function \function{DocFileSuite()} above.
|
|
|
|
\versionadded{2.3}
|
|
|
|
\versionchanged[The parameters \var{globs}, \var{extraglobs},
|
|
\var{test_finder}, \var{setUp}, \var{tearDown}, and
|
|
\var{optionflags} were added; this function now uses the same search
|
|
technique as \function{testmod()}]{2.4}
|
|
\end{funcdesc}
|
|
|
|
Under the covers, \function{DocTestSuite()} creates a
|
|
\class{\refmodule{unittest}.TestSuite} out of \class{doctest.DocTestCase}
|
|
instances, and \class{DocTestCase} is a subclass of
|
|
\class{\refmodule{unittest}.TestCase}. \class{DocTestCase} isn't documented
|
|
here (it's an internal detail), but studying its code can answer questions
|
|
about the exact details of \refmodule{unittest} integration.
|
|
|
|
Similarly, \function{DocFileSuite()} creates a
|
|
\class{\refmodule{unittest}.TestSuite} out of \class{doctest.DocFileCase}
|
|
instances, and \class{DocFileCase} is a subclass of \class{DocTestCase}.
|
|
|
|
So both ways of creating a \class{\refmodule{unittest}.TestSuite} run
|
|
instances of \class{DocTestCase}. This is important for a subtle reason:
|
|
when you run \refmodule{doctest} functions yourself, you can control the
|
|
\refmodule{doctest} options in use directly, by passing option flags to
|
|
\refmodule{doctest} functions. However, if you're writing a
|
|
\refmodule{unittest} framework, \refmodule{unittest} ultimately controls
|
|
when and how tests get run. The framework author typically wants to
|
|
control \refmodule{doctest} reporting options (perhaps, e.g., specified by
|
|
command line options), but there's no way to pass options through
|
|
\refmodule{unittest} to \refmodule{doctest} test runners.
|
|
|
|
For this reason, \refmodule{doctest} also supports a notion of
|
|
\refmodule{doctest} reporting flags specific to \refmodule{unittest}
|
|
support, via this function:
|
|
|
|
\begin{funcdesc}{set_unittest_reportflags}{flags}
|
|
Set the \refmodule{doctest} reporting flags to use.
|
|
|
|
Argument \var{flags} or's together option flags. See
|
|
section~\ref{doctest-options}. Only "reporting flags" can be used.
|
|
|
|
This is a module-global setting, and affects all future doctests run by
|
|
module \refmodule{unittest}: the \method{runTest()} method of
|
|
\class{DocTestCase} looks at the option flags specified for the test case
|
|
when the \class{DocTestCase} instance was constructed. If no reporting
|
|
flags were specified (which is the typical and expected case),
|
|
\refmodule{doctest}'s \refmodule{unittest} reporting flags are or'ed into
|
|
the option flags, and the option flags so augmented are passed to the
|
|
\class{DocTestRunner} instance created to run the doctest. If any
|
|
reporting flags were specified when the \class{DocTestCase} instance was
|
|
constructed, \refmodule{doctest}'s \refmodule{unittest} reporting flags
|
|
are ignored.
|
|
|
|
The value of the \refmodule{unittest} reporting flags in effect before the
|
|
function was called is returned by the function.
|
|
|
|
\versionadded{2.4}
|
|
\end{funcdesc}
|
|
|
|
|
|
\subsection{Advanced API\label{doctest-advanced-api}}
|
|
|
|
The basic API is a simple wrapper that's intended to make doctest easy
|
|
to use. It is fairly flexible, and should meet most users' needs;
|
|
however, if you require more fine-grained control over testing, or
|
|
wish to extend doctest's capabilities, then you should use the
|
|
advanced API.
|
|
|
|
The advanced API revolves around two container classes, which are used
|
|
to store the interactive examples extracted from doctest cases:
|
|
|
|
\begin{itemize}
|
|
\item \class{Example}: A single python statement, paired with its
|
|
expected output.
|
|
\item \class{DocTest}: A collection of \class{Example}s, typically
|
|
extracted from a single docstring or text file.
|
|
\end{itemize}
|
|
|
|
Additional processing classes are defined to find, parse, and run, and
|
|
check doctest examples:
|
|
|
|
\begin{itemize}
|
|
\item \class{DocTestFinder}: Finds all docstrings in a given module,
|
|
and uses a \class{DocTestParser} to create a \class{DocTest}
|
|
from every docstring that contains interactive examples.
|
|
\item \class{DocTestParser}: Creates a \class{DocTest} object from
|
|
a string (such as an object's docstring).
|
|
\item \class{DocTestRunner}: Executes the examples in a
|
|
\class{DocTest}, and uses an \class{OutputChecker} to verify
|
|
their output.
|
|
\item \class{OutputChecker}: Compares the actual output from a
|
|
doctest example with the expected output, and decides whether
|
|
they match.
|
|
\end{itemize}
|
|
|
|
The relationships among these processing classes are summarized in the
|
|
following diagram:
|
|
|
|
\begin{verbatim}
|
|
list of:
|
|
+------+ +---------+
|
|
|module| --DocTestFinder-> | DocTest | --DocTestRunner-> results
|
|
+------+ | ^ +---------+ | ^ (printed)
|
|
| | | Example | | |
|
|
v | | ... | v |
|
|
DocTestParser | Example | OutputChecker
|
|
+---------+
|
|
\end{verbatim}
|
|
|
|
\subsubsection{DocTest Objects\label{doctest-DocTest}}
|
|
\begin{classdesc}{DocTest}{examples, globs, name, filename, lineno,
|
|
docstring}
|
|
A collection of doctest examples that should be run in a single
|
|
namespace. The constructor arguments are used to initialize the
|
|
member variables of the same names.
|
|
\versionadded{2.4}
|
|
\end{classdesc}
|
|
|
|
\class{DocTest} defines the following member variables. They are
|
|
initialized by the constructor, and should not be modified directly.
|
|
|
|
\begin{memberdesc}{examples}
|
|
A list of \class{Example} objects encoding the individual
|
|
interactive Python examples that should be run by this test.
|
|
\end{memberdesc}
|
|
|
|
\begin{memberdesc}{globs}
|
|
The namespace (aka globals) that the examples should be run in.
|
|
This is a dictionary mapping names to values. Any changes to the
|
|
namespace made by the examples (such as binding new variables)
|
|
will be reflected in \member{globs} after the test is run.
|
|
\end{memberdesc}
|
|
|
|
\begin{memberdesc}{name}
|
|
A string name identifying the \class{DocTest}. Typically, this is
|
|
the name of the object or file that the test was extracted from.
|
|
\end{memberdesc}
|
|
|
|
\begin{memberdesc}{filename}
|
|
The name of the file that this \class{DocTest} was extracted from;
|
|
or \code{None} if the filename is unknown, or if the
|
|
\class{DocTest} was not extracted from a file.
|
|
\end{memberdesc}
|
|
|
|
\begin{memberdesc}{lineno}
|
|
The line number within \member{filename} where this
|
|
\class{DocTest} begins, or \code{None} if the line number is
|
|
unavailable. This line number is zero-based with respect to the
|
|
beginning of the file.
|
|
\end{memberdesc}
|
|
|
|
\begin{memberdesc}{docstring}
|
|
The string that the test was extracted from, or `None` if the
|
|
string is unavailable, or if the test was not extracted from a
|
|
string.
|
|
\end{memberdesc}
|
|
|
|
\subsubsection{Example Objects\label{doctest-Example}}
|
|
\begin{classdesc}{Example}{source, want\optional{,
|
|
exc_msg}\optional{, lineno}\optional{,
|
|
indent}\optional{, options}}
|
|
A single interactive example, consisting of a Python statement and
|
|
its expected output. The constructor arguments are used to
|
|
initialize the member variables of the same names.
|
|
\versionadded{2.4}
|
|
\end{classdesc}
|
|
|
|
\class{Example} defines the following member variables. They are
|
|
initialized by the constructor, and should not be modified directly.
|
|
|
|
\begin{memberdesc}{source}
|
|
A string containing the example's source code. This source code
|
|
consists of a single Python statement, and always ends with a
|
|
newline; the constructor adds a newline when necessary.
|
|
\end{memberdesc}
|
|
|
|
\begin{memberdesc}{want}
|
|
The expected output from running the example's source code (either
|
|
from stdout, or a traceback in case of exception). \member{want}
|
|
ends with a newline unless no output is expected, in which case
|
|
it's an empty string. The constructor adds a newline when
|
|
necessary.
|
|
\end{memberdesc}
|
|
|
|
\begin{memberdesc}{exc_msg}
|
|
The exception message generated by the example, if the example is
|
|
expected to generate an exception; or \code{None} if it is not
|
|
expected to generate an exception. This exception message is
|
|
compared against the return value of
|
|
\function{traceback.format_exception_only()}. \member{exc_msg}
|
|
ends with a newline unless it's \code{None}. The constructor adds
|
|
a newline if needed.
|
|
\end{memberdesc}
|
|
|
|
\begin{memberdesc}{lineno}
|
|
The line number within the string containing this example where
|
|
the example begins. This line number is zero-based with respect
|
|
to the beginning of the containing string.
|
|
\end{memberdesc}
|
|
|
|
\begin{memberdesc}{indent}
|
|
The example's indentation in the containing string, i.e., the
|
|
number of space characters that precede the example's first
|
|
prompt.
|
|
\end{memberdesc}
|
|
|
|
\begin{memberdesc}{options}
|
|
A dictionary mapping from option flags to \code{True} or
|
|
\code{False}, which is used to override default options for this
|
|
example. Any option flags not contained in this dictionary are
|
|
left at their default value (as specified by the
|
|
\class{DocTestRunner}'s \member{optionflags}).
|
|
By default, no options are set.
|
|
\end{memberdesc}
|
|
|
|
\subsubsection{DocTestFinder objects\label{doctest-DocTestFinder}}
|
|
\begin{classdesc}{DocTestFinder}{\optional{verbose}\optional{,
|
|
parser}\optional{, recurse}\optional{,
|
|
exclude_empty}}
|
|
A processing class used to extract the \class{DocTest}s that are
|
|
relevant to a given object, from its docstring and the docstrings
|
|
of its contained objects. \class{DocTest}s can currently be
|
|
extracted from the following object types: modules, functions,
|
|
classes, methods, staticmethods, classmethods, and properties.
|
|
|
|
The optional argument \var{verbose} can be used to display the
|
|
objects searched by the finder. It defaults to \code{False} (no
|
|
output).
|
|
|
|
The optional argument \var{parser} specifies the
|
|
\class{DocTestParser} object (or a drop-in replacement) that is
|
|
used to extract doctests from docstrings.
|
|
|
|
If the optional argument \var{recurse} is false, then
|
|
\method{DocTestFinder.find()} will only examine the given object,
|
|
and not any contained objects.
|
|
|
|
If the optional argument \var{exclude_empty} is false, then
|
|
\method{DocTestFinder.find()} will include tests for objects with
|
|
empty docstrings.
|
|
|
|
\versionadded{2.4}
|
|
\end{classdesc}
|
|
|
|
\class{DocTestFinder} defines the following method:
|
|
|
|
\begin{methoddesc}{find}{obj\optional{, name}\optional{,
|
|
module}\optional{, globs}\optional{, extraglobs}}
|
|
Return a list of the \class{DocTest}s that are defined by
|
|
\var{obj}'s docstring, or by any of its contained objects'
|
|
docstrings.
|
|
|
|
The optional argument \var{name} specifies the object's name; this
|
|
name will be used to construct names for the returned
|
|
\class{DocTest}s. If \var{name} is not specified, then
|
|
\code{\var{obj}.__name__} is used.
|
|
|
|
The optional parameter \var{module} is the module that contains
|
|
the given object. If the module is not specified or is None, then
|
|
the test finder will attempt to automatically determine the
|
|
correct module. The object's module is used:
|
|
|
|
\begin{itemize}
|
|
\item As a default namespace, if \var{globs} is not specified.
|
|
\item To prevent the DocTestFinder from extracting DocTests
|
|
from objects that are imported from other modules. (Contained
|
|
objects with modules other than \var{module} are ignored.)
|
|
\item To find the name of the file containing the object.
|
|
\item To help find the line number of the object within its file.
|
|
\end{itemize}
|
|
|
|
If \var{module} is \code{False}, no attempt to find the module
|
|
will be made. This is obscure, of use mostly in testing doctest
|
|
itself: if \var{module} is \code{False}, or is \code{None} but
|
|
cannot be found automatically, then all objects are considered to
|
|
belong to the (non-existent) module, so all contained objects will
|
|
(recursively) be searched for doctests.
|
|
|
|
The globals for each \class{DocTest} is formed by combining
|
|
\var{globs} and \var{extraglobs} (bindings in \var{extraglobs}
|
|
override bindings in \var{globs}). A new shallow copy of the globals
|
|
dictionary is created for each \class{DocTest}. If \var{globs} is
|
|
not specified, then it defaults to the module's \var{__dict__}, if
|
|
specified, or \code{\{\}} otherwise. If \var{extraglobs} is not
|
|
specified, then it defaults to \code{\{\}}.
|
|
\end{methoddesc}
|
|
|
|
\subsubsection{DocTestParser objects\label{doctest-DocTestParser}}
|
|
\begin{classdesc}{DocTestParser}{}
|
|
A processing class used to extract interactive examples from a
|
|
string, and use them to create a \class{DocTest} object.
|
|
\versionadded{2.4}
|
|
\end{classdesc}
|
|
|
|
\class{DocTestParser} defines the following methods:
|
|
|
|
\begin{methoddesc}{get_doctest}{string, globs, name, filename, lineno}
|
|
Extract all doctest examples from the given string, and collect
|
|
them into a \class{DocTest} object.
|
|
|
|
\var{globs}, \var{name}, \var{filename}, and \var{lineno} are
|
|
attributes for the new \class{DocTest} object. See the
|
|
documentation for \class{DocTest} for more information.
|
|
\end{methoddesc}
|
|
|
|
\begin{methoddesc}{get_examples}{string\optional{, name}}
|
|
Extract all doctest examples from the given string, and return
|
|
them as a list of \class{Example} objects. Line numbers are
|
|
0-based. The optional argument \var{name} is a name identifying
|
|
this string, and is only used for error messages.
|
|
\end{methoddesc}
|
|
|
|
\begin{methoddesc}{parse}{string\optional{, name}}
|
|
Divide the given string into examples and intervening text, and
|
|
return them as a list of alternating \class{Example}s and strings.
|
|
Line numbers for the \class{Example}s are 0-based. The optional
|
|
argument \var{name} is a name identifying this string, and is only
|
|
used for error messages.
|
|
\end{methoddesc}
|
|
|
|
\subsubsection{DocTestRunner objects\label{doctest-DocTestRunner}}
|
|
\begin{classdesc}{DocTestRunner}{\optional{checker}\optional{,
|
|
verbose}\optional{, optionflags}}
|
|
A processing class used to execute and verify the interactive
|
|
examples in a \class{DocTest}.
|
|
|
|
The comparison between expected outputs and actual outputs is done
|
|
by an \class{OutputChecker}. This comparison may be customized
|
|
with a number of option flags; see section~\ref{doctest-options}
|
|
for more information. If the option flags are insufficient, then
|
|
the comparison may also be customized by passing a subclass of
|
|
\class{OutputChecker} to the constructor.
|
|
|
|
The test runner's display output can be controlled in two ways.
|
|
First, an output function can be passed to
|
|
\method{TestRunner.run()}; this function will be called with
|
|
strings that should be displayed. It defaults to
|
|
\code{sys.stdout.write}. If capturing the output is not
|
|
sufficient, then the display output can be also customized by
|
|
subclassing DocTestRunner, and overriding the methods
|
|
\method{report_start}, \method{report_success},
|
|
\method{report_unexpected_exception}, and \method{report_failure}.
|
|
|
|
The optional keyword argument \var{checker} specifies the
|
|
\class{OutputChecker} object (or drop-in replacement) that should
|
|
be used to compare the expected outputs to the actual outputs of
|
|
doctest examples.
|
|
|
|
The optional keyword argument \var{verbose} controls the
|
|
\class{DocTestRunner}'s verbosity. If \var{verbose} is
|
|
\code{True}, then information is printed about each example, as it
|
|
is run. If \var{verbose} is \code{False}, then only failures are
|
|
printed. If \var{verbose} is unspecified, or \code{None}, then
|
|
verbose output is used iff the command-line switch \programopt{-v}
|
|
is used.
|
|
|
|
The optional keyword argument \var{optionflags} can be used to
|
|
control how the test runner compares expected output to actual
|
|
output, and how it displays failures. For more information, see
|
|
section~\ref{doctest-options}.
|
|
|
|
\versionadded{2.4}
|
|
\end{classdesc}
|
|
|
|
\class{DocTestParser} defines the following methods:
|
|
|
|
\begin{methoddesc}{report_start}{out, test, example}
|
|
Report that the test runner is about to process the given example.
|
|
This method is provided to allow subclasses of
|
|
\class{DocTestRunner} to customize their output; it should not be
|
|
called directly.
|
|
|
|
\var{example} is the example about to be processed. \var{test} is
|
|
the test containing \var{example}. \var{out} is the output
|
|
function that was passed to \method{DocTestRunner.run()}.
|
|
\end{methoddesc}
|
|
|
|
\begin{methoddesc}{report_success}{out, test, example, got}
|
|
Report that the given example ran successfully. This method is
|
|
provided to allow subclasses of \class{DocTestRunner} to customize
|
|
their output; it should not be called directly.
|
|
|
|
\var{example} is the example about to be processed. \var{got} is
|
|
the actual output from the example. \var{test} is the test
|
|
containing \var{example}. \var{out} is the output function that
|
|
was passed to \method{DocTestRunner.run()}.
|
|
\end{methoddesc}
|
|
|
|
\begin{methoddesc}{report_failure}{out, test, example, got}
|
|
Report that the given example failed. This method is provided to
|
|
allow subclasses of \class{DocTestRunner} to customize their
|
|
output; it should not be called directly.
|
|
|
|
\var{example} is the example about to be processed. \var{got} is
|
|
the actual output from the example. \var{test} is the test
|
|
containing \var{example}. \var{out} is the output function that
|
|
was passed to \method{DocTestRunner.run()}.
|
|
\end{methoddesc}
|
|
|
|
\begin{methoddesc}{report_unexpected_exception}{out, test, example, exc_info}
|
|
Report that the given example raised an unexpected exception.
|
|
This method is provided to allow subclasses of
|
|
\class{DocTestRunner} to customize their output; it should not be
|
|
called directly.
|
|
|
|
\var{example} is the example about to be processed.
|
|
\var{exc_info} is a tuple containing information about the
|
|
unexpected exception (as returned by \function{sys.exc_info()}).
|
|
\var{test} is the test containing \var{example}. \var{out} is the
|
|
output function that was passed to \method{DocTestRunner.run()}.
|
|
\end{methoddesc}
|
|
|
|
\begin{methoddesc}{run}{test\optional{, compileflags}\optional{,
|
|
out}\optional{, clear_globs}}
|
|
Run the examples in \var{test} (a \class{DocTest} object), and
|
|
display the results using the writer function \var{out}.
|
|
|
|
The examples are run in the namespace \code{test.globs}. If
|
|
\var{clear_globs} is true (the default), then this namespace will
|
|
be cleared after the test runs, to help with garbage collection.
|
|
If you would like to examine the namespace after the test
|
|
completes, then use \var{clear_globs=False}.
|
|
|
|
\var{compileflags} gives the set of flags that should be used by
|
|
the Python compiler when running the examples. If not specified,
|
|
then it will default to the set of future-import flags that apply
|
|
to \var{globs}.
|
|
|
|
The output of each example is checked using the
|
|
\class{DocTestRunner}'s output checker, and the results are
|
|
formatted by the \method{DocTestRunner.report_*} methods.
|
|
\end{methoddesc}
|
|
|
|
\begin{methoddesc}{summarize}{\optional{verbose}}
|
|
Print a summary of all the test cases that have been run by this
|
|
DocTestRunner, and return a tuple \samp{(\var{failure_count},
|
|
\var{test_count})}.
|
|
|
|
The optional \var{verbose} argument controls how detailed the
|
|
summary is. If the verbosity is not specified, then the
|
|
\class{DocTestRunner}'s verbosity is used.
|
|
\end{methoddesc}
|
|
|
|
\subsubsection{OutputChecker objects\label{doctest-OutputChecker}}
|
|
|
|
\begin{classdesc}{OutputChecker}{}
|
|
A class used to check the whether the actual output from a doctest
|
|
example matches the expected output. \class{OutputChecker}
|
|
defines two methods: \method{check_output}, which compares a given
|
|
pair of outputs, and returns true if they match; and
|
|
\method{output_difference}, which returns a string describing the
|
|
differences between two outputs.
|
|
\versionadded{2.4}
|
|
\end{classdesc}
|
|
|
|
\class{OutputChecker} defines the following methods:
|
|
|
|
\begin{methoddesc}{check_output}{want, got, optionflags}
|
|
Return \code{True} iff the actual output from an example
|
|
(\var{got}) matches the expected output (\var{want}). These
|
|
strings are always considered to match if they are identical; but
|
|
depending on what option flags the test runner is using, several
|
|
non-exact match types are also possible. See
|
|
section~\ref{doctest-options} for more information about option
|
|
flags.
|
|
\end{methoddesc}
|
|
|
|
\begin{methoddesc}{output_difference}{example, got, optionflags}
|
|
Return a string describing the differences between the expected
|
|
output for a given example (\var{example}) and the actual output
|
|
(\var{got}). \var{optionflags} is the set of option flags used to
|
|
compare \var{want} and \var{got}.
|
|
\end{methoddesc}
|
|
|
|
\subsection{Debugging\label{doctest-debugging}}
|
|
|
|
Doctest provides several mechanisms for debugging doctest examples:
|
|
|
|
\begin{itemize}
|
|
\item Several functions convert doctests to executable Python
|
|
programs, which can be run under the Python debugger, \refmodule{pdb}.
|
|
\item The \class{DebugRunner} class is a subclass of
|
|
\class{DocTestRunner} that raises an exception for the first
|
|
failing example, containing information about that example.
|
|
This information can be used to perform post-mortem debugging on
|
|
the example.
|
|
\item The \refmodule{unittest} cases generated by \function{DocTestSuite()}
|
|
support the \method{debug()} method defined by
|
|
\class{\refmodule{unittest}.TestCase}.
|
|
\item You can add a call to \function{\refmodule{pdb}.set_trace()} in a
|
|
doctest example, and you'll drop into the Python debugger when that
|
|
line is executed. Then you can inspect current values of variables,
|
|
and so on. For example, suppose \file{a.py} contains just this
|
|
module docstring:
|
|
|
|
\begin{verbatim}
|
|
"""
|
|
>>> def f(x):
|
|
... g(x*2)
|
|
>>> def g(x):
|
|
... print x+3
|
|
... import pdb; pdb.set_trace()
|
|
>>> f(3)
|
|
9
|
|
"""
|
|
\end{verbatim}
|
|
|
|
Then an interactive Python session may look like this:
|
|
|
|
\begin{verbatim}
|
|
>>> import a, doctest
|
|
>>> doctest.testmod(a)
|
|
--Return--
|
|
> <doctest a[1]>(3)g()->None
|
|
-> import pdb; pdb.set_trace()
|
|
(Pdb) list
|
|
1 def g(x):
|
|
2 print x+3
|
|
3 -> import pdb; pdb.set_trace()
|
|
[EOF]
|
|
(Pdb) print x
|
|
6
|
|
(Pdb) step
|
|
--Return--
|
|
> <doctest a[0]>(2)f()->None
|
|
-> g(x*2)
|
|
(Pdb) list
|
|
1 def f(x):
|
|
2 -> g(x*2)
|
|
[EOF]
|
|
(Pdb) print x
|
|
3
|
|
(Pdb) step
|
|
--Return--
|
|
> <doctest a[2]>(1)?()->None
|
|
-> f(3)
|
|
(Pdb) cont
|
|
(0, 3)
|
|
>>>
|
|
\end{verbatim}
|
|
|
|
\versionchanged[The ability to use \code{\refmodule{pdb}.set_trace()}
|
|
usefully inside doctests was added]{2.4}
|
|
\end{itemize}
|
|
|
|
Functions that convert doctests to Python code, and possibly run
|
|
the synthesized code under the debugger:
|
|
|
|
\begin{funcdesc}{script_from_examples}{s}
|
|
Convert text with examples to a script.
|
|
|
|
Argument \var{s} is a string containing doctest examples. The string
|
|
is converted to a Python script, where doctest examples in \var{s}
|
|
are converted to regular code, and everything else is converted to
|
|
Python comments. The generated script is returned as a string.
|
|
For example,
|
|
|
|
\begin{verbatim}
|
|
import doctest
|
|
print doctest.script_from_examples(r"""
|
|
Set x and y to 1 and 2.
|
|
>>> x, y = 1, 2
|
|
|
|
Print their sum:
|
|
>>> print x+y
|
|
3
|
|
""")
|
|
\end{verbatim}
|
|
|
|
displays:
|
|
|
|
\begin{verbatim}
|
|
# Set x and y to 1 and 2.
|
|
x, y = 1, 2
|
|
#
|
|
# Print their sum:
|
|
print x+y
|
|
# Expected:
|
|
## 3
|
|
\end{verbatim}
|
|
|
|
This function is used internally by other functions (see below), but
|
|
can also be useful when you want to transform an interactive Python
|
|
session into a Python script.
|
|
|
|
\versionadded{2.4}
|
|
\end{funcdesc}
|
|
|
|
\begin{funcdesc}{testsource}{module, name}
|
|
Convert the doctest for an object to a script.
|
|
|
|
Argument \var{module} is a module object, or dotted name of a module,
|
|
containing the object whose doctests are of interest. Argument
|
|
\var{name} is the name (within the module) of the object with the
|
|
doctests of interest. The result is a string, containing the
|
|
object's docstring converted to a Python script, as described for
|
|
\function{script_from_examples()} above. For example, if module
|
|
\file{a.py} contains a top-level function \function{f()}, then
|
|
|
|
\begin{verbatim}
|
|
import a, doctest
|
|
print doctest.testsource(a, "a.f")
|
|
\end{verbatim}
|
|
|
|
prints a script version of function \function{f()}'s docstring,
|
|
with doctests converted to code, and the rest placed in comments.
|
|
|
|
\versionadded{2.3}
|
|
\end{funcdesc}
|
|
|
|
\begin{funcdesc}{debug}{module, name\optional{, pm}}
|
|
Debug the doctests for an object.
|
|
|
|
The \var{module} and \var{name} arguments are the same as for function
|
|
\function{testsource()} above. The synthesized Python script for the
|
|
named object's docstring is written to a temporary file, and then that
|
|
file is run under the control of the Python debugger, \refmodule{pdb}.
|
|
|
|
A shallow copy of \code{\var{module}.__dict__} is used for both local
|
|
and global execution context.
|
|
|
|
Optional argument \var{pm} controls whether post-mortem debugging is
|
|
used. If \var{pm} has a true value, the script file is run directly, and
|
|
the debugger gets involved only if the script terminates via raising an
|
|
unhandled exception. If it does, then post-mortem debugging is invoked,
|
|
via \code{\refmodule{pdb}.post_mortem()}, passing the traceback object
|
|
from the unhandled exception. If \var{pm} is not specified, or is false,
|
|
the script is run under the debugger from the start, via passing an
|
|
appropriate \function{execfile()} call to \code{\refmodule{pdb}.run()}.
|
|
|
|
\versionadded{2.3}
|
|
|
|
\versionchanged[The \var{pm} argument was added]{2.4}
|
|
\end{funcdesc}
|
|
|
|
\begin{funcdesc}{debug_src}{src\optional{, pm}\optional{, globs}}
|
|
Debug the doctests in a string.
|
|
|
|
This is like function \function{debug()} above, except that
|
|
a string containing doctest examples is specified directly, via
|
|
the \var{src} argument.
|
|
|
|
Optional argument \var{pm} has the same meaning as in function
|
|
\function{debug()} above.
|
|
|
|
Optional argument \var{globs} gives a dictionary to use as both
|
|
local and global execution context. If not specified, or \code{None},
|
|
an empty dictionary is used. If specified, a shallow copy of the
|
|
dictionary is used.
|
|
|
|
\versionadded{2.4}
|
|
\end{funcdesc}
|
|
|
|
The \class{DebugRunner} class, and the special exceptions it may raise,
|
|
are of most interest to testing framework authors, and will only be
|
|
sketched here. See the source code, and especially \class{DebugRunner}'s
|
|
docstring (which is a doctest!) for more details:
|
|
|
|
\begin{classdesc}{DebugRunner}{\optional{checker}\optional{,
|
|
verbose}\optional{, optionflags}}
|
|
|
|
A subclass of \class{DocTestRunner} that raises an exception as
|
|
soon as a failure is encountered. If an unexpected exception
|
|
occurs, an \exception{UnexpectedException} exception is raised,
|
|
containing the test, the example, and the original exception. If
|
|
the output doesn't match, then a \exception{DocTestFailure}
|
|
exception is raised, containing the test, the example, and the
|
|
actual output.
|
|
|
|
For information about the constructor parameters and methods, see
|
|
the documentation for \class{DocTestRunner} in
|
|
section~\ref{doctest-advanced-api}.
|
|
\end{classdesc}
|
|
|
|
There are two exceptions that may be raised by \class{DebugRunner}
|
|
instances:
|
|
|
|
\begin{excclassdesc}{DocTestFailure}{test, example, got}
|
|
An exception thrown by \class{DocTestRunner} to signal that a
|
|
doctest example's actual output did not match its expected output.
|
|
The constructor arguments are used to initialize the member
|
|
variables of the same names.
|
|
\end{excclassdesc}
|
|
\exception{DocTestFailure} defines the following member variables:
|
|
\begin{memberdesc}{test}
|
|
The \class{DocTest} object that was being run when the example failed.
|
|
\end{memberdesc}
|
|
\begin{memberdesc}{example}
|
|
The \class{Example} that failed.
|
|
\end{memberdesc}
|
|
\begin{memberdesc}{got}
|
|
The example's actual output.
|
|
\end{memberdesc}
|
|
|
|
\begin{excclassdesc}{UnexpectedException}{test, example, exc_info}
|
|
An exception thrown by \class{DocTestRunner} to signal that a
|
|
doctest example raised an unexpected exception. The constructor
|
|
arguments are used to initialize the member variables of the same
|
|
names.
|
|
\end{excclassdesc}
|
|
\exception{UnexpectedException} defines the following member variables:
|
|
\begin{memberdesc}{test}
|
|
The \class{DocTest} object that was being run when the example failed.
|
|
\end{memberdesc}
|
|
\begin{memberdesc}{example}
|
|
The \class{Example} that failed.
|
|
\end{memberdesc}
|
|
\begin{memberdesc}{exc_info}
|
|
A tuple containing information about the unexpected exception, as
|
|
returned by \function{sys.exc_info()}.
|
|
\end{memberdesc}
|
|
|
|
\subsection{Soapbox\label{doctest-soapbox}}
|
|
|
|
As mentioned in the introduction, \refmodule{doctest} has grown to have
|
|
three primary uses:
|
|
|
|
\begin{enumerate}
|
|
\item Checking examples in docstrings.
|
|
\item Regression testing.
|
|
\item Executable documentation / literate testing.
|
|
\end{enumerate}
|
|
|
|
These uses have different requirements, and it is important to
|
|
distinguish them. In particular, filling your docstrings with obscure
|
|
test cases makes for bad documentation.
|
|
|
|
When writing a docstring, choose docstring examples with care.
|
|
There's an art to this that needs to be learned---it may not be
|
|
natural at first. Examples should add genuine value to the
|
|
documentation. A good example can often be worth many words.
|
|
If done with care, the examples will be invaluable for your users, and
|
|
will pay back the time it takes to collect them many times over as the
|
|
years go by and things change. I'm still amazed at how often one of
|
|
my \refmodule{doctest} examples stops working after a "harmless"
|
|
change.
|
|
|
|
Doctest also makes an excellent tool for regression testing, especially if
|
|
you don't skimp on explanatory text. By interleaving prose and examples,
|
|
it becomes much easier to keep track of what's actually being tested, and
|
|
why. When a test fails, good prose can make it much easier to figure out
|
|
what the problem is, and how it should be fixed. It's true that you could
|
|
write extensive comments in code-based testing, but few programmers do.
|
|
Many have found that using doctest approaches instead leads to much clearer
|
|
tests. Perhaps this is simply because doctest makes writing prose a little
|
|
easier than writing code, while writing comments in code is a little
|
|
harder. I think it goes deeper than just that: the natural attitude
|
|
when writing a doctest-based test is that you want to explain the fine
|
|
points of your software, and illustrate them with examples. This in
|
|
turn naturally leads to test files that start with the simplest features,
|
|
and logically progress to complications and edge cases. A coherent
|
|
narrative is the result, instead of a collection of isolated functions
|
|
that test isolated bits of functionality seemingly at random. It's
|
|
a different attitude, and produces different results, blurring the
|
|
distinction between testing and explaining.
|
|
|
|
Regression testing is best confined to dedicated objects or files. There
|
|
are several options for organizing tests:
|
|
|
|
\begin{itemize}
|
|
\item Write text files containing test cases as interactive examples,
|
|
and test the files using \function{testfile()} or
|
|
\function{DocFileSuite()}. This is recommended, although is
|
|
easiest to do for new projects, designed from the start to use
|
|
doctest.
|
|
\item Define functions named \code{_regrtest_\textit{topic}} that
|
|
consist of single docstrings, containing test cases for the
|
|
named topics. These functions can be included in the same file
|
|
as the module, or separated out into a separate test file.
|
|
\item Define a \code{__test__} dictionary mapping from regression test
|
|
topics to docstrings containing test cases.
|
|
\end{itemize}
|