catch_warnings(), and clean up the API.
While expanding the test suite, a bug was found where a warning about the
'line' argument to showwarning() was not letting functions with '*args' go
without a warning.
Closes issue 3602.
Code review by Benjamin Peterson.
triggered warnings should be captured. This allows the context manager to be
used to just prevent the internal state of the 'warnings' framework and thus
allow triggered warnings to be displayed.
'warnings' code in places where it was previously not possible (e.g., the
parser). It could also potentially lead to a speed-up in interpreter start-up
if the C version of the code (_warnings) is imported over the use of the
Python version in key places.
Closes issue #1631171.
to listen on in network-oriented tests has been refined in an effort to
facilitate running multiple instances of the entire regression test suite
in parallel without issue. test_support.bind_port() has been fixed such
that it will always return a unique port -- which wasn't always the case
with the previous implementation, especially if socket options had been
set that affected address reuse (i.e. SO_REUSEADDR, SO_REUSEPORT). The
new implementation of bind_port() will actually raise an exception if it
is passed an AF_INET/SOCK_STREAM socket with either the SO_REUSEADDR or
SO_REUSEPORT socket option set. Furthermore, if available, bind_port()
will set the SO_EXCLUSIVEADDRUSE option on the socket it's been passed.
This currently only applies to Windows. This option prevents any other
sockets from binding to the host/port we've bound to, thus removing the
possibility of the 'non-deterministic' behaviour, as Microsoft puts it,
that occurs when a second SOCK_STREAM socket binds and accepts to a
host/port that's already been bound by another socket. The optional
preferred port parameter to bind_port() has been removed. Under no
circumstances should tests be hard coding ports!
test_support.find_unused_port() has also been introduced, which will pass
a temporary socket object to bind_port() in order to obtain an unused port.
The temporary socket object is then closed and deleted, and the port is
returned. This method should only be used for obtaining an unused port
in order to pass to an external program (i.e. the -accept [port] argument
to openssl's s_server mode) or as a parameter to a server-oriented class
that doesn't give you direct access to the underlying socket used.
Finally, test_support.HOST has been introduced, which should be used for
the host argument of any relevant socket calls (i.e. bind and connect).
The following tests were updated to following the new conventions:
test_socket, test_smtplib, test_asyncore, test_ssl, test_httplib,
test_poplib, test_ftplib, test_telnetlib, test_socketserver,
test_asynchat and test_socket_ssl.
It is now possible for multiple instances of the regression test suite to
run in parallel without issue.
run the test simultaneously. The simplest thing I found that worked
on both Windows and Unix was to use the PID. It's unique so should be
sufficient. This should prevent many of the spurious failures of
the automated tests since they run as different users.
Also cleanup the directory consistenly in the tearDown methods.
It would be nice if someone ensured that the directories are always
created with a consistent name.
This should make the tests more robust at the expense of allowing
tests to be sloppier by not requiring them to cleanup after themselves.
(It will legitamitely help when running two test suites simultaneously
or if another process is already using one of the predefined ports.)
Also simplifies (slightLy) the exception handling elsewhere.
test.test_support.catch_warning is more full-featured and provides the same
functionality.
Since guard_warnings_filter was added in 2.6 there is no
backwards-compatibility issues.
If the call to requires() doesn't precede the filesystem check, we get the following situation:
1. ./python Lib/test/regrtest.py test_foo # test needs urlfetch, not enabled, so skipped
2. ./python Lib/test/regrtest.py -u urlfetch test_foo # test runs
3. ./python Lib/test/regrtest.py test_foo # test runs (!)
By moving the call to requires() *before* the filesystem check, the fact that fetched files are cached on the local disk becomes an implementation detail, rather than a semantics-changing point of note.
nests test.test_support.TransientResource context managers that capture
exceptions raised when the Internet connection is flaky.
Initially using in test_socket_ssl but should probably be expanded to cover any
test that should not raise the captured exceptions if the Internet connection
works.
surround calls to resources that may or may not be available. Specifying the
expected exception and attributes to be raised if the resource is not available
prevents overly broad catching of exceptions.
This is meant to help suppress spurious failures by raising
test.test_support.ResourceDenied if the exception matches. It would probably
be good to go through the various network tests and surround the calls to catch
connection timeouts (as done with test_socket_ssl in this commit).
be called at the end of each test that spawns children (perhaps it
should be called from regrtest instead?). This will hopefully prevent
some of the unexplained failures in the buildbots (hppa and alpha)
during tests that spawn children. The problems were not reproducible.
There were many zombies that remained at the end of several tests.
In the worst case, this shouldn't cause any more problems,
though it may not help either. Time will tell.
issues that crop up from time to time, but this change seems to have been
pretty stable (no spurious warnings) for about a week.
Other modules which use threads may require similar use of
threading_setup/threading_cleanup from test_support.
tuples. Lots to be added, still, but this will give big-memory people
something to play with in 2.5 alpha 2, and hopefully get more people to
write these tests.
and test_support.run_classtests() into run_unittest()
and use it wherever possible.
Also don't use "from test.test_support import ...", but
"from test import test_support" in a few spots.
From SF patch #662807.
exception, ResourceDenied. This is used to distinguish between tests that
are skipped for other reasons (platform support, missing data, etc.) from
those that are skipped because a "resource" has not been enabled. This
prevents those tests from being reported as unexpected skips for the
platform; those should only be considered unexpected skips if the resource
were enabled.
imports of test modules now import from the test package. Other
related oddities are also fixed (like DeprecationWarning filters that
weren't specifying the full import part, etc.). Also did a general
code cleanup to remove all "from test.test_support import *"'s. Other
from...import *'s weren't changed.
Taught doctest about static methods, class methods, and property docstrings
in new-style classes. As for inspect.py/pydoc.py before it, the new stuff
needed didn't really fit into the old architecture (but was less of a
strain to force-fit here).
New-style class docstrings still aren't found, but that's the subject
of a different bug and I want to fix that right instead of hacking around
it in doctest.
the local save/modify/restore of sys.stdout, but add machinery so that
regrtest can tell test_support the value of sys.stdout at the time
regrtest.main() started, and test_support can pass that out later to anyone
who needs a "visible" stdout.
the first difference, let the test run till completion, then gather
all the output and compare it to the expected output using difflib.
XXX Still to do: produce diff output that only shows the sections that
differ; currently it produces ndiff-style output because that's the
easiest to produce with difflib, but this becomes a liability when the
output is voluminous and there are only a few differences.
their own test suite from a multitude of classes (like test_email.py
will be doing).
run_unittest(): Call run_suite() after making a suite from the
testclass.
horridly inefficient hack in regrtest's Compare class, but it's about as
clean as can be: regrtest has to set up the Compare instance before
importing a test module, and by the time the module *is* imported it's too
late to change that decision. The good news is that the more tests we
convert to unittest and doctest, the less the inefficiency here matters.
Even now there are few tests with large expected-output files (the new
cost here is a Python-level call per .write() when there's an expected-
output file).
saving instead a traceback string, but test_support's run_unittest was
still peeking into unittest internals and trying to pick apart unittest's
errors and failures vectors as if they contained exc_info() tuples instead
of strings.
Whatever, when a unittest-based test failed, test_support blew up. I'm
not sure this is the right way to fix it; it simply gets me unstuck.
(1) Allow multiple -u options to extend each other (and the initial
value of use_resources passed into regrtest.main()).
(2) When a test is run stand-alone (not via regrtest.py), needed
resources are always granted.
requires(): New function which can be used to `assert' that a specific
-u/--use resource flag is present. Raises a TestSkipped if not.
This is used in test_largefile and test_socket_ssl to enable
external or resource consumptive tests that are normally
disabled.
- Do not compile unicodeobject, unicodectype, and unicodedata if Unicode is disabled
- check for Py_USING_UNICODE in all places that use Unicode functions
- disables unicode literals, and the builtin functions
- add the types.StringTypes list
- remove Unicode literals from most tests.
The comment following used to say:
/* We use ~hash instead of hash, as degenerate hash functions, such
as for ints <sigh>, can have lots of leading zeros. It's not
really a performance risk, but better safe than sorry.
12-Dec-00 tim: so ~hash produces lots of leading ones instead --
what's the gain? */
That is, there was never a good reason for doing it. And to the contrary,
as explained on Python-Dev last December, it tended to make the *sum*
(i + incr) & mask (which is the first table index examined in case of
collison) the same "too often" across distinct hashes.
Changing to the simpler "i = hash & mask" reduced the number of string-dict
collisions (== # number of times we go around the lookup for-loop) from about
6 million to 5 million during a full run of the test suite (these are
approximate because the test suite does some random stuff from run to run).
The number of collisions in non-string dicts also decreased, but not as
dramatically.
Note that this may, for a given dict, change the order (wrt previous
releases) of entries exposed by .keys(), .values() and .items(). A number
of std tests suffered bogus failures as a result. For dicts keyed by
small ints, or (less so) by characters, the order is much more likely to be
in increasing order of key now; e.g.,
>>> d = {}
>>> for i in range(10):
... d[i] = i
...
>>> d
{0: 0, 1: 1, 2: 2, 3: 3, 4: 4, 5: 5, 6: 6, 7: 7, 8: 8, 9: 9}
>>>
Unfortunately. people may latch on to that in small examples and draw a
bogus conclusion.
test_support.py
Moved test_extcall's sortdict() into test_support, made it stronger,
and imported sortdict into other std tests that needed it.
test_unicode.py
Excluced cp875 from the "roundtrip over range(128)" test, because
cp875 doesn't have a well-defined inverse for unicode("?", "cp875").
See Python-Dev for excruciating details.
Cookie.py
Chaged various output functions to sort dicts before building
strings from them.
test_extcall
Fiddled the expected-result file. This remains sensitive to native
dict ordering, because, e.g., if there are multiple errors in a
keyword-arg dict (and test_extcall sets up many cases like that), the
specific error Python complains about first depends on native dict
ordering.
Benjamin Peterson
Nick Coghlan
Brett Cannon
Neal Norwitz
Georg Brandl
Alexandre Vassalotti
Skip Montanaro
Trent Nelson
Christian Heimes
Collin Winter
Walter Dörwald
Tim Peters
Armin Rigo
Thomas Wouters
Hye-Shik Chang
Martin v. Löwis
Mark Hammond
Raymond Hettinger
Fred Drake
Finn Bock
Barry Warsaw
Guido van Rossum
Steve Purcell