Replace it with correct logic that raises ValueError for bad input.
Issues #8478 and #12691 reported the incorrect logic.
Add an Untokenize test case and an initial test method.
svn+ssh://pythondev@svn.python.org/python/branches/py3k
........
r85482 | antoine.pitrou | 2010-10-14 17:34:31 +0200 (jeu., 14 oct. 2010) | 4 lines
Replace the "compiler" resource with the more generic "cpu", so
as to mark CPU-heavy tests.
........
Debian sparc buildbots. Since this goes through a lot of tests
and hits the disk a lot it could be slow (especially if NFS is involved).
I'm not sure if that's the problem, but printing periodic msgs shouldn't hurt.
The code was stolen from test_compiler.
Small: Always generate a NL or NEWLINE token following
a COMMENT token. The old code did not generate an NL token if
the comment was on a line by itself.
Large: The output of untokenize() will now match the
input exactly if it is passed the full token sequence. The
old, crufty output is still generated if a limited input
sequence is provided, where limited means that it does not
include position information for tokens.
Remaining bug: There is no CONTINUATION token (\) so there is no way
for untokenize() to handle such code.
Also, expanded the number of doctests in hopes of eventually removing
the old-style tests that compare against a golden file.
Bug fix candidate for Python 2.5.1. (Sigh.)
- The doctests in decistmt() weren't run at all when
test_tokenize was run via regrtest.py.
- Some expected output in decistmt() was Windows-specific
(but nobody noticed because the doctests weren't getting
run).
- test_roundtrip() didn't actually test anything when
running the tests with -O. Now it does.
- Changed test_roundtrip() to show the name of the input
file when it fails. That would have saved a lot of
time earlier today.
- Added a bunch of comments.
Should significantly enhance the utility of the module by supporting
the creation of tools that modify the token stream and writeback the
modified result.
This file isn't meant to be executed, it's data input for test_tokenize.py.
The problem with the .py extension is that it uses "non-standard"
indentation, and it's good to test that, but reindent.py keeps wanting
to fix it. But fixing the indentation causes the expected-output file to
change, since exact line and column numbers are part of the
tokenize.tokenize() output getting tested.
imports e.g. test_support must do so using an absolute package name
such as "import test.test_support" or "from test import test_support".
This also updates the README in Lib/test, and gets rid of the
duplicate data dirctory in Lib/test/data (replaced by
Lib/email/test/data).
Now Tim and Jack can have at it. :)
and replaces them with a new API verify(). As a result the regression
suite will also perform its tests in optimization mode.
Written by Marc-Andre Lemburg. Copyright assigned to Guido van Rossum.
Terry Jan Reedy
Ezio Melotti
Meador Inge
Antoine Pitrou
Mark Dickinson
Georg Brandl
Benjamin Peterson
Amaury Forgeot d'Arc
Christian Heimes
Eric Smith
Brett Cannon
Neal Norwitz
Tim Peters
Jeremy Hylton
Raymond Hettinger
Barry Warsaw
Guido van Rossum
Fredrik Lundh
Marc-André Lemburg
Fred Drake