traverseda
/
cpython
1
0
Fork 0
Code Issues Pull Requests Releases Wiki Activity
38,383 Commits 9 Branches 456 Tags 1.7 GiB
Commit Graph

10 Commits

Author SHA1 Message Date
Jeremy Hylton 76467ba6d6 Bug fixes large and small for tokenize. 
Small: Always generate a NL or NEWLINE token following
       a COMMENT token.  The old code did not generate an NL token if
       the comment was on a line by itself.

Large: The output of untokenize() will now match the
       input exactly if it is passed the full token sequence.  The
       old, crufty output is still generated if a limited input
       sequence is provided, where limited means that it does not
       include position information for tokens.

Remaining bug: There is no CONTINUATION token (\) so there is no way
for untokenize() to handle such code.

Also, expanded the number of doctests in hopes of eventually removing
the old-style tests that compare against a golden file.

Bug fix candidate for Python 2.5.1. (Sigh.)
 2006-08-23 21:14:03 +00:00
Anthony Baxter c2a5a63654 PEP-0318, @decorator-style. In Guido's words: 
"@ seems the syntax that everybody can hate equally"
Implementation by Mark Russell, from SF #979728.
 2004-08-02 06:10:11 +00:00
Guido van Rossum fc26c0730c Undo Barry's change. This file is not imported, it's fed as input to 
the tokenize module by test_tokenize.py.  The FutureWarnings only
appeared during installation, and I've figured out a way to suppress
those in a different way.
 2002-08-29 15:10:30 +00:00
Barry Warsaw c6f80fd995 The test_tokenize output has changed slightly, by the addition of some 
trailing `L's.
 2002-08-29 12:56:59 +00:00
Jeremy Hylton 3bee2f6011 Update to reflect new tokenize_test.py 2001-04-13 14:55:18 +00:00
Ka-Ping Yee fa004ad36c Show '\011', '\012', and '\015' as '\t', '\n', '\r' in strings. 
Switch from octal escapes to hex escapes for other nonprintable characters.
 2001-01-24 17:19:08 +00:00
Ka-Ping Yee 10bc59320c Ugh. Sorry. Checked in the wrong file. Please ignore revision 1.3; 
it anticipates another patch i was about to propose.
 2001-01-15 22:27:06 +00:00
Ka-Ping Yee 1ff08b1243 Add tokenizer support and tests for u'', U"", uR'', Ur"", etc. 2001-01-15 22:04:30 +00:00
Guido van Rossum a50547e0c0 Track changes in tokenize.py 1998-04-03 19:56:40 +00:00
Guido van Rossum 0874f7fdaf Tests for tokenize.py (Ka-Ping Yee) 1997-10-27 22:15:06 +00:00