Commit Graph

3 Commits

Author SHA1 Message Date
Ned Deily 2ea6fccf64 Issue #12587: Correct faulty test file and reference in test_tokenize.
(Patch by Robert Xiao)
2011-07-19 16:15:27 -07:00
Martin v. Löwis 3c6938d22b Ran svneol.py 2008-06-13 23:34:35 +00:00
Trent Nelson 428de65ca9 - Issue #719888: Updated tokenize to use a bytes API. generate_tokens has been
renamed tokenize and now works with bytes rather than strings. A new
  detect_encoding function has been added for determining source file encoding
  according to PEP-0263. Token sequences returned by tokenize always start
  with an ENCODING token which specifies the encoding used to decode the file.
  This token is used to encode the output of untokenize back to bytes.

Credit goes to Michael "I'm-going-to-name-my-first-child-unittest" Foord from Resolver Systems for this work.
2008-03-18 22:41:35 +00:00