Commit Graph

7 Commits

Author SHA1 Message Date
Gregory P. Smith 2e23e08f04 fix typo. \var vs \variable 2005-06-11 08:16:04 +00:00
Raymond Hettinger 68c0453418 Add untokenize() function to allow full round-trip tokenization.
Should significantly enhance the utility of the module by supporting
the creation of tools that modify the token stream and writeback the
modified result.
2005-06-10 11:05:19 +00:00
Tim Peters 4efb6e9643 Turns out Neil didn't intend for *all* of his gen-branch work to get
committed.

tokenize.py:  I like these changes, and have tested them extensively
without even realizing it, so I just updated the docstring and the docs.

tabnanny.py:  Also liked this, but did a little code fiddling.  I should
really rewrite this to *exploit* generators, but that's near the bottom
of my effort/benefit scale so doubt I'll get to it anytime soon (it
would be most useful as a non-trivial example of ideal use of generators;
but test_generators.py has already grown plenty of food-for-thought
examples).

inspect.py:  I'm sure Ping intended for this to continue running even
under 1.5.2, so I reverted this to the last pre-gen-branch version.  The
"bugfix" I checked in in-between was actually repairing a bug *introduced*
by the conversion to generators, so it's OK that the reverted version
doesn't reflect that checkin.
2001-06-29 23:51:08 +00:00
Ka-Ping Yee ce7298ab42 Explain the difference between NL and NEWLINE. 2001-03-23 05:22:12 +00:00
Skip Montanaro 58177b9975 added description of NL token 2001-02-28 22:05:41 +00:00
Fred Drake 16214fb017 Fix a couple of grammatical errors reported by Detlef Lannert
<lannert@lannert.rz.uni-duesseldorf.de>.
1999-04-23 20:00:53 +00:00
Fred Drake 6b103f1e12 Documentation for pyclbr and tokenize modules. 1999-02-18 21:06:50 +00:00