7 Commits

Author	SHA1	Message	Date
Gregory P. Smith	2e23e08f04	fix typo. \var vs \variable	2005-06-11 08:16:04 +00:00
Raymond Hettinger	68c0453418	Add untokenize() function to allow full round-trip tokenization. ... Should significantly enhance the utility of the module by supporting the creation of tools that modify the token stream and writeback the modified result.	2005-06-10 11:05:19 +00:00
Tim Peters	4efb6e9643	Turns out Neil didn't intend for *all* of his gen-branch work to get ... committed. tokenize.py: I like these changes, and have tested them extensively without even realizing it, so I just updated the docstring and the docs. tabnanny.py: Also liked this, but did a little code fiddling. I should really rewrite this to *exploit* generators, but that's near the bottom of my effort/benefit scale so doubt I'll get to it anytime soon (it would be most useful as a non-trivial example of ideal use of generators; but test_generators.py has already grown plenty of food-for-thought examples). inspect.py: I'm sure Ping intended for this to continue running even under 1.5.2, so I reverted this to the last pre-gen-branch version. The "bugfix" I checked in in-between was actually repairing a bug *introduced* by the conversion to generators, so it's OK that the reverted version doesn't reflect that checkin.	2001-06-29 23:51:08 +00:00
Ka-Ping Yee	ce7298ab42	Explain the difference between NL and NEWLINE.	2001-03-23 05:22:12 +00:00
Skip Montanaro	58177b9975	added description of NL token	2001-02-28 22:05:41 +00:00
Fred Drake	16214fb017	Fix a couple of grammatical errors reported by Detlef Lannert ... <lannert@lannert.rz.uni-duesseldorf.de>.	1999-04-23 20:00:53 +00:00
Fred Drake	6b103f1e12	Documentation for pyclbr and tokenize modules.	1999-02-18 21:06:50 +00:00