Commit Graph

15 Commits

Author SHA1 Message Date
Tim Peters de49583a0d Possible fix for Skip's bug 116136 (sre recursion limit hit in tokenize.py).
tokenize.py has always used naive regexps for matching string literals,
and that appears to trigger the sre recursion limit on Skip's platform (he
has very long single-line string literals).  Replaced all of tokenize.py's
string regexps with the "unrolled" forms used in IDLE, where they're known to
handle even absurd (multi-megabyte!) string literals without trouble.  See
Friedl's book for explanation (at heart, the naive regexps create a backtracking
choice point for each character in the literal, while the unrolled forms create
none).
2000-10-07 05:09:39 +00:00
Thomas Wouters e1519a1b4d Update for augmented assignment, tested & approved by Guido. 2000-08-24 21:44:52 +00:00
Fred Drake 9b8d801c37 Convert some old-style string exceptions to class exceptions. 2000-08-17 04:45:13 +00:00
Guido van Rossum a90c78b918 Differentiate between NEWLINE token (an official newline) and NL token
(a newline that the grammar ignores).
1998-04-03 16:05:38 +00:00
Guido van Rossum fefc922cef New, fixed version with proper r"..." and R"..." support from Ka-Ping. 1997-10-27 21:17:24 +00:00
Guido van Rossum 3b631775b2 Redone (by Ka-Ping) using the new re module, and adding recognition
for r"..." raw strings.  (And R"..." string support added by Guido.)
1997-10-27 20:44:15 +00:00
Guido van Rossum 2b1566be9d Correct typo in last line (test program invocation). 1997-06-03 22:05:15 +00:00
Guido van Rossum de65527e4b Ping's latest. Fixes triple quoted strings ending in odd
#backslashes, and other stuff I don't know.
1997-04-09 17:15:54 +00:00
Guido van Rossum 1aec32363f Ka-Ping's muich improved version of March 26, 1997:
#     Ignore now accepts \f as whitespace.  Operator now includes '**'.
#     Ignore and Special now accept \n or \r\n at the end of a line.
#     Imagnumber is new.  Expfloat is corrected to reject '0e4'.
1997-04-08 14:24:39 +00:00
Guido van Rossum b5dc5e3d7e Added support for imaginary constants (e.g. 0j, 1j, 1.0j). 1997-03-10 23:17:01 +00:00
Guido van Rossum b51eaa183e Fixed doc string, added __version__, fixed 1 bug. 1997-03-07 00:21:55 +00:00
Guido van Rossum fc6f5339a9 Ka-Ping's version. 1997-03-07 00:21:12 +00:00
Guido van Rossum b31c7f732a * test_select.py: (some) tests for built-in select module
* test_grammar.py, testall.out: added test for funny things in string literals
* token.py, symbol.py: definitions used with built-in parser module.
* tokenize.py: added double-quote recognition
1993-11-11 10:31:23 +00:00
Guido van Rossum 10d10ffb1b Change the order in which Floatnumber and Intnumber are tried
so it will correctly recognize floats.
Fix the test program so it works again.
1992-03-16 18:30:24 +00:00
Guido van Rossum 4d8e859e8f Initial revision 1992-01-01 19:34:47 +00:00