Minor modernization and readability improvement to the tokenizer example (GH-19558)

This commit is contained in:
Raymond Hettinger 2020-04-16 19:54:13 -07:00 committed by GitHub
parent a75e730075
commit bf1a81258c
No known key found for this signature in database
GPG Key ID: 4AEE18F83AFDEB23
1 changed files with 6 additions and 2 deletions

View File

@ -1617,10 +1617,14 @@ The text categories are specified with regular expressions. The technique is
to combine those into a single master regular expression and to loop over
successive matches::
import collections
from typing import NamedTuple
import re
Token = collections.namedtuple('Token', ['type', 'value', 'line', 'column'])
class Token(NamedTuple):
type: str
value: str
line: int
column: int
def tokenize(code):
keywords = {'IF', 'THEN', 'ENDIF', 'FOR', 'NEXT', 'GOSUB', 'RETURN'}