cpython

Commit Graph

Author	SHA1	Message	Date
Lysandros Nikolaou	a1ac5590e0	gh-107450: Check for overflow in the tokenizer and fix overflow test (#110832 ) Co-authored-by: Filipe Laíns <lains@riseup.net> Co-authored-by: Serhiy Storchaka <storchaka@gmail.com>	2023-10-16 16:42:49 +02:00
Lysandros Nikolaou	01481f2dc1	gh-104169: Refactor tokenizer into lexer and wrappers (#110684 ) * The lexer, which include the actual lexeme producing logic, goes into the `lexer` directory. * The wrappers, one wrapper per input mode (file, string, utf-8, and readline), go into the `tokenizer` directory and include logic for creating a lexer instance and managing the buffer for different modes. --------- Co-authored-by: Pablo Galindo <pablogsal@gmail.com> Co-authored-by: blurb-it[bot] <43283697+blurb-it[bot]@users.noreply.github.com>	2023-10-11 15:14:44 +00:00

Author

SHA1

Message

Date

Lysandros Nikolaou

a1ac5590e0

gh-107450: Check for overflow in the tokenizer and fix overflow test (#110832 )

Co-authored-by: Filipe Laíns <lains@riseup.net>
Co-authored-by: Serhiy Storchaka <storchaka@gmail.com>

2023-10-16 16:42:49 +02:00

Lysandros Nikolaou

01481f2dc1

gh-104169: Refactor tokenizer into lexer and wrappers (#110684 )

* The lexer, which include the actual lexeme producing logic, goes into
  the `lexer` directory.
* The wrappers, one wrapper per input mode (file, string, utf-8, and
  readline), go into the `tokenizer` directory and include logic for
  creating a lexer instance and managing the buffer for different modes.
---------

Co-authored-by: Pablo Galindo <pablogsal@gmail.com>
Co-authored-by: blurb-it[bot] <43283697+blurb-it[bot]@users.noreply.github.com>

2023-10-11 15:14:44 +00:00

2 Commits