Marta Gómez Macías 6715f91edc gh-102856: Python tokenizer implementation for PEP 701 (#104323) ... This commit replaces the Python implementation of the tokenize module with an implementation that reuses the real C tokenizer via a private extension module. The tokenize module now implements a compatibility layer that transforms tokens from the C tokenizer into Python tokenize tokens for backward compatibility. As the C tokenizer does not emit some tokens that the Python tokenizer provides (such as comments and non-semantic newlines), a new special mode has been added to the C tokenizer mode that currently is only used via the extension module that exposes it to the Python layer. This new mode forces the C tokenizer to emit these new extra tokens and add the appropriate metadata that is needed to match the old Python implementation. Co-authored-by: Pablo Galindo <pablogsal@gmail.com>		2023-05-21 01:03:02 +01:00
..
__phello__	…
asyncio	gh-104340: Suppress warning about unawaited exception for closed pipe stdin (#104586)	2023-05-17 16:45:11 -07:00
collections	gh-91896: Revert some very noisy DeprecationWarnings for `ByteString` (#104424)	2023-05-13 09:55:35 +01:00
concurrent	gh-94440: Fix issue of ProcessPoolExecutor shutdown hanging (#94468)	2023-03-15 21:44:52 -07:00
ctypes	Fix typo in _swappedbytes_ in ctypes comment (#102773)	2023-03-25 16:38:24 -07:00
curses	gh-60436: fix curses textbox backspace/del (#103783)	2023-04-26 22:54:07 +02:00
dbm	…
email	GH-103857: Deprecate utcnow and utcfromtimestamp (#103858)	2023-04-27 11:32:30 -06:00
encodings	…
ensurepip	gh-103872: Update bundled pip version to 23.1.2 (gh-103873)	2023-04-26 10:22:25 +00:00
html	gh-100210: Correct the comment link for unescaping HTML (#100212)	2023-02-19 11:18:12 +01:00
http	gh-69152: add method get_proxy_response_headers to HTTPConnection class (#104248)	2023-05-16 06:20:30 +00:00
idlelib	gh-104499: Fix typo. (#104598)	2023-05-18 02:35:46 +00:00
importlib	gh-103763: Implement PEP 695 (#103764)	2023-05-15 20:36:23 -07:00
json	…
lib2to3	…
logging	gh-103606: raise RuntimeError if config file is invalid or empty (#104701)	2023-05-20 22:26:49 +00:00
msilib	…
multiprocessing	gh-61460: Stronger HMAC in multiprocessing (#20380)	2023-05-20 23:33:09 +00:00
pydoc_data	gh-104301: Allow leading whitespace in disambiguated pdb statements (#104342)	2023-05-11 18:12:02 +01:00
re	…
site-packages	…
sqlite3	gh-102628: Fix sqlite3 CLI prompt in IDLE on Windows (#103945)	2023-04-27 21:22:26 +00:00
test	gh-102856: Python tokenizer implementation for PEP 701 (#104323)	2023-05-21 01:03:02 +01:00
tkinter	gh-103685: Fix tkinter.Menu.index() for Tk 8.7 (#103686)	2023-04-23 21:31:44 -04:00
tomllib	…
turtledemo	Rewrite the turtledemo makeGraphFrame method (#104224)	2023-05-06 11:09:08 -04:00
unittest	gh-68968: Correcting message display issue with assertEqual (#103937)	2023-05-04 16:37:17 -06:00
urllib	gh-102153: Start stripping C0 control and space chars in `urlsplit` (#102508)	2023-05-17 01:49:20 -07:00
venv	gh-95299: Stop installing setuptools as a part of ensurepip and venv (#101039)	2023-04-17 23:43:34 -05:00
wsgiref	…
xml	…
xmlrpc	…
zipfile	gh-103861: Fix Zip64 extensions not being properly applied in some cases (#103863)	2023-05-16 00:43:44 -07:00
zoneinfo	Fix typos in documentation and comments (GH-102374)	2023-03-02 07:26:49 -08:00
__future__.py	…
__hello__.py	…
_aix_support.py	…
_collections_abc.py	gh-91896: Deprecate collections.abc.ByteString (#102096)	2023-05-04 09:39:33 -07:00
_compat_pickle.py	…
_compression.py	…
_markupbase.py	…
_osx_support.py	…
_py_abc.py	…
_pydatetime.py	GH-84976: Re-introduce `datetime.py` and fix reprs	2023-05-03 03:09:45 -06:00
_pydecimal.py	…
_pyio.py	…
_pylong.py	gh-102515: Remove unused imports in the `Lib/` directory (#102516)	2023-03-08 11:45:38 +00:00
_sitebuiltins.py	…
_strptime.py	GH-90750: Use datetime.fromisocalendar in _strptime (#103802)	2023-04-27 10:27:27 -06:00
_threading_local.py	…
_weakrefset.py	…
abc.py	gh-87864: Use correct function definition syntax in the docs (#103312)	2023-04-11 16:50:25 +03:00
aifc.py	…
antigravity.py	…
argparse.py	gh-92248: Deprecate `type`, `choices`, `metavar` parameters of `argparse.BooleanOptionalAction` (#103678)	2023-05-19 16:44:43 +00:00
ast.py	gh-103763: Implement PEP 695 (#103764)	2023-05-15 20:36:23 -07:00
base64.py	GH-102456: Fix docstring and getopt options for base64 (gh-102457)	2023-04-02 14:05:50 +09:00
bdb.py	gh-101517: fix line number propagation in code generated for except* (#103550)	2023-04-24 21:58:51 +01:00
bisect.py	GH-102833: Mention the key function in the docstrings (GH-103009)	2023-03-25 02:19:20 -05:00
bz2.py	…
cProfile.py	gh-103935: Use `io.open_code()` when executing code in trace and profile modules (GH-103947)	2023-04-27 20:29:35 +00:00
calendar.py	gh-103822: [Calendar] change return value to enum for day and month APIs (GH-103827)	2023-05-02 13:13:31 -07:00
cgi.py	…
cgitb.py	…
chunk.py	…
cmd.py	…
code.py	gh-102778: Add sys.last_exc, deprecate sys.last_type, sys.last_value,sys.last_traceback (#102779)	2023-03-18 11:47:11 +00:00
codecs.py	…
codeop.py	…
colorsys.py	…
compileall.py	…
configparser.py	…
contextlib.py	gh-103791: Make contextlib.suppress also act on exceptions within an ExceptionGroup (#103792)	2023-04-24 22:17:02 +00:00
contextvars.py	…
copy.py	…
copyreg.py	…
crypt.py	…
csv.py	gh-67230: add quoting rules to csv module (GH-29469)	2023-04-12 15:32:30 -07:00
dataclasses.py	gh-103000: Optimise `dataclasses.asdict` for the common case (#104364)	2023-05-10 22:43:51 +01:00
datetime.py	GH-84976: Re-introduce `datetime.py` and fix reprs	2023-05-03 03:09:45 -06:00
decimal.py	…
difflib.py	…
dis.py	GH-103963: Make dis display names of args for intrinsics opcodes (#104029)	2023-05-02 19:00:17 -07:00
doctest.py	…
enum.py	gh-103596: [Enum] do not shadow mixed-in methods/attributes (GH-103600)	2023-04-18 16:19:23 -07:00
filecmp.py	…
fileinput.py	gh-101961 fileinput.hookcompressed should not set the encoding value for the binary mode (gh-102068)	2023-02-21 12:10:29 +09:00
fnmatch.py	…
fractions.py	gh-101825: Clarify that as_integer_ratio() output is always normalized (#101843)	2023-02-27 19:11:28 +00:00
ftplib.py	…
functools.py	gh-104600: Make function.__type_params__ writable (#104601)	2023-05-18 16:45:37 -07:00
genericpath.py	gh-101196: Make isdir/isfile/exists faster on Windows (GH-101324)	2023-02-08 14:34:24 +00:00
getopt.py	…
getpass.py	…
gettext.py	…
glob.py	…
graphlib.py	…
gzip.py	gh-89550: Buffer GzipFile.write to reduce execution time by ~15% (#101251)	2023-05-08 17:55:59 +00:00
hashlib.py	gh-99108: Refactor _sha256 & _sha512 into _sha2. (#101924)	2023-02-15 22:08:20 -08:00
heapq.py	…
hmac.py	…
imaplib.py	…
imghdr.py	…
inspect.py	gh-102856: Python tokenizer implementation for PEP 701 (#104323)	2023-05-21 01:03:02 +01:00
io.py	…
ipaddress.py	ipaddress: Remove non-existent ip_str param from docstring (#103461)	2023-04-14 00:44:10 -07:00
keyword.py	gh-103763: Implement PEP 695 (#103764)	2023-05-15 20:36:23 -07:00
linecache.py	…
locale.py	gh-74940: Allow fallback to UTF-8 encoding on systems with no locales installed (GH-14925)	2023-04-26 15:33:09 +09:00
lzma.py	…
mailbox.py	…
mailcap.py	…
mimetypes.py	…
modulefinder.py	…
netrc.py	…
nntplib.py	…
ntpath.py	GH-103220: Fix `ntpath.join()` of partial UNC drive with trailing slash (GH-103221)	2023-04-11 17:26:45 +01:00
nturl2path.py	…
numbers.py	…
opcode.py	gh-103865: add monitoring support to LOAD_SUPER_ATTR (#103866)	2023-05-16 10:29:00 -06:00
operator.py	…
optparse.py	…
os.py	…
pathlib.py	GH-104484: Add case_sensitive argument to `pathlib.PurePath.match()` (GH-104565)	2023-05-18 18:59:31 +01:00
pdb.py	gh-104301: Allow leading whitespace in disambiguated pdb statements (#104342)	2023-05-11 18:12:02 +01:00
pickle.py	gh-102799: use exception instance instead of sys.exc_info() (#102885)	2023-03-31 11:23:02 +01:00
pickletools.py	…
pipes.py	…
pkgutil.py	gh-97850: Deprecate `find_loader` and `get_loader` in `pkgutil` (GH-98520)	2023-05-03 16:11:54 -07:00
platform.py	gh-103977: compile re expressions in platform.py only if required (#103981)	2023-04-30 08:36:19 -07:00
plistlib.py	gh-101992: update pstlib module documentation (#102133)	2023-03-04 13:35:25 -08:00
poplib.py	…
posixpath.py	gh-101196: Make isdir/isfile/exists faster on Windows (GH-101324)	2023-02-08 14:34:24 +00:00
pprint.py	…
profile.py	gh-103935: Use `io.open_code()` when executing code in trace and profile modules (GH-103947)	2023-04-27 20:29:35 +00:00
pstats.py	…
pty.py	gh-96522: Fix deadlock in pty.spawn (#96639)	2023-05-19 13:22:43 +00:00
py_compile.py	…
pyclbr.py	…
pydoc.py	pydoc.safeimport: Use importlib.import_module instead of __import__ (GH-103118)	2023-05-03 16:26:39 -07:00
queue.py	…
quopri.py	…
random.py	Move binomialvariate() to a section for discrete distributions (GH-102955)	2023-03-23 12:10:12 -05:00
reprlib.py	…
rlcompleter.py	…
runpy.py	gh-98040: Remove just the `imp` module (#98573)	2023-04-28 16:17:58 -07:00
sched.py	…
secrets.py	…
selectors.py	…
shelve.py	…
shlex.py	…
shutil.py	gh-82814: fix shutil access error on WSL (#103790)	2023-04-25 00:45:38 +00:00
signal.py	…
site.py	gh-102038: Skip a sometimes unnecessary stat in site.py (#102039)	2023-04-02 15:47:31 -07:00
smtplib.py	…
sndhdr.py	…
socket.py	Fix broken link to MSDN (#102355)	2023-03-13 15:15:52 +02:00
socketserver.py	gh-103673: Add missing ForkingUnixStreamServer and ForkingUnixDatagramServer socketservers (#103674)	2023-04-24 22:35:52 +00:00
sre_compile.py	…
sre_constants.py	…
sre_parse.py	…
ssl.py	…
stat.py	…
statistics.py	GH-102670: Use sumprod() to simplify, speed up, and improve accuracy of statistics functions (GH-102649)	2023-03-13 20:06:43 -05:00
string.py	…
stringprep.py	…
struct.py	…
subprocess.py	gh-87474: Fix file descriptor leaks in subprocess.Popen (#96351)	2023-05-16 20:23:53 +00:00
sunau.py	…
symtable.py	…
sysconfig.py	gh-102515: Remove unused imports in the `Lib/` directory (#102516)	2023-03-08 11:45:38 +00:00
tabnanny.py	gh-102856: Python tokenizer implementation for PEP 701 (#104323)	2023-05-21 01:03:02 +01:00
tarfile.py	gh-102950: Implement PEP 706 – Filter for tarfile.extractall (#102953)	2023-04-24 10:58:06 +02:00
telnetlib.py	…
tempfile.py	gh-51574: Make tempfile.mkdtemp() always return absolute paths (#94612)	2023-04-25 16:05:59 +00:00
textwrap.py	…
this.py	…
threading.py	…
timeit.py	…
token.py	gh-102856: Python tokenizer implementation for PEP 701 (#104323)	2023-05-21 01:03:02 +01:00
tokenize.py	gh-102856: Python tokenizer implementation for PEP 701 (#104323)	2023-05-21 01:03:02 +01:00
trace.py	gh-103935: Use `io.open_code()` when executing code in trace and profile modules (GH-103947)	2023-04-27 20:29:35 +00:00
traceback.py	gh-102799: replace sys.exc_info by sys.exception in inspect and traceback modules (#104032)	2023-05-01 18:19:47 +01:00
tracemalloc.py	…
tty.py	gh-85984: New additions and improvements to the tty library. (#101832)	2023-05-19 18:13:30 +00:00
turtle.py	gh-65772: Clean-up turtle module (#104218)	2023-05-06 11:04:41 -04:00
types.py	gh-101688: Implement types.get_original_bases (#101827)	2023-04-23 20:24:30 +01:00
typing.py	Improve readability of `typing._ProtocolMeta.__instancecheck__` (#104649)	2023-05-19 14:30:02 +01:00
uu.py	gh-99889: Fix directory traversal security flaw in uu.decode() (#104096)	2023-05-09 16:01:58 +00:00
uuid.py	gh-104396: uuid.py to skip platform check for emscripten and wasi (gh-104397)	2023-05-12 07:45:55 +09:00
warnings.py	…
wave.py	…
weakref.py	…
webbrowser.py	bpo-44844: Enable detection of Microsoft Edge browser in webbrowser module (GH-29908)	2023-04-04 17:00:03 +01:00
xdrlib.py	…
zipapp.py	…
zipimport.py	…