Mirror
/
cpython
mirror of https://github.com/python/cpython
4
1
Fork 0
Code Issues Releases Wiki Activity
cpython/Misc/NEWS.d
History
Marta Gómez Macías 6715f91edc
gh-102856: Python tokenizer implementation for PEP 701 (#104323) 
This commit replaces the Python implementation of the tokenize module with an implementation
that reuses the real C tokenizer via a private extension module. The tokenize module now implements
a compatibility layer that transforms tokens from the C tokenizer into Python tokenize tokens for backward
compatibility.

As the C tokenizer does not emit some tokens that the Python tokenizer provides (such as comments and non-semantic newlines), a new special mode has been added to the C tokenizer mode that currently is only used via
the extension module that exposes it to the Python layer. This new mode forces the C tokenizer to emit these new extra tokens and add the appropriate metadata that is needed to match the old Python implementation.

Co-authored-by: Pablo Galindo <pablogsal@gmail.com>
 		2023-05-21 01:03:02 +01:00
..
next gh-102856: Python tokenizer implementation for PEP 701 (#104323) 2023-05-21 01:03:02 +01:00
3.5.0.rst
…
3.5.0a1.rst gh-93738: Documentation C syntax (:c:type:<C type> -> :c:expr:<C type>) (#97768) 2022-10-05 11:01:14 -07:00
3.5.0a2.rst
…
3.5.0a3.rst
…
3.5.0a4.rst
…
3.5.0b1.rst
…
3.5.0b2.rst
…
3.5.0b3.rst
…
3.5.0b4.rst
…
3.5.0rc1.rst
…
3.5.0rc2.rst
…
3.5.0rc3.rst
…
3.5.0rc4.rst
…
3.5.1.rst
…
3.5.1rc1.rst gh-95454: Replace truthy/falsy with true/false (GH-95456) 2022-07-30 00:42:21 -07:00
3.5.2.rst
…
3.5.2rc1.rst NEWS: Reorder items by section (GH-92373) 2022-05-06 13:58:58 +09:00
3.5.3.rst
…
3.5.3rc1.rst gh-95454: Replace truthy/falsy with true/false (GH-95456) 2022-07-30 00:42:21 -07:00
3.5.4.rst
…
3.5.4rc1.rst
…
3.5.5.rst
…
3.5.5rc1.rst
…
3.6.0.rst
…
3.6.0a1.rst NEWS: Reorder items by section (GH-92373) 2022-05-06 13:58:58 +09:00
3.6.0a2.rst NEWS: Reorder items by section (GH-92373) 2022-05-06 13:58:58 +09:00
3.6.0a3.rst NEWS: Reorder items by section (GH-92373) 2022-05-06 13:58:58 +09:00
3.6.0a4.rst
…
3.6.0b1.rst
…
3.6.0b2.rst gh-95454: Replace truthy/falsy with true/false (GH-95456) 2022-07-30 00:42:21 -07:00
3.6.0b3.rst
…
3.6.0b4.rst
…
3.6.0rc1.rst
…
3.6.0rc2.rst
…
3.6.1.rst
…
3.6.1rc1.rst
…
3.6.2.rst
…
3.6.2rc1.rst NEWS: Reorder items by section (GH-92373) 2022-05-06 13:58:58 +09:00
3.6.2rc2.rst
…
3.6.3.rst
…
3.6.3rc1.rst
…
3.6.4.rst
…
3.6.4rc1.rst [docs] Improve the markup of powers (GH-28598) 2021-09-28 22:40:57 +02:00
3.6.5.rst
…
3.6.5rc1.rst
…
3.6.6.rst
…
3.6.6rc1.rst
…
3.7.0.rst
…
3.7.0a1.rst GH-103484: Fix redirected permanently URLs (#104001) 2023-05-02 09:34:44 +03:00
3.7.0a2.rst
…
3.7.0a3.rst gh-92448: Update the documentation builder to render the GitHub issue. (GH-92449) 2022-05-08 21:50:40 +09:00
3.7.0a4.rst [docs] Use full names for time units (GH-28611) 2021-09-29 12:09:56 +03:00
3.7.0b1.rst
…
3.7.0b2.rst GH-103484: Fix broken links reported by linkcheck (#103608) 2023-04-22 08:24:47 -06:00
3.7.0b3.rst
…
3.7.0b4.rst
…
3.7.0b5.rst
…
3.7.0rc1.rst
…
3.8.0a1.rst GH-98040: Suppress cross-references to the removed ``imp`` module (#104131) 2023-05-03 13:29:42 +01:00
3.8.0a2.rst Docs: remove redundant "adverb-adjective" hyphens from compound modifiers (GH-94551) 2022-07-05 11:16:10 +02:00
3.8.0a3.rst
…
3.8.0a4.rst gh-93738: Documentation C syntax (:c:type: to :c:expr:, misc. cases) (#97775) 2022-10-04 16:32:27 -07:00
3.8.0b1.rst [codemod] Fix non-matching bracket pairs (GH-28473) 2021-09-22 01:09:00 +02:00
3.9.0a1.rst GH-103484: Fix broken links reported by linkcheck (#103608) 2023-04-22 08:24:47 -06:00
3.9.0a2.rst GH-103484: Fix broken links reported by linkcheck (#103608) 2023-04-22 08:24:47 -06:00
3.9.0a3.rst docs: Change links to label refs (#98454) 2022-10-25 20:26:28 -07:00
3.9.0a4.rst GH-103484: Fix broken links reported by linkcheck (#103608) 2023-04-22 08:24:47 -06:00
3.9.0a5.rst gh-85454: Remove links from historical mentions of distutils (GH-95192) 2022-07-25 12:20:09 +02:00
3.9.0a6.rst
…
3.9.0b1.rst gh-93738: Documentation C syntax (:c:type:<C type> -> :c:expr:<C type>) (#97768) 2022-10-05 11:01:14 -07:00
3.10.0a1.rst gh-85454: Remove links from historical mentions of distutils (GH-95192) 2022-07-25 12:20:09 +02:00
3.10.0a2.rst gh-101100: Test docs in nit-picky mode (#102513) 2023-03-24 13:23:35 +02:00
3.10.0a3.rst Docs: remove redundant "adverb-adjective" hyphens from compound modifiers (GH-94551) 2022-07-05 11:16:10 +02:00
3.10.0a4.rst Docs: remove redundant "adverb-adjective" hyphens from compound modifiers (GH-94551) 2022-07-05 11:16:10 +02:00
3.10.0a5.rst GH-97850: Suppress cross-references to removed ``importlib.util`` functions (#104134) 2023-05-03 14:13:03 +01:00
3.10.0a6.rst gh-92448: Update the documentation builder to render the GitHub issue. (GH-92449) 2022-05-08 21:50:40 +09:00
3.10.0a7.rst gh-92448: Update the documentation builder to render the GitHub issue. (GH-92449) 2022-05-08 21:50:40 +09:00
3.10.0b1.rst gh-85454: Remove links from historical mentions of distutils (GH-95192) 2022-07-25 12:20:09 +02:00
3.11.0a1.rst gh-85454: Remove links from historical mentions of distutils (GH-95192) 2022-07-25 12:20:09 +02:00
3.11.0a2.rst Fix name of removed `inspect.Signature.from_builtin` method in 3.11.0a2 changelog (#100525) 2022-12-26 17:04:18 +05:30
3.11.0a3.rst Remove a NEWS entry for bpo-45878 (GH-30259) 2021-12-26 12:38:19 +02:00
3.11.0a4.rst Docs: remove redundant "adverb-adjective" hyphens from compound modifiers (GH-94551) 2022-07-05 11:16:10 +02:00
3.11.0a5.rst Python 3.11.0a5 2022-02-03 18:37:08 +00:00
3.11.0a6.rst Fix duplicated words in the docs (GH-102629) 2023-03-12 11:03:59 -07:00
3.11.0a7.rst no-issue: Fix typo in 3.11.0a7.rst (gh-96547) 2022-09-04 22:17:26 +09:00
3.11.0b1.rst Docs: remove redundant "adverb-adjective" hyphens from compound modifiers (GH-94551) 2022-07-05 11:16:10 +02:00
3.12.0a1.rst GH-97850: Suppress cross-references to removed ``importlib.util`` functions (#104134) 2023-05-03 14:13:03 +01:00
3.12.0a2.rst Remove misformatted exclamation marks in docs (#102694) 2023-03-15 09:06:32 +02:00
3.12.0a3.rst Python 3.12.0a3 2022-12-06 19:33:02 +01:00
3.12.0a4.rst Python 3.12.0a4 2023-01-10 13:09:15 +01:00
3.12.0a5.rst Python 3.12.0a5 2023-02-07 13:21:15 +01:00
3.12.0a6.rst GH-84783: Mention Author for GH-101264 (make slices hashable) (#103146) 2023-03-31 09:54:17 -04:00
3.12.0a7.rst Python 3.12.0a7 2023-04-04 17:52:42 +02:00