Mirror
/
cpython
mirror of https://github.com/python/cpython
4
1
Fork 0
Code Issues Releases Wiki Activity
cpython/Python
History
Marta Gómez Macías 6715f91edc
gh-102856: Python tokenizer implementation for PEP 701 (#104323) 
This commit replaces the Python implementation of the tokenize module with an implementation
that reuses the real C tokenizer via a private extension module. The tokenize module now implements
a compatibility layer that transforms tokens from the C tokenizer into Python tokenize tokens for backward
compatibility.

As the C tokenizer does not emit some tokens that the Python tokenizer provides (such as comments and non-semantic newlines), a new special mode has been added to the C tokenizer mode that currently is only used via
the extension module that exposes it to the Python layer. This new mode forces the C tokenizer to emit these new extra tokens and add the appropriate metadata that is needed to match the old Python implementation.

Co-authored-by: Pablo Galindo <pablogsal@gmail.com>
 		2023-05-21 01:03:02 +01:00
..
clinic gh-102856: Python tokenizer implementation for PEP 701 (#104323) 2023-05-21 01:03:02 +01:00
deepfreeze gh-97669: Create Tools/build/ directory (#97963) 2022-10-17 12:01:00 +02:00
frozen_modules gh-97669: Create Tools/build/ directory (#97963) 2022-10-17 12:01:00 +02:00
Python-ast.c gh-103763: Implement PEP 695 (#103764) 2023-05-15 20:36:23 -07:00
Python-tokenize.c gh-102856: Python tokenizer implementation for PEP 701 (#104323) 2023-05-21 01:03:02 +01:00
README
…
_warnings.c gh-104341: Adjust tstate_must_exit() to Respect Interpreter Finalization (gh-104437) 2023-05-15 13:59:26 -06:00
adaptive.md GH-98686: Get rid of "adaptive" and "quick" instructions (GH-99182) 2022-11-09 10:50:09 -08:00
asdl.c bpo-43244: Remove ast.h, asdl.h, Python-ast.h headers (GH-24933) 2021-03-23 20:47:40 +01:00
asm_trampoline.S gh-96143: Move the perf trampoline files to the Python directory (#98675) 2022-10-25 23:34:22 +01:00
assemble.c gh-87092: avoid gcc warning on uninitialized struct field in assemble.c (#104460) 2023-05-14 12:03:30 +01:00
ast.c gh-103763: Implement PEP 695 (#103764) 2023-05-15 20:36:23 -07:00
ast_opt.c gh-103763: Implement PEP 695 (#103764) 2023-05-15 20:36:23 -07:00
ast_unparse.c gh-100227: Move _str_replace_inf to PyInterpreterState (gh-102333) 2023-02-28 14:16:39 -07:00
bltinmodule.c gh-103886: Improve `builtins.__doc__` (#104179) 2023-05-06 19:05:34 -07:00
bootstrap_hash.c gh-81057: Move More Globals in Core Code to _PyRuntimeState (gh-99516) 2022-11-16 09:37:14 -07:00
bytecodes.c GH-104580: Don't cache eval breaker in interpreter (GH-104581) 2023-05-18 10:08:33 +01:00
ceval.c GH-104580: Don't cache eval breaker in interpreter (GH-104581) 2023-05-18 10:08:33 +01:00
ceval_gil.c gh-104341: Adjust tstate_must_exit() to Respect Interpreter Finalization (gh-104437) 2023-05-15 13:59:26 -06:00
ceval_macros.h GH-104580: Don't cache eval breaker in interpreter (GH-104581) 2023-05-18 10:08:33 +01:00
codecs.c gh-77757: replace exception wrapping by PEP-678 notes in typeobject's __set_name__ (#103402) 2023-04-11 11:53:06 +01:00
compile.c gh-104619: never leak comprehension locals to outer locals() (#104637) 2023-05-18 18:50:24 -07:00
condvar.h gh-74953: _PyThread_cond_after() uses _PyTime_t (#94056) 2022-06-21 15:45:49 +02:00
context.c gh-94673: Hide Objects in PyTypeObject Behind Accessors (gh-104074) 2023-05-01 20:34:43 -06:00
dtoa.c gh-104263: Rely on Py_NAN and introduce Py_INFINITY (GH-104202) 2023-05-10 17:44:52 +01:00
dup2.c gh-95174: Handle missing dup() and constants in WASI (GH-95229) 2022-07-26 11:16:51 +02:00
dynamic_annotations.c
…
dynload_hpux.c bpo-44959: Add fallback to extension modules with '.sl' suffix on HP-UX (GH-27857) 2021-09-08 14:43:00 +02:00
dynload_shlib.c gh-101758: Clean Up Uses of Import State (gh-101919) 2023-02-15 15:32:31 -07:00
dynload_stub.c
…
dynload_win.c gh-102255: Improve build support for Windows API partitions (GH-102256) 2023-03-09 21:09:12 +00:00
emscripten_signal.c bpo-47176: Interrupt handling for wasm32-emscripten builds without pthreads (GH-32209) 2022-04-03 22:58:52 +02:00
errors.c gh-94673: Properly Initialize and Finalize Static Builtin Types for Each Interpreter (gh-104072) 2023-05-01 19:36:00 -06:00
fileutils.c gh-99726: Improves correctness of stat results for Windows, and uses faster API when available (GH-102149) 2023-03-16 17:27:21 +00:00
flowgraph.c gh-104615: don't make unsafe swaps in apply_static_swaps (#104620) 2023-05-18 21:22:03 +00:00
formatter_unicode.c gh-89653: Use int type for Unicode kind (#92704) 2022-05-13 12:41:05 +02:00
frame.c GH-96803: Document and test new unstable internal frame API functions (GH-104211) 2023-05-18 10:10:15 +01:00
frozen.c gh-100227: Only Use deepfreeze for the Main Interpreter (gh-103794) 2023-04-24 21:48:05 +00:00
frozenmain.c gh-93103: Parser uses PyConfig.parser_debug instead of Py_DebugFlag (#93106) 2022-05-24 22:35:08 +02:00
future.c gh-87092: do not allocate PyFutureFeatures dynamically (GH-98913) 2022-11-02 15:13:07 +00:00
generated_cases.c.h GH-104580: Don't cache eval breaker in interpreter (GH-104581) 2023-05-18 10:08:33 +01:00
getargs.c gh-99240: Reset pointer to NULL when the pointed memory is freed in argument parsing (#99890) 2022-12-17 12:07:44 +05:30
getcompiler.c closes bpo-43278: remove unnecessary leading '\n' from COMPILER when build with GCC/Clang (GH-24606) 2021-02-25 20:24:21 -08:00
getcopyright.c Update copyright years to 2023. (gh-100848) 2023-01-08 09:13:25 -06:00
getopt.c gh-90300: split --help output into separate options (#30331) 2022-06-01 05:50:01 -04:00
getplatform.c
…
getversion.c gh-81057: Move Globals in Core Code to _PyRuntimeState (gh-99496) 2022-11-15 09:45:11 -07:00
hamt.c gh-99537: Use Py_SETREF() function in C code (#99657) 2022-11-22 13:39:11 +01:00
hashtable.c bpo-41061: Fix incorrect expressions in hashtable (GH-21028) 2020-06-22 00:41:48 -07:00
import.c gh-99113: Add Py_MOD_PER_INTERPRETER_GIL_SUPPORTED (gh-104205) 2023-05-05 21:11:27 +00:00
importdl.c gh-98627: Add an Optional Check for Extension Module Subinterpreter Compatibility (gh-99040) 2023-02-15 18:16:00 -07:00
importdl.h bpo-47162: Add call trampoline to mitigate bad fpcasts on Emscripten (GH-32189) 2022-03-30 12:28:33 -07:00
initconfig.c gh-103167: Fix `-Wstrict-prototypes` warnings by using `(void)` for functions with no args (GH-103168) 2023-04-05 09:22:33 +02:00
instrumentation.c gh-103865: add monitoring support to LOAD_SUPER_ATTR (#103866) 2023-05-16 10:29:00 -06:00
intrinsics.c gh-103763: Implement PEP 695 (#103764) 2023-05-15 20:36:23 -07:00
legacy_tracing.c GH-103082: Filter LINE events in VM, to simplify tool implementation. (GH-104387) 2023-05-12 12:21:20 +01:00
makeopcodetargets.py gh-98040: Remove just the `imp` module (#98573) 2023-04-28 16:17:58 -07:00
marshal.c gh-99113: Add Py_MOD_PER_INTERPRETER_GIL_SUPPORTED (gh-104205) 2023-05-05 21:11:27 +00:00
modsupport.c gh-94673: Ensure Builtin Static Types are Readied Properly (gh-103940) 2023-04-27 16:19:43 -06:00
mysnprintf.c Add a warning message about PyOS_snprintf (#95993) 2022-10-07 11:49:53 -07:00
mystrtoul.c bpo-35134: Add Include/cpython/longobject.h (GH-29044) 2021-10-19 02:04:52 +02:00
opcode_metadata.h gh-103865: add monitoring support to LOAD_SUPER_ATTR (#103866) 2023-05-16 10:29:00 -06:00
opcode_targets.h gh-103865: add monitoring support to LOAD_SUPER_ATTR (#103866) 2023-05-16 10:29:00 -06:00
pathconfig.c gh-98978: Fix Py_SetPythonHome(NULL) (#99066) 2022-11-03 18:34:32 +01:00
perf_trampoline.c gh-103650: Fix perf maps address format (#103651) 2023-05-07 20:42:26 +01:00
preconfig.c gh-77782: Deprecate global configuration variable (#93943) 2022-06-17 16:12:00 +02:00
pyarena.c bpo-43244: Remove the pyarena.h header (GH-25007) 2021-03-24 02:23:01 +01:00
pyctype.c
…
pyfpe.c bpo-46315: Add ifdef HAVE_ feature checks for WASI compatibility (GH-30507) 2022-01-13 09:46:04 +01:00
pyhash.c bpo-29410: Change the default hash algorithm to SipHash13. (GH-28752) 2021-10-10 17:29:46 +09:00
pylifecycle.c GH-101520: Move tracemalloc functionality into core, leaving interface in Modules. (#104508) 2023-05-17 14:17:16 +01:00
pymath.c bpo-45440: Remove pymath.c fallbacks (GH-28977) 2021-10-15 19:45:34 +02:00
pystate.c gh-104341: Adjust tstate_must_exit() to Respect Interpreter Finalization (gh-104437) 2023-05-15 13:59:26 -06:00
pystrcmp.c bpo-41524: fix pointer bug in PyOS_mystr{n}icmp (GH-21845) 2020-08-27 14:45:25 +09:00
pystrhex.c gh-91768: C API no longer use "const PyObject*" type (#91769) 2022-04-21 22:07:19 +02:00
pystrtod.c gh-104263: Rely on Py_NAN and introduce Py_INFINITY (GH-104202) 2023-05-10 17:44:52 +01:00
pythonrun.c gh-103895: Improve how invalid `Exception.__notes__` are displayed (#103897) 2023-05-01 08:32:04 +01:00
pytime.c gh-97786: Fix compiler warnings in pytime.c (#101826) 2023-02-19 17:16:11 -08:00
specialize.c GH-104405: Add missing PEP 523 checks (GH-104406) 2023-05-12 22:23:13 +00:00
stdlib_module_names.h GH-84976: Move Lib/datetime.py to Lib/_pydatetime 2023-05-03 03:09:45 -06:00
structmember.c gh-103509: PEP 697 -- Limited C API for Extending Opaque Types (GH-103511) 2023-05-04 09:56:53 +02:00
suggestions.c GH-81057: remove static state from suggestions.c (#99411) 2022-11-30 16:55:16 +05:30
symtable.c gh-104640: Disallow walrus in comprehension within type scopes (#104641) 2023-05-19 07:31:09 -07:00
sysmodule.c GH-102818: Do not call `PyTraceBack_Here` in sys.settrace trampoline. (GH-104579) 2023-05-19 12:40:48 +01:00
thread.c gh-94673: Properly Initialize and Finalize Static Builtin Types for Each Interpreter (gh-104072) 2023-05-01 19:36:00 -06:00
thread_nt.h gh-81057: Move Threading-Related Globals to _PyRuntimeState (#100084) 2022-12-08 17:50:58 -07:00
thread_pthread.h gh-81057: Fix the wasm32-wasi Buildbot (gh-100139) 2022-12-09 10:17:54 -07:00
thread_pthread_stubs.h gh-81057: Move Threading-Related Globals to _PyRuntimeState (#100084) 2022-12-08 17:50:58 -07:00
traceback.c GH-96803: Add three C-API functions to make _PyInterpreterFrame less opaque for users of PEP 523. (GH-96849) 2023-05-05 17:53:07 +01:00
tracemalloc.c GH-101520: Move tracemalloc functionality into core, leaving interface in Modules. (#104508) 2023-05-17 14:17:16 +01:00

README 

Miscellaneous source files for the main Python shared library