cpython/Tools
Lysandros Nikolaou 01481f2dc1
gh-104169: Refactor tokenizer into lexer and wrappers (#110684)
* The lexer, which include the actual lexeme producing logic, goes into
  the `lexer` directory.
* The wrappers, one wrapper per input mode (file, string, utf-8, and
  readline), go into the `tokenizer` directory and include logic for
  creating a lexer instance and managing the buffer for different modes.
---------

Co-authored-by: Pablo Galindo <pablogsal@gmail.com>
Co-authored-by: blurb-it[bot] <43283697+blurb-it[bot]@users.noreply.github.com>
2023-10-11 15:14:44 +00:00
..
build gh-110014: Fix _POSIX_THREADS and _POSIX_SEMAPHORES usage (#110139) 2023-09-30 19:25:54 +02:00
buildbot gh-109566, regrtest: Add --fast-ci and --slow-ci options (#109570) 2023-09-26 17:22:50 +02:00
c-analyzer gh-104169: Refactor tokenizer into lexer and wrappers (#110684) 2023-10-11 15:14:44 +00:00
cases_generator gh-109287: fix overrides in cases generator (#110419) 2023-10-05 15:05:29 -07:00
ccbench bpo-43723: Fix deprecation error caused by thread.setDaemon() (GH-25361) 2021-04-12 13:12:36 +02:00
clinic gh-110558: Enable ruff's pyupgrade rules when running on Argument Clinic (#110603) 2023-10-10 09:35:36 +00:00
freeze gh-103053: Fix test_tools.test_freeze on FreeBSD (#110451) 2023-10-06 03:08:34 +02:00
gdb GH-100987: Allow objects other than code objects as the "executable" of an internal frame. (GH-105727) 2023-06-14 13:46:37 +01:00
i18n gh-102507 Remove invisible pagebreak characters (#102531) 2023-03-08 13:58:14 +00:00
importbench gh-98040: Fix importbench: use types.ModuleType() (#105743) 2023-06-14 00:32:12 +02:00
iobench gh-100176: Tools/iobench: Remove redundant compat code for Python <= 3.2 (#100197) 2023-04-08 12:04:47 +03:00
lockbench gh-108724: Add PyMutex and _PyParkingLot APIs (gh-109344) 2023-09-19 09:54:29 -06:00
msi gh-106765: Updates Windows installer build instructions to clarify system requirements (GH-110566) 2023-10-09 22:15:19 +01:00
nuget bpo-41744: Package python.props with correct name in NuGet package (GH-22154) 2020-09-14 20:30:15 +01:00
patchcheck GH-109408: Move the C file whitespace check from patchcheck to pre-commit (#109890) 2023-10-10 07:40:08 -06:00
peg_generator gh-109596: Ensure repeated rules in the grammar are not allowed and fix incorrect soft keywords (#109606) 2023-09-22 19:03:23 +01:00
scripts gh-109329: Add stat for "trace too short" (GH-110402) 2023-10-05 16:12:06 +01:00
ssl gh-109649: Use os.process_cpu_count() (#110165) 2023-10-01 03:14:57 +02:00
stringbench [codemod] Fix non-matching bracket pairs (GH-28473) 2021-09-22 01:09:00 +02:00
tz bpo-29919: Remove unused imports found by pyflakes (#137) 2017-03-27 16:05:26 +02:00
unicode Code: Update Donghee Na's name (#109744) 2023-09-25 18:17:34 +03:00
unittestgui Remove a redundant assignment in Tools/unittestgui/unittestgui.py (GH-21438) 2021-05-16 16:55:06 +01:00
wasm gh-109649: Use os.process_cpu_count() (#110165) 2023-10-01 03:14:57 +02:00
README gh-104780: Remove 2to3 program and lib2to3 module (#104781) 2023-05-23 19:40:02 +02:00
requirements-dev.txt PEG generator: bump types-setuptools from 68.1.0.1 to 68.2.0.0 (#110175) 2023-10-01 12:38:46 +01:00
requirements-hypothesis.txt build(deps): bump hypothesis from 6.84.0 to 6.87.1 in /Tools (#110174) 2023-10-01 17:25:29 +01:00

README

This directory contains a number of Python programs that are useful
while building or extending Python.

build           Automatically generated directory by the build system
                contain build artifacts and intermediate files.

buildbot        Batchfiles for running on Windows buildbot workers.

c-analyzer      Tools to check no new global variables have been added.

cases_generator Tooling to generate interpreters.

ccbench         A Python threads-based concurrency benchmark. (*)

clinic          A preprocessor for CPython C files in order to automate
                the boilerplate involved with writing argument parsing
                code for "builtins".

freeze          Create a stand-alone executable from a Python program.

gdb             Python code to be run inside gdb, to make it easier to
                debug Python itself (by David Malcolm).

i18n            Tools for internationalization. pygettext.py
                parses Python source code and generates .pot files,
                and msgfmt.py generates a binary message catalog
                from a catalog in text format.

importbench     A set of micro-benchmarks for various import scenarios.

iobench         Benchmark for the new Python I/O system. (*)

msi             Support for packaging Python as an MSI package on Windows.

nuget           Files for the NuGet package manager for .NET.

patchcheck      Tools for checking and applying patches to the Python source code
                and verifying the integrity of patch files.

peg_generator   PEG-based parser generator (pegen) used for new parser.

scripts         A number of useful single-file programs, e.g. run_tests.py
                which runs the Python test suite.

ssl             Scripts to generate ssl_data.h from OpenSSL sources, and run
                tests against multiple installations of OpenSSL and LibreSSL.

stringbench     A suite of micro-benchmarks for various operations on
                strings (both 8-bit and unicode). (*)

tz              A script to dump timezone from /usr/share/zoneinfo.

unicode         Tools for generating unicodedata and codecs from unicode.org
                and other mapping files (by Fredrik Lundh, Marc-Andre Lemburg
                and Martin von Loewis).

unittestgui     A Tkinter based GUI test runner for unittest, with test
                discovery.

wasm            Config and helpers to facilitate cross compilation of CPython
                to WebAssembly (WASM).

(*) A generic benchmark suite is maintained separately at https://github.com/python/performance

Note: The pynche color editor has moved to https://gitlab.com/warsaw/pynche