Commit Graph

4301 Commits

Author SHA1 Message Date
Michael Droettboom b05afdd5ec
gh-115168: Add pystats counter for invalidated executors (GH-115169) 2024-02-26 17:51:47 +00:00
Guido van Rossum c0fdfba7ff
Rename tier 2 redundancy eliminator to optimizer (#115888)
The original name is just too much of a mouthful.
2024-02-26 08:42:53 -08:00
Brandt Bucher 7259480957
GH-115802: JIT "small" code for macOS and Linux (GH-115826) 2024-02-26 08:32:44 -08:00
Kirill Podoprigora e4561e0501
gh-115778: Add `tierN` annotation for instruction definitions (#115815)
This replaces the old `TIER_{ONE,TWO}_ONLY` macros. Note that `specialized` implies `tier1`.

Co-authored-by: Alex Waygood <Alex.Waygood@Gmail.com>
2024-02-23 17:31:57 +00:00
Nikita Sobolev e3f462c9a7
Remove `ConverterKeywordDict` alias in `clinic.py` (#115843) 2024-02-23 09:00:07 +00:00
Brandt Bucher 17dab2e572
GH-113464: Clean up JIT stencil generation (GH-115800) 2024-02-22 10:22:23 -08:00
Victor Stinner 52d1477566
gh-110850: Rename internal PyTime C API functions (#115734)
Rename functions:

* _PyTime_GetSystemClock() => _PyTime_TimeUnchecked()
* _PyTime_GetPerfCounter() => _PyTime_PerfCounterUnchecked()
* _PyTime_GetMonotonicClock() => _PyTime_MonotonicUnchecked()
* _PyTime_GetSystemClockWithInfo() => _PyTime_TimeWithInfo()
* _PyTime_GetMonotonicClockWithInfo() => _PyTime_MonotonicWithInfo()
* _PyTime_GetMonotonicClockWithInfo() => _PyTime_MonotonicWithInfo()

Changes:

* Remove "typedef PyTime_t PyTime_t;" which was
  "typedef PyTime_t _PyTime_t;" before a previous rename.
* Update comments of "Unchecked" functions.
* Remove invalid PyTime_Time() comment.
2024-02-20 22:16:37 +00:00
Guido van Rossum 142502ea8d
Tier 2 cleanups and tweaks (#115534)
* Rename `_testinternalcapi.get_{uop,counter}_optimizer` to `new_*_optimizer`
* Use `_PyUOpName()` instead of` _PyOpcode_uop_name[]`
* Add `target` to executor iterator items -- `list(ex)` now returns `(opcode, oparg, target, operand)` quadruples
* Add executor methods `get_opcode()` and `get_oparg()` to get `vmdata.opcode`, `vmdata.oparg`
* Define a helper for printing uops, and unify various places where they are printed
* Add a hack to summarize_stats.py to fix legacy uop names (e.g. `POP_TOP` -> `_POP_TOP`)
* Define helpers in `test_opt.py` for accessing the set or list of opnames of an executor
2024-02-20 20:24:35 +00:00
Mark Shannon 626c414995
GH-115457: Support splitting and replication of micro ops. (GH-115558) 2024-02-20 10:50:59 +00:00
Mark Shannon 7b21403ccd
GH-112354: Initial implementation of warm up on exits and trace-stitching (GH-114142) 2024-02-20 09:39:55 +00:00
Nikita Sobolev f9154f8f23
gh-108303: Move `Lib/test/sortperf.py` to `Tools/scripts` (#114687) 2024-02-18 10:27:14 +03:00
Hugo van Kemenade aba37d451f
gh-100176: Remove outdated Tools/{io,cc,string}bench (#101853)
Co-authored-by: Hugo van Kemenade <hugovk@users.noreply.github.com>
2024-02-17 14:17:21 -07:00
Łukasz Langa 711f42de2e
gh-115556: Remove quotes from command-line arguments in test.bat and rt.bat (#115557)
This change essentially replaces usage of `%1` with `%~1`, which removes
quotes, if any. Without this change, the if statements fail due to
the quotes mangling the syntax.

Additionally, this change works around comma being treated as a parameter
delimiter in test.bat by escaping commas at time of parsing. Tested
combinations of rt and regrtest arguments, all seems to work as before
but now you can specify commas in arguments like "-uall,extralargefile".
2024-02-16 21:24:56 +01:00
Michael Droettboom fbb0169731
gh-115362: Add documentation to pystats output (#115365) 2024-02-16 17:06:07 +00:00
Erlend E. Aasland 351c103134
gh-113317: Argument Clinic: move C/Py identifier helpers into libclinic (#115520) 2024-02-16 07:42:15 +01:00
Erlend E. Aasland 58cb634632
gh-113317: Argument Clinic: move linear_format into libclinic (#115518) 2024-02-15 23:52:20 +01:00
Erlend E. Aasland e74fa294c9
gh-113317: Argument Clinic: inline required_type_for_self_for_parser() in self converter (#115522)
Co-authored-by: Alex Waygood <Alex.Waygood@Gmail.com>
2024-02-15 16:03:58 +00:00
Erlend E. Aasland a0149fa6cf
gh-113317: Argument Clinic: remove global clinic instance (#115517) 2024-02-15 13:21:31 +00:00
Erlend E. Aasland 7f074a771b
gh-113317: Argument Clinic: don't use global state in warn() and fail() (#115510) 2024-02-15 13:22:21 +01:00
Erlend E. Aasland 98ee4ecdbc
gh-113317: Argument Clinic: don't use fail() in CLI (#115513) 2024-02-15 12:10:32 +00:00
Erlend E. Aasland 32f8ab1ab6
gh-114258: Refactor Argument Clinic function name parser (#114930)
Refactor state_modulename_name() of the parsing state machine, by
adding helpers for the sections that deal with ...:

1. parsing the function name
2. normalizing "function kind"
3. dealing with cloned functions
4. resolving return converters
5. adding the function to the DSL parser
2024-02-15 09:45:21 +01:00
Brett Cannon 18343c0985
GH-113516: don't set `LDSHARED` when building for WASI (GH-115495) 2024-02-15 00:51:23 +00:00
Victor Stinner 3e7b7df5cb
gh-114570: Add PythonFinalizationError exception (#115352)
Add PythonFinalizationError exception. This exception derived from
RuntimeError is raised when an operation is blocked during the Python
finalization.

The following functions now raise PythonFinalizationError, instead of
RuntimeError:

* _thread.start_new_thread()
* subprocess.Popen
* os.fork()
* os.fork1()
* os.forkpty()

Morever, _winapi.Overlapped finalizer now logs an unraisable
PythonFinalizationError, instead of an unraisable RuntimeError.
2024-02-14 23:35:06 +01:00
Seth Michael Larson 889cc43cb1
gh-112302: Move pip SBOM discovery to release-tools (#115360) 2024-02-14 12:47:15 -07:00
Seth Michael Larson 4b2d1786cc
gh-115399: Upgrade bundled libexpat to 2.6.0 (#115431) 2024-02-14 16:29:06 +00:00
Eric Snow 514b1c91b8
gh-76785: Improved Subinterpreters Compatibility with 3.12 (gh-115424)
For the most part, these changes make is substantially easier to backport subinterpreter-related code to 3.12, especially the related modules (e.g. _xxsubinterpreters). The main motivation is to support releasing a PyPI package with the 3.13 capabilities compiled for 3.12.

A lot of the changes here involve either hiding details behind macros/functions or splitting up some files.
2024-02-13 14:56:49 -07:00
Ken Jin 7cce857622
gh-114058: Foundations of the Tier2 redundancy eliminator (GH-115085)
---------

Co-authored-by: Mark Shannon <9448417+markshannon@users.noreply.github.com>
Co-authored-by: Jules <57632293+JuliaPoo@users.noreply.github.com>
Co-authored-by: Guido van Rossum <gvanrossum@users.noreply.github.com>
2024-02-13 21:24:48 +08:00
Mark Shannon 8144661017
GH-113710: Fix updating of dict version tag and add watched dict stats (GH-115221) 2024-02-12 16:07:38 +00:00
Mark Shannon 8a3c499ffe
GH-108362: Revert "GH-108362: Incremental GC implementation (GH-108038)" (#115132)
Revert "GH-108362: Incremental GC implementation (GH-108038)"

This reverts commit 36518e69d7.
2024-02-07 12:38:34 +00:00
Seth Michael Larson 4bf41879d0
gh-112302: Change 'licenseConcluded' field to 'NOASSERTION' (#115038) 2024-02-06 12:25:58 +02:00
Ned Deily 299e16ca0f
gh-109991: Update GitHub CI workflows to use OpenSSL 3.0.13. (#115050)
Also update multissltests to use 1.1.1w, 3.0.13, 3.1.5, and 3.2.1.
2024-02-05 21:10:11 -05:00
Erlend E. Aasland 09096a1647
gh-115015: Argument Clinic: fix generated code for METH_METHOD methods without params (#115016) 2024-02-05 21:49:17 +01:00
Mark Shannon 36518e69d7
GH-108362: Incremental GC implementation (GH-108038) 2024-02-05 18:28:51 +00:00
Nikita Sobolev 87cd20a567
gh-115026: Argument Clinic: handle PyBuffer_FillInfo errors in generated code (#115027) 2024-02-05 11:45:09 +01:00
Skip Montanaro 80734a6872
Update README.md (#114974)
Trivial edit

Co-authored-by: Carol Willing <carolcode@willingconsulting.com>
2024-02-04 00:16:30 +00:00
Kirill Podoprigora f3cdd64de8
``Tools/cases_generator``: Fix typos and incorrect comments. (#114892) 2024-02-02 17:52:58 -08:00
Alex Waygood 920b89f627
Bump ruff to 0.2.0 (#114932) 2024-02-02 21:04:15 +00:00
Sam Gross 587d480203
gh-112529: Remove PyGC_Head from object pre-header in free-threaded build (#114564)
* gh-112529: Remove PyGC_Head from object pre-header in free-threaded build

This avoids allocating space for PyGC_Head in the free-threaded build.
The GC implementation for free-threaded CPython does not use the
PyGC_Head structure.

 * The trashcan mechanism uses the `ob_tid` field instead of `_gc_prev`
   in the free-threaded build.
 * The GDB libpython.py file now determines the offset of the managed
   dict field based on whether the running process is a free-threaded
   build. Those are identified by the `ob_ref_local` field in PyObject.
 * Fixes `_PySys_GetSizeOf()` which incorrectly incorrectly included the
   size of `PyGC_Head` in the size of static `PyTypeObject`.
2024-02-01 12:29:19 -08:00
Kirill Podoprigora 6d7ad57385
Update outdated info in ``Tools/cases_generator/README.md`` (#114844) 2024-02-01 08:56:24 -08:00
dependabot[bot] 59ae215387
build(deps-dev): bump types-psutil from 5.9.5.17 to 5.9.5.20240106 in /Tools (#114852)
build(deps-dev): bump types-psutil in /Tools

Bumps [types-psutil](https://github.com/python/typeshed) from 5.9.5.17 to 5.9.5.20240106.
- [Commits](https://github.com/python/typeshed/commits)

---
updated-dependencies:
- dependency-name: types-psutil
  dependency-type: direct:development
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2024-02-01 10:50:08 +00:00
dependabot[bot] 93bfaa858c
build(deps): bump hypothesis from 6.92.2 to 6.97.4 in /Tools (#114851)
Bumps [hypothesis](https://github.com/HypothesisWorks/hypothesis) from 6.92.2 to 6.97.4.
- [Release notes](https://github.com/HypothesisWorks/hypothesis/releases)
- [Commits](https://github.com/HypothesisWorks/hypothesis/compare/hypothesis-python-6.92.2...hypothesis-python-6.97.4)

---
updated-dependencies:
- dependency-name: hypothesis
  dependency-type: direct:production
  update-type: version-update:semver-minor
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2024-02-01 10:42:35 +00:00
dependabot[bot] 21c01a009f
build(deps-dev): bump types-setuptools from 69.0.0.0 to 69.0.0.20240125 in /Tools (#114853)
build(deps-dev): bump types-setuptools in /Tools

Bumps [types-setuptools](https://github.com/python/typeshed) from 69.0.0.0 to 69.0.0.20240125.
- [Commits](https://github.com/python/typeshed/commits)

---
updated-dependencies:
- dependency-name: types-setuptools
  dependency-type: direct:development
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2024-02-01 09:52:05 +00:00
Pablo Galindo Salgado 39d102c2ee
gh-113744: Add a new IncompleteInputError exception to improve incomplete input detection in the codeop module (#113745)
Signed-off-by: Pablo Galindo <pablogsal@gmail.com>
2024-01-30 16:21:30 +00:00
Sviatoslav Sydorenko (Святослав Сидоренко) 963904335e
GH-80789: Get rid of the ``ensurepip`` infra for many wheels (#109245)
Co-authored-by: vstinner@python.org
Co-authored-by: Pradyun Gedam <pradyunsg@gmail.com>
Co-authored-by: Adam Turner <9087854+aa-turner@users.noreply.github.com>
2024-01-30 01:25:31 +00:00
Petr Viktorin 15fe8cea17
gh-91325: Skip Stable ABI checks with Py_TRACE_REFS special build (GH-92046)
Skip Stable ABI checks with Py_TRACE_REFS special build

This build is not compatible with Py_LIMITED_API nor with
the stable ABI.
2024-01-29 16:45:31 +01:00
Brandt Bucher f6d9e5926b
GH-113464: Add a JIT backend for tier 2 (GH-113465)
Add an option (--enable-experimental-jit for configure-based builds
or --experimental-jit for PCbuild-based ones) to build an
*experimental* just-in-time compiler, based on copy-and-patch (https://fredrikbk.com/publications/copy-and-patch.pdf).

See Tools/jit/README.md for more information on how to install the required build-time tooling.
2024-01-28 18:48:48 -08:00
Neil Schemenauer 7a7bce5a0a
gh-113055: Use pointer for interp->obmalloc state (gh-113412)
For interpreters that share state with the main interpreter, this points
to the same static memory structure.  For interpreters with their own
obmalloc state, it is heap allocated.  Add free_obmalloc_arenas() which
will free the obmalloc arenas and radix tree structures for interpreters
with their own obmalloc state.

Co-authored-by: Eric Snow <ericsnowcurrently@gmail.com>
2024-01-26 19:38:14 -08:00
Seth Michael Larson 582d95e8bb
gh-114250: Fetch metadata for pip and its vendored dependencies from PyPI (#114450) 2024-01-26 09:48:13 +00:00
Michael Droettboom ea3cd0498c
gh-114312: Collect stats for unlikely events (GH-114493) 2024-01-25 11:10:51 +00:00
Mark Shannon 981d172f7f
GH-112354: `END_FOR` instruction to only pop one value. (GH-114247)
* Compiler emits END_FOR; POP_TOP instead of END_FOR. To support tier 2 side exits in loops.
2024-01-24 15:10:17 +00:00
Sam Gross 5f1997896d
gh-112984: Fix link error on free-threaded Windows build (GH-114455)
The test_peg_generator test tried to link the python313_d.lib library,
which failed because the library is now named python313t_d.lib. The
underlying problem is that the "compiler" attribute was not set when
we call get_libraries() from distutils.
2024-01-23 18:05:15 +00:00
Erlend E. Aasland e14930ff63
gh-113317: Don't use global clinic instance in bad_argument() (#114330)
Make it possible for a converter to have multiple includes, by collecting
them in a list on the converter instance. This implies converter includes
are added during template generation, so we have to add them to the
clinic instance at the end of the template generation instead of in the
beginning.
2024-01-23 11:07:56 +01:00
Michael Droettboom e45bae7a45
GH-114448: Don't sort summarize_stats.py histograms by amount of change (GH-114449) 2024-01-22 11:45:15 -08:00
Brett Cannon 5cd9c6b1fc
Fix `wasi.py build` after adding the `clean` subcommand. ({GH-114447) 2024-01-22 10:28:57 -08:00
Brett Cannon 681e9e85a2
Add a `clean` subcommand to `Tools/wasm/wasi.py` (GH-114274) 2024-01-19 11:38:52 -08:00
Steve Dower f56d132deb
gh-112984 Update Windows build and installer for free-threaded builds (GH-113129) 2024-01-17 21:52:23 +00:00
Erlend E. Aasland 7a0ac89b29
gh-114178: Fix generate_sbom.py for out-of-tree builds (#114179) 2024-01-17 17:25:39 +01:00
Brett Cannon 03f7839703
GH-114013: fix setting `HOSTRUNNER` for `Tools/wasm/wasi.py` (GH-114097)
Also fix tests found failing under a pydebug build of WASI thanks to `make test` working due to this change.
2024-01-16 11:36:41 -08:00
Mark Shannon ac10947ba7
GH-112354: `_GUARD_IS_TRUE_POP` side-exits to target the next instruction, not themselves. (GH-114078) 2024-01-15 11:41:06 +00:00
Erlend E. Aasland 1709020e8e
gh-113317: Move FormatCounterFormatter into libclinic (#114066) 2024-01-15 00:09:26 +01:00
Erlend E. Aasland 5dbcdfdeb8
gh-113317: Move global utility functions into libclinic (#113986)
Establish Tools/clinic/libclinic/utils.py and move the following
functions over there:

- compute_checksum()
- create_regex()
- write_file()
2024-01-14 18:26:09 +00:00
Ken Jin ac92527c08
gh-113710: Add types to the interpreter DSL (#113711)
Co-authored-by: Jules <57632293+JuliaPoo@users.noreply.github.com>
Co-authored-by: blurb-it[bot] <43283697+blurb-it[bot]@users.noreply.github.com>
2024-01-13 01:30:27 +08:00
Seth Michael Larson e82b096335
gh-112302: Point core developers to SBOM devguide on errors (#113490)
Co-authored-by: Hugo van Kemenade <1324225+hugovk@users.noreply.github.com>
2024-01-10 19:21:04 +00:00
Serhiy Storchaka b3d2427f22
gh-58032: Do not use argparse.FileType in module CLIs and scripts (GH-113649)
Open and close files manually. It prevents from leaking files,
preliminary creation of output files, and accidental closing of stdin
and stdout.
2024-01-10 15:07:19 +02:00
Rami 84d1f76092
gh-89532: Remove LibreSSL workarounds (#28728)
Remove LibreSSL specific workaround ifdefs from `_ssl.c` and delete the non-version-specific `_ssl_data.h` file (relevant for OpenSSL < 1.1.1, which we no longer support per PEP 644).

Co-authored-by: Christian Heimes <christian@python.org>
Co-authored-by: Gregory P. Smith <greg@krypto.org>
2024-01-06 23:25:58 +00:00
Itamar Oren 178919cf21
gh-113258: Write frozen modules to the build tree on Windows (GH-113303)
This ensures the source directory is not modified at build time, and different builds (e.g. different versions or GIL vs no-GIL) do not have conflicts.
2024-01-03 17:30:20 +00:00
dependabot[bot] 4036e48d59
build(deps): bump hypothesis from 6.91.0 to 6.92.2 in /Tools (#113615)
Bumps [hypothesis](https://github.com/HypothesisWorks/hypothesis) from 6.91.0 to 6.92.2.
- [Release notes](https://github.com/HypothesisWorks/hypothesis/releases)
- [Commits](https://github.com/HypothesisWorks/hypothesis/compare/hypothesis-python-6.91.0...hypothesis-python-6.92.2)

---
updated-dependencies:
- dependency-name: hypothesis
  dependency-type: direct:production
  update-type: version-update:semver-minor
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2024-01-01 09:42:44 +00:00
Erlend E. Aasland 7ab9efdd6a
gh-113299: Move cpp.py into libclinic (#113526) 2023-12-28 00:20:57 +01:00
Erlend E. Aasland 87295b4068
gh-113317: Rework Argument Clinic cpp.py error handling (#113525)
Rework error handling in the C preprocessor helper. Instead of monkey-
patching the cpp.Monitor.fail() method from within clinic.py, rewrite
cpp.py to use a subclass of the ClinicError exception. As a side-effect,
ClinicError is moved into Tools/clinic/libclinic/errors.py.

Yak-shaving in preparation for putting cpp.py into libclinic.
2023-12-27 21:43:19 +00:00
Donghee Na e5cce70df7
gh-112532: Fix peg generator build.py for mimalloc build (gh-113492)
gh-112532: Fix peg generator for mimalloc build
2023-12-27 02:54:16 +09:00
Erlend E. Aasland ca71987f4e
gh-113317: Move more formatting helpers into libclinic (#113438)
Move the following global helpers into libclinic:
- format_escape()
- normalize_snippet()
- wrap_declarations()

Also move strip_leading_and_trailing_blank_lines() and make it internal to libclinic.
2023-12-23 17:08:10 +00:00
Erlend E. Aasland c3f92f6a75
gh-113317: Clean up Argument Clinic global namespace (#113414)
Split up clinic.py by establishing libclinic as a support package for
Argument Clinic. Get rid of clinic.py globals by either making them
class members, or by putting them into libclinic.

- Move INCLUDE_COMMENT_COLUMN to BlockPrinter
- Move NO_VARARG to CLanguage
- Move formatting helpers to libclinic
- Move some constants to libclinic (and annotate them as Final)
2023-12-23 00:37:39 +00:00
Erlend E. Aasland 9c3ddf31a3
gh-113317: Remove TextAccumulator type alias from clinic.py (#113413)
Clean-up after gh-113402.
2023-12-22 22:35:16 +00:00
Erlend E. Aasland daa658aba5
gh-113317: Argument Clinic: tear out internal text accumulator APIs (#113402)
Replace the internal accumulator APIs by using lists of strings and join().

Yak-shaving for separating out formatting code into a separate file.

Co-authored-by: Alex Waygood <Alex.Waygood@Gmail.com>
2023-12-22 21:28:15 +01:00
Alex Waygood 7de9855410
Bump mypy to 1.8.0 (#113385) 2023-12-21 23:14:24 +00:00
Ronald Oussoren df1eec3dae
gh-65701: document that freeze doesn't work with framework builds on macOS (#113352)
* gh-65701: document that freeze doesn't work with framework builds on macOS

The framework install is inherently incompatible with freeze. Document
that that freeze doesn't work with framework builds and bail out
early when trying to run freeze anyway.

Co-authored-by: Erlend E. Aasland <erlend.aasland@protonmail.com>
2023-12-21 16:28:00 +01:00
Mark Shannon 723f4d6698
GH-111485: Delete the old generator code. (GH-113321) 2023-12-21 12:46:28 +00:00
Erlend E. Aasland fae096cd4b
gh-113336: Remove the 'version' directive from Argument Clinic (#113341)
The 'version' directive was introduced with gh-63929 in Nov 2013. It has
not been in use in the CPython code base, and the 'version' variable has
never been bumped.
2023-12-21 13:10:41 +01:00
Erlend E. Aasland 4b90b5d857
Docs: update URL in Argument Clinic CLI help text (#113351)
The Argument Clinic docs was moved to the devguide earlier in 2023.
2023-12-21 09:28:43 +00:00
Guido van Rossum 713e42822f
GH-111485: Fix DEFAULT_OUTPUT in opcode_metadata_generator.py (#113324) 2023-12-20 10:05:48 -08:00
Seth Michael Larson b221e03010
gh-113257: Automatically generate pip SBOM metadata from wheel (#113295)
Co-authored-by: Hugo van Kemenade <hugovk@users.noreply.github.com>
2023-12-20 17:28:20 +00:00
Mark Shannon de8a4e52a5
GH-111485: Generate `TARGET` table for computed goto dispatch. (GH-113319) 2023-12-20 15:09:12 +00:00
Mark Shannon e96f26083b
GH-111485: Generate instruction and uop metadata (GH-113287) 2023-12-20 14:27:25 +00:00
Donghee Na 57b7e52790
gh-112205: Support docstring for `@getter` (#113160)
---------

Co-authored-by: Erlend E. Aasland <erlend@python.org>
2023-12-20 21:52:12 +09:00
Erlend E. Aasland 6a69b80d1b
gh-112205: Require @getter and @setter to be methods (#113278)
Co-authored-by: Alex Waygood <Alex.Waygood@Gmail.com>
2023-12-19 12:32:28 +01:00
Seth Michael Larson 4658464e9c
gh-113257: Fix SBOM metadata for pip 23.3.2 (#113262)
Fix SBOM metadata for pip 23.3.2
2023-12-19 08:34:53 +02:00
Mark Shannon 70d378cdaa
GH-111485: Break up instructions with unused cache entries into component micro-ops (GH-113169) 2023-12-18 13:16:45 +00:00
Mark Shannon 771903596b
GH-111485: Test the new cases generator (GH-113252) 2023-12-18 11:14:40 +00:00
Stéphane Bidoul 4a24bf9a13
gh-113246: Updated bundled pip to 23.3.2 (gh-113249)
Updated bundled pip to 23.3.2
2023-12-18 10:21:46 +00:00
Mark Shannon e24eccbc1c
GH-111485: Sort metadata tables for easier checking of future diffs (GH-113101) 2023-12-14 16:41:52 +00:00
Serhiy Storchaka 29f7eb4859
gh-59616: Support os.chmod(follow_symlinks=True) and os.lchmod() on Windows (GH-113049) 2023-12-14 13:28:37 +02:00
Steve Dower 79dad03747
gh-111650: Ensure pyconfig.h includes Py_GIL_DISABLED on Windows (GH-112778) 2023-12-13 15:38:45 +00:00
Donghee Na 498a096a51
gh-112205: Support `@setter` annotation from AC (gh-112922)
---------

Co-authored-by: Erlend E. Aasland <erlend.aasland@protonmail.com>
Co-authored-by: Alex Waygood <Alex.Waygood@Gmail.com>
2023-12-13 14:00:34 +00:00
Mark Shannon 9263173280
Fix whitespace in generated code 2023-12-13 12:31:41 +00:00
Guido van Rossum 7316dfb0eb
gh-112320: Implement on-trace confidence tracking for branches (#112321)
We track the confidence as a scaled int.
2023-12-12 21:43:08 +00:00
Eric Snow a49b427b02
gh-76785: More Fixes for test.support.interpreters (gh-113012)
This brings the module (along with the associated extension modules) mostly in sync with PEP 734.  There are only a few small things to wrap up.
2023-12-12 17:43:30 +00:00
Eric Snow 86a77f4e1a
gh-76785: Fixes for test.support.interpreters (gh-112982)
This involves a number of changes for PEP 734.
2023-12-12 08:24:31 -07:00
Mark Shannon 0c55f27060
GH-111485: Factor out tier 2 code generation from the rest of the interpreter code generator (GH-112968) 2023-12-12 12:12:17 +00:00
Mark Shannon c27e9d5d17
GH-111485: Factor out generation of uop IDs from cases generator. (GH-112877) 2023-12-11 14:14:36 +00:00
Brad Larsen eb27c9a99e
Add a fuzzer for `Py_CompileStringExFlags` (#111721) 2023-12-10 12:16:15 -05:00
Jelle Zijlstra 1f9cd3c1e5
Argument Clinic: fix bare "type" in annotations (#112915)
Bare "type" in annotations should be equivalent to "type[Any]"; see
https://discuss.python.org/t/inconsistencies-between-type-and-type/37404
and python/mypy#16366. It's better to use "type[object]", which is
safer and unambiguous.
2023-12-10 15:30:02 +00:00
Mark Shannon aefdebdef1
GH-111485: Factor out opcode ID generator from the main cases generator. (GH-112831) 2023-12-08 11:48:30 +00:00
Seth Michael Larson 21221c398f
gh-112302: Add Software Bill-of-Materials (SBOM) tracking for dependencies (#112303) 2023-12-07 18:01:58 +02:00
Mark Shannon b449415b2f
GH-111485: Separate out parsing, analysis and code-gen phases of tier 1 code generator (GH-112299) 2023-12-07 12:49:40 +00:00
Victor Stinner d9e444dbb8
gh-106560: Fix redundant declarations in Python/frozen.c (#112612)
Avoid duplicated declarations of "extern" functions in
Python/frozen.c.

Compiler warnings seen by building Python with gcc -Wredundant-decls.
2023-12-03 12:18:24 +01:00
dependabot[bot] 847e4fe0e8
build(deps): bump hypothesis from 6.88.1 to 6.91.0 in /Tools (#112580)
Bumps [hypothesis](https://github.com/HypothesisWorks/hypothesis) from 6.88.1 to 6.91.0.
- [Release notes](https://github.com/HypothesisWorks/hypothesis/releases)
- [Commits](https://github.com/HypothesisWorks/hypothesis/compare/hypothesis-python-6.88.1...hypothesis-python-6.91.0)

---
updated-dependencies:
- dependency-name: hypothesis
  dependency-type: direct:production
  update-type: version-update:semver-minor
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2023-12-01 11:17:24 +00:00
Yang Hau 707c37e373
Fix typos in variable names, function names, and comments (GH-101868) 2023-12-01 09:37:40 +00:00
dependabot[bot] 467e3f9417
build(deps-dev): bump types-setuptools from 68.2.0.0 to 69.0.0.0 in /Tools (#112582)
build(deps-dev): bump types-setuptools in /Tools

Bumps [types-setuptools](https://github.com/python/typeshed) from 68.2.0.0 to 69.0.0.0.
- [Commits](https://github.com/python/typeshed/commits)

---
updated-dependencies:
- dependency-name: types-setuptools
  dependency-type: direct:development
  update-type: version-update:semver-major
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2023-12-01 09:29:04 +00:00
dependabot[bot] 19a86148e6
build(deps-dev): bump mypy from 1.7.0 to 1.7.1 in /Tools (#112581)
Bumps [mypy](https://github.com/python/mypy) from 1.7.0 to 1.7.1.
- [Changelog](https://github.com/python/mypy/blob/master/CHANGELOG.md)
- [Commits](https://github.com/python/mypy/compare/v1.7.0...v1.7.1)

---
updated-dependencies:
- dependency-name: mypy
  dependency-type: direct:development
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2023-12-01 09:21:45 +00:00
Brett Cannon 482b0ee8f6
Clarify that WASI tool requirements are included in the devcontainer (GH-112561) 2023-11-30 13:01:07 -08:00
Irit Katriel 07ebd46f9e
gh-112519: Make it possible to specify instruction flags for pseudo instructions in bytecodes.c (#112520) 2023-11-30 11:03:30 +00:00
Donghee Na 7eeea13403
gh-112205: Support @getter annotation from AC (gh-112396) 2023-11-30 19:40:53 +09:00
Brett Cannon 37589d76bb
GH-103065, GH-106704, GH-105253: Provide a `Tools/wasm/wasi.py` script to simplify doing a WASI build (GH-112473) 2023-11-29 16:18:25 -08:00
Donghee Na 4ec849bba8
gh-112213: Add missing declaration of target_critical_section (gh-112374)
Co-authored-by: Alex Waygood <Alex.Waygood@Gmail.com>
2023-11-25 00:58:39 +09:00
Mark Shannon 1619f4350e
GH-111485: Sort cases in the case generator output (GH-112315) 2023-11-22 15:19:50 +00:00
Guido van Rossum 8deb8bc2e5
gh-112287: Speed up Tier 2 (uop) interpreter a little (#112286)
This makes the Tier 2 interpreter a little faster.
I calculated by about 3%,
though I hesitate to claim an exact number.

This starts by doubling the trace size limit (to 512),
making it more likely that loops fit in a trace.

The rest of the approach is to only load
`oparg` and `operand` in cases that use them.
The code generator know when these are used.

For `oparg`, it will conditionally emit
```
oparg = CURRENT_OPARG();
```
at the top of the case block.
(The `oparg` variable may be referenced multiple times
by the instructions code block, so it must be in a variable.)

For `operand`, it will use `CURRENT_OPERAND()` directly
instead of referencing the `operand` variable,
which no longer exists.
(There is only one place where this will be used.)
2023-11-20 11:25:32 -08:00
Brett Cannon 56e59a49ae
GH-111807: Lower the parser stack depth under WASI debug builds (#112225) 2023-11-20 13:27:33 +00:00
Donghee Na b8c952af72
gh-111903: Update AC to support "pycore_critical_section.h" header (gh-112251) 2023-11-19 10:13:58 +09:00
Donghee Na e52cc80f7f
gh-112213: Add @critical_section target directive to Argument Clinic (gh-112232)
Co-authored-by: Alex Waygood <Alex.Waygood@Gmail.com>
2023-11-19 08:05:49 +09:00
Guido van Rossum eb3c94ea66
gh-110319: Assert type_version != 0 before using it (#112226)
- Ensure that `assert(type_version != 0);` always comes *before* using `type_version`

Also:
- In cases_generator, rename `-v` to from `--verbose` to `--viable`
2023-11-17 20:58:13 -08:00
Guido van Rossum da314f7c8d
A few more cases_generator cleanups (#112220) 2023-11-17 22:36:37 +00:00
Guido van Rossum be0bd54c6b
gh-106529: Cleanups split off gh-112134 (#112214)
- Double max trace size to 256
- Add a dependency on executor_cases.c.h for ceval.o
- Mark `_SPECIALIZE_UNPACK_SEQUENCE` as `TIER_ONE_ONLY`
- Add debug output back showing the optimized trace
- Bunch of cleanups to Tools/cases_generator/
2023-11-17 11:49:42 -08:00
Nikita Sobolev 762eb58220
Remove `imp_dummy_def` from `Tools/c-analyzer/cpython/ignored.tsv` (gh-112122)
It was removed in 3.12, no need to keep the ignore.
2023-11-16 12:21:23 -07:00
Victor Stinner d9fd33a869
gh-112088: Run autoreconf in GHA check_generated_files (#112090)
The "Check if generated files are up to date" job of GitHub Actions
now runs the "autoreconf -ivf -Werror" command instead of the "make
regen-configure" command to avoid depending on the external quay.io
server.

Add Tools/build/regen-configure.sh script to regenerate the configure
with an Ubuntu container image. The
"quay.io/tiran/cpython_autoconf:271" container image
(https://github.com/tiran/cpython_autoconf) is no longer used.
2023-11-15 21:47:14 +01:00
Mark Shannon 4bbb367ba6
GH-111848: Set the IP when de-optimizing (GH-112065)
* Replace jumps with deopts in tier 2

* Fewer special cases of uop names

* Add target field to uop IR

* Remove more redundant SET_IP and _CHECK_VALIDITY micro-ops

* Extend whitelist of non-escaping API functions.
2023-11-15 15:48:58 +00:00
Sam Gross 324531df90
gh-111903: Add `@critical_section` directive to Argument Clinic. (#111904)
The `@critical_section` directive instructs Argument Clinic to generate calls
to `Py_BEGIN_CRITICAL_SECTION()` and `Py_END_CRITICAL_SECTION()` around the
bound function. In `--disable-gil` builds, these calls will lock and unlock
the `self` object. They are no-ops in the default build.

This is used in one place (`_io._Buffered.close`) as a demonstration.
Subsequent PRs will use it more widely in the `_io.Buffered` bindings.
2023-11-14 10:47:46 +00:00
Brandt Bucher 31ad7e061e
GH-111520: Add back the operand local (GH-111813) 2023-11-13 17:27:19 -08:00
Alex Waygood 2e7f070080
Bump mypy to 1.7.0 (#111961) 2023-11-10 16:00:50 +00:00
Mark Shannon 25c4956488
GH-109369: Exit tier 2 if executor is invalid (GH-111657) 2023-11-09 11:19:51 +00:00
Mark Shannon 06efb60264
GH-111848: Tidy up tier 2 handling of FOR_ITER specialization by using DEOPT_IF instead of jumps. (GH-111849) 2023-11-08 13:31:55 +00:00
Victor Stinner 11e83488c5
gh-111089: Revert PyUnicode_AsUTF8() changes (#111833)
* Revert "gh-111089: Use PyUnicode_AsUTF8() in Argument Clinic (#111585)"

This reverts commit d9b606b3d0.

* Revert "gh-111089: Use PyUnicode_AsUTF8() in getargs.c (#111620)"

This reverts commit cde1071b2a.

* Revert "gh-111089: PyUnicode_AsUTF8() now raises on embedded NUL (#111091)"

This reverts commit d731579bfb.

* Revert "gh-111089: Add PyUnicode_AsUTF8() to the limited C API (#111121)"

This reverts commit d8f32be5b6.

* Revert "gh-111089: Use PyUnicode_AsUTF8() in sqlite3 (#111122)"

This reverts commit 37e4e20eaa.
2023-11-07 22:36:13 +00:00
Mark Shannon 931f4438c9
GH-111485: Allow arbitrary annotations on instructions and micro-ops. (GH-111697) 2023-11-07 09:42:39 +00:00
CF Bolz-Tereick 9573d14215
gh-96954: use a directed acyclic word graph for storing the unicodedata codepoint names (#97906)
Co-authored-by: Łukasz Langa <lukasz@langa.pl>
Co-authored-by: Pieter Eendebak <pieter.eendebak@gmail.com>
Co-authored-by: Dennis Sweeney <36520290+sweeneyde@users.noreply.github.com>
2023-11-04 15:56:58 +01:00
Brad Larsen f21b23058e
Add a fuzz target for `_elementtree.XMLParser._parse_whole` (#111477)
* Add a fuzzer for `_elementtree.XMLParser._parse_whole`
2023-11-03 14:01:56 -07:00
dependabot[bot] 7215f173f5
build(deps): bump hypothesis from 6.87.1 to 6.88.1 in /Tools (#111596)
Bumps [hypothesis](https://github.com/HypothesisWorks/hypothesis) from 6.87.1 to 6.88.1.
- [Release notes](https://github.com/HypothesisWorks/hypothesis/releases)
- [Commits](https://github.com/HypothesisWorks/hypothesis/compare/hypothesis-python-6.87.1...hypothesis-python-6.88.1)

---
updated-dependencies:
- dependency-name: hypothesis
  dependency-type: direct:production
  update-type: version-update:semver-minor
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2023-11-02 10:53:40 +00:00
Guido van Rossum 7e135a48d6
gh-111520: Integrate the Tier 2 interpreter in the Tier 1 interpreter (#111428)
- There is no longer a separate Python/executor.c file.
- Conventions in Python/bytecodes.c are slightly different -- don't use `goto error`,
  you must use `GOTO_ERROR(error)` (same for others like `unused_local_error`).
- The `TIER_ONE` and `TIER_TWO` symbols are only valid in the generated (.c.h) files.
- In Lib/test/support/__init__.py, `Py_C_RECURSION_LIMIT` is imported from `_testcapi`.
- On Windows, in debug mode, stack allocation grows from 8MiB to 12MiB.
- **Beware!** This changes the env vars to enable uops and their debugging
  to `PYTHON_UOPS` and `PYTHON_LLTRACE`.
2023-11-01 13:13:02 -07:00
Victor Stinner d9b606b3d0
gh-111089: Use PyUnicode_AsUTF8() in Argument Clinic (#111585)
Replace PyUnicode_AsUTF8AndSize() with PyUnicode_AsUTF8() to remove
the explicit check for embedded null characters.

The change avoids to have to include explicitly <string.h> to get the
strlen() function when using a recent version of the limited C API.
2023-11-01 16:34:42 +01:00
Mark Shannon b14e882428
GH-111485: Use micro-ops to split specialization code from base action (GH-111561) 2023-11-01 10:53:27 +00:00
Mark Shannon 2904d99839
GH-111485: Remove some special cases from the code generator and bytecodes.c (GH-111540) 2023-10-31 13:21:07 +00:00
Mark Shannon d27acd4461
GH-111485: Increment `next_instr` consistently at the start of the instruction. (GH-111486) 2023-10-31 10:09:54 +00:00
Michael Droettboom 9495bcaf59
gh-111277: In summarize_stats.py, don't fail fast on invalid ratios (#111278) 2023-10-30 17:10:07 -07:00
Michael Droettboom 84b4533e84
gh-109329: Count tier2 opcode misses (#110561)
This keeps a separate 'miss' counter for each micro-opcode, incremented whenever a guard uop takes a deoptimization side exit.
2023-10-30 17:02:45 -07:00
Riccardo Ghetta 8eaa206fec
gh-109515: When generating deep frozen modules on Windows, use a list file instead of arguments (GH-109516) 2023-10-30 16:14:26 +00:00
Sam Gross 6dfb8fe023
gh-110481: Implement biased reference counting (gh-110764) 2023-10-30 16:06:09 +00:00
Dino Viehland 05f2f0ac92
gh-90815: Add mimalloc memory allocator (#109914)
* Add mimalloc v2.12

Modified src/alloc.c to remove include of alloc-override.c and not
compile new handler.

Did not include the following files:

 - include/mimalloc-new-delete.h
 - include/mimalloc-override.h
 - src/alloc-override-osx.c
 - src/alloc-override.c
 - src/static.c
 - src/region.c

mimalloc is thread safe and shares a single heap across all runtimes,
therefore finalization and getting global allocated blocks across all
runtimes is different.

* mimalloc: minimal changes for use in Python:

 - remove debug spam for freeing large allocations
 - use same bytes (0xDD) for freed allocations in CPython and mimalloc
   This is important for the test_capi debug memory tests

* Don't export mimalloc symbol in libpython.
* Enable mimalloc as Python allocator option.
* Add mimalloc MIT license.
* Log mimalloc in Lib/test/pythoninfo.py.
* Document new mimalloc support.
* Use macro defs for exports as done in:
  https://github.com/python/cpython/pull/31164/

Co-authored-by: Sam Gross <colesbury@gmail.com>
Co-authored-by: Christian Heimes <christian@python.org>
Co-authored-by: Victor Stinner <vstinner@python.org>
2023-10-30 15:43:11 +00:00
Savannah Ostrowski 4a929d432b
GH-111339: Fix initialization and finalization of static optimizer types (GH-111430) 2023-10-29 13:53:25 -07:00
Irit Katriel 67a91f78e4
gh-109094: replace frame->prev_instr by frame->instr_ptr (#109095) 2023-10-26 13:43:10 +00:00
Mark Shannon b0699aa544
GH-111213: Fix a few broken stats (GH-111216) 2023-10-26 11:33:12 +01:00
Artyom Romanov 0d1cbff833
Bump test deps: `ruff` and `mypy` (#111288) 2023-10-25 01:05:37 -07:00
Michael Droettboom 81eba76450
gh-110019: Refactor summarize_stats (GH-110398) 2023-10-24 09:57:39 +01:00