Commit Graph

1197 Commits

Author SHA1 Message Date
Sam Gross a3af3cb4f4
gh-110481: Implement inter-thread queue for biased reference counting (#114824)
Biased reference counting maintains two refcount fields in each object:
`ob_ref_local` and `ob_ref_shared`. The true refcount is the sum of these two
fields. In some cases, when refcounting operations are split across threads,
the ob_ref_shared field can be negative (although the total refcount must be
at least zero). In this case, the thread that decremented the refcount
requests that the owning thread give up ownership and merge the refcount
fields.
2024-02-09 17:08:32 -05:00
Skip Montanaro 2afc7182e6
gh-114505: Add missing header file dependencies (#114513)
Also move PYTHON_HEADERS up and make _testembed.o depend on it.
2024-02-07 08:50:24 +00:00
Zachary Ware 1032326fe4
gh-114883: Fix Makefile dependency tree for non-jit builds (GH-114884) 2024-02-03 23:16:03 +00:00
Brandt Bucher f6d9e5926b
GH-113464: Add a JIT backend for tier 2 (GH-113465)
Add an option (--enable-experimental-jit for configure-based builds
or --experimental-jit for PCbuild-based ones) to build an
*experimental* just-in-time compiler, based on copy-and-patch (https://fredrikbk.com/publications/copy-and-patch.pdf).

See Tools/jit/README.md for more information on how to install the required build-time tooling.
2024-01-28 18:48:48 -08:00
Sam Gross b52fc70d1a
gh-112529: Implement GC for free-threaded builds (#114262)
* gh-112529: Implement GC for free-threaded builds

This implements a mark and sweep GC for the free-threaded builds of
CPython. The implementation relies on mimalloc to find GC tracked
objects (i.e., "containers").
2024-01-25 10:27:36 -08:00
Nikita Sobolev ba253a4794
gh-108303: Move `.whl` test files to `Lib/test/wheeldata/` (#114343) 2024-01-23 16:33:12 +01:00
Sam Gross 1d6d5e854c
gh-112529: Use GC heaps for GC allocations in free-threaded builds (gh-114157)
* gh-112529: Use GC heaps for GC allocations in free-threaded builds

The free-threaded build's garbage collector implementation will need to
find GC objects by traversing mimalloc heaps. This hooks up the
allocation calls with the correct heaps by using a thread-local
"current_obj_heap" variable.

* Refactor out setting heap based on type
2024-01-21 01:14:45 +09:00
Nikita Sobolev 9c93350f58
gh-108303: Move all doctest related files and tests to `Lib/test/test_doctest/` (#112109)
Co-authored-by: Brett Cannon <brett@python.org>
2024-01-18 16:58:11 +01:00
Brandt Bucher 30e6cbdba2
GH-113860: Get rid of `_PyUOpExecutorObject` (GH-113954) 2024-01-12 11:58:23 +00:00
Donghee Na 57bdc6c30d
gh-111968: Introduce _PyFreeListState and _PyFreeListState_GET API (gh-113584) 2024-01-10 08:04:41 +09:00
Jakub Kulík bc71ae2b97
gh-113688: fix dtrace build on Solaris (#113814)
(the gcmodule -> gc refactoring broke it)
2024-01-08 11:28:09 -08:00
Rami 84d1f76092
gh-89532: Remove LibreSSL workarounds (#28728)
Remove LibreSSL specific workaround ifdefs from `_ssl.c` and delete the non-version-specific `_ssl_data.h` file (relevant for OpenSSL < 1.1.1, which we no longer support per PEP 644).

Co-authored-by: Christian Heimes <christian@python.org>
Co-authored-by: Gregory P. Smith <greg@krypto.org>
2024-01-06 23:25:58 +00:00
Sam Gross 99854ce170
gh-113688: Split up gcmodule.c (gh-113715)
This splits part of Modules/gcmodule.c of into Python/gc.c, which
now contains the core garbage collection implementation. The Python
module remain in the Modules/gcmodule.c file.
2024-01-05 12:17:16 -08:00
Sam Gross 31d8757b60
gh-113370: Add missing obmalloc.o dependencies on mimalloc (#113371) 2023-12-21 22:44:55 +01:00
Mark Shannon 723f4d6698
GH-111485: Delete the old generator code. (GH-113321) 2023-12-21 12:46:28 +00:00
Gregory P. Smith 103c4ea274
gh-112305: Fix check-clean-src to detect frozen_modules .h files. (#113344)
A typo left this check broken so many of us who do out-of-tree builds
were seeing strange failures due to bad `Python/frozen_modules/*.h`
files being picked up from the source tree and used at build time from
different Python versions leading to errors like:

`Fatal Python error: _PyImport_InitCore: failed to initialize importlib`

Or similar once our build got to an "invoke the interpreter"
bootstrapping step due to incorrect bytecode being embedded.
2023-12-20 23:32:13 -08:00
Sam Gross a3e8afe0a3
gh-113330: Fix mimalloc headers reference (#113331)
The `MIMALLOC_HEADERS` variable is defined in the Makefile.pre.in, not
the configure script, so we should use the `$(MIMALLOC_HEADERS)` syntax
instead of the `@MIMALLOC_HEADERS@` syntax.
2023-12-20 22:07:17 +01:00
Mark Shannon de8a4e52a5
GH-111485: Generate `TARGET` table for computed goto dispatch. (GH-113319) 2023-12-20 15:09:12 +00:00
Mark Shannon e96f26083b
GH-111485: Generate instruction and uop metadata (GH-113287) 2023-12-20 14:27:25 +00:00
Barney Gale d91e43ed78
GH-110109: Move tests for pathlib ABCs to new module. (#112904) 2023-12-16 19:04:33 +00:00
Mark Shannon 0c55f27060
GH-111485: Factor out tier 2 code generation from the rest of the interpreter code generator (GH-112968) 2023-12-12 12:12:17 +00:00
Mark Shannon c27e9d5d17
GH-111485: Factor out generation of uop IDs from cases generator. (GH-112877) 2023-12-11 14:14:36 +00:00
Alex Waygood c1652d6d62
gh-110109: Fix installed buildbots now `pathlib` is a package (#112901) 2023-12-09 18:05:33 +00:00
Mark Shannon aefdebdef1
GH-111485: Factor out opcode ID generator from the main cases generator. (GH-112831) 2023-12-08 11:48:30 +00:00
Sam Gross db460735af
gh-112538: Add internal-only _PyThreadStateImpl "wrapper" for PyThreadState (gh-112560)
Every PyThreadState instance is now actually a _PyThreadStateImpl.
It is safe to cast from `PyThreadState*` to `_PyThreadStateImpl*` and back.
The _PyThreadStateImpl will contain fields that we do not want to expose
in the public C API.
2023-12-07 12:11:45 -07:00
Seth Michael Larson 21221c398f
gh-112302: Add Software Bill-of-Materials (SBOM) tracking for dependencies (#112303) 2023-12-07 18:01:58 +02:00
Mark Shannon b449415b2f
GH-111485: Separate out parsing, analysis and code-gen phases of tier 1 code generator (GH-112299) 2023-12-07 12:49:40 +00:00
Serhiy Storchaka f8c0198e3b
gh-108927: Include new dir test/regrtestdata in the installation (GH-112765)
Co-authored-by: Victor Stinner <vstinner@python.org>
2023-12-06 09:48:27 +02:00
Brett Cannon 37589d76bb
GH-103065, GH-106704, GH-105253: Provide a `Tools/wasm/wasi.py` script to simplify doing a WASI build (GH-112473) 2023-11-29 16:18:25 -08:00
Guido van Rossum be0bd54c6b
gh-106529: Cleanups split off gh-112134 (#112214)
- Double max trace size to 256
- Add a dependency on executor_cases.c.h for ceval.o
- Mark `_SPECIALIZE_UNPACK_SEQUENCE` as `TIER_ONE_ONLY`
- Add debug output back showing the optimized trace
- Bunch of cleanups to Tools/cases_generator/
2023-11-17 11:49:42 -08:00
Victor Stinner d9fd33a869
gh-112088: Run autoreconf in GHA check_generated_files (#112090)
The "Check if generated files are up to date" job of GitHub Actions
now runs the "autoreconf -ivf -Werror" command instead of the "make
regen-configure" command to avoid depending on the external quay.io
server.

Add Tools/build/regen-configure.sh script to regenerate the configure
with an Ubuntu container image. The
"quay.io/tiran/cpython_autoconf:271" container image
(https://github.com/tiran/cpython_autoconf) is no longer used.
2023-11-15 21:47:14 +01:00
Victor Stinner e5dfcc2b6e
gh-96954: Don't run regen-unicodedata in regen-all (#112120)
The "make regen-unicodedata" should now be run manually. By the
default, it requires an Internet connection, which is not always the
case. Some Linux distributions build Linux packages in isolated
environment (without network).
2023-11-15 18:14:08 +01:00
Miro Hrončok ec05507367
gh-96954: Fix `make regen-unicodedata` in out-of-tree builds (#112118)
This avoids:

    python3.13 Tools/unicode/makeunicodedata.py
    python3.13: can't open file '.../build/debug/Tools/unicode/makeunicodedata.py': [Errno 2] No such file or directory
    make: *** [Makefile:1498: regen-unicodedata] Error 2

Re-run `make regen-unicodedata` to update the script path in generated files.
2023-11-15 16:42:17 +00:00
Victor Stinner 62802b6228
gh-111545: Add Include/cpython/pyhash.h header file (#112063)
Move non-limited C API to a new Include/cpython/pyhash.h header file.
2023-11-15 01:19:20 +01:00
Nikita Sobolev 65d6dc2715
gh-108303: Install `Lib/test/configdata` (#111899) 2023-11-10 13:00:50 +01:00
Sam Gross 31c90d5838
gh-111569: Implement Python critical section API (gh-111571)
Critical sections are helpers to replace the global interpreter lock
with finer grained locking.  They provide similar guarantees to the GIL
and avoid the deadlock risk that plain locking involves.  Critical
sections are implicitly ended whenever the GIL would be released.  They
are resumed when the GIL would be acquired.  Nested critical sections
behave as if the sections were interleaved.
2023-11-08 15:39:29 -07:00
CF Bolz-Tereick 9573d14215
gh-96954: use a directed acyclic word graph for storing the unicodedata codepoint names (#97906)
Co-authored-by: Łukasz Langa <lukasz@langa.pl>
Co-authored-by: Pieter Eendebak <pieter.eendebak@gmail.com>
Co-authored-by: Dennis Sweeney <36520290+sweeneyde@users.noreply.github.com>
2023-11-04 15:56:58 +01:00
Nikita Sobolev 890ef1b035
gh-108303: Move `zip` and `tar` archives to `Lib/test/archivetestdata` (#111549) 2023-11-04 10:57:53 +01:00
Guido van Rossum 7e135a48d6
gh-111520: Integrate the Tier 2 interpreter in the Tier 1 interpreter (#111428)
- There is no longer a separate Python/executor.c file.
- Conventions in Python/bytecodes.c are slightly different -- don't use `goto error`,
  you must use `GOTO_ERROR(error)` (same for others like `unused_local_error`).
- The `TIER_ONE` and `TIER_TWO` symbols are only valid in the generated (.c.h) files.
- In Lib/test/support/__init__.py, `Py_C_RECURSION_LIMIT` is imported from `_testcapi`.
- On Windows, in debug mode, stack allocation grows from 8MiB to 12MiB.
- **Beware!** This changes the env vars to enable uops and their debugging
  to `PYTHON_UOPS` and `PYTHON_LLTRACE`.
2023-11-01 13:13:02 -07:00
Eric Snow c6fe0869ab
gh-76785: Move the Cross-Interpreter Code to Its Own File (gh-111502)
This is partly to clear this stuff out of pystate.c, but also in preparation for moving some code out of _xxsubinterpretersmodule.c.  This change also moves this stuff to the internal API (new: Include/internal/pycore_crossinterp.h).  @vstinner did this previously and I undid it.  Now I'm re-doing it. :/
2023-10-30 16:53:10 -06:00
Dino Viehland 05f2f0ac92
gh-90815: Add mimalloc memory allocator (#109914)
* Add mimalloc v2.12

Modified src/alloc.c to remove include of alloc-override.c and not
compile new handler.

Did not include the following files:

 - include/mimalloc-new-delete.h
 - include/mimalloc-override.h
 - src/alloc-override-osx.c
 - src/alloc-override.c
 - src/static.c
 - src/region.c

mimalloc is thread safe and shares a single heap across all runtimes,
therefore finalization and getting global allocated blocks across all
runtimes is different.

* mimalloc: minimal changes for use in Python:

 - remove debug spam for freeing large allocations
 - use same bytes (0xDD) for freed allocations in CPython and mimalloc
   This is important for the test_capi debug memory tests

* Don't export mimalloc symbol in libpython.
* Enable mimalloc as Python allocator option.
* Add mimalloc MIT license.
* Log mimalloc in Lib/test/pythoninfo.py.
* Document new mimalloc support.
* Use macro defs for exports as done in:
  https://github.com/python/cpython/pull/31164/

Co-authored-by: Sam Gross <colesbury@gmail.com>
Co-authored-by: Christian Heimes <christian@python.org>
Co-authored-by: Victor Stinner <vstinner@python.org>
2023-10-30 15:43:11 +00:00
Donghee Na 2dcc57008b
gh-109693: Remove pycore_atomic.h (gh-110992) 2023-10-18 00:33:50 +09:00
Victor Stinner be5e8a0103
gh-110964: Remove private _PyArg functions (#110966)
Move the following private functions and structures to
pycore_modsupport.h internal C API:

* _PyArg_BadArgument()
* _PyArg_CheckPositional()
* _PyArg_NoKeywords()
* _PyArg_NoPositional()
* _PyArg_ParseStack()
* _PyArg_ParseStackAndKeywords()
* _PyArg_Parser structure
* _PyArg_UnpackKeywords()
* _PyArg_UnpackKeywordsWithVararg()
* _PyArg_UnpackStack()
* _Py_ANY_VARARGS()

Changes:

* Python/getargs.h now includes pycore_modsupport.h to export
  functions.
* clinic.py now adds pycore_modsupport.h when one of these functions
  is used.
* Add pycore_modsupport.h includes when a C extension uses one of
  these functions.
* Define Py_BUILD_CORE_MODULE in C extensions which now include
  directly or indirectly (via code generated by Argument Clinic)
  pycore_modsupport.h:

  * _csv
  * _curses_panel
  * _dbm
  * _gdbm
  * _multiprocessing.posixshmem
  * _sqlite.row
  * _statistics
  * grp
  * resource
  * syslog

* _testcapi: bad_get() no longer uses METH_FASTCALL calling
  convention but METH_VARARGS. Replace _PyArg_UnpackStack() with
  PyArg_ParseTuple().
* _testcapi: add PYTESTCAPI_NEED_INTERNAL_API macro which is defined
  by _testcapi sub-modules which need the internal C API
  (pycore_modsupport.h): exceptions.c, float.c, vectorcall.c,
  watchers.c.
* Remove Include/cpython/modsupport.h header file.
  Include/modsupport.h no longer includes the removed header file.
* Fix mypy clinic.py
2023-10-17 14:30:31 +02:00
Filipe Laíns b883cad06b
GH-103480: fix sysconfig package not being installed (#110792) 2023-10-13 01:13:28 +00:00
Victor Stinner 467abfe1d4
gh-110313: Revert "gh-90108: Disable LTO on _freeze_module and _teste… (#110720)
gh-110313: Revert "gh-90108: Disable LTO on _freeze_module and _testembed (#109581)"

This reverts commit 3e3a7da590.
2023-10-11 15:54:40 +00:00
Lysandros Nikolaou 01481f2dc1
gh-104169: Refactor tokenizer into lexer and wrappers (#110684)
* The lexer, which include the actual lexeme producing logic, goes into
  the `lexer` directory.
* The wrappers, one wrapper per input mode (file, string, utf-8, and
  readline), go into the `tokenizer` directory and include logic for
  creating a lexer instance and managing the buffer for different modes.
---------

Co-authored-by: Pablo Galindo <pablogsal@gmail.com>
Co-authored-by: blurb-it[bot] <43283697+blurb-it[bot]@users.noreply.github.com>
2023-10-11 15:14:44 +00:00
Nikita Sobolev 732532b0af
gh-108303: Move all inspect test files to `test_inspect/` (#109607) 2023-10-10 22:15:11 +02:00
Eric Snow 7bd560ce8d
gh-76785: Add SendChannel.send_buffer() (#110246)
(This is still a test module.)
2023-10-09 07:39:51 -06:00
Victor Stinner a155f9f342
gh-103053: Fix make check-clean-src: check "python" program (#110449)
"make check-clean-src" now also checks if the "python" program is
found in the source directory: fail with an error if it does exist.
2023-10-06 00:46:52 +00:00
Victor Stinner fb6c4ed2bb
gh-110429: Fix race condition in "make regen-all" (#110433)
"make regen-pegen" now creates a temporary file called "parser.c.new"
instead of "parser.new.c". Previously, if "make clinic" was run in
parallel with "make regen-all", clinic may try but fail to open
"parser.new.c" if the temporay file was removed in the meanwhile.
2023-10-05 22:26:37 +02:00