Commit Graph

1563 Commits

Author SHA1 Message Date
Sam Gross a3af3cb4f4
gh-110481: Implement inter-thread queue for biased reference counting (#114824)
Biased reference counting maintains two refcount fields in each object:
`ob_ref_local` and `ob_ref_shared`. The true refcount is the sum of these two
fields. In some cases, when refcounting operations are split across threads,
the ob_ref_shared field can be negative (although the total refcount must be
at least zero). In this case, the thread that decremented the refcount
requests that the owning thread give up ownership and merge the refcount
fields.
2024-02-09 17:08:32 -05:00
adang1345 5914a211ef
gh-115167: Exclude vcruntime140_threads.dll from Windows build output (GH-115176) 2024-02-08 21:42:45 +00:00
Erlend E. Aasland 11ac6f5354
gh-115009: Update Windows installer to use SQLite 3.45.1 (#115065) 2024-02-06 23:44:14 +01:00
Zachary Ware 01dceba13e
gh-109991: Update Windows build to use OpenSSL 3.0.13 (#115043) 2024-02-05 23:10:55 +00:00
Zachary Ware 618d7256e7
gh-111239: Update Windows build to use zlib 1.3.1 (GH-114877) 2024-02-01 17:54:02 -06:00
Brandt Bucher f6d9e5926b
GH-113464: Add a JIT backend for tier 2 (GH-113465)
Add an option (--enable-experimental-jit for configure-based builds
or --experimental-jit for PCbuild-based ones) to build an
*experimental* just-in-time compiler, based on copy-and-patch (https://fredrikbk.com/publications/copy-and-patch.pdf).

See Tools/jit/README.md for more information on how to install the required build-time tooling.
2024-01-28 18:48:48 -08:00
Sam Gross b52fc70d1a
gh-112529: Implement GC for free-threaded builds (#114262)
* gh-112529: Implement GC for free-threaded builds

This implements a mark and sweep GC for the free-threaded builds of
CPython. The implementation relies on mimalloc to find GC tracked
objects (i.e., "containers").
2024-01-25 10:27:36 -08:00
Steve Dower 665b8f365e
gh-113655: Revert extra stack reserve in PGO builds unless UseExtraStackReserve=true (GH-114263) 2024-01-22 21:19:16 +00:00
Sam Gross 1d6d5e854c
gh-112529: Use GC heaps for GC allocations in free-threaded builds (gh-114157)
* gh-112529: Use GC heaps for GC allocations in free-threaded builds

The free-threaded build's garbage collector implementation will need to
find GC objects by traversing mimalloc heaps. This hooks up the
allocation calls with the correct heaps by using a thread-local
"current_obj_heap" variable.

* Refactor out setting heap based on type
2024-01-21 01:14:45 +09:00
Steve Dower f56d132deb
gh-112984 Update Windows build and installer for free-threaded builds (GH-113129) 2024-01-17 21:52:23 +00:00
Steve Dower 2e672f7ca6
gh-113655: Increase default stack size for PGO builds to avoid C stack exhaustion (GH-114148) 2024-01-16 22:02:20 +00:00
Brandt Bucher 30e6cbdba2
GH-113860: Get rid of `_PyUOpExecutorObject` (GH-113954) 2024-01-12 11:58:23 +00:00
Donghee Na 57bdc6c30d
gh-111968: Introduce _PyFreeListState and _PyFreeListState_GET API (gh-113584) 2024-01-10 08:04:41 +09:00
Steve Dower ad849b4ba0
gh-113650: Add workaround option for MSVC ARM64 bug affecting string encoding (GH-113836) 2024-01-09 17:32:22 +00:00
Steve Dower 92f96240d7
gh-113827: Move Windows frozen modules directory to allow PGO builds (GH-113828) 2024-01-08 21:51:39 +00:00
Pablo Galindo Salgado a03ec20bcd
gh-110721: Remove unused code from suggestions.c after moving PyErr_Display to use the traceback module (#113712) 2024-01-08 15:10:45 +00:00
Sam Gross 99854ce170
gh-113688: Split up gcmodule.c (gh-113715)
This splits part of Modules/gcmodule.c of into Python/gc.c, which
now contains the core garbage collection implementation. The Python
module remain in the Modules/gcmodule.c file.
2024-01-05 12:17:16 -08:00
Itamar Oren 178919cf21
gh-113258: Write frozen modules to the build tree on Windows (GH-113303)
This ensures the source directory is not modified at build time, and different builds (e.g. different versions or GIL vs no-GIL) do not have conflicts.
2024-01-03 17:30:20 +00:00
Erlend E. Aasland 14d4c7742a
gh-111973: Update Windows installer to use SQLite 3.44.2 (#113281) 2023-12-19 23:31:52 +01:00
Itamar Oren 2feec0fc7f
gh-113039: Avoid using leading dots in the include path for frozen getpath.py (GH-113022) 2023-12-18 17:04:40 +00:00
Steve Dower 79dad03747
gh-111650: Ensure pyconfig.h includes Py_GIL_DISABLED on Windows (GH-112778) 2023-12-13 15:38:45 +00:00
Eric Snow a49b427b02
gh-76785: More Fixes for test.support.interpreters (gh-113012)
This brings the module (along with the associated extension modules) mostly in sync with PEP 734.  There are only a few small things to wrap up.
2023-12-12 17:43:30 +00:00
Steve Dower 1c5fc02fd0
gh-71383: Update Tcl/Tk version in Windows to our patched build containing a targeted upstream fix (GH-112973) 2023-12-11 21:54:17 +00:00
Sam Gross db460735af
gh-112538: Add internal-only _PyThreadStateImpl "wrapper" for PyThreadState (gh-112560)
Every PyThreadState instance is now actually a _PyThreadStateImpl.
It is safe to cast from `PyThreadState*` to `_PyThreadStateImpl*` and back.
The _PyThreadStateImpl will contain fields that we do not want to expose
in the public C API.
2023-12-07 12:11:45 -07:00
Hugo van Kemenade 3b3ec0d77f
gh-111863: Rename `Py_NOGIL` to `Py_GIL_DISABLED` (#111864)
Rename Py_NOGIL to Py_GIL_DISABLED
2023-11-20 15:52:00 +02:00
Victor Stinner 55f3cce821
gh-111545: Test PyHash_GetFuncDef() function (#112098)
Add Modules/_testcapi/hash.c and Lib/test/test_capi/test_hash.py.
2023-11-15 03:41:29 +01:00
Victor Stinner 62802b6228
gh-111545: Add Include/cpython/pyhash.h header file (#112063)
Move non-limited C API to a new Include/cpython/pyhash.h header file.
2023-11-15 01:19:20 +01:00
Michael Droettboom bc12f79112
gh-111786: Optimize for space for _PyEval_EvalFrameDefault on MSVC for PGO (#111794)
In PGO mode, this function caused a compiler error in MSVC.
It turns out that optimizing for space only save the day, and is even faster.
However, without PGO, this is neither necessary nor slower.
2023-11-09 18:41:40 +00:00
Sam Gross 31c90d5838
gh-111569: Implement Python critical section API (gh-111571)
Critical sections are helpers to replace the global interpreter lock
with finer grained locking.  They provide similar guarantees to the GIL
and avoid the deadlock risk that plain locking involves.  Critical
sections are implicitly ended whenever the GIL would be released.  They
are resumed when the GIL would be acquired.  Nested critical sections
behave as if the sections were interleaved.
2023-11-08 15:39:29 -07:00
Guido van Rossum 7e135a48d6
gh-111520: Integrate the Tier 2 interpreter in the Tier 1 interpreter (#111428)
- There is no longer a separate Python/executor.c file.
- Conventions in Python/bytecodes.c are slightly different -- don't use `goto error`,
  you must use `GOTO_ERROR(error)` (same for others like `unused_local_error`).
- The `TIER_ONE` and `TIER_TWO` symbols are only valid in the generated (.c.h) files.
- In Lib/test/support/__init__.py, `Py_C_RECURSION_LIMIT` is imported from `_testcapi`.
- On Windows, in debug mode, stack allocation grows from 8MiB to 12MiB.
- **Beware!** This changes the env vars to enable uops and their debugging
  to `PYTHON_UOPS` and `PYTHON_LLTRACE`.
2023-11-01 13:13:02 -07:00
Sergey B Kirpichev 33ed5fa69d
gh-111495: add stub files for C API test modules (GH-111586)
This is to reduce merge conflicts (Modules/Setup.stdlib.in) for
subsequent pull requests for the issue.
2023-11-01 07:44:54 +02:00
Dino Viehland c42347d025
gh-90815: Exclude mimalloc .c files from Windows build (#111532)
* Don't include mimalloc .c's in Windows build
* Fix warnings on Windows related to mimalloc
2023-10-31 11:54:35 -07:00
Eric Snow c6fe0869ab
gh-76785: Move the Cross-Interpreter Code to Its Own File (gh-111502)
This is partly to clear this stuff out of pystate.c, but also in preparation for moving some code out of _xxsubinterpretersmodule.c.  This change also moves this stuff to the internal API (new: Include/internal/pycore_crossinterp.h).  @vstinner did this previously and I undid it.  Now I'm re-doing it. :/
2023-10-30 16:53:10 -06:00
Riccardo Ghetta 8eaa206fec
gh-109515: When generating deep frozen modules on Windows, use a list file instead of arguments (GH-109516) 2023-10-30 16:14:26 +00:00
Dino Viehland 05f2f0ac92
gh-90815: Add mimalloc memory allocator (#109914)
* Add mimalloc v2.12

Modified src/alloc.c to remove include of alloc-override.c and not
compile new handler.

Did not include the following files:

 - include/mimalloc-new-delete.h
 - include/mimalloc-override.h
 - src/alloc-override-osx.c
 - src/alloc-override.c
 - src/static.c
 - src/region.c

mimalloc is thread safe and shares a single heap across all runtimes,
therefore finalization and getting global allocated blocks across all
runtimes is different.

* mimalloc: minimal changes for use in Python:

 - remove debug spam for freeing large allocations
 - use same bytes (0xDD) for freed allocations in CPython and mimalloc
   This is important for the test_capi debug memory tests

* Don't export mimalloc symbol in libpython.
* Enable mimalloc as Python allocator option.
* Add mimalloc MIT license.
* Log mimalloc in Lib/test/pythoninfo.py.
* Document new mimalloc support.
* Use macro defs for exports as done in:
  https://github.com/python/cpython/pull/31164/

Co-authored-by: Sam Gross <colesbury@gmail.com>
Co-authored-by: Christian Heimes <christian@python.org>
Co-authored-by: Victor Stinner <vstinner@python.org>
2023-10-30 15:43:11 +00:00
Serhiy Storchaka b2ba298527
gh-111065: Add more tests for the C API with the PySys_ prefix (GH-111067)
* Move existing tests for PySys_GetObject() and PySys_SetObject() into
  specialized files.
* Add test for PySys_GetXOptions() using _testcapi.
* Add tests for PySys_FormatStdout(), PySys_FormatStderr(),
  PySys_WriteStdout() and PySys_WriteStderr() using ctypes.
2023-10-25 11:33:12 +03:00
Donghee Na 2dcc57008b
gh-109693: Remove pycore_atomic.h (gh-110992) 2023-10-18 00:33:50 +09:00
Victor Stinner be5e8a0103
gh-110964: Remove private _PyArg functions (#110966)
Move the following private functions and structures to
pycore_modsupport.h internal C API:

* _PyArg_BadArgument()
* _PyArg_CheckPositional()
* _PyArg_NoKeywords()
* _PyArg_NoPositional()
* _PyArg_ParseStack()
* _PyArg_ParseStackAndKeywords()
* _PyArg_Parser structure
* _PyArg_UnpackKeywords()
* _PyArg_UnpackKeywordsWithVararg()
* _PyArg_UnpackStack()
* _Py_ANY_VARARGS()

Changes:

* Python/getargs.h now includes pycore_modsupport.h to export
  functions.
* clinic.py now adds pycore_modsupport.h when one of these functions
  is used.
* Add pycore_modsupport.h includes when a C extension uses one of
  these functions.
* Define Py_BUILD_CORE_MODULE in C extensions which now include
  directly or indirectly (via code generated by Argument Clinic)
  pycore_modsupport.h:

  * _csv
  * _curses_panel
  * _dbm
  * _gdbm
  * _multiprocessing.posixshmem
  * _sqlite.row
  * _statistics
  * grp
  * resource
  * syslog

* _testcapi: bad_get() no longer uses METH_FASTCALL calling
  convention but METH_VARARGS. Replace _PyArg_UnpackStack() with
  PyArg_ParseTuple().
* _testcapi: add PYTESTCAPI_NEED_INTERNAL_API macro which is defined
  by _testcapi sub-modules which need the internal C API
  (pycore_modsupport.h): exceptions.c, float.c, vectorcall.c,
  watchers.c.
* Remove Include/cpython/modsupport.h header file.
  Include/modsupport.h no longer includes the removed header file.
* Fix mypy clinic.py
2023-10-17 14:30:31 +02:00
Lysandros Nikolaou 01481f2dc1
gh-104169: Refactor tokenizer into lexer and wrappers (#110684)
* The lexer, which include the actual lexeme producing logic, goes into
  the `lexer` directory.
* The wrappers, one wrapper per input mode (file, string, utf-8, and
  readline), go into the `tokenizer` directory and include logic for
  creating a lexer instance and managing the buffer for different modes.
---------

Co-authored-by: Pablo Galindo <pablogsal@gmail.com>
Co-authored-by: blurb-it[bot] <43283697+blurb-it[bot]@users.noreply.github.com>
2023-10-11 15:14:44 +00:00
Nikita Sobolev 9cfb4e0d1e
gh-110525: Add tests for internal `set` CAPI (GH-110630) 2023-10-10 19:00:05 +03:00
Steve Dower 12cc6792d0
gh-110437: Allow overriding VCRuntimeDLL with a semicolon separated list of DLLs to bundle (GH-110470) 2023-10-09 12:17:42 +01:00
Nikita Sobolev c49edd7d9c
gh-110525: Add CAPI tests for set and frozenset objects (GH-110526) 2023-10-09 11:57:48 +03:00
jtranquilli 201dc11aeb
gh-109286: Update Windows installer to use SQLite 3.43.1 (#110403)
Co-authored-by: blurb-it[bot] <43283697+blurb-it[bot]@users.noreply.github.com>
Co-authored-by: Erlend E. Aasland <erlend@python.org>
2023-10-06 17:12:15 +02:00
Sam Gross cf6f23b0e3
gh-88402: Add new sysconfig variables on Windows (GH-110049)
Co-authored-by: Filipe Laíns <filipe.lains@gmail.com>
2023-10-04 22:50:29 +00:00
Victor Stinner 6387b5313c
gh-108494: Document how to add a project in PCbuild/readme.txt (#110077)
Add _testclinic_limited to Tools/msi/test/test_files.wxs.
2023-10-02 18:53:38 +02:00
Victor Stinner f3df8fa669
gh-109566: PCbuild/rt.bat now uses --fast-ci (#110120)
Replace "--fail-env-changed --fail-rerun" with "--fast-ci".

Tools/buildbot/test.bat pass --slow-ci which has the priority over
--fast-ci.
2023-09-29 19:16:29 +00:00
Zachary Ware cf4c297256
gh-109991: Update Windows build to use OpenSSL 3.0.11 (GH-110054) 2023-09-28 22:58:13 +00:00
Victor Stinner fbfec5642e
gh-109566: regrtest reexecutes the process (#109909)
When --fast-ci or --slow-ci option is used, regrtest now replaces the
current process with a new process to add "-u -W default -bb -E"
options to Python.

Changes:

* PCbuild/rt.bat and Tools/scripts/run_tests.py no longer need to add
  "-u -W default -bb -E" options to Python: it's now done by
  regrtest.
* Fix Tools/scripts/run_tests.py: flush stdout before replacing the
  process. Previously, buffered messages were lost.
2023-09-26 20:46:52 +02:00
Victor Stinner 3cce6be06a
gh-109566: Fix typo in PCbuild/rt.bat (#109701) 2023-09-22 02:43:48 +00:00
Sam Gross 2aceb21ae6
gh-109693: Remove pycore_atomic_funcs.h (#109694)
_PyUnicode_FromId() now uses pyatomic.h functions instead.
2023-09-21 22:57:20 +02:00