* Add mimalloc v2.12
Modified src/alloc.c to remove include of alloc-override.c and not
compile new handler.
Did not include the following files:
- include/mimalloc-new-delete.h
- include/mimalloc-override.h
- src/alloc-override-osx.c
- src/alloc-override.c
- src/static.c
- src/region.c
mimalloc is thread safe and shares a single heap across all runtimes,
therefore finalization and getting global allocated blocks across all
runtimes is different.
* mimalloc: minimal changes for use in Python:
- remove debug spam for freeing large allocations
- use same bytes (0xDD) for freed allocations in CPython and mimalloc
This is important for the test_capi debug memory tests
* Don't export mimalloc symbol in libpython.
* Enable mimalloc as Python allocator option.
* Add mimalloc MIT license.
* Log mimalloc in Lib/test/pythoninfo.py.
* Document new mimalloc support.
* Use macro defs for exports as done in:
https://github.com/python/cpython/pull/31164/
Co-authored-by: Sam Gross <colesbury@gmail.com>
Co-authored-by: Christian Heimes <christian@python.org>
Co-authored-by: Victor Stinner <vstinner@python.org>
Move the following private functions and structures to
pycore_modsupport.h internal C API:
* _PyArg_BadArgument()
* _PyArg_CheckPositional()
* _PyArg_NoKeywords()
* _PyArg_NoPositional()
* _PyArg_ParseStack()
* _PyArg_ParseStackAndKeywords()
* _PyArg_Parser structure
* _PyArg_UnpackKeywords()
* _PyArg_UnpackKeywordsWithVararg()
* _PyArg_UnpackStack()
* _Py_ANY_VARARGS()
Changes:
* Python/getargs.h now includes pycore_modsupport.h to export
functions.
* clinic.py now adds pycore_modsupport.h when one of these functions
is used.
* Add pycore_modsupport.h includes when a C extension uses one of
these functions.
* Define Py_BUILD_CORE_MODULE in C extensions which now include
directly or indirectly (via code generated by Argument Clinic)
pycore_modsupport.h:
* _csv
* _curses_panel
* _dbm
* _gdbm
* _multiprocessing.posixshmem
* _sqlite.row
* _statistics
* grp
* resource
* syslog
* _testcapi: bad_get() no longer uses METH_FASTCALL calling
convention but METH_VARARGS. Replace _PyArg_UnpackStack() with
PyArg_ParseTuple().
* _testcapi: add PYTESTCAPI_NEED_INTERNAL_API macro which is defined
by _testcapi sub-modules which need the internal C API
(pycore_modsupport.h): exceptions.c, float.c, vectorcall.c,
watchers.c.
* Remove Include/cpython/modsupport.h header file.
Include/modsupport.h no longer includes the removed header file.
* Fix mypy clinic.py
* The lexer, which include the actual lexeme producing logic, goes into
the `lexer` directory.
* The wrappers, one wrapper per input mode (file, string, utf-8, and
readline), go into the `tokenizer` directory and include logic for
creating a lexer instance and managing the buffer for different modes.
---------
Co-authored-by: Pablo Galindo <pablogsal@gmail.com>
Co-authored-by: blurb-it[bot] <43283697+blurb-it[bot]@users.noreply.github.com>
"make regen-pegen" now creates a temporary file called "parser.c.new"
instead of "parser.new.c". Previously, if "make clinic" was run in
parallel with "make regen-all", clinic may try but fail to open
"parser.new.c" if the temporay file was removed in the meanwhile.
* _add_python_opts() now handles cross compilation and HOSTRUNNER.
* display_header() now tells if Python is cross-compiled, display
HOSTRUNNER, and get the host platform.
* Remove Tools/scripts/run_tests.py script.
* Remove "make hostrunnertest": use "make buildbottest"
or "make test" instead.
Split test_gdb.py file into a test_gdb package made of multiple
tests, so tests can now be run in parallel.
* Create Lib/test/test_gdb/ directory.
* Split test_gdb.py into multiple files in Lib/test/test_gdb/
directory.
* Move Lib/test/gdb_sample.py to Lib/test/test_gdb/ directory.
Update get_sample_script(): use __file__ to locate gdb_sample.py.
* Move gdb_has_frame_select() and HAS_PYUP_PYDOWN to test_misc.py.
* Explicitly skip test_gdb on Windows. Previously, test_gdb was
skipped even if gdb was available because of
gdb_has_frame_select().
* Add --fast-ci and --slow-ci options to libregrtest:
* --fast-ci uses a default timeout of 10 minutes and "-u all,-cpu"
(skip slowest tests).
* --slow-ci uses a default timeout of 20 minues and "-u all" (run
all tests).
* regrtest header now lists test resources.
* Makefile changes:
* "make test", "make hostrunnertest" and "make coverage-report" now
use --fast-ci option and TESTTIMEOUT variable.
* "make buildbottest" now uses "--slow-ci". Remove options which
became redundant with "--slow-ci".
* "make testall" and "make testuniversal" now use --slow-ci option
and TESTTIMEOUT variable.
* "make testall" now uses "find -exec rm ..." instead of
"find ... -print|xargs rm ...", same as "make clean".
* GitHub Actions workflow:
* Ubuntu and Address Sanitizer jobs now use "make test". Remove
options which became redundant with "--fast-ci".
* Windows jobs now use --fast-ci option.
* Use -j0 to detect the number of CPUs.
* Set Makefile TESTTIMEOUT default to an empty string, since
--slow-ci and --fast-ci use different default timeout. It's now
accepted to pass "--timeout=" to regrtest: treated as not timeout.
* Tools/scripts/run_tests.py now uses --fast-ci option.
* Tools/buildbot/test.bat now uses --slow-ci option. Remove
--timeout=1200 option, redundant with --slow-ci.
LTO optimization is nice to make Python faster, but _freeze_module
and _testembed performance is not important. Using LTO to build these
two programs make a whole Python build way slower, especially
combined with a sanitizer (like ASAN).
PyMutex is a one byte lock with fast, inlineable lock and unlock functions for the common uncontended case. The design is based on WebKit's WTF::Lock.
PyMutex is built using the _PyParkingLot APIs, which provides a cross-platform futex-like API (based on WebKit's WTF::ParkingLot). This internal API will be used for building other synchronization primitives used to implement PEP 703, such as one-time initialization and events.
This also includes tests and a mini benchmark in Tools/lockbench/lockbench.py to compare with the existing PyThread_type_lock.
Uncontended acquisition + release:
* Linux (x86-64): PyMutex: 11 ns, PyThread_type_lock: 44 ns
* macOS (arm64): PyMutex: 13 ns, PyThread_type_lock: 18 ns
* Windows (x86-64): PyMutex: 13 ns, PyThread_type_lock: 38 ns
PR Overview:
The primary purpose of this PR is to implement PyMutex, but there are a number of support pieces (described below).
* PyMutex: A 1-byte lock that doesn't require memory allocation to initialize and is generally faster than the existing PyThread_type_lock. The API is internal only for now.
* _PyParking_Lot: A futex-like API based on the API of the same name in WebKit. Used to implement PyMutex.
* _PyRawMutex: A word sized lock used to implement _PyParking_Lot.
* PyEvent: A one time event. This was used a bunch in the "nogil" fork and is useful for testing the PyMutex implementation, so I've included it as part of the PR.
* pycore_llist.h: Defines common operations on doubly-linked list. Not strictly necessary (could do the list operations manually), but they come up frequently in the "nogil" fork. ( Similar to https://man.freebsd.org/cgi/man.cgi?queue)
---------
Co-authored-by: Eric Snow <ericsnowcurrently@gmail.com>
Fix a race condition in "make regen-all". The deepfreeze.c source and
files generated by Argument Clinic are now generated or updated
before generating "global objects". Previously, some identifiers may
miss depending on the order in which these files were generated.
* "make regen-global-objects": Make sure that deepfreeze.c is
generated and up to date, and always run "make clinic".
* "make clinic" no longer runs generate_global_objects.py script.
* "make regen-deepfreeze" now only updates deepfreeze.c (C file).
It doesn't build deepfreeze.o (object) anymore.
* Remove misleading messages in "make regen-global-objects" and
"make clinic". They are now outdated, these commands are now
safe to use.
* Document generates files in Doc/using/configure.rst.
Co-authored-by: Erlend E. Aasland <erlend@python.org>
Statistics gathering is now off by default. Use the "-X pystats"
command line option or set the new PYTHONSTATS environment variable
to 1 to turn statistics gathering on at Python startup.
Statistics are no longer dumped at exit if statistics gathering was
off or statistics have been cleared.
Changes:
* Add PYTHONSTATS environment variable.
* sys._stats_dump() now returns False if statistics are not dumped
because they are all equal to zero.
* Add PyConfig._pystats member.
* Add tests on sys functions and on setting PyConfig._pystats to 1.
* Add Include/cpython/pystats.h and Include/internal/pycore_pystats.h
header files.
* Rename '_py_stats' variable to '_Py_stats'.
* Exclude Include/cpython/pystats.h from the Py_LIMITED_API.
* Move pystats.h include from object.h to Python.h.
* Add _Py_StatsOn() and _Py_StatsOff() functions. Remove
'_py_stats_struct' variable from the API: make it static in
specialize.c.
* Document API in Include/pystats.h and Include/cpython/pystats.h.
* Complete pystats documentation in Doc/using/configure.rst.
* Don't write "all zeros" stats: if _stats_off() and _stats_clear()
or _stats_dump() were called.
* _PyEval_Fini() now always call _Py_PrintSpecializationStats() which
does nothing if stats are all zeros.
Co-authored-by: Michael Droettboom <mdboom@gmail.com>
This adds a new header that provides atomic operations on common data
types. The intention is that this will be exposed through Python.h,
although that is not the case yet. The only immediate use is in
the test file.
Co-authored-by: Sam Gross <colesbury@gmail.com>
Remove these private functions from the public C API:
* _PyRun_AnyFileObject()
* _PyRun_InteractiveLoopObject()
* _PyRun_SimpleFileObject()
* _Py_SourceAsString()
Move them to the internal C API: add a new pycore_pythonrun.h header
file. No longer export these functions.
Remove the private _Py_Identifier type and related private functions
from the public C API:
* _PyObject_GetAttrId()
* _PyObject_LookupSpecialId()
* _PyObject_SetAttrId()
* _PyType_LookupId()
* _Py_IDENTIFIER()
* _Py_static_string()
* _Py_static_string_init()
Move them to the internal C API: add a new pycore_identifier.h header
file. No longer export these functions.
* Move test_cppext to its own directory
* Rename setup_testcppext.py to setup.py
* Rename _testcppext.cpp to extension.cpp
* The source (extension.cpp) is now also copied by the test.
* Move Python scripts related to test_module to this new directory:
good_getattr.py and bad_getattrX.py scripts.
* Move Lib/test/test_module.py to Lib/test/test_module/__init__.py.
* Instead of calling get_identifiers_and_strings(), extract identifiers and strings from pycore_global_strings.h.
* Avoid ast.literal_eval(), it's very slow.
The _xxsubinterpreters module should not rely on internal API. Some of the functions it uses were recently moved there however. Here we move them back (and expose them properly).
Move the private _PyInterpreterID C API to the internal C API: add a
new pycore_interp_id.h header file.
Remove Include/interpreteridobject.h and
Include/cpython/interpreteridobject.h header files.
in the case of an empty FRAMEWORKALTINSTALLLAST, this patch prevents leaving
an astray linebreak and two tabs in the resulting Makefile.
Before change:
```
.PHONY: commoninstall
commoninstall: check-clean-src \
altbininstall libinstall inclinstall libainstall \
sharedinstall altmaninstall \
```
After change (with empty FRAMEWORKALTINSTALLLAST):
```
.PHONY: commoninstall
commoninstall: check-clean-src \
altbininstall libinstall inclinstall libainstall \
sharedinstall altmaninstall
```
Dino Viehland
Donghee Na
Victor Stinner
Filipe Laíns
Lysandros Nikolaou
Nikita Sobolev
Eric Snow
Sam Gross
Mark Shannon
Serhiy Storchaka
Irit Katriel
Ken Jin
Guido van Rossum
Erlend E. Aasland
Moritz Neeb
Brandt Bucher