The libexpat 2.4.1 upgrade from introduced the following new exported symbols:
* `testingAccountingGetCountBytesDirect`
* `testingAccountingGetCountBytesIndirect`
* `unsignedCharToPrintable`
* `XML_SetBillionLaughsAttackProtectionActivationThreshold`
* `XML_SetBillionLaughsAttackProtectionMaximumAmplification`
We need to adjust [Modules/expat/pyexpatns.h](https://github.com/python/cpython/blob/master/Modules/expat/pyexpatns.h)
(The newer libexpat upgrade has no new symbols).
Automerge-Triggered-By: GH:gpshead
asyncio/taskgroups.py is an adaptation of taskgroup.py from EdgeDb, with the following key changes:
- Allow creating new tasks as long as the last task hasn't finished
- Raise [Base]ExceptionGroup (directly) rather than TaskGroupError deriving from MultiError
- Instead of monkey-patching the parent task's cancel() method,
add a new public API to Task
The Task class has a new internal flag, `_cancel_requested`, which is set when `.cancel()` is called successfully. The `.cancelling()` method returns the value of this flag. Further `.cancel()` calls while this flag is set return False. To reset this flag, call `.uncancel()`.
Thus, a Task that catches and ignores `CancelledError` should call `.uncancel()` if it wants to be cancellable again; until it does so, it is deemed to be busy with uninterruptible cleanup.
This new Task API helps solve the problem where TaskGroup needs to distinguish between whether the parent task being cancelled "from the outside" vs. "from inside".
Co-authored-by: Yury Selivanov <yury@edgedb.com>
Co-authored-by: Andrew Svetlov <andrew.svetlov@gmail.com>
We're no longer using _Py_IDENTIFIER() (or _Py_static_string()) in any core CPython code. It is still used in a number of non-builtin stdlib modules.
The replacement is: PyUnicodeObject (not pointer) fields under _PyRuntimeState, statically initialized as part of _PyRuntime. A new _Py_GET_GLOBAL_IDENTIFIER() macro facilitates lookup of the fields (along with _Py_GET_GLOBAL_STRING() for non-identifier strings).
https://bugs.python.org/issue46541#msg411799 explains the rationale for this change.
The core of the change is in:
* (new) Include/internal/pycore_global_strings.h - the declarations for the global strings, along with the macros
* Include/internal/pycore_runtime_init.h - added the static initializers for the global strings
* Include/internal/pycore_global_objects.h - where the struct in pycore_global_strings.h is hooked into _PyRuntimeState
* Tools/scripts/generate_global_objects.py - added generation of the global string declarations and static initializers
I've also added a --check flag to generate_global_objects.py (along with make check-global-objects) to check for unused global strings. That check is added to the PR CI config.
The remainder of this change updates the core code to use _Py_GET_GLOBAL_IDENTIFIER() instead of _Py_IDENTIFIER() and the related _Py*Id functions (likewise for _Py_GET_GLOBAL_STRING() instead of _Py_static_string()). This includes adding a few functions where there wasn't already an alternative to _Py*Id(), replacing the _Py_Identifier * parameter with PyObject *.
The following are not changed (yet):
* stop using _Py_IDENTIFIER() in the stdlib modules
* (maybe) get rid of _Py_IDENTIFIER(), etc. entirely -- this may not be doable as at least one package on PyPI using this (private) API
* (maybe) intern the strings during runtime init
https://bugs.python.org/issue46541
Cast PY_TIMEOUT_MAX to double, not to _PyTime_t.
Fix the clang warning:
Modules/_threadmodule.c:1648:26: warning: implicit conversion from
'_PyTime_t' (aka 'long') to 'double' changes value from
9223372036854775 to 9223372036854776
[-Wimplicit-const-int-float-conversion]
double timeout_max = (_PyTime_t)PY_TIMEOUT_MAX * 1e-6;
^~~~~~~~~~~~~~~~~~~~~~~~~ ~
ctypes.CFUNCTYPE() and ctypes.WINFUNCTYPE() now fail to create the
type if its "_argtypes_" member contains too many arguments.
Previously, the error was only raised when calling a function.
Change also how CFUNCTYPE() and WINFUNCTYPE() handle KeyError to
prevent creating a chain of exceptions if ctypes.CFuncPtr raises an
error.
Convert the PyType_SUPPORTS_WEAKREFS() macro to a regular function.
It no longer access the PyTypeObject.tp_weaklistoffset member
directly.
Add _PyType_SUPPORTS_WEAKREFS() static inline functions, used
internally by Python for best performance.
Add a new _PyType_GetSubclasses() function to get type's subclasses.
_PyType_GetSubclasses(type) returns a list which holds strong
refererences to subclasses. It is safer than iterating on
type->tp_subclasses which yields weak references and can be modified
in the loop.
_PyType_GetSubclasses(type) now holds a reference to the tp_subclasses
dict while creating the list of subclasses.
set_collection_flag_recursive() of _abc.c now uses
_PyType_GetSubclasses().
The signal module now creates its struct_siginfo type as a heap type
using PyStructSequence_NewType(), rather than using a static type.
Add 'siginfo_type' member to the global signal_state_t structure.
The _curses module now creates its ncurses_version type as a heap
type using PyStructSequence_NewType(), rather than using a static
type.
* Move _PyStructSequence_FiniType() definition to pycore_structseq.h.
* test.pythoninfo: log curses.ncurses_version.
The time module now creates its struct_time type as a heap
type using PyStructSequence_NewType(), rather than using a static
type.
* Add a module state to the time module: add traverse, clear and free
functions.
* Use PyModule_AddType().
* Remove the 'initialized' variable.
Py_EndInterpreter() now explicitly untracks all objects currently
tracked by the GC. Previously, if an object was used later by another
interpreter, calling PyObject_GC_UnTrack() on the object crashed if
the previous or the next object of the PyGC_Head structure became a
dangling pointer.
Previously, the main interpreter was allocated on the heap during runtime initialization. Here we instead embed it into _PyRuntimeState, which means it is statically allocated as part of the _PyRuntime global. The same goes for the initial thread state (of each interpreter, including the main one). Consequently there are fewer allocations during runtime/interpreter init, fewer possible failures, and better memory locality.
FYI, this also helps efforts to consolidate globals, which in turns helps work on subinterpreter isolation.
https://bugs.python.org/issue45953
Move almost all private functions of Include/cpython/fileutils.h to
the internal C API Include/internal/pycore_fileutils.h.
Only keep _Py_fopen_obj() in Include/cpython/fileutils.h, since it's
used by _testcapi which must not use the internal C API.
Move EncodeLocaleEx() and DecodeLocaleEx() functions from _testcapi
to _testinternalcapi, since the C API moved to the internal C API.
The idea is to ensure that module `xml.parsers.expat.errors`
contains all known error codes and messages,
even when CPython is compiled or run with an outdated version of libexpat.
https://bugs.python.org/issue45321
* Do not PUSH/POP traceback or type to the stack as part of exc_info
* Remove exc_traceback and exc_type from _PyErr_StackItem
* Add to what's new, because this change breaks things like Cython
This defines VPATH differently in PGO instrumentation builds, to account for a different default output directory. It also adds sys._vpath on Windows to make the value available to sysconfig so that it can be used in tests.
When Python is embedded in other applications, it is not easy to determine which version of Python is being used. This change exposes the Python version as part of the API data. Tools like Austin (https://github.com/P403n1x87/austin) can benefit from this data when targeting applications like uWSGI, as the Python version can then be inferred systematically by looking at the exported symbols rather than relying on unreliable pattern matching or other hacks (like remote code execution etc...).
Automerge-Triggered-By: GH:pablogsal
PyInterpreterState_Main() is a plain function exposed in the public C-API. For internal usage we can take the more efficient approach in this PR.
https://bugs.python.org/issue46008
This simplifies new_threadstate(). We also rename _PyThreadState_Init() to _PyThreadState_SetCurrent() to reflect what it actually does.
https://bugs.python.org/issue46008
This parallels _PyRuntimeState.interpreters. Doing this helps make it more clear what part of PyInterpreterState relates to its threads.
https://bugs.python.org/issue46008
* Check NS API return values for NULL to prevent segfault in
``_bootstrap_python``.
* Set modPathInitialized to 1 so the ``decode_to_dict`` path is used.
Signed-off-by: Christian Heimes <christian@python.org>
For very large numbers use divide-and-conquer algorithm for getting
benefit of Karatsuba multiplication of large numbers.
Do calculations completely in C unsigned long long instead of Python
integers if possible.
The getpath.py file is frozen at build time and executed as code over a namespace. It is never imported, nor is it meant to be importable or reusable. However, it should be easier to read, modify, and patch than the previous code.
This commit attempts to preserve every previously tested quirk, but these may be changed in the future to better align platforms.
``Modules/Setup.stdlib`` contains ``Setup`` lines for all stdlib extension modules for which ``configure`` has detected their dependencies. The file is not used yet and still under development. To use the file, do ``ln -sfr Modules/Setup.stdlib Modules/Setup.local``.
``configure`` now uses a standardized format to forward state, compiler
flags, and linker flags to ``Makefile``, ``setup.py``, and
``Modules/Setup``. ``makesetup`` use the new variables by default if a
module line does not contain any compiler or linker flags. ``setup.py``
has a new function ``addext()``.
For a module ``egg``, configure adds:
* ``MODULE_EGG`` with value yes, missing, disabled, or n/a
* ``MODULE_EGG_CFLAGS``
* ``MODULE_EGG_LDFLAGS``
``Makefile.pre.in`` may also provide ``MODULE_EGG_DEPS`` that lists
dependencies such as header files and static libs.
Signed-off-by: Christian Heimes <christian@python.org>
* bpo-45512: Raise sqlite3.Connection.__init__ is called with bad isolation level
* Also explicitly test allowed isolation levels
* Use subTest for better error messages if something goes wrong
* Update Lib/test/test_sqlite3/test_dbapi.py
Co-authored-by: Dong-hee Na <donghee.na92@gmail.com>
Co-authored-by: Dong-hee Na <donghee.na92@gmail.com>
Settings for :mod:`pyexpat` C extension are now detected by ``configure``.
The bundled ``expat`` library is built in ``Makefile``.
Signed-off-by: Christian Heimes <christian@python.org>
Settings for :mod:`decimal` internal C extension are now detected by
:program:`configure`. The bundled `libmpdec` library is built in
``Makefile``.
Signed-off-by: Christian Heimes <christian@python.org>
Some test cases don't work when test modules are static extensions.
Add dependency on Modules/config.c to trigger a rebuild whenever a
module build type is changed.
``makesetup`` puts shared extensions into ``Modules/`` directory. Create
symlinks from pybuilddir so the extensions can be imported.
Note: It is not possible to use the content of pybuilddir.txt as a build
target. Makefile evaluates target variables in the first pass. The
pybuilddir.txt file does not exist at that point.
Add Modules subdirs to SRCDIRS to generate directories for out-of-tree
object files.
Debian wants ncurses lib. Works on Fedora, too.
Debian also needs pkg-config to detect correct flags.
Remove more outdated comments. Makefile now tracks header dependencies
-lintl is injected by configure when needed. Build _dbm with
gdbm-compat.
Group some modules by purpose. socket, select, and mmap work on Windows,
too.
(original patch by eryksun)
Correctly hand various failure modes when resizing an mmap on Windows:
* Resizing a pagefile-backed mmap now creates a new mmap and copies data
* Attempting to resize when another mapping is held on the same file raises an OSError
* Attempting to resize a nametagged mmap raises an OSError if another mapping is held with the same nametag
* record which modules are build as shared extensions
* put object files in same directory as source files
* remove dependency on deleted _math.c
Signed-off-by: Christian Heimes <christian@python.org>
Adds _ctypes, _decimal, _multiprocessing, _posixshmem, _scproxy, _sqlite3,
and _uuid.
Use Makefile variables to build OpenSSL and TCL/TK modules.
Drop ``-lm`` from math lines. Extensions are always linked against libm.
Signed-off-by: Christian Heimes <christian@python.org>
``setup.py`` and ``makesetup`` now track build dependencies on all Python
header files and module specific header files.
Signed-off-by: Christian Heimes <christian@python.org>
The :mod:`math` and :mod:`cmath` implementation now require a C99 compatible
``libm`` and no longer ship with workarounds for missing acosh, asinh,
expm1, and log1p functions.
The changeset also removes ``_math.c`` and moves the last remaining
workaround into ``_math.h``. This simplifies static builds with
``Modules/Setup`` and resolves symbol conflicts.
Co-authored-by: Mark Dickinson <mdickinson@enthought.com>
Co-authored-by: Brett Cannon <brett@python.org>
Signed-off-by: Christian Heimes <christian@python.org>
The recently added PyConfig.stdlib_dir was being set with ".." entries. When __file__ was added for from modules this caused a problem on out-of-tree builds. This PR fixes that by normalizing "stdlib_dir" when it is calculated in getpath.c.
https://bugs.python.org/issue45506
* ``HAVE_EXPAT_CONFIG_H`` is not used by our code and not used by
system-wide expat header files
* ``USE_PYEXPAT_CAPI`` is no longer used by our code
* ``XML_POOR_ENTROPY`` should be defined in expat_config.h
Signed-off-by: Christian Heimes <christian@python.org>
setup.py no longer defines Py_BUILD_CORE_MODULE. Instead every
module defines the macro before #include "Python.h" unless
Py_BUILD_CORE_BUILTIN is already defined.
Py_BUILD_CORE_BUILTIN is defined for every module that is built by
Modules/Setup.
The PR also simplifies Modules/Setup. Makefile and makesetup
already define Py_BUILD_CORE_BUILTIN and include Modules/internal
for us.
Signed-off-by: Christian Heimes <christian@python.org>
The Python 3.11 limited C API no longer includes stdlib.h, stdio.h,
string.h and errno.h.
* Exclude Py_MEMCPY() from Python 3.11 limited C API.
* xxlimited C extension is now built with Python 3.11 limited C API.
Move Include/longobject.h non-limited API to a new
Include/cpython/longobject.h header file.
Move the following definitions to the internal C API:
* _PyLong_DigitValue
* _PyLong_FormatAdvancedWriter()
* _PyLong_FormatWriter()
Add PyThreadState_EnterTracing() and PyThreadState_LeaveTracing()
functions to the limited C API to suspend and resume tracing and
profiling.
Add an unit test on the PyThreadState C API to _testcapi.
Add also internal _PyThreadState_DisableTracing() and
_PyThreadState_ResetTracing().
Rename Include/namespaceobject.h to
Include/internal/pycore_namespace.h.
The _testmultiphase extension is now built with the
Py_BUILD_CORE_MODULE macro defined to access _PyNamespace_Type.
object.c: remove unused "pycore_context.h" include.
Move classobject.h, context.h, genobject.h and longintrepr.h header
files from Include/ to Include/cpython/.
Remove redundant "#ifndef Py_LIMITED_API" in context.h.
Remove explicit #include "longintrepr.h" in C files. It's not needed,
Python.h already includes it.
* Move _PyObject_VectorcallTstate() and _PyObject_FastCallTstate() to
pycore_call.h (internal C API).
* Convert PyObject_CallOneArg(), PyObject_Vectorcall(),
_PyObject_FastCall() and PyVectorcall_Function() static inline
functions to regular functions.
* Add _PyVectorcall_FunctionInline() static inline function.
* PyObject_Vectorcall(), _PyObject_FastCall(), and
PyObject_CallOneArg() now call _PyThreadState_GET() rather
than PyThreadState_Get().
They support now splitting escape sequences between input chunks.
Add the third parameter "final" in codecs.raw_unicode_escape_decode().
It is True by default to match the former behavior.
They support now splitting escape sequences between input chunks.
Add the third parameter "final" in codecs.unicode_escape_decode().
It is True by default to match the former behavior.
Move Include/pystrhex.h to Include/internal/pycore_strhex.h.
The header file only contains private functions.
The following C extensions are now built with Py_BUILD_CORE_MODULE
macro defined to get access to the internal C API:
* _blake2
* _hashopenssl
* _md5
* _sha1
* _sha3
* _ssl
* binascii
Redefining the PyThreadState_GET() macro in pycore_pystate.h is
useless since it doesn't affect files not including it. Either use
_PyThreadState_GET() directly, or don't use pycore_pystate.h internal
C API. For example, the _testcapi extension don't use the internal C
API, but use the public PyThreadState_Get() function instead.
Replace PyThreadState_Get() with _PyThreadState_GET(). The
_PyThreadState_GET() macro is more efficient than PyThreadState_Get()
and PyThreadState_GET() function calls which call fail with a fatal
Python error.
posixmodule.c and _ctypes extension now include <windows.h> before
pycore header files (like pycore_call.h).
_PyTraceback_Add() now uses _PyErr_Fetch()/_PyErr_Restore() instead
of PyErr_Fetch()/PyErr_Restore().
The _decimal and _xxsubinterpreters extensions are now built with the
Py_BUILD_CORE_MODULE macro defined to get access to the internal C
API.
* Move _PyObject_CallNoArgs() to pycore_call.h (internal C API).
* _ssl, _sqlite and _testcapi extensions now call the public
PyObject_CallNoArgs() function, rather than _PyObject_CallNoArgs().
* _lsprof extension is now built with Py_BUILD_CORE_MODULE macro
defined to get access to internal _PyObject_CallNoArgs().
Fix typo in the private _PyObject_CallNoArg() function name: rename
it to _PyObject_CallNoArgs() to be consistent with the public
function PyObject_CallNoArgs().
This supercedes https://github.com/python/cpython/pull/3437 and fuzzes the method we recommend for unsafe inputs, `ast.literal_eval`. This should exercise the tokenizer and parser.
Add a private C API for deadlines: add _PyDeadline_Init() and
_PyDeadline_Get() functions.
* Add _PyTime_Add() and _PyTime_Mul() functions which compute t1+t2
and t1*t2 and clamp the result on overflow.
* _PyTime_MulDiv() now uses _PyTime_Add() and _PyTime_Mul().
If the DEBUG_STATS debug flag is set, gc_collect_main() now uses
_PyTime_GetPerfCounter() instead of _PyTime_GetMonotonicClock() to
measure the elapsed time.
On Windows, _PyTime_GetMonotonicClock() only has a resolution of 15.6
ms, whereas _PyTime_GetPerfCounter() is closer to a resolution of 100
ns.
PyThread_acquire_lock_timed() now clamps the timeout into the
[_PyTime_MIN; _PyTime_MAX] range (_PyTime_t type) if it is too large,
rather than calling Py_FatalError() which aborts the process.
PyThread_acquire_lock_timed() no longer uses
MICROSECONDS_TO_TIMESPEC() to compute sem_timedwait() argument, but
_PyTime_GetSystemClock() and _PyTime_AsTimespec_truncate().
Fix _thread.TIMEOUT_MAX value on Windows: the maximum timeout is
0x7FFFFFFF milliseconds (around 24.9 days), not 0xFFFFFFFF
milliseconds (around 49.7 days).
Set PY_TIMEOUT_MAX to 0x7FFFFFFF milliseconds, rather than 0xFFFFFFFF
milliseconds.
Fix PY_TIMEOUT_MAX overflow test: replace (us >= PY_TIMEOUT_MAX) with
(us > PY_TIMEOUT_MAX).
Add the _PyTime_AsTimespec_clamp() function: similar to
_PyTime_AsTimespec(), but clamp to _PyTime_t min/max and don't raise
an exception.
PyThread_acquire_lock_timed() now uses _PyTime_AsTimespec_clamp() to
remove the Py_UNREACHABLE() code path.
* Add _PyTime_AsTime_t() function.
* Add PY_TIME_T_MIN and PY_TIME_T_MAX constants.
* Replace _PyTime_AsTimeval_noraise() with _PyTime_AsTimeval_clamp().
* Add pytime_divide_round_up() function.
* Fix integer overflow in pytime_divide().
* Add pytime_divmod() function.
During runtime startup we figure out the stdlib dir but currently throw that information away. This change preserves it and exposes it via PyConfig.stdlib_dir, _Py_GetStdlibDir(), and sys._stdlib_dir.
https://bugs.python.org/issue45211
This accomplishes 2 things:
* consolidates some common code between getpath.c and getpathp.c
* makes the helpers available to code in other files
FWIW, the signature of the join_relfile() function (in fileutils.c) intentionally mirrors that of Windows' PathCchCombineEx().
Note that this change is mostly moving code around. No behavior is meant to change.
https://bugs.python.org/issue45211
Having `operator.call(obj, arg)` mean `type(obj).__call__(obj, arg)` is
consistent with the other dunder operators. The semantics with `*args,
**kwargs` then follow naturally from the single-arg semantics.
On Windows, time.sleep() now uses a waitable timer which has a
resolution of 100 ns (10^-7 sec). Previously, it had a solution of 1
ms (10^-3 sec).
* On Windows, time.sleep() now calls PyErr_CheckSignals() before
resetting the SIGINT event.
* Add _PyTime_As100Nanoseconds() function.
* Complete and update time.sleep() documentation.
Co-authored-by: Livius <egyszeregy@freemail.hu>
Refactor pytime.c:
* Add pytime_from_nanoseconds() and pytime_as_nanoseconds(),
and use explicitly these functions
* Add two empty lines between functions
* PEP 7: add braces { ... }
* C99: declare variables where they are set
* Rename private functions to lowercase
* Rename error_time_t_overflow() to pytime_time_t_overflow()
* Rename win_perf_counter_frequency() to py_win_perf_counter_frequency()
* py_get_monotonic_clock(): add an assertion to detect overflow when
mach_absolute_time() unsigned uint64_t is casted to _PyTime_t
(signed int64_t).
_testcapi: use _PyTime_FromNanoseconds().