Remove _random.Random.randbytes(): the C implementation of
randbytes(). Implement the method in Python to ease subclassing:
randbytes() now directly reuses getrandbits().
test.pythoninfo logs OpenSSL FIPS_mode() and Linux
/proc/sys/crypto/fips_enabled in a new "fips" section.
Co-Authored-By: Petr Viktorin <encukou@gmail.com>
New PyFrame_GetBack() function: get the frame next outer frame.
Replace frame->f_back with PyFrame_GetBack(frame) in most code but
frameobject.c, ceval.c and genobject.c.
Remove the following function from the C API:
* PyAsyncGen_ClearFreeLists()
* PyContext_ClearFreeList()
* PyDict_ClearFreeList()
* PyFloat_ClearFreeList()
* PyFrame_ClearFreeList()
* PyList_ClearFreeList()
* PySet_ClearFreeList()
* PyTuple_ClearFreeList()
Make these functions private, move them to the internal C API and
change their return type to void.
Call explicitly PyGC_Collect() to free all free lists.
Note: PySet_ClearFreeList() did nothing.
This allows the caller to avoid creation of an exception when the channel is empty (just like `dict.get()` works). `ChannelEmptyError` is still raised if no default is provided.
Automerge-Triggered-By: @ericsnowcurrently
PyFrame_GetCode(frame): return a borrowed reference to the frame
code.
Replace frame->f_code with PyFrame_GetCode(frame) in most code,
except in frameobject.c, genobject.c and ceval.c.
Also add PyFrame_GetLineNumber() to the limited C API.
This commit also allows to pass flags to the new parser in all interfaces and fixes a bug in the parser generator that was causing to inline rules with actions, making them disappear.
Optimize the subprocess module on FreeBSD using closefrom().
A single close(fd) syscall is cheap, but when sysconf(_SC_OPEN_MAX)
is high, the loop calling close(fd) on each file descriptor can take
several milliseconds.
The workaround on FreeBSD to improve performance was to load and
mount the fdescfs kernel module, but this is not enabled by default.
Initial patch by Ed Maste (emaste), Conrad Meyer (cem), Kyle Evans
(kevans) and Kubilay Kocak (koobs):
https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=242274
On FreeBSD, os.closerange(fd_low, fd_high) now calls
closefrom(fd_low) if fd_high is greater than or equal to
sysconf(_SC_OPEN_MAX).
Initial patch by Ed Maste (emaste), Conrad Meyer (cem), Kyle Evans
(kevans) and Kubilay Kocak (koobs):
https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=242274
Fix the Windows implementation of os.waitpid() for exit code
larger than "INT_MAX >> 8". The exit status is now interpreted as an
unsigned number.
os.waitstatus_to_exitcode() now accepts wait status larger than
INT_MAX.
* Replace PY_INT64_T with int64_t
* Replace PY_UINT32_T with uint32_t
* Replace PY_UINT64_T with uint64_t
sha3module.c no longer checks if PY_UINT64_T is defined since it's
always defined and uint64_t is always available on platforms
supported by Python.
Add random.randbytes() function and random.Random.randbytes()
method to generate random bytes.
Modify secrets.token_bytes() to use SystemRandom.randbytes()
rather than calling directly os.urandom().
Rename also genrand_int32() to genrand_uint32(), since it returns an
unsigned 32-bit integer, not a signed integer.
The _random module is now built with Py_BUILD_CORE_MODULE defined.
Add a new internal pycore_byteswap.h header file with the following
functions:
* _Py_bswap16()
* _Py_bswap32()
* _Py_bswap64()
Use these functions in _ctypes, sha256 and sha512 modules,
and also use in the UTF-32 encoder.
sha256, sha512 and _ctypes modules are now built with the internal
C API.
Rename _PyInterpreterState_GET_UNSAFE() to _PyInterpreterState_GET()
for consistency with _PyThreadState_GET() and to have a shorter name
(help to fit into 80 columns).
Add also "assert(tstate != NULL);" to the function.
Move the PyGC_Head structure and the following private macros to the
internal C API:
* _PyGCHead_FINALIZED()
* _PyGCHead_NEXT()
* _PyGCHead_PREV()
* _PyGCHead_SET_FINALIZED()
* _PyGCHead_SET_NEXT()
* _PyGCHead_SET_PREV()
* _PyGC_FINALIZED()
* _PyGC_PREV_MASK
* _PyGC_PREV_MASK_COLLECTING
* _PyGC_PREV_MASK_FINALIZED
* _PyGC_PREV_SHIFT
* _PyGC_SET_FINALIZED()
* _PyObject_GC_IS_TRACKED()
* _PyObject_GC_MAY_BE_TRACKED()
* _Py_AS_GC(o)
Keep the private _PyGC_FINALIZED() macro in the public C API for
backward compatibility with Python 3.8: make it an alias to the new
PyObject_GC_IsFinalized() function.
Move the SIZEOF_PYGC_HEAD constant from _testcapi module to
_testinternalcapi module.
Don't access PyInterpreterState.config member directly anymore, but
use new functions:
* _PyInterpreterState_GetConfig()
* _PyInterpreterState_SetConfig()
* _Py_GetConfig()
Add the functions PyObject_GC_IsTracked and PyObject_GC_IsFinalized to the public API to allow to query if Python objects are being currently tracked or have been already finalized by the garbage collector respectively.
Fix the signal handler: it now always uses the main interpreter,
rather than trying to get the current Python thread state.
The following function now accepts an interpreter, instead of a
Python thread state:
* _PyEval_SignalReceived()
* _Py_ThreadCanHandleSignals()
* _PyEval_AddPendingCall()
* COMPUTE_EVAL_BREAKER()
* SET_GIL_DROP_REQUEST(), RESET_GIL_DROP_REQUEST()
* SIGNAL_PENDING_CALLS(), UNSIGNAL_PENDING_CALLS()
* SIGNAL_PENDING_SIGNALS(), UNSIGNAL_PENDING_SIGNALS()
* SIGNAL_ASYNC_EXC(), UNSIGNAL_ASYNC_EXC()
Py_AddPendingCall() now uses the main interpreter if it fails to the
current Python thread state.
Convert _PyThreadState_GET() and PyInterpreterState_GET_UNSAFE()
macros to static inline functions.
The PyObject_NEW() macro becomes an alias to the PyObject_New()
macro, and the PyObject_NEW_VAR() macro becomes an alias to the
PyObject_NewVar() macro, to hide implementation details. They no
longer access directly the PyTypeObject.tp_basicsize member.
Exclude _PyObject_SIZE() and _PyObject_VAR_SIZE() macros from
the limited C API.
Replace PyObject_NEW() with PyObject_New() and replace
PyObject_NEW_VAR() with PyObject_NewVar().
Add a private _at_fork_reinit() method to _thread.Lock,
_thread.RLock, threading.RLock and threading.Condition classes:
reinitialize the lock after fork in the child process; reset the lock
to the unlocked state.
Rename also the private _reset_internal_locks() method of
threading.Event to _at_fork_reinit().
* Add _PyThread_at_fork_reinit() private function. It is excluded
from the limited C API.
* threading.Thread._reset_internal_locks() now calls
_at_fork_reinit() on self._tstate_lock rather than creating a new
Python lock object.
This implements things like `list[int]`,
which returns an object of type `types.GenericAlias`.
This object mostly acts as a proxy for `list`,
but has attributes `__origin__` and `__args__`
that allow recovering the parts (with values `list` and `(int,)`.
There is also an approximate notion of type variables;
e.g. `list[T]` has a `__parameters__` attribute equal to `(T,)`.
Type variables are objects of type `typing.TypeVar`.
Convert the PyObject_GET_WEAKREFS_LISTPTR() macro to a function to
hide implementation details: the macro accessed directly to the
PyTypeObject.tp_weaklistoffset member.
Add _PyObject_GET_WEAKREFS_LISTPTR() static inline function to the
internal C API.
Add os.waitstatus_to_exitcode() function to convert a wait status to an
exitcode.
Suggest waitstatus_to_exitcode() usage in the documentation when
appropriate.
Use waitstatus_to_exitcode() in:
* multiprocessing, os, subprocess and _bootsubprocess modules;
* test.support.wait_process();
* setup.py: run_command();
* and many tests.
Changes on 7dd549eb08 made _functools compatible with
PEP-489 and we could have multiple modules instances loaded.
But, right now there is no way to make `kwd_mark` global into
a per module instance variable. kwd_mark is used on lru_cache_new
which does not have a reference to a PyModule*, necessary to use
PyModule_GetState.
PEP-573 will solve this problem and will allow us to move the global
state to per-module data and properly clear the state when unloading
a module instance.
This change temporarily disable cleaning of kwd_mark to avoid NULL
pointer dereference if we clear kwd_mark and other module instances
still alive use it.
Replace statically allocated types with heap allocated types:
use PyType_FromSpec().
Add a module state to store the _abc_data_type.
Add traverse, clear and free functions to the module.
Replace statically allocated types with heap allocated types:
use PyType_FromSpec().
Add a module state to store the Scanner and Encoder types.
Add traverse, clear and free functions to the module.
_tracemalloc.c and _xxsubinterpretersmodule.c use
PyThreadState_GetFrame() and PyThreadState_GetInterpreter() to no
longer depend on the PyThreadState structure.
Update _asyncio, _bz2, _csv, _curses, _datetime,
_io, _operator, _pickle, _queue, blake2,
multibytecodec and overlapped C extension modules
to use PyModule_AddType().
Fix os.getgrouplist(): if getgrouplist() function fails because the
group list is too small, retry with a larger group list.
On failure, the glibc implementation of getgrouplist() sets ngroups
to the total number of groups. For other implementations, double the
group list size.
On macOS, getgrouplist() returns a non-zero value without setting
errno if the group list is too small. Double the list size and call
it again in this case.
COMPUTE_EVAL_BREAKER() now also checks if the Python thread state
belongs to the main interpreter. Don't break the evaluation loop if
there are pending signals but the Python thread state it belongs to a
subinterpeter.
* Add _Py_IsMainThread() function.
* Add _Py_ThreadCanHandleSignals() function.
If Py_AddPendingCall() is called in a subinterpreter, the function is
now scheduled to be called from the subinterpreter, rather than being
called from the main interpreter.
Each subinterpreter now has its own list of scheduled calls.
* Move pending and eval_breaker fields from _PyRuntimeState.ceval
to PyInterpreterState.ceval.
* new_interpreter() now calls _PyEval_InitThreads() to create
pending calls lock.
* Fix Py_AddPendingCall() for subinterpreters. It now calls
_PyThreadState_GET() which works in a subinterpreter if the
caller holds the GIL, and only falls back on
PyGILState_GetThisThreadState() if _PyThreadState_GET()
returns NULL.
Dong-hee Na
Victor Stinner
Lewis Gaul
Eric Snow
Pablo Galindo
Lysandros Nikolaou
Raymond Hettinger
Hai Shi
Serhiy Storchaka
Antoine Pitrou
Jeffrey Quesnelle
Ammar Askar
Ethan Smith
Benjamin Peterson
Oren Milman
Batuhan Taşkaya
Zackery Spytz
Guido van Rossum
Paulo Henrique Silva
Steve Dower
Andy Lester
Stefan Krah
Russell Owen
AlphaHot