mirror of https://github.com/python/cpython
PyMem_Malloc() now uses the fast pymalloc allocator
Issue #26249: PyMem_Malloc() allocator family now uses the pymalloc allocator rather than system malloc(). Applications calling PyMem_Malloc() without holding the GIL can now crash: use PYTHONMALLOC=debug environment variable to validate the usage of memory allocators in your application.
This commit is contained in:
parent
5439fc4901
commit
f5c4b99034
|
@ -165,15 +165,17 @@ The following function sets, modeled after the ANSI C standard, but specifying
|
||||||
behavior when requesting zero bytes, are available for allocating and releasing
|
behavior when requesting zero bytes, are available for allocating and releasing
|
||||||
memory from the Python heap.
|
memory from the Python heap.
|
||||||
|
|
||||||
The default memory block allocator uses the following functions:
|
By default, these functions use :ref:`pymalloc memory allocator <pymalloc>`.
|
||||||
:c:func:`malloc`, :c:func:`calloc`, :c:func:`realloc` and :c:func:`free`; call
|
|
||||||
``malloc(1)`` (or ``calloc(1, 1)``) when requesting zero bytes.
|
|
||||||
|
|
||||||
.. warning::
|
.. warning::
|
||||||
|
|
||||||
The :term:`GIL <global interpreter lock>` must be held when using these
|
The :term:`GIL <global interpreter lock>` must be held when using these
|
||||||
functions.
|
functions.
|
||||||
|
|
||||||
|
.. versionchanged:: 3.6
|
||||||
|
|
||||||
|
The default allocator is now pymalloc instead of system :c:func:`malloc`.
|
||||||
|
|
||||||
.. c:function:: void* PyMem_Malloc(size_t n)
|
.. c:function:: void* PyMem_Malloc(size_t n)
|
||||||
|
|
||||||
Allocates *n* bytes and returns a pointer of type :c:type:`void\*` to the
|
Allocates *n* bytes and returns a pointer of type :c:type:`void\*` to the
|
||||||
|
@ -295,15 +297,32 @@ Customize Memory Allocators
|
||||||
|
|
||||||
Enum used to identify an allocator domain. Domains:
|
Enum used to identify an allocator domain. Domains:
|
||||||
|
|
||||||
* :c:data:`PYMEM_DOMAIN_RAW`: functions :c:func:`PyMem_RawMalloc`,
|
.. c:var:: PYMEM_DOMAIN_RAW
|
||||||
:c:func:`PyMem_RawRealloc`, :c:func:`PyMem_RawCalloc` and
|
|
||||||
:c:func:`PyMem_RawFree`
|
|
||||||
* :c:data:`PYMEM_DOMAIN_MEM`: functions :c:func:`PyMem_Malloc`,
|
|
||||||
:c:func:`PyMem_Realloc`, :c:func:`PyMem_Calloc` and :c:func:`PyMem_Free`
|
|
||||||
* :c:data:`PYMEM_DOMAIN_OBJ`: functions :c:func:`PyObject_Malloc`,
|
|
||||||
:c:func:`PyObject_Realloc`, :c:func:`PyObject_Calloc` and
|
|
||||||
:c:func:`PyObject_Free`
|
|
||||||
|
|
||||||
|
Functions:
|
||||||
|
|
||||||
|
* :c:func:`PyMem_RawMalloc`
|
||||||
|
* :c:func:`PyMem_RawRealloc`
|
||||||
|
* :c:func:`PyMem_RawCalloc`
|
||||||
|
* :c:func:`PyMem_RawFree`
|
||||||
|
|
||||||
|
.. c:var:: PYMEM_DOMAIN_MEM
|
||||||
|
|
||||||
|
Functions:
|
||||||
|
|
||||||
|
* :c:func:`PyMem_Malloc`,
|
||||||
|
* :c:func:`PyMem_Realloc`
|
||||||
|
* :c:func:`PyMem_Calloc`
|
||||||
|
* :c:func:`PyMem_Free`
|
||||||
|
|
||||||
|
.. c:var:: PYMEM_DOMAIN_OBJ
|
||||||
|
|
||||||
|
Functions:
|
||||||
|
|
||||||
|
* :c:func:`PyObject_Malloc`
|
||||||
|
* :c:func:`PyObject_Realloc`
|
||||||
|
* :c:func:`PyObject_Calloc`
|
||||||
|
* :c:func:`PyObject_Free`
|
||||||
|
|
||||||
.. c:function:: void PyMem_GetAllocator(PyMemAllocatorDomain domain, PyMemAllocatorEx *allocator)
|
.. c:function:: void PyMem_GetAllocator(PyMemAllocatorDomain domain, PyMemAllocatorEx *allocator)
|
||||||
|
|
||||||
|
@ -328,18 +347,12 @@ Customize Memory Allocators
|
||||||
|
|
||||||
.. c:function:: void PyMem_SetupDebugHooks(void)
|
.. c:function:: void PyMem_SetupDebugHooks(void)
|
||||||
|
|
||||||
Setup hooks to detect bugs in the following Python memory allocator
|
Setup hooks to detect bugs in the Python memory allocator functions.
|
||||||
functions:
|
|
||||||
|
|
||||||
- :c:func:`PyMem_RawMalloc`, :c:func:`PyMem_RawRealloc`,
|
|
||||||
:c:func:`PyMem_RawCalloc`, :c:func:`PyMem_RawFree`
|
|
||||||
- :c:func:`PyMem_Malloc`, :c:func:`PyMem_Realloc`, :c:func:`PyMem_Calloc`,
|
|
||||||
:c:func:`PyMem_Free`
|
|
||||||
- :c:func:`PyObject_Malloc`, :c:func:`PyObject_Realloc`,
|
|
||||||
:c:func:`PyObject_Calloc`, :c:func:`PyObject_Free`
|
|
||||||
|
|
||||||
Newly allocated memory is filled with the byte ``0xCB``, freed memory is
|
Newly allocated memory is filled with the byte ``0xCB``, freed memory is
|
||||||
filled with the byte ``0xDB``. Additional checks:
|
filled with the byte ``0xDB``.
|
||||||
|
|
||||||
|
Runtime checks:
|
||||||
|
|
||||||
- Detect API violations, ex: :c:func:`PyObject_Free` called on a buffer
|
- Detect API violations, ex: :c:func:`PyObject_Free` called on a buffer
|
||||||
allocated by :c:func:`PyMem_Malloc`
|
allocated by :c:func:`PyMem_Malloc`
|
||||||
|
@ -377,8 +390,9 @@ to 512 bytes) with a short lifetime. It uses memory mappings called "arenas"
|
||||||
with a fixed size of 256 KB. It falls back to :c:func:`PyMem_RawMalloc` and
|
with a fixed size of 256 KB. It falls back to :c:func:`PyMem_RawMalloc` and
|
||||||
:c:func:`PyMem_RawRealloc` for allocations larger than 512 bytes.
|
:c:func:`PyMem_RawRealloc` for allocations larger than 512 bytes.
|
||||||
|
|
||||||
*pymalloc* is the default allocator of the :c:data:`PYMEM_DOMAIN_OBJ` domain
|
*pymalloc* is the default allocator of the :c:data:`PYMEM_DOMAIN_MEM` (ex:
|
||||||
(ex: :c:func:`PyObject_Malloc`).
|
:c:func:`PyObject_Malloc`) and :c:data:`PYMEM_DOMAIN_OBJ` (ex:
|
||||||
|
:c:func:`PyObject_Malloc`) domains.
|
||||||
|
|
||||||
The arena allocator uses the following functions:
|
The arena allocator uses the following functions:
|
||||||
|
|
||||||
|
|
|
@ -628,12 +628,11 @@ conflict.
|
||||||
Set the family of memory allocators used by Python:
|
Set the family of memory allocators used by Python:
|
||||||
|
|
||||||
* ``malloc``: use the :c:func:`malloc` function of the C library
|
* ``malloc``: use the :c:func:`malloc` function of the C library
|
||||||
for all Python memory allocators (ex: :c:func:`PyMem_RawMalloc`,
|
for all domains (:c:data:`PYMEM_DOMAIN_RAW`, :c:data:`PYMEM_DOMAIN_MEM`,
|
||||||
:c:func:`PyMem_Malloc` and :c:func:`PyObject_Malloc`).
|
:c:data:`PYMEM_DOMAIN_OBJ`).
|
||||||
* ``pymalloc``: :c:func:`PyObject_Malloc`, :c:func:`PyObject_Calloc` and
|
* ``pymalloc``: use the :ref:`pymalloc allocator <pymalloc>` for
|
||||||
:c:func:`PyObject_Realloc` use the :ref:`pymalloc allocator <pymalloc>`.
|
:c:data:`PYMEM_DOMAIN_MEM` and :c:data:`PYMEM_DOMAIN_OBJ` domains and use
|
||||||
Other Python memory allocators (ex: :c:func:`PyMem_RawMalloc` and
|
the :c:func:`malloc` function for the :c:data:`PYMEM_DOMAIN_RAW` domain.
|
||||||
:c:func:`PyMem_Malloc`) use :c:func:`malloc`.
|
|
||||||
|
|
||||||
Install debug hooks:
|
Install debug hooks:
|
||||||
|
|
||||||
|
|
|
@ -531,5 +531,11 @@ Changes in the Python API
|
||||||
Changes in the C API
|
Changes in the C API
|
||||||
--------------------
|
--------------------
|
||||||
|
|
||||||
|
* :c:func:`PyMem_Malloc` allocator family now uses the :ref:`pymalloc allocator
|
||||||
|
<pymalloc>` rather than system :c:func:`malloc`. Applications calling
|
||||||
|
:c:func:`PyMem_Malloc` without holding the GIL can now crash. Set the
|
||||||
|
:envvar:`PYTHONMALLOC` environment variable to ``debug`` to validate the
|
||||||
|
usage of memory allocators in your application. See :issue:`26249`.
|
||||||
|
|
||||||
* :c:func:`Py_Exit` (and the main interpreter) now override the exit status
|
* :c:func:`Py_Exit` (and the main interpreter) now override the exit status
|
||||||
with 120 if flushing buffered data failed. See :issue:`5319`.
|
with 120 if flushing buffered data failed. See :issue:`5319`.
|
||||||
|
|
|
@ -10,6 +10,13 @@ Release date: tba
|
||||||
Core and Builtins
|
Core and Builtins
|
||||||
-----------------
|
-----------------
|
||||||
|
|
||||||
|
- Issue #26249: Memory functions of the :c:func:`PyMem_Malloc` domain
|
||||||
|
(:c:data:`PYMEM_DOMAIN_MEM`) now use the :ref:`pymalloc allocator <pymalloc>`
|
||||||
|
rather than system :c:func:`malloc`. Applications calling
|
||||||
|
:c:func:`PyMem_Malloc` without holding the GIL can now crash: use
|
||||||
|
``PYTHONMALLOC=debug`` environment variable to validate the usage of memory
|
||||||
|
allocators in your application.
|
||||||
|
|
||||||
- Issue #26802: Optimize function calls only using unpacking like
|
- Issue #26802: Optimize function calls only using unpacking like
|
||||||
``func(*tuple)`` (no other positional argument, no keyword): avoid copying
|
``func(*tuple)`` (no other positional argument, no keyword): avoid copying
|
||||||
the tuple. Patch written by Joe Jevnik.
|
the tuple. Patch written by Joe Jevnik.
|
||||||
|
|
|
@ -198,9 +198,9 @@ static PyMemAllocatorEx _PyMem_Raw = {
|
||||||
|
|
||||||
static PyMemAllocatorEx _PyMem = {
|
static PyMemAllocatorEx _PyMem = {
|
||||||
#ifdef Py_DEBUG
|
#ifdef Py_DEBUG
|
||||||
&_PyMem_Debug.mem, PYDBG_FUNCS
|
&_PyMem_Debug.obj, PYDBG_FUNCS
|
||||||
#else
|
#else
|
||||||
NULL, PYMEM_FUNCS
|
NULL, PYOBJ_FUNCS
|
||||||
#endif
|
#endif
|
||||||
};
|
};
|
||||||
|
|
||||||
|
@ -256,7 +256,7 @@ _PyMem_SetupAllocators(const char *opt)
|
||||||
PyMemAllocatorEx obj_alloc = {NULL, PYOBJ_FUNCS};
|
PyMemAllocatorEx obj_alloc = {NULL, PYOBJ_FUNCS};
|
||||||
|
|
||||||
PyMem_SetAllocator(PYMEM_DOMAIN_RAW, &mem_alloc);
|
PyMem_SetAllocator(PYMEM_DOMAIN_RAW, &mem_alloc);
|
||||||
PyMem_SetAllocator(PYMEM_DOMAIN_MEM, &mem_alloc);
|
PyMem_SetAllocator(PYMEM_DOMAIN_MEM, &obj_alloc);
|
||||||
PyMem_SetAllocator(PYMEM_DOMAIN_OBJ, &obj_alloc);
|
PyMem_SetAllocator(PYMEM_DOMAIN_OBJ, &obj_alloc);
|
||||||
|
|
||||||
if (strcmp(opt, "pymalloc_debug") == 0)
|
if (strcmp(opt, "pymalloc_debug") == 0)
|
||||||
|
|
Loading…
Reference in New Issue