Issue #3329: Implement the PEP 445
Add new enum: * PyMemAllocatorDomain Add new structures: * PyMemAllocator * PyObjectArenaAllocator Add new functions: * PyMem_RawMalloc(), PyMem_RawRealloc(), PyMem_RawFree() * PyMem_GetAllocator(), PyMem_SetAllocator() * PyObject_GetArenaAllocator(), PyObject_SetArenaAllocator() * PyMem_SetupDebugHooks() Changes: * PyMem_Malloc()/PyObject_Realloc() now always call malloc()/realloc(), instead of calling PyObject_Malloc()/PyObject_Realloc() in debug mode. * PyObject_Malloc()/PyObject_Realloc() now falls back to PyMem_Malloc()/PyMem_Realloc() for allocations larger than 512 bytes. * Redesign debug checks on memory block allocators as hooks, instead of using C macros
This commit is contained in:
Victor Stinner
parent
df715ba54d
commit
0507bf56f0
|
|
@ -84,6 +84,48 @@ the C library allocator as shown in the previous example, the allocated memory
|
|||
for the I/O buffer escapes completely the Python memory manager.
|
||||
|
||||
|
||||
Raw Memory Interface
|
||||
====================
|
||||
|
||||
The following function sets are wrappers to the system allocator. These
|
||||
functions are thread-safe, the :term:`GIL <global interpreter lock>` does not
|
||||
need to be held.
|
||||
|
||||
The default raw memory block allocator uses the following functions:
|
||||
:c:func:`malloc`, :c:func:`realloc` and :c:func:`free`; call ``malloc(1)`` when
|
||||
requesting zero bytes.
|
||||
|
||||
.. versionadded:: 3.4
|
||||
|
||||
.. c:function:: void* PyMem_RawMalloc(size_t n)
|
||||
|
||||
Allocates *n* bytes and returns a pointer of type :c:type:`void\*` to the
|
||||
allocated memory, or *NULL* if the request fails. Requesting zero bytes
|
||||
returns a distinct non-*NULL* pointer if possible, as if
|
||||
``PyMem_RawMalloc(1)`` had been called instead. The memory will not have
|
||||
been initialized in any way.
|
||||
|
||||
|
||||
.. c:function:: void* PyMem_RawRealloc(void *p, size_t n)
|
||||
|
||||
Resizes the memory block pointed to by *p* to *n* bytes. The contents will
|
||||
be unchanged to the minimum of the old and the new sizes. If *p* is *NULL*,
|
||||
the call is equivalent to ``PyMem_RawMalloc(n)``; else if *n* is equal to
|
||||
zero, the memory block is resized but is not freed, and the returned pointer
|
||||
is non-*NULL*. Unless *p* is *NULL*, it must have been returned by a
|
||||
previous call to :c:func:`PyMem_RawMalloc` or :c:func:`PyMem_RawRealloc`. If
|
||||
the request fails, :c:func:`PyMem_RawRealloc` returns *NULL* and *p* remains
|
||||
a valid pointer to the previous memory area.
|
||||
|
||||
|
||||
.. c:function:: void PyMem_RawFree(void *p)
|
||||
|
||||
Frees the memory block pointed to by *p*, which must have been returned by a
|
||||
previous call to :c:func:`PyMem_RawMalloc` or :c:func:`PyMem_RawRealloc`.
|
||||
Otherwise, or if ``PyMem_Free(p)`` has been called before, undefined
|
||||
behavior occurs. If *p* is *NULL*, no operation is performed.
|
||||
|
||||
|
||||
.. _memoryinterface:
|
||||
|
||||
Memory Interface
|
||||
|
|
@ -91,8 +133,16 @@ Memory Interface
|
|||
|
||||
The following function sets, modeled after the ANSI C standard, but specifying
|
||||
behavior when requesting zero bytes, are available for allocating and releasing
|
||||
memory from the Python heap:
|
||||
memory from the Python heap.
|
||||
|
||||
The default memory block allocator uses the following functions:
|
||||
:c:func:`malloc`, :c:func:`realloc` and :c:func:`free`; call ``malloc(1)`` when
|
||||
requesting zero bytes.
|
||||
|
||||
.. warning::
|
||||
|
||||
The :term:`GIL <global interpreter lock>` must be held when using these
|
||||
functions.
|
||||
|
||||
.. c:function:: void* PyMem_Malloc(size_t n)
|
||||
|
||||
|
|
@ -155,6 +205,125 @@ versions and is therefore deprecated in extension modules.
|
|||
:c:func:`PyMem_NEW`, :c:func:`PyMem_RESIZE`, :c:func:`PyMem_DEL`.
|
||||
|
||||
|
||||
Customize Memory Allocators
|
||||
===========================
|
||||
|
||||
.. versionadded:: 3.4
|
||||
|
||||
.. c:type:: PyMemAllocator
|
||||
|
||||
Structure used to describe a memory block allocator. The structure has
|
||||
four fields:
|
||||
|
||||
+----------------------------------------------------------+---------------------------------------+
|
||||
| Field | Meaning |
|
||||
+==========================================================+=======================================+
|
||||
| ``void *ctx`` | user context passed as first argument |
|
||||
+----------------------------------------------------------+---------------------------------------+
|
||||
| ``void* malloc(void *ctx, size_t size)`` | allocate a memory block |
|
||||
+----------------------------------------------------------+---------------------------------------+
|
||||
| ``void* realloc(void *ctx, void *ptr, size_t new_size)`` | allocate or resize a memory block |
|
||||
+----------------------------------------------------------+---------------------------------------+
|
||||
| ``void free(void *ctx, void *ptr)`` | free a memory block |
|
||||
+----------------------------------------------------------+---------------------------------------+
|
||||
|
||||
.. c:type:: PyMemAllocatorDomain
|
||||
|
||||
Enum used to identify an allocator domain. Domains:
|
||||
|
||||
* :c:data:`PYMEM_DOMAIN_RAW`: functions :c:func:`PyMem_RawMalloc`,
|
||||
:c:func:`PyMem_RawRealloc` and :c:func:`PyMem_RawFree`
|
||||
* :c:data:`PYMEM_DOMAIN_MEM`: functions :c:func:`PyMem_Malloc`,
|
||||
:c:func:`PyMem_Realloc` and :c:func:`PyMem_Free`
|
||||
* :c:data:`PYMEM_DOMAIN_OBJ`: functions :c:func:`PyObject_Malloc`,
|
||||
:c:func:`PyObject_Realloc` and :c:func:`PyObject_Free`
|
||||
|
||||
|
||||
.. c:function:: void PyMem_GetAllocator(PyMemAllocatorDomain domain, PyMemAllocator *allocator)
|
||||
|
||||
Get the memory block allocator of the specified domain.
|
||||
|
||||
|
||||
.. c:function:: void PyMem_SetAllocator(PyMemAllocatorDomain domain, PyMemAllocator *allocator)
|
||||
|
||||
Set the memory block allocator of the specified domain.
|
||||
|
||||
The new allocator must return a distinct non-NULL pointer when requesting
|
||||
zero bytes.
|
||||
|
||||
For the :c:data:`PYMEM_DOMAIN_RAW` domain, the allocator must be
|
||||
thread-safe: the :term:`GIL <global interpreter lock>` is not held when the
|
||||
allocator is called.
|
||||
|
||||
If the new allocator is not a hook (does not call the previous allocator),
|
||||
the :c:func:`PyMem_SetupDebugHooks` function must be called to reinstall the
|
||||
debug hooks on top on the new allocator.
|
||||
|
||||
|
||||
.. c:function:: void PyMem_SetupDebugHooks(void)
|
||||
|
||||
Setup hooks to detect bugs in the following Python memory allocator
|
||||
functions:
|
||||
|
||||
- :c:func:`PyMem_RawMalloc`, :c:func:`PyMem_RawRealloc`,
|
||||
:c:func:`PyMem_RawFree`
|
||||
- :c:func:`PyMem_Malloc`, :c:func:`PyMem_Realloc`, :c:func:`PyMem_Free`
|
||||
- :c:func:`PyObject_Malloc`, :c:func:`PyObject_Realloc`,
|
||||
:c:func:`PyObject_Free`
|
||||
|
||||
Newly allocated memory is filled with the byte ``0xCB``, freed memory is
|
||||
filled with the byte ``0xDB``. Additionnal checks:
|
||||
|
||||
- detect API violations, ex: :c:func:`PyObject_Free` called on a buffer
|
||||
allocated by :c:func:`PyMem_Malloc`
|
||||
- detect write before the start of the buffer (buffer underflow)
|
||||
- detect write after the end of the buffer (buffer overflow)
|
||||
|
||||
The function does nothing if Python is not compiled is debug mode.
|
||||
|
||||
|
||||
Customize PyObject Arena Allocator
|
||||
==================================
|
||||
|
||||
Python has a *pymalloc* allocator for allocations smaller than 512 bytes. This
|
||||
allocator is optimized for small objects with a short lifetime. It uses memory
|
||||
mappings called "arenas" with a fixed size of 256 KB. It falls back to
|
||||
:c:func:`PyMem_Malloc` and :c:func:`PyMem_Realloc` for allocations larger than
|
||||
512 bytes. *pymalloc* is the default allocator used by
|
||||
:c:func:`PyObject_Malloc`.
|
||||
|
||||
The default arena allocator uses the following functions:
|
||||
|
||||
* :c:func:`VirtualAlloc` and :c:func:`VirtualFree` on Windows,
|
||||
* :c:func:`mmap` and :c:func:`munmap` if available,
|
||||
* :c:func:`malloc` and :c:func:`free` otherwise.
|
||||
|
||||
.. versionadded:: 3.4
|
||||
|
||||
.. c:type:: PyObjectArenaAllocator
|
||||
|
||||
Structure used to describe an arena allocator. The structure has
|
||||
three fields:
|
||||
|
||||
+--------------------------------------------------+---------------------------------------+
|
||||
| Field | Meaning |
|
||||
+==================================================+=======================================+
|
||||
| ``void *ctx`` | user context passed as first argument |
|
||||
+--------------------------------------------------+---------------------------------------+
|
||||
| ``void* alloc(void *ctx, size_t size)`` | allocate an arena of size bytes |
|
||||
+--------------------------------------------------+---------------------------------------+
|
||||
| ``void free(void *ctx, size_t size, void *ptr)`` | free an arena |
|
||||
+--------------------------------------------------+---------------------------------------+
|
||||
|
||||
.. c:function:: PyObject_GetArenaAllocator(PyObjectArenaAllocator *allocator)
|
||||
|
||||
Get the arena allocator.
|
||||
|
||||
.. c:function:: PyObject_SetArenaAllocator(PyObjectArenaAllocator *allocator)
|
||||
|
||||
Set the arena allocator.
|
||||
|
||||
|
||||
.. _memoryexamples:
|
||||
|
||||
Examples
|
||||
|
|
|
|||
|
|
@ -112,21 +112,11 @@ Security improvements:
|
|||
Please read on for a comprehensive list of user-facing changes.
|
||||
|
||||
|
||||
.. PEP-sized items next.
|
||||
|
||||
.. _pep-4XX:
|
||||
|
||||
.. PEP 4XX: Example PEP
|
||||
.. ====================
|
||||
|
||||
|
||||
.. (Implemented by Foo Bar.)
|
||||
|
||||
.. .. seealso::
|
||||
|
||||
:pep:`4XX` - Example PEP
|
||||
PEP written by Example Author
|
||||
PEP 445: Add new APIs to customize Python memory allocators
|
||||
===========================================================
|
||||
|
||||
The :pep:`445` adds new Application Programming Interfaces (API) to customize
|
||||
Python memory allocators.
|
||||
|
||||
|
||||
|
||||
|
|
|
|||
|
|
@ -94,9 +94,9 @@ PyObject_{New, NewVar, Del}.
|
|||
the object gets initialized via PyObject_{Init, InitVar} after obtaining
|
||||
the raw memory.
|
||||
*/
|
||||
PyAPI_FUNC(void *) PyObject_Malloc(size_t);
|
||||
PyAPI_FUNC(void *) PyObject_Realloc(void *, size_t);
|
||||
PyAPI_FUNC(void) PyObject_Free(void *);
|
||||
PyAPI_FUNC(void *) PyObject_Malloc(size_t size);
|
||||
PyAPI_FUNC(void *) PyObject_Realloc(void *ptr, size_t new_size);
|
||||
PyAPI_FUNC(void) PyObject_Free(void *ptr);
|
||||
|
||||
/* This function returns the number of allocated memory blocks, regardless of size */
|
||||
PyAPI_FUNC(Py_ssize_t) _Py_GetAllocatedBlocks(void);
|
||||
|
|
@ -106,41 +106,15 @@ PyAPI_FUNC(Py_ssize_t) _Py_GetAllocatedBlocks(void);
|
|||
#ifndef Py_LIMITED_API
|
||||
PyAPI_FUNC(void) _PyObject_DebugMallocStats(FILE *out);
|
||||
#endif /* #ifndef Py_LIMITED_API */
|
||||
#ifdef PYMALLOC_DEBUG /* WITH_PYMALLOC && PYMALLOC_DEBUG */
|
||||
PyAPI_FUNC(void *) _PyObject_DebugMalloc(size_t nbytes);
|
||||
PyAPI_FUNC(void *) _PyObject_DebugRealloc(void *p, size_t nbytes);
|
||||
PyAPI_FUNC(void) _PyObject_DebugFree(void *p);
|
||||
PyAPI_FUNC(void) _PyObject_DebugDumpAddress(const void *p);
|
||||
PyAPI_FUNC(void) _PyObject_DebugCheckAddress(const void *p);
|
||||
PyAPI_FUNC(void *) _PyObject_DebugMallocApi(char api, size_t nbytes);
|
||||
PyAPI_FUNC(void *) _PyObject_DebugReallocApi(char api, void *p, size_t nbytes);
|
||||
PyAPI_FUNC(void) _PyObject_DebugFreeApi(char api, void *p);
|
||||
PyAPI_FUNC(void) _PyObject_DebugCheckAddressApi(char api, const void *p);
|
||||
PyAPI_FUNC(void *) _PyMem_DebugMalloc(size_t nbytes);
|
||||
PyAPI_FUNC(void *) _PyMem_DebugRealloc(void *p, size_t nbytes);
|
||||
PyAPI_FUNC(void) _PyMem_DebugFree(void *p);
|
||||
#define PyObject_MALLOC _PyObject_DebugMalloc
|
||||
#define PyObject_Malloc _PyObject_DebugMalloc
|
||||
#define PyObject_REALLOC _PyObject_DebugRealloc
|
||||
#define PyObject_Realloc _PyObject_DebugRealloc
|
||||
#define PyObject_FREE _PyObject_DebugFree
|
||||
#define PyObject_Free _PyObject_DebugFree
|
||||
#endif
|
||||
|
||||
#else /* WITH_PYMALLOC && ! PYMALLOC_DEBUG */
|
||||
/* Macros */
|
||||
#define PyObject_MALLOC PyObject_Malloc
|
||||
#define PyObject_REALLOC PyObject_Realloc
|
||||
#define PyObject_FREE PyObject_Free
|
||||
#endif
|
||||
|
||||
#else /* ! WITH_PYMALLOC */
|
||||
#define PyObject_MALLOC PyMem_MALLOC
|
||||
#define PyObject_REALLOC PyMem_REALLOC
|
||||
#define PyObject_FREE PyMem_FREE
|
||||
|
||||
#endif /* WITH_PYMALLOC */
|
||||
|
||||
#define PyObject_Del PyObject_Free
|
||||
#define PyObject_DEL PyObject_FREE
|
||||
#define PyObject_DEL PyObject_Free
|
||||
|
||||
|
||||
/*
|
||||
* Generic object allocator interface
|
||||
|
|
@ -224,6 +198,26 @@ PyAPI_FUNC(PyVarObject *) _PyObject_NewVar(PyTypeObject *, Py_ssize_t);
|
|||
constructor you would start directly with PyObject_Init/InitVar
|
||||
*/
|
||||
|
||||
#ifndef Py_LIMITED_API
|
||||
typedef struct {
|
||||
/* user context passed as the first argument to the 2 functions */
|
||||
void *ctx;
|
||||
|
||||
/* allocate an arena of size bytes */
|
||||
void* (*alloc) (void *ctx, size_t size);
|
||||
|
||||
/* free an arena */
|
||||
void (*free) (void *ctx, void *ptr, size_t size);
|
||||
} PyObjectArenaAllocator;
|
||||
|
||||
/* Get the arena allocator. */
|
||||
PyAPI_FUNC(void) PyObject_GetArenaAllocator(PyObjectArenaAllocator *allocator);
|
||||
|
||||
/* Set the arena allocator. */
|
||||
PyAPI_FUNC(void) PyObject_SetArenaAllocator(PyObjectArenaAllocator *allocator);
|
||||
#endif
|
||||
|
||||
|
||||
/*
|
||||
* Garbage Collection Support
|
||||
* ==========================
|
||||
|
|
|
|||
|
|
@ -11,6 +11,11 @@
|
|||
extern "C" {
|
||||
#endif
|
||||
|
||||
PyAPI_FUNC(void *) PyMem_RawMalloc(size_t size);
|
||||
PyAPI_FUNC(void *) PyMem_RawRealloc(void *ptr, size_t new_size);
|
||||
PyAPI_FUNC(void) PyMem_RawFree(void *ptr);
|
||||
|
||||
|
||||
/* BEWARE:
|
||||
|
||||
Each interface exports both functions and macros. Extension modules should
|
||||
|
|
@ -49,21 +54,11 @@ extern "C" {
|
|||
performed on failure (no exception is set, no warning is printed, etc).
|
||||
*/
|
||||
|
||||
PyAPI_FUNC(void *) PyMem_Malloc(size_t);
|
||||
PyAPI_FUNC(void *) PyMem_Realloc(void *, size_t);
|
||||
PyAPI_FUNC(void) PyMem_Free(void *);
|
||||
|
||||
/* Starting from Python 1.6, the wrappers Py_{Malloc,Realloc,Free} are
|
||||
no longer supported. They used to call PyErr_NoMemory() on failure. */
|
||||
PyAPI_FUNC(void *) PyMem_Malloc(size_t size);
|
||||
PyAPI_FUNC(void *) PyMem_Realloc(void *ptr, size_t new_size);
|
||||
PyAPI_FUNC(void) PyMem_Free(void *ptr);
|
||||
|
||||
/* Macros. */
|
||||
#ifdef PYMALLOC_DEBUG
|
||||
/* Redirect all memory operations to Python's debugging allocator. */
|
||||
#define PyMem_MALLOC _PyMem_DebugMalloc
|
||||
#define PyMem_REALLOC _PyMem_DebugRealloc
|
||||
#define PyMem_FREE _PyMem_DebugFree
|
||||
|
||||
#else /* ! PYMALLOC_DEBUG */
|
||||
|
||||
/* PyMem_MALLOC(0) means malloc(1). Some systems would return NULL
|
||||
for malloc(0), which would be treated as an error. Some platforms
|
||||
|
|
@ -71,13 +66,9 @@ PyAPI_FUNC(void) PyMem_Free(void *);
|
|||
pymalloc. To solve these problems, allocate an extra byte. */
|
||||
/* Returns NULL to indicate error if a negative size or size larger than
|
||||
Py_ssize_t can represent is supplied. Helps prevents security holes. */
|
||||
#define PyMem_MALLOC(n) ((size_t)(n) > (size_t)PY_SSIZE_T_MAX ? NULL \
|
||||
: malloc((n) ? (n) : 1))
|
||||
#define PyMem_REALLOC(p, n) ((size_t)(n) > (size_t)PY_SSIZE_T_MAX ? NULL \
|
||||
: realloc((p), (n) ? (n) : 1))
|
||||
#define PyMem_FREE free
|
||||
|
||||
#endif /* PYMALLOC_DEBUG */
|
||||
#define PyMem_MALLOC(n) PyMem_Malloc(n)
|
||||
#define PyMem_REALLOC(p, n) PyMem_Realloc(p, n)
|
||||
#define PyMem_FREE(p) PyMem_Free(p)
|
||||
|
||||
/*
|
||||
* Type-oriented memory interface
|
||||
|
|
@ -115,6 +106,69 @@ PyAPI_FUNC(void) PyMem_Free(void *);
|
|||
#define PyMem_Del PyMem_Free
|
||||
#define PyMem_DEL PyMem_FREE
|
||||
|
||||
#ifndef Py_LIMITED_API
|
||||
typedef enum {
|
||||
/* PyMem_RawMalloc(), PyMem_RawRealloc() and PyMem_RawFree() */
|
||||
PYMEM_DOMAIN_RAW,
|
||||
|
||||
/* PyMem_Malloc(), PyMem_Realloc() and PyMem_Free() */
|
||||
PYMEM_DOMAIN_MEM,
|
||||
|
||||
/* PyObject_Malloc(), PyObject_Realloc() and PyObject_Free() */
|
||||
PYMEM_DOMAIN_OBJ
|
||||
} PyMemAllocatorDomain;
|
||||
|
||||
typedef struct {
|
||||
/* user context passed as the first argument to the 3 functions */
|
||||
void *ctx;
|
||||
|
||||
/* allocate a memory block */
|
||||
void* (*malloc) (void *ctx, size_t size);
|
||||
|
||||
/* allocate or resize a memory block */
|
||||
void* (*realloc) (void *ctx, void *ptr, size_t new_size);
|
||||
|
||||
/* release a memory block */
|
||||
void (*free) (void *ctx, void *ptr);
|
||||
} PyMemAllocator;
|
||||
|
||||
/* Get the memory block allocator of the specified domain. */
|
||||
PyAPI_FUNC(void) PyMem_GetAllocator(PyMemAllocatorDomain domain,
|
||||
PyMemAllocator *allocator);
|
||||
|
||||
/* Set the memory block allocator of the specified domain.
|
||||
|
||||
The new allocator must return a distinct non-NULL pointer when requesting
|
||||
zero bytes.
|
||||
|
||||
For the PYMEM_DOMAIN_RAW domain, the allocator must be thread-safe: the GIL
|
||||
is not held when the allocator is called.
|
||||
|
||||
If the new allocator is not a hook (don't call the previous allocator), the
|
||||
PyMem_SetupDebugHooks() function must be called to reinstall the debug hooks
|
||||
on top on the new allocator. */
|
||||
PyAPI_FUNC(void) PyMem_SetAllocator(PyMemAllocatorDomain domain,
|
||||
PyMemAllocator *allocator);
|
||||
|
||||
/* Setup hooks to detect bugs in the following Python memory allocator
|
||||
functions:
|
||||
|
||||
- PyMem_RawMalloc(), PyMem_RawRealloc(), PyMem_RawFree()
|
||||
- PyMem_Malloc(), PyMem_Realloc(), PyMem_Free()
|
||||
- PyObject_Malloc(), PyObject_Realloc() and PyObject_Free()
|
||||
|
||||
Newly allocated memory is filled with the byte 0xCB, freed memory is filled
|
||||
with the byte 0xDB. Additionnal checks:
|
||||
|
||||
- detect API violations, ex: PyObject_Free() called on a buffer allocated
|
||||
by PyMem_Malloc()
|
||||
- detect write before the start of the buffer (buffer underflow)
|
||||
- detect write after the end of the buffer (buffer overflow)
|
||||
|
||||
The function does nothing if Python is not compiled is debug mode. */
|
||||
PyAPI_FUNC(void) PyMem_SetupDebugHooks(void);
|
||||
#endif
|
||||
|
||||
#ifdef __cplusplus
|
||||
}
|
||||
#endif
|
||||
|
|
|
|||
|
|
@ -10,6 +10,9 @@ What's New in Python 3.4.0 Alpha 1?
|
|||
Core and Builtins
|
||||
-----------------
|
||||
|
||||
- Issue #3329: Implement the PEP 445: Add new APIs to customize Python memory
|
||||
allocators.
|
||||
|
||||
- Issue #18328: Reorder ops in PyThreadState_Delete*() functions. Now the
|
||||
tstate is first removed from TLS and then deallocated.
|
||||
|
||||
|
|
|
|||
|
|
@ -2511,6 +2511,161 @@ test_decref_doesnt_leak(PyObject *ob)
|
|||
Py_RETURN_NONE;
|
||||
}
|
||||
|
||||
static PyObject *
|
||||
test_pymem_alloc0(PyObject *self)
|
||||
{
|
||||
void *ptr;
|
||||
|
||||
ptr = PyMem_Malloc(0);
|
||||
if (ptr == NULL) {
|
||||
PyErr_SetString(PyExc_RuntimeError, "PyMem_Malloc(0) returns NULL");
|
||||
return NULL;
|
||||
}
|
||||
PyMem_Free(ptr);
|
||||
|
||||
ptr = PyObject_Malloc(0);
|
||||
if (ptr == NULL) {
|
||||
PyErr_SetString(PyExc_RuntimeError, "PyObject_Malloc(0) returns NULL");
|
||||
return NULL;
|
||||
}
|
||||
PyObject_Free(ptr);
|
||||
|
||||
Py_RETURN_NONE;
|
||||
}
|
||||
|
||||
typedef struct {
|
||||
PyMemAllocator alloc;
|
||||
|
||||
size_t malloc_size;
|
||||
void *realloc_ptr;
|
||||
size_t realloc_new_size;
|
||||
void *free_ptr;
|
||||
} alloc_hook_t;
|
||||
|
||||
static void* hook_malloc (void* ctx, size_t size)
|
||||
{
|
||||
alloc_hook_t *hook = (alloc_hook_t *)ctx;
|
||||
hook->malloc_size = size;
|
||||
return hook->alloc.malloc(hook->alloc.ctx, size);
|
||||
}
|
||||
|
||||
static void* hook_realloc (void* ctx, void* ptr, size_t new_size)
|
||||
{
|
||||
alloc_hook_t *hook = (alloc_hook_t *)ctx;
|
||||
hook->realloc_ptr = ptr;
|
||||
hook->realloc_new_size = new_size;
|
||||
return hook->alloc.realloc(hook->alloc.ctx, ptr, new_size);
|
||||
}
|
||||
|
||||
static void hook_free (void *ctx, void *ptr)
|
||||
{
|
||||
alloc_hook_t *hook = (alloc_hook_t *)ctx;
|
||||
hook->free_ptr = ptr;
|
||||
hook->alloc.free(hook->alloc.ctx, ptr);
|
||||
}
|
||||
|
||||
static PyObject *
|
||||
test_setallocators(PyMemAllocatorDomain domain)
|
||||
{
|
||||
PyObject *res = NULL;
|
||||
const char *error_msg;
|
||||
alloc_hook_t hook;
|
||||
PyMemAllocator alloc;
|
||||
size_t size, size2;
|
||||
void *ptr, *ptr2;
|
||||
|
||||
hook.malloc_size = 0;
|
||||
hook.realloc_ptr = NULL;
|
||||
hook.realloc_new_size = 0;
|
||||
hook.free_ptr = NULL;
|
||||
|
||||
alloc.ctx = &hook;
|
||||
alloc.malloc = &hook_malloc;
|
||||
alloc.realloc = &hook_realloc;
|
||||
alloc.free = &hook_free;
|
||||
PyMem_GetAllocator(domain, &hook.alloc);
|
||||
PyMem_SetAllocator(domain, &alloc);
|
||||
|
||||
size = 42;
|
||||
switch(domain)
|
||||
{
|
||||
case PYMEM_DOMAIN_RAW: ptr = PyMem_RawMalloc(size); break;
|
||||
case PYMEM_DOMAIN_MEM: ptr = PyMem_Malloc(size); break;
|
||||
case PYMEM_DOMAIN_OBJ: ptr = PyObject_Malloc(size); break;
|
||||
default: ptr = NULL; break;
|
||||
}
|
||||
|
||||
if (ptr == NULL) {
|
||||
error_msg = "malloc failed";
|
||||
goto fail;
|
||||
}
|
||||
|
||||
if (hook.malloc_size != size) {
|
||||
error_msg = "malloc invalid size";
|
||||
goto fail;
|
||||
}
|
||||
|
||||
size2 = 200;
|
||||
switch(domain)
|
||||
{
|
||||
case PYMEM_DOMAIN_RAW: ptr2 = PyMem_RawRealloc(ptr, size2); break;
|
||||
case PYMEM_DOMAIN_MEM: ptr2 = PyMem_Realloc(ptr, size2); break;
|
||||
case PYMEM_DOMAIN_OBJ: ptr2 = PyObject_Realloc(ptr, size2); break;
|
||||
}
|
||||
|
||||
if (ptr2 == NULL) {
|
||||
error_msg = "realloc failed";
|
||||
goto fail;
|
||||
}
|
||||
|
||||
if (hook.realloc_ptr != ptr
|
||||
|| hook.realloc_new_size != size2) {
|
||||
error_msg = "realloc invalid parameters";
|
||||
goto fail;
|
||||
}
|
||||
|
||||
switch(domain)
|
||||
{
|
||||
case PYMEM_DOMAIN_RAW: PyMem_RawFree(ptr2); break;
|
||||
case PYMEM_DOMAIN_MEM: PyMem_Free(ptr2); break;
|
||||
case PYMEM_DOMAIN_OBJ: PyObject_Free(ptr2); break;
|
||||
}
|
||||
|
||||
if (hook.free_ptr != ptr2) {
|
||||
error_msg = "free invalid pointer";
|
||||
goto fail;
|
||||
}
|
||||
|
||||
Py_INCREF(Py_None);
|
||||
res = Py_None;
|
||||
goto finally;
|
||||
|
||||
fail:
|
||||
PyErr_SetString(PyExc_RuntimeError, error_msg);
|
||||
|
||||
finally:
|
||||
PyMem_SetAllocator(domain, &hook.alloc);
|
||||
return res;
|
||||
}
|
||||
|
||||
static PyObject *
|
||||
test_pymem_setrawallocators(PyObject *self)
|
||||
{
|
||||
return test_setallocators(PYMEM_DOMAIN_RAW);
|
||||
}
|
||||
|
||||
static PyObject *
|
||||
test_pymem_setallocators(PyObject *self)
|
||||
{
|
||||
return test_setallocators(PYMEM_DOMAIN_MEM);
|
||||
}
|
||||
|
||||
static PyObject *
|
||||
test_pyobject_setallocators(PyObject *self)
|
||||
{
|
||||
return test_setallocators(PYMEM_DOMAIN_OBJ);
|
||||
}
|
||||
|
||||
static PyMethodDef TestMethods[] = {
|
||||
{"raise_exception", raise_exception, METH_VARARGS},
|
||||
{"raise_memoryerror", (PyCFunction)raise_memoryerror, METH_NOARGS},
|
||||
|
|
@ -2611,6 +2766,14 @@ static PyMethodDef TestMethods[] = {
|
|||
{"pytime_object_to_time_t", test_pytime_object_to_time_t, METH_VARARGS},
|
||||
{"pytime_object_to_timeval", test_pytime_object_to_timeval, METH_VARARGS},
|
||||
{"pytime_object_to_timespec", test_pytime_object_to_timespec, METH_VARARGS},
|
||||
{"test_pymem",
|
||||
(PyCFunction)test_pymem_alloc0, METH_NOARGS},
|
||||
{"test_pymem_alloc0",
|
||||
(PyCFunction)test_pymem_setrawallocators, METH_NOARGS},
|
||||
{"test_pymem_setallocators",
|
||||
(PyCFunction)test_pymem_setallocators, METH_NOARGS},
|
||||
{"test_pyobject_setallocators",
|
||||
(PyCFunction)test_pyobject_setallocators, METH_NOARGS},
|
||||
{NULL, NULL} /* sentinel */
|
||||
};
|
||||
|
||||
|
|
|
|||
|
|
@ -1859,26 +1859,6 @@ PyTypeObject *_PyCapsule_hack = &PyCapsule_Type;
|
|||
Py_ssize_t (*_Py_abstract_hack)(PyObject *) = PyObject_Size;
|
||||
|
||||
|
||||
/* Python's malloc wrappers (see pymem.h) */
|
||||
|
||||
void *
|
||||
PyMem_Malloc(size_t nbytes)
|
||||
{
|
||||
return PyMem_MALLOC(nbytes);
|
||||
}
|
||||
|
||||
void *
|
||||
PyMem_Realloc(void *p, size_t nbytes)
|
||||
{
|
||||
return PyMem_REALLOC(p, nbytes);
|
||||
}
|
||||
|
||||
void
|
||||
PyMem_Free(void *p)
|
||||
{
|
||||
PyMem_FREE(p);
|
||||
}
|
||||
|
||||
void
|
||||
_PyObject_DebugTypeStats(FILE *out)
|
||||
{
|
||||
|
|
|
|||
|
|
@ -1,18 +1,326 @@
|
|||
#include "Python.h"
|
||||
|
||||
/* Python's malloc wrappers (see pymem.h) */
|
||||
|
||||
#ifdef PYMALLOC_DEBUG /* WITH_PYMALLOC && PYMALLOC_DEBUG */
|
||||
/* Forward declaration */
|
||||
static void* _PyMem_DebugMalloc(void *ctx, size_t size);
|
||||
static void _PyMem_DebugFree(void *ctx, void *p);
|
||||
static void* _PyMem_DebugRealloc(void *ctx, void *ptr, size_t size);
|
||||
|
||||
static void _PyObject_DebugDumpAddress(const void *p);
|
||||
static void _PyMem_DebugCheckAddress(char api_id, const void *p);
|
||||
#endif
|
||||
|
||||
#ifdef WITH_PYMALLOC
|
||||
|
||||
#ifdef HAVE_MMAP
|
||||
#include <sys/mman.h>
|
||||
#ifdef MAP_ANONYMOUS
|
||||
#define ARENAS_USE_MMAP
|
||||
#endif
|
||||
#ifdef MS_WINDOWS
|
||||
# include <windows.h>
|
||||
#elif defined(HAVE_MMAP)
|
||||
# include <sys/mman.h>
|
||||
# ifdef MAP_ANONYMOUS
|
||||
# define ARENAS_USE_MMAP
|
||||
# endif
|
||||
#endif
|
||||
|
||||
#ifdef MS_WINDOWS
|
||||
#include <windows.h>
|
||||
/* Forward declaration */
|
||||
static void* _PyObject_Malloc(void *ctx, size_t size);
|
||||
static void _PyObject_Free(void *ctx, void *p);
|
||||
static void* _PyObject_Realloc(void *ctx, void *ptr, size_t size);
|
||||
#endif
|
||||
|
||||
|
||||
static void *
|
||||
_PyMem_RawMalloc(void *ctx, size_t size)
|
||||
{
|
||||
/* PyMem_Malloc(0) means malloc(1). Some systems would return NULL
|
||||
for malloc(0), which would be treated as an error. Some platforms would
|
||||
return a pointer with no memory behind it, which would break pymalloc.
|
||||
To solve these problems, allocate an extra byte. */
|
||||
if (size == 0)
|
||||
size = 1;
|
||||
return malloc(size);
|
||||
}
|
||||
|
||||
static void *
|
||||
_PyMem_RawRealloc(void *ctx, void *ptr, size_t size)
|
||||
{
|
||||
if (size == 0)
|
||||
size = 1;
|
||||
return realloc(ptr, size);
|
||||
}
|
||||
|
||||
static void
|
||||
_PyMem_RawFree(void *ctx, void *ptr)
|
||||
{
|
||||
free(ptr);
|
||||
}
|
||||
|
||||
|
||||
#ifdef MS_WINDOWS
|
||||
static void *
|
||||
_PyObject_ArenaVirtualAlloc(void *ctx, size_t size)
|
||||
{
|
||||
return VirtualAlloc(NULL, size,
|
||||
MEM_COMMIT | MEM_RESERVE, PAGE_READWRITE);
|
||||
}
|
||||
|
||||
static void
|
||||
_PyObject_ArenaVirtualFree(void *ctx, void *ptr, size_t size)
|
||||
{
|
||||
VirtualFree(ptr, size, MEM_RELEASE);
|
||||
}
|
||||
|
||||
#elif defined(ARENAS_USE_MMAP)
|
||||
static void *
|
||||
_PyObject_ArenaMmap(void *ctx, size_t size)
|
||||
{
|
||||
void *ptr;
|
||||
ptr = mmap(NULL, size, PROT_READ|PROT_WRITE,
|
||||
MAP_PRIVATE|MAP_ANONYMOUS, -1, 0);
|
||||
if (ptr == MAP_FAILED)
|
||||
return NULL;
|
||||
assert(ptr != NULL);
|
||||
return ptr;
|
||||
}
|
||||
|
||||
static void
|
||||
_PyObject_ArenaMunmap(void *ctx, void *ptr, size_t size)
|
||||
{
|
||||
munmap(ptr, size);
|
||||
}
|
||||
|
||||
#else
|
||||
static void *
|
||||
_PyObject_ArenaMalloc(void *ctx, size_t size)
|
||||
{
|
||||
return malloc(size);
|
||||
}
|
||||
|
||||
static void
|
||||
_PyObject_ArenaFree(void *ctx, void *ptr, size_t size)
|
||||
{
|
||||
free(ptr);
|
||||
}
|
||||
#endif
|
||||
|
||||
|
||||
#define PYRAW_FUNCS _PyMem_RawMalloc, _PyMem_RawRealloc, _PyMem_RawFree
|
||||
#ifdef WITH_PYMALLOC
|
||||
#define PYOBJECT_FUNCS _PyObject_Malloc, _PyObject_Realloc, _PyObject_Free
|
||||
#else
|
||||
#define PYOBJECT_FUNCS PYRAW_FUNCS
|
||||
#endif
|
||||
|
||||
#ifdef PYMALLOC_DEBUG
|
||||
typedef struct {
|
||||
/* We tag each block with an API ID in order to tag API violations */
|
||||
char api_id;
|
||||
PyMemAllocator alloc;
|
||||
} debug_alloc_api_t;
|
||||
static struct {
|
||||
debug_alloc_api_t raw;
|
||||
debug_alloc_api_t mem;
|
||||
debug_alloc_api_t obj;
|
||||
} _PyMem_Debug = {
|
||||
{'r', {NULL, PYRAW_FUNCS}},
|
||||
{'m', {NULL, PYRAW_FUNCS}},
|
||||
{'o', {NULL, PYOBJECT_FUNCS}}
|
||||
};
|
||||
|
||||
#define PYDEBUG_FUNCS _PyMem_DebugMalloc, _PyMem_DebugRealloc, _PyMem_DebugFree
|
||||
#endif
|
||||
|
||||
static PyMemAllocator _PyMem_Raw = {
|
||||
#ifdef PYMALLOC_DEBUG
|
||||
&_PyMem_Debug.raw, PYDEBUG_FUNCS
|
||||
#else
|
||||
NULL, PYRAW_FUNCS
|
||||
#endif
|
||||
};
|
||||
|
||||
static PyMemAllocator _PyMem = {
|
||||
#ifdef PYMALLOC_DEBUG
|
||||
&_PyMem_Debug.mem, PYDEBUG_FUNCS
|
||||
#else
|
||||
NULL, PYRAW_FUNCS
|
||||
#endif
|
||||
};
|
||||
|
||||
static PyMemAllocator _PyObject = {
|
||||
#ifdef PYMALLOC_DEBUG
|
||||
&_PyMem_Debug.obj, PYDEBUG_FUNCS
|
||||
#else
|
||||
NULL, PYOBJECT_FUNCS
|
||||
#endif
|
||||
};
|
||||
|
||||
#undef PYRAW_FUNCS
|
||||
#undef PYOBJECT_FUNCS
|
||||
#undef PYDEBUG_FUNCS
|
||||
|
||||
static PyObjectArenaAllocator _PyObject_Arena = {NULL,
|
||||
#ifdef MS_WINDOWS
|
||||
_PyObject_ArenaVirtualAlloc, _PyObject_ArenaVirtualFree
|
||||
#elif defined(ARENAS_USE_MMAP)
|
||||
_PyObject_ArenaMmap, _PyObject_ArenaMunmap
|
||||
#else
|
||||
_PyObject_ArenaMalloc, _PyObject_ArenaFree
|
||||
#endif
|
||||
};
|
||||
|
||||
void
|
||||
PyMem_SetupDebugHooks(void)
|
||||
{
|
||||
#ifdef PYMALLOC_DEBUG
|
||||
PyMemAllocator alloc;
|
||||
|
||||
alloc.malloc = _PyMem_DebugMalloc;
|
||||
alloc.realloc = _PyMem_DebugRealloc;
|
||||
alloc.free = _PyMem_DebugFree;
|
||||
|
||||
if (_PyMem_Raw.malloc != _PyMem_DebugMalloc) {
|
||||
alloc.ctx = &_PyMem_Debug.raw;
|
||||
PyMem_GetAllocator(PYMEM_DOMAIN_RAW, &_PyMem_Debug.raw.alloc);
|
||||
PyMem_SetAllocator(PYMEM_DOMAIN_RAW, &alloc);
|
||||
}
|
||||
|
||||
if (_PyMem.malloc != _PyMem_DebugMalloc) {
|
||||
alloc.ctx = &_PyMem_Debug.mem;
|
||||
PyMem_GetAllocator(PYMEM_DOMAIN_MEM, &_PyMem_Debug.mem.alloc);
|
||||
PyMem_SetAllocator(PYMEM_DOMAIN_MEM, &alloc);
|
||||
}
|
||||
|
||||
if (_PyObject.malloc != _PyMem_DebugMalloc) {
|
||||
alloc.ctx = &_PyMem_Debug.obj;
|
||||
PyMem_GetAllocator(PYMEM_DOMAIN_OBJ, &_PyMem_Debug.obj.alloc);
|
||||
PyMem_SetAllocator(PYMEM_DOMAIN_OBJ, &alloc);
|
||||
}
|
||||
#endif
|
||||
}
|
||||
|
||||
void
|
||||
PyMem_GetAllocator(PyMemAllocatorDomain domain, PyMemAllocator *allocator)
|
||||
{
|
||||
switch(domain)
|
||||
{
|
||||
case PYMEM_DOMAIN_RAW: *allocator = _PyMem_Raw; break;
|
||||
case PYMEM_DOMAIN_MEM: *allocator = _PyMem; break;
|
||||
case PYMEM_DOMAIN_OBJ: *allocator = _PyObject; break;
|
||||
default:
|
||||
/* unknown domain */
|
||||
allocator->ctx = NULL;
|
||||
allocator->malloc = NULL;
|
||||
allocator->realloc = NULL;
|
||||
allocator->free = NULL;
|
||||
}
|
||||
}
|
||||
|
||||
void
|
||||
PyMem_SetAllocator(PyMemAllocatorDomain domain, PyMemAllocator *allocator)
|
||||
{
|
||||
switch(domain)
|
||||
{
|
||||
case PYMEM_DOMAIN_RAW: _PyMem_Raw = *allocator; break;
|
||||
case PYMEM_DOMAIN_MEM: _PyMem = *allocator; break;
|
||||
case PYMEM_DOMAIN_OBJ: _PyObject = *allocator; break;
|
||||
/* ignore unknown domain */
|
||||
}
|
||||
|
||||
}
|
||||
|
||||
void
|
||||
PyObject_GetArenaAllocator(PyObjectArenaAllocator *allocator)
|
||||
{
|
||||
*allocator = _PyObject_Arena;
|
||||
}
|
||||
|
||||
void
|
||||
PyObject_SetArenaAllocator(PyObjectArenaAllocator *allocator)
|
||||
{
|
||||
_PyObject_Arena = *allocator;
|
||||
}
|
||||
|
||||
void *
|
||||
PyMem_RawMalloc(size_t size)
|
||||
{
|
||||
/*
|
||||
* Limit ourselves to PY_SSIZE_T_MAX bytes to prevent security holes.
|
||||
* Most python internals blindly use a signed Py_ssize_t to track
|
||||
* things without checking for overflows or negatives.
|
||||
* As size_t is unsigned, checking for size < 0 is not required.
|
||||
*/
|
||||
if (size > (size_t)PY_SSIZE_T_MAX)
|
||||
return NULL;
|
||||
|
||||
return _PyMem_Raw.malloc(_PyMem_Raw.ctx, size);
|
||||
}
|
||||
|
||||
void*
|
||||
PyMem_RawRealloc(void *ptr, size_t new_size)
|
||||
{
|
||||
/* see PyMem_RawMalloc() */
|
||||
if (new_size > (size_t)PY_SSIZE_T_MAX)
|
||||
return NULL;
|
||||
return _PyMem_Raw.realloc(_PyMem_Raw.ctx, ptr, new_size);
|
||||
}
|
||||
|
||||
void PyMem_RawFree(void *ptr)
|
||||
{
|
||||
_PyMem_Raw.free(_PyMem_Raw.ctx, ptr);
|
||||
}
|
||||
|
||||
void *
|
||||
PyMem_Malloc(size_t size)
|
||||
{
|
||||
/* see PyMem_RawMalloc() */
|
||||
if (size > (size_t)PY_SSIZE_T_MAX)
|
||||
return NULL;
|
||||
return _PyMem.malloc(_PyMem.ctx, size);
|
||||
}
|
||||
|
||||
void *
|
||||
PyMem_Realloc(void *ptr, size_t new_size)
|
||||
{
|
||||
/* see PyMem_RawMalloc() */
|
||||
if (new_size > (size_t)PY_SSIZE_T_MAX)
|
||||
return NULL;
|
||||
return _PyMem.realloc(_PyMem.ctx, ptr, new_size);
|
||||
}
|
||||
|
||||
void
|
||||
PyMem_Free(void *ptr)
|
||||
{
|
||||
_PyMem.free(_PyMem.ctx, ptr);
|
||||
}
|
||||
|
||||
void *
|
||||
PyObject_Malloc(size_t size)
|
||||
{
|
||||
/* see PyMem_RawMalloc() */
|
||||
if (size > (size_t)PY_SSIZE_T_MAX)
|
||||
return NULL;
|
||||
return _PyObject.malloc(_PyObject.ctx, size);
|
||||
}
|
||||
|
||||
void *
|
||||
PyObject_Realloc(void *ptr, size_t new_size)
|
||||
{
|
||||
/* see PyMem_RawMalloc() */
|
||||
if (new_size > (size_t)PY_SSIZE_T_MAX)
|
||||
return NULL;
|
||||
return _PyObject.realloc(_PyObject.ctx, ptr, new_size);
|
||||
}
|
||||
|
||||
void
|
||||
PyObject_Free(void *ptr)
|
||||
{
|
||||
_PyObject.free(_PyObject.ctx, ptr);
|
||||
}
|
||||
|
||||
|
||||
#ifdef WITH_PYMALLOC
|
||||
|
||||
#ifdef WITH_VALGRIND
|
||||
#include <valgrind/valgrind.h>
|
||||
|
||||
|
|
@ -549,7 +857,6 @@ new_arena(void)
|
|||
struct arena_object* arenaobj;
|
||||
uint excess; /* number of bytes above pool alignment */
|
||||
void *address;
|
||||
int err;
|
||||
|
||||
#ifdef PYMALLOC_DEBUG
|
||||
if (Py_GETENV("PYTHONMALLOCSTATS"))
|
||||
|
|
@ -571,7 +878,7 @@ new_arena(void)
|
|||
return NULL; /* overflow */
|
||||
#endif
|
||||
nbytes = numarenas * sizeof(*arenas);
|
||||
arenaobj = (struct arena_object *)realloc(arenas, nbytes);
|
||||
arenaobj = (struct arena_object *)PyMem_Realloc(arenas, nbytes);
|
||||
if (arenaobj == NULL)
|
||||
return NULL;
|
||||
arenas = arenaobj;
|
||||
|
|
@ -602,19 +909,8 @@ new_arena(void)
|
|||
arenaobj = unused_arena_objects;
|
||||
unused_arena_objects = arenaobj->nextarena;
|
||||
assert(arenaobj->address == 0);
|
||||
#ifdef MS_WINDOWS
|
||||
address = (void*)VirtualAlloc(NULL, ARENA_SIZE,
|
||||
MEM_COMMIT | MEM_RESERVE, PAGE_READWRITE);
|
||||
err = (address == NULL);
|
||||
#elif defined(ARENAS_USE_MMAP)
|
||||
address = mmap(NULL, ARENA_SIZE, PROT_READ|PROT_WRITE,
|
||||
MAP_PRIVATE|MAP_ANONYMOUS, -1, 0);
|
||||
err = (address == MAP_FAILED);
|
||||
#else
|
||||
address = malloc(ARENA_SIZE);
|
||||
err = (address == 0);
|
||||
#endif
|
||||
if (err) {
|
||||
address = _PyObject_Arena.alloc(_PyObject_Arena.ctx, ARENA_SIZE);
|
||||
if (address == NULL) {
|
||||
/* The allocation failed: return NULL after putting the
|
||||
* arenaobj back.
|
||||
*/
|
||||
|
|
@ -777,9 +1073,8 @@ int Py_ADDRESS_IN_RANGE(void *P, poolp pool) Py_NO_INLINE;
|
|||
* Unless the optimizer reorders everything, being too smart...
|
||||
*/
|
||||
|
||||
#undef PyObject_Malloc
|
||||
void *
|
||||
PyObject_Malloc(size_t nbytes)
|
||||
static void *
|
||||
_PyObject_Malloc(void *ctx, size_t nbytes)
|
||||
{
|
||||
block *bp;
|
||||
poolp pool;
|
||||
|
|
@ -795,17 +1090,6 @@ PyObject_Malloc(size_t nbytes)
|
|||
goto redirect;
|
||||
#endif
|
||||
|
||||
/*
|
||||
* Limit ourselves to PY_SSIZE_T_MAX bytes to prevent security holes.
|
||||
* Most python internals blindly use a signed Py_ssize_t to track
|
||||
* things without checking for overflows or negatives.
|
||||
* As size_t is unsigned, checking for nbytes < 0 is not required.
|
||||
*/
|
||||
if (nbytes > PY_SSIZE_T_MAX) {
|
||||
_Py_AllocatedBlocks--;
|
||||
return NULL;
|
||||
}
|
||||
|
||||
/*
|
||||
* This implicitly redirects malloc(0).
|
||||
*/
|
||||
|
|
@ -978,10 +1262,8 @@ redirect:
|
|||
* last chance to serve the request) or when the max memory limit
|
||||
* has been reached.
|
||||
*/
|
||||
if (nbytes == 0)
|
||||
nbytes = 1;
|
||||
{
|
||||
void *result = malloc(nbytes);
|
||||
void *result = PyMem_Malloc(nbytes);
|
||||
if (!result)
|
||||
_Py_AllocatedBlocks--;
|
||||
return result;
|
||||
|
|
@ -990,9 +1272,8 @@ redirect:
|
|||
|
||||
/* free */
|
||||
|
||||
#undef PyObject_Free
|
||||
void
|
||||
PyObject_Free(void *p)
|
||||
static void
|
||||
_PyObject_Free(void *ctx, void *p)
|
||||
{
|
||||
poolp pool;
|
||||
block *lastfree;
|
||||
|
|
@ -1101,13 +1382,8 @@ PyObject_Free(void *p)
|
|||
unused_arena_objects = ao;
|
||||
|
||||
/* Free the entire arena. */
|
||||
#ifdef MS_WINDOWS
|
||||
VirtualFree((void *)ao->address, 0, MEM_RELEASE);
|
||||
#elif defined(ARENAS_USE_MMAP)
|
||||
munmap((void *)ao->address, ARENA_SIZE);
|
||||
#else
|
||||
free((void *)ao->address);
|
||||
#endif
|
||||
_PyObject_Arena.free(_PyObject_Arena.ctx,
|
||||
(void *)ao->address, ARENA_SIZE);
|
||||
ao->address = 0; /* mark unassociated */
|
||||
--narenas_currently_allocated;
|
||||
|
||||
|
|
@ -1216,7 +1492,7 @@ PyObject_Free(void *p)
|
|||
redirect:
|
||||
#endif
|
||||
/* We didn't allocate this address. */
|
||||
free(p);
|
||||
PyMem_Free(p);
|
||||
}
|
||||
|
||||
/* realloc. If p is NULL, this acts like malloc(nbytes). Else if nbytes==0,
|
||||
|
|
@ -1224,9 +1500,8 @@ redirect:
|
|||
* return a non-NULL result.
|
||||
*/
|
||||
|
||||
#undef PyObject_Realloc
|
||||
void *
|
||||
PyObject_Realloc(void *p, size_t nbytes)
|
||||
static void *
|
||||
_PyObject_Realloc(void *ctx, void *p, size_t nbytes)
|
||||
{
|
||||
void *bp;
|
||||
poolp pool;
|
||||
|
|
@ -1236,16 +1511,7 @@ PyObject_Realloc(void *p, size_t nbytes)
|
|||
#endif
|
||||
|
||||
if (p == NULL)
|
||||
return PyObject_Malloc(nbytes);
|
||||
|
||||
/*
|
||||
* Limit ourselves to PY_SSIZE_T_MAX bytes to prevent security holes.
|
||||
* Most python internals blindly use a signed Py_ssize_t to track
|
||||
* things without checking for overflows or negatives.
|
||||
* As size_t is unsigned, checking for nbytes < 0 is not required.
|
||||
*/
|
||||
if (nbytes > PY_SSIZE_T_MAX)
|
||||
return NULL;
|
||||
return _PyObject_Malloc(ctx, nbytes);
|
||||
|
||||
#ifdef WITH_VALGRIND
|
||||
/* Treat running_on_valgrind == -1 the same as 0 */
|
||||
|
|
@ -1273,10 +1539,10 @@ PyObject_Realloc(void *p, size_t nbytes)
|
|||
}
|
||||
size = nbytes;
|
||||
}
|
||||
bp = PyObject_Malloc(nbytes);
|
||||
bp = _PyObject_Malloc(ctx, nbytes);
|
||||
if (bp != NULL) {
|
||||
memcpy(bp, p, size);
|
||||
PyObject_Free(p);
|
||||
_PyObject_Free(ctx, p);
|
||||
}
|
||||
return bp;
|
||||
}
|
||||
|
|
@ -1294,14 +1560,14 @@ PyObject_Realloc(void *p, size_t nbytes)
|
|||
* at p. Instead we punt: let C continue to manage this block.
|
||||
*/
|
||||
if (nbytes)
|
||||
return realloc(p, nbytes);
|
||||
return PyMem_Realloc(p, nbytes);
|
||||
/* C doesn't define the result of realloc(p, 0) (it may or may not
|
||||
* return NULL then), but Python's docs promise that nbytes==0 never
|
||||
* returns NULL. We don't pass 0 to realloc(), to avoid that endcase
|
||||
* to begin with. Even then, we can't be sure that realloc() won't
|
||||
* return NULL.
|
||||
*/
|
||||
bp = realloc(p, 1);
|
||||
bp = PyMem_Realloc(p, 1);
|
||||
return bp ? bp : p;
|
||||
}
|
||||
|
||||
|
|
@ -1311,24 +1577,6 @@ PyObject_Realloc(void *p, size_t nbytes)
|
|||
/* pymalloc not enabled: Redirect the entry points to malloc. These will
|
||||
* only be used by extensions that are compiled with pymalloc enabled. */
|
||||
|
||||
void *
|
||||
PyObject_Malloc(size_t n)
|
||||
{
|
||||
return PyMem_MALLOC(n);
|
||||
}
|
||||
|
||||
void *
|
||||
PyObject_Realloc(void *p, size_t n)
|
||||
{
|
||||
return PyMem_REALLOC(p, n);
|
||||
}
|
||||
|
||||
void
|
||||
PyObject_Free(void *p)
|
||||
{
|
||||
PyMem_FREE(p);
|
||||
}
|
||||
|
||||
Py_ssize_t
|
||||
_Py_GetAllocatedBlocks(void)
|
||||
{
|
||||
|
|
@ -1354,10 +1602,6 @@ _Py_GetAllocatedBlocks(void)
|
|||
#define DEADBYTE 0xDB /* dead (newly freed) memory */
|
||||
#define FORBIDDENBYTE 0xFB /* untouchable bytes at each end of a block */
|
||||
|
||||
/* We tag each block with an API ID in order to tag API violations */
|
||||
#define _PYMALLOC_MEM_ID 'm' /* the PyMem_Malloc() API */
|
||||
#define _PYMALLOC_OBJ_ID 'o' /* The PyObject_Malloc() API */
|
||||
|
||||
static size_t serialno = 0; /* incremented on each debug {m,re}alloc */
|
||||
|
||||
/* serialno is always incremented via calling this routine. The point is
|
||||
|
|
@ -1440,58 +1684,18 @@ p[2*S: 2*S+n]
|
|||
p[2*S+n: 2*S+n+S]
|
||||
Copies of FORBIDDENBYTE. Used to catch over- writes and reads.
|
||||
p[2*S+n+S: 2*S+n+2*S]
|
||||
A serial number, incremented by 1 on each call to _PyObject_DebugMalloc
|
||||
and _PyObject_DebugRealloc.
|
||||
A serial number, incremented by 1 on each call to _PyMem_DebugMalloc
|
||||
and _PyMem_DebugRealloc.
|
||||
This is a big-endian size_t.
|
||||
If "bad memory" is detected later, the serial number gives an
|
||||
excellent way to set a breakpoint on the next run, to capture the
|
||||
instant at which this block was passed out.
|
||||
*/
|
||||
|
||||
/* debug replacements for the PyMem_* memory API */
|
||||
void *
|
||||
_PyMem_DebugMalloc(size_t nbytes)
|
||||
{
|
||||
return _PyObject_DebugMallocApi(_PYMALLOC_MEM_ID, nbytes);
|
||||
}
|
||||
void *
|
||||
_PyMem_DebugRealloc(void *p, size_t nbytes)
|
||||
{
|
||||
return _PyObject_DebugReallocApi(_PYMALLOC_MEM_ID, p, nbytes);
|
||||
}
|
||||
void
|
||||
_PyMem_DebugFree(void *p)
|
||||
{
|
||||
_PyObject_DebugFreeApi(_PYMALLOC_MEM_ID, p);
|
||||
}
|
||||
|
||||
/* debug replacements for the PyObject_* memory API */
|
||||
void *
|
||||
_PyObject_DebugMalloc(size_t nbytes)
|
||||
{
|
||||
return _PyObject_DebugMallocApi(_PYMALLOC_OBJ_ID, nbytes);
|
||||
}
|
||||
void *
|
||||
_PyObject_DebugRealloc(void *p, size_t nbytes)
|
||||
{
|
||||
return _PyObject_DebugReallocApi(_PYMALLOC_OBJ_ID, p, nbytes);
|
||||
}
|
||||
void
|
||||
_PyObject_DebugFree(void *p)
|
||||
{
|
||||
_PyObject_DebugFreeApi(_PYMALLOC_OBJ_ID, p);
|
||||
}
|
||||
void
|
||||
_PyObject_DebugCheckAddress(const void *p)
|
||||
{
|
||||
_PyObject_DebugCheckAddressApi(_PYMALLOC_OBJ_ID, p);
|
||||
}
|
||||
|
||||
|
||||
/* generic debug memory api, with an "id" to identify the API in use */
|
||||
void *
|
||||
_PyObject_DebugMallocApi(char id, size_t nbytes)
|
||||
static void *
|
||||
_PyMem_DebugMalloc(void *ctx, size_t nbytes)
|
||||
{
|
||||
debug_alloc_api_t *api = (debug_alloc_api_t *)ctx;
|
||||
uchar *p; /* base address of malloc'ed block */
|
||||
uchar *tail; /* p + 2*SST + nbytes == pointer to tail pad bytes */
|
||||
size_t total; /* nbytes + 4*SST */
|
||||
|
|
@ -1502,14 +1706,14 @@ _PyObject_DebugMallocApi(char id, size_t nbytes)
|
|||
/* overflow: can't represent total as a size_t */
|
||||
return NULL;
|
||||
|
||||
p = (uchar *)PyObject_Malloc(total);
|
||||
p = (uchar *)api->alloc.malloc(api->alloc.ctx, total);
|
||||
if (p == NULL)
|
||||
return NULL;
|
||||
|
||||
/* at p, write size (SST bytes), id (1 byte), pad (SST-1 bytes) */
|
||||
write_size_t(p, nbytes);
|
||||
p[SST] = (uchar)id;
|
||||
memset(p + SST + 1 , FORBIDDENBYTE, SST-1);
|
||||
p[SST] = (uchar)api->api_id;
|
||||
memset(p + SST + 1, FORBIDDENBYTE, SST-1);
|
||||
|
||||
if (nbytes > 0)
|
||||
memset(p + 2*SST, CLEANBYTE, nbytes);
|
||||
|
|
@ -1527,25 +1731,27 @@ _PyObject_DebugMallocApi(char id, size_t nbytes)
|
|||
Then fills the original bytes with DEADBYTE.
|
||||
Then calls the underlying free.
|
||||
*/
|
||||
void
|
||||
_PyObject_DebugFreeApi(char api, void *p)
|
||||
static void
|
||||
_PyMem_DebugFree(void *ctx, void *p)
|
||||
{
|
||||
debug_alloc_api_t *api = (debug_alloc_api_t *)ctx;
|
||||
uchar *q = (uchar *)p - 2*SST; /* address returned from malloc */
|
||||
size_t nbytes;
|
||||
|
||||
if (p == NULL)
|
||||
return;
|
||||
_PyObject_DebugCheckAddressApi(api, p);
|
||||
_PyMem_DebugCheckAddress(api->api_id, p);
|
||||
nbytes = read_size_t(q);
|
||||
nbytes += 4*SST;
|
||||
if (nbytes > 0)
|
||||
memset(q, DEADBYTE, nbytes);
|
||||
PyObject_Free(q);
|
||||
api->alloc.free(api->alloc.ctx, q);
|
||||
}
|
||||
|
||||
void *
|
||||
_PyObject_DebugReallocApi(char api, void *p, size_t nbytes)
|
||||
static void *
|
||||
_PyMem_DebugRealloc(void *ctx, void *p, size_t nbytes)
|
||||
{
|
||||
debug_alloc_api_t *api = (debug_alloc_api_t *)ctx;
|
||||
uchar *q = (uchar *)p;
|
||||
uchar *tail;
|
||||
size_t total; /* nbytes + 4*SST */
|
||||
|
|
@ -1553,9 +1759,9 @@ _PyObject_DebugReallocApi(char api, void *p, size_t nbytes)
|
|||
int i;
|
||||
|
||||
if (p == NULL)
|
||||
return _PyObject_DebugMallocApi(api, nbytes);
|
||||
return _PyMem_DebugMalloc(ctx, nbytes);
|
||||
|
||||
_PyObject_DebugCheckAddressApi(api, p);
|
||||
_PyMem_DebugCheckAddress(api->api_id, p);
|
||||
bumpserialno();
|
||||
original_nbytes = read_size_t(q - 2*SST);
|
||||
total = nbytes + 4*SST;
|
||||
|
|
@ -1572,12 +1778,12 @@ _PyObject_DebugReallocApi(char api, void *p, size_t nbytes)
|
|||
* case we didn't get the chance to mark the old memory with DEADBYTE,
|
||||
* but we live with that.
|
||||
*/
|
||||
q = (uchar *)PyObject_Realloc(q - 2*SST, total);
|
||||
q = (uchar *)api->alloc.realloc(api->alloc.ctx, q - 2*SST, total);
|
||||
if (q == NULL)
|
||||
return NULL;
|
||||
|
||||
write_size_t(q, nbytes);
|
||||
assert(q[SST] == (uchar)api);
|
||||
assert(q[SST] == (uchar)api->api_id);
|
||||
for (i = 1; i < SST; ++i)
|
||||
assert(q[SST + i] == FORBIDDENBYTE);
|
||||
q += 2*SST;
|
||||
|
|
@ -1599,8 +1805,8 @@ _PyObject_DebugReallocApi(char api, void *p, size_t nbytes)
|
|||
* and call Py_FatalError to kill the program.
|
||||
* The API id, is also checked.
|
||||
*/
|
||||
void
|
||||
_PyObject_DebugCheckAddressApi(char api, const void *p)
|
||||
static void
|
||||
_PyMem_DebugCheckAddress(char api, const void *p)
|
||||
{
|
||||
const uchar *q = (const uchar *)p;
|
||||
char msgbuf[64];
|
||||
|
|
@ -1652,7 +1858,7 @@ error:
|
|||
}
|
||||
|
||||
/* Display info to stderr about the memory block at p. */
|
||||
void
|
||||
static void
|
||||
_PyObject_DebugDumpAddress(const void *p)
|
||||
{
|
||||
const uchar *q = (const uchar *)p;
|
||||
|
|
|
|||
Loading…
Reference in New Issue