Issue #3329: Add new APIs to customize memory allocators

* Add a new PyMemAllocators structure
* New functions:

  - PyMem_RawMalloc(), PyMem_RawRealloc(), PyMem_RawFree(): GIL-free memory
    allocator functions
  - PyMem_GetRawAllocators(), PyMem_SetRawAllocators()
  - PyMem_GetAllocators(), PyMem_SetAllocators()
  - PyMem_SetupDebugHooks()
  - _PyObject_GetArenaAllocators(), _PyObject_SetArenaAllocators()

* Add unit test for PyMem_Malloc(0) and PyObject_Malloc(0)
* Add unit test for new get/set allocators functions
* PyObject_Malloc() now falls back on PyMem_Malloc() instead of malloc() if
  size is bigger than SMALL_REQUEST_THRESHOLD, and PyObject_Realloc() falls
  back on PyMem_Realloc() instead of realloc()
* PyMem_Malloc() and PyMem_Realloc() now always call malloc() and realloc(),
  instead of calling PyObject_Malloc() and PyObject_Realloc() in debug mode

Doc/c-api/memory.rst

@@ -84,6 +84,46 @@ the C library allocator as shown in the previous example, the allocated memory
 for the I/O buffer escapes completely the Python memory manager.
 
 
+Raw Memory Interface
+====================
+
+The following function are wrappers to system allocators: :c:func:`malloc`,
+:c:func:`realloc`, :c:func:`free`. These functions are thread-safe, the
+:term:`GIL <global interpreter lock>` does not need to be held to use these
+functions.
+
+The behaviour of requesting zero bytes is not defined: return *NULL* or a
+distinct non-*NULL* pointer depending on the platform. Use
+:c:func:`PyMem_Malloc` and :c:func:`PyMem_Realloc` to have a well defined
+behaviour.
+
+.. versionadded:: 3.4
+
+.. c:function:: void* PyMem_RawMalloc(size_t n)
+
+   Allocates *n* bytes and returns a pointer of type :c:type:`void\*` to the
+   allocated memory, or *NULL* if the request fails. The memory
+   will not have been initialized in any way.
+
+
+.. c:function:: void* PyMem_RawRealloc(void *p, size_t n)
+
+   Resizes the memory block pointed to by *p* to *n* bytes. The contents will
+   be unchanged to the minimum of the old and the new sizes. If *p* is *NULL*,
+   the call is equivalent to ``PyMem_RawMalloc(n)``. Unless *p* is *NULL*, it
+   must have been returned by a previous call to :c:func:`PyMem_RawMalloc` or
+   :c:func:`PyMem_RawRealloc`. If the request fails, :c:func:`PyMem_RawRealloc`
+   returns *NULL* and *p* remains a valid pointer to the previous memory area.
+
+
+.. c:function:: void PyMem_RawFree(void *p)
+
+   Frees the memory block pointed to by *p*, which must have been returned by a
+   previous call to :c:func:`PyMem_RawMalloc` or :c:func:`PyMem_RawRealloc`.
+   Otherwise, or if ``PyMem_Free(p)`` has been called before, undefined
+   behavior occurs. If *p* is *NULL*, no operation is performed.
+
+
 .. _memoryinterface:
 
 Memory Interface
@@ -91,8 +131,12 @@ Memory Interface
 
 The following function sets, modeled after the ANSI C standard, but specifying
 behavior when requesting zero bytes, are available for allocating and releasing
-memory from the Python heap:
+memory from the Python heap.
 
+.. warning::
+
+   The :term:`GIL <global interpreter lock>` must be held when using these
+   functions.
 
 .. c:function:: void* PyMem_Malloc(size_t n)
 
@@ -155,6 +199,81 @@ versions and is therefore deprecated in extension modules.
 :c:func:`PyMem_NEW`, :c:func:`PyMem_RESIZE`, :c:func:`PyMem_DEL`.
 
 
+Customize Memory Allocators
+===========================
+
+.. versionadded:: 3.4
+
+.. c:type:: PyMemAllocators
+
+   Structure used to describe memory allocator.  This structure has
+   four fields:
+
+   +----------------------------------------------------------+-----------------+
+   | Field                                                    | Meaning         |
+   +==========================================================+=================+
+   | ``void *ctx``                                            | user data       |
+   +----------------------------------------------------------+-----------------+
+   | ``void* malloc(void *ctx, size_t size)``                 | allocate memory |
+   +----------------------------------------------------------+-----------------+
+   | ``void* realloc(void *ctx, void *ptr, size_t new_size)`` | allocate memory |
+   |                                                          | or resize a     |
+   |                                                          | memory block    |
+   +----------------------------------------------------------+-----------------+
+   | ``void free(void *ctx, void *ptr)``                      | release memory  |
+   +----------------------------------------------------------+-----------------+
+
+.. c:function:: void PyMem_GetRawAllocators(PyMemAllocators *allocators)
+
+   Get internal functions of :c:func:`PyMem_RawMalloc`, :c:func:`PyMem_RawRealloc`
+   and :c:func:`PyMem_RawFree`.
+
+.. c:function:: void PyMem_SetRawAllocators(PyMemAllocators *allocators)
+
+   Set internal functions of :c:func:`PyMem_RawMalloc`, :c:func:`PyMem_RawRealloc`
+   and :c:func:`PyMem_RawFree`.
+
+   :c:func:`PyMem_SetupDebugHooks` should be called to reinstall debug hooks if
+   new functions do no call original functions anymore.
+
+.. c:function:: void PyMem_GetAllocators(PyMemAllocators *allocators)
+
+   Get internal functions of :c:func:`PyMem_Malloc`, :c:func:`PyMem_Realloc`
+   and :c:func:`PyMem_Free`.
+
+.. c:function:: void PyMem_SetAllocators(PyMemAllocators *allocators)
+
+   Set internal functions of :c:func:`PyMem_Malloc`, :c:func:`PyMem_Realloc`
+   and :c:func:`PyMem_Free`.
+
+   ``malloc(ctx, 0)`` and ``realloc(ctx, ptr, 0)`` must not return *NULL*: it
+   would be treated as an error.
+
+   :c:func:`PyMem_SetupDebugHooks` should be called to reinstall debug hooks if
+   new functions do no call original functions anymore.
+
+.. c:function:: void PyMem_SetupDebugHooks(void)
+
+   Setup hooks to detect bugs in the following Python memory allocator
+   functions:
+
+   - :c:func:`PyMem_RawMalloc`, :c:func:`PyMem_RawRealloc`,
+     :c:func:`PyMem_RawFree`
+   - :c:func:`PyMem_Malloc`, :c:func:`PyMem_Realloc`, :c:func:`PyMem_Free`
+   - :c:func:`PyObject_Malloc`, :c:func:`PyObject_Realloc`,
+     :c:func:`PyObject_Free`
+
+   Newly allocated memory is filled with the byte ``0xCB``, freed memory is
+   filled with the byte ``0xDB``. Additionnal checks:
+
+   - detect API violations, ex: :c:func:`PyObject_Free` called on a buffer
+     allocated by :c:func:`PyMem_Malloc`
+   - detect write before the start of the buffer (buffer underflow)
+   - detect write after the end of the buffer (buffer overflow)
+
+   The function does nothing if Python is not compiled is debug mode.
+
+
 .. _memoryexamples:
 
 Examples

Include/objimpl.h

@@ -94,9 +94,9 @@ PyObject_{New, NewVar, Del}.
    the object gets initialized via PyObject_{Init, InitVar} after obtaining
    the raw memory.
 */
-PyAPI_FUNC(void *) PyObject_Malloc(size_t);
+PyAPI_FUNC(void *) PyObject_Malloc(size_t size);
-PyAPI_FUNC(void *) PyObject_Realloc(void *, size_t);
+PyAPI_FUNC(void *) PyObject_Realloc(void *ptr, size_t new_size);
-PyAPI_FUNC(void) PyObject_Free(void *);
+PyAPI_FUNC(void) PyObject_Free(void *ptr);
 
 /* This function returns the number of allocated memory blocks, regardless of size */
 PyAPI_FUNC(Py_ssize_t) _Py_GetAllocatedBlocks(void);
@@ -106,41 +106,46 @@ PyAPI_FUNC(Py_ssize_t) _Py_GetAllocatedBlocks(void);
 #ifndef Py_LIMITED_API
 PyAPI_FUNC(void) _PyObject_DebugMallocStats(FILE *out);
 #endif /* #ifndef Py_LIMITED_API */
-#ifdef PYMALLOC_DEBUG   /* WITH_PYMALLOC && PYMALLOC_DEBUG */
+#endif
-PyAPI_FUNC(void *) _PyObject_DebugMalloc(size_t nbytes);
-PyAPI_FUNC(void *) _PyObject_DebugRealloc(void *p, size_t nbytes);
-PyAPI_FUNC(void) _PyObject_DebugFree(void *p);
-PyAPI_FUNC(void) _PyObject_DebugDumpAddress(const void *p);
-PyAPI_FUNC(void) _PyObject_DebugCheckAddress(const void *p);
-PyAPI_FUNC(void *) _PyObject_DebugMallocApi(char api, size_t nbytes);
-PyAPI_FUNC(void *) _PyObject_DebugReallocApi(char api, void *p, size_t nbytes);
-PyAPI_FUNC(void) _PyObject_DebugFreeApi(char api, void *p);
-PyAPI_FUNC(void) _PyObject_DebugCheckAddressApi(char api, const void *p);
-PyAPI_FUNC(void *) _PyMem_DebugMalloc(size_t nbytes);
-PyAPI_FUNC(void *) _PyMem_DebugRealloc(void *p, size_t nbytes);
-PyAPI_FUNC(void) _PyMem_DebugFree(void *p);
-#define PyObject_MALLOC         _PyObject_DebugMalloc
-#define PyObject_Malloc         _PyObject_DebugMalloc
-#define PyObject_REALLOC        _PyObject_DebugRealloc
-#define PyObject_Realloc        _PyObject_DebugRealloc
-#define PyObject_FREE           _PyObject_DebugFree
-#define PyObject_Free           _PyObject_DebugFree
 
-#else   /* WITH_PYMALLOC && ! PYMALLOC_DEBUG */
+/* Macros */
 #define PyObject_MALLOC         PyObject_Malloc
 #define PyObject_REALLOC        PyObject_Realloc
 #define PyObject_FREE           PyObject_Free
-#endif
-
-#else   /* ! WITH_PYMALLOC */
-#define PyObject_MALLOC         PyMem_MALLOC
-#define PyObject_REALLOC        PyMem_REALLOC
-#define PyObject_FREE           PyMem_FREE
-
-#endif  /* WITH_PYMALLOC */
-
 #define PyObject_Del            PyObject_Free
-#define PyObject_DEL            PyObject_FREE
+#define PyObject_DEL            PyObject_Free
+
+/* Get internal functions of PyObject_Malloc(), PyObject_Realloc() and
+   PyObject_Free(). *ctx_p is an arbitrary user value. */
+PyAPI_FUNC(void) PyObject_GetAllocators(PyMemAllocators *allocators);
+
+/* Set internal functions of PyObject_Malloc(), PyObject_Realloc() and PyObject_Free().
+   ctx is an arbitrary user value.
+
+   malloc(ctx, 0) and realloc(ctx, ptr, 0) must not return NULL: it would be
+   treated as an error.
+
+   PyMem_SetupDebugHooks() should be called to reinstall debug hooks if new
+   functions do no call original functions anymore. */
+PyAPI_FUNC(void) PyObject_SetAllocators(PyMemAllocators *allocators);
+
+/* Get internal functions allocating and deallocating arenas for
+   PyObject_Malloc(), PyObject_Realloc() and PyObject_Free().
+   *ctx_p is an arbitrary user value. */
+PyAPI_FUNC(void) _PyObject_GetArenaAllocators(
+    void **ctx_p,
+    void* (**malloc_p) (void *ctx, size_t size),
+    void (**free_p) (void *ctx, void *ptr, size_t size)
+    );
+
+/* Get internal functions allocating and deallocating arenas for
+   PyObject_Malloc(), PyObject_Realloc() and PyObject_Free().
+   ctx is an arbitrary user value. */
+PyAPI_FUNC(void) _PyObject_SetArenaAllocators(
+    void *ctx,
+    void* (*malloc) (void *ctx, size_t size),
+    void (*free) (void *ctx, void *ptr, size_t size)
+    );
 
 /*
  * Generic object allocator interface

Include/pymem.h

@@ -11,6 +11,40 @@
 extern "C" {
 #endif
 
+typedef struct {
+    /* user context passed as the first argument to the 3 functions */
+    void *ctx;
+
+    /* allocate memory */
+    void* (*malloc) (void *ctx, size_t size);
+
+    /* allocate memory or resize a memory buffer */
+    void* (*realloc) (void *ctx, void *ptr, size_t new_size);
+
+    /* release memory */
+    void (*free) (void *ctx, void *ptr);
+} PyMemAllocators;
+
+/* Raw memory allocators, system functions: malloc(), realloc(), free().
+
+   These functions are thread-safe, the GIL does not need to be held. */
+
+/* Get internal functions of PyMem_RawMalloc(), PyMem_RawRealloc() and
+   PyMem_RawFree(). *ctx_p is an arbitrary user value. */
+PyAPI_FUNC(void) PyMem_GetRawAllocators(PyMemAllocators *allocators);
+
+/* Set internal functions of PyMem_RawMalloc(), PyMem_RawRealloc() and
+   PyMem_RawFree(). ctx is an arbitrary user value.
+
+   PyMem_SetupDebugHooks() should be called to reinstall debug hooks if new
+   functions do no call original functions anymore. */
+PyAPI_FUNC(void) PyMem_SetRawAllocators(PyMemAllocators *allocators);
+
+PyAPI_FUNC(void *) PyMem_RawMalloc(size_t size);
+PyAPI_FUNC(void *) PyMem_RawRealloc(void *ptr, size_t new_size);
+PyAPI_FUNC(void) PyMem_RawFree(void *ptr);
+
+
 /* BEWARE:
 
    Each interface exports both functions and macros.  Extension modules should
@@ -49,21 +83,11 @@ extern "C" {
    performed on failure (no exception is set, no warning is printed, etc).
 */
 
-PyAPI_FUNC(void *) PyMem_Malloc(size_t);
+PyAPI_FUNC(void *) PyMem_Malloc(size_t size);
-PyAPI_FUNC(void *) PyMem_Realloc(void *, size_t);
+PyAPI_FUNC(void *) PyMem_Realloc(void *ptr, size_t new_size);
-PyAPI_FUNC(void) PyMem_Free(void *);
+PyAPI_FUNC(void) PyMem_Free(void *ptr);
-
-/* Starting from Python 1.6, the wrappers Py_{Malloc,Realloc,Free} are
-   no longer supported. They used to call PyErr_NoMemory() on failure. */
 
 /* Macros. */
-#ifdef PYMALLOC_DEBUG
-/* Redirect all memory operations to Python's debugging allocator. */
-#define PyMem_MALLOC		_PyMem_DebugMalloc
-#define PyMem_REALLOC		_PyMem_DebugRealloc
-#define PyMem_FREE		_PyMem_DebugFree
-
-#else	/* ! PYMALLOC_DEBUG */
 
 /* PyMem_MALLOC(0) means malloc(1). Some systems would return NULL
    for malloc(0), which would be treated as an error. Some platforms
@@ -71,13 +95,9 @@ PyAPI_FUNC(void) PyMem_Free(void *);
    pymalloc. To solve these problems, allocate an extra byte. */
 /* Returns NULL to indicate error if a negative size or size larger than
    Py_ssize_t can represent is supplied.  Helps prevents security holes. */
-#define PyMem_MALLOC(n)		((size_t)(n) > (size_t)PY_SSIZE_T_MAX ? NULL \
+#define PyMem_MALLOC(n)         PyMem_Malloc(n)
-				: malloc((n) ? (n) : 1))
+#define PyMem_REALLOC(p, n)     PyMem_Realloc(p, n)
-#define PyMem_REALLOC(p, n)	((size_t)(n) > (size_t)PY_SSIZE_T_MAX  ? NULL \
+#define PyMem_FREE(p)           PyMem_Free(p)
-				: realloc((p), (n) ? (n) : 1))
-#define PyMem_FREE		free
-
-#endif	/* PYMALLOC_DEBUG */
 
 /*
  * Type-oriented memory interface
@@ -115,6 +135,37 @@ PyAPI_FUNC(void) PyMem_Free(void *);
 #define PyMem_Del		PyMem_Free
 #define PyMem_DEL		PyMem_FREE
 
+/* Get internal functions of PyMem_Malloc(), PyMem_Realloc()
+   and PyMem_Free() */
+PyAPI_FUNC(void) PyMem_GetAllocators(PyMemAllocators *allocators);
+
+/* Set internal functions of PyMem_Malloc(), PyMem_Realloc() and PyMem_Free().
+
+   malloc(ctx, 0) and realloc(ctx, ptr, 0) must not return NULL: it would be
+   treated as an error.
+
+   PyMem_SetupDebugHooks() should be called to reinstall debug hooks if new
+   functions do no call original functions anymore. */
+PyAPI_FUNC(void) PyMem_SetAllocators(PyMemAllocators *allocators);
+
+/* Setup hooks to detect bugs in the following Python memory allocator
+   functions:
+
+   - PyMem_RawMalloc(), PyMem_RawRealloc(), PyMem_RawFree()
+   - PyMem_Malloc(), PyMem_Realloc(), PyMem_Free()
+   - PyObject_Malloc(), PyObject_Realloc() and PyObject_Free()
+
+   Newly allocated memory is filled with the byte 0xCB, freed memory is filled
+   with the byte 0xDB. Additionnal checks:
+
+   - detect API violations, ex: PyObject_Free() called on a buffer allocated
+     by PyMem_Malloc()
+   - detect write before the start of the buffer (buffer underflow)
+   - detect write after the end of the buffer (buffer overflow)
+
+   The function does nothing if Python is not compiled is debug mode. */
+PyAPI_FUNC(void) PyMem_SetupDebugHooks(void);
+
 #ifdef __cplusplus
 }
 #endif

Modules/_testcapimodule.c

@@ -2511,6 +2511,176 @@ test_decref_doesnt_leak(PyObject *ob)
     Py_RETURN_NONE;
 }
 
+static PyObject *
+test_pymem_alloc0(PyObject *self)
+{
+    void *ptr;
+
+    ptr = PyMem_Malloc(0);
+    if (ptr == NULL) {
+        PyErr_SetString(PyExc_RuntimeError, "PyMem_Malloc(0) returns NULL");
+        return NULL;
+    }
+    PyMem_Free(ptr);
+
+    ptr = PyObject_Malloc(0);
+    if (ptr == NULL) {
+        PyErr_SetString(PyExc_RuntimeError, "PyObject_Malloc(0) returns NULL");
+        return NULL;
+    }
+    PyObject_Free(ptr);
+
+    Py_RETURN_NONE;
+}
+
+typedef struct {
+    PyMemAllocators alloc;
+
+    size_t malloc_size;
+    void *realloc_ptr;
+    size_t realloc_new_size;
+    void *free_ptr;
+} alloc_hook_t;
+
+static void* hook_malloc (void* ctx, size_t size)
+{
+    alloc_hook_t *hook = (alloc_hook_t *)ctx;
+    hook->malloc_size = size;
+    return hook->alloc.malloc(hook->alloc.ctx, size);
+}
+
+static void* hook_realloc (void* ctx, void* ptr, size_t new_size)
+{
+    alloc_hook_t *hook = (alloc_hook_t *)ctx;
+    hook->realloc_ptr = ptr;
+    hook->realloc_new_size = new_size;
+    return hook->alloc.realloc(hook->alloc.ctx, ptr, new_size);
+}
+
+static void hook_free (void *ctx, void *ptr)
+{
+    alloc_hook_t *hook = (alloc_hook_t *)ctx;
+    printf("HOOK\n");
+    hook->free_ptr = ptr;
+    hook->alloc.free(hook->alloc.ctx, ptr);
+}
+
+static PyObject *
+test_setallocators(char api)
+{
+    PyObject *res = NULL;
+    const char *error_msg;
+    alloc_hook_t hook;
+    PyMemAllocators alloc;
+    size_t size, size2;
+    void *ptr, *ptr2;
+
+    hook.malloc_size = 0;
+    hook.realloc_ptr = NULL;
+    hook.realloc_new_size = 0;
+    hook.free_ptr = NULL;
+
+    alloc.ctx = &hook;
+    alloc.malloc = &hook_malloc;
+    alloc.realloc = &hook_realloc;
+    alloc.free = &hook_free;
+    if (api == 'o') {
+        PyObject_GetAllocators(&hook.alloc);
+        PyObject_SetAllocators(&alloc);
+    }
+    else if (api == 'r') {
+        PyMem_GetRawAllocators(&hook.alloc);
+        PyMem_SetRawAllocators(&alloc);
+    }
+    else {
+        PyMem_GetAllocators(&hook.alloc);
+        PyMem_SetAllocators(&alloc);
+    }
+
+    size = 42;
+    if (api == 'o')
+        ptr = PyObject_Malloc(size);
+    else if (api == 'r')
+        ptr = PyMem_RawMalloc(size);
+    else
+        ptr = PyMem_Malloc(size);
+    if (ptr == NULL) {
+        error_msg = "malloc failed";
+        goto fail;
+    }
+
+    if (hook.malloc_size != size) {
+        error_msg = "malloc invalid size";
+        goto fail;
+    }
+
+    size2 = 200;
+    if (api == 'o')
+        ptr2 = PyObject_Realloc(ptr, size2);
+    else if (api == 'r')
+        ptr2 = PyMem_RawRealloc(ptr, size2);
+    else
+        ptr2 = PyMem_Realloc(ptr, size2);
+    if (ptr2 == NULL) {
+        error_msg = "realloc failed";
+        goto fail;
+    }
+
+    if (hook.realloc_ptr != ptr
+        || hook.realloc_new_size != size2) {
+        error_msg = "realloc invalid parameters";
+        goto fail;
+    }
+
+    if (api == 'o')
+        PyObject_Free(ptr2);
+    else if (api == 'r')
+        PyMem_RawFree(ptr2);
+    else {
+        printf("PyMem_Free\n");
+        PyMem_Free(ptr2);
+    }
+
+    if (hook.free_ptr != ptr2) {
+        error_msg = "free invalid pointer";
+        goto fail;
+    }
+
+    Py_INCREF(Py_None);
+    res = Py_None;
+    goto finally;
+
+fail:
+    PyErr_SetString(PyExc_RuntimeError, error_msg);
+
+finally:
+    if (api == 'o')
+        PyObject_SetAllocators(&hook.alloc);
+    else if (api == 'r')
+        PyMem_SetRawAllocators(&hook.alloc);
+    else
+        PyMem_SetAllocators(&hook.alloc);
+    return res;
+}
+
+static PyObject *
+test_pymem_setrawallocators(PyObject *self)
+{
+    return test_setallocators('r');
+}
+
+static PyObject *
+test_pymem_setallocators(PyObject *self)
+{
+    return test_setallocators('m');
+}
+
+static PyObject *
+test_pyobject_setallocators(PyObject *self)
+{
+    return test_setallocators('o');
+}
+
 static PyMethodDef TestMethods[] = {
     {"raise_exception",         raise_exception,                 METH_VARARGS},
     {"raise_memoryerror",   (PyCFunction)raise_memoryerror,  METH_NOARGS},
@@ -2611,6 +2781,14 @@ static PyMethodDef TestMethods[] = {
     {"pytime_object_to_time_t", test_pytime_object_to_time_t,  METH_VARARGS},
     {"pytime_object_to_timeval", test_pytime_object_to_timeval,  METH_VARARGS},
     {"pytime_object_to_timespec", test_pytime_object_to_timespec,  METH_VARARGS},
+    {"test_pymem",
+     (PyCFunction)test_pymem_alloc0, METH_NOARGS},
+    {"test_pymem_alloc0",
+     (PyCFunction)test_pymem_setrawallocators, METH_NOARGS},
+    {"test_pymem_setallocators",
+     (PyCFunction)test_pymem_setallocators, METH_NOARGS},
+    {"test_pyobject_setallocators",
+     (PyCFunction)test_pyobject_setallocators, METH_NOARGS},
     {NULL, NULL} /* sentinel */
 };
 

Objects/object.c

@@ -1859,26 +1859,6 @@ PyTypeObject *_PyCapsule_hack = &PyCapsule_Type;
 Py_ssize_t (*_Py_abstract_hack)(PyObject *) = PyObject_Size;
 
 
-/* Python's malloc wrappers (see pymem.h) */
-
-void *
-PyMem_Malloc(size_t nbytes)
-{
-    return PyMem_MALLOC(nbytes);
-}
-
-void *
-PyMem_Realloc(void *p, size_t nbytes)
-{
-    return PyMem_REALLOC(p, nbytes);
-}
-
-void
-PyMem_Free(void *p)
-{
-    PyMem_FREE(p);
-}
-
 void
 _PyObject_DebugTypeStats(FILE *out)
 {

Objects/obmalloc.c

@@ -1,5 +1,327 @@
 #include "Python.h"
 
+/* Python's malloc wrappers (see pymem.h) */
+
+/* Forward declaration */
+
+#ifdef PYMALLOC_DEBUG   /* WITH_PYMALLOC && PYMALLOC_DEBUG */
+static void* _PyMem_DebugMalloc(void *ctx, size_t size);
+static void _PyMem_DebugFree(void *ctx, void *p);
+static void* _PyMem_DebugRealloc(void *ctx, void *ptr, size_t size);
+
+static void _PyObject_DebugDumpAddress(const void *p);
+static void _PyMem_DebugCheckAddress(char api_id, const void *p);
+#endif
+
+#ifdef WITH_PYMALLOC
+static void* _PyObject_Malloc(void *ctx, size_t size);
+static void _PyObject_Free(void *ctx, void *p);
+static void* _PyObject_Realloc(void *ctx, void *ptr, size_t size);
+#endif
+
+
+static void *
+_PyMem_RawMalloc(void *ctx, size_t size)
+{
+    return malloc(size);
+}
+
+static void *
+_PyMem_RawRealloc(void *ctx, void *ptr, size_t size)
+{
+    return realloc(ptr, size);
+}
+
+static void
+_PyMem_RawFree(void *ctx, void *ptr)
+{
+    return free(ptr);
+}
+
+static void *
+_PyMem_Malloc(void *ctx, size_t size)
+{
+    /* PyMem_Malloc(0) means malloc(1). Some systems would return NULL
+       for malloc(0), which would be treated as an error. Some platforms would
+       return a pointer with no memory behind it, which would break pymalloc.
+       To solve these problems, allocate an extra byte. */
+    if (size == 0)
+        size = 1;
+    return malloc(size);
+}
+
+static void *
+_PyMem_Realloc(void *ctx, void *ptr, size_t size)
+{
+    if (size == 0)
+        size = 1;
+    return realloc(ptr, size);
+}
+
+#ifdef ARENAS_USE_MMAP
+static void *
+_PyObject_ArenaMmap(void *ctx, size_t size)
+{
+    void *ptr;
+    ptr = mmap(NULL, size, PROT_READ|PROT_WRITE,
+               MAP_PRIVATE|MAP_ANONYMOUS, -1, 0);
+    if (ptr == MAP_FAILED)
+        return NULL;
+    assert(ptr != NULL);
+    return ptr;
+}
+
+static void
+_PyObject_ArenaMunmap(void *ctx, void *ptr, size_t size)
+{
+    return munmap(ptr, size);
+}
+#else
+static void *
+_PyObject_ArenaMalloc(void *ctx, size_t size)
+{
+    return malloc(size);
+}
+
+static void
+_PyObject_ArenaFree(void *ctx, void *ptr, size_t size)
+{
+    free(ptr);
+}
+#endif
+
+#define PYRAW_FUNCS _PyMem_RawMalloc, _PyMem_RawRealloc, _PyMem_RawFree
+#define PYMEM_FUNCS _PyMem_Malloc, _PyMem_Realloc, _PyMem_RawFree
+#ifdef WITH_PYMALLOC
+#define PYOBJECT_FUNCS _PyObject_Malloc, _PyObject_Realloc, _PyObject_Free
+#else
+#define PYOBJECT_FUNCS PYMEM_FUNCS
+#endif
+
+#ifdef PYMALLOC_DEBUG
+typedef struct {
+    /* We tag each block with an API ID in order to tag API violations */
+    char api_id;
+    PyMemAllocators alloc;
+} debug_alloc_api_t;
+static struct {
+    debug_alloc_api_t raw;
+    debug_alloc_api_t mem;
+    debug_alloc_api_t obj;
+} _PyMem_Debug = {
+    {'r', {NULL, PYRAW_FUNCS}},
+    {'m', {NULL, PYMEM_FUNCS}},
+    {'o', {NULL, PYOBJECT_FUNCS}}
+    };
+
+#define PYDEBUG_FUNCS _PyMem_DebugMalloc, _PyMem_DebugRealloc, _PyMem_DebugFree
+#endif
+
+static PyMemAllocators _PyMem_Raw = {
+#ifdef PYMALLOC_DEBUG
+    &_PyMem_Debug.raw, PYDEBUG_FUNCS
+#else
+    NULL, PYMEM_FUNCS
+#endif
+    };
+
+static PyMemAllocators _PyMem = {
+#ifdef PYMALLOC_DEBUG
+    &_PyMem_Debug.mem, PYDEBUG_FUNCS
+#else
+    NULL, PYMEM_FUNCS
+#endif
+    };
+
+static PyMemAllocators _PyObject = {
+#ifdef PYMALLOC_DEBUG
+    &_PyMem_Debug.obj, PYDEBUG_FUNCS
+#else
+    NULL, PYOBJECT_FUNCS
+#endif
+    };
+
+#undef PYRAW_FUNCS
+#undef PYMEM_FUNCS
+#undef PYOBJECT_FUNCS
+#undef PYDEBUG_FUNCS
+
+static struct {
+    void *ctx;
+    void* (*malloc) (void*, size_t);
+    void (*free) (void*, void*, size_t);
+} _PyObject_Arena = {NULL,
+#ifdef ARENAS_USE_MMAP
+    _PyObject_ArenaMmap, _PyObject_ArenaMunmap
+#else
+    _PyObject_ArenaMalloc, _PyObject_ArenaFree
+#endif
+    };
+
+void
+PyMem_SetupDebugHooks(void)
+{
+#ifdef PYMALLOC_DEBUG
+    PyMemAllocators alloc;
+
+    alloc.malloc = _PyMem_DebugMalloc;
+    alloc.realloc = _PyMem_DebugRealloc;
+    alloc.free = _PyMem_DebugFree;
+
+    if (_PyMem_Raw.malloc != _PyMem_DebugMalloc) {
+        alloc.ctx = &_PyMem_Debug.raw;
+        PyMem_GetAllocators(&_PyMem_Debug.raw.alloc);
+        PyMem_SetAllocators(&alloc);
+    }
+
+    if (_PyMem.malloc != _PyMem_DebugMalloc) {
+        alloc.ctx = &_PyMem_Debug.mem;
+        PyMem_GetAllocators(&_PyMem_Debug.mem.alloc);
+        PyMem_SetAllocators(&alloc);
+    }
+
+    if (_PyObject.malloc != _PyMem_DebugMalloc) {
+        alloc.ctx = &_PyMem_Debug.obj;
+        PyObject_GetAllocators(&_PyMem_Debug.obj.alloc);
+        PyObject_SetAllocators(&alloc);
+    }
+#endif
+}
+
+void
+PyMem_GetRawAllocators(PyMemAllocators *allocators)
+{
+    *allocators = _PyMem_Raw;
+}
+
+void
+PyMem_SetRawAllocators(PyMemAllocators *allocators)
+{
+    _PyMem_Raw = *allocators;
+}
+
+void
+PyMem_GetAllocators(PyMemAllocators *allocators)
+{
+    *allocators = _PyMem;
+}
+
+void
+PyMem_SetAllocators(PyMemAllocators *allocators)
+{
+    _PyMem = *allocators;
+}
+
+void
+PyObject_GetAllocators(PyMemAllocators *allocators)
+{
+    *allocators = _PyObject;
+}
+
+void
+PyObject_SetAllocators(PyMemAllocators *allocators)
+{
+    _PyObject = *allocators;
+}
+
+void
+_PyObject_GetArenaAllocators(void **ctx_p,
+                             void* (**malloc_p) (void *ctx, size_t size),
+                             void (**free_p) (void *ctx, void *ptr, size_t size))
+{
+    *malloc_p = _PyObject_Arena.malloc;
+    *free_p = _PyObject_Arena.free;
+    *ctx_p = _PyObject_Arena.ctx;
+}
+
+void
+_PyObject_SetArenaAllocators(void *ctx,
+                             void* (*malloc) (void *ctx, size_t size),
+                             void (*free) (void *ctx, void *ptr, size_t size))
+{
+    _PyObject_Arena.malloc = malloc;
+    _PyObject_Arena.free = free;
+    _PyObject_Arena.ctx = ctx;
+}
+
+void *
+PyMem_RawMalloc(size_t size)
+{
+    return _PyMem_Raw.malloc(_PyMem_Raw.ctx, size);
+}
+
+void*
+PyMem_RawRealloc(void *ptr, size_t new_size)
+{
+    return _PyMem_Raw.realloc(_PyMem_Raw.ctx, ptr, new_size);
+}
+
+void PyMem_RawFree(void *ptr)
+{
+    _PyMem_Raw.free(_PyMem_Raw.ctx, ptr);
+}
+
+void *
+PyMem_Malloc(size_t size)
+{
+    /*
+     * Limit ourselves to PY_SSIZE_T_MAX bytes to prevent security holes.
+     * Most python internals blindly use a signed Py_ssize_t to track
+     * things without checking for overflows or negatives.
+     * As size_t is unsigned, checking for size < 0 is not required.
+     */
+    if (size > (size_t)PY_SSIZE_T_MAX)
+        return NULL;
+
+    return _PyMem.malloc(_PyMem.ctx, size);
+}
+
+void *
+PyMem_Realloc(void *ptr, size_t new_size)
+{
+    if (new_size > (size_t)PY_SSIZE_T_MAX)
+        return NULL;
+
+    return _PyMem.realloc(_PyMem.ctx, ptr, new_size);
+}
+
+void
+PyMem_Free(void *ptr)
+{
+    _PyMem.free(_PyMem.ctx, ptr);
+}
+
+void *
+PyObject_Malloc(size_t size)
+{
+    /*
+     * Limit ourselves to PY_SSIZE_T_MAX bytes to prevent security holes.
+     * Most python internals blindly use a signed Py_ssize_t to track
+     * things without checking for overflows or negatives.
+     * As size_t is unsigned, checking for size < 0 is not required.
+     */
+    if (size > (size_t)PY_SSIZE_T_MAX)
+        return NULL;
+
+    return _PyObject.malloc(_PyObject.ctx, size);
+}
+
+void *
+PyObject_Realloc(void *ptr, size_t new_size)
+{
+    if (new_size > (size_t)PY_SSIZE_T_MAX)
+        return NULL;
+
+    return _PyObject.realloc(_PyObject.ctx, ptr, new_size);
+}
+
+void
+PyObject_Free(void *ptr)
+{
+    _PyObject.free(_PyObject.ctx, ptr);
+}
+
+
 #ifdef WITH_PYMALLOC
 
 #ifdef HAVE_MMAP
@@ -545,7 +867,6 @@ new_arena(void)
     struct arena_object* arenaobj;
     uint excess;        /* number of bytes above pool alignment */
     void *address;
-    int err;
 
 #ifdef PYMALLOC_DEBUG
     if (Py_GETENV("PYTHONMALLOCSTATS"))
@@ -567,11 +888,12 @@ new_arena(void)
             return NULL;                /* overflow */
 #endif
         nbytes = numarenas * sizeof(*arenas);
-        arenaobj = (struct arena_object *)realloc(arenas, nbytes);
+        arenaobj = (struct arena_object *)PyMem_Realloc(arenas, nbytes);
         if (arenaobj == NULL)
             return NULL;
         arenas = arenaobj;
 
+
         /* We might need to fix pointers that were copied.  However,
          * new_arena only gets called when all the pages in the
          * previous arenas are full.  Thus, there are *no* pointers
@@ -598,15 +920,8 @@ new_arena(void)
     arenaobj = unused_arena_objects;
     unused_arena_objects = arenaobj->nextarena;
     assert(arenaobj->address == 0);
-#ifdef ARENAS_USE_MMAP
+    address = _PyObject_Arena.malloc(_PyObject_Arena.ctx, ARENA_SIZE);
-    address = mmap(NULL, ARENA_SIZE, PROT_READ|PROT_WRITE,
+    if (address == NULL) {
-                                   MAP_PRIVATE|MAP_ANONYMOUS, -1, 0);
-    err = (address == MAP_FAILED);
-#else
-    address = malloc(ARENA_SIZE);
-    err = (address == 0);
-#endif
-    if (err) {
         /* The allocation failed: return NULL after putting the
          * arenaobj back.
          */
@@ -769,9 +1084,8 @@ int Py_ADDRESS_IN_RANGE(void *P, poolp pool) Py_NO_INLINE;
  * Unless the optimizer reorders everything, being too smart...
  */
 
-#undef PyObject_Malloc
+static void *
-void *
+_PyObject_Malloc(void *ctx, size_t nbytes)
-PyObject_Malloc(size_t nbytes)
 {
     block *bp;
     poolp pool;
@@ -787,17 +1101,6 @@ PyObject_Malloc(size_t nbytes)
         goto redirect;
 #endif
 
-    /*
-     * Limit ourselves to PY_SSIZE_T_MAX bytes to prevent security holes.
-     * Most python internals blindly use a signed Py_ssize_t to track
-     * things without checking for overflows or negatives.
-     * As size_t is unsigned, checking for nbytes < 0 is not required.
-     */
-    if (nbytes > PY_SSIZE_T_MAX) {
-        _Py_AllocatedBlocks--;
-        return NULL;
-    }
-
     /*
      * This implicitly redirects malloc(0).
      */
@@ -970,10 +1273,8 @@ redirect:
      * last chance to serve the request) or when the max memory limit
      * has been reached.
      */
-    if (nbytes == 0)
-        nbytes = 1;
     {
-        void *result = malloc(nbytes);
+        void *result = PyMem_Malloc(nbytes);
         if (!result)
             _Py_AllocatedBlocks--;
         return result;
@@ -982,9 +1283,8 @@ redirect:
 
 /* free */
 
-#undef PyObject_Free
+static void
-void
+_PyObject_Free(void *ctx, void *p)
-PyObject_Free(void *p)
 {
     poolp pool;
     block *lastfree;
@@ -1093,11 +1393,8 @@ PyObject_Free(void *p)
                 unused_arena_objects = ao;
 
                 /* Free the entire arena. */
-#ifdef ARENAS_USE_MMAP
+                _PyObject_Arena.free(_PyObject_Arena.ctx,
-                munmap((void *)ao->address, ARENA_SIZE);
+                                     (void *)ao->address, ARENA_SIZE);
-#else
-                free((void *)ao->address);
-#endif
                 ao->address = 0;                        /* mark unassociated */
                 --narenas_currently_allocated;
 
@@ -1206,7 +1503,7 @@ PyObject_Free(void *p)
 redirect:
 #endif
     /* We didn't allocate this address. */
-    free(p);
+    PyMem_Free(p);
 }
 
 /* realloc.  If p is NULL, this acts like malloc(nbytes).  Else if nbytes==0,
@@ -1214,9 +1511,8 @@ redirect:
  * return a non-NULL result.
  */
 
-#undef PyObject_Realloc
+static void *
-void *
+_PyObject_Realloc(void *ctx, void *p, size_t nbytes)
-PyObject_Realloc(void *p, size_t nbytes)
 {
     void *bp;
     poolp pool;
@@ -1226,16 +1522,7 @@ PyObject_Realloc(void *p, size_t nbytes)
 #endif
 
     if (p == NULL)
-        return PyObject_Malloc(nbytes);
+        return _PyObject_Malloc(ctx, nbytes);
-
-    /*
-     * Limit ourselves to PY_SSIZE_T_MAX bytes to prevent security holes.
-     * Most python internals blindly use a signed Py_ssize_t to track
-     * things without checking for overflows or negatives.
-     * As size_t is unsigned, checking for nbytes < 0 is not required.
-     */
-    if (nbytes > PY_SSIZE_T_MAX)
-        return NULL;
 
 #ifdef WITH_VALGRIND
     /* Treat running_on_valgrind == -1 the same as 0 */
@@ -1263,10 +1550,10 @@ PyObject_Realloc(void *p, size_t nbytes)
             }
             size = nbytes;
         }
-        bp = PyObject_Malloc(nbytes);
+        bp = _PyObject_Malloc(ctx, nbytes);
         if (bp != NULL) {
             memcpy(bp, p, size);
-            PyObject_Free(p);
+            _PyObject_Free(ctx, p);
         }
         return bp;
     }
@@ -1284,14 +1571,14 @@ PyObject_Realloc(void *p, size_t nbytes)
      * at p.  Instead we punt:  let C continue to manage this block.
      */
     if (nbytes)
-        return realloc(p, nbytes);
+        return PyMem_Realloc(p, nbytes);
     /* C doesn't define the result of realloc(p, 0) (it may or may not
      * return NULL then), but Python's docs promise that nbytes==0 never
      * returns NULL.  We don't pass 0 to realloc(), to avoid that endcase
      * to begin with.  Even then, we can't be sure that realloc() won't
      * return NULL.
      */
-    bp = realloc(p, 1);
+    bp = PyMem_Realloc(p, 1);
     return bp ? bp : p;
 }
 
@@ -1301,24 +1588,6 @@ PyObject_Realloc(void *p, size_t nbytes)
 /* pymalloc not enabled:  Redirect the entry points to malloc.  These will
  * only be used by extensions that are compiled with pymalloc enabled. */
 
-void *
-PyObject_Malloc(size_t n)
-{
-    return PyMem_MALLOC(n);
-}
-
-void *
-PyObject_Realloc(void *p, size_t n)
-{
-    return PyMem_REALLOC(p, n);
-}
-
-void
-PyObject_Free(void *p)
-{
-    PyMem_FREE(p);
-}
-
 Py_ssize_t
 _Py_GetAllocatedBlocks(void)
 {
@@ -1344,10 +1613,6 @@ _Py_GetAllocatedBlocks(void)
 #define DEADBYTE       0xDB    /* dead (newly freed) memory */
 #define FORBIDDENBYTE  0xFB    /* untouchable bytes at each end of a block */
 
-/* We tag each block with an API ID in order to tag API violations */
-#define _PYMALLOC_MEM_ID 'm'   /* the PyMem_Malloc() API */
-#define _PYMALLOC_OBJ_ID 'o'   /* The PyObject_Malloc() API */
-
 static size_t serialno = 0;     /* incremented on each debug {m,re}alloc */
 
 /* serialno is always incremented via calling this routine.  The point is
@@ -1430,58 +1695,18 @@ p[2*S: 2*S+n]
 p[2*S+n: 2*S+n+S]
     Copies of FORBIDDENBYTE.  Used to catch over- writes and reads.
 p[2*S+n+S: 2*S+n+2*S]
-    A serial number, incremented by 1 on each call to _PyObject_DebugMalloc
+    A serial number, incremented by 1 on each call to _PyMem_DebugMalloc
-    and _PyObject_DebugRealloc.
+    and _PyMem_DebugRealloc.
     This is a big-endian size_t.
     If "bad memory" is detected later, the serial number gives an
     excellent way to set a breakpoint on the next run, to capture the
     instant at which this block was passed out.
 */
 
-/* debug replacements for the PyMem_* memory API */
+static void *
-void *
+_PyMem_DebugMalloc(void *ctx, size_t nbytes)
-_PyMem_DebugMalloc(size_t nbytes)
-{
-    return _PyObject_DebugMallocApi(_PYMALLOC_MEM_ID, nbytes);
-}
-void *
-_PyMem_DebugRealloc(void *p, size_t nbytes)
-{
-    return _PyObject_DebugReallocApi(_PYMALLOC_MEM_ID, p, nbytes);
-}
-void
-_PyMem_DebugFree(void *p)
-{
-    _PyObject_DebugFreeApi(_PYMALLOC_MEM_ID, p);
-}
-
-/* debug replacements for the PyObject_* memory API */
-void *
-_PyObject_DebugMalloc(size_t nbytes)
-{
-    return _PyObject_DebugMallocApi(_PYMALLOC_OBJ_ID, nbytes);
-}
-void *
-_PyObject_DebugRealloc(void *p, size_t nbytes)
-{
-    return _PyObject_DebugReallocApi(_PYMALLOC_OBJ_ID, p, nbytes);
-}
-void
-_PyObject_DebugFree(void *p)
-{
-    _PyObject_DebugFreeApi(_PYMALLOC_OBJ_ID, p);
-}
-void
-_PyObject_DebugCheckAddress(const void *p)
-{
-    _PyObject_DebugCheckAddressApi(_PYMALLOC_OBJ_ID, p);
-}
-
-
-/* generic debug memory api, with an "id" to identify the API in use */
-void *
-_PyObject_DebugMallocApi(char id, size_t nbytes)
 {
+    debug_alloc_api_t *api = (debug_alloc_api_t *)ctx;
     uchar *p;           /* base address of malloc'ed block */
     uchar *tail;        /* p + 2*SST + nbytes == pointer to tail pad bytes */
     size_t total;       /* nbytes + 4*SST */
@@ -1492,14 +1717,14 @@ _PyObject_DebugMallocApi(char id, size_t nbytes)
         /* overflow:  can't represent total as a size_t */
         return NULL;
 
-    p = (uchar *)PyObject_Malloc(total);
+    p = (uchar *)api->alloc.malloc(api->alloc.ctx, total);
     if (p == NULL)
         return NULL;
 
     /* at p, write size (SST bytes), id (1 byte), pad (SST-1 bytes) */
     write_size_t(p, nbytes);
-    p[SST] = (uchar)id;
+    p[SST] = (uchar)api->api_id;
-    memset(p + SST + 1 , FORBIDDENBYTE, SST-1);
+    memset(p + SST + 1, FORBIDDENBYTE, SST-1);
 
     if (nbytes > 0)
         memset(p + 2*SST, CLEANBYTE, nbytes);
@@ -1517,25 +1742,27 @@ _PyObject_DebugMallocApi(char id, size_t nbytes)
    Then fills the original bytes with DEADBYTE.
    Then calls the underlying free.
 */
-void
+static void
-_PyObject_DebugFreeApi(char api, void *p)
+_PyMem_DebugFree(void *ctx, void *p)
 {
+    debug_alloc_api_t *api = (debug_alloc_api_t *)ctx;
     uchar *q = (uchar *)p - 2*SST;  /* address returned from malloc */
     size_t nbytes;
 
     if (p == NULL)
         return;
-    _PyObject_DebugCheckAddressApi(api, p);
+    _PyMem_DebugCheckAddress(api->api_id, p);
     nbytes = read_size_t(q);
     nbytes += 4*SST;
     if (nbytes > 0)
         memset(q, DEADBYTE, nbytes);
-    PyObject_Free(q);
+    api->alloc.free(api->alloc.ctx, q);
 }
 
-void *
+static void *
-_PyObject_DebugReallocApi(char api, void *p, size_t nbytes)
+_PyMem_DebugRealloc(void *ctx, void *p, size_t nbytes)
 {
+    debug_alloc_api_t *api = (debug_alloc_api_t *)ctx;
     uchar *q = (uchar *)p;
     uchar *tail;
     size_t total;       /* nbytes + 4*SST */
@@ -1543,9 +1770,9 @@ _PyObject_DebugReallocApi(char api, void *p, size_t nbytes)
     int i;
 
     if (p == NULL)
-        return _PyObject_DebugMallocApi(api, nbytes);
+        return _PyMem_DebugMalloc(ctx, nbytes);
 
-    _PyObject_DebugCheckAddressApi(api, p);
+    _PyMem_DebugCheckAddress(api->api_id, p);
     bumpserialno();
     original_nbytes = read_size_t(q - 2*SST);
     total = nbytes + 4*SST;
@@ -1562,12 +1789,12 @@ _PyObject_DebugReallocApi(char api, void *p, size_t nbytes)
      * case we didn't get the chance to mark the old memory with DEADBYTE,
      * but we live with that.
      */
-    q = (uchar *)PyObject_Realloc(q - 2*SST, total);
+    q = (uchar *)api->alloc.realloc(api->alloc.ctx, q - 2*SST, total);
     if (q == NULL)
         return NULL;
 
     write_size_t(q, nbytes);
-    assert(q[SST] == (uchar)api);
+    assert(q[SST] == (uchar)api->api_id);
     for (i = 1; i < SST; ++i)
         assert(q[SST + i] == FORBIDDENBYTE);
     q += 2*SST;
@@ -1589,8 +1816,8 @@ _PyObject_DebugReallocApi(char api, void *p, size_t nbytes)
  * and call Py_FatalError to kill the program.
  * The API id, is also checked.
  */
- void
+static void
-_PyObject_DebugCheckAddressApi(char api, const void *p)
+_PyMem_DebugCheckAddress(char api, const void *p)
 {
     const uchar *q = (const uchar *)p;
     char msgbuf[64];
@@ -1642,7 +1869,7 @@ error:
 }
 
 /* Display info to stderr about the memory block at p. */
-void
+static void
 _PyObject_DebugDumpAddress(const void *p)
 {
     const uchar *q = (const uchar *)p;