mirror of https://github.com/python/cpython
bpo-43774: Remove unused PYMALLOC_DEBUG macro (GH-25711)
Enhance also the documentation of debug hooks on memory allocators.
This commit is contained in:
Victor Stinner
GitHub
parent
b1f413e6cf
commit
645ed62fb4
|
|
@ -229,17 +229,20 @@ PyPreConfig
|
|||
Name of the Python memory allocators:
|
||||
|
||||
* ``PYMEM_ALLOCATOR_NOT_SET`` (``0``): don't change memory allocators
|
||||
(use defaults)
|
||||
* ``PYMEM_ALLOCATOR_DEFAULT`` (``1``): default memory allocators
|
||||
* ``PYMEM_ALLOCATOR_DEBUG`` (``2``): default memory allocators with
|
||||
debug hooks
|
||||
* ``PYMEM_ALLOCATOR_MALLOC`` (``3``): force usage of ``malloc()``
|
||||
(use defaults).
|
||||
* ``PYMEM_ALLOCATOR_DEFAULT`` (``1``): :ref:`default memory allocators
|
||||
<default-memory-allocators>`.
|
||||
* ``PYMEM_ALLOCATOR_DEBUG`` (``2``): :ref:`default memory allocators
|
||||
<default-memory-allocators>` with :ref:`debug hooks
|
||||
<pymem-debug-hooks>`.
|
||||
* ``PYMEM_ALLOCATOR_MALLOC`` (``3``): use ``malloc()`` of the C library.
|
||||
* ``PYMEM_ALLOCATOR_MALLOC_DEBUG`` (``4``): force usage of
|
||||
``malloc()`` with debug hooks
|
||||
``malloc()`` with :ref:`debug hooks <pymem-debug-hooks>`.
|
||||
* ``PYMEM_ALLOCATOR_PYMALLOC`` (``5``): :ref:`Python pymalloc memory
|
||||
allocator <pymalloc>`
|
||||
allocator <pymalloc>`.
|
||||
* ``PYMEM_ALLOCATOR_PYMALLOC_DEBUG`` (``6``): :ref:`Python pymalloc
|
||||
memory allocator <pymalloc>` with debug hooks
|
||||
memory allocator <pymalloc>` with :ref:`debug hooks
|
||||
<pymem-debug-hooks>`.
|
||||
|
||||
``PYMEM_ALLOCATOR_PYMALLOC`` and ``PYMEM_ALLOCATOR_PYMALLOC_DEBUG`` are
|
||||
not supported if Python is :option:`configured using --without-pymalloc
|
||||
|
|
|
|||
|
|
@ -389,7 +389,8 @@ Legend:
|
|||
* ``malloc``: system allocators from the standard C library, C functions:
|
||||
:c:func:`malloc`, :c:func:`calloc`, :c:func:`realloc` and :c:func:`free`.
|
||||
* ``pymalloc``: :ref:`pymalloc memory allocator <pymalloc>`.
|
||||
* "+ debug": with debug hooks installed by :c:func:`PyMem_SetupDebugHooks`.
|
||||
* "+ debug": with :ref:`debug hooks on the Python memory allocators
|
||||
<pymem-debug-hooks>`.
|
||||
* "Debug build": :ref:`Python build in debug mode <debug-build>`.
|
||||
|
||||
.. _customize-memory-allocators:
|
||||
|
|
@ -478,45 +479,113 @@ Customize Memory Allocators
|
|||
|
||||
.. c:function:: void PyMem_SetupDebugHooks(void)
|
||||
|
||||
Setup hooks to detect bugs in the Python memory allocator functions.
|
||||
Setup :ref:`debug hooks in the Python memory allocators <pymem-debug-hooks>`
|
||||
to detect memory errors.
|
||||
|
||||
Newly allocated memory is filled with the byte ``0xCD`` (``CLEANBYTE``),
|
||||
freed memory is filled with the byte ``0xDD`` (``DEADBYTE``). Memory blocks
|
||||
are surrounded by "forbidden bytes" (``FORBIDDENBYTE``: byte ``0xFD``).
|
||||
|
||||
Runtime checks:
|
||||
.. _pymem-debug-hooks:
|
||||
|
||||
- Detect API violations, ex: :c:func:`PyObject_Free` called on a buffer
|
||||
allocated by :c:func:`PyMem_Malloc`
|
||||
- Detect write before the start of the buffer (buffer underflow)
|
||||
- Detect write after the end of the buffer (buffer overflow)
|
||||
- Check that the :term:`GIL <global interpreter lock>` is held when
|
||||
allocator functions of :c:data:`PYMEM_DOMAIN_OBJ` (ex:
|
||||
:c:func:`PyObject_Malloc`) and :c:data:`PYMEM_DOMAIN_MEM` (ex:
|
||||
:c:func:`PyMem_Malloc`) domains are called
|
||||
Debug hooks on the Python memory allocators
|
||||
===========================================
|
||||
|
||||
On error, the debug hooks use the :mod:`tracemalloc` module to get the
|
||||
traceback where a memory block was allocated. The traceback is only
|
||||
displayed if :mod:`tracemalloc` is tracing Python memory allocations and the
|
||||
memory block was traced.
|
||||
When :ref:`Python is built is debug mode <debug-build>`, the
|
||||
:c:func:`PyMem_SetupDebugHooks` function is called at the :ref:`Python
|
||||
preinitialization <c-preinit>` to setup debug hooks on Python memory allocators
|
||||
to detect memory errors.
|
||||
|
||||
These hooks are :ref:`installed by default <default-memory-allocators>` if
|
||||
:ref:`Python is built in debug mode <debug-build>`.
|
||||
The :envvar:`PYTHONMALLOC` environment variable can be used to install
|
||||
debug hooks on a Python compiled in release mode.
|
||||
The :envvar:`PYTHONMALLOC` environment variable can be used to install debug
|
||||
hooks on a Python compiled in release mode (ex: ``PYTHONMALLOC=debug``).
|
||||
|
||||
.. versionchanged:: 3.6
|
||||
This function now also works on Python compiled in release mode.
|
||||
On error, the debug hooks now use :mod:`tracemalloc` to get the traceback
|
||||
where a memory block was allocated. The debug hooks now also check
|
||||
if the GIL is held when functions of :c:data:`PYMEM_DOMAIN_OBJ` and
|
||||
:c:data:`PYMEM_DOMAIN_MEM` domains are called.
|
||||
The :c:func:`PyMem_SetupDebugHooks` function can be used to set debug hooks
|
||||
after calling :c:func:`PyMem_SetAllocator`.
|
||||
|
||||
.. versionchanged:: 3.8
|
||||
Byte patterns ``0xCB`` (``CLEANBYTE``), ``0xDB`` (``DEADBYTE``) and
|
||||
``0xFB`` (``FORBIDDENBYTE``) have been replaced with ``0xCD``, ``0xDD``
|
||||
and ``0xFD`` to use the same values than Windows CRT debug ``malloc()``
|
||||
and ``free()``.
|
||||
These debug hooks fill dynamically allocated memory blocks with special,
|
||||
recognizable bit patterns. Newly allocated memory is filled with the byte
|
||||
``0xCD`` (``PYMEM_CLEANBYTE``), freed memory is filled with the byte ``0xDD``
|
||||
(``PYMEM_DEADBYTE``). Memory blocks are surrounded by "forbidden bytes"
|
||||
filled with the byte ``0xFD`` (``PYMEM_FORBIDDENBYTE``). Strings of these bytes
|
||||
are unlikely to be valid addresses, floats, or ASCII strings.
|
||||
|
||||
Runtime checks:
|
||||
|
||||
- Detect API violations. For example, detect if :c:func:`PyObject_Free` is
|
||||
called on a memory block allocated by :c:func:`PyMem_Malloc`.
|
||||
- Detect write before the start of the buffer (buffer underflow).
|
||||
- Detect write after the end of the buffer (buffer overflow).
|
||||
- Check that the :term:`GIL <global interpreter lock>` is held when
|
||||
allocator functions of :c:data:`PYMEM_DOMAIN_OBJ` (ex:
|
||||
:c:func:`PyObject_Malloc`) and :c:data:`PYMEM_DOMAIN_MEM` (ex:
|
||||
:c:func:`PyMem_Malloc`) domains are called.
|
||||
|
||||
On error, the debug hooks use the :mod:`tracemalloc` module to get the
|
||||
traceback where a memory block was allocated. The traceback is only displayed
|
||||
if :mod:`tracemalloc` is tracing Python memory allocations and the memory block
|
||||
was traced.
|
||||
|
||||
Let *S* = ``sizeof(size_t)``. ``2*S`` bytes are added at each end of each block
|
||||
of *N* bytes requested. The memory layout is like so, where p represents the
|
||||
address returned by a malloc-like or realloc-like function (``p[i:j]`` means
|
||||
the slice of bytes from ``*(p+i)`` inclusive up to ``*(p+j)`` exclusive; note
|
||||
that the treatment of negative indices differs from a Python slice):
|
||||
|
||||
``p[-2*S:-S]``
|
||||
Number of bytes originally asked for. This is a size_t, big-endian (easier
|
||||
to read in a memory dump).
|
||||
``p[-S]``
|
||||
API identifier (ASCII character):
|
||||
|
||||
* ``'r'`` for :c:data:`PYMEM_DOMAIN_RAW`.
|
||||
* ``'m'`` for :c:data:`PYMEM_DOMAIN_MEM`.
|
||||
* ``'o'`` for :c:data:`PYMEM_DOMAIN_OBJ`.
|
||||
|
||||
``p[-S+1:0]``
|
||||
Copies of PYMEM_FORBIDDENBYTE. Used to catch under- writes and reads.
|
||||
|
||||
``p[0:N]``
|
||||
The requested memory, filled with copies of PYMEM_CLEANBYTE, used to catch
|
||||
reference to uninitialized memory. When a realloc-like function is called
|
||||
requesting a larger memory block, the new excess bytes are also filled with
|
||||
PYMEM_CLEANBYTE. When a free-like function is called, these are
|
||||
overwritten with PYMEM_DEADBYTE, to catch reference to freed memory. When
|
||||
a realloc- like function is called requesting a smaller memory block, the
|
||||
excess old bytes are also filled with PYMEM_DEADBYTE.
|
||||
|
||||
``p[N:N+S]``
|
||||
Copies of PYMEM_FORBIDDENBYTE. Used to catch over- writes and reads.
|
||||
|
||||
``p[N+S:N+2*S]``
|
||||
Only used if the ``PYMEM_DEBUG_SERIALNO`` macro is defined (not defined by
|
||||
default).
|
||||
|
||||
A serial number, incremented by 1 on each call to a malloc-like or
|
||||
realloc-like function. Big-endian ``size_t``. If "bad memory" is detected
|
||||
later, the serial number gives an excellent way to set a breakpoint on the
|
||||
next run, to capture the instant at which this block was passed out. The
|
||||
static function bumpserialno() in obmalloc.c is the only place the serial
|
||||
number is incremented, and exists so you can set such a breakpoint easily.
|
||||
|
||||
A realloc-like or free-like function first checks that the PYMEM_FORBIDDENBYTE
|
||||
bytes at each end are intact. If they've been altered, diagnostic output is
|
||||
written to stderr, and the program is aborted via Py_FatalError(). The other
|
||||
main failure mode is provoking a memory error when a program reads up one of
|
||||
the special bit patterns and tries to use it as an address. If you get in a
|
||||
debugger then and look at the object, you're likely to see that it's entirely
|
||||
filled with PYMEM_DEADBYTE (meaning freed memory is getting used) or
|
||||
PYMEM_CLEANBYTE (meaning uninitialized memory is getting used).
|
||||
|
||||
.. versionchanged:: 3.6
|
||||
The :c:func:`PyMem_SetupDebugHooks` function now also works on Python
|
||||
compiled in release mode. On error, the debug hooks now use
|
||||
:mod:`tracemalloc` to get the traceback where a memory block was allocated.
|
||||
The debug hooks now also check if the GIL is held when functions of
|
||||
:c:data:`PYMEM_DOMAIN_OBJ` and :c:data:`PYMEM_DOMAIN_MEM` domains are
|
||||
called.
|
||||
|
||||
.. versionchanged:: 3.8
|
||||
Byte patterns ``0xCB`` (``PYMEM_CLEANBYTE``), ``0xDB`` (``PYMEM_DEADBYTE``)
|
||||
and ``0xFB`` (``PYMEM_FORBIDDENBYTE``) have been replaced with ``0xCD``,
|
||||
``0xDD`` and ``0xFD`` to use the same values than Windows CRT debug
|
||||
``malloc()`` and ``free()``.
|
||||
|
||||
|
||||
.. _pymalloc:
|
||||
|
|
@ -539,6 +608,10 @@ The arena allocator uses the following functions:
|
|||
* :c:func:`mmap` and :c:func:`munmap` if available,
|
||||
* :c:func:`malloc` and :c:func:`free` otherwise.
|
||||
|
||||
This allocator is disabled if Python is configured with the
|
||||
:option:`--without-pymalloc` option. It can also be disabled at runtime using
|
||||
the :envvar:`PYTHONMALLOC` environment variable (ex: ``PYTHONMALLOC=malloc``).
|
||||
|
||||
Customize pymalloc Arena Allocator
|
||||
----------------------------------
|
||||
|
||||
|
|
|
|||
|
|
@ -799,17 +799,13 @@ conflict.
|
|||
:c:data:`PYMEM_DOMAIN_MEM` and :c:data:`PYMEM_DOMAIN_OBJ` domains and use
|
||||
the :c:func:`malloc` function for the :c:data:`PYMEM_DOMAIN_RAW` domain.
|
||||
|
||||
Install debug hooks:
|
||||
Install :ref:`debug hooks <pymem-debug-hooks>`:
|
||||
|
||||
* ``debug``: install debug hooks on top of the :ref:`default memory
|
||||
allocators <default-memory-allocators>`.
|
||||
* ``malloc_debug``: same as ``malloc`` but also install debug hooks.
|
||||
* ``pymalloc_debug``: same as ``pymalloc`` but also install debug hooks.
|
||||
|
||||
See the :ref:`default memory allocators <default-memory-allocators>` and the
|
||||
:c:func:`PyMem_SetupDebugHooks` function (install debug hooks on Python
|
||||
memory allocators).
|
||||
|
||||
.. versionchanged:: 3.7
|
||||
Added the ``"default"`` allocator.
|
||||
|
||||
|
|
|
|||
|
|
@ -72,15 +72,6 @@
|
|||
# endif
|
||||
#endif
|
||||
|
||||
/* Debug-mode build with pymalloc implies PYMALLOC_DEBUG.
|
||||
* PYMALLOC_DEBUG is in error if pymalloc is not in use.
|
||||
*/
|
||||
#if defined(Py_DEBUG) && defined(WITH_PYMALLOC) && !defined(PYMALLOC_DEBUG)
|
||||
#define PYMALLOC_DEBUG
|
||||
#endif
|
||||
#if defined(PYMALLOC_DEBUG) && !defined(WITH_PYMALLOC)
|
||||
#error "PYMALLOC_DEBUG requires WITH_PYMALLOC"
|
||||
#endif
|
||||
#include "pymath.h"
|
||||
#include "pymem.h"
|
||||
|
||||
|
|
|
|||
|
|
@ -48,8 +48,8 @@ Functions and macros for modules that implement new object types.
|
|||
|
||||
Note that objects created with PyObject_{New, NewVar} are allocated using the
|
||||
specialized Python allocator (implemented in obmalloc.c), if WITH_PYMALLOC is
|
||||
enabled. In addition, a special debugging allocator is used if PYMALLOC_DEBUG
|
||||
is also #defined.
|
||||
enabled. In addition, a special debugging allocator is used if Py_DEBUG
|
||||
macro is also defined.
|
||||
|
||||
In case a specific form of memory management is needed (for example, if you
|
||||
must use the platform malloc heap(s), or shared memory, or C++ local storage or
|
||||
|
|
|
|||
|
|
@ -25,8 +25,8 @@ extern "C" {
|
|||
heap used by the Python DLL; it could be a disaster if you free()'ed that
|
||||
directly in your own extension. Using PyMem_Free instead ensures Python
|
||||
can return the memory to the proper heap. As another example, in
|
||||
PYMALLOC_DEBUG mode, Python wraps all calls to all PyMem_ and PyObject_
|
||||
memory functions in special debugging wrappers that add additional
|
||||
a debug build (Py_DEBUG macro), Python wraps all calls to all PyMem_ and
|
||||
PyObject_ memory functions in special debugging wrappers that add additional
|
||||
debugging info to dynamic memory blocks. The system routines have no idea
|
||||
what to do with that stuff, and the Python wrappers have no idea what to do
|
||||
with raw blocks obtained directly by the system routines then.
|
||||
|
|
|
|||
|
|
@ -0,0 +1,5 @@
|
|||
Remove the now unused ``PYMALLOC_DEBUG`` macro. Debug hooks on memory
|
||||
allocators are now installed by default if Python is built in debug mode (if
|
||||
``Py_DEBUG`` macro is defined). Moreover, they can now be used on Python
|
||||
build in release mode (ex: using ``PYTHONMALLOC=debug`` environment
|
||||
variable).
|
||||
|
|
@ -77,90 +77,14 @@ envvar PYTHONDUMPREFS
|
|||
combinerefs.py, were new in Python 2.3b1.
|
||||
|
||||
|
||||
PYMALLOC_DEBUG
|
||||
--------------
|
||||
|
||||
When pymalloc is enabled (WITH_PYMALLOC is defined), calls to the PyObject_
|
||||
memory routines are handled by Python's own small-object allocator, while calls
|
||||
to the PyMem_ memory routines are directed to the system malloc/ realloc/free.
|
||||
If PYMALLOC_DEBUG is also defined, calls to both PyObject_ and PyMem_ memory
|
||||
routines are directed to a special debugging mode of Python's small-object
|
||||
allocator.
|
||||
|
||||
This mode fills dynamically allocated memory blocks with special, recognizable
|
||||
bit patterns, and adds debugging info on each end of dynamically allocated
|
||||
memory blocks. The special bit patterns are:
|
||||
|
||||
#define CLEANBYTE 0xCB /* clean (newly allocated) memory */
|
||||
#define DEADBYTE 0xDB /* dead (newly freed) memory */
|
||||
#define FORBIDDENBYTE 0xFB /* forbidden -- untouchable bytes */
|
||||
|
||||
Strings of these bytes are unlikely to be valid addresses, floats, or 7-bit
|
||||
ASCII strings.
|
||||
|
||||
Let S = sizeof(size_t). 2*S bytes are added at each end of each block of N bytes
|
||||
requested. The memory layout is like so, where p represents the address
|
||||
returned by a malloc-like or realloc-like function (p[i:j] means the slice of
|
||||
bytes from *(p+i) inclusive up to *(p+j) exclusive; note that the treatment of
|
||||
negative indices differs from a Python slice):
|
||||
|
||||
p[-2*S:-S]
|
||||
Number of bytes originally asked for. This is a size_t, big-endian (easier
|
||||
to read in a memory dump).
|
||||
p[-S]
|
||||
API ID. See PEP 445. This is a character, but seems undocumented.
|
||||
p[-S+1:0]
|
||||
Copies of FORBIDDENBYTE. Used to catch under- writes and reads.
|
||||
p[0:N]
|
||||
The requested memory, filled with copies of CLEANBYTE, used to catch
|
||||
reference to uninitialized memory. When a realloc-like function is called
|
||||
requesting a larger memory block, the new excess bytes are also filled with
|
||||
CLEANBYTE. When a free-like function is called, these are overwritten with
|
||||
DEADBYTE, to catch reference to freed memory. When a realloc- like function
|
||||
is called requesting a smaller memory block, the excess old bytes are also
|
||||
filled with DEADBYTE.
|
||||
p[N:N+S]
|
||||
Copies of FORBIDDENBYTE. Used to catch over- writes and reads.
|
||||
p[N+S:N+2*S]
|
||||
A serial number, incremented by 1 on each call to a malloc-like or
|
||||
realloc-like function. Big-endian size_t. If "bad memory" is detected
|
||||
later, the serial number gives an excellent way to set a breakpoint on the
|
||||
next run, to capture the instant at which this block was passed out. The
|
||||
static function bumpserialno() in obmalloc.c is the only place the serial
|
||||
number is incremented, and exists so you can set such a breakpoint easily.
|
||||
|
||||
A realloc-like or free-like function first checks that the FORBIDDENBYTEs at
|
||||
each end are intact. If they've been altered, diagnostic output is written to
|
||||
stderr, and the program is aborted via Py_FatalError(). The other main failure
|
||||
mode is provoking a memory error when a program reads up one of the special bit
|
||||
patterns and tries to use it as an address. If you get in a debugger then and
|
||||
look at the object, you're likely to see that it's entirely filled with 0xDB
|
||||
(meaning freed memory is getting used) or 0xCB (meaning uninitialized memory is
|
||||
getting used).
|
||||
|
||||
Note that PYMALLOC_DEBUG requires WITH_PYMALLOC. Py_DEBUG implies
|
||||
PYMALLOC_DEBUG (if WITH_PYMALLOC is enabled).
|
||||
|
||||
Special gimmicks:
|
||||
|
||||
envvar PYTHONMALLOCSTATS
|
||||
If this envvar exists, a report of pymalloc summary statistics is printed to
|
||||
stderr whenever a new arena is allocated, and also by Py_FinalizeEx().
|
||||
|
||||
Changed in 2.5: The number of extra bytes allocated is 4*sizeof(size_t).
|
||||
Before it was 16 on all boxes, reflecting that Python couldn't make use of
|
||||
allocations >= 2**32 bytes even on 64-bit boxes before 2.5.
|
||||
|
||||
|
||||
Py_DEBUG
|
||||
--------
|
||||
|
||||
This is what is generally meant by "a debug build" of Python.
|
||||
|
||||
Py_DEBUG implies LLTRACE, Py_REF_DEBUG, and PYMALLOC_DEBUG (if
|
||||
WITH_PYMALLOC is enabled). In addition, C assert()s are enabled (via the C way:
|
||||
by not defining NDEBUG), and some routines do additional sanity checks inside
|
||||
"#ifdef Py_DEBUG" blocks.
|
||||
Py_DEBUG implies LLTRACE and Py_REF_DEBUG. In addition, C assert()s are enabled
|
||||
(via the C way: by not defining NDEBUG), and some routines do additional sanity
|
||||
checks inside "#ifdef Py_DEBUG" blocks.
|
||||
|
||||
|
||||
LLTRACE
|
||||
|
|
|
|||
Loading…
Reference in New Issue