2002-04-28 01:11:46 -03:00
|
|
|
/* The PyMem_ family: low-level memory allocation interfaces.
|
|
|
|
|
See objimpl.h for the PyObject_ memory family.
|
|
|
|
|
*/
|
2000-07-31 19:19:30 -03:00
|
|
|
|
|
|
|
|
#ifndef Py_PYMEM_H
|
|
|
|
|
#define Py_PYMEM_H
|
|
|
|
|
|
|
|
|
|
#include "pyport.h"
|
|
|
|
|
|
|
|
|
|
#ifdef __cplusplus
|
|
|
|
|
extern "C" {
|
|
|
|
|
#endif
|
|
|
|
|
|
|
|
|
|
/* BEWARE:
|
|
|
|
|
|
2002-04-28 01:11:46 -03:00
|
|
|
Each interface exports both functions and macros. Extension modules should
|
|
|
|
|
use the functions, to ensure binary compatibility across Python versions.
|
|
|
|
|
Because the Python implementation is free to change internal details, and
|
|
|
|
|
the macros may (or may not) expose details for speed, if you do use the
|
|
|
|
|
macros you must recompile your extensions with each Python release.
|
|
|
|
|
|
|
|
|
|
Never mix calls to PyMem_ with calls to the platform malloc/realloc/
|
|
|
|
|
calloc/free. For example, on Windows different DLLs may end up using
|
|
|
|
|
different heaps, and if you use PyMem_Malloc you'll get the memory from the
|
|
|
|
|
heap used by the Python DLL; it could be a disaster if you free()'ed that
|
|
|
|
|
directly in your own extension. Using PyMem_Free instead ensures Python
|
|
|
|
|
can return the memory to the proper heap. As another example, in
|
|
|
|
|
PYMALLOC_DEBUG mode, Python wraps all calls to all PyMem_ and PyObject_
|
|
|
|
|
memory functions in special debugging wrappers that add additional
|
|
|
|
|
debugging info to dynamic memory blocks. The system routines have no idea
|
|
|
|
|
what to do with that stuff, and the Python wrappers have no idea what to do
|
|
|
|
|
with raw blocks obtained directly by the system routines then.
|
2007-05-02 01:47:55 -03:00
|
|
|
|
|
|
|
|
The GIL must be held when using these APIs.
|
2002-04-28 01:11:46 -03:00
|
|
|
*/
|
2000-07-31 19:19:30 -03:00
|
|
|
|
|
|
|
|
/*
|
|
|
|
|
* Raw memory interface
|
|
|
|
|
* ====================
|
|
|
|
|
*/
|
|
|
|
|
|
2002-04-28 01:11:46 -03:00
|
|
|
/* Functions
|
2000-07-31 19:19:30 -03:00
|
|
|
|
2002-04-28 01:11:46 -03:00
|
|
|
Functions supplying platform-independent semantics for malloc/realloc/
|
|
|
|
|
free. These functions make sure that allocating 0 bytes returns a distinct
|
First stab at rationalizing the PyMem_ API. Mixing PyObject_xyz with
PyMem_{Del, DEL} doesn't work yet (compilation problems).
pyport.h: _PyMem_EXTRA is gone.
pmem.h: Repaired comments. PyMem_{Malloc, MALLOC} and
PyMem_{Realloc, REALLOC} now make the same x-platform guarantees when
asking for 0 bytes, and when passing a NULL pointer to the latter.
object.c: PyMem_{Malloc, Realloc} just call their macro versions
now, since the latter take care of the x-platform 0 and NULL stuff
by themselves now.
pypcre.c, grow_stack(): So sue me. On two lines, this called
PyMem_RESIZE to grow a "const" area. It's not legit to realloc a
const area, so the compiler warned given the new expansion of
PyMem_RESIZE. It would have gotten the same warning before if it
had used PyMem_Resize() instead; the older macro version, but not the
function version, silently cast away the constness. IMO that was a wrong
thing to do, and the docs say the macro versions of PyMem_xyz are
deprecated anyway. If somebody else is resizing const areas with the
macro spelling, they'll get a warning when they recompile now too.
2002-04-12 04:22:56 -03:00
|
|
|
non-NULL pointer (whenever possible -- if we're flat out of memory, NULL
|
|
|
|
|
may be returned), even if the platform malloc and realloc don't.
|
|
|
|
|
Returned pointers must be checked for NULL explicitly. No action is
|
2002-04-28 01:11:46 -03:00
|
|
|
performed on failure (no exception is set, no warning is printed, etc).
|
|
|
|
|
*/
|
First stab at rationalizing the PyMem_ API. Mixing PyObject_xyz with
PyMem_{Del, DEL} doesn't work yet (compilation problems).
pyport.h: _PyMem_EXTRA is gone.
pmem.h: Repaired comments. PyMem_{Malloc, MALLOC} and
PyMem_{Realloc, REALLOC} now make the same x-platform guarantees when
asking for 0 bytes, and when passing a NULL pointer to the latter.
object.c: PyMem_{Malloc, Realloc} just call their macro versions
now, since the latter take care of the x-platform 0 and NULL stuff
by themselves now.
pypcre.c, grow_stack(): So sue me. On two lines, this called
PyMem_RESIZE to grow a "const" area. It's not legit to realloc a
const area, so the compiler warned given the new expansion of
PyMem_RESIZE. It would have gotten the same warning before if it
had used PyMem_Resize() instead; the older macro version, but not the
function version, silently cast away the constness. IMO that was a wrong
thing to do, and the docs say the macro versions of PyMem_xyz are
deprecated anyway. If somebody else is resizing const areas with the
macro spelling, they'll get a warning when they recompile now too.
2002-04-12 04:22:56 -03:00
|
|
|
|
2002-08-12 04:21:58 -03:00
|
|
|
PyAPI_FUNC(void *) PyMem_Malloc(size_t);
|
|
|
|
|
PyAPI_FUNC(void *) PyMem_Realloc(void *, size_t);
|
|
|
|
|
PyAPI_FUNC(void) PyMem_Free(void *);
|
2000-07-31 19:19:30 -03:00
|
|
|
|
|
|
|
|
/* Starting from Python 1.6, the wrappers Py_{Malloc,Realloc,Free} are
|
|
|
|
|
no longer supported. They used to call PyErr_NoMemory() on failure. */
|
|
|
|
|
|
First stab at rationalizing the PyMem_ API. Mixing PyObject_xyz with
PyMem_{Del, DEL} doesn't work yet (compilation problems).
pyport.h: _PyMem_EXTRA is gone.
pmem.h: Repaired comments. PyMem_{Malloc, MALLOC} and
PyMem_{Realloc, REALLOC} now make the same x-platform guarantees when
asking for 0 bytes, and when passing a NULL pointer to the latter.
object.c: PyMem_{Malloc, Realloc} just call their macro versions
now, since the latter take care of the x-platform 0 and NULL stuff
by themselves now.
pypcre.c, grow_stack(): So sue me. On two lines, this called
PyMem_RESIZE to grow a "const" area. It's not legit to realloc a
const area, so the compiler warned given the new expansion of
PyMem_RESIZE. It would have gotten the same warning before if it
had used PyMem_Resize() instead; the older macro version, but not the
function version, silently cast away the constness. IMO that was a wrong
thing to do, and the docs say the macro versions of PyMem_xyz are
deprecated anyway. If somebody else is resizing const areas with the
macro spelling, they'll get a warning when they recompile now too.
2002-04-12 04:22:56 -03:00
|
|
|
/* Macros. */
|
2002-04-21 23:33:27 -03:00
|
|
|
#ifdef PYMALLOC_DEBUG
|
|
|
|
|
/* Redirect all memory operations to Python's debugging allocator. */
|
|
|
|
|
#define PyMem_MALLOC PyObject_MALLOC
|
|
|
|
|
#define PyMem_REALLOC PyObject_REALLOC
|
Years in the making.
objimpl.h, pymem.h: Stop mapping PyMem_{Del, DEL} and PyMem_{Free, FREE}
to PyObject_{Free, FREE} in a release build. They're aliases for the
system free() now.
_subprocess.c/sp_handle_dealloc(): Since the memory was originally
obtained via PyObject_NEW, it must be released via PyObject_FREE (or
_DEL).
pythonrun.c, tokenizer.c, parsermodule.c: I lost count of the number of
PyObject vs PyMem mismatches in these -- it's like the specific
function called at each site was picked at random, sometimes even with
memory obtained via PyMem getting released via PyObject. Changed most
to use PyObject uniformly, since the blobs allocated are predictably
small in most cases, and obmalloc is generally faster than system
mallocs then.
If extension modules in real life prove as sloppy as Python's front
end, we'll have to revert the objimpl.h + pymem.h part of this patch.
Note that no problems will show up in a debug build (all calls still go
thru obmalloc then). Problems will show up only in a release build, most
likely segfaults.
2006-03-26 19:27:58 -04:00
|
|
|
#define PyMem_FREE PyObject_FREE
|
2002-04-21 23:33:27 -03:00
|
|
|
|
|
|
|
|
#else /* ! PYMALLOC_DEBUG */
|
|
|
|
|
|
2002-11-23 05:13:40 -04:00
|
|
|
/* PyMem_MALLOC(0) means malloc(1). Some systems would return NULL
|
|
|
|
|
for malloc(0), which would be treated as an error. Some platforms
|
|
|
|
|
would return a pointer with no memory behind it, which would break
|
|
|
|
|
pymalloc. To solve these problems, allocate an extra byte. */
|
2008-07-22 01:46:32 -03:00
|
|
|
/* Returns NULL to indicate error if a negative size or size larger than
|
|
|
|
|
Py_ssize_t can represent is supplied. Helps prevents security holes. */
|
|
|
|
|
#define PyMem_MALLOC(n) (((n) < 0 || (n) > PY_SSIZE_T_MAX) ? NULL \
|
|
|
|
|
: malloc((n) ? (n) : 1))
|
|
|
|
|
#define PyMem_REALLOC(p, n) (((n) < 0 || (n) > PY_SSIZE_T_MAX) ? NULL \
|
|
|
|
|
: realloc((p), (n) ? (n) : 1))
|
Years in the making.
objimpl.h, pymem.h: Stop mapping PyMem_{Del, DEL} and PyMem_{Free, FREE}
to PyObject_{Free, FREE} in a release build. They're aliases for the
system free() now.
_subprocess.c/sp_handle_dealloc(): Since the memory was originally
obtained via PyObject_NEW, it must be released via PyObject_FREE (or
_DEL).
pythonrun.c, tokenizer.c, parsermodule.c: I lost count of the number of
PyObject vs PyMem mismatches in these -- it's like the specific
function called at each site was picked at random, sometimes even with
memory obtained via PyMem getting released via PyObject. Changed most
to use PyObject uniformly, since the blobs allocated are predictably
small in most cases, and obmalloc is generally faster than system
mallocs then.
If extension modules in real life prove as sloppy as Python's front
end, we'll have to revert the objimpl.h + pymem.h part of this patch.
Note that no problems will show up in a debug build (all calls still go
thru obmalloc then). Problems will show up only in a release build, most
likely segfaults.
2006-03-26 19:27:58 -04:00
|
|
|
#define PyMem_FREE free
|
First stab at rationalizing the PyMem_ API. Mixing PyObject_xyz with
PyMem_{Del, DEL} doesn't work yet (compilation problems).
pyport.h: _PyMem_EXTRA is gone.
pmem.h: Repaired comments. PyMem_{Malloc, MALLOC} and
PyMem_{Realloc, REALLOC} now make the same x-platform guarantees when
asking for 0 bytes, and when passing a NULL pointer to the latter.
object.c: PyMem_{Malloc, Realloc} just call their macro versions
now, since the latter take care of the x-platform 0 and NULL stuff
by themselves now.
pypcre.c, grow_stack(): So sue me. On two lines, this called
PyMem_RESIZE to grow a "const" area. It's not legit to realloc a
const area, so the compiler warned given the new expansion of
PyMem_RESIZE. It would have gotten the same warning before if it
had used PyMem_Resize() instead; the older macro version, but not the
function version, silently cast away the constness. IMO that was a wrong
thing to do, and the docs say the macro versions of PyMem_xyz are
deprecated anyway. If somebody else is resizing const areas with the
macro spelling, they'll get a warning when they recompile now too.
2002-04-12 04:22:56 -03:00
|
|
|
|
2002-04-21 23:33:27 -03:00
|
|
|
#endif /* PYMALLOC_DEBUG */
|
First stab at rationalizing the PyMem_ API. Mixing PyObject_xyz with
PyMem_{Del, DEL} doesn't work yet (compilation problems).
pyport.h: _PyMem_EXTRA is gone.
pmem.h: Repaired comments. PyMem_{Malloc, MALLOC} and
PyMem_{Realloc, REALLOC} now make the same x-platform guarantees when
asking for 0 bytes, and when passing a NULL pointer to the latter.
object.c: PyMem_{Malloc, Realloc} just call their macro versions
now, since the latter take care of the x-platform 0 and NULL stuff
by themselves now.
pypcre.c, grow_stack(): So sue me. On two lines, this called
PyMem_RESIZE to grow a "const" area. It's not legit to realloc a
const area, so the compiler warned given the new expansion of
PyMem_RESIZE. It would have gotten the same warning before if it
had used PyMem_Resize() instead; the older macro version, but not the
function version, silently cast away the constness. IMO that was a wrong
thing to do, and the docs say the macro versions of PyMem_xyz are
deprecated anyway. If somebody else is resizing const areas with the
macro spelling, they'll get a warning when they recompile now too.
2002-04-12 04:22:56 -03:00
|
|
|
|
2000-07-31 19:19:30 -03:00
|
|
|
/*
|
|
|
|
|
* Type-oriented memory interface
|
|
|
|
|
* ==============================
|
First stab at rationalizing the PyMem_ API. Mixing PyObject_xyz with
PyMem_{Del, DEL} doesn't work yet (compilation problems).
pyport.h: _PyMem_EXTRA is gone.
pmem.h: Repaired comments. PyMem_{Malloc, MALLOC} and
PyMem_{Realloc, REALLOC} now make the same x-platform guarantees when
asking for 0 bytes, and when passing a NULL pointer to the latter.
object.c: PyMem_{Malloc, Realloc} just call their macro versions
now, since the latter take care of the x-platform 0 and NULL stuff
by themselves now.
pypcre.c, grow_stack(): So sue me. On two lines, this called
PyMem_RESIZE to grow a "const" area. It's not legit to realloc a
const area, so the compiler warned given the new expansion of
PyMem_RESIZE. It would have gotten the same warning before if it
had used PyMem_Resize() instead; the older macro version, but not the
function version, silently cast away the constness. IMO that was a wrong
thing to do, and the docs say the macro versions of PyMem_xyz are
deprecated anyway. If somebody else is resizing const areas with the
macro spelling, they'll get a warning when they recompile now too.
2002-04-12 04:22:56 -03:00
|
|
|
*
|
2008-07-22 01:46:32 -03:00
|
|
|
* Allocate memory for n objects of the given type. Returns a new pointer
|
|
|
|
|
* or NULL if the request was too large or memory allocation failed. Use
|
|
|
|
|
* these macros rather than doing the multiplication yourself so that proper
|
|
|
|
|
* overflow checking is always done.
|
2000-07-31 19:19:30 -03:00
|
|
|
*/
|
|
|
|
|
|
|
|
|
|
#define PyMem_New(type, n) \
|
2010-02-14 10:00:19 -04:00
|
|
|
( ((size_t)(n) > PY_SSIZE_T_MAX / sizeof(type)) ? NULL : \
|
2008-06-11 04:41:16 -03:00
|
|
|
( (type *) PyMem_Malloc((n) * sizeof(type)) ) )
|
2000-07-31 19:19:30 -03:00
|
|
|
#define PyMem_NEW(type, n) \
|
2010-02-14 10:00:19 -04:00
|
|
|
( ((size_t)(n) > PY_SSIZE_T_MAX / sizeof(type)) ? NULL : \
|
2008-06-11 04:41:16 -03:00
|
|
|
( (type *) PyMem_MALLOC((n) * sizeof(type)) ) )
|
2002-04-28 01:11:46 -03:00
|
|
|
|
2008-07-22 01:46:32 -03:00
|
|
|
/*
|
|
|
|
|
* The value of (p) is always clobbered by this macro regardless of success.
|
|
|
|
|
* The caller MUST check if (p) is NULL afterwards and deal with the memory
|
|
|
|
|
* error if so. This means the original value of (p) MUST be saved for the
|
|
|
|
|
* caller's memory error handler to not lose track of it.
|
|
|
|
|
*/
|
2002-04-28 01:11:46 -03:00
|
|
|
#define PyMem_Resize(p, type, n) \
|
2010-02-14 10:00:19 -04:00
|
|
|
( (p) = ((size_t)(n) > PY_SSIZE_T_MAX / sizeof(type)) ? NULL : \
|
2008-07-22 01:46:32 -03:00
|
|
|
(type *) PyMem_Realloc((p), (n) * sizeof(type)) )
|
First stab at rationalizing the PyMem_ API. Mixing PyObject_xyz with
PyMem_{Del, DEL} doesn't work yet (compilation problems).
pyport.h: _PyMem_EXTRA is gone.
pmem.h: Repaired comments. PyMem_{Malloc, MALLOC} and
PyMem_{Realloc, REALLOC} now make the same x-platform guarantees when
asking for 0 bytes, and when passing a NULL pointer to the latter.
object.c: PyMem_{Malloc, Realloc} just call their macro versions
now, since the latter take care of the x-platform 0 and NULL stuff
by themselves now.
pypcre.c, grow_stack(): So sue me. On two lines, this called
PyMem_RESIZE to grow a "const" area. It's not legit to realloc a
const area, so the compiler warned given the new expansion of
PyMem_RESIZE. It would have gotten the same warning before if it
had used PyMem_Resize() instead; the older macro version, but not the
function version, silently cast away the constness. IMO that was a wrong
thing to do, and the docs say the macro versions of PyMem_xyz are
deprecated anyway. If somebody else is resizing const areas with the
macro spelling, they'll get a warning when they recompile now too.
2002-04-12 04:22:56 -03:00
|
|
|
#define PyMem_RESIZE(p, type, n) \
|
2010-02-14 10:00:19 -04:00
|
|
|
( (p) = ((size_t)(n) > PY_SSIZE_T_MAX / sizeof(type)) ? NULL : \
|
2008-07-22 01:46:32 -03:00
|
|
|
(type *) PyMem_REALLOC((p), (n) * sizeof(type)) )
|
First stab at rationalizing the PyMem_ API. Mixing PyObject_xyz with
PyMem_{Del, DEL} doesn't work yet (compilation problems).
pyport.h: _PyMem_EXTRA is gone.
pmem.h: Repaired comments. PyMem_{Malloc, MALLOC} and
PyMem_{Realloc, REALLOC} now make the same x-platform guarantees when
asking for 0 bytes, and when passing a NULL pointer to the latter.
object.c: PyMem_{Malloc, Realloc} just call their macro versions
now, since the latter take care of the x-platform 0 and NULL stuff
by themselves now.
pypcre.c, grow_stack(): So sue me. On two lines, this called
PyMem_RESIZE to grow a "const" area. It's not legit to realloc a
const area, so the compiler warned given the new expansion of
PyMem_RESIZE. It would have gotten the same warning before if it
had used PyMem_Resize() instead; the older macro version, but not the
function version, silently cast away the constness. IMO that was a wrong
thing to do, and the docs say the macro versions of PyMem_xyz are
deprecated anyway. If somebody else is resizing const areas with the
macro spelling, they'll get a warning when they recompile now too.
2002-04-12 04:22:56 -03:00
|
|
|
|
Years in the making.
objimpl.h, pymem.h: Stop mapping PyMem_{Del, DEL} and PyMem_{Free, FREE}
to PyObject_{Free, FREE} in a release build. They're aliases for the
system free() now.
_subprocess.c/sp_handle_dealloc(): Since the memory was originally
obtained via PyObject_NEW, it must be released via PyObject_FREE (or
_DEL).
pythonrun.c, tokenizer.c, parsermodule.c: I lost count of the number of
PyObject vs PyMem mismatches in these -- it's like the specific
function called at each site was picked at random, sometimes even with
memory obtained via PyMem getting released via PyObject. Changed most
to use PyObject uniformly, since the blobs allocated are predictably
small in most cases, and obmalloc is generally faster than system
mallocs then.
If extension modules in real life prove as sloppy as Python's front
end, we'll have to revert the objimpl.h + pymem.h part of this patch.
Note that no problems will show up in a debug build (all calls still go
thru obmalloc then). Problems will show up only in a release build, most
likely segfaults.
2006-03-26 19:27:58 -04:00
|
|
|
/* PyMem{Del,DEL} are left over from ancient days, and shouldn't be used
|
|
|
|
|
* anymore. They're just confusing aliases for PyMem_{Free,FREE} now.
|
|
|
|
|
*/
|
|
|
|
|
#define PyMem_Del PyMem_Free
|
|
|
|
|
#define PyMem_DEL PyMem_FREE
|
Give Python a debug-mode pymalloc, much as sketched on Python-Dev.
When WITH_PYMALLOC is defined, define PYMALLOC_DEBUG to enable the debug
allocator. This can be done independent of build type (release or debug).
A debug build automatically defines PYMALLOC_DEBUG when pymalloc is
enabled. It's a detected error to define PYMALLOC_DEBUG when pymalloc
isn't enabled.
Two debugging entry points defined only under PYMALLOC_DEBUG:
+ _PyMalloc_DebugCheckAddress(const void *p) can be used (e.g., from gdb)
to sanity-check a memory block obtained from pymalloc. It sprays
info to stderr (see next) and dies via Py_FatalError if the block is
detectably damaged.
+ _PyMalloc_DebugDumpAddress(const void *p) can be used to spray info
about a debug memory block to stderr.
A tiny start at implementing "API family" checks isn't good for
anything yet.
_PyMalloc_DebugRealloc() has been optimized to do little when the new
size is <= old size. However, if the new size is larger, it really
can't call the underlying realloc() routine without either violating its
contract, or knowing something non-trivial about how the underlying
realloc() works. A memcpy is always done in this case.
This was a disaster for (and only) one of the std tests: test_bufio
creates single text file lines up to a million characters long. On
Windows, fileobject.c's get_line() uses the horridly funky
getline_via_fgets(), which keeps growing and growing a string object
hoping to find a newline. It grew the string object 1000 bytes each
time, so for a million-character string it took approximately forever
(I gave up after a few minutes).
So, also:
fileobject.c, getline_via_fgets(): When a single line is outrageously
long, grow the string object at a mildly exponential rate, instead of
just 1000 bytes at a time.
That's enough so that a debug-build test_bufio finishes in about 5 seconds
on my Win98SE box. I'm curious to try this on Win2K, because it has very
different memory behavior than Win9X, and test_bufio always took a factor
of 10 longer to complete on Win2K. It *could* be that the endless
reallocs were simply killing it on Win2K even in the release build.
2002-03-23 06:03:50 -04:00
|
|
|
|
2000-07-31 19:19:30 -03:00
|
|
|
#ifdef __cplusplus
|
|
|
|
|
}
|
|
|
|
|
#endif
|
|
|
|
|
|
|
|
|
|
#endif /* !Py_PYMEM_H */
|
