Commit Graph

66 Commits

Author SHA1 Message Date
Gregory P. Smith 0470bab697 Issue #2620: Overflow checking when allocating or reallocating memory
was not always being done properly in some python types and extension
modules.  PyMem_MALLOC, PyMem_REALLOC, PyMem_NEW and PyMem_RESIZE have
all been updated to perform better checks and places in the code that
would previously leak memory on the error path when such an allocation
failed have been fixed.
2008-07-22 04:46:32 +00:00
Gregory P. Smith 9d53457e59 Merge in release25-maint r60793:
Added checks for integer overflows, contributed by Google. Some are
 only available if asserts are left in the code, in cases where they
 can't be triggered from Python code.
2008-06-11 07:41:16 +00:00
Martin v. Löwis 6819210b9e PEP 3123: Provide forward compatibility with Python 3.0, while keeping
backwards compatibility. Add Py_Refcnt, Py_Type, Py_Size, and
PyVarObject_HEAD_INIT.
2007-07-21 06:55:02 +00:00
Neal Norwitz ab77227470 Don't inline Py_ADDRESS_IN_RANGE with gcc 4+ either.
Will backport.
2006-10-28 21:21:00 +00:00
Tim Peters 3eeb17346c _PyObject_DebugMalloc(): The return value should add
2*sizeof(size_t) now, not 8.  This probably accounts for
current disasters on the 64-bit buildbot slaves.
2006-06-04 03:38:04 +00:00
Tim Peters 9ea89d2a19 In a PYMALLOC_DEBUG build obmalloc adds extra debugging info
to each allocated block.  This was using 4 bytes for each such
piece of info regardless of platform.  This didn't really matter
before (proof: no bug reports, and the debug-build obmalloc would
have assert-failed if it was ever asked for a chunk of memory
>= 2**32 bytes), since container indices were plain ints.  But after
the Py_ssize_t changes, it's at least theoretically possible to
allocate a list or string whose guts exceed 2**32 bytes, and the
PYMALLOC_DEBUG routines would fail then (having only 4 bytes
to record the originally requested size).

Now we use sizeof(size_t) bytes for each of a PYMALLOC_DEBUG
build's extra debugging fields.  This won't make any difference
on 32-bit boxes, but will add 16 bytes to each allocation in
a debug build on a 64-bit box.
2006-06-04 03:26:02 +00:00
Neal Norwitz 9b26122ec0 Get compiling again 2006-04-11 07:58:54 +00:00
Anthony Baxter a62862120d More low-hanging fruit. Still need to re-arrange some code (or find a better
solution) in the same way as listobject.c got changed. Hoping for a better
solution.
2006-04-11 07:42:36 +00:00
Tim Peters cf79aace07 Merge the tim-obmalloc branch to the trunk.
This is a heavily altered derivative of SF patch 1123430, Evan
Jones's heroic effort to make obmalloc return unused arenas to
the system free(), with some heuristic strategies to make it
more likley that arenas eventually _can_ be freed.
2006-03-16 01:14:46 +00:00
Neal Norwitz 82c5a86d7c Oops, this is supposed to be disabled by default. 2006-02-16 07:30:11 +00:00
Martin v. Löwis 18e165558b Merge ssize_t branch. 2006-02-15 17:27:45 +00:00
Neal Norwitz e5e5aa4ea6 Do a better job of not inlining Py_ADDRESS_IN_RANGE() for newer gcc's.
Perhaps Py_NO_INLINE should be moved to pyport.h or some other header?
2005-11-13 18:55:39 +00:00
Tim Peters ecc6e6a54e SF bug 1185883: PyObject_Realloc can't safely take over a block currently
managed by C, because it's possible for the block to be smaller than the
new requested size, and at the end of allocated VM.  Trying to copy over
nbytes bytes to a Python small-object block can segfault then, and there's
no portable way to avoid this (we would have to know how many bytes
starting at p are addressable, and std C has no means to determine that).

Bugfix candidate.  Should be backported to 2.4, but I'm out of time.
2005-07-10 22:30:55 +00:00
Neal Norwitz b5d7702e5c whoops, I wanted that commented out by default, will add doc to Misc 2004-06-06 19:21:34 +00:00
Neal Norwitz 7eb3c9196d SF bug 881641, make it easier to use valgrind 2004-06-06 19:20:22 +00:00
Walter Dörwald e0a1bb6341 Whitespace normalization. 2003-06-17 15:48:11 +00:00
Martin v. Löwis 39f59b089d Remove MALLOC_ZERO_RETURNS_NULL. 2002-11-23 09:13:40 +00:00
Martin v. Löwis 8c1402869b Patch #627105: Document that SYSTEM_PAGE_SIZE really should not be
larger than the system page size.
2002-10-26 15:01:53 +00:00
Guido van Rossum efc1188239 Fix warnings on 64-bit platforms about casts from pointers to ints.
Two of these were real bugs.
2002-09-12 14:43:41 +00:00
Jeremy Hylton d1fedb6ab5 Remove extraneous semicolon.
(Silences compiler warning for Compaq C++ 6.5 on Tru64.)
2002-07-18 18:49:52 +00:00
Tim Peters 889f61dcfb Documented PYMALLOC_DEBUG. This completes primary coverage of all the
"special builds" I ever use.  If you use others, document them here, or
don't be surprised if I rip out the code for them <0.5 wink>.
2002-07-10 19:29:49 +00:00
Tim Peters 4ce71f77c3 PyObject_Realloc(): If a small block is shrinking, bite the expense of
copying it if at least 25% of the input block can be reclaimed.
2002-05-02 20:19:34 +00:00
Tim Peters 449b5a8da1 _PyObject_DebugCheckAddress(): If the leading pad bytes are corrupt,
display a msg warning that the count of bytes requested may be bogus,
and that a segfault may happen next.
2002-04-28 06:14:45 +00:00
Tim Peters 08d821582f _PyObject_DebugMallocStats(): Added some potentially expensive internal
consistency checks, enabled only in a debug (Py_DEBUG) build.  Note that
this never gets called automatically unless PYMALLOC_DEBUG is #define'd
too, and the envar PYTHONMALLOCSTATS exists.
2002-04-18 22:25:03 +00:00
Tim Peters 64d80c9f40 PyObject_Malloc: make a tiny bit faster for platforms where malloc(0)
doesn't return NULL.

PyObject_Realloc:  better comment for why we don't call PyObject_Malloc(0).
2002-04-18 21:58:56 +00:00
Tim Peters 3e12071dbe Remove some long-disabled debugging boilerplate. 2002-04-18 21:37:03 +00:00
Tim Peters 0e871188e8 _PyObject_DebugDumpStats: renamed to _PyObject_DebugMallocStats.
Added code to call this when PYMALLOC_DEBUG is enabled, and envar
PYTHONMALLOCSTATS is set, whenever a new arena is obtained and once
late in the Python shutdown process.
2002-04-13 08:29:14 +00:00
Tim Peters 8a8cdfd0f5 Small anal correctness tweaks:
_PyObject_DebugMalloc:  explicitly cast PyObject_Malloc's result to the
target pointer type.

_PyObject_DebugDumpStats:  change decl of arena_alignment from unsigned
int to unsigned long.

This is for the 2.3 release only (it's new code).
2002-04-12 20:49:36 +00:00
Tim Peters 85cc1c4368 _PyObject_DebugRealloc(): rewritten to let the underlying realloc do
most of the work.  In particular, if the underlying realloc is able to
grow the memory block in place, great (this routine used to do a fresh
malloc + memcpy every time a block grew).  BTW, I'm not so keen here on
avoiding possible quadratic-time realloc patterns as I am on making
the debug pymalloc more invisible (the more it uses memory "just like"
the underlying allocator, the better the chance that a suspected memory
corruption bug won't vanish when the debug malloc is turned on).
2002-04-12 08:52:50 +00:00
Tim Peters f539c68ccd _PyObject_DebugDumpAddress(): clarify an output message. 2002-04-12 07:43:07 +00:00
Tim Peters f6fb501c57 PYMALLOC_{CLEAN, DEAD, FORBIDDEN}BYTE symbols: remove the PYMALLOC_
prefix.  These symbols are private to the file, and the PYMALLOC_ gets
in the way (overly long code lines, comments, and error messages).
2002-04-12 07:38:53 +00:00
Neil Schemenauer d2560cd37c Move PyObject_Malloc and PyObject_Free here from object.c. Remove
PyMalloc_ prefix and use PyObject_ instead.  I'm not sure about the
debugging functions.  Perhaps they should stay as PyMalloc_.
2002-04-12 03:10:20 +00:00
Tim Peters 52aefc8a7b SF bug 542181: Realloc behavior
The bug report pointed out a bogosity in the comment block explaining
thread safety for arena management.  Repaired that comment, repaired a
couple others while I was at it, and added an assert.

_PyMalloc_DebugRealloc:  If this needed to get more memory, but couldn't,
it erroneously freed the original memory.  Repaired that.

This is for 2.3 only (unless we decide to backport the new pymalloc).
2002-04-11 06:36:45 +00:00
Tim Peters 49f26817eb Minor improvements to the stats output dump, including adding commas to
the big numbers.
2002-04-06 01:45:35 +00:00
Tim Peters ffdd22f1af Repair an incomprehensible comment. 2002-04-05 06:24:54 +00:00
Tim Peters 16bcb6b1af _PyMalloc_DebugDumpStats(): vastly improved the output, and it now
accounts for every byte.
2002-04-05 05:45:31 +00:00
Tim Peters e70ddf3a99 Widespread, but mostly in _PyMalloc_Malloc: optimize away all expensive
runtime multiplications and divisions, via the scheme developed with
Vladimir Marangozov on Python-Dev.  The pool_header struct loses its
capacity member, but gains nextoffset and maxnextoffset members; this
still leaves it at 32 bytes on a 32-bit box (it has to be padded to a
multiple of 8 bytes).
2002-04-05 04:32:29 +00:00
Tim Peters b7265dbe3e _PyMalloc_Realloc(): removed a now-pointless cast. 2002-04-04 05:08:31 +00:00
Tim Peters 84c1b97467 _PyMalloc_{Malloc, Realloc}: Strive to meet the doc's promises about
what these do given a 0 size argument.  This is so that when pymalloc
is enabled, we don't need to wrap pymalloc calls in goofy little
routines special-casing 0.  Note that it's virtually impossible to meet
the doc's promise that malloc(0) will never return NULL; this makes a
best effort, but not an insane effort.  The code does promise that
realloc(not-NULL, 0) will never return NULL (malloc(0) is much harder).

_PyMalloc_Realloc:  Changed to take over all requests for 0 bytes, and
rearranged to be a little quicker in expected cases.

All over the place:  when resorting to the platform allocator, call
free/malloc/realloc directly, without indirecting thru macros.  This
should avoid needing a nightmarish pile of #ifdef-ery if PYMALLOC_DEBUG
is changed so that pymalloc takes over all Py(Mem, Object} memory
operations (which would add useful debugging info to PyMem_xyz
allocations too).
2002-04-04 04:44:32 +00:00
Tim Peters 6169f09d64 Fixed errors in two comments. 2002-04-01 20:12:59 +00:00
Tim Peters 338e010b45 Restructured my pool-management overview in terms of the three
possible pool states.  I think it's much clearer now.

Added a new long overdue block-management overview comment block.

I believe the comments are in good shape now.

Added two comments about possible small optimizations (one getting rid
of runtime multiplications at the cost of a new pool_header member; the
other getting rid of runtime divisions and the pool_header capacity
member, at the cost of a static const vector of 32 uints).
2002-04-01 19:23:44 +00:00
Tim Peters 7ccfadf3a8 New PYMALLOC_DEBUG function void _PyMalloc_DebugDumpStats(void).
This displays stats about the # of arenas, pools, blocks and bytes, to
stderr, both used and reserved but unused.

CAUTION:  Because PYMALLOC_DEBUG is on, the debug malloc routine adds
16 bytes to each request.  This makes each block appear two size classes
higher than it would be if PYMALLOC_DEBUG weren't on.

So far, playing with this confirms the obvious:  there's a lot of activity
in the "small dict" size class, but nothing in the core makes any use of
the 8-byte or 16-byte classes.
2002-04-01 06:04:21 +00:00
Tim Peters 57b17ad6ae Add one more assert that indirectly interlocking conditions are consistent
with each other.
2002-03-31 02:59:48 +00:00
Tim Peters 4c5be0ce09 Fixed an error in a new assert. 2002-03-31 02:52:29 +00:00
Tim Peters b1da050131 Fixed a typo in a new comment. 2002-03-31 02:51:40 +00:00
Tim Peters 2c95c99a64 _PyMalloc_Free(): As was already done for _PyMalloc_Malloc, rearranged
the code so that the most frequent cases come first.  Added comments.
Found a hidden assumption that a pool contains room for at least two
blocks, and added an assert to catch a violation if it ever happens in
a place where that matters.  Gave the normal "I allocated this block"
case a longer basic block to work with before it has to do its first
branch (via breaking apart an embedded assignment in an "if", and
hoisting common code out of both branches).
2002-03-31 02:18:01 +00:00
Tim Peters 1e16db6d3b Added a long-overdue comment block giving an overview of pool operations
and terminology, plus explanation of some extreme obscurities.
2002-03-31 01:05:22 +00:00
Tim Peters c2ce91af5f It's once again thought safe to call the pymalloc free/realloc with an
address obtained from system malloc/realloc without holding the GIL.

When the vector of arena base addresses has to grow, the old vector is
deliberately leaked.  This makes "stale" x-thread references safe.
arenas and narenas are also declared volatile, and changed in an order
that prevents a thread from picking up a value of narenas too large
for the value of arenas it sees.

Added more asserts.

Fixed an old inaccurate comment.

Added a comment explaining why it's safe to call pymalloc free/realloc
with an address obtained from system malloc/realloc even when arenas is
still NULL (this is obscure, since the ADDRESS_IN_RANGE macro
appears <wink> to index into arenas).
2002-03-30 21:36:04 +00:00
Tim Peters 7b85b4aa7f new_arena(): In error cases, reset the number of available pools to 0.
Else the pymalloc malloc will go insane the next time it's called.
2002-03-30 10:42:09 +00:00
Tim Peters 1d99af8d69 Changed the #-of-arenas counters to uints -- no need to be insane about
this.  But added an overflow check just in case there is.

Got rid of the ushort macro.  It wasn't used anymore (it was only used
in the no-longer-exists off_t macro), and there's no plausible use for it.
2002-03-30 10:35:09 +00:00