* Optimize tracemalloc_add_trace(): modify hashtable entry data (trace) if the
memory block is already tracked, rather than trying to remove the old trace
and then add a new trace.
* Add _Py_HASHTABLE_ENTRY_WRITE_DATA() macro
Issue #26530:
* Add C functions _PyTraceMalloc_Track() and _PyTraceMalloc_Untrack() to track
memory blocks using the tracemalloc module.
* Add _PyTraceMalloc_GetTraceback() to get the traceback of an object.
Issue #26516:
* Add PYTHONMALLOC environment variable to set the Python memory
allocators and/or install debug hooks.
* PyMem_SetupDebugHooks() can now also be used on Python compiled in release
mode.
* The PYTHONMALLOCSTATS environment variable can now also be used on Python
compiled in release mode. It now has no effect if set to an empty string.
* In debug mode, debug hooks are now also installed on Python memory allocators
when Python is configured without pymalloc.
PyObject_Calloc(), _PyObject_GC_Calloc(). bytes(int) and bytearray(int) are now
using ``calloc()`` instead of ``malloc()`` for large objects which is faster
and use less memory (until the bytearray buffer is filled with data).
Add new enum:
* PyMemAllocatorDomain
Add new structures:
* PyMemAllocator
* PyObjectArenaAllocator
Add new functions:
* PyMem_RawMalloc(), PyMem_RawRealloc(), PyMem_RawFree()
* PyMem_GetAllocator(), PyMem_SetAllocator()
* PyObject_GetArenaAllocator(), PyObject_SetArenaAllocator()
* PyMem_SetupDebugHooks()
Changes:
* PyMem_Malloc()/PyObject_Realloc() now always call malloc()/realloc(), instead
of calling PyObject_Malloc()/PyObject_Realloc() in debug mode.
* PyObject_Malloc()/PyObject_Realloc() now falls back to
PyMem_Malloc()/PyMem_Realloc() for allocations larger than 512 bytes.
* Redesign debug checks on memory block allocators as hooks, instead of using C
macros
* Add a new PyMemAllocators structure
* New functions:
- PyMem_RawMalloc(), PyMem_RawRealloc(), PyMem_RawFree(): GIL-free memory
allocator functions
- PyMem_GetRawAllocators(), PyMem_SetRawAllocators()
- PyMem_GetAllocators(), PyMem_SetAllocators()
- PyMem_SetupDebugHooks()
- _PyObject_GetArenaAllocators(), _PyObject_SetArenaAllocators()
* Add unit test for PyMem_Malloc(0) and PyObject_Malloc(0)
* Add unit test for new get/set allocators functions
* PyObject_Malloc() now falls back on PyMem_Malloc() instead of malloc() if
size is bigger than SMALL_REQUEST_THRESHOLD, and PyObject_Realloc() falls
back on PyMem_Realloc() instead of realloc()
* PyMem_Malloc() and PyMem_Realloc() now always call malloc() and realloc(),
instead of calling PyObject_Malloc() and PyObject_Realloc() in debug mode
svn+ssh://pythondev@svn.python.org/python/trunk
........
r78189 | mark.dickinson | 2010-02-14 13:40:30 +0000 (Sun, 14 Feb 2010) | 1 line
Silence more 'comparison between signed and unsigned' warnings.
........
svn+ssh://pythondev@svn.python.org/python/trunk
........
r76740 | mark.dickinson | 2009-12-10 10:36:32 +0000 (Thu, 10 Dec 2009) | 8 lines
Replace the size check for PyMem_MALLOC and PyMem_REALLOC with an almost
equivalent[*] check that doesn't produce compiler warnings about a 'x < 0'
check on an unsigned type.
[*] it's equivalent for inputs of type size_t or Py_ssize_t, or any smaller
unsigned or signed integer type.
........
svn+ssh://pythondev@svn.python.org/python/trunk
........
r65182 | gregory.p.smith | 2008-07-22 06:46:32 +0200 (Tue, 22 Jul 2008) | 7 lines
Issue #2620: Overflow checking when allocating or reallocating memory
was not always being done properly in some python types and extension
modules. PyMem_MALLOC, PyMem_REALLOC, PyMem_NEW and PyMem_RESIZE have
all been updated to perform better checks and places in the code that
would previously leak memory on the error path when such an allocation
failed have been fixed.
........
svn+ssh://pythondev@svn.python.org/python/trunk
........
r64114 | gregory.p.smith | 2008-06-11 09:41:16 +0200 (mer., 11 juin 2008) | 6 lines
Merge in release25-maint r60793:
Added checks for integer overflows, contributed by Google. Some are
only available if asserts are left in the code, in cases where they
can't be triggered from Python code.
........
svn+ssh://pythondev@svn.python.org/python/branches/p3yk
................
r55326 | guido.van.rossum | 2007-05-14 15:07:35 -0700 (Mon, 14 May 2007) | 2 lines
Don't use err.message, use err.args[0].
................
r55327 | guido.van.rossum | 2007-05-14 15:11:37 -0700 (Mon, 14 May 2007) | 259 lines
Merged revisions 54988-55226,55228-55323 via svnmerge from
svn+ssh://pythondev@svn.python.org/python/trunk
........
r54995 | neal.norwitz | 2007-04-26 23:45:32 -0700 (Thu, 26 Apr 2007) | 3 lines
This gets the test working on Solaris. It seems a little hokey to me,
but the test passed on Linux and Solaris, hopefully other platforms too.
........
r55002 | georg.brandl | 2007-04-27 12:20:00 -0700 (Fri, 27 Apr 2007) | 2 lines
Version fix (bug #1708710)
........
r55021 | neal.norwitz | 2007-04-29 16:53:24 -0700 (Sun, 29 Apr 2007) | 1 line
There really are some tests that are problematic.
........
r55024 | kristjan.jonsson | 2007-04-30 08:17:46 -0700 (Mon, 30 Apr 2007) | 1 line
Complete revamp of PCBuild8 directory. Use subdirectories for each project under the main pcbuild solution. Now make extensive use of property sheets to simplify project configuration. x64 build fully supported, and the process for building PGO version (Profiler Guided Optimization) simplified. All projects are now present, except _ssl, which needs to be reimplemented. Also, some of the projects that require external libraries need extra work to fully compile on x64.
........
r55025 | thomas.heller | 2007-04-30 08:44:17 -0700 (Mon, 30 Apr 2007) | 4 lines
Make sure to call PyErr_NoMemory() in several places where
PyMem_Malloc() could potentially fail.
Will backport to the release25-maint branch.
........
r55027 | thomas.heller | 2007-04-30 09:04:57 -0700 (Mon, 30 Apr 2007) | 8 lines
When accessing the .value attribute of a c_wchar_p instance, and the
instance does not point to a valid wchar_t zero-terminated string,
raise a ValueError. c_char_p does this already.
The ValueError message now contains the correct pointer address.
Will backport to release25-maint.
........
r55036 | georg.brandl | 2007-04-30 23:04:11 -0700 (Mon, 30 Apr 2007) | 2 lines
Bug #1710295: exceptions are now new-style classes.
........
r55038 | georg.brandl | 2007-04-30 23:08:15 -0700 (Mon, 30 Apr 2007) | 2 lines
Patch #1710352: add missing public functions to locale.__all__.
........
r55041 | vinay.sajip | 2007-05-01 03:20:03 -0700 (Tue, 01 May 2007) | 1 line
Added new optional credentials argument to SMTPHandler.__init__, and smtp.login() is now called in SMTPHandler.emit() if credentials are specified.
........
r55042 | vinay.sajip | 2007-05-01 03:21:45 -0700 (Tue, 01 May 2007) | 1 line
Added documentation for new optional credentials argument to SMTPHandler.__init__().
........
r55070 | neal.norwitz | 2007-05-01 21:47:55 -0700 (Tue, 01 May 2007) | 3 lines
Stop using PyMem_FREE while the GIL is not held. For details see:
http://mail.python.org/pipermail/python-dev/2007-May/072896.html
........
r55080 | armin.rigo | 2007-05-02 12:23:31 -0700 (Wed, 02 May 2007) | 6 lines
Fix for #1303614 and #1174712:
- __dict__ descriptor abuse for subclasses of built-in types
- subclassing from both ModuleType and another built-in types
Thanks zseil for the patch.
........
r55083 | georg.brandl | 2007-05-02 13:02:29 -0700 (Wed, 02 May 2007) | 3 lines
Actually raise an exception before calling ast_error_finish.
Triggers an assertion otherwise.
........
r55087 | neal.norwitz | 2007-05-02 23:47:18 -0700 (Wed, 02 May 2007) | 1 line
Handle a couple of uncaught errors. This should be backported
........
r55090 | neal.norwitz | 2007-05-03 00:20:57 -0700 (Thu, 03 May 2007) | 4 lines
Remove dead code. This code couldn't be reached because earlier in
the function there is another check for z != Py_None.
........
r55092 | thomas.heller | 2007-05-03 05:02:08 -0700 (Thu, 03 May 2007) | 1 line
Fix building _ctypes.pyd for x64 / Windows.
........
r55093 | thomas.heller | 2007-05-03 05:05:20 -0700 (Thu, 03 May 2007) | 1 line
Don't truncate pointers to integers (on win64 platform).
........
r55094 | walter.doerwald | 2007-05-03 08:13:55 -0700 (Thu, 03 May 2007) | 3 lines
Clarify the behaviour of PyUnicode_DecodeUTF16(): A BOM is only skipped
in native order mode, and only if it's the first two bytes.
........
r55101 | kristjan.jonsson | 2007-05-03 13:04:53 -0700 (Thu, 03 May 2007) | 2 lines
Fix pcbuild8 after recent overhaul: Added the version resource to python26.dll. Adjust stacksize to 2Mb and made large address aware for 32 bits, and set stacksize to 3Mb for 64 bits.
Todo: Set .dll optimized load addresses, and side-by-side packaging of the python26.dll.
........
r55102 | kristjan.jonsson | 2007-05-03 13:09:56 -0700 (Thu, 03 May 2007) | 1 line
Fix those parts in the testsuite that assumed that sys.maxint would cause overflow on x64. Now the testsuite is well behaved on that platform.
........
r55103 | kristjan.jonsson | 2007-05-03 13:27:03 -0700 (Thu, 03 May 2007) | 11 lines
Fix problems in x64 build that were discovered by the testsuite:
- Reenable modules on x64 that had been disabled aeons ago for Itanium.
- Cleared up confusion about compilers for 64 bit windows. There is only Itanium and x64. Added macros MS_WINI64 and MS_WINX64 for those rare cases where it matters, such as the disabling of modules above.
- Set target platform (_WIN32_WINNT and WINVER) to 0x0501 (XP) for x64, and 0x0400 (NT 4.0) otherwise, which are the targeted minimum platforms.
- Fixed thread_nt.h. The emulated InterlockedCompareExchange function didn?\194?\180t work on x64, probaby due to the lack of a "volatile" specifier. Anyway, win95 is no longer a target platform.
- Itertools module used wrong constant to check for overflow in count()
- PyInt_AsSsize_t couldn't deal with attribute error when accessing the __long__ member.
- PyLong_FromSsize_t() incorrectly specified that the operand were unsigned.
With these changes, the x64 passes the testsuite, for those modules present.
........
r55107 | kristjan.jonsson | 2007-05-03 17:25:08 -0700 (Thu, 03 May 2007) | 1 line
Revert compiler comment to AMD64 for x64/AMD64 builds.
........
r55115 | thomas.heller | 2007-05-04 00:14:39 -0700 (Fri, 04 May 2007) | 4 lines
Fix some ctypes test crashes, when running with a debug Python
version on win64 by using proper argtypes and restype function
attributes.
........
r55117 | thomas.heller | 2007-05-04 01:20:41 -0700 (Fri, 04 May 2007) | 4 lines
On 64-bit Windows, ffi_arg must be 8 bytes long. This fixes the
remaining crashes in the ctypes tests, when functions return float or
double types.
........
r55120 | kristjan.jonsson | 2007-05-04 08:48:15 -0700 (Fri, 04 May 2007) | 1 line
Update the pcbuild8 solution. Straightened out the _ctypes project by using a .vsproj file and a masm64.rules file to avoid redundancy
........
r55121 | kristjan.jonsson | 2007-05-04 10:28:06 -0700 (Fri, 04 May 2007) | 1 line
Minor fix of PCBuild8/_ctypes vcproj, moving include dir into the .vsprops file.
........
r55129 | thomas.heller | 2007-05-04 12:54:22 -0700 (Fri, 04 May 2007) | 3 lines
Do not truncate 64-bit pointers to 32-bit integers.
Fixes SF #1703286, will backport to release25-maint.
........
r55131 | thomas.heller | 2007-05-04 12:56:32 -0700 (Fri, 04 May 2007) | 1 line
Oops, these tests do not run on Windows CE.
........
r55140 | brett.cannon | 2007-05-04 18:34:02 -0700 (Fri, 04 May 2007) | 2 lines
Deprecate BaseException.message as per PEP 352.
........
r55154 | georg.brandl | 2007-05-05 11:55:37 -0700 (Sat, 05 May 2007) | 2 lines
Bug #1713535: typo in logging example.
........
r55158 | vinay.sajip | 2007-05-06 10:53:37 -0700 (Sun, 06 May 2007) | 1 line
Updates of recent changes to logging.
........
r55165 | neal.norwitz | 2007-05-07 00:02:26 -0700 (Mon, 07 May 2007) | 1 line
Verify changes to the trunk go to the normal checkins list
........
r55169 | kristjan.jonsson | 2007-05-07 09:46:54 -0700 (Mon, 07 May 2007) | 1 line
As per Armin Rigo's suggestion, remove special handing from intobject.c to deal with the peculiarities of classobject's implementation of the number protocol. The nb_long method of classobject now falls back to nb_int if there is no __long__ attribute present.
........
r55197 | collin.winter | 2007-05-08 21:14:36 -0700 (Tue, 08 May 2007) | 9 lines
Fix a bug in test.test_support.open_urlresource().
If the call to requires() doesn't precede the filesystem check, we get the following situation:
1. ./python Lib/test/regrtest.py test_foo # test needs urlfetch, not enabled, so skipped
2. ./python Lib/test/regrtest.py -u urlfetch test_foo # test runs
3. ./python Lib/test/regrtest.py test_foo # test runs (!)
By moving the call to requires() *before* the filesystem check, the fact that fetched files are cached on the local disk becomes an implementation detail, rather than a semantics-changing point of note.
........
r55198 | neal.norwitz | 2007-05-08 23:43:15 -0700 (Tue, 08 May 2007) | 1 line
Add markup for True/False. Will backport
........
r55205 | walter.doerwald | 2007-05-09 11:10:47 -0700 (Wed, 09 May 2007) | 4 lines
Backport checkin:
Fix a segfault when b"" was passed to b2a_qp() -- it was using strchr()
instead of memchr().
........
r55241 | neal.norwitz | 2007-05-10 22:55:15 -0700 (Thu, 10 May 2007) | 6 lines
Don't ever report a failure when the sum of the reference count differences
are zero. This should help reduce the false positives.
The message about references leaking is maintained to provide as much
info as possible rather than simply suppressing the message at the source.
........
r55242 | neal.norwitz | 2007-05-10 23:23:01 -0700 (Thu, 10 May 2007) | 1 line
Fix typo in docstring (the module is popen2, not 3).
........
r55244 | neal.norwitz | 2007-05-10 23:56:52 -0700 (Thu, 10 May 2007) | 1 line
Remove trailing whitespace in docstring
........
r55245 | neal.norwitz | 2007-05-10 23:57:33 -0700 (Thu, 10 May 2007) | 1 line
Deprecate os.popen* and popen2 module in favor of the subprocess module.
........
r55247 | neal.norwitz | 2007-05-11 00:13:30 -0700 (Fri, 11 May 2007) | 1 line
Deprecate os.popen* and popen2 module in favor of the subprocess module. (forgot the doc)
........
r55253 | georg.brandl | 2007-05-11 02:41:37 -0700 (Fri, 11 May 2007) | 3 lines
Remove an XXX that is unnecessary.
........
r55258 | georg.brandl | 2007-05-11 04:04:26 -0700 (Fri, 11 May 2007) | 2 lines
Patch #1714700: clarify os.linesep vs. tfiles opened in text mode.
(backport)
........
r55259 | georg.brandl | 2007-05-11 04:43:56 -0700 (Fri, 11 May 2007) | 2 lines
Update DDJ link.
........
r55273 | raymond.hettinger | 2007-05-11 10:59:59 -0700 (Fri, 11 May 2007) | 1 line
Better tests for posixpath.commonprefix
........
r55287 | georg.brandl | 2007-05-12 14:06:41 -0700 (Sat, 12 May 2007) | 2 lines
Bug #1046945: document SWIG options of distutils.
........
r55290 | georg.brandl | 2007-05-13 01:04:07 -0700 (Sun, 13 May 2007) | 2 lines
Add bz2 to content encodings.
........
r55297 | neal.norwitz | 2007-05-13 13:45:05 -0700 (Sun, 13 May 2007) | 3 lines
Remove Amoeba doc which was removed in version 1.0! according to Misc/HISTORY.
Hopefully Guido won't shed a tear. :-)
........
r55298 | neal.norwitz | 2007-05-13 13:54:19 -0700 (Sun, 13 May 2007) | 1 line
Remove references to stdwin which was removed long ago.
........
r55299 | neal.norwitz | 2007-05-13 14:13:42 -0700 (Sun, 13 May 2007) | 3 lines
Remove support for freebsd[23] which haven't been released since 2000
or earlier. http://www.freebsd.org/releases/index.html
........
r55320 | raymond.hettinger | 2007-05-14 13:52:31 -0700 (Mon, 14 May 2007) | 1 line
Small speedup.
........
................
number of tests, all because of the codecs/_multibytecodecs issue described
here (it's not a Py3K issue, just something Py3K discovers):
http://mail.python.org/pipermail/python-dev/2006-April/064051.html
Hye-Shik Chang promised to look for a fix, so no need to fix it here. The
tests that are expected to break are:
test_codecencodings_cn
test_codecencodings_hk
test_codecencodings_jp
test_codecencodings_kr
test_codecencodings_tw
test_codecs
test_multibytecodec
This merge fixes an actual test failure (test_weakref) in this branch,
though, so I believe merging is the right thing to do anyway.
As threatened, PyMem_{Free, FREE} also invoke the object deallocator now
when pymalloc is enabled (well, it does when pymalloc isn't enabled too,
but in that case "the object deallocator" is plain free()).
This is maximally backward-compatible, but it leaves a bitter aftertaste.
Also massive reworking of comments.
+ Redirect PyMem_{Del, DEL} to the object allocator's free() when
pymalloc is enabled. Needed so old extensions can continue to
mix PyObject_New with PyMem_DEL.
+ This implies that pgen needs to be able to see the PyObject_XYZ
declarations too. pgenheaders.h now includes Python.h. An
implication is that I expect obmalloc.o needs to get linked into
pgen on non-Windows boxes.
+ When PYMALLOC_DEBUG is defined, *all* Py memory API functions
now funnel through the debug allocator wrapper around pymalloc.
This is the default in a debug build.
+ That caused compile.c to fail: it indirectly mixed PyMem_Malloc
with raw platform free() in one place. This is verbotten.
PyMem_{Del, DEL} doesn't work yet (compilation problems).
pyport.h: _PyMem_EXTRA is gone.
pmem.h: Repaired comments. PyMem_{Malloc, MALLOC} and
PyMem_{Realloc, REALLOC} now make the same x-platform guarantees when
asking for 0 bytes, and when passing a NULL pointer to the latter.
object.c: PyMem_{Malloc, Realloc} just call their macro versions
now, since the latter take care of the x-platform 0 and NULL stuff
by themselves now.
pypcre.c, grow_stack(): So sue me. On two lines, this called
PyMem_RESIZE to grow a "const" area. It's not legit to realloc a
const area, so the compiler warned given the new expansion of
PyMem_RESIZE. It would have gotten the same warning before if it
had used PyMem_Resize() instead; the older macro version, but not the
function version, silently cast away the constness. IMO that was a wrong
thing to do, and the docs say the macro versions of PyMem_xyz are
deprecated anyway. If somebody else is resizing const areas with the
macro spelling, they'll get a warning when they recompile now too.
This displays stats about the # of arenas, pools, blocks and bytes, to
stderr, both used and reserved but unused.
CAUTION: Because PYMALLOC_DEBUG is on, the debug malloc routine adds
16 bytes to each request. This makes each block appear two size classes
higher than it would be if PYMALLOC_DEBUG weren't on.
So far, playing with this confirms the obvious: there's a lot of activity
in the "small dict" size class, but nothing in the core makes any use of
the 8-byte or 16-byte classes.
When WITH_PYMALLOC is defined, define PYMALLOC_DEBUG to enable the debug
allocator. This can be done independent of build type (release or debug).
A debug build automatically defines PYMALLOC_DEBUG when pymalloc is
enabled. It's a detected error to define PYMALLOC_DEBUG when pymalloc
isn't enabled.
Two debugging entry points defined only under PYMALLOC_DEBUG:
+ _PyMalloc_DebugCheckAddress(const void *p) can be used (e.g., from gdb)
to sanity-check a memory block obtained from pymalloc. It sprays
info to stderr (see next) and dies via Py_FatalError if the block is
detectably damaged.
+ _PyMalloc_DebugDumpAddress(const void *p) can be used to spray info
about a debug memory block to stderr.
A tiny start at implementing "API family" checks isn't good for
anything yet.
_PyMalloc_DebugRealloc() has been optimized to do little when the new
size is <= old size. However, if the new size is larger, it really
can't call the underlying realloc() routine without either violating its
contract, or knowing something non-trivial about how the underlying
realloc() works. A memcpy is always done in this case.
This was a disaster for (and only) one of the std tests: test_bufio
creates single text file lines up to a million characters long. On
Windows, fileobject.c's get_line() uses the horridly funky
getline_via_fgets(), which keeps growing and growing a string object
hoping to find a newline. It grew the string object 1000 bytes each
time, so for a million-character string it took approximately forever
(I gave up after a few minutes).
So, also:
fileobject.c, getline_via_fgets(): When a single line is outrageously
long, grow the string object at a mildly exponential rate, instead of
just 1000 bytes at a time.
That's enough so that a debug-build test_bufio finishes in about 5 seconds
on my Win98SE box. I'm curious to try this on Win2K, because it has very
different memory behavior than Win9X, and test_bufio always took a factor
of 10 longer to complete on Win2K. It *could* be that the endless
reallocs were simply killing it on Win2K even in the release build.
Windows some modules are considered (by me, and I don't care what anyone
else thinks about this <wink>) to be part of "the core" despite that they
happen to be compiled into separate DLLs (the "to DLL or not to DLL?"
question on Windows is nearly arbitrary). Making the pymalloc entry
points available to them allows the Windows build to complete without
incident when WITH_PYMALLOC is #define'd.
Note that this isn't unprecedented. Other "private API" functions we
export include _PySequence_IterSearch, _PyEval_SliceIndex, _PyCodec_Lookup,
_Py_ZeroStruct, _Py_TrueStruct, _PyLong_New and _PyModule_Clear.
platform realloc(p, 0) returns NULL, so MALLOC_ZERO_RETURNS_NULL can
be correctly undefined yet realloc(p, 0) can return NULL anyway.
Prevent realloc(p, 0) doing free(p) and returning NULL via a different
hack. Would probably be better to get rid of MALLOC_ZERO_RETURNS_NULL
entirely.
Bugfix candidate.
Serhiy Storchaka
Benjamin Peterson
Victor Stinner
Martin v. Löwis
Mark Dickinson
Kristján Valur Jónsson
Georg Brandl
Amaury Forgeot d'Arc
Guido van Rossum
Thomas Wouters
Mark Hammond
Tim Peters
Neil Schemenauer
Vladimir Marangozov
Peter Schneider-Kamp