[ 400998 ] experimental support for extended slicing on lists
somewhat spruced up and better tested than it was when I wrote it.
Includes docs & tests. The whatsnew section needs expanding, and arrays
should support extended slices -- later.
discovered that subtype_traverse must traverse the type if it is a
heap type, because otherwise some cycles involving a type and its
instance would not be collected. Simplest example:
while 1:
class C(object): pass
C.ref = C()
This program grows without bounds before this fix. (It grows ever
slower since it spends ever more time in the collector.)
Simply adding the right visit() call to subtype_traverse() revealed
other problems. With MvL's help we re-learned that type_clear()
doesn't have to clear *all* references, only the ones that may not be
cleared by other means. Careful analysis (see comments in the code)
revealed that only tp_mro needs to be cleared. (The previous checkin
to this file adds a test for tp_mro==NULL to _PyType_Lookup() that's
essential to prevent crashes due to tp_mro being NULL when
subtype_dealloc() tries to look for a __del__ method.) The same kind
of analysis also revealed that subtype_clear() doesn't need to clear
the instance dict.
With this fix, a useful property of the collector is once again
guaranteed: a single gc.collect() call will clear out all garbage.
(It didn't always before, which put us on the track of this bug.)
Will backport to 2.2.
about the test case, slot_nb_power gets called on behalf of its second
argument, but with a non-None modulus it wouldn't check this, and
believes it is called on behalf of its first argument. Fix this
properly, and get rid of the code in _PyType_Lookup() that tries to
call _PyType_Ready(). But do leave a check for a NULL tp_mro there,
because this can still legitimately occur.
I'll fix this in 2.2.x too.
While I was at it, I added a tp_clear handler and changed the
tp_dealloc handler to use the clear_slots helper for the tp_clear
handler.
Also tightened the rules for slot names: they must now be proper
identifiers (ignoring the dirty little fact that <ctype.h> is locale
sensitive).
Also set mp->flags = READONLY for the __weakref__ pseudo-slot.
Most of this is a 2.2 bugfix candidate; I'll apply it there myself.
Change the module constructor (module_init) to have the signature
__init__(name:str, doc=None); this prevents the call from type_new()
to succeed. While we're at it, prevent repeated calling of
module_init for the same module from leaking the dict, changing the
semantics so that __dict__ is only initialized if NULL.
Also adding a unittest, test_module.py.
This is an incompatibility with 2.2, if anybody was instantiating the
module class before, their argument list was probably empty; so this
can't be backported to 2.2.x.
In the past, an object's tp_compare could return any value. In 2.2
the docs were tightened to require it to return -1, 0 or 1; and -1 for
an error.
We now issue a warning if the value is not in this range. When an
exception is raised, we allow -1 or -2 as return value, since -2 will
the recommended return value for errors in the future. (Eventually
tp_compare will also be allowed to return +2, to indicate
NotImplemented; but that can only be implemented once we know all
extensions return a value in [-2...1]. Or perhaps it will require the
type to set a flag bit.)
I haven't decided yet whether to backport this to 2.2.x. The patch
applies fine. But is it fair to start warning in 2.2.2 about code
that worked flawlessly in 2.2.1?
for 'str' and 'unicode', and can be used instead of
types.StringTypes, e.g. to test whether something is "a string":
isinstance(x, string) is True for Unicode and 8-bit strings. This
is an abstract base class and cannot be instantiated directly.
A MemoryError is now raised when the list cannot be created.
There is a test, but as the comment says, it really only
works for 32 bit systems. I don't know how to improve
the test for other systems (ie, 64 bit or systems
where the data size != addressable size,
e.g. 64 bit data, but 48 bit addressable memory)
returned a proxy for __class__ whose __bases__ was also a proxy. The
merge_class_dict() helper for dir() assumed incorrectly that __bases__
would always be a tuple and used the in-line tuple API on the proxy.
I will backport this to 2.2 as well.
handlers were both set, but were not compatible. This change uses only the
tp_getattro handler with a more "modern" approach.
This fixes SF bug #551285.
don't understand how this function works, also beefed up the docs. The
most common usage error is of this form (often spread out across gotos):
if (_PyString_Resize(&s, n) < 0) {
Py_DECREF(s);
s = NULL;
goto outtahere;
}
The error is that if _PyString_Resize runs out of memory, it automatically
decrefs the input string object s (which also deallocates it, since its
refcount must be 1 upon entry), and sets s to NULL. So if the "if"
branch ever triggers, it's an error to call Py_DECREF(s): s is already
NULL! A correct way to write the above is the simpler (and intended)
if (_PyString_Resize(&s, n) < 0)
goto outtahere;
Bugfix candidate.
This implements ideas from Marc-Andre, Martin, Guido and me on Python-Dev.
"Short" Unicode strings are encoded into a "big enough" stack buffer,
then exactly as much string space as they turn out to need is allocated
at the end. This should have speed benefits akin to Martin's "measure
once, allocate once" strategy, but without needing a distinct measuring
pass.
"Long" Unicode strings allocate as much heap space as they could possibly
need (4 x # Unicode chars), and do a realloc at the end to return the
untouched excess. Since the overallocation is likely to be substantial,
this shouldn't burden the platform realloc with unusably small excess
blocks.
Also simplified uses of the PyString_xyz functions. Also added a release-
build check that 4*size doesn't overflow a C int. Sooner or later, that's
going to happen.
left and right type were of the same type and not classic instances.
This shortcut is dangerous for proxy types, because it means that
coerce(Proxy(1), Proxy(2.1)) leaves Proxy(1) unchanged rather than
turning it into Proxy(1.0).
In an ever-so-slight change of semantics, I now only take the shortcut
when the left and right types are of the same type and don't have the
CHECKTYPES feature. It so happens that classic instances have this
flag, so the shortcut is still skipped in this case (i.e. nothing
changes for classic instances). Proxies also have this flag set
(otherwise implementing numeric operations on proxies would become
nightmarish) and this means that the shortcut is also skipped there,
as desired. It so happens that int, long and float also have this
flag set; that means that e.g. coerce(1, 1) will now invoke
int_coerce(). This is fine: int_coerce() can deal with this, and I'm
not worried about the performance; int_coerce() is only invoked when
the user explicitly calls coerce(), which should be rarer than rare.
This fixes the problem that Barry reported on python-dev:
>>> 23000 .__class__ = bool
crashes in the deallocator. This was because int inherited tp_free
from object, which uses the default allocator.
2.2. Bugfix candidate.
states can be for this function, and ensure that only AttributeErrors
are masked. Any other exception raised via the equivalent of
getattr(cls, '__bases__') should be propagated up.
abstract_issubclass(): If abstract_get_bases() returns NULL, we must
call PyErr_Occurred() to see if an exception is being propagated, and
return -1 or 0 as appropriate. This is the specific fix for a problem
whereby if getattr(derived, '__bases__') raised an exception, an
"undetected error" would occur (under a debug build). This nasty
situation was uncovered when writing a security proxy extension type
for the Zope3 project, where the security proxy raised a Forbidden
exception on getattr of __bases__.
PyObject_IsInstance(), PyObject_IsSubclass(): After both calls to
abstract_get_bases(), where we're setting the TypeError if the return
value is NULL, we must first check to see if an exception occurred,
and /not/ mask an existing exception.
Neil Schemenauer should double check that these changes don't break
his ExtensionClass examples (there aren't any test cases for those
examples and abstract_get_bases() was added by him in response to
problems with ExtensionClass). Neil, please add test cases if
possible!
I belive this is a bug fix candidate for Python 2.2.2.
http://www.python.org/sf/444708
This adds the optional argument for str.strip
to unicode.strip too and makes it possible
to call str.strip with a unicode argument
and unicode.strip with a str argument.
+ Continued looping until n bytes in the buffer have been filled, not
just when n bytes have been read from the file. This repairs the
bug that f.readlines() only sucked up the first 8192 bytes of the file
on Windows when universal newlines was enabled and f was opened in
U mode (see Python-Dev -- this was the ultimate cause of the
test_inspect.py failure).
+ Changed prototye to take a char* buffer (void* doesn't make much sense).
+ Squashed size_t vs int mismatches (in particular, besides the unsigned
vs signed distinction, size_t may be larger than int).
+ Gets out under all error conditions now (it's possible for fread() to
suffer an error even if it returns a number larger than 0 -- any
"short read" is an error or EOF condition).
+ Rearranged and simplified declarations.
pointers is a signed type. Changing "allocated" to a signed int makes
undetected overflow more likely, but there was no overflow detection
before either.
PyFrame_FastToLocals() and PyFrame_LocalsToFast() had a return if
f_nlocals was 0. I think this was a holdover from the pre 2.1 days
when regular locals were the only kind of local variables.
The change makes it possible to use a free variable in eval or exec if
it the variable is also used elsewhere in the same block, which is
what the documentation says.
consistency checks, enabled only in a debug (Py_DEBUG) build. Note that
this never gets called automatically unless PYMALLOC_DEBUG is #define'd
too, and the envar PYTHONMALLOCSTATS exists.
Change type_get_doc (the get function for __doc__) to look in tp_dict
more often, and if it finds a descriptor in tp_dict, to call it (with
a NULL instance). This means you can add a __doc__ descriptor to a
new-style class that returns instance docs when called on an instance,
and class docs when called on a class -- or the same docs in either
case, but lazily computed.
I'll also check this into the 2.2 maintenance branch.
PyNumber_InPlaceMultiply insisted on calling sq_inplace_repeat if it
existed, even if nb_inplace_multiply also existed and the arguments
weren't right for sq_inplace_repeat. Change this to only use
sq_inplace_repeat if nb_inplace_multiply isn't defined.
Bugfix candidate.
Add a method zfill to str, unicode and UserString and change
Lib/string.py accordingly.
This activates the zfill version in unicodeobject.c that was
commented out and implements the same in stringobject.c. It also
adds the test for unicode support in Lib/string.py back in and
uses repr() instead() of str() (as it was before Lib/string.py 1.62)
Complex numbers implement divmod() and //, neither of which makes one
lick of sense. Unfortunately this is documented, so I'm adding a
deprecation warning now, so we can delete this silliness, oh, around
2005 or so.
Bugfix candidate (At least for 2.2.2, I think.)
Highlights: import and friends will understand any of \r, \n and \r\n
as end of line. Python file input will do the same if you use mode 'U'.
Everything can be disabled by configuring with --without-universal-newlines.
See PEP278 for details.
Added code to call this when PYMALLOC_DEBUG is enabled, and envar
PYTHONMALLOCSTATS is set, whenever a new arena is obtained and once
late in the Python shutdown process.
Put a bound on the number of frameobjects that can live in the
frameobject free_list.
Am also backporting to 2.2. I don't intend to backport to 2.1 (too
much work -- lots of cyclic structures leak there, and the GC API).
Add optional arg to string methods strip(), lstrip(), rstrip().
The optional arg specifies characters to delete.
Also for UserString.
Still to do:
- Misc/NEWS
- LaTeX docs (I did the docstrings though)
- Unicode methods, and Unicode support in the string methods.
_PyObject_DebugMalloc: explicitly cast PyObject_Malloc's result to the
target pointer type.
_PyObject_DebugDumpStats: change decl of arena_alignment from unsigned
int to unsigned long.
This is for the 2.3 release only (it's new code).
most of the work. In particular, if the underlying realloc is able to
grow the memory block in place, great (this routine used to do a fresh
malloc + memcpy every time a block grew). BTW, I'm not so keen here on
avoiding possible quadratic-time realloc patterns as I am on making
the debug pymalloc more invisible (the more it uses memory "just like"
the underlying allocator, the better the chance that a suspected memory
corruption bug won't vanish when the debug malloc is turned on).
PyMem_{Del, DEL} doesn't work yet (compilation problems).
pyport.h: _PyMem_EXTRA is gone.
pmem.h: Repaired comments. PyMem_{Malloc, MALLOC} and
PyMem_{Realloc, REALLOC} now make the same x-platform guarantees when
asking for 0 bytes, and when passing a NULL pointer to the latter.
object.c: PyMem_{Malloc, Realloc} just call their macro versions
now, since the latter take care of the x-platform 0 and NULL stuff
by themselves now.
pypcre.c, grow_stack(): So sue me. On two lines, this called
PyMem_RESIZE to grow a "const" area. It's not legit to realloc a
const area, so the compiler warned given the new expansion of
PyMem_RESIZE. It would have gotten the same warning before if it
had used PyMem_Resize() instead; the older macro version, but not the
function version, silently cast away the constness. IMO that was a wrong
thing to do, and the docs say the macro versions of PyMem_xyz are
deprecated anyway. If somebody else is resizing const areas with the
macro spelling, they'll get a warning when they recompile now too.
The bug report pointed out a bogosity in the comment block explaining
thread safety for arena management. Repaired that comment, repaired a
couple others while I was at it, and added an assert.
_PyMalloc_DebugRealloc: If this needed to get more memory, but couldn't,
it erroneously freed the original memory. Repaired that.
This is for 2.3 only (unless we decide to backport the new pymalloc).
open_the_file: Some (not all) flavors of Windows set errno to EINVAL
when passed a syntactically invalid filename. Python turned that into an
incomprehensible complaint about the mode string. Fixed by special-casing
MSVC.
when PyType_Ready() was called, if ob_type was found to be NULL, it
was always set to &PyType_Type; now it is set to base->ob_type,
where base is tp_base, defaulting to &PyObject_Type.
- PyType_Ready() accidentally did not inherit tp_is_gc; now it does.
Bugfix candidate.
runtime multiplications and divisions, via the scheme developed with
Vladimir Marangozov on Python-Dev. The pool_header struct loses its
capacity member, but gains nextoffset and maxnextoffset members; this
still leaves it at 32 bytes on a 32-bit box (it has to be padded to a
multiple of 8 bytes).
speeds up __getitem__ and __setitem__ in subclasses of built-in
sequences.
It's much revised because I took the opportunity to refactor the code
somewhat (moving a large section of duplicated code to a helper
function) and added comments to a series of functions.
what these do given a 0 size argument. This is so that when pymalloc
is enabled, we don't need to wrap pymalloc calls in goofy little
routines special-casing 0. Note that it's virtually impossible to meet
the doc's promise that malloc(0) will never return NULL; this makes a
best effort, but not an insane effort. The code does promise that
realloc(not-NULL, 0) will never return NULL (malloc(0) is much harder).
_PyMalloc_Realloc: Changed to take over all requests for 0 bytes, and
rearranged to be a little quicker in expected cases.
All over the place: when resorting to the platform allocator, call
free/malloc/realloc directly, without indirecting thru macros. This
should avoid needing a nightmarish pile of #ifdef-ery if PYMALLOC_DEBUG
is changed so that pymalloc takes over all Py(Mem, Object} memory
operations (which would add useful debugging info to PyMem_xyz
allocations too).
PEP 285. Everything described in the PEP is here, and there is even
some documentation. I had to fix 12 unit tests; all but one of these
were printing Boolean outcomes that changed from 0/1 to False/True.
(The exception is test_unicode.py, which did a type(x) == type(y)
style comparison. I could've fixed that with a single line using
issubtype(x, type(y)), but instead chose to be explicit about those
places where a bool is expected.
Still to do: perhaps more documentation; change standard library
modules to return False/True from predicates.
possible pool states. I think it's much clearer now.
Added a new long overdue block-management overview comment block.
I believe the comments are in good shape now.
Added two comments about possible small optimizations (one getting rid
of runtime multiplications at the cost of a new pool_header member; the
other getting rid of runtime divisions and the pool_header capacity
member, at the cost of a static const vector of 32 uints).
This displays stats about the # of arenas, pools, blocks and bytes, to
stderr, both used and reserved but unused.
CAUTION: Because PYMALLOC_DEBUG is on, the debug malloc routine adds
16 bytes to each request. This makes each block appear two size classes
higher than it would be if PYMALLOC_DEBUG weren't on.
So far, playing with this confirms the obvious: there's a lot of activity
in the "small dict" size class, but nothing in the core makes any use of
the 8-byte or 16-byte classes.
the code so that the most frequent cases come first. Added comments.
Found a hidden assumption that a pool contains room for at least two
blocks, and added an assert to catch a violation if it ever happens in
a place where that matters. Gave the normal "I allocated this block"
case a longer basic block to work with before it has to do its first
branch (via breaking apart an embedded assignment in an "if", and
hoisting common code out of both branches).
address obtained from system malloc/realloc without holding the GIL.
When the vector of arena base addresses has to grow, the old vector is
deliberately leaked. This makes "stale" x-thread references safe.
arenas and narenas are also declared volatile, and changed in an order
that prevents a thread from picking up a value of narenas too large
for the value of arenas it sees.
Added more asserts.
Fixed an old inaccurate comment.
Added a comment explaining why it's safe to call pymalloc free/realloc
with an address obtained from system malloc/realloc even when arenas is
still NULL (this is obscure, since the ADDRESS_IN_RANGE macro
appears <wink> to index into arenas).
this. But added an overflow check just in case there is.
Got rid of the ushort macro. It wasn't used anymore (it was only used
in the no-longer-exists off_t macro), and there's no plausible use for it.
waste the first pool if malloc happens to return a pool-aligned address.
This means the number of pools per arena can now vary by 1. Unfortunately,
the code counted up from 0 to a presumed constant number of pools. So
changed the increasing "watermark" counter to a decreasing "nfreepools"
counter instead, and fiddled various stuff accordingly. This also allowed
getting rid of two more macros.
Also changed the code to align the first address to a pool boundary
instead of a page boundary. These are two parallel sets of macro #defines
that happen to be identical now, but the page macros are in theory more
restrictive (bigger), and there's simply no reason I can see that it
wasn't aligning to the less restrictive pool size all along (the code
only relies on pool alignment).
Hmm. The "page size" macros aren't used for anything *except* defining
the pool size macros, and the comments claim the latter isn't necessary.
So this has the feel of a layer of indirection that doesn't serve a
purpose; should probably get rid of the page macros now.
are called without the GIL. It's incredibly unlikely to fail, but I can't
make this bulletproof without either adding a lock for exclusion, or
giving up on growing the arena base-address vector (it would be safe if
this were a static array).
+ A new scheme for determining whether an address belongs to a pymalloc
arena. This should be 100% reliable. The poolp->pooladdr and
poolp->magic members are gone. A new poolp->arenaindex member takes
their place. Note that the pool header overhead doesn't actually
shrink, though, since the header is padded to a multiple of 8 bytes.
+ _PyMalloc_Free and _PyMalloc_Realloc should now be safe to call for
any legit address, whether obtained from a _PyMalloc function or from
the system malloc/realloc. It should even be safe to call
_PyMalloc_Free when *not* holding the GIL, provided that the passed-in
address was obtained from system malloc/realloc. Since this is
accomplished without any locks, you better believe the code is subtle.
I hope it's sufficiently commented.
+ The above implies we don't need the new PyMalloc_{New, NewVar, Del}
API anymore, and could switch back to PyObject_XXX without breaking
existing code mixing PyObject_XXX with PyMem_{Del, DEL, Free, FREE}.
Nothing is done here about that yet, and I'd like to see this new
code exercised more first.
+ The small object threshhold is boosted to 256 (the max). We should
play with that some more, but the old 64 was way too small for 2.3.
+ Getting a new arena is now done via new function new_arena().
+ Removed some unused macros, and squashed out some macros that were
used only once to define other macros.
+ Arenas are no longer linked together. A new vector of arena base
addresses had to be created anyway to make address classification
bulletproof.
+ A lot of the patch size is an illusion: given the way address
classification works now, it was more convenient to switch the
sense of the prime "if" tests in the realloc and free functions,
so the "if" and "else" blocks got swapped.
+ Assorted minor code, comment and whitespace cleanup.
Back to the Windows installer <wink>.
The fix makes it possible to call PyObject_GC_UnTrack() more than once
on the same object, and then move the PyObject_GC_UnTrack() call to
*before* the trashcan code is invoked.
BUGFIX CANDIDATE!
descriptor, as used for the tp_methods slot of a type. These new flag
bits are both optional, and mutually exclusive. Most methods will not
use either. These flags are used to create special method types which
exist in the same namespace as normal methods without having to use
tedious construction code to insert the new special method objects in
the type's tp_dict after PyType_Ready() has been called.
If METH_CLASS is specified, the method will represent a class method
like that returned by the classmethod() built-in.
If METH_STATIC is specified, the method will represent a static method
like that returned by the staticmethod() built-in.
These flags may not be used in the PyMethodDef table for modules since
these special method types are not meaningful in that case; a
ValueError will be raised if these flags are found in that context.
Assorted: bump the serial number via a trivial new bumpserialno()
function. The point is to give a single place to set a breakpoint when
waiting for a specific serial number.
of get_line. This makes test_bufio finish in 1.7 seconds instead of 57
seconds on my machine (with Py_DEBUG defined).
Also, rename the local variables n1 and n2 to used_v_size and
total_v_size.
When WITH_PYMALLOC is defined, define PYMALLOC_DEBUG to enable the debug
allocator. This can be done independent of build type (release or debug).
A debug build automatically defines PYMALLOC_DEBUG when pymalloc is
enabled. It's a detected error to define PYMALLOC_DEBUG when pymalloc
isn't enabled.
Two debugging entry points defined only under PYMALLOC_DEBUG:
+ _PyMalloc_DebugCheckAddress(const void *p) can be used (e.g., from gdb)
to sanity-check a memory block obtained from pymalloc. It sprays
info to stderr (see next) and dies via Py_FatalError if the block is
detectably damaged.
+ _PyMalloc_DebugDumpAddress(const void *p) can be used to spray info
about a debug memory block to stderr.
A tiny start at implementing "API family" checks isn't good for
anything yet.
_PyMalloc_DebugRealloc() has been optimized to do little when the new
size is <= old size. However, if the new size is larger, it really
can't call the underlying realloc() routine without either violating its
contract, or knowing something non-trivial about how the underlying
realloc() works. A memcpy is always done in this case.
This was a disaster for (and only) one of the std tests: test_bufio
creates single text file lines up to a million characters long. On
Windows, fileobject.c's get_line() uses the horridly funky
getline_via_fgets(), which keeps growing and growing a string object
hoping to find a newline. It grew the string object 1000 bytes each
time, so for a million-character string it took approximately forever
(I gave up after a few minutes).
So, also:
fileobject.c, getline_via_fgets(): When a single line is outrageously
long, grow the string object at a mildly exponential rate, instead of
just 1000 bytes at a time.
That's enough so that a debug-build test_bufio finishes in about 5 seconds
on my Win98SE box. I'm curious to try this on Win2K, because it has very
different memory behavior than Win9X, and test_bufio always took a factor
of 10 longer to complete on Win2K. It *could* be that the endless
reallocs were simply killing it on Win2K even in the release build.
Also move all _PyMalloc_XXX entry points into obmalloc.c.
The Windows build works fine.
The Unix build is changed here (Makefile.pre.in), but not tested.
No other platform's build process has been fiddled.
Konrad was too kind. Not only did it raise an exception, the specific
exception it raised made no sense. These are old bugs in complex_pow()
and friends:
1. Raising 0 to a negative power isn't a range error, it's a domain
error, so changed c_pow() to set errno to EDOM in that case instead
of ERANGE.
2. Changed complex_pow() to:
A. Used the Py_ADJUST_ERANGE2 macro to try to clear errno of a spurious
ERANGE error due to underflow in the libm pow() called by c_pow().
B. Produced different exceptions depending on the errno value:
i) For errno==EDOM, raise ZeroDivisionError instead of ValueError.
This is for consistency with the non-complex cases 0.0**-2 and
0**-2 and 0L**-2.
ii) For errno==ERANGE, raise OverflowError.
Bugfix candidate.
The proper fix is not quite what was submitted; it's really better to
take the class of the object passed rather than calling PyMethod_New
with NULL pointer args, because that can then cause other core dumps
later.
I also added a testcase for the fix to classmethods() in test_descr.py.
I've already applied this to the 2.2 branch.
As promised in my response to the bug report, I'm not really fixing
it; in fact, one could argule over what the proper fix should do.
Instead, I'm adding a little magic that raises TypeError if you try to
pickle an instance of a class that has __slots__ but doesn't define or
override __getstate__. This is done by adding a bozo __getstate__
that always raises TypeError.
There were several places that assumed the md_dict field was always
set, but it needn't be. Fixed these to be more careful.
I changed PyModule_GetDict() to initialize md_dict to a new dictionary
if it's NULL.
Bugfix candidate.
and (b) stop trying to prevent file growth.
Beef up the file.truncate() docs.
Change test_largefile.py to stop assuming that f.truncate() moves the
file pointer to the truncation point, and to verify instead that it leaves
the file position alone. Remove the test for what happens when a
specified size exceeds the original file size (it's ill-defined, according
to the Single Unix Spec).
dropping MS's inadequate _chsize() function. This was inspired by
SF patch 498109 ("fileobject truncate support for win32"), which I
rejected.
libstdtypes.tex: Someone who knows should update the availability
blurb. For example, if it's available on Linux, it would be good to
say so.
test_largefile: Uncommented the file.truncate() tests, and reworked to
do more. The old comment about "permission errors" in the truncation
tests under Windows was almost certainly due to that the file wasn't open
for *write* access at this point, so of course MS wouldn't let you
truncate it. I'd be appalled if a Unixish system did.
CAUTION: Someone should run this test on Linux (etc) too. The
truncation part was commented out before. Note that test_largefile isn't
run by default.
Adapter from SF patch 528038; fixes SF bug 527816.
The wrapper for __nonzero__ should be wrap_inquiry rather than
wrap_unaryfunc, since the slot returns an int, not a PyObject *.
Another year in the quest to out-guess random C behavior.
Added macros Py_ADJUST_ERANGE1(X) and Py_ADJUST_ERANGE2(X, Y). The latter
is useful for functions with complex results. Two corrections to errno-
after-libm-call are attempted:
1. If the platform set errno to ERANGE due to underflow, clear errno.
Some unknown subset of libm versions and link options do this. It's
allowed by C89, but I never figured anyone would do it.
2. If the platform did not set errno but overflow occurred, force
errno to ERANGE. C89 required setting errno to ERANGE, but C99
doesn't. Some unknown subset of libm versions and link options do
it the C99 way now.
Bugfix candidate, but hold off until some Linux people actually try it,
with and without -lieee. I'll send a help plea to Python-Dev.
PyNumber_Add() tries the nb_add slot first, then falls back to
sq_concat. However, tt didn't check the return value of sq_concat.
If sq_concat returns NotImplemented, raise the standard TypeError.
[ 526072 ] pickling os.stat results round II
structseq's constructors can now take "invisible" fields in a dict.
Gave the constructors better error messages.
their __reduce__ method puts these fields in a dict.
(this is all in aid of getting os.stat_result's to pickle portably)
Also fixes
[ 526039 ] devious code can crash structseqs
Thought needed about how much of this counts as a bugfix. Certainly
#526039 needs to be fixed.
platform realloc(p, 0) returns NULL, so MALLOC_ZERO_RETURNS_NULL can
be correctly undefined yet realloc(p, 0) can return NULL anyway.
Prevent realloc(p, 0) doing free(p) and returning NULL via a different
hack. Would probably be better to get rid of MALLOC_ZERO_RETURNS_NULL
entirely.
Bugfix candidate.
Due to the bizarre definition of _PyLong_Copy(), creating an instance
of a subclass of long with a negative value could cause core dumps
later on. Unfortunately it looks like the behavior of _PyLong_Copy()
is quite intentional, so the fix is more work than feels comfortable.
This fix is almost, but not quite, the code that Naofumi Honda added;
in addition, I added a test case.
Objects/
fileobject.c
stringobject.c
unicodeobject.c
This commit doesn't include the cleanup patches for stringobject.c and
unicodeobject.c which are shown separately in the patch manager. Those
patches will be regenerated and applied in a subsequent commit, so as
to preserve a fallback position (this commit to those files).
Fix for the UTF-8 decoder: it will now accept isolated surrogates
(previously it raised an exception which causes round-trips to
fail).
Added new tests for UTF-8 round-trip safety (we rely on UTF-8 for
marshalling Unicode objects, so we better make sure it works for
all Unicode code points, including isolated surrogates).
Bumped the PYC magic in a non-standard way -- please review. This
was needed because the old PYC format used illegal UTF-8 sequences
for isolated high surrogates which now raise an exception.
NULL, so that you can call PyType_Ready() to initialize a type that
is to be separately compiled with C on Windows.
inherit_special(): Add a long comment explaining that you have to set
tp_new if your base class is PyBaseObject_Type.
Fix for SF bug #492345. (I could've sworn I checked this in, but
apparently I didn't!)
This code:
class Classic:
pass
class New(Classic):
__metaclass__ = type
attempts to create a new-style class with only classic bases -- but it
doesn't work right. Attempts to fix it so it works caused problems
elsewhere, so I'm now raising a TypeError in this case.
delivered bizarre results. Check float_divmod for a Py_NotImplemented
return and pass it along (instead of treating Py_NotImplemented as a
2-tuple).
CONVERT_TO_DOUBLE: Added comments; this macro is obscure.
PyDict_UpdateFromSeq2(): removed it.
PyDict_MergeFromSeq2(): made it public and documented it.
PyDict_Merge() docs: updated to reveal <wink> that the second
argument can be any mapping object.
no get function was defined, the property's doc string was
inaccessible. This was because the test for prop_get was made
*before* the test for a NULL/None object argument.
Also changed the property class defined in Python in a comment to test
for NULL to decide between get and delete; this makes it less Python
but then, assigning None to a property doesn't delete it!
PyString_FromString():
Since the length of the string is already being stored in size,
changed the strcpy() to a memcpy() for a small speed improvement.
out the for loop at the end intended to zero out new items wasn't
doing anything, because sv->ob_size was already equal to newsize. The
fix slightly refactors the function, introducing a variable oldsize
and doing away with sizediff (which was used only once), and using
oldsize and newsize consistently. I also added comments explaining
what the two for loops do. (Looking at the CVS annotation of this
function, it's no miracle a bug crept in -- this has been patched by
many different folks! :-)
This is best reproduced by
while 1:
class U(unicode):
pass
U(u"xxxxxx")
The unicode_dealloc() code wasn't properly freeing the str and defenc
fields of the Unicode object when freeing a subtype instance. Fixed
this by a subtle refactoring that actually reduces the amount of code
slightly.
PyCell_Set() incremenets the reference count, so the earlier XINCREF
causes a leak.
Also make a number of small performance improvements to the code on
the assumption that most of the time variables are not rebound across
a FastToLocals() / LocalsToFast() pair.
Replace uses of PyCell_Set() and PyCell_Get() with PyCell_SET() and
PyCell_GET(), since the frame is guaranteed to contain cells.
Add a missing DECREF in an obscure corner. If the str() or repr() of
an object passed to a string interpolation -- e.g. "%s" % obj --
returns a non-string, the returned object was leaked.
Repair an indentation glitch.
Replace a bunch of PyString_AsString() calls (and their ilk) with
macros.
It was easier than I thought, assuming that no other things contribute
to the instance size besides slots -- a pretty good bet. With a test
suite, no less!
happy if one could delete the __dict__ attribute of an instance. I
love to make Jim happy, so here goes...
- New-style objects now support deleting their __dict__. This is for
all intents and purposes equivalent to assigning a brand new empty
dictionary, but saves space if the object is not used further.
int_mul(): new and vastly simpler overflow checking. Whether it's
faster or slower will likely vary across platforms, favoring boxes
with fast floating point. OTOH, we no longer have to worry about
people shipping broken LONG_BIT definitions <0.9 wink>.
There's now a new structmember code, T_OBJECT_EX, which is used for
all __slot__ variables (except __weakref__, which has special behavior
anyway). This new code raises AttributeError when the variable is
NULL rather than converting NULL to None.
string object (or a Unicode that's trivially converted to ASCII).
PyObject_GetAttr(): add an 'else' to the Unicode test like
PyObject_SetAttr() already has.
Rather than tweaking the inheritance of type object slots (which turns
out to be too messy to try), this fix adds a __hash__ to the list and
dict types (the only mutable types I'm aware of) that explicitly
raises an error. This has the advantage that list.__hash__([]) also
raises an error (previously, this would invoke object.__hash__([]),
returning the argument's address); ditto for dict.__hash__.
The disadvantage for this fix is that 3rd party mutable types aren't
automatically fixed. This should be added to the rules for creating
subclassable extension types: if you don't want your object to be
hashable, add a tp_hash function that raises an exception.
Also, it's possible that I've forgotten about other mutable types for
which this should be done.
SF patch #480716 by Greg Chapman fixes the problem that super's
__get__ method always returns an instance of super, even when the
instance whose __get__ method is called is an instance of a subclass
of super.
Other issues fixed:
- super(C, C()).__class__ would return the __class__ attribute of C()
rather than the __class__ attribute of the super object. This is
confusing. To fix this, I decided to change the semantics of super
so that it only applies to code attributes, not to data attributes.
After all, overriding data attributes is not supported anyway.
- While super(C, x) carefully checked that x is an instance of C,
super(C).__get__(x) made no such check, allowing for a loophole.
This is now fixed.
slot_tp_descr_set(): When deleting an attribute described by a
descriptor implemented in Python, the descriptor's __del__ method is
called by the slot_tp_descr_set dispatch function. This is bogus --
__del__ already has a different meaning. Renaming this use of __del__
is renamed to __delete__.
Bugfix candidate.
int_repr(): we've never had a buffer big enough to hold the largest
possible result on a 64-bit box. Now that we're using snprintf instead
of sprintf, this can lead to nonsense results instead of random stack
corruption.
pass the buffer length. Stop using it. It should be deprecated, but too
late in the release cycle to do that now.
New static format_float() does the same thing but requires passing the
buffer length too. Use it instead.
const char* instead of char*. The change is conceptually correct, and
indirectly fixes a compiler wng introduced when somebody else innocently
passed a const char* to this function.
sprintf() to PyOS_snprintf() for buffer overrun avoidance.
complex_print(), complex_repr(), complex_str(): Call complex_to_buf()
passing in sizeof(buf).
confusing error messages. If a new-style class has no sequence or
mapping behavior, attempting to use the indexing notation with a
non-integer key would complain that the sequence index must be an
integer, rather than complaining that the operation is not supported.
of multiple inheritance from a mix of new- and classic-style classes.
This is his patch, plus a start at some test cases from me. Will check
in more, plus a NEWS blurb, later tonight.
object, so the "Metroworks only" section should not decref it in case
of error (the caller is responsible for decref'ing in case of error --
and does).
helping for types that defined tp_richcmp but not tp_compare, although
that's when it's most valuable, and strings moved into that category
since the fast path was first introduced. Now it helps for same-type
non-Instance objects that define rich or 3-way compares.
For all the edits here, the rest just amounts to moving the fast path from
do_richcmp into PyObject_RichCompare, saving a layer of function call
(measurable on my box!). This loses when NESTING_LIMIT is exceeded, but I
don't care about that (fast-paths are for normal cases, not pathologies).
Also added a tasteful <wink> label to get out of PyObject_RichCompare, as
the if/else nesting in this routine was getting incomprehensible.
Michael W. Hudson
Guido van Rossum
Raymond Hettinger
Neal Norwitz
Thomas Heller
Tim Peters
Marc-André Lemburg
Barry Warsaw
Walter Dörwald
Martin v. Löwis
Fred Drake
Skip Montanaro
Jeremy Hylton
Jack Jansen
Neil Schemenauer
Andrew MacIntyre
Sjoerd Mullender