As promised in my response to the bug report, I'm not really fixing
it; in fact, one could argule over what the proper fix should do.
Instead, I'm adding a little magic that raises TypeError if you try to
pickle an instance of a class that has __slots__ but doesn't define or
override __getstate__. This is done by adding a bozo __getstate__
that always raises TypeError.
There were several places that assumed the md_dict field was always
set, but it needn't be. Fixed these to be more careful.
I changed PyModule_GetDict() to initialize md_dict to a new dictionary
if it's NULL.
Bugfix candidate.
and (b) stop trying to prevent file growth.
Beef up the file.truncate() docs.
Change test_largefile.py to stop assuming that f.truncate() moves the
file pointer to the truncation point, and to verify instead that it leaves
the file position alone. Remove the test for what happens when a
specified size exceeds the original file size (it's ill-defined, according
to the Single Unix Spec).
dropping MS's inadequate _chsize() function. This was inspired by
SF patch 498109 ("fileobject truncate support for win32"), which I
rejected.
libstdtypes.tex: Someone who knows should update the availability
blurb. For example, if it's available on Linux, it would be good to
say so.
test_largefile: Uncommented the file.truncate() tests, and reworked to
do more. The old comment about "permission errors" in the truncation
tests under Windows was almost certainly due to that the file wasn't open
for *write* access at this point, so of course MS wouldn't let you
truncate it. I'd be appalled if a Unixish system did.
CAUTION: Someone should run this test on Linux (etc) too. The
truncation part was commented out before. Note that test_largefile isn't
run by default.
Adapter from SF patch 528038; fixes SF bug 527816.
The wrapper for __nonzero__ should be wrap_inquiry rather than
wrap_unaryfunc, since the slot returns an int, not a PyObject *.
Another year in the quest to out-guess random C behavior.
Added macros Py_ADJUST_ERANGE1(X) and Py_ADJUST_ERANGE2(X, Y). The latter
is useful for functions with complex results. Two corrections to errno-
after-libm-call are attempted:
1. If the platform set errno to ERANGE due to underflow, clear errno.
Some unknown subset of libm versions and link options do this. It's
allowed by C89, but I never figured anyone would do it.
2. If the platform did not set errno but overflow occurred, force
errno to ERANGE. C89 required setting errno to ERANGE, but C99
doesn't. Some unknown subset of libm versions and link options do
it the C99 way now.
Bugfix candidate, but hold off until some Linux people actually try it,
with and without -lieee. I'll send a help plea to Python-Dev.
PyNumber_Add() tries the nb_add slot first, then falls back to
sq_concat. However, tt didn't check the return value of sq_concat.
If sq_concat returns NotImplemented, raise the standard TypeError.
[ 526072 ] pickling os.stat results round II
structseq's constructors can now take "invisible" fields in a dict.
Gave the constructors better error messages.
their __reduce__ method puts these fields in a dict.
(this is all in aid of getting os.stat_result's to pickle portably)
Also fixes
[ 526039 ] devious code can crash structseqs
Thought needed about how much of this counts as a bugfix. Certainly
#526039 needs to be fixed.
platform realloc(p, 0) returns NULL, so MALLOC_ZERO_RETURNS_NULL can
be correctly undefined yet realloc(p, 0) can return NULL anyway.
Prevent realloc(p, 0) doing free(p) and returning NULL via a different
hack. Would probably be better to get rid of MALLOC_ZERO_RETURNS_NULL
entirely.
Bugfix candidate.
Due to the bizarre definition of _PyLong_Copy(), creating an instance
of a subclass of long with a negative value could cause core dumps
later on. Unfortunately it looks like the behavior of _PyLong_Copy()
is quite intentional, so the fix is more work than feels comfortable.
This fix is almost, but not quite, the code that Naofumi Honda added;
in addition, I added a test case.
Objects/
fileobject.c
stringobject.c
unicodeobject.c
This commit doesn't include the cleanup patches for stringobject.c and
unicodeobject.c which are shown separately in the patch manager. Those
patches will be regenerated and applied in a subsequent commit, so as
to preserve a fallback position (this commit to those files).
Fix for the UTF-8 decoder: it will now accept isolated surrogates
(previously it raised an exception which causes round-trips to
fail).
Added new tests for UTF-8 round-trip safety (we rely on UTF-8 for
marshalling Unicode objects, so we better make sure it works for
all Unicode code points, including isolated surrogates).
Bumped the PYC magic in a non-standard way -- please review. This
was needed because the old PYC format used illegal UTF-8 sequences
for isolated high surrogates which now raise an exception.
NULL, so that you can call PyType_Ready() to initialize a type that
is to be separately compiled with C on Windows.
inherit_special(): Add a long comment explaining that you have to set
tp_new if your base class is PyBaseObject_Type.
Fix for SF bug #492345. (I could've sworn I checked this in, but
apparently I didn't!)
This code:
class Classic:
pass
class New(Classic):
__metaclass__ = type
attempts to create a new-style class with only classic bases -- but it
doesn't work right. Attempts to fix it so it works caused problems
elsewhere, so I'm now raising a TypeError in this case.
delivered bizarre results. Check float_divmod for a Py_NotImplemented
return and pass it along (instead of treating Py_NotImplemented as a
2-tuple).
CONVERT_TO_DOUBLE: Added comments; this macro is obscure.
PyDict_UpdateFromSeq2(): removed it.
PyDict_MergeFromSeq2(): made it public and documented it.
PyDict_Merge() docs: updated to reveal <wink> that the second
argument can be any mapping object.
no get function was defined, the property's doc string was
inaccessible. This was because the test for prop_get was made
*before* the test for a NULL/None object argument.
Also changed the property class defined in Python in a comment to test
for NULL to decide between get and delete; this makes it less Python
but then, assigning None to a property doesn't delete it!
PyString_FromString():
Since the length of the string is already being stored in size,
changed the strcpy() to a memcpy() for a small speed improvement.
out the for loop at the end intended to zero out new items wasn't
doing anything, because sv->ob_size was already equal to newsize. The
fix slightly refactors the function, introducing a variable oldsize
and doing away with sizediff (which was used only once), and using
oldsize and newsize consistently. I also added comments explaining
what the two for loops do. (Looking at the CVS annotation of this
function, it's no miracle a bug crept in -- this has been patched by
many different folks! :-)
This is best reproduced by
while 1:
class U(unicode):
pass
U(u"xxxxxx")
The unicode_dealloc() code wasn't properly freeing the str and defenc
fields of the Unicode object when freeing a subtype instance. Fixed
this by a subtle refactoring that actually reduces the amount of code
slightly.
PyCell_Set() incremenets the reference count, so the earlier XINCREF
causes a leak.
Also make a number of small performance improvements to the code on
the assumption that most of the time variables are not rebound across
a FastToLocals() / LocalsToFast() pair.
Replace uses of PyCell_Set() and PyCell_Get() with PyCell_SET() and
PyCell_GET(), since the frame is guaranteed to contain cells.
Add a missing DECREF in an obscure corner. If the str() or repr() of
an object passed to a string interpolation -- e.g. "%s" % obj --
returns a non-string, the returned object was leaked.
Repair an indentation glitch.
Replace a bunch of PyString_AsString() calls (and their ilk) with
macros.
It was easier than I thought, assuming that no other things contribute
to the instance size besides slots -- a pretty good bet. With a test
suite, no less!
happy if one could delete the __dict__ attribute of an instance. I
love to make Jim happy, so here goes...
- New-style objects now support deleting their __dict__. This is for
all intents and purposes equivalent to assigning a brand new empty
dictionary, but saves space if the object is not used further.
int_mul(): new and vastly simpler overflow checking. Whether it's
faster or slower will likely vary across platforms, favoring boxes
with fast floating point. OTOH, we no longer have to worry about
people shipping broken LONG_BIT definitions <0.9 wink>.
There's now a new structmember code, T_OBJECT_EX, which is used for
all __slot__ variables (except __weakref__, which has special behavior
anyway). This new code raises AttributeError when the variable is
NULL rather than converting NULL to None.
string object (or a Unicode that's trivially converted to ASCII).
PyObject_GetAttr(): add an 'else' to the Unicode test like
PyObject_SetAttr() already has.
Rather than tweaking the inheritance of type object slots (which turns
out to be too messy to try), this fix adds a __hash__ to the list and
dict types (the only mutable types I'm aware of) that explicitly
raises an error. This has the advantage that list.__hash__([]) also
raises an error (previously, this would invoke object.__hash__([]),
returning the argument's address); ditto for dict.__hash__.
The disadvantage for this fix is that 3rd party mutable types aren't
automatically fixed. This should be added to the rules for creating
subclassable extension types: if you don't want your object to be
hashable, add a tp_hash function that raises an exception.
Also, it's possible that I've forgotten about other mutable types for
which this should be done.
SF patch #480716 by Greg Chapman fixes the problem that super's
__get__ method always returns an instance of super, even when the
instance whose __get__ method is called is an instance of a subclass
of super.
Other issues fixed:
- super(C, C()).__class__ would return the __class__ attribute of C()
rather than the __class__ attribute of the super object. This is
confusing. To fix this, I decided to change the semantics of super
so that it only applies to code attributes, not to data attributes.
After all, overriding data attributes is not supported anyway.
- While super(C, x) carefully checked that x is an instance of C,
super(C).__get__(x) made no such check, allowing for a loophole.
This is now fixed.
slot_tp_descr_set(): When deleting an attribute described by a
descriptor implemented in Python, the descriptor's __del__ method is
called by the slot_tp_descr_set dispatch function. This is bogus --
__del__ already has a different meaning. Renaming this use of __del__
is renamed to __delete__.
Bugfix candidate.
int_repr(): we've never had a buffer big enough to hold the largest
possible result on a 64-bit box. Now that we're using snprintf instead
of sprintf, this can lead to nonsense results instead of random stack
corruption.
pass the buffer length. Stop using it. It should be deprecated, but too
late in the release cycle to do that now.
New static format_float() does the same thing but requires passing the
buffer length too. Use it instead.
const char* instead of char*. The change is conceptually correct, and
indirectly fixes a compiler wng introduced when somebody else innocently
passed a const char* to this function.
sprintf() to PyOS_snprintf() for buffer overrun avoidance.
complex_print(), complex_repr(), complex_str(): Call complex_to_buf()
passing in sizeof(buf).
confusing error messages. If a new-style class has no sequence or
mapping behavior, attempting to use the indexing notation with a
non-integer key would complain that the sequence index must be an
integer, rather than complaining that the operation is not supported.
of multiple inheritance from a mix of new- and classic-style classes.
This is his patch, plus a start at some test cases from me. Will check
in more, plus a NEWS blurb, later tonight.
object, so the "Metroworks only" section should not decref it in case
of error (the caller is responsible for decref'ing in case of error --
and does).
helping for types that defined tp_richcmp but not tp_compare, although
that's when it's most valuable, and strings moved into that category
since the fast path was first introduced. Now it helps for same-type
non-Instance objects that define rich or 3-way compares.
For all the edits here, the rest just amounts to moving the fast path from
do_richcmp into PyObject_RichCompare, saving a layer of function call
(measurable on my box!). This loses when NESTING_LIMIT is exceeded, but I
don't care about that (fast-paths are for normal cases, not pathologies).
Also added a tasteful <wink> label to get out of PyObject_RichCompare, as
the if/else nesting in this routine was getting incomprehensible.
Try to ensure that divmod(-0.0, 1.0) -> (-0.0, +0.0) across platforms.
It always did on Windows, and still does. It didn't on Linux. Alas,
there's no platform-independent way to write a test case for this.
Bugfix candidate.
presence of NaNs. So pass the issue on to the platform libm fabs();
after all, fabs() is a std C function because you can't implement it
correctly in portable C89.
should just avoid calling it in the first place to avoid waiting for a repr
of a large object like a dict or list. The result of PyObject_Repr() was
being leaked as well.
Bugfix candidate!
XXX Remaining problems:
- The GC module doesn't know about these; I think it has its reasons
to disallow calling __del__, but for now, __del__ on new-style
objects is called when the GC module discards an object, for better
or for worse.
- The code to call a __del__ handler is really ridiculously
complicated, due to all the different debug #ifdefs. I've copied
this from the similar code in classobject.c, so I'm pretty sure I
did it right, but it's not pretty. :-(
- No tests yet.
object.h: Added PyType_CheckExact macro.
typeobject.c, type_new():
+ Use the new macro.
+ Assert that the arguments have the right types rather than do incomplete
runtime checks "sometimes".
+ If this isn't the 1-argument flavor() of type, and there aren't 3 args
total, produce a "types() takes 1 or 3 args" msg before
PyArg_ParseTupleAndKeywords produces a "takes exactly 3" msg.
the va_list until we are sure we have a format string and need to use it;
this avoid premature initialization and having to finalize it several
different places because of error returns.
and functions: we only need to call PyObject_ClearWeakRefs() if the weakref
list is non-NULL. Since these objects are common but weakrefs are still
unusual, saving the call at deallocation time makes a lot of sense.
PyObject_CallFunctionObArgs() and PyObject_CallMethodObArgs() have the
advantage that no format strings need to be parsed. The CallMethod
variant also avoids creating a new string object in order to retrieve
a method from an object as well.
outer level, the iterator protocol is used for memory-efficiency (the
outer sequence may be very large if fully materialized); at the inner
level, PySequence_Fast() is used for time-efficiency (these should
always be sequences of length 2).
dictobject.c, new functions PyDict_{Merge,Update}FromSeq2. These are
wholly analogous to PyDict_{Merge,Update}, but process a sequence-of-2-
sequences argument instead of a mapping object. For now, I left these
functions file static, so no corresponding doc changes. It's tempting
to change dict.update() to allow a sequence-of-2-seqs argument too.
Also changed the name of dictionary's keyword argument from "mapping"
to "x". Got a better name? "mapping_or_sequence_of_pairs" isn't
attractive, although more so than "mosop" <wink>.
abstract.h, abstract.tex: Added new PySequence_Fast_GET_SIZE function,
much faster than going thru the all-purpose PySequence_Size.
libfuncs.tex:
- Document dictionary().
- Fiddle tuple() and list() to admit that their argument is optional.
- The long-winded repetitions of "a sequence, a container that supports
iteration, or an iterator object" is getting to be a PITA. Many
months ago I suggested factoring this out into "iterable object",
where the definition of that could include being explicit about
generators too (as is, I'm not sure a reader outside of PythonLabs
could guess that "an iterator object" includes a generator call).
- Please check my curly braces -- I'm going blind <0.9 wink>.
abstract.c, PySequence_Tuple(): When PyObject_GetIter() fails, leave
its error msg alone now (the msg it produces has improved since
PySequence_Tuple was generalized to accept iterable objects, and
PySequence_Tuple was also stomping on the msg in cases it shouldn't
have even before PyObject_GetIter grew a better msg).
of the if block where it was before. The name is only used inside
that if block, but the storage is referenced outside it via the 's'
variable.
(This patch was part of SF patch #474590 -- RISC OS support.)
The C-code in fileobject.readinto(buffer) which parses
the arguments assumes that size_t is interchangeable
with int:
size_t ntodo, ndone, nnow;
if (f->f_fp == NULL)
return err_closed();
if (!PyArg_Parse(args, "w#", &ptr, &ntodo))
return NULL;
This causes a problem on Alpha / Tru64 / OSF1 v5.1
where size_t is a long and sizeof(long) != sizeof(int).
The patch I'm proposing declares ntodo as an int. An
alternative might be to redefine w# to expect size_t.
[We can't change w# because there are probably third party modules
relying on it. GvR]
response to a message by Laura Creighton on c.l.py. E.g.
>>> 0+''
TypeError: unsupported operand types for +: 'int' and 'str'
(previously this did not mention the operand types)
>>> ''+0
TypeError: cannot concatenate 'str' and 'int' objects
There really isn't a good reason for instance method objects to have
their own __dict__, __doc__ and __name__ properties that just delegate
the request to the function (callable); the default attribute behavior
already does this.
The test suite had to be fixed because the error changes from
TypeError to AttributeError.
'slotdef' structure typedef and 'struct wrapperbase'. By adding the
wrapper docstrings to the slotdef structure, the slotdefs array can
serve as the data structure that drives add_operators(); the wrapper
descriptor contains a pointer to slotdef structure. This replaces
lots of custom code from add_operators() by a loop over the slotdefs
array, and does away with all the tab_xxx tables.
This patch implements what we have discussed on python-dev late in
September: str(obj) and unicode(obj) should behave similar, while
the old behaviour is retained for unicode(obj, encoding, errors).
The patch also adds a new feature with which objects can provide
unicode(obj) with input data: the __unicode__ method. Currently no
new tp_unicode slot is implemented; this is left as option for the
future.
Note that PyUnicode_FromEncodedObject() no longer accepts Unicode
objects as input. The API name already suggests that Unicode
objects do not belong in the list of acceptable objects and the
functionality was only needed because
PyUnicode_FromEncodedObject() was being used directly by
unicode(). The latter was changed in the discussed way:
* unicode(obj) calls PyObject_Unicode()
* unicode(obj, encoding, errors) calls PyUnicode_FromEncodedObject()
One thing left open to discussion is whether to leave the
PyUnicode_FromObject() API as a thin API extension on top of
PyUnicode_FromEncodedObject() or to turn it into a (macro) alias
for PyObject_Unicode() and deprecate it. Doing so would have some
surprising consequences though, e.g. u"abc" + 123 would turn out
as u"abc123"...
[Marc-Andre didn't have time to check this in before the deadline. I
hope this is OK, Marc-Andre! You can still make changes and commit
them on the trunk after the branch has been made, but then please mail
Barry a context diff if you want the change to be merged into the
2.2b1 release branch. GvR]
the left-hand operand may not be the proxy in all cases. If it isn't,
we end up doing two things: a) unwrapping something that isn't a
PyWeakReference (later resulting in a core dump) and b) passing a
proxy as the right-hand operand anyway, even though that can't be
handled by the actual handler (maybe eventually causing a core dump).
This is fixed by always unwrapping all the proxies involved before
passing anything to the actual handler.
isinstance() now allows any object as the first argument and a class, a
type or something with a __bases__ tuple attribute for the second
argument. This closes SF patch #464992.
object.c, PyObject_Str: Don't try to optimize anything except exact
string objects here; in particular, let str subclasses go thru tp_str,
same as non-str objects. This allows overrides of tp_str to take
effect.
stringobject.c:
+ string_print (str's tp_print): If the argument isn't an exact string
object, get one from PyObject_Str.
+ string_str (str's tp_str): Make a genuine-string copy of the object if
it's of a proper str subclass type. str() applied to a str subclass
that doesn't override __str__ ends up here.
test_descr.py: New str_of_str_subclass() test.
efficient:
- recurse down subclasses only once rather than for each affected
slot;
- short-circuit recursing down subclasses when a subclass has its own
definition of the name that caused the update_slot() calls in the
first place;
- inline collect_ptrs().
using the same algorithm as the slot updates. The slotdefs array is
now sorted by slot offset and has an interned string object corresponding
to the name added to each item. More can be done but I need to commit
this first as a working intermediate stage.
The problem is that if fread() returns a short count, we attempt
another fread() the next time through the loop, and apparently glibc
clears or ignores the eof condition so the second fread() requires
another ^D to make it see the eof condition.
According to the man page (and the C std, I hope) fread() can only
return a short count on error or eof. I'm using that in the band-aid
solution to avoid calling fread() a second time after a short read.
Note that xreadlines() still has this problem: it calls
readlines(sizehint) until it gets a zero-length return. Since
xreadlines() is mostly used for reading real files, I won't worry
about this until we get a bug report.
inherit_slots(): tp_as_buffer was getting inherited as if it were a
method pointer, rather than a pointer to a vector of method pointers. As
a result, inheriting from a type that implemented buffer methods was
ineffective, leaving all the tp_as_buffer slots NULL in the subclass.
corresponding to a dispatch slot (e.g. __getitem__ or __add__) is set,
calculate the proper dispatch slot and propagate the change to all
subclasses. Because of multiple inheritance, there's no easy way to
avoid always recursing down the tree of subclasses. Who cares?
(There's more to do, but this works. There's also a test for this now.)
lseek(fp, 0L, SEEK_CUR) can make a filedescriptor unusable.
This workaround is expected to last only a few weeks (until GUSI
is fixed), but without it test_email fails.
the problem that slots weren't inherited properly. override_slots()
no longer exists; in its place comes fixup_slot_dispatchers() which
does more and different work and is table-based. (Eventually I want
this table also to replace all the little tab_foo tables.)
Also add a wrapper for __delslice__; this required a change in
test_descrtut.py.
without the Py_TPFLAGS_CHECKTYPES flag) in the wrappers. This
required a few changes in test_descr.py to cope with the fact that the
complex type has __int__, __long__ and __float__ methods that always
raise an exception.
is a list of weak references to types (new-style classes). Make this
accessible to Python as the function __subclasses__ which returns a
list of types -- we don't want Python programmers to be able to
manipulate the raw list.
In order to make this possible, I also had to add weak reference
support to type objects.
This will eventually be used together with a trap on attribute
assignment for dynamic classes for a major speed-up without losing the
dynamic properties of types: when a __foo__ method is added to a
class, the class and all its subclasses will get an appropriate tp_foo
slot function.
Tim Peters
Martin v. Löwis
Guido van Rossum
Jeremy Hylton
Michael W. Hudson
Andrew MacIntyre
Marc-André Lemburg
Neal Norwitz
Fred Drake
Sjoerd Mullender
Jack Jansen
Barry Warsaw
Neil Schemenauer
Thomas Heller