Mostly by Toby Dickenson and Titus Brown.
Add an optional argument to a decompression object's decompress()
method. The argument specifies the maximum length of the return
value. If the uncompressed data exceeds this length, the excess data
is stored as the unconsumed_tail attribute. (Not to be confused with
unused_data, which is a separate issue.)
Difference from SF patch: Default value for unconsumed_tail is ""
rather than None. It's simpler if the attribute is always a string.
object.c, PyObject_Str: Don't try to optimize anything except exact
string objects here; in particular, let str subclasses go thru tp_str,
same as non-str objects. This allows overrides of tp_str to take
effect.
stringobject.c:
+ string_print (str's tp_print): If the argument isn't an exact string
object, get one from PyObject_Str.
+ string_str (str's tp_str): Make a genuine-string copy of the object if
it's of a proper str subclass type. str() applied to a str subclass
that doesn't override __str__ ends up here.
test_descr.py: New str_of_str_subclass() test.
Remove the log file after we are done with it. This should clean up after
the test even on Windows, since the file is now closed before we attempt
removal.
Simply commented it out, and then test_hotshot passes on Windows.
Leaving to Fred to fix "the right way" (it seems to be a feature of
unittest that all unittests try to unlink open files <wink>).
inherit_slots(): tp_as_buffer was getting inherited as if it were a
method pointer, rather than a pointer to a vector of method pointers. As
a result, inheriting from a type that implemented buffer methods was
ineffective, leaving all the tp_as_buffer slots NULL in the subclass.
corresponding to a dispatch slot (e.g. __getitem__ or __add__) is set,
calculate the proper dispatch slot and propagate the change to all
subclasses. Because of multiple inheritance, there's no easy way to
avoid always recursing down the tree of subclasses. Who cares?
(There's more to do, but this works. There's also a test for this now.)
the problem that slots weren't inherited properly. override_slots()
no longer exists; in its place comes fixup_slot_dispatchers() which
does more and different work and is table-based. (Eventually I want
this table also to replace all the little tab_foo tables.)
Also add a wrapper for __delslice__; this required a change in
test_descrtut.py.
without the Py_TPFLAGS_CHECKTYPES flag) in the wrappers. This
required a few changes in test_descr.py to cope with the fact that the
complex type has __int__, __long__ and __float__ methods that always
raise an exception.
this type of test fails, vereq() does a better job of reporting than
verify().
Change vereq(x, y) to use "not x == y" rather than "x != y" -- it
makes a difference is some overloading tests.
many types were subclassable but had a xxx_dealloc function that
called PyObject_DEL(self) directly instead of deferring to
self->ob_type->tp_free(self). It is permissible to set tp_free in the
type object directly to _PyObject_Del, for non-GC types, or to
_PyObject_GC_Del, for GC types. Still, PyObject_DEL was a tad faster,
so I'm fearing that our pystone rating is going down again. I'm not
sure if doing something like
void xxx_dealloc(PyObject *self)
{
if (PyXxxCheckExact(self))
PyObject_DEL(self);
else
self->ob_type->tp_free(self);
}
is any faster than always calling the else branch, so I haven't
attempted that -- however those types whose own dealloc is fancier
(int, float, unicode) do use this pattern.
For a dynamically constructed type object, fill in the tp_doc slot with
a copy of the argument dict's "__doc__" value, provided the latter exists
and is a string.
NOTE: I don't know what to do if it's a Unicode string, so in that case
tp_doc is left NULL (which shows up as Py_None if you do Class.__doc__).
Note that tp_doc holds a char*, not a general PyObject*.
it deals correctly with some anomalous cases; according to this test
suite I've fixed it right.
The anomalous cases had to do with 'exception' events: these aren't
generated when they would be most helpful, and the profiler has to
work hard to recover the right information. The problems occur when C
code (such as hasattr(), which is used as the example here) calls back
into Python code and clears an exception raised by that Python code.
Consider this example:
def foo():
hasattr(obj, "bar")
Where obj is an instance from a class like this:
class C:
def __getattr__(self, name):
raise AttributeError
The profiler sees the following sequence of events:
call (foo)
call (__getattr__)
exception (in __getattr__)
return (from foo)
Previously, the profiler would assume the return event returned from
__getattr__. An if statement checking for this condition and raising
an exception was commented out... This version does the right thing.
test for modifying __getattr__ works, now that slot_tp_getattr_hook
zaps the slot if there's no hook. Added an XXX comment with a ref
back to slot_tp_getattr_hook.
Taught doctest about static methods, class methods, and property docstrings
in new-style classes. As for inspect.py/pydoc.py before it, the new stuff
needed didn't really fit into the old architecture (but was less of a
strain to force-fit here).
New-style class docstrings still aren't found, but that's the subject
of a different bug and I want to fix that right instead of hacking around
it in doctest.
instances).
Also added GC support to various auxiliary types: super, property,
descriptors, wrappers, dictproxy. (Only type objects have a tp_clear
field; the other types are.)
One change was necessary to the GC infrastructure. We have statically
allocated type objects that don't have a GC header (and can't easily
be given one) and heap-allocated type objects that do have a GC
header. Giving these different metatypes would be really ugly: I
tried, and I had to modify pickle.py, cPickle.c, copy.py, add a new
invent a new name for the new metatype and make it a built-in, change
affected tests... In short, a mess. So instead, we add a new type
slot tp_is_gc, which is a simple Boolean function that determines
whether a particular instance has GC headers or not. This slot is
only relevant for types that have the (new) GC flag bit set. If the
tp_is_gc slot is NULL (by far the most common case), all instances of
the type are deemed to have GC headers. This slot is called by the
PyObject_IS_GC() macro (which is only used twice, both times in
gcmodule.c).
I also changed the extern declarations for a bunch of GC-related
functions (_PyObject_GC_Del etc.): these always exist but objimpl.h
only declared them when WITH_CYCLE_GC was defined, but I needed to be
able to reference them without #ifdefs. (When WITH_CYCLE_GC is not
defined, they do the same as their non-GC counterparts anyway.)
- The test for deepcopy() in pickles() was indented wrongly, so it got
run twice (one for binary pickle mode, one for text pickle mode; but
the test doesn't depend on the pickle mode).
- In verbose mode, show which subtest (pickle/cPickle/deepcopy, text/bin).
in run_test() referenced two non-existent variables, and in
non-verbose mode, the tests didn't report the actual number, when it
differed from the expected number. Fixed this.
Also added an extra call to gc.collect() at the start of test_all().
This will be needed when I check in the changes to add GC to new-style
classes.
from Tim Hochberg. Also mucho fiddling to change the way doctest
determines whether a thing is a function, module or class. Under 2.2,
this really requires the functions in inspect.py (e.g., types.ClassType
is close to meaningless now, if not outright misleading).
staticness when __dynamic__ = 1 becomes the default:
- Some classes which are used to test the difference between static
and dynamic.
- Subclasses of complex: complex uses old-style numbers and the slot
wrappers used by dynamic classes only support new-style numbers.
(Ideally, the complex type should be fixed, but that looks like a
labor-intensive job.)
__rop__ now takes precendence over __op__. Those circumstances are:
- Both arguments are new-style classes
- Both arguments are new-style numbers
- Their implementation slots for tp_op differ
- Their types differ
- The right argument's type is a subtype of the left argument's type
Also did this for the ternary operator (pow) -- only the binary case
is dealt with properly though, since __rpow__ is not supported anyway.
fallback for objects that are neither supported by our dispatch table
nor have a __copy__ or __deepcopy__ method.
Changes to _reduce() in copy_reg.py to support reducing objects that
don't have a __dict__ -- copy.copy(complex()) now invokes _reduce().
Add tests for copy.copy() and copy.deepcopy() to test_regrtest.py.
Add raise_exception() to the _testcapi module. It isn't a test, but
the C API exists only to support test_exceptions. raise_exception()
takes two arguments -- an exception class and an integer specifying
how many arguments it should be called with.
test_exceptions uses BadException() to test the interpreter's behavior
when there is a problem instantiating the exception. test_capi1()
calls it with too many arguments. test_capi2() causes an exception to
be raised in the Python code of the constructor.
the local save/modify/restore of sys.stdout, but add machinery so that
regrtest can tell test_support the value of sys.stdout at the time
regrtest.main() started, and test_support can pass that out later to anyone
who needs a "visible" stdout.
- Made cls.__module__ writable.
- Ensure that obj.__dict__ is returned as {}, not None, even upon first
reference; it simply springs into life when you ask for it.
(*) The pickling support is provisional for the following reasons:
- It doesn't support classes with __slots__.
- It relies on additional support in copy_reg.py: the C method
__reduce__, defined in the object class, really calls calling
copy_reg._reduce(obj). Eventually the Python code in copy_reg.py
needs to be migrated to C, but I'd like to experiment with the
Python implementation first. The _reduce() code also relies on an
additional helper function, _reconstructor(), defined in
copy_reg.py; this should also be reimplemented in C.
than <type 'ClassName'>. Exception: if it's a built-in type or an
extension type, continue to call it <type 'ClassName>. Call me a
wimp, but I don't want to break more user code than necessary.
same. I hope the test for structural equivalence is stringent enough.
It only allows the assignment if the old and new types:
- have the same basic size
- have the same item size
- have the same dict offset
- have the same weaklist offset
- have the same GC flag bit
- have a common base that is the same except for maybe the dict and
weaklist (which may have been added separately at the same offsets
in both types)
- property() now takes 4 keyword arguments: fget, fset, fdel, doc.
Note that the real purpose of the 'f' prefix is to make fdel fit in
('del' is a keyword, so can't used as a keyword argument name).
- These map to visible readonly attributes 'fget', 'fset', 'fdel',
and '__doc__' in the property object.
- fget/fset/fdel weren't discoverable from Python before.
- __doc__ is new, and allows to associate a docstring with a property.
declarations and weird markup that we used to accept & ignore that recent
versions raised an exception for; the original behavior has been restored
and augmented (the user can decide what to do if they care; the default is
to ignore it as done in early versions).
point out, pydoc doesn't tell you where class attributes were defined,
gets several new 2.2 features wrong, and isn't aware of some new features
checked in on Thursday <wink>. pydoc is hampered in part because
inspect.py has the same limitations. Alas, I can't think of a way to
fix this within the current architecture of inspect/pydoc: it's simply
not possible in 2.2 to figure out everything needed just from examining
the object you get back from class.attr. You also need the class
context, and the method resolution order, and tests against various things
that simply didn't exist before. OTOH, knowledge of how to do that is
getting quite complex, so doesn't belong in pydoc.
classify_class_attrs takes a different approach, analyzing all
the class attrs "at once", and returning the most interesting stuff for
each, all in one gulp. pydoc needs to be reworked to use this for
classes (instead of the current "filter dir(class) umpteen times against
assorted predicates" approach).
easy for 2.2 new-style classes, but trickier for classic classes, and
different approaches are needed "depending". The function will allow
later code to treat all flavors of classes uniformly.
somewhere inside a line, use ndiff so that intraline difference marking
can point out what changed within a line. I don't remember diff-style
abbreviations either (haven't used it since '94, except to produce
patches), so say the rest in English too.
Lib/test/output/test_StringIO is no longer necessary.
Also, added a test of the iterator protocol that's just been added to
StringIO's and cStringIO's.
- if __getattribute__ exists, it is called first;
if it doesn't exists, PyObject_GenericGetAttr is called first.
- if the above raises AttributeError, and __getattr__ exists,
it is called.
the first difference, let the test run till completion, then gather
all the output and compare it to the expected output using difflib.
XXX Still to do: produce diff output that only shows the sections that
differ; currently it produces ndiff-style output because that's the
easiest to produce with difflib, but this becomes a liability when the
output is voluminous and there are only a few differences.
classes to __getattribute__, to make it crystal-clear that it doesn't
have the same semantics as overriding __getattr__ on classic classes.
This is a halfway checkin -- I'll proceed to add a __getattr__ hook
that works the way it works in classic classes.
elements which are not Unicode objects or strings. (This matches
the string.join() behaviour.)
Fix a memory leak in the .join() method which occurs in case
the Unicode resize fails.
Restore the test_unicode output.
their own test suite from a multitude of classes (like test_email.py
will be doing).
run_unittest(): Call run_suite() after making a suite from the
testclass.
instance.
Split a string comparison test in two halves, replacing "a==b==a" with
separate tests for a==b and b==a. (Reason: while experimenting, this
test failed, and I wanted to know if it was the first or the second ==
operator that failed.)
hack, and it's even more disgusting than a PyInstance_Check() call.
If the tp_compare slot is the slot used for overrides in Python,
it's always called.
Add some tests that show what should work too.
#462270: sub-tle difference between pre.sub and sre.sub. PRE ignored
an empty match at the previous location, SRE didn't.
also synced with Secret Labs "sreopen" codebase.
to raise TypeError. In practice, a disallowed attribute assignment
can raise either TypeError or AttributeError (and it's unclear which
is better). So allow either. (Yes, this is in anticipation of a
code change that switches the exception raised. :-)
- Add a utility function, cantset(), which verifies that setting a
particular attribute to a given value is disallowed, and also that
deleting that same attribute is disallowed. Use this in the
test_func_*() tests.
- Add a new set of tests that test conformance of various instance
method attributes. (Also in anticipation of code that changes their
implementation.)
and are lists, and then just the string elements (if any)).
There are good and bad reasons for this. The good reason is to support
dir() "like before" on objects of extension types that haven't migrated
to the class introspection API yet. The bad reason is that Python's own
method objects are such a type, and this is the quickest way to get their
im_self etc attrs to "show up" via dir(). It looks much messier to move
them to the new scheme, as their current getattr implementation presents
a view of their attrs that's a untion of their own attrs plus their
im_func's attrs. In particular, methodobject.__dict__ actually returns
methodobject.im_func.__dict__, and if that's important to preserve it
doesn't seem to fit the class introspection model at all.
Both int and long multiplication are changed to be more careful in
their assumptions about when one of the arguments is a sequence: the
assumption that at least one of the arguments must be an int (or long,
respectively) is still held, but the assumption that these don't smell
like sequences is no longer true: a subtype of int or long may well
have a sequence-repeat thingie!
keys are true strings -- no subclasses need apply. This may be debatable.
The problem is that a str subclass may very well want to override __eq__
and/or __hash__ (see the new example of case-insensitive strings in
test_descr), but go-fast shortcuts for strings are ubiquitous in our dicts
(and subclass overrides aren't even looked for then). Another go-fast
reason for the change is that PyCheck_StringExact() is a quicker test
than PyCheck_String(), and we make such a test on virtually every access
to every dict.
OTOH, a str subclass may also be perfectly happy using the base str eq
and hash, and this change slows them a lot. But those cases are still
hypothetical, while Python's own reliance on true-string dicts is not.
just by doing type(f) where f is any file object. This left a hole in
restricted execution mode that rexec.py can't plug by itself (although it
can plug part of it; the rest is plugged in fileobject.c now).
on to the tp_new slot (if non-NULL), as well as to the tp_init slot (if
any). A sane type implementing both tp_new and tp_init should probably
pay attention to the arguments in only one of them.
with the same value instead. This ensures that a string (or string
subclass) object's ob_sinterned pointer is always a str (or NULL), and
that the dict of interned strings only has strs as keys.
+ These were leaving the hash fields at 0, which all string and unicode
routines believe is a legitimate hash code. As a result, hash() applied
to str and unicode subclass instances always returned 0, which in turn
confused dict operations, etc.
+ Changed local names "new"; no point to antagonizing C++ compilers.
subclasses, all "the usual" ones (slicing etc), plus replace, translate,
ljust, rjust, center and strip. I don't know how to be sure they've all
been caught.
Question: Should we complain if someone tries to intern an instance of
a string subclass? I hate to slow any code on those paths.
tuple(i) repaired to return a true tuple when i is an instance of a
tuple subclass.
Added PyTuple_CheckExact macro.
PySequence_Tuple(): if a tuple-like object isn't exactly a tuple, it's
not safe to return the object as-is -- make a new tuple of it instead.
Given an immutable type M, and an instance I of a subclass of M, the
constructor call M(I) was just returning I as-is; but it should return a
new instance of M. This fixes it for M in {int, long}. Strings, floats
and tuples remain to be done.
Added new macros PyInt_CheckExact and PyLong_CheckExact, to more easily
distinguish between "is" and "is a" (i.e., only an int passes
PyInt_CheckExact, while any sublass of int passes PyInt_Check).
Added private API function _PyLong_Copy.
If on Windows, we require the 'largefile' resource.
If not on Windows, we use a test that actually writes a byte beyond
the 2BG limit -- seeking alone is not sufficient, since on some
systems (e.g. Linux with glibc 2.2) the sytem call interface supports
large seek offsets but not all filesystem implementations do.
Note that on Windows, we do not use the write test: on Win2K, that
test can take a minute trying to zero all those blocks on disk, and on
Windows our code always supports large seek offsets (but again, not
all filesystems do). This may mean that on Win95, or on certain other
backward filesystems, test_largefile will *fail*.
horridly inefficient hack in regrtest's Compare class, but it's about as
clean as can be: regrtest has to set up the Compare instance before
importing a test module, and by the time the module *is* imported it's too
late to change that decision. The good news is that the more tests we
convert to unittest and doctest, the less the inefficiency here matters.
Even now there are few tests with large expected-output files (the new
cost here is a Python-level call per .write() when there's an expected-
output file).
iterable object. I'm not sure how that got overlooked before!
Got rid of the internal _PySequence_IterContains, introduced a new
internal _PySequence_IterSearch, and rewrote all the iteration-based
"count of", "index of", and "is the object in it or not?" routines to
just call the new function. I suppose it's slower this way, but the
code duplication was getting depressing.
saving instead a traceback string, but test_support's run_unittest was
still peeking into unittest internals and trying to pick apart unittest's
errors and failures vectors as if they contained exc_info() tuples instead
of strings.
Whatever, when a unittest-based test failed, test_support blew up. I'm
not sure this is the right way to fix it; it simply gets me unstuck.
capabilities of the Pentium FPU, so what should have been (and were on
Windows) exact results got fuzzy. Then it turns out test_support.fcmp()
isn't tolerant of tiny errors when *one* of the comparands is 0, but
test_complex's old check_close_real() is. Rather than fix gcc <wink>,
easier to revert this test and revisit after the release.
(1) Allow multiple -u options to extend each other (and the initial
value of use_resources passed into regrtest.main()).
(2) When a test is run stand-alone (not via regrtest.py), needed
resources are always granted.
1. That seeking beyond the end of a file increases the size of a file.
2. That files so extended are magically filled with null bytes.
I find no support for either in the C std, and #2 in particular turns out
not to be true on Win32 (you apparently see whatever trash happened to be
on disk). Left #1 intact, but changed the test to check only bytes it
explicitly wrote. Also fiddled the "expected" vs "got" failure reports
to consistently use repr (%r) -- they weren't readable otherwise.
Curious: the MS docs say stati64 etc are supported even on Win95, but
Win95 doesn't support a filesystem that allows partitions > 2 Gb.
test_largefile: This was opening its test file in text mode. I have no
idea how that worked under Win64, but it sure needs binary mode on Win98.
BTW, on Win98 test_largefile runs quickly (under a second).
requires that errno ever get set, and it looks like glibc is already
playing that game. New rules:
+ Never use HUGE_VAL. Use the new Py_HUGE_VAL instead.
+ Never believe errno. If overflow is the only thing you're interested in,
use the new Py_OVERFLOWED(x) macro. If you're interested in any libm
errors, use the new Py_SET_ERANGE_IF_OVERFLOW(x) macro, which attempts
to set errno the way C89 said it worked.
Unfortunately, none of these are reliable, but they work on Windows and I
*expect* under glibc too.
32 characters per component. This makes mkdir() calls and such fail with EINVAL.
For now I am disabling the test on the Mac, and I'll open a bugreport.
getting Infs, NaNs, or nonsense in 2.1 and before; in yesterday's CVS we
were getting OverflowError; but these functions always make good sense
for positive arguments, no matter how large).
"/" and "//", and doesn't really care what they *mean*, just that both
are tried (and that, whatever they mean, they act similarly for int and
long arguments).
the fiddling is simply due to that no caller of PyLong_AsDouble ever
checked for failure (so that's fixing old bugs). PyLong_AsDouble is much
faster for big inputs now too, but that's more of a happy consequence
than a design goal.
of PyMapping_Keys because we know we have a real dict. Tolerate that
objects may have an attr named "__dict__" that's not a dict (Py_None
popped up during testing).
test_descr.py, test_dir(): Test the new classic-class behavior; beef up
the new-style class test similarly.
test_pyclbr.py, checkModule(): dir(C) is no longer a synonym for
C.__dict__.keys() when C is a classic class (looks like the same thing
that burned distutils! -- should it be *made* a synoym again? Then it
would be inconsistent with new-style class behavior.).
bag. It's clearly wrong for classic classes, at heart because a classic
class doesn't have a __class__ attribute, and I'm unclear on whether
that's feature or bug. I'll repair this once I find out (in the
meantime, dir() applied to classic classes won't find the base classes,
while dir() applied to a classic-class instance *will* find the base
classes but not *their* base classes).
Please give the new dir() a try and see whether you love it or hate it.
The new dir([]) behavior is something I could come to love. Here's
something to hate:
>>> class C:
... pass
...
>>> c = C()
>>> dir(c)
['__doc__', '__module__']
>>>
The idea that an instance has a __doc__ attribute is jarring (of course
it's really c.__class__.__doc__ == C.__doc__; likewise for __module__).
OTOH, the code already has too many special cases, and dir(x) doesn't
have a compelling or clear purpose when x isn't a module.
mapping object", in the same sense dict.update(x) requires of x (that x
has a keys() method and a getitem).
Questionable: The other type constructors accept a keyword argument, so I
did that here too (e.g., dictionary(mapping={1:2}) works). But type_call
doesn't pass the keyword args to the tp_new slot (it passes NULL), it only
passes them to the tp_init slot, so getting at them required adding a
tp_init slot to dicts. Looks like that makes the normal case (i.e., no
args at all) a little slower (the time it takes to call dict.tp_init and
have it figure out there's nothing to do).
Jeremy Hylton
Tim Peters
Guido van Rossum
Barry Warsaw
Fred Drake
Skip Montanaro
Martin v. Löwis
Thomas Wouters
Marc-André Lemburg
Fredrik Lundh
Jack Jansen