which can be reviewed via
http://coding.derkeiler.com/Archive/Python/comp.lang.python/2003-12/1011.html
Duncan Booth investigated, and discovered that an "optimisation" was
in fact a pessimisation for small numbers of elements in a source list,
compared to not having the optimisation, although with large numbers
of elements in the source list the optimisation was quite beneficial.
He posted his change to comp.lang.python (but not to SF).
Further research has confirmed his assessment that the optimisation only
becomes a net win when the source list has more than 100 elements.
I also found that the optimisation could apply to tuples as well,
but the gains only arrive with source tuples larger than about 320
elements and are nowhere near as significant as the gains with lists,
(~95% gain @ 10000 elements for lists, ~20% gain @ 10000 elements for
tuples) so I haven't proceeded with this.
The code as it was applied the optimisation to list subclasses as
well, and this also appears to be a net loss for all reasonable sized
sources (~80-100% for up to 100 elements, ~20% for more than 500
elements; I tested up to 10000 elements).
Duncan also suggested special casing empty lists, which I've extended
to all empty sequences.
On the basis that list_fill() is only ever called with a list for the
result argument, testing for the source being the destination has
now happens before testing source types.
Original idea by Guido van Rossum.
Idea for skipable inner iterators by Raymond Hettinger.
Idea for argument order and identity function default by Alex Martelli.
Implementation by Hye-Shik Chang (with tweaks by Raymond Hettinger).
comments about why both calls to cyclic gc here can cause problems.
I'll backport to 2.3 maint. Since the calls were introduced in 2.3,
that will be the end of it.
and left shifts. (Thanks to Kalle Svensson for SF patch 849227.)
This addresses most of the remaining semantic changes promised by
PEP 237, except for repr() of a long, which still shows the trailing
'L'. The PEP appears to promise warnings for operations that
changed semantics compared to Python 2.3, but this is not
implemented; we've suffered through enough warnings related to
hex/oct literals and I think it's best to be silent now.
by the function object or by the method object, the function
object's attribute usually wins. Christian Tismer pointed out that
that this is really a mistake, because this only happens for special
methods (like __reduce__) where the method object's version is
really more appropriate than the function's attribute. So from now
on, all method attributes will have precedence over function
attributes with the same name.
Also SF patch 843455.
This is a critical bugfix.
I'll backport to 2.3 maint, but not beyond that. The bugs this fixes
have been there since weakrefs were introduced.
* Install the unittests, docs, newsitem, include file, and makefile update.
* Exercise the new functions whereever sets.py was being used.
Includes the docs for libfuncs.tex. Separate docs for the types are
forthcoming.
subtype_dealloc(): This left the dying object exposed to gc, so that
if cyclic gc triggered during the weakref callback, gc tried to delete
the dying object a second time. That's a disaster. subtype_dealloc()
had a (I hope!) unique problem here, as every normal dealloc routine
untracks the object (from gc) before fiddling with weakrefs etc. But
subtype_dealloc has obscure technical reasons for re-registering the
dying object with gc (already explained in a large comment block at
the bottom of the function).
The fix amounts to simply refraining from reregistering the dying object
with gc until after the weakref callback (if any) has been called.
This is a critical bug (hard to predict, and causes seemingly random
memory corruption when it occurs). I'll backport it to 2.3 later.
It works like the pure python verion except:
* it stops storing data after of the iterators gets deallocated
* the data queue is implemented with two stacks instead of one dictionary.
key provides C support for the decorate-sort-undecorate pattern.
reverse provide a stable sort of the list with the comparisions reversed.
* Amended the docs to guarantee sort stability.
* Added C coded getrandbits(k) method that runs in linear time.
* Call the new method from randrange() for ranges >= 2**53.
* Adds a warning for generators not defining getrandbits() whenever they
have a call to randrange() with too large of a population.
If a length-1 Unicode string was in the freelist and it was
uninitialized or pointed to a very large (magnitude) negative number,
the check
unicode_latin1[unicode->str[0]] == unicode
could cause a segmentation violation, e.g. unicode->str[0] is 0xcbcbcbcb.
Fix this in two ways:
1. Change guard befor unicode_latin1[] to test against 256U. If I
understand correctly, the unsigned long used to store UCS4 on my
box was getting converted to a signed long to compare with the
signed constant 256.
2. Change _PyUnicode_New() to make sure the first element of str is
always initialized to zero. There are several places in the code
where the caller can exit with an error before initializing any
of str, which would leave junk in str[0].
Also, silence a compiler warning on pointer vs. int arithmetic.
Bug fix candidate.
Add support for the iterator and mapping protocols.
For Py2.3, this was done for shelve, dumbdbm and other mapping objects, but
not for bsddb and dbhash which were inadvertently missed.
file_truncate(): C doesn't define what fflush(fp) does if fp is open
for update, and the preceding I/O operation on fp was input. On Windows,
fflush() actually changes the current file position then. Because
Windows doesn't support ftruncate() directly, this not only caused
Python's file.truncate() to change the file position (contra our docs),
it also caused the file not to change size.
Repaired by getting the initial file position at the start, restoring
it at the end, and tossing all the complicated micro-efficiency checks
trying to avoid "provably unnecessary" seeks. file.truncate() can't
be a frequent operation, and seeking to the current file position has
got to be cheap anyway.
Bugfix candidate.
* Relaxed the argument restrictions for non-operator methods. They now
allow any iterable instead of requiring a set. This makes the module
a little easier to use and paves the way for an efficient C
implementation which can take better advantage of iterable arguments
while screening out immutables.
* Deprecated Set.update() because it now duplicates Set.union_update()
* Adapted the tests and docs to include the above changes.
* Added more test coverage including testing identities and checking
to make sure non-restartable generators work as arguments.
Will backport to Py2.3.1 so that the interface remains consistent
across versions. The deprecation of update() will be changed to
a FutureWarning.
number. This accounts for the 2 refcount leaks per test_complex run
Michael Hudson discovered (I figured only I would have the stomach to
look for leaks in floating-point code <wink>).
The default seed is time.time().
Multiplied by 256 before truncating so that fractional seconds are used.
This way, two successive calls to random.seed() are much more likely
to produce different sequences.
The fix is confined to the Windows installer.
Not a bugfix candidate: the need for the new -n switch added here was
introduced by moving to the idlefork IDLE (so this change isn't needed
or helpful before 2.3).
arbitrary bytes before the actual zip compatible archive. Zipfiles
containing comments at the end of the file are still not supported.
Add a testcase to test_zipimport, and update NEWS.
This closes sf #775637 and sf #669036.
are satisfied in a case-insensitive manner, the attempt to import (the
non-existent) fcntl gets satisfied by FCNTL.py instead, and the tempfile
module defines a Unix-specific _set_cloexec() function in that case. As
a result, temp files can't be created then (blows up with an AttributeError
trying to reference fcntl.fcntl). This just popped up in the spambayes
project, where there is no apparent workaround (which is why I'm pushing
this in now).
New Plan (releases to be made off the head, ongoing random 2.4 stuff
to be done on a short-lived branch, provided anyone is motivated enough
to create one).
skip over functions with private names (as indicated by the underscore
naming convention). The old default created too much of a risk that
user tests were being skipped inadvertently. Note, this change could
break code in the unlikely case that someone had intentionally put
failing tests in the docstrings of private functions. The breakage
is easily fixable by specifying the old behavior when calling testmod()
or Tester(). The more likely case is that the silent failure was
unintended and that the user needed to be informed so the test could be
fixed.
Related to SF patch 723231 (which pointed out the problem, but didn't
fix it, just shut up the warning msg -- which was pointing out a dead-
serious bug!).
Bugfix candidate.
behavior, creating many threads very quickly. A long debugging session
revealed that the Windows implementation of PyThread_start_new_thread()
was choked with "laziness" errors:
1. It checked MS _beginthread() for a failure return, but when that
happened it returned heap trash as the function result, instead of
an id of -1 (the proper error-return value).
2. It didn't consider that the Win32 CreateSemaphore() can fail.
3. When creating a great many threads very quickly, it's quite possible
that any particular bootstrap call can take virtually any amount of
time to return. But the code waited for a maximum of 5 seconds, and
didn't check to see whether the semaphore it was waiting for got
signaled. If it in fact timed out, the function could again return
heap trash as the function result. This is actually what confused
the test program, as the heap trash usually turned out to be 0, and
then multiple threads all got id 0 simultaneously, confusing the
hell out of threading.py's _active dict (mapping id to thread
object). A variety of baffling behaviors followed from that.
WRT #1 and #2, error returns are checked now, and "thread.error: can't
start new thread" gets raised now if a new thread (or new semaphore)
can't be created. WRT #3, we now wait for the semaphore without a
timeout.
Also removed useless local vrbls, folded long lines, and changed callobj
to a stack auto (it was going thru malloc/free instead, for no discernible
reason).
Bugfix candidate.
I won't have time to write real docs, but spent a lot of time adding
comments to his code and fleshing out the exported functions' docstrings.
There's probably opportunity to consolidate how docstrings get extracted
too, and the new code for that is probably better than the old code for
that (which strained mightily to recover from 2.2's new class/type
gimmicks).
SF bug #760703: SocketHandler and LogRecord don't work well together
SF bug #757821: logging module docs
Applied Vinay Sajip's patch with a few minor fixups and a NEWS item.
Patched __init__.py - added new function
makeLogRecord (for bug report 760703).
Patched handlers.py - updated some docstrings and
deleted some old commented-out code.
Patched test_logging.py to make use of makeLogRecord.
Patched liblogging.tex to fill documentation gaps (both
760703 and bug 757821).
now accepts "True" when a test expects "1", and similarly for "False"
versus "0". This is un-doctest-like, but on balance makes it much
more pleasant to write doctests that pass under 2.2 and 2.3. I expect
it to go away again, when 2.2 is forgotten. In the meantime, there's
a new doctest module constant that can be passed to a new optional
argument, if you want to turn this behavior off.
Note that this substitution is very simple-minded: the expected and
actual outputs have to consist of single tokens. No attempt is made,
e.g., to accept [True, False] when a test expects [1, 0]. This is a
simple hack for simple tests, and I intend to keep it that way.
fix the hangs on Win98SE when starting IDLE via "python" from a DOS box,
but did appear to make them harder to provoke. I closed that bug report
as being hopeless (and if someone wants to open it again, don't dare
assign it to me again <0.1 wink>).
have to insert it in front of other classes, nor do dirty tricks like
inserting a "dummy" HTTPHandler after a ProxyHandler when building an
opener with proxy support.
Python-Dev. Fixed typos in test comments. Added some trivial new test
guts to show the parallelism (now) among __delitem__, __setitem__ and
__getitem__ wrt error conditions.
Still a bugfix candidate for 2.2.3 final, but waiting for Fred to get a
chance to chime in.
Someone review this, please! Final releases are getting close, Fred
(the weakref guy) won't be around until Tuesday, and the pre-patch
code can indeed raise spurious RuntimeErrors in the presence of
threads or mutating comparison functions.
See the bug report for my confusions: I can't see any reason for why
__delitem__ iterated over the keys. The new one-liner implementation
is much faster, can't raise RuntimeError, and should be better-behaved
in all respects wrt threads.
New tests test_weak_keyed_bad_delitem and
test_weak_keyed_cascading_deletes fail before this patch.
Bugfix candidate for 2.2.3 too, if someone else agrees with this patch.
float_pow(): Don't let the platform pow() raise -1.0 to an integer power
anymore; at least glibc gets it wrong in some cases. Note that
math.pow() will continue to deliver wrong (but platform-native) results
in such cases.
tp_free is NULL or PyObject_Del at the end. Because it's a base type
it must call tp_free in its dealloc function, and because it's gc'able
it must not call PyObject_Del.
inherit_slots(): Don't inherit tp_free unless the type and its base
agree about whether they're gc'able. If the type is gc'able and the
base is not, and the base uses the default PyObject_Del for its
tp_free, give the type PyObject_GC_Del for its tp_free (the appropriate
default for a gc'able type).
cPickle.c: The Pickler and Unpickler types claim to be base classes
and gc'able, but their dealloc functions didn't call tp_free.
Repaired that. Also call PyType_Ready() on these typeobjects, so
that the correct (PyObject_GC_Del) default memory-freeing function
gets plugged into these types' tp_free slots.
one good use: a subclass adding a method to express the duration as
a number of hours (or minutes, or whatever else you want to add). The
native breakdown into days+seconds+us is often clumsy. Incidentally
moved a large chunk of object-initialization code closer to the top of
the file, to avoid worse forward-reference trickery.
- When redirecting, always use GET. This is common practice and
more-or-less sanctioned by the HTTP standard.
- Add a handler for 307 redirection, which becomes an error for POST,
but a regular redirect for GET and HEAD.
After removing that, two testers on machines where C: is not the system
drive reported that the installer suggested their system drive instead
of C:, and that's what they wanted it to do.
Reverted a Py2.3b1 change to iterator in subclasses of list and tuple.
They had been changed to use __getitem__ whenever it had been overriden
in the subclass.
This caused some usabilty and performance problems. Also, it was
inconsistent with the rest of python where many container methods
access the underlying object directly without first checking for
an overridden getter. Users needing a change in iterator behavior
should override it directly.
- The socket module now provides the functions inet_pton and inet_ntop
for converting between string and packed representation of IP addresses.
See SF patch #658327.
This still needs a bit of work in the doc area, because it is not
available on all platforms (especially not on Windows).
Revised netrc.py to include the additional ascii punctuation
characters. Omitted the other logic changes. See
Lib/netrc.py 1.17.
Since this is more of a feature request than a bug,
including in Py2.3 but not recommending for backporting.
docs here are best-guess: the MS docs I could find weren't clear, and
some even claimed _commit() has no effect on Win32 systems (which is
easily shown to be false just by trying it).
* Can now test for basic blocks.
* Optimize inverted comparisions.
* Optimize unary_not followed by a conditional jump.
* Added a new opcode, NOP, to keep code size constant.
* Applied NOP to previous transformations where appropriate.
Note, the NOP would not be necessary if other functions were
added to re-target jump addresses and update the co_lnotab mapping.
That would yield slightly faster and cleaner bytecode at the
expense of optimizer simplicity and of keeping it decoupled
from the line-numbering structure.
raising an exception. This is consistent with calling the
constructors for the other builtin types -- called without argument
they all return the false value of that type. (SF patch #724135)
Thanks to Alex Martelli.
I'm finding some pretty baffling output, like reprs consisting entirely
of three left parens. At least this will let us know what type the object
is (it's not str -- there's no quote character in the repr).
New tool combinerefs.py, to combine the two output blocks produced via
PYTHONDUMPREFS.
even farther down, to just before the call to
_PyObject_DebugMallocStats(). This required the following changes:
- pystate.c, PyThreadState_GetDict(): changed not to raise an
exception or issue a fatal error when no current thread state is
available, but simply return NULL without raising an exception
(ever).
- object.c, Py_ReprEnter(): when PyThreadState_GetDict() returns NULL,
don't raise an exception but return 0. This means that when
printing a container that's recursive, printing will go on and on
and on. But that shouldn't happen in the case we care about (see
first bullet).
- Updated Misc/NEWS and Doc/api/init.tex to reflect changes to
PyThreadState_GetDict() definition.
interpreted by slicing, so negative values count from the end of the
list. This was the only place where such an interpretation was not
placed on a list index.
A small fix for bug #545855 and Greg Chapman's
addition of op code SRE_OP_MIN_REPEAT_ONE for
eliminating recursion on simple uses of pattern '*?' on a
long string.
- range() now works even if the arguments are longs with magnitude
larger than sys.maxint, as long as the total length of the sequence
fits. E.g., range(2**100, 2**101, 2**100) is the following list:
[1267650600228229401496703205376L]. (SF patch #707427.)
of PyObject_HasAttr(); the former promises never to execute
arbitrary Python code. Undid many of the changes recently made to
worm around the worst consequences of that PyObject_HasAttr() could
execute arbitrary Python code.
Compatibility is hard to discuss, because the dangerous cases are
so perverse, and much of this appears to rely on implementation
accidents.
To start with, using hasattr() to check for __del__ wasn't only
dangerous, in some cases it was wrong: if an instance of an old-
style class didn't have "__del__" in its instance dict or in any
base class dict, but a getattr hook said __del__ existed, then
hasattr() said "yes, this object has a __del__". But
instance_dealloc() ignores the possibility of getattr hooks when
looking for a __del__, so while object.__del__ succeeds, no
__del__ method is called when the object is deleted. gc was
therefore incorrect in believing that the object had a finalizer.
The new method doesn't suffer that problem (like instance_dealloc(),
_PyObject_Lookup() doesn't believe __del__ exists in that case), but
does suffer a somewhat opposite-- and even more obscure --oddity:
if an instance of an old-style class doesn't have "__del__" in its
instance dict, and a base class does have "__del__" in its dict,
and the first base class with a "__del__" associates it with a
descriptor (an object with a __get__ method), *and* if that
descriptor raises an exception when __get__ is called, then
(a) the current method believes the instance does have a __del__,
but (b) hasattr() does not believe the instance has a __del__.
While these disagree, I believe the new method is "more correct":
because the descriptor *will* be called when the object is
destructed, it can execute arbitrary Python code at the time the
object is destructed, and that's really what gc means by "has a
finalizer": not specifically a __del__ method, but more generally
the possibility of executing arbitrary Python code at object
destruction time. Code in a descriptor's __get__() executed at
destruction time can be just as problematic as code in a
__del__() executed then.
So I believe the new method is better on all counts.
Bugfix candidate, but it's unclear to me how all this differs in
the 2.2 branch (e.g., new-style and old-style classes already
took different gc paths in 2.3 before this last round of patches,
but don't in the 2.2 branch).
platforms which have dup(2). The makefile() method is built directly on top
of the socket without duplicating the file descriptor, allowing timeouts to
work properly. Includes a new test case (urllibnet) which requires the
network resource.
Closes bug 707074.
pack_float, pack_double, save_float: All the routines for creating
IEEE-format packed representations of floats and doubles simply ignored
that rounding can (in rare cases) propagate out of a long string of
1 bits. At worst, the end-off carry can (by mistake) interfere with
the exponent value, and then unpacking yields a result wrong by a factor
of 2. In less severe cases, it can end up losing more low-order bits
than intended, or fail to catch overflow *caused* by rounding.
Bugfix candidate, but I already backported this to 2.2.
In 2.3, this code remains in severe need of refactoring.
variables to store internal data. As a result, any atempts to use the
unicode system with multiple active interpreters, or successive
interpreter executions, would fail.
Now that information is stored into members of the PyInterpreterState
structure.
- Implement the behavior as specified in PEP 277, meaning os.listdir()
will only return unicode strings if it is _called_ with a unicode
argument.
- And then return only unicode, don't attempt to convert to ASCII.
- Don't switch on Py_FileSystemDefaultEncoding, but simply use the
default encoding if Py_FileSystemDefaultEncoding is NULL. This means
os.listdir() can now raise UnicodeDecodeError if the default encoding
can't represent the directory entry. (This seems better than silcencing
the error and fall back to a byte string.)
- Attempted to decribe the above in Doc/lib/libos.tex.
- Reworded the Misc/NEWS items to reflect the current situation.
This checkin also fixes bug #696261, which was due to os.listdir() not
using Py_FileSystemDefaultEncoding, like all file system calls are
supposed to.
[ 555817 ] Flawed fcntl.ioctl implementation.
with my patch that allows for an array to be mutated when passed
as the buffer argument to ioctl() (details complicated by
backwards compatibility considerations -- read the docs!).
Allow mixed-type __eq__ and __ne__ for Set objects. This is messier than
I'd like because Set *also* implements __cmp__. I know of one glitch now:
cmp(s, t) returns 0 now when s and t are both Sets and s == t, despite
that Set.__cmp__ unconditionally raises TypeError (and by intent). The
rub is that __eq__ gets tried first, and the x.__eq__(y) True result
convinces Python that cmp(x, y) is 0 without even calling Set.__cmp__.
rarely needed, but can sometimes be useful to release objects
referenced by the traceback held in sys.exc_info()[2]. (SF patch
#693195.) Thanks to Kevin Jacobs!
test_linuxaudiodev.py) are no longer run by default. This is
because they don't always work, depending on your hardware and
software. To run these tests, you must use an invocation like
./python Lib/test/regrtest.py -u audio test_ossaudiodev
with an indented code block but no newline would raise SyntaxError.
This would have been a four-line change in parsetok.c... Except
codeop.py depends on this behavior, so a compilation flag had to be
invented that causes the tokenizer to revert to the old behavior;
this required extra changes to 2 .h files, 2 .c files, and 2 .py
files. (Fixes SF bug #501622.)
This changes the default __new__ to refuse arguments iff tp_init is the
default __init__ implementation -- thus making it a TypeError when you
try to pass arguments to a constructor if the class doesn't override at
least __init__ or __new__.
mostly from SF patch #683257, but I had to change unlock_import() to
return an error value to avoid fatal error.
Should this be backported? The patch requested this, but it's a new
feature.
folded; this will change in Python 2.4. On a 32-bit machine, this
happens for 0x80000000 through 0xffffffff, and for octal constants in
the same value range. No warning is issued if an explicit base is
given, *or* if the string contains a sign (since in those cases no
sign folding ever happens).
"Unsigned" (i.e., positive-looking, but really negative) hex/oct
constants with a leading minus sign are once again properly negated.
The micro-optimization for negated numeric constants did the wrong
thing for such hex/oct constants. The patch avoids the optimization
for all hex/oct constants.
This needs to be backported to Python 2.2!
__ne__ no longer complain if they don't know how to compare to the other
thing. If no meaningful way to compare is known, saying "not equal" is
sensible. This allows things like
if adatetime in some_sequence:
and
somedict[adatetime] = whatever
to work as expected even if some_sequence contains non-datetime objects,
or somedict non-datetime keys, because they only call __eq__.
It still complains (raises TypeError) for mixed-type comparisons in
contexts that require a total ordering, such as list.sort(), use as a
key in a BTree-based data structure, and cmp().
extension implemented flush() was fixed. Scott also rewrite the
zlib test suite using the unittest module. (SF bug #640230 and
patch #678531.)
Backport candidate I think.
anymore either, so don't. This also allows to get rid of obscure code
making __getnewargs__ identical to __getstate__ (hmm ... hope there
wasn't more to this than I realize!).
(pickling no longer needs them, and immutable objects shouldn't have
visible __setstate__() methods regardless). Rearranged the code to
put the internal setstate functions in the constructor sections.
Repaired the timedelta reduce() method, which was still producing
stuff that required a public timedelta.__setstate__() when unpickling.
compare against "the other" argument, we raise TypeError,
in order to prevent comparison from falling back to the
default (and worse than useless, in this case) comparison
by object address.
That's fine so far as it goes, but leaves no way for
another date/datetime object to make itself comparable
to our objects. For example, it leaves Marc-Andre no way
to teach mxDateTime dates how to compare against Python
dates.
Discussion on Python-Dev raised a number of impractical
ideas, and the simple one implemented here: when we don't
know how to compare against "the other" argument, we raise
TypeError *unless* the other object has a timetuple attr.
In that case, we return NotImplemented instead, and Python
will give the other object a shot at handling the
comparison then.
Note that comparisons of time and timedelta objects still
suffer the original problem, though.
This gives much the same treatment to datetime.fromtimestamp(stamp, tz) as
the last batch of checkins gave to datetime.now(tz): do "the obvious"
thing with the tz argument instead of a senseless thing.
tzinfo.fromutc() method. The C code doesn't implement any of this
yet (well, not the C code on the machine I'm using now), nor does
the test suite reflect it. The Python datetime.py implementation and
test suite in the sandbox do match these doc changes. The C
implementation probably won't catch up before Thursday (Wednesday is
a scheduled "black hole" day this week <0.4 wink>).
Andrew MacIntyre
Raymond Hettinger
Hye-Shik Chang
Jeremy Hylton
Thomas Heller
Fred Drake
Walter Dörwald
Tim Peters
Guido van Rossum
Martin v. Löwis
Neil Schemenauer
Anthony Baxter
Skip Montanaro
Armin Rigo
Alex Martelli
Brett Cannon
Gustavo Niemeyer
Marc-André Lemburg
Barry Warsaw
Andrew M. Kuchling
Jack Jansen
Neal Norwitz
Just van Rossum
Michael W. Hudson
David Goodger