This was mostly a matter of adding comments and light code rearrangement.
Upon untracking, gc_next is still set to NULL. It's a cheap way to
provoke memory faults if calling code is insane. It's also used in some
way by the trashcan mechanism.
object should now have a well-defined gc_refs value, with clear transitions
among gc_refs states. As a result, none of the visit_XYZ traversal
callbacks need to check IS_TRACKED() anymore, and those tests were removed.
(They were already looking for objects with specific gc_refs states, and
the gc_refs state of an untracked object can no longer match any other
gc_refs state by accident.)
Added more asserts.
I expect that the gc_next == NULL indicator for an untracked object is
now redundant and can also be removed, but I ran out of time for this.
in gc_refs, even at the cost of putting back a test+branch in
visit_decref.
The good news: since gc_refs became utterly tame then, it became
clear that another special value could be useful. The move_roots() and
move_root_reachable() passes have now been replaced by a single
move_unreachable() pass. Besides saving a pass over the generation, this
has a better effect: most of the time everything turns out to be
reachable, so we were breaking the generation list apart and moving it
into into the reachable list, one element at a time. Now the reachable
stuff stays in the generation list, and the unreachable stuff is moved
instead. This isn't quite as good as it sounds, since sometimes we
guess wrongly that a thing is unreachable, and have to move it back again.
Still, overall, it yields a significant (but not dramatic) boost in
collection speed.
1. You're not supposed to call this with a NULL argument, although the
docs could be clearer about that. The other visit_XYZ() functions
don't bother to check. This doesn't either now, although it does
assert non-NULL-ness now.
2. It doesn't matter whether the object is currently tracked, so don't
bother checking that either (if it isn't currently tracked, it may
have some nonsense value in gc_refs, but it doesn't hurt to
decrement gibberish, and it's cheaper to do so than to make everyone
test for trackedness).
It would be nice to get rid of the other tests on IS_TRACKED. Perhaps
trackedness should not be a matter of not being in any gc list, but
should be a matter of being in a new "untracked" gc list. This list
simply wouldn't be involved in the collection mechanism. A newly
created object would be put in the untracked list. Tracking would
simply unlink it and move it into the gen0 list. Untracking would do
the reverse. No test+branch needed then. visit_move() may be vulnerable
then, though, and I don't know how this would work with the trashcan.
"The regression" is actually due to that 2.2.1 had a bug that prevented
the regression (which isn't a regression at all) from showing up. "The
regression" is actually a glitch in cyclic gc that's been there forever.
As the generation being collected is analyzed, objects that can't be
collected (because, e.g., we find they're externally referenced, or
are in an unreachable cycle but have a __del__ method) are moved out
of the list of candidates. A tricksy scheme uses negative values of
gc_refs to mark such objects as being moved. However, the exact
negative value set at the start may become "more negative" over time
for objects not in the generation being collected, and the scheme was
checking for an exact match on the negative value originally assigned.
As a result, objects in generations older than the one being collected
could get scanned too, and yanked back into a younger generation. Doing
so doesn't lead to an error, but doesn't do any good, and can burn an
unbounded amount of time doing useless work.
A test case is simple (thanks to Kevin Jacobs for finding it!):
x = []
for i in xrange(200000):
x.append((1,))
Without the patch, this ends up scanning all of x on every gen0 collection,
scans all of x twice on every gen1 collection, and x gets yanked back into
gen1 on every gen0 collection. With the patch, once x gets to gen2, it's
never scanned again until another gen2 collection, and stays in gen2.
Bugfix candidate, although the code has changed enough that I think I'll
need to port it by hand. 2.2.1 also has a different bug that causes
bound method objects not to get tracked at all (so the test case doesn't
burn absurd amounts of time in 2.2.1, but *should* <wink>).
Setting the buffer_text attribute to true causes the parser to collect
character data, waiting as long as possible to report it to the Python
callback. This can save an enormous number of callbacks from C to
Python, which can be a substantial performance improvement.
buffer_text defaults to false.
The handlers array on each parser now has the invariant that None will
never be set as a handler; it will always be NULL or a Python-level
value passed in for the specific handler.
have_handler(): Return true if there is a Python handler for a
particular event.
get_handler_name(): Return a string object giving the name of a
particular handler. This caches the string object so it doesn't
need to be created more than once.
get_parse_result(): Helper to allow the Parse() and ParseFile()
methods to share the same logic for determining the return value
or exception state.
PyUnknownEncodingHandler(), PyModule_AddIntConstant():
Made these helpers static. (The later is only defined for older
versions of Python.)
pyxml_UpdatePairedHandlers(), pyxml_SetStartElementHandler(),
pyxml_SetEndElementHandler(), pyxml_SetStartNamespaceDeclHandler(),
pyxml_SetEndNamespaceDeclHandler(), pyxml_SetStartCdataSection(),
pyxml_SetEndCdataSection(), pyxml_SetStartDoctypeDeclHandler(),
pyxml_SetEndDoctypeDeclHandler():
Removed. These are no longer needed with Expat 1.95.x.
handler_info:
Use the setter functions provided by Expat 1.95.x instead of the
pyxml_Set*Handler() functions which have been removed.
Minor code formatting changes for consistency.
Trailing whitespace removed.
- Include a blank line between the signature line and the description
(Guido sez).
- Don't include "-> None" for API functions that always return None
because they don't have a meaningful return value.
These built-in functions are replaced by their (now callable) type:
slice()
buffer()
and these types can also be called (but have no built-in named
function named after them)
classobj (type name used to be "class")
code
function
instance
instancemethod (type name used to be "instance method")
The module "new" has been replaced with a small backward compatibility
placeholder in Python.
A large portion of the patch simply removes the new module from
various platform-specific build recipes. The following binary Mac
project files still have references to it:
Mac/Build/PythonCore.mcp
Mac/Build/PythonStandSmall.mcp
Mac/Build/PythonStandalone.mcp
[I've tweaked the code layout and the doc strings here and there, and
added a comment to types.py about StringTypes vs. basestring. --Guido]
library. Since multiple versions can be installed simultaneously, it's
crucial that you only select libraries and header files which are compatible
with each other. Version checking is done from highest version to lowest.
Building using version 1 of Berkeley DB is disabled by default because of
the hash file bugs people keep rediscovering. It can be enabled by
uncommenting a few lines in setup.py. Closes patch 553108.
that retries the connect() call in timeout mode so it can be shared
between connect() and connect_ex(), and needs only a single #ifdef.
The test for this was doing funky stuff I don't approve of,
so I removed it in favor of a simpler test. This allowed me
to implement a simpler, "purer" form of the timeout retry code.
Hopefully that's enough (if you want to be fancy, use non-blocking
mode and decode the errors yourself, like before).
- setblocking(0) and settimeout(0) are now equivalent, and ditto for
setblocking(1) and settimeout(None).
- Don't raise an exception from internal_select(); let the final call
report the error (this means you will get an EAGAIN error instead of
an ETIMEDOUT error -- I don't care).
- Move the select to inside the Py_{BEGIN,END}_ALLOW_THREADS brackets,
so other theads can run (this was a bug in the original code).
- Redid the retry logic in connect() and connect_ex() to avoid masking
errors. This probably doesn't work for Windows yet; I'll fix that
next. It may also fail on other platforms, depending on what
retrying a connect does; I need help with this.
- Get rid of the retry logic in accept(). I don't think it was needed
at all. But I may be wrong.
settimeout(). Already, settimeout() canceled non-blocking mode; now,
setblocking() also cancels the timeout. This is easier to document.
(XXX should settimeout(0) be an alias for setblocking(0)? They seem
to have roughly the same effect. Also, I'm not sure that the code in
connect() and accept() is correct in all cases. We'll sort this out
soon enough.)
not testing it -- apparently test_timeout.py doesn't test anything
useful):
In internal_select():
- The tv_usec part of the timeout for select() was calculated wrong.
- The first argument to select() was one too low.
- The sense of the direction argument to internal_select() was
inverted.
In PySocketSock_settimeout():
- The calls to internal_setblocking() were swapped.
Also, repaired some comments and fixed the test for the return value
of internal_select() in sendall -- this was in the original patch.
I've made considerable changes to Michael's code, specifically to use
the select() system call directly and to store the timeout as a C
double instead of a Python object; internally, -1.0 (or anything
negative) represents the None from the API.
I'm not 100% sure that all corner cases are covered correctly, so
please keep an eye on this. Next I'm going to try it Windows before
Tim complains.
No way is this a bugfix candidate. :-)
readline in all python versions is configured
to append a 'space' character for a successful
completion. But for almost all python expressions
'space' is not wanted (see coding conventions PEP 8).
For example if you have a function 'longfunction'
and you type 'longf<TAB>' you get 'longfunction '
as a completion. note the unwanted space at the
end.
The patch fixes this behaviour by setting readline's
append_character to '\0' which means don't append
anything. This doesn't work with readline < 2.1
(AFAIK nowadays readline2.2 is in good use).
An alternative approach would be to make the
append_character
accessable from python so that modules like
the rlcompleter.py can set it to '\0'.
[Ed.: I think expecting readline >= 2.2 is fine. If a completer wants
another character they can append that to the keyword in the list.]
[ 559250 ] more POSIX signal stuff
Adds support (and docs and tests and autoconfery) for posix signal
mask handling -- sigpending, sigprocmask and sigsuspend.
The old syntax suggested that a trailing comma was OK inside backticks,
but in fact (due to ideosyncrasies of pgen) it was not. Fix the grammar
to avoid the ambiguity. Fred: you may want to update the refman.
generations is now an array. This cleans up some code and makes it easy
to change the number of generations. Also, implemented a
gc_list_is_empty() function. This makes the logic a little clearer in
places. The performance impact of these changes should be negligible.
One functional change is that allocation/collection counters are always
zeroed at the start of a collection. This should fix SF bug #551915.
This change is too big for back-porting but the minimal patch on SF
looks good for a bugfix release.
Unicode objects are currently taken as binary data by the write()
method. This is not what Unicode users expect, nor what the
StringIO.py code does. Until somebody adds a way to specify binary or
text mode for cStringIO objects, change the format string to use "t#"
instead of "s#", so that it will request the "text buffer" version.
This will try the default encoding for Unicode objects.
This is *not* a 2.2 bugfix (since it *is* a semantic change).
don't understand how this function works, also beefed up the docs. The
most common usage error is of this form (often spread out across gotos):
if (_PyString_Resize(&s, n) < 0) {
Py_DECREF(s);
s = NULL;
goto outtahere;
}
The error is that if _PyString_Resize runs out of memory, it automatically
decrefs the input string object s (which also deallocates it, since its
refcount must be 1 upon entry), and sets s to NULL. So if the "if"
branch ever triggers, it's an error to call Py_DECREF(s): s is already
NULL! A correct way to write the above is the simpler (and intended)
if (_PyString_Resize(&s, n) < 0)
goto outtahere;
Bugfix candidate.
closes SF #514433
can now pass 'None' as the filename for the bsddb.*open functions,
and you'll get an in-memory temporary store.
docs are ripped out of the bsddb dbopen man page. Fred may want to
clean them up.
Considering this for 2.2, but not 2.1.
and returns None. This allows any object that supports the fileno()
method to be passed as a file descriptor, not just an integer.
posix_fchdir(): New exposed function: implements posix.fchdir(). This
closes SF feature #536796.
posix_fsync(), posix_fdatasync(): Convert to use posix_fildes() instead
of posix_int(). This also changes them from METH_VARARGS to METH_O
functions.
setup_confname_table(): Remove unused variable. Change to take a module
rather than a dict to save the resulting table into.
setup_confname_tables(): Change to take a module instead of a dict to
pass to setup_confname_table().
PyMem_{Del, DEL} doesn't work yet (compilation problems).
pyport.h: _PyMem_EXTRA is gone.
pmem.h: Repaired comments. PyMem_{Malloc, MALLOC} and
PyMem_{Realloc, REALLOC} now make the same x-platform guarantees when
asking for 0 bytes, and when passing a NULL pointer to the latter.
object.c: PyMem_{Malloc, Realloc} just call their macro versions
now, since the latter take care of the x-platform 0 and NULL stuff
by themselves now.
pypcre.c, grow_stack(): So sue me. On two lines, this called
PyMem_RESIZE to grow a "const" area. It's not legit to realloc a
const area, so the compiler warned given the new expansion of
PyMem_RESIZE. It would have gotten the same warning before if it
had used PyMem_Resize() instead; the older macro version, but not the
function version, silently cast away the constness. IMO that was a wrong
thing to do, and the docs say the macro versions of PyMem_xyz are
deprecated anyway. If somebody else is resizing const areas with the
macro spelling, they'll get a warning when they recompile now too.
compatibility function.
Make PyObject_GC_Track and PyObject_GC_UnTrack functions instead of
trivial macros wrapping functions. Provide binary compatibility
functions.
Change pickling format for bools to use a backwards compatible
encoding. This means you can pickle True or False on Python 2.3
and Python 2.2 or before will read it back as 1 or 0. The code
used for pickling bools before would create pickles that could
not be read in previous Python versions.
PEP 285. Everything described in the PEP is here, and there is even
some documentation. I had to fix 12 unit tests; all but one of these
were printing Boolean outcomes that changed from 0/1 to False/True.
(The exception is test_unicode.py, which did a type(x) == type(y)
style comparison. I could've fixed that with a single line using
issubtype(x, type(y)), but instead chose to be explicit about those
places where a bool is expected.
Still to do: perhaps more documentation; change standard library
modules to return False/True from predicates.
457466: popenx() argument mangling hangs python
226766: popen('python -c"...."') tends to hang
Fixes argument quoting in w9xpopen.exe for Windows 9x. w9xpopen.exe
also never attempts to display a MessageBox when not executed
interactively.
Added test_popen() test. This test currently just executes
"python -c ..." as a child process, and checks that the expected
arguments were all recieved correctly by the child process. This
test succeeds for me on Win9x, win2k and Linux, and I hope it does
for other popen supported platforms too :)
Jeremy Hylton
Tim Peters
Fred Drake
Jack Jansen
Martin v. Löwis
Neil Schemenauer
Michael W. Hudson
Guido van Rossum
Skip Montanaro
Neal Norwitz
Andrew MacIntyre
Raymond Hettinger
Anthony Baxter
Thomas Heller
Mark Hammond