Cygwin's pthread_sigmask() implementation appears to be buggy. This
patch works around this problem by using sigprocmask() instead.
This patch is implemented in a general way so it could be used by other
platforms too. If this approach is deemed too risky, then I can work up
a patch that just hacks Python/thread_pthread.h for Cygwin.
Note that I tested this patch against 2.3c1 under Red Hat Linux 8.0 too.
[snip]
And finally, I need someone to regenerate pyconfig.h.in and configure
with the same versions of the autotools that are normally used by
Python.
Neal kindly regenerated pyconfig.h.in and configure for me.
If the initial import of warnings fails, clear the error. When the module
is actually needed, if the original import failed, see if it has managed
to find its way to sys.modules yet and if so, remember it.
Fixes for three related bugs, including errors that caused a script to
be ignored without printing an error message. The key problem was a bad
interaction between syntax warnings and syntax errors. If an
exception was already set when a warning was issued, the warning could
clobber the exception.
The PyErr_Occurred() check in issue_warning() isn't entirely
satisfying (the caller should know whether there was already an
error), but a better solution isn't immediately obvious.
Bug fix candidate.
behavior, creating many threads very quickly. A long debugging session
revealed that the Windows implementation of PyThread_start_new_thread()
was choked with "laziness" errors:
1. It checked MS _beginthread() for a failure return, but when that
happened it returned heap trash as the function result, instead of
an id of -1 (the proper error-return value).
2. It didn't consider that the Win32 CreateSemaphore() can fail.
3. When creating a great many threads very quickly, it's quite possible
that any particular bootstrap call can take virtually any amount of
time to return. But the code waited for a maximum of 5 seconds, and
didn't check to see whether the semaphore it was waiting for got
signaled. If it in fact timed out, the function could again return
heap trash as the function result. This is actually what confused
the test program, as the heap trash usually turned out to be 0, and
then multiple threads all got id 0 simultaneously, confusing the
hell out of threading.py's _active dict (mapping id to thread
object). A variety of baffling behaviors followed from that.
WRT #1 and #2, error returns are checked now, and "thread.error: can't
start new thread" gets raised now if a new thread (or new semaphore)
can't be created. WRT #3, we now wait for the semaphore without a
timeout.
Also removed useless local vrbls, folded long lines, and changed callobj
to a stack auto (it was going thru malloc/free instead, for no discernible
reason).
Bugfix candidate.
A new API (only accessible from C) to interrupt a thread by sending it
an exception. This is not always effective, but might help some people.
Requested by Just van Rossum and Alex Martelli. It is intentional
that you have to write your own C extension to call it from Python.
Docs will have to wait.
It depended on the previously removed basic block checker to
prevent a jump into the middle of the transformed block.
Clears SF 757818: tuple assignment -- SystemError: unknown opcode
The compiler was reseting the list comprehension tmpname counter for each function, but the symtable was using the same counter for the entire module. Repair by move tmpname into the symtable entry.
Bugfix candidate.
[ 708901 ] Lineno calculation sometimes broken
A one line patch to compile.c and a rather-more-than-one-line patch
to test_dis. Hey ho.
Possibly a backport candidate -- tho' lnotab is less used in 2.2...
* Can now test for basic blocks.
* Optimize inverted comparisions.
* Optimize unary_not followed by a conditional jump.
* Added a new opcode, NOP, to keep code size constant.
* Applied NOP to previous transformations where appropriate.
Note, the NOP would not be necessary if other functions were
added to re-target jump addresses and update the co_lnotab mapping.
That would yield slightly faster and cleaner bytecode at the
expense of optimizer simplicity and of keeping it decoupled
from the line-numbering structure.
particular leaving the traceback object (and everything reachable
from it) alive throughout shutdown. The patch is mostly from Guido.
Bugfix candidate.
new line.
New pvt API function _Py_PrintReferenceAddresses(): Prints only the
addresses and refcnts of the live objects. This is always safe to call,
because it has no dependence on Python's C API.
Py_Finalize(): If envar PYTHONDUMPREFS is set, call (the new)
_Py_PrintReferenceAddresses() right before dumping final pymalloc stats.
We can't print the reprs of the objects here because too much of the
interpreter has been shut down. You need to correlate the addresses
displayed here with the object reprs printed by the earlier
PYTHONDUMPREFS call to _Py_PrintReferences().
New functions:
unsigned long PyInt_AsUnsignedLongMask(PyObject *);
unsigned PY_LONG_LONG) PyInt_AsUnsignedLongLongMask(PyObject *);
unsigned long PyLong_AsUnsignedLongMask(PyObject *);
unsigned PY_LONG_LONG) PyLong_AsUnsignedLongLongMask(PyObject *);
New and changed format codes:
b unsigned char 0..UCHAR_MAX
B unsigned char none **
h unsigned short 0..USHRT_MAX
H unsigned short none **
i int INT_MIN..INT_MAX
I * unsigned int 0..UINT_MAX
l long LONG_MIN..LONG_MAX
k * unsigned long none
L long long LLONG_MIN..LLONG_MAX
K * unsigned long long none
Notes:
* New format codes.
** Changed from previous "range-and-a-half" to "none"; the
range-and-a-half checking wasn't particularly useful.
New test test_getargs2.py, to verify all this.
PYTHONDUMPREFS output after most teardown. Attempts to use
PYTHONDUMPREFS with the Zope3 test suite died with Py_FatalError(),
since _Py_PrintReferences() can end up executing arbitrary Python code
(for objects that override __repr__), and that requires an intact
interpreter.
even farther down, to just before the call to
_PyObject_DebugMallocStats(). This required the following changes:
- pystate.c, PyThreadState_GetDict(): changed not to raise an
exception or issue a fatal error when no current thread state is
available, but simply return NULL without raising an exception
(ever).
- object.c, Py_ReprEnter(): when PyThreadState_GetDict() returns NULL,
don't raise an exception but return 0. This means that when
printing a container that's recursive, printing will go on and on
and on. But that shouldn't happen in the case we care about (see
first bullet).
- Updated Misc/NEWS and Doc/api/init.tex to reflect changes to
PyThreadState_GetDict() definition.
the code erroneously decrefed the istep argument in an error case. This
caused a co_consts tuple to lose a float constant prematurely, which
eventually caused gc to try executing static data in floatobject.c (don't
ask <wink>). So reworked this extensively to ensure refcount correctness.
- range() now works even if the arguments are longs with magnitude
larger than sys.maxint, as long as the total length of the sequence
fits. E.g., range(2**100, 2**101, 2**100) is the following list:
[1267650600228229401496703205376L]. (SF patch #707427.)
return. Setting an exception can mess with the exception state, and
continuing is definitely wrong (since type is dereferenced later on).
Some code that calls this seems to be prepared for a NULL exception
type, so let's be safe rather than sorry and simply assume there's
nothing to normalize in this case.
Adds a single function to improve generated bytecode. Has a single line
attachment point, so it is completely de-coupled from both the compiler
and ceval.c.
Makes three simple transforms that do not require a basic block analysis
or re-ordering of code. Gives improved timings on pystone, pybench,
and any code using either "while 1" or "x,y=y,x".
Arranged that all the objects exposed by __builtin__ appear in the list
of all objects. I basically peed away two days tracking down a mystery
leak in sys.gettotalrefcount() in a ZODB app (== tons of code), because
the object leaking the references didn't appear in the sys.getobjects(0)
list. The object happened to be False. Now False is in the list, along
with other popular & previously missing leak candidates (like None).
Alas, we still don't have a choke point covering *all* Python objects,
so the list of all objects may still be incomplete.
variables to store internal data. As a result, any atempts to use the
unicode system with multiple active interpreters, or successive
interpreter executions, would fail.
Now that information is stored into members of the PyInterpreterState
structure.
Added two predictions:
GET_ITER --> FOR_ITER
FOR_ITER --> STORE_FAST or UNPACK_SEQUENCE
Improves timings on pybench and timeit.py. Pystone results are neutral.
Applied to common cases:
COMPARE_OP is often followed by a JUMP_IF.
JUMP_IF is usually followed by POP_TOP.
Shows improved timings on PyStone, PyBench, and specific tests
using timeit.py:
python timeit.py -s "x=1" "if x==1: pass"
python timeit.py -s "x=1" "if x==2: pass"
python timeit.py -s "x=1" "if x: pass"
python timeit.py -s "x=100" "while x!=1: x-=1"
Potential future candidates:
GET_ITER predicts FOR_ITER
FOR_ITER predicts STORE_FAST or UNPACK_SEQUENCE
Also, applied missing goto fast_next_opcode to DUP_TOPX.
My previous patches should have used fast_next_opcode
in a few places instead of continue.
Also, applied one PyInt_AS_LONG macro in a place where
the type had already been checked.
rarely needed, but can sometimes be useful to release objects
referenced by the traceback held in sys.exc_info()[2]. (SF patch
#693195.) Thanks to Kevin Jacobs!
import warnings.py _after_ site.py has run. This ensures that site.py
is again the first .py to be imported, giving it back full control over
sys.path.
with an indented code block but no newline would raise SyntaxError.
This would have been a four-line change in parsetok.c... Except
codeop.py depends on this behavior, so a compilation flag had to be
invented that causes the tokenizer to revert to the old behavior;
this required extra changes to 2 .h files, 2 .c files, and 2 .py
files. (Fixes SF bug #501622.)
mostly from SF patch #683257, but I had to change unlock_import() to
return an error value to avoid fatal error.
Should this be backported? The patch requested this, but it's a new
feature.
"Unsigned" (i.e., positive-looking, but really negative) hex/oct
constants with a leading minus sign are once again properly negated.
The micro-optimization for negated numeric constants did the wrong
thing for such hex/oct constants. The patch avoids the optimization
for all hex/oct constants.
This needs to be backported to Python 2.2!
object is not a real str or unicode but an instance
of a subclass, construct the output via looping
over __getitem__. This guarantees that the result
is the same for function==None and function==lambda x:x
This doesn't happen for tuples, because filtertuple()
uses PyTuple_GetItem().
(This was discussed on SF bug #665835).
-DCALL_PROFILE: Count the number of function calls executed.
When this symbol is defined, the ceval mainloop and helper functions
count the number of function calls made. It keeps detailed statistics
about what kind of object was called and whether the call hit any of
the special fast paths in the code.
Optimization:
When we take the fast_function() path, which seems to be taken for
most function calls, and there is minimal frame setup to do, avoid
call PyEval_EvalCodeEx(). The eval code ex function does a lot of
work to handle keywords args and star args, free variables,
generators, etc. The inlined version simply allocates the frame and
copies the arguments values into the frame.
The optimization gets a little help from compile.c which adds a
CO_NOFREE flag to code objects that don't have free variables or cell
variables. This change allows fast_function() to get into the fast
path with fewer tests.
I measure a couple of percent speedup in pystone with this change, but
there's surely more that can be done.
blindly assumed that tp_as_sequence->sq_item always returns
a str or unicode object. This might fail with str or unicode
subclasses.
This patch checks whether the object returned from __getitem__
is a str/unicode object and raises a TypeError if not (and
the filter function returned true).
Furthermore the result for __getitem__ can be more than one
character long, so checks for enough memory have to be done.
__module__ is the string name of the module the function was defined
in, just like __module__ of classes. In some cases, particularly for
C functions, the __module__ may be None.
Change PyCFunction_New() from a function to a macro, but keep an
unused copy of the function around so that we don't change the binary
API.
Change pickle's save_global() to use whichmodule() if __module__ is
None, but add the __module__ logic to whichmodule() since it might be
used outside of pickle.
Make the code slightly shorter, faster, and easier to
read.
* Eliminate unused DUP_TOPX code for x==1.
compile.c always generates DUP_TOP instead.
* Since only two cases remain for DUP_TOPX, replace
the switch-case with if-elseif.
* The in-lined integer compare does a CheckExact on
both arguments. Since the second is a little more
likely to fail, test it first.
* The switch-case for IS/IS_NOT and IN/NOT_IN can
separate the regular and inverted cases with no
additional work. For all four paths, saves a test and
jump.
the AEDesc data shouldn't be disposed when the Python object is.
Added a C call AEDesc_NewBorrowed() to create these objects and a Python
method old=AEDesc.AutoDispose(onoff) to change auto-dispose state.
The two are semantically equivalent, but the first triggered a compiler
warning about an unused variable. Note, the preceding steps had already
accessed and decreffed the variable so the reference counts were fine.
parameter being either four or five. Currently, compile.c does not
generate calls with a parameter higher than three.
May have to be reverted if the second alpha or beta shakes out some
other tool generating this op code with a parameter of four or five.
Replaced groups of pushes and pops with indexed access to the stack and
a single adjustment (if needed) to the stacklevel.
Avoids scores of unnecessary increments and decrements to the stackpointer.
Removes unnecessary sequential dependencies so that the compiler has more
freedom for optimizations. Frees the processor for more parallel and
pipelined execution by using mostly read-only access and having few pointer
adjustments just prior to a read or write.
This fixes the problem on Windows - that's the only system where I can
test it.
It leaves sys.argv alone and only changes sys.path[0] to an absolute
pathname.
A variety of changes from Michael Hudson to get the compiler working
with 2.3. The primary change is the handling of SET_LINENO:
# The set_lineno() function and the explicit emit() calls for
# SET_LINENO below are only used to generate the line number table.
# As of Python 2.3, the interpreter does not have a SET_LINENO
# instruction. pyassem treats SET_LINENO opcodes as a special case.
A few other small changes:
- Remove unused code from pycodegen and pyassem.
- Fix error handling in parsermodule. When PyParser_SimplerParseString()
fails, it sets an exception with detailed info. The parsermodule
was clobbering that exception and replacing it was a generic
"could not parse string" exception. Keep the original exception.
I can't test this on the snake farm (no aix box is working).
This change works for the submitter seems correct.
Can anybody test this on 32- and 64- bit AIX?
Initialize the small integers and __builtins__ in startup.
This removes some if conditions.
Change XDECREF to DECREF for values which shouldn't be NULL.
- new import hooks in import.c, exposed in the sys module
- new module called 'zipimport'
- various changes to allow bootstrapping from zip files
I hope I didn't break the Windows build (or anything else for that
matter), but then again, it's been sitting on sf long enough...
Regarding the latest discussions on python-dev: zipimport sets
pkg.__path__ as specified in PEP 273, and likewise, sys.path item such as
/path/to/Archive.zip/subdir/ are supported again.
Obtain cleaner coding and a system wide
performance boost by using the fast, pre-parsed
PyArg_Unpack function instead of PyArg_ParseTuple
function which is driven by a format string.
[#448679] Left to right
* Python/compile.c
(com_dictmaker): Reordered evaluation of dictionaries to follow strict
LTR evaluation.
* Lib/compiler/pycodegen.py
(CodeGenerator.visitDict): Reordered evaluation of dictionaries to
follow strict LTR evaluation.
* Doc/ref/ref5.tex
Documented the general LTR evaluation order idea.
* Misc/NEWS
Documented change in evaluation order of dictionaries.
supported as the second argument. This has the same meaning as
for isinstance(), i.e. issubclass(X, (A, B)) is equivalent
to issubclass(X, A) or issubclass(X, B). Compared to isinstance(),
this patch does not search the tuple recursively for classes, i.e.
any entry in the tuple that is not a class, will result in a
TypeError.
This closes SF patch #649608.
Py_Init crash". refchain cannot be cleared because objects can live across
Py_Finalize() and Py_Initialize() if they are kept alive by circular
references.
dialogs are now stored in Mac/Lib, and loaded on demand through macresource.
Not only does this simplify a MacPython based on Apple's Python, but
it also makes Mac error codes come out symbolically when running command
line python (if you have Mac/Lib in your path).
The resource files are copied from Mac/Resources. The old ones will disappear
after the OS9 build procedure has been adjusted.
sys.getwindowsversion() on Windows (new enahanced Tim-proof <wink>
version), and fix test_pep277.py in a few minor ways.
Including doc and NEWS entries.
all along. Before instr_lb tended to be too high.
I don't think this actually makes any difference, given what the compiler
produces, but it makes me a bit happier.
patch #617312, both on the trunk and the 22-maint branch.
Also added a test case, and ported the test_trace I wrote for HEAD
to 2.2.2 (with all those horrible extra 'line' events ;-).
The switch in Exception__str__ didn't clear the error if
PySequence_Size() raised an exception. Added a case -1 which clears
the error and falls through to the default case.
Definite backport candidate (this dates all the way to Python 2.0).
than when this interval was first established. Checking too frequently just
adds needless overhead because most of the time there is nothing to do and
no other threads ready to run.
globals, _Py_Ticker and _Py_CheckInterval. This also implements Jeremy's
shortcut in Py_AddPendingCall that zeroes out _Py_Ticker. This allows the
test in the main loop to only test a single value.
The gory details are at
http://python.org/sf/602191
Use a slightly different strategy to determine when not to call the line
trace function. This removes the need for the RETURN_NONE opcode, so
that's gone again. Update docs and comments to match.
Thanks to Neal and Armin!
Also add a test suite. This should have come with the original patch...
in LOAD_GLOBAL. Besides saving a C function call, it saves checks
whether f_globals and f_builtins are dicts, and extracting and testing
the string object's hash code is done only once. We bail out of the
inlining if the name is not exactly a string, or when its hash is -1;
because of interning, neither should ever happen. I believe interning
guarantees that the hash code is set, and I believe that the 'names'
tuple of a code object always contains interned strings, but I'm not
assuming that -- I'm simply testing hash != -1.
On my home machine, this makes a pystone variant with new-style
classes and slots run at the same speed as classic pystone! (With
new-style classes but without slots, it is still a lot slower.)
Also, don't handle METH_OLDARGS on the fast path. All the interesting
builtins have been converted to use METH_NOARGS, METH_O, or
METH_VARARGS.
Result is another 1-2% speedup. If I can cobble together 10 of these,
it might make a difference.
This makes the code much easier to ready, because it is at a sane
indentation level. On my box this shows a 1-2% speedup, which means
nothing, except that I'm not going to worry about the performance
effects of the change.
nothing special done if keyword arguments were present, so test for
that earlier and fall through to the normal case if there are any.
This ought to slow down CFunction calls with keyword args, but I don't
care; it's a tiny (1%) improvement for pystone.
- Use PyObject_Call() instead of PyEval_CallObject(), saves several
layers of calls and checks.
- Pre-allocate the argument tuple rather than calling Py_BuildValue()
each time round the loop.
- For filter(None, seq), avoid an INCREF and a DECREF.
warning for 'global None', but that's either accompanied by an
assignment to None, which will trigger a warning, or not, in which
case it's harmless. :-)
[ 587993 ] SET_LINENO killer
Remove SET_LINENO. Tracing is now supported by inspecting co_lnotab.
Many sundry changes to document and adapt to this change.
currently return inconsistent results for ints and longs; in
particular: hex/oct/%u/%o/%x/%X of negative short ints, and x<<n that
either loses bits or changes sign. (No warnings for repr() of a long,
though that will also change to lose the trailing 'L' eventually.)
This introduces some warnings in the test suite; I'll take care of
those later.
Change the parser and compiler to use PyMalloc.
Only the files implementing processes that will request memory
allocations small enough for PyMalloc to be a win have been
changed, which are:-
- Python/compile.c
- Parser/acceler.c
- Parser/node.c
- Parser/parsetok.c
This augments the aggressive overallocation strategy implemented by
Tim Peters in PyNode_AddChild() [Parser/node.c], in reducing the
impact of platform malloc()/realloc()/free() corner case behaviour.
Such corner cases are known to be triggered by test_longexp and
test_import.
Jeremy Hylton, in accepting this patch, recommended this as a
bugfix candidate for 2.2. While the changes to Python/compile.c
and Parser/node.c backport easily (and could go in), the changes
to Parser/acceler.c and Parser/parsetok.c require other not
insignificant changes as a result of the differences in the memory
APIs between 2.3 and 2.2, which I'm not in a position to work
through at the moment. This is a pity, as the Parser/parsetok.c
changes are the most important after the Parser/node.c changes, due
to the size of the memory requests involved and their frequency.
actual script to run in case we are running from an applet. If we are indeed
running an applet we skip the normal option processing leaving it all to the
applet code.
This allows us to get use the normal python binary in the Python.app bundle,
giving us all the normal command line options through PythonLauncher while
still allowing Python.app to be used as the template for building applets.
Consequently, pythonforbundle is gone, and Mac/Python/macmain.c isn't used
on OSX anymore.
See there for a description.
Added test case.
Bugfix candidate for 2.2.x, not sure about previous versions:
probably low priority, because virtually no one runs debug builds.
PyErr_SetExcFromWindowsErr(), PyErr_SetExcFromWindowsErrWithFilename().
Similar to PyErr_SetFromWindowsErrWithFilename() and
PyErr_SetFromWindowsErr(), but they allow to specify
the exception type to raise. Available on Windows.
See SF patch #576458.
is slow things down unnecessarily and make tracing much more verbose.
Something like
def f(n):
return [i for i in range(n) if i%2]
should have at most two SET_LINENO instructions, not four. When tracing,
the current line number should be printed once, not 2*n+1 times.
The staticforward define was needed to support certain broken C
compilers (notably SCO ODT 3.0, perhaps early AIX as well) botched the
static keyword when it was used with a forward declaration of a static
initialized structure. Standard C allows the forward declaration with
static, and we've decided to stop catering to broken C compilers. (In
fact, we expect that the compilers are all fixed eight years later.)
I'm leaving staticforward and statichere defined in object.h as
static. This is only for backwards compatibility with C extensions
that might still use it.
XXX I haven't updated the documentation.
more trivial lexical helper macros so that uses of these guys expand
to nothing at all when they're not enabled. This should help sub-
standard compilers that can't do a good job of optimizing away the
previous "(void)0" expressions.
Py_DECREF: There's only one definition of this now. Yay! That
was that last one in the family defined multiple times in an #ifdef
maze.
Py_FatalError(): Changed the char* signature to const char*.
_Py_NegativeRefcount(): New helper function for the Py_REF_DEBUG
expansion of Py_DECREF. Calling an external function cuts down on
the volume of generated code. The previous inline expansion of abort()
didn't work as intended on Windows (the program often kept going, and
the error msg scrolled off the screen unseen). _Py_NegativeRefcount
calls Py_FatalError instead, which captures our best knowledge of
how to abort effectively across platforms.
that have taken me "too long" to reverse-engineer over the years.
Vastly reduced the nesting level and redundancy of #ifdef-ery.
Took a light stab at repairing comments that are no longer true.
sys_gettotalrefcount(): Changed to enable under Py_REF_DEBUG.
It was enabled under Py_TRACE_REFS, which was much heavier than
necessary. sys.gettotalrefcount() is now available in a
Py_REF_DEBUG-only build.
This patch enhances Python/import.c/find_module() so
that unicode objects found in sys.path will be treated
as legal directory names (The current code ignores
anything that is not a str). The unicode name is
converted to str using Py_FileSystemDefaultEncoding.
These built-in functions are replaced by their (now callable) type:
slice()
buffer()
and these types can also be called (but have no built-in named
function named after them)
classobj (type name used to be "class")
code
function
instance
instancemethod (type name used to be "instance method")
The module "new" has been replaced with a small backward compatibility
placeholder in Python.
A large portion of the patch simply removes the new module from
various platform-specific build recipes. The following binary Mac
project files still have references to it:
Mac/Build/PythonCore.mcp
Mac/Build/PythonStandSmall.mcp
Mac/Build/PythonStandalone.mcp
[I've tweaked the code layout and the doc strings here and there, and
added a comment to types.py about StringTypes vs. basestring. --Guido]
Write 4 bytes for co_stacksize, etc. to prevent writing out
bad .pyc files which can cause a crash when read back in.
(I forgot that frozen needs to be updated too for the test.)
This was a simple typo. Strange that the compiler didn't catch it!
Instead of WHY_CONTINUE, two tests used CONTINUE_LOOP, which isn't a
why_code at all, but an opcode; but even though 'why' is declared as
an enum, comparing it to an int is apparently not even worth a
warning -- not in gcc, and not in VC++. :-(
Will fix in 2.2 too.
get_file() must convert 'U' to "r" PY_STDIOTEXTMODE before calling
fopen().
imp_load_module() must accept 'r' or 'U' or something with '+'.
Also reflow some long lines.
for 'str' and 'unicode', and can be used instead of
types.StringTypes, e.g. to test whether something is "a string":
isinstance(x, string) is True for Unicode and 8-bit strings. This
is an abstract base class and cannot be instantiated directly.
The old syntax suggested that a trailing comma was OK inside backticks,
but in fact (due to ideosyncrasies of pgen) it was not. Fix the grammar
to avoid the ambiguity. Fred: you may want to update the refman.
[ 558249 ] softspace vs --disable-unicode
And #endif was in the wrong place.
Bugfix candidate, almost surely.
I think I will embark on squashing test failures in --disable-unicode builds --
a Real Bug was hiding under them.
don't understand how this function works, also beefed up the docs. The
most common usage error is of this form (often spread out across gotos):
if (_PyString_Resize(&s, n) < 0) {
Py_DECREF(s);
s = NULL;
goto outtahere;
}
The error is that if _PyString_Resize runs out of memory, it automatically
decrefs the input string object s (which also deallocates it, since its
refcount must be 1 upon entry), and sets s to NULL. So if the "if"
branch ever triggers, it's an error to call Py_DECREF(s): s is already
NULL! A correct way to write the above is the simpler (and intended)
if (_PyString_Resize(&s, n) < 0)
goto outtahere;
Bugfix candidate.
The SIGXFSZ signal is sent when the maximum file size limit is
exceeded (RLIMIT_FSIZE). Apparently, it is also sent when the 2GB
file limit is reached on platforms without large file support.
The default action for SIGXFSZ is to terminate the process and dump
core. When it is ignored, the system call that caused the limit to be
exceeded returns an error and sets errno to EFBIG. Python
always checks errno on I/O syscalls, so there is nothing to do with
the signal.
+ Redirect PyMem_{Del, DEL} to the object allocator's free() when
pymalloc is enabled. Needed so old extensions can continue to
mix PyObject_New with PyMem_DEL.
+ This implies that pgen needs to be able to see the PyObject_XYZ
declarations too. pgenheaders.h now includes Python.h. An
implication is that I expect obmalloc.o needs to get linked into
pgen on non-Windows boxes.
+ When PYMALLOC_DEBUG is defined, *all* Py memory API functions
now funnel through the debug allocator wrapper around pymalloc.
This is the default in a debug build.
+ That caused compile.c to fail: it indirectly mixed PyMem_Malloc
with raw platform free() in one place. This is verbotten.
Highlights: import and friends will understand any of \r, \n and \r\n
as end of line. Python file input will do the same if you use mode 'U'.
Everything can be disabled by configuring with --without-universal-newlines.
See PEP278 for details.
Added code to call this when PYMALLOC_DEBUG is enabled, and envar
PYTHONMALLOCSTATS is set, whenever a new arena is obtained and once
late in the Python shutdown process.
PEP 285. Everything described in the PEP is here, and there is even
some documentation. I had to fix 12 unit tests; all but one of these
were printing Boolean outcomes that changed from 0/1 to False/True.
(The exception is test_unicode.py, which did a type(x) == type(y)
style comparison. I could've fixed that with a single line using
issubtype(x, type(y)), but instead chose to be explicit about those
places where a bool is expected.
Still to do: perhaps more documentation; change standard library
modules to return False/True from predicates.
SF bug 535905 (Evil Trashcan and GC interaction).
The SETLOCAL() macro should not DECREF the local variable in-place and
then store the new value; it should copy the old value to a temporary
value, then store the new value, and then DECREF the temporary value.
This is because it is possible that during the DECREF the frame is
accessed by other code (e.g. a __del__ method or gc.collect()) and the
variable would be pointing to already-freed memory.
BUGFIX CANDIDATE!
descriptor, as used for the tp_methods slot of a type. These new flag
bits are both optional, and mutually exclusive. Most methods will not
use either. These flags are used to create special method types which
exist in the same namespace as normal methods without having to use
tedious construction code to insert the new special method objects in
the type's tp_dict after PyType_Ready() has been called.
If METH_CLASS is specified, the method will represent a class method
like that returned by the classmethod() built-in.
If METH_STATIC is specified, the method will represent a static method
like that returned by the staticmethod() built-in.
These flags may not be used in the PyMethodDef table for modules since
these special method types are not meaningful in that case; a
ValueError will be raised if these flags are found in that context.
This fixes the symptom, but PRINT_ITEM has no way to know what (if
anything) PyFile_WriteObject() writes unless the object being printed
is a string. When the object isn't a string, this fix retains the
guess that softspace should be set after PyFile_WriteObject().
We might want to say that it's the job of filelike-object write methods
to leave the file's softspace in the correct state. That would probably
be better -- but everyone relies on PRINT_ITEM to guess for them now.
A file-static "threads" dict mapped thread IDs to Windows handles, but
was never referenced, and entries never got removed. This gets rid of
the YAGNI-dict entirely.
Bugfix candidate.
Python/
dynload_shlib.c // EMX port emulates dlopen() etc. for DL extensions
import.c // changes to support 8.3 DLL name limit (VACPP+EMX)
// and case sensitive import semantics
importdl.h
thread_os2.h
Fix for the UTF-8 decoder: it will now accept isolated surrogates
(previously it raised an exception which causes round-trips to
fail).
Added new tests for UTF-8 round-trip safety (we rely on UTF-8 for
marshalling Unicode objects, so we better make sure it works for
all Unicode code points, including isolated surrogates).
Bumped the PYC magic in a non-standard way -- please review. This
was needed because the old PYC format used illegal UTF-8 sequences
for isolated high surrogates which now raise an exception.
By default every module is imported in its own namespace, but this can
be changed by defining USE_DYLD_GLOBAL_NAMESPACE. In a future version this
define will be replaced by a runtime setting, but that needs a bit more
thought.
This code is largely based on code and feedback from Steven Majewski,
Marcel Prastawa, Manoj Plakal and other on pythonmac-sig.
type.__module__ problems (again?)
This simply initializes the __module__ local in a class statement from
the __name__ global. I'm not 100% sure that this is the correct fix,
although it usually does the right thing. The problem is that if the
class statement executes in a custom namespace, the __name__ global
may be taken from __builtins__, in which case it would have the value
__builtin__, or it may not exist at all (if the custom namespace also
has a custom __builtins__), in which case the class statement will
fail.
Nevertheless, unless someone finds a better solution, this is a 2.2.1
bugfix too.
eval_frame(): Under -Qnew, INPLACE_DIVIDE wasn't getting handed off to
INPLACE_TRUE_DIVIDE (like BINARY_DIVIDE was getting handed off to
BINARY_TRUE_DIVIDE).
Bugfix candidate.
(ditto for PyMem_Free() -> PyMem_FREE()) to fix and close SF bug
#495875 on systems that HAVE_SNPRINTF=0.
Check in on both release-22 branch and trunk.
Based on the patch from Danny Yoo. The fix is in exec_statement() in
ceval.c.
There are also changes to introduce use of PyCode_GetNumFree() in
several places.
was obviously leaking an int object when whatever the heck it's looking for
was found. Repaired that. This accounts for why entering function and
class definitions at an interactive prompt leaked a reference to the
integer 1 each time.
Bugfix candidate.
still fail on importing modules that link with libraries that fail
their initialization code (such as windowing libraries when we don't have
access to the window server) and that is what I really wanted to fix.
Had nothing to do with rich comparisons -- some stack cleanup code was
lost as a result of merging in Neil Schemenauer's generators patch.
Reinserted the stack cleanup code, skipping it when yielding.
leak when a class defined a __metaclass__. This fixes the problem
reported on python-dev by Ping; I dunno if it's the same as SF bug
#489669 (since that mentions Unicode).
Big Hammer to implement -Qnew as PEP 238 says it should work (a global
option affecting all instances of "/").
pydebug.h, main.c, pythonrun.c: define a private _Py_QnewFlag flag, true
iff -Qnew is passed on the command line. This should go away (as the
comments say) when true division becomes The Rule. This is
deliberately not exposed to runtime inspection or modification: it's
a one-way one-shot switch to pretend you're using Python 3.
ceval.c: when _Py_QnewFlag is set, treat BINARY_DIVIDE as
BINARY_TRUE_DIVIDE.
test_{descr, generators, zipfile}.py: fiddle so these pass under
-Qnew too. This was just a matter of s!/!//! in test_generators and
test_zipfile. test_descr was trickier, as testbinop() is passed
assumptions that "/" is the same as calling a "__div__" method; put
a temporary hack there to call "__truediv__" instead when the method
name is "__div__" and 1/2 evaluates to 0.5.
Three standard tests still fail under -Qnew (on Windows; somebody
please try the Linux tests with -Qnew too! Linux runs a whole bunch
of tests Windows doesn't):
test_augassign
test_class
test_coercion
I can't stay awake longer to stare at this (be my guest). Offhand
cures weren't obvious, nor was it even obvious that cures are possible
without major hackery.
Question: when -Qnew is in effect, should calls to __div__ magically
change into calls to __truediv__? See "major hackery" at tail end of
last paragraph <wink>.
There's now a new structmember code, T_OBJECT_EX, which is used for
all __slot__ variables (except __weakref__, which has special behavior
anyway). This new code raises AttributeError when the variable is
NULL rather than converting NULL to None.
The error for assignment to __debug__ used ste->ste_opt_lineno instead
of n->n_lineno. The latter was at best incorrect; often the slot was
uninitialized. Two fixes here: Use the correct lineno for the error.
Initialize ste_opt_lineno in PySymtable_New(); while there are no
current cases where it is referenced unless it has already been
assigned to, there is no harm in initializing it.
use wrappers on all platforms, to make this as consistent as possible x-
platform (in particular, make sure there's at least one \0 byte in
the output buffer). Also document more of the truth about what these do.
getargs.c, seterror(): Three computations of remaining buffer size were
backwards, thus telling PyOS_snprintf the buffer is larger than it
actually is. This matters a lot now that PyOS_snprintf ensures there's a
trailing \0 byte (because it didn't get the truth about the buffer size,
it was storing \0 beyond the true end of the buffer).
sysmodule.c, mywrite(): Simplify, now that PyOS_vsnprintf guarantees to
produce a \0 byte.
than the argument string size, copy as many bytes as will fit
(including a terminating '\0'), rather than not copying anything.
This to make it satisfy the C99 spec.
vgetargskeywords(): Now that this routine is checking for bad input
(rather than dump core in some cases), some bad calls are raising errors
that previously "worked". This patch makes the error strings more
revealing, and changes the exceptions from SystemError to RuntimeError
(under the theory that SystemError is more of a "can't happen!" assert-
like thing, and so inappropriate for bad arguments to a public C API
function).
seterror() uses a char array and a pointer to the current position in
that array. Use snprintf() and compute the amount of space left in
the buffer based on the current pointer position.
If it returns -1 (which indicates overflow on old Linux platforms and
perhaps on Windows) or size greater than buffer, write a message
indicating that the previous message was truncated.
com_arglist(), symtable_check_unoptimized(), symtable_params(),
symtable_global(), symtable_list_comprehension():
Conversion of sprintf() to PyOS_snprintf() for buffer overrun
avoidance.
PyEval_EvalCodeEx(): increment tstate->recursion_depth around the
decref of the frame, because the C stack for this call is still in
use and the decref can lead to __del__ methods getting called.
While this gives tstate->recursion_depth a value proportional to the
depth of the C stack (instead of a small constant no matter how
deeply __del__s recurse), it's not enough to stop the reported crash
when using the default recursion limit on Windows.
Bugfix candidate.
Bugfix candidate.
tb_displayline(): the sprintf format was choking off the file name, but
used plain %s for the function name (which can be arbitrarily long).
Limit both to 500 chars max.
uninitialized memory reads reported in bug #478001.
Note that this doesn't address the following larger issues:
- Error conditions are not documented for PyOS_*sig() in the C API.
- Nothing that actually calls PyOS_*sig() in the core interpreter and
extension modules actually /checks/ the return value of the call.
Fixing those is left as an exercise for a later day.
This patch boosts performance for comparing identical string object
by some 20% on my machine while not causing any noticable slow-down
for other operations (according to tests done with pybench).
routines. As of 10.1 using Carbon will crash Python if no window server is
available (ssh connection, console mode, MacOSX Server). This fixes bug
#466907.
A result of this mod is that the default 8bit encoding on OSX is now ASCII,
for the time being. Also, the extension modules that need the Carbon
framework now explicitly include it in setup.py.
+ Squash another potential buffer overrun.
+ Simplify the keyword-arg loop by decrementing the count of keywords
remaining instead of incrementing Yet Another Variable; also break
out early if the number of keyword args remaining hits 0.
Since I hit the function's closing curly brace with this patch, that's
enough of this for now <wink>.
The "need" for this was probably removed by an earlier patch that stopped
the loop right before it from passing NULL to a dict lookup routine.
I still haven't convinced myself that the next loop is correct, so am
leaving the next mysterious PyErr_Clear() call in for now.
+ Generally test nkeywords against 0 instead of keywords against NULL
(saves a little work if an empty keywords dict is passed, and is
conceptually more on-target regardless).
+ When a call erroneously specifies a keyword argument both by position
and by keyword name:
- It was easy to provoke this routine into an internal buffer overrun
by using a long argument name. Now uses PyErr_format instead (which
computes a safe buffer size).
- Improved the error msg.
+ Got rid of now-redundant dict typecheck.
+ Renamed nkwds to nkwlist. Now all the "counting" vrbls have names
related to the things they're counting in an obvious way.
+ Renamed argslen to nargs.
+ Renamed kwlen to nkeywords. This one was especially confusing because
kwlen wasn't the length of the kwlist argument, but of the keywords
argument.
+ Removed now-redundant tuple typecheck.
+ Renamed "tplen" local to "argslen" (it's the length of the "args"
argument; I suppose "tp" was for "Tim Peters should rename me
someday <wink>).
introduced this bug just a little while ago, when *adding* internal error
checks).
vgetargskeywords: Rewrote the section that crawls over the format string.
+ Added block comment so it won't take the next person 15 minutes to
reverse-engineer what it's doing.
+ Lined up the "else" clauses.
+ Rearranged the ifs in decreasing order of likelihood (for speed).
and raise an error if they're insane.
vgetargskeywords: the same, except that since this is an internal routine,
just assert that the arguments are sane.
the kwlist vector whenever there was a mix of positional and keyword
arguments, and the number of positional arguments exceeded the length
of the kwlist vector. If there was just one more positional arg than
keyword, the kwlist-terminating NULL got passed to PyMapping_HasKeyString,
which set an internal error that vgetargskeywords() then squashed (but
it's impossible to say whether it knew it was masking an error). If
more than one more positional argument, it went on to pass random trash
to PyMapping_HasKeyString, which is why the example at the start
happened to kill the process.
Pure bugfix candidate.
to call the corresponding methods. This is not a performance improvement
since the times are still swamped by disk I/O, but cleans up the code just
a little.
In Include/, marshal.h declares both
PyMarshal_ReadLongFromFile()
and PyMarshal_ReadShortFromFile(),
but the second is missing from marshal.c.
[Shouldn't the return type be declared as 'short' instead of 'int'?
But 'int' is what was in marshal.h all those years... --Guido]
This fixes the behavior reported by SF bug #404545, where a file
x.y.py could be imported by the statement "import x.y" when there's a
frozen package x (I believe even if x.y also exists as a frozen
module).
:-).
Add a test that prevents the __hello__ bytecode from going stale
unnoticed again.
The test also tests the loophole noted in SF bug #404545. This test
will fail right now; I'll check in the fix in a minute.
The symbol table pass didn't have an explicit case for the list_iter
node which is used only for a nested list comprehension. As a result,
the target of the list comprehension was treated as a use instead of
an assignment. Fix is to add a case to symtable_node() to handle
list_iter.
Also, rework and document a couple of the subtler implementation
issues in the symbol table pass. The symtable_node() switch statement
depends on falling through the last several cases, in order to handle
some of the more complicated nodes like atom. Add a comment
explaining the behavior before the first fall through case. Add a
comment /* fall through */ at the end of case so that it is explicitly
marked as such.
Move the for_stmt case out of the fall through logic, which simplifies
both for_stmt and default. (The default used the local variable start
to skip the first three nodes of a for_stmt when it fell through.)
Rename the flag argument to symtable_assign() to def_flag and add a
comment explaining its use:
The third argument to symatble_assign() is a flag to be passed to
symtable_add_def() if it is eventually called. The flag is useful
to specify the particular type of assignment that should be
recorded, e.g. an assignment caused by import.
Also minor tweaks to internal routines.
Use PyCF_MASK instead of explicit list of flags.
For the MAKE_CLOSURE opcode, the number of items popped off the stack
depends on both the oparg and the number of free variables for the
code object. Fix the code so it accounts for the free variables.
In com_classdef(), record an extra pop to account for the STORE call
after the BUILD_CLASS.
Get rid of some commented out debugging code in com_push() and
com_pop().
Factor string resize logic into helper routine com_check_size().
In com_addbyte(), remove redudant if statement after assert. (They
test the same condition.)
In several routines, use string macros instead of string functions.
This changes Pythread_start_thread() to return the thread ID, or -1
for an error. (It's technically an incompatible API change, but I
doubt anyone calls it.)
When an extension imports another extension in its
initXXX() function, the variable _Py_PackageContext is
prematurely reset to NULL. If the outer extension then
calls Py_InitModule(), the extension is installed in
sys.modules without its package name. The
manifestation of this bug is a "SystemError:
_PyImport_FixupExtension: module <package>.<extension>
not loaded".
To fix this, importdl.c just needs to retain the old
value of _Py_PackageContext and restore it after the
initXXX() method is called. The attached patch does this.
This patch applies to Python 2.1.1 and the current CVS.
"for <var> in <testlist> may no longer be a single test followed by
a comma. This solves SF bug #431886. Note that if the testlist
contains more than one test, a trailing comma is still allowed, for
maximum backward compatibility; but this example is not:
[(x, y) for x in range(10), for y in range(10)]
^
The fix involved creating a new nonterminal 'testlist_safe' whose
definition doesn't allow the trailing comma if there's only one test:
testlist_safe: test [(',' test)+ [',']]
This patch changes to logic to:
if env.var. set and non-empty:
if env.var. is an integer:
set flag to that integer
if flag is zero: # [actually, <= 0 --GvR]
set flag to 1
Under this patch, anyone currently using
PYTHONVERBOSE=yes will get the same output as before.
PYTHONVERBNOSE=2 will generate more verbosity than
before.
The only unusual case that the following three are
still all equivalent:
PYTHONVERBOSE=yespleas
PYTHONVERBOSE=1
PYTHONVERBOSE=0
This patch updates Python/thread_pthread.h to mask all
signals for any thread created. This will keep all
signals masked for any thread that isn't the initial
thread. For Solaris and Linux, the two platforms I was
able to test it on, it solves bug #465673 (pthreads
need signal protection) and probably will solve bug
#219772 (Interactive InterPreter+ Thread -> core dump
at exit).
I'd be great if this could get some testing on other
platforms, especially HP-UX pre 11.00 and post 11.00,
as I had to make some guesses for the DCE thread case.
AIX is also a concern as I saw some mention of using
sigthreadmask() as a pthread_sigmask() equivalent, but
this patch doesn't use sigthreadmask(). I don't have
access to AIX.
The new profiler event stream includes a "return" event even when an
exception is being propogated, but the machinery that called the profile
hook did not save & restore the exception. In debug mode, the exception
was detected during the execution of the profile callback, which did not
have the proper internal flags set for the exception. Saving & restoring
the exception state solves the problem.
The profiler does not need to know anything about the exception state,
so we no longer call it when an exception is raised. We do, however,
make sure we *always* call the profiler when we exit a frame. This
ensures that timing events are more easily isolated by a profiler and
finally clauses that do a lot of work don't have their time
mis-allocated.
When an exception is propogated out of the frame, the C callback for
the profiler now receives a PyTrace_RETURN event with an arg of NULL;
the Python-level profile hook function will see a 'return' event with
an arg of None. This means that from Python it is impossible for the
profiler to determine if the frame exited with an exception or if it
returned None, but this doesn't matter for profiling. A C-based
profiler could tell the difference, but this doesn't seem important.
ceval.c:eval_frame(): Simplify the code in two places so that the
profiler is called for every exit from a frame
and not for exceptions.
sysmodule.c:profile_trampoline(): Make sure we don't expose Python
code to NULL; use None instead.
Unknown whether this fixes it.
- stringobject.c, PyString_FromFormatV: don't assume that va_list is of
a type that can be copied via an initializer.
- errors.c, PyErr_Format: add a va_end() to balance the va_start().
If a new exception occurs while an exception instance is being
created, try harder to make sure there is a traceback. If the
original exception had a traceback associated with it and the new
exception does not, keep the old exception.
Of course, callers to PyErr_NormalizeException() must still be
prepared to have tb set to NULL.
XXX This isn't an ideal solution, but it's better than no traceback at
all. It occurs if, for example, the exception occurs when the call to
the constructor fails before any Python code is executed. Guido
suggests that it there is Python code that was about to be executed
-- but wasn't, say, because it was called with the wrong number of
arguments -- then we should point at the first line of the code object
anyway.
It's possible for PyErr_NormalizeException() to set the traceback
pointer to NULL. I'm not sure how to provoke this directly from
Python, although it may be possible. The error occurs when an
exception is set using PyErr_SetObject() and another exception occurs
while PyErr_NormalizeException() is creating the exception instance.
XXX As a result of this change, it's possible for an exception to
occur but sys.last_traceback to be left undefined. Not sure if this
is a problem.
popped frame-block. What an embarrassing bug! Especially for Jeremy, since
he accepted the patch :-)
This fixes SF bugs #463359 and #462937, and possibly other, *very* obscure
bugs with very deeply nested loops that continue the loop and then break out
of it or raise an exception.
compatibility, this required all places where an array of "struct
memberlist" structures was declared that is referenced from a type's
tp_members slot to change the type of the structure to PyMemberDef;
"struct memberlist" is now only used by old code that still calls
PyMember_Get/Set. The code in PyObject_GenericGetAttr/SetAttr now
calls the new APIs PyMember_GetOne/SetOne, which take a PyMemberDef
argument.
As examples, I added actual docstrings to the attributes of a few
types: file, complex, instance method, super, and xxsubtype.spamlist.
Also converted the symtable to new style getattr.
Renamed the 'readonly' field to 'flags' and defined some new flag
bits: READ_RESTRICTED and WRITE_RESTRICTED, as well as a shortcut
RESTRICTED that means both.
backwards compatibility. When using the class of the first base as
the metaclass, use its __class__ attribute in preference over its
ob_type slot. This ensures that we can still use classic classes as
metaclasse, as shown in the original "Metaclasses" essay. This also
makes all the examples in Demo/metaclasses/ work again (maybe these
should be turned into a test suite?).
parameter for the return string (as unix pathnames are not limited
by the 255 char pstring limit).
Implemented the function for MachO-Python, where it returns unix pathnames.
by bbrox@bbrox.org / lionel.ulmer@free.fr.
This adds a configure check and if all goes well turns on the
PTHREAD_SCOPE_SYSTEM thread attribute for new threads.
This should remove the need to add tiny sleeps at the start of threads
to allow other threads to be scheduled.
Reported by Fredrik Lundh on python-dev.
The conversimple() code that handles Unicode arguments and converts
them to the default encoding now calls converterr() with the original
Unicode argument instead of the NULL returned by the failed encoding
attempt.
com_factor(): when a unary minus is attached to a float or imaginary zero,
don't optimize the UNARY_MINUS opcode away: the const dict can't
distinguish between +0.0 and -0.0, so ended up treating both like the
first one added to it. Optimizing UNARY_PLUS away isn't a problem.
(BTW, I already uploaded the 2.2a3 Windows installer, and this isn't
important enough to delay the release.)
of PyMapping_Keys because we know we have a real dict. Tolerate that
objects may have an attr named "__dict__" that's not a dict (Py_None
popped up during testing).
test_descr.py, test_dir(): Test the new classic-class behavior; beef up
the new-style class test similarly.
test_pyclbr.py, checkModule(): dir(C) is no longer a synonym for
C.__dict__.keys() when C is a classic class (looks like the same thing
that burned distutils! -- should it be *made* a synoym again? Then it
would be inconsistent with new-style class behavior.).
bag. It's clearly wrong for classic classes, at heart because a classic
class doesn't have a __class__ attribute, and I'm unclear on whether
that's feature or bug. I'll repair this once I find out (in the
meantime, dir() applied to classic classes won't find the base classes,
while dir() applied to a classic-class instance *will* find the base
classes but not *their* base classes).
Please give the new dir() a try and see whether you love it or hate it.
The new dir([]) behavior is something I could come to love. Here's
something to hate:
>>> class C:
... pass
...
>>> c = C()
>>> dir(c)
['__doc__', '__module__']
>>>
The idea that an instance has a __doc__ attribute is jarring (of course
it's really c.__class__.__doc__ == C.__doc__; likewise for __module__).
OTOH, the code already has too many special cases, and dir(x) doesn't
have a compelling or clear purpose when x isn't a module.
PEP 238. Changes:
- add a new flag variable Py_DivisionWarningFlag, declared in
pydebug.h, defined in object.c, set in main.c, and used in
{int,long,float,complex}object.c. When this flag is set, the
classic division operator issues a DeprecationWarning message.
- add a new API PyRun_SimpleStringFlags() to match
PyRun_SimpleString(). The main() function calls this so that
commands run with -c can also benefit from -Dnew.
- While I was at it, I changed the usage message in main() somewhat:
alphabetized the options, split it in *four* parts to fit in under
512 bytes (not that I still believe this is necessary -- doc strings
elsewhere are much longer), and perhaps most visibly, don't display
the full list of options on each command line error. Instead, the
full list is only displayed when -h is used, and otherwise a brief
reminder of -h is displayed. When -h is used, write to stdout so
that you can do `python -h | more'.
Notes:
- I don't want to use the -W option to control whether the classic
division warning is issued or not, because the machinery to decide
whether to display the warning or not is very expensive (it involves
calling into the warnings.py module). You can use -Werror to turn
the warnings into exceptions though.
- The -Dnew option doesn't select future division for all of the
program -- only for the __main__ module. I don't know if I'll ever
change this -- it would require changes to the .pyc file magic
number to do it right, and a more global notion of compiler flags.
- You can usefully combine -Dwarn and -Dnew: this gives the __main__
module new division, and warns about classic division everywhere
else.
pyport.h: typedef a new Py_intptr_t type.
DELICATE ASSUMPTION: That HAVE_UINTPTR_T implies intptr_t is
available as well as uintptr_t. If that turns out not to be
true, things must get uglier (C99 wants both, so I think it's
an assumption we're *likely* to get away with).
thread_nt.h, PyThread_start_new_thread: MS _beginthread is documented
as returning unsigned long; no idea why uintptr_t was being used.
Others: Always use Py_[u]intptr_t, never [u]intptr_t directly.
Check return value from future_parse() in for loop for file_input to
accomodate multiple future statements on separate lines.
Add several comments explaining how the code works.
Remove out-dated XXX comment.
Change to get/set/del slice operations so that if the object doesn't
support slicing, *or* if either of the slice arguments is not an int
or long, we construct a slice object and call the get/set/del item
operation instead. This makes it possible to design classes that
support slice arguments of non-integral types.
builtin_eval wasn't merging in the compiler flags from the current frame;
I suppose we never noticed this before because future division is the
first future-feature that can affect expressions (nested_scopes and
generators had only statement-level effects).
CO_FUTURE_DIVISION flag. Redid this to use Jeremy's PyCF_MASK #define
instead, so we dont have to remember to fiddle individual feature names
here again.
pythonrun.h: Also #define a PyCF_MASK_OBSOLETE mask. This isn't used
yet, but will be as part of the PEP 264 implementation (compile() mustn't
raise an error just because old code uses a flag name that's become
obsolete; a warning may be appropriate, but not an error; so compile() has
to know about obsolete flags too, but nobody is going to remember to
update compile() with individual obsolete flag names across releases either
-- i.e., this is the flip side of PyEval_MergeCompilerFlags's oversight).
- Do not compile unicodeobject, unicodectype, and unicodedata if Unicode is disabled
- check for Py_USING_UNICODE in all places that use Unicode functions
- disables unicode literals, and the builtin functions
- add the types.StringTypes list
- remove Unicode literals from most tests.
When code is compiled and compiler flags are passed in, be sure to
update cf_flags with any features defined by future statements in the
compiled code.
_PyImport_FixupExtension() on the exceptions module. Now
reload(exceptions) acts just like reload(sys) instead of raising
an ImportError.
This closes SF bug #422004.
The descr changes moved the dispatch for calling objects from
call_object() in ceval.c to PyObject_Call() in abstract.c.
call_object() and the many functions it used in ceval.c were no longer
used, but were not removed.
Rename meth_call() as PyCFunction_Call() so that it can be called by
the CALL_FUNCTION opcode in ceval.c.
Also, fix error message that referred to PyEval_EvalCodeEx() by its
old name eval_code2(). (I'll probably refer to it by its old name,
too.)
Revised version of Fred's patch, including support for ~ operator.
If the unary +, -, or ~ operator is applied to a constant, don't
generate a UNARY_xxx opcode. Just store the approriate value as a
constant. If the value is negative, extend the string containing the
constant and insert a negative in the 0th position.
For ~, compute the inverse of int and longs and use them directly, but
be prepared to generate code for all other possibilities (invalid
numbers, floats, complex).
same module twice, which apparently crashes Python. I could not test the
error condition, but in normal life it seems to have no adverse effects.
Also removed an unsued variable, and corrected 2 glaring errors (missing
'case' in front of a label).
Replace uses of PyCF_xxx with CO_xxx.
Replace individual feature slots in PyFutureFeatures with single
bitmask ff_features.
When flags must be transfered among the three parts of the interpreter
that care about them -- the pythonrun layer, the compiler, and the
future feature parser -- can simply or (|) the definitions.
with functionality needed for both unix-Python and MacPython and a
new smaller ./Mac/Python/macglue.c which contains MacPython stuff only.
pymactoolbox.h has moved to ./Include from ./Mac/Include and now also
contains the relevant stuff from macglue.h.
The net effect of this is that the ./Mac subdirectory is not needed
anymore for building the unix-Python core on MacOSX (it is needed
for building the extension modules).
This introduces:
- A new operator // that means floor division (the kind of division
where 1/2 is 0).
- The "future division" statement ("from __future__ import division)
which changes the meaning of the / operator to implement "true
division" (where 1/2 is 0.5).
- New overloadable operators __truediv__ and __floordiv__.
- New slots in the PyNumberMethods struct for true and floor division,
new abstract APIs for them, new opcodes, and so on.
I emphasize that without the future division statement, the semantics
of / will remain unchanged until Python 3.0.
Not yet implemented are warnings (default off) when / is used with int
or long arguments.
This has been on display since 7/31 as SF patch #443474.
Flames to /dev/null.
- Add an explicit call to PyType_Ready(&PyList_Type) to pythonrun.c
(just for the heck of it, really -- we should either explicitly
ready all types, or none).
Python warning which can be catched by means of the Python warning
framework.
It also adds two new APIs which hopefully make it easier for Python
to switch to buffer overflow safe [v]snprintf() APIs for error
reporting et al. The two new APIs are PyOS_snprintf() and
PyOS_vsnprintf() and work just like the standard ones in many
C libs. On platforms which have snprintf(), the native APIs are used,
on all other an emulation with snprintf() tries to do its best.
Fix suggested by Michael Hudson: Raise TypeError if attribute name
passed to getattr() is not a string or Unicode. There is some
unfortunate duplication of code between builtin_getattr() and
PyObject_GetAttr(), but it appears to be unavoidable.
And remove all the extern decls in the middle of .c files.
Apparently, it was excluded from the header file because it is
intended for internal use by the interpreter. It's still intended for
internal use and documented as such in the header file.
exception in the execution of bar, ensure that foo.bar exists.
(Previously, while sys.modules['foo.bar'] would exist, foo.bar would
only be created upon successful execution of bar. This is
inconvenient; some would say wrong. :-)
that 'yield' is a keyword. This doesn't help test_generators at all! I
don't know why not. These things do work now (and didn't before this
patch):
1. "from __future__ import generators" now works in a native shell.
2. Similarly "python -i xxx.py" now has generators enabled in the
shell if xxx.py had them enabled.
3. This program (which was my doctest proxy) works fine:
from __future__ import generators
source = """\
def f():
yield 1
"""
exec compile(source, "", "single") in globals()
print type(f())
that info to code dynamically compiled *by* code compiled with generators
enabled. Doesn't yet work because there's still no way to tell the parser
that "yield" is OK (unlike nested_scopes, the parser has its fingers in
this too).
Replaced PyEval_GetNestedScopes by a more-general
PyEval_MergeCompilerFlags. Perhaps I should not have? I doubted it was
*intended* to be part of the public API, so just did.
the yield statement. I figure we have to have this in before I can
release 2.2a1 on Wednesday.
Note: test_generators is currently broken, I'm counting on Tim to fix
this.
Probable fix (the bug report doesn't have enough info to say for sure).
find_init_module(): Insist on a case-sensitive match for __init__ files.
Given __INIT__.PY instead, find_init_module() thought that was fine, but
the later attempt to do find_module("__INIT__.PY") didn't and its caller
silently suppressed the resulting ImportError. Now find_init_module()
refuses to accept __INIT__.PY to begin with.
Bugfix candidate; specific to platforms with case-insensitive filesystems.
path (with no profile/trace function) through eval_code2() and
eval_frame() avoids several checks.
In the common cases of calls, returns, and exception propogation,
eval_code2() and eval_frame() used to test two values in the
thread-state: the profiling function and the tracing function. With
this change, a flag is set in the thread-state if either of these is
active, allowing a single check to suffice when both are NULL. This
also simplifies the code needed when either function is in use but is
already active (to avoid profiling/tracing the profiler/tracer); the
flag is set to 0 when the profile/trace code is entered, allowing the
same check to suffice for "already in the tracer" for call/return/
exception events.
"return expr" instances in generators (which latter may be generators
due to otherwise invisible "yield" stmts hiding in "if 0" blocks).
This was fun the first time, but this has gotten truly ugly now.
Python interpreter.
This change adds two new C-level APIs: PyEval_SetProfile() and
PyEval_SetTrace(). These can be used to install profile and trace
functions implemented in C, which can operate at much higher speeds
than Python-based functions. The overhead for calling a C-based
profile function is a very small fraction of a percent of the overhead
involved in calling a Python-based function.
The machinery required to call a Python-based profile or trace
function been moved to sysmodule.c, where sys.setprofile() and
sys.setprofile() simply become users of the new interface.
As a side effect, SF bug #436058 is fixed; there is no longer a
_PyTrace_Init() function to declare.
Implement sys.maxunicode.
Explicitly wrap around upper/lower computations for wide Py_UNICODE.
When decoding large characters with UTF-8, represent expected test
results using the \U notation.
- the correct range for the error message is range(0x110000);
- put the 4-byte Unicode-size code inside the same else branch as the
2-byte code, rather generating unreachable code in the 2-byte case.
- Don't hide the 'else' behine the '}'.
(I would prefer that in 4-byte mode, any value should be accepted, but
reasonable people can argue about that, so I'll put that off.)
Add configure option --enable-unicode.
Add config.h macros Py_USING_UNICODE, PY_UNICODE_TYPE, Py_UNICODE_SIZE,
SIZEOF_WCHAR_T.
Define Py_UCS2.
Encode and decode large UTF-8 characters into single Py_UNICODE values
for wide Unicode types; likewise for UTF-16.
Remove test whether sizeof Py_UNICODE is two.
Iterators list and Python-Dev; e.g., these all pass now:
def g1():
try:
return
except:
yield 1
assert list(g1()) == []
def g2():
try:
return
finally:
yield 1
assert list(g2()) == [1]
def g3():
for i in range(3):
yield None
yield None
assert list(g3()) == [None] * 4
compile.c: compile_funcdef and com_return_stmt: Just van Rossum's patch
to compile the same code for "return" regardless of function type (this
goes back to the previous scheme of returning Py_None).
ceval.c: gen_iternext: take a return (but not a yield) of Py_None as
meaning the generator is exhausted.
the next free valuestack slot, not to the base (in America, stacks push
and pop at the top -- they mutate at the bottom in Australia <winK>).
eval_frame(): assert that f_stacktop isn't NULL upon entry.
frame_delloc(): avoid ordered pointer comparisons involving f_stacktop
when f_stacktop is NULL.
reference to f_back when its really needed. Do a little whitespace
normalization as well. This whole file is a big war between tabs and spaces
but now is probably not the time to reindent everything.
NeilS, please check! This came from staring at your genbug.py, but I'm
not sure it plugs all possible holes. Without this, I caught a
frameobject refcount going negative, and it was also the cause (in debug
build) of _Py_ForgetReference's attempt to forget an object with already-
NULL _ob_prev and _ob_next pointers -- although I'm still not entirely
sure how! Part of the difficulty is that frameobjects are stored on a
free list that gets recycled very quickly, so if there's a stray pointer
to one of them it never looks like an insane frameobject (never goes
trough the free() mangling MS debug forces, etc).
and trace functions lazily, which incurs extra argument pushing and checks
in the C overhead for profiling/tracing, create the strings semi-lazily
when the Python code first registers a profile or trace function. This
simplifies the trampoline into the profile/trace functions.
Armin Rigo pointed out that the way the line-# table got built didn't work
for lines generating more than 255 bytes of bytecode. Fixed as he
suggested, plus corresponding changes to pyassem.py, plus added some
long overdue docs about this subtle table to compile.c.
Bugfix candidate (line numbers may be off in tracebacks under -O).
that should be used to cache an interned version of the event
string passed to the profile/trace function. call_trace() will
create interned strings and cache them in using the storage
specified by this additional parameter, avoiding a lot of string
object creation at runtime when using the profiling or tracing
functions.
All call sites are modified to pass the additional parameter, and four
static PyObject* variables are allocated to cache the interned string
objects.
This closes SF patch #431257.
in release builds. Suggested by Martin v. Loewis.
I'm half tempted to macroize PyErr_Occurred too, as the whole thing could
collapse to just
_PyThreadState_Current->curexc_type
In the default branch, keep three ifs that are used if level == 0, the
most common case. Note that first if here is a slight optimization
for the 'O' format.
Second part of SF patch 426072.
Note that lots of code was re-indented.
Replace two-step of convertsimple() and convertsimple1() with
convertsimple() and helper converterr(), which is called to format
error messages when convertsimple() fails. The old code did all the
real work in convertsimple1(), but deferred error message formatting
to conversimple(). The result was paying the price of a second
function call on every call just to format error messages in the
failure cases.
Factor out of the buffer-handling code in convertsimple() and package
it as convertbuffer().
Add two macros to ease readability of Unicode coversions,
UNICODE_DEFAULT_ENCODING() and CONV_UNICODE, an error string.
The convertsimple() routine had awful indentation problems, primarily
because there were two tabs between the case line and the body of the
case statements. This patch reformats the entire function to have a
single tab between case line and case body, which makes the code
easier to read (and consistent with ceval). The introduction of
converterr() exacerbated the problem and prompted this fix.
Also, eliminate non-standard whitespace after opening paren and before
closing paren in a few if statements.
(This checkin is part of SF patch 426072.)
Keyword arguments passed to builtin functions that don't take them are
ignored.
>>> {}.clear(x=2)
>>>
instead of
>>> {}.clear(x=2)
Traceback (most recent call last):
File "<stdin>", line 1, in ?
TypeError: clear() takes no keyword arguments
If we have a PyCFunction (builtin) and it is METH_VARARGS only, load
the args and dispatch to call_cfunction() directly. This provides a
small speedup for perhaps the most common function calls -- builtins.
Store floats and doubles to full precision in marshal.
Test that floats read from .pyc/.pyo closely match those read from .py.
Declare PyFloat_AsString() in floatobject header file.
Add new PyFloat_AsReprString() API function.
Document the functions declared in floatobject.h.
Check for free in class and method only if nested scopes are enabled.
Add assertion to verify that no free variables occur when nested
scopes are disabled.
XXX When should nested scopes by made non-optional on the trunk?
NEEDS DOC CHANGES.
More AttributeErrors transmuted into TypeErrors, in test_b2.py, and,
again, this strikes me as a good thing.
This checkin completes the iterator generalization work that obviously
needed to be done. Can anyone think of others that should be changed?
NEEDS DOC CHANGES.
Possibly contentious: The first time s.next() yields StopIteration (for
a given map argument s) is the last time map() *tries* s.next(). That
is, if other sequence args are longer, s will never again contribute
anything but None values to the result, even if trying s.next() again
could yield another result. This is the same behavior map() used to have
wrt IndexError, so it's the only way to be wholly backward-compatible.
I'm not a fan of letting StopIteration mean "try again later" anyway.
Directory containing
Spam.py
spam/__init__.py
Then "import Spam" caused a SystemError, because code checking for
the existence of "Spam/__init__.py" finds it on a case-insensitive
filesystem, but then bails because the directory it finds it in
doesn't match case, and then old code assumed that was still an error
even though it isn't anymore. Changed the code to just continue
looking in this case (instead of calling it an error). So
import Spam
and
import spam
both work now.
Also a 2.1 bugfix candidate (am I supposed to do something with those?).
Took away map()'s insistence that sequences support __len__, and cleaned
up the convoluted code that made it *look* like it really cared about
__len__ (in fact the old ->len field was only *used* as a flag bit, as
the main loop only looked at its sign bit, setting the field to -1 when
IndexError got raised; renamed the field to ->saw_IndexError instead).
The new test case demonstrates the bug. Be more careful in
symtable_resolve_free() to add a var to cells or frees only if it
won't be added under some other rule.
XXX Add new assertion that will catch this bug.
sees it (test_iter.py is unchanged).
- Added a tp_iternext slot, which calls the iterator's next() method;
this is much faster for built-in iterators over built-in types
such as lists and dicts, speeding up pybench's ForLoop with about
25% compared to Python 2.1. (Now there's a good argument for
iterators. ;-)
- Renamed the built-in sequence iterator SeqIter, affecting the C API
functions for it. (This frees up the PyIter prefix for generic
iterator operations.)
- Added PyIter_Check(obj), which checks that obj's type has a
tp_iternext slot and that the proper feature flag is set.
- Added PyIter_Next(obj) which calls the tp_iternext slot. It has a
somewhat complex return condition due to the need for speed: when it
returns NULL, it may not have set an exception condition, meaning
the iterator is exhausted; when the exception StopIteration is set
(or a derived exception class), it means the same thing; any other
exception means some other error occurred.
new slot tp_iter in type object, plus new flag Py_TPFLAGS_HAVE_ITER
new C API PyObject_GetIter(), calls tp_iter
new builtin iter(), with two forms: iter(obj), and iter(function, sentinel)
new internal object types iterobject and calliterobject
new exception StopIteration
new opcodes for "for" loops, GET_ITER and FOR_ITER (also supported by dis.py)
new magic number for .pyc files
new special method for instances: __iter__() returns an iterator
iteration over dictionaries: "for x in dict" iterates over the keys
iteration over files: "for x in file" iterates over lines
TODO:
documentation
test suite
decide whether to use a different way to spell iter(function, sentinal)
decide whether "for key in dict" is a good idea
use iterators in map/filter/reduce, min/max, and elsewhere (in/not in?)
speed tuning (make next() a slot tp_next???)
now raises NameError instead of UnboundLocalError, because the var in
question is definitely not local. (This affects test_scope.py)
Also update the recent fix by Ping using get_func_name(). Replace
tests of get_func_name() return value with call to get_func_desc() to
match all the other uses.
Calling an unbound method on a C extension class without providing
an instance can yield a segfault. Try "Exception.__init__()" or
"ValueError.__init__()".
This is a simple fix. The error-reporting bits in call_method
mistakenly treat the misleadingly-named variable "func" as a
function, when in fact it is a method.
If we let get_func_name take care of the work, all is fine.
Fix based on patch #414750 by Michael Hudson.
New functions get_func_name() and get_func_desc() return reasonable
names and descriptions for all objects. XXX Even objects that aren't
actually callable.
pickle.py
The code implicitly assumed that all ints fit in 4 bytes, causing all
sorts of mischief (from nonsense results to corrupted pickles).
Repaired that.
marshal.c
The int marshaling code assumed that right shifts of signed longs
sign-extend. Repaired that.
Jeffery Collins pointed out that filterstring decrefs a character object
before it's done using it. This works by accident today because another
module always happens to have an active reference too at the time. The
accident doesn't work after his Pippy modifications, and since it *is*
an accident even in the mainline Python, it should work by design there too.
The patch accomplishes that.
but apparently he had to go to school, so I am checking it in for him.
This makes PyRun_HandleSystemExit() a static instead, called
handle_system_exit(), and let it use the current exception rather than
passing in an exception. This slightly simplifies the code.
Update docstring and library reference section on 'sys' module.
New API PyErr_Display, just for displaying errors, called by excepthook.
Uncaught exceptions now call sys.excepthook; if that fails, we fall back
to calling PyErr_Display directly.
Also comes with sys.__excepthook__ and sys.__displayhook__.
If a module has a future statement enabling nested scopes, they are
also enable for the exec statement and the functions compile() and
execfile() if they occur in the module.
If Python is run with the -i option, which enters interactive mode
after executing a script, and the script it runs enables nested
scopes, they are also enabled in interactive mode.
XXX The use of -i with -c "from __future__ import nested_scopes" is
not supported. What's the point?
To support these changes, many function variants have been added to
pythonrun.c. All the variants names end with Flags and they take an
extra PyCompilerFlags * argument. It is possible that this complexity
will be eliminated in a future version of the interpreter in which
nested scopes are not optional.
frees. Note there doesn't seem to be any way to test LocalsToFast(),
because the instructions that trigger it are illegal in nested scopes
with free variables.
Fix allocation strategy for cells that are also formal parameters.
Instead of emitting LOAD_FAST / STORE_DEREF pairs for each parameter,
have the argument handling code in eval_code2() do the right thing.
A side-effect of this change is that cell variables that are also
arguments are listed at the front of co_cellvars in the order they
appear in the argument list.
has a binding for the name. The fix is in two places:
- in symtable_update_free_vars, ignore a global stmt in a class scope
- in symtable_load_symbols, add extra handling for names that are
defined at class scope and free in a method
Closes SF bug 407800
with free variables. Thanks to Martin v. Loewis for finding two of
the problems. This fixes SF buf 405583.
There is also a C API change: PyFrame_New() is reverting to its
pre-2.1 signature. The change introduced by nested scopes was a
mistake. XXX Is this okay between beta releases?
cell_clear(), the GC helper, must decref its reference to break
cycles.
frame_dealloc() must dealloc all cell vars and free vars in addition
to locals.
eval_code2() setup code must INCREF cells it copies out of the
closure.
The STORE_DEREF opcode implementation must DECREF the object it passes
to PyCell_Set().
Made sure that the warnings issued by symtable_check_unoptimized()
(about import * and exec) contain the proper filename and line number,
and are transformed into SyntaxError exceptions with -Werror.
(Also remove warning about module-level global decl, because we can't
distinguish from code passed to exec.)
Define PyCompilerFlags type contains a single element,
cf_nested_scopes, that is true if a nested scopes future statement has
been entered at the interactive prompt.
New API functions:
PyNode_CompileFlags()
PyRun_InteractiveOneFlags()
-- same as their non Flags counterparts except that the take an
optional PyCompilerFlags pointer
compile.c: In jcompile() use PyCompilerFlags argument. If
cf_nested_scopes is true, compile code with nested scopes. If it
is false, but the code has a valid future nested scopes statement,
set it to true.
pythonrun.c: Create a new PyCompilerFlags object in
PyRun_InteractiveLoop() and thread it through to
PyRun_InteractiveOneFlags().
the more recent versions of that platform, so we use the value (time_t)(-1)
as the error value. This is the type used in the OpenVMS documentation:
http://www.openvms.compaq.com/commercial/c/5763p048.htm#inde
This closes SF tracker bug #404240.
Also clean up an exception message when detecting overflow of time_t values
beyond 4 bytes.
from __future__ import nested_scopes
x=7
def f():
x=1
def g():
global x
def i():
def h():
return x
return h()
return i()
return g()
print f()
print x
This kind of code didn't work correctly because x was treated as free
in i, leading to an attempt to load x in g to make a closure for i.
Solution is to make global decl apply to nested scopes unless their is
an assignment. Thus, x in h is global.
described in PEP 227.
symtable_check_unoptimized() warns about import * and exec with "in"
when it is used in a function that contains a nested function with
free variables. Warnings are issued unless nested scopes are in
effect, in which case these are SyntaxErrors.
symtable_check_shadow() warns about assignments in a function scope
that shadow free variables defined in a nested scope. This will
always generate a warning -- and will behave differently with nested
scopes than without.
Restore full checking for free vars in children, even when nested
scopes are not enabled. This is needed to support warnings for
shadowing.
Change symtable_warn() to return an int-- the return value of
PyErr_WarnExplicit.
Sundry cleanup: Remove commented out code. Break long lines.
global after assign / use.
Note: I'm not updating the PyErr_Warn() call for import * / exec
combined with a function, because I can't trigger it with an example.
Jeremy, just follow the example of the call to PyErr_WarnExplicit()
that I *did* include.
for errors raised in future.c.
Move some helper functions from compile.c to errors.c and make them
API functions: PyErr_SyntaxLocation() and PyErr_ProgramText().
raised by the compiler.
XXX For now, text entered into the interactive intepreter is not
printed in the traceback.
Inspired by a patch from Roman Sulzhyk
compile.c:
Add helper fetch_program_text() that opens a file and reads until it
finds the specified line number. The code is a near duplicate of
similar code in traceback.c.
Modify com_error() to pass two arguments to SyntaxError constructor,
where the second argument contains the offending text when possible.
Modify set_error_location(), now used only by the symtable pass, to
set the text attribute on existing exceptions.
pythonrun.c:
Change parse_syntax_error() to continue of the offset attribute of a
SyntaxError is None. In this case, it sets offset to -1.
Move code from PyErr_PrintEx() into helper function
print_error_text(). In the helper, only print the caret for a
SyntaxError if offset > 0.
http://python.sourceforge.net/peps/pep-0235.html
Renamed check_case to case_ok. Substantial code rearrangement to get
this stuff in one place in the file. Innermost loop of find_module()
now much simpler and #ifdef-free, and I want to keep it that way (it's
bad enough that the innermost loop is itself still in an #ifdef!).
Windows semantics tested and are fine.
Jason, Cygwin *should* be fine if and only if what you did before "worked"
for case_ok.
Jack, the semantics on your flavor of Mac have definitely changed (see
the PEP), and need to be tested. The intent is that your flavor of Mac
now work the same as everything else in the "lower left" box, including
respecting PYTHONCASEOK.
Steven, sorry, you did the most work here so far but you got screwed the
worst. Happy to work with you on repairing it, but I don't understand
anything about all your Mac variants. We need to add another branch (or
two, three, ...?) inside case_ok. But we should not need to change
anything else.
XXX still need to integrate into symtable API
compile.h: Remove ff_n_simple_stmt; obsolete.
Add ff_found_docstring used internally to skip one and only
one string at the beginning of a module.
compile.c: Add check for from __future__ imports to far into the file.
In symtable_global() check for -1 returned from
symtable_lookup(), which signifies name not defined.
Add missing DECERF in symtable_add_def.
Free c->c_future.
future.c: Add special handling for multiple statements joined on a
single line using one or more semicolons; this form can
include an illegal future statement that would otherwise be
hard to detect.
Add support for detecting and skipping doc strings.
Makefile.pre.in: add target future.o
Include/compile.h: define PyFutureFeaters and PyNode_Future()
add c_future slot to struct compiling
Include/symtable.h: add st_future slot to struct symtable
Python/future.c: implementation of PyNode_Future()
Python/compile.c: use PyNode_Future() for nested_scopes support
Python/symtable.c: include compile.h to pick up PyFutureFeatures decl
compile.h: #define NESTED_SCOPES_DEFAULT 0 for Python 2.1
__future__ feature name: "nested_scopes"
symtable.h: Add st_nested_scopes slot. Define flags to track exec and
import star.
Lib/test/test_scope.py: requires nested scopes
compile.c: Fiddle with error messages.
Reverse the sense of ste_optimized flag on
PySymtableEntryObjects. If it is true, there is an optimization
conflict.
Modify get_ref_type to respect st_nested_scopes flags.
Refactor symtable_load_symbols() into several smaller functions,
which use struct symbol_info to share variables. In new function
symtable_update_flags(), raise an error or warning for import * or
bare exec that conflicts with nested scopes. Also, modify handle
for free variables to respect st_nested_scopes flag.
In symtable_init() assign st_nested_scopes flag to
NESTED_SCOPES_DEFAULT (defined in compile.h).
Add preliminary and often incorrect implementation of
symtable_check_future().
Add symtable_lookup() helper for future use.
Two different but related problems:
1. PySymtable_Free() must explicitly DECREF(st->st_cur), which should
always point to the global symtable entry. This entry is setup by the
first enter_scope() call, but there is never a corresponding
exit_scope() call.
Since each entry has a reference to scopes defined within it, the
missing DECREF caused all symtable entries to be leaked.
2. The leak here masked a separate problem with
PySymtableEntry_New(). When the requested entry was found in
st->st_symbols, the entry was returned without doing an INCREF.
And problem c) The ste_children slot was getting two copies of each
child entry, because it was populating the slot on the first and
second passes. Now only populate on the first pass.
save the __builtin__ module in a static variable. But this doesn't
work across Py_Finalise()/Py_Initialize()! It also doesn't work when
using multiple interpreter states created with PyInterpreterState_New().
So I'm ripping out this small optimization.
This was probably broken since PyImport_Import() was introduced in
1997! We really need a better test suite for multiple interpreter
states and repeatedly initializing.
This fixes the problems Barry reported in Demo/embed/loop.c.
the symbol table pass. These blocks were already ignored by the code
gen pass. Both passes must visit the same set of blocks in the same
order.
Fixes SF buf 132820
They're actually complaining about something more specific, an assignment
in a lambda as an actual argument, so that Python parses the
lambda as if it were a keyword argument. Like f(lambda x: x[0]=42).
The "lambda x: x[0]" part gets parsed as if it were a keyword, being
bound to 42, and the resulting error msg didn't make much sense.
_testcapimodule.c
make sure PyList_Reverse doesn't blow up again
getargs.c
assert args isn't NULL at the top of vgetargs1 instead of
waiting for a NULL-pointer dereference at the end
Bug was introduced by tricks played to make .pyc files executable
via cmdline arg. Then again, -x worked via a trick to begin with.
If anyone can think of a portable way to test -x, be my guest!
create an empty dictionary if it is called without keyword args. Just
pass NULL.
XXX I had believed that this caused weird errors, but the test suite
runs cleanly.
of nested functions. Either is allowed in a function if it contains
no defs or lambdas or the defs and lambdas it contains have no free
variables. If a function is itself nested and has free variables,
either is illegal.
Revise the symtable to use a PySymtableEntryObject, which holds all
the revelent information for a scope, rather than using a bunch of
st_cur_XXX pointers in the symtable struct. The changes simplify the
internal management of the current symtable scope and of the stack.
Added new C source file: Python/symtable.c. (Does the Windows build
process need to be updated?)
As part of these changes, the initial _symtable module interface
introduced in 2.1a2 is replaced. A dictionary of
PySymtableEntryObjects are returned.
hooks to take over the Python import machinery at a very early stage
in the Python startup phase.
If there are still places in the Python interpreter which need to
bypass the __import__ hook, these places must now use
PyImport_ImportModuleEx() instead. So far no other places than in
the import mechanism itself have been identified.
symtable.h, so that they can be used by external module.
Improve error handling in symtable_enter_scope(), which return an
error code that went unchecked by most callers. XXX The error handling
in symtable code is sloppy in general.
Modify symtable to record the line number that begins each scope.
This can help to identify which code block is being referred to when
multiple blocks are bound to the same name.
Add st_scopes dict that is used to preserve scope info when
PyNode_CompileSymtable() is called. Otherwise, this information is
tossed as soon as it is no longer needed.
Add Py_SymtableString() to pythonrun; analogous to Py_CompileString().
discussion on python-dev. 'from mod import *' is still banned except
at the module level.
Fix value for special NOOPT entry in symtable. Initialze to 0 instead
of None, so that later uses of PyInt_AS_LONG() are valid. (Bug
reported by Donn Cave.)
replace local REPR macros with PyObject_REPR in object.h
reference manual but not checked: Names bound by import statemants may
not occur in global statements in the same scope. The from ... import *
form may only occur in a module scope.
I guess these changes could break code, but the reference manual
warned about them.
Several other small changes
If a variable is declared global in the nearest enclosing scope of a
free variable, then treat it is a global in the nested scope too.
Get rid of com_mangle and symtable_mangle functions and call mangle
directly.
If errors occur during symtable table creation, return -1 from
symtable_build().
Do not increment st_errors in assignment to lambda, because exception
is not set.
Add extra argument to symtable_assign(); the argument, flag, is ORed
with DEF_LOCAL for each symtable_add_def() call.
This change eliminates an extra malloc/free when a frame with free
variables is created. Any cell vars or free vars are stored in
f_localsplus after the locals and before the stack.
eval_code2() fills in the appropriate values after handling
initialization of locals.
To track the size the frame has an f_size member that tracks the total
size of f_localsplus. It used to be implicitly f_nlocals + f_stacksize.
They're named as if public, so I did a Bad Thing by changing
PyMarshal_ReadObjectFromFile() to suck up the remainder of the file in one
gulp: anyone who counted on that leaving the file pointer merely at the
end of the next object would be screwed. So restored
PyMarshal_ReadObjectFromFile() to its earlier state, renamed the new greedy
code to PyMarshal_ReadLastObjectFromFile(), and changed Python internals to
call the latter instead.
SF patch http://sourceforge.net/patch/?func=detailpatch&patch_id=103453&group_id=5470
PyMember_Set of T_CHAR always raises exception.
Unfortunately, this is a use of a C API function that Python itself never makes, so
there's no .py test I can check in to verify this stays fixed. But the fault in the
code is obvious, and Dave Cole's patch just as obviously fixes it.
The majority of the changes are in the compiler. The mainloop changes
primarily to implement the new opcodes and to pass a function's
closure to eval_code2(). Frames and functions got new slots to hold
the closure.
Include/compile.h
Add co_freevars and co_cellvars slots to code objects.
Update PyCode_New() to take freevars and cellvars as arguments
Include/funcobject.h
Add func_closure slot to function objects.
Add GetClosure()/SetClosure() functions (and corresponding
macros) for getting at the closure.
Include/frameobject.h
PyFrame_New() now takes a closure.
Include/opcode.h
Add four new opcodes: MAKE_CLOSURE, LOAD_CLOSURE, LOAD_DEREF,
STORE_DEREF.
Remove comment about old requirement for opcodes to fit in 7
bits.
compile.c
Implement changes to code objects for co_freevars and co_cellvars.
Modify symbol table to use st_cur_name (string object for the name
of the current scope) and st_cur_children (list of nested blocks).
Also define st_nested, which might more properly be called
st_cur_nested. Add several DEF_XXX flags to track def-use
information for free variables.
New or modified functions of note:
com_make_closure(struct compiling *, PyCodeObject *)
Emit LOAD_CLOSURE opcodes as needed to pass cells for free
variables into nested scope.
com_addop_varname(struct compiling *, int, char *)
Emits opcodes for LOAD_DEREF and STORE_DEREF.
get_ref_type(struct compiling *, char *name)
Return NAME_CLOSURE if ref type is FREE or CELL
symtable_load_symbols(struct compiling *)
Decides what variables are cell or free based on def-use info.
Can now raise SyntaxError if nested scopes are mixed with
exec or from blah import *.
make_scope_info(PyObject *, PyObject *, int, int)
Helper functions for symtable scope stack.
symtable_update_free_vars(struct symtable *)
After a code block has been analyzed, it must check each of
its children for free variables that are not defined in the
block. If a variable is free in a child and not defined in
the parent, then it is defined by block the enclosing the
current one or it is a global. This does the right logic.
symtable_add_use() is now a macro for symtable_add_def()
symtable_assign(struct symtable *, node *)
Use goto instead of for (;;)
Fixed bug in symtable where name of keyword argument in function
call was treated as assignment in the scope of the call site. Ex:
def f():
g(a=2) # a was considered a local of f
ceval.c
eval_code2() now take one more argument, a closure.
Implement LOAD_CLOSURE, LOAD_DEREF, STORE_DEREF, MAKE_CLOSURE>
Also: When name error occurs for global variable, report that the
name was global in the error mesage.
Objects/frameobject.c
Initialize f_closure to be a tuple containing space for cellvars
and freevars. f_closure is NULL if neither are present.
Objects/funcobject.c
Add support for func_closure.
Python/import.c
Change the magic number.
Python/marshal.c
Track changes to code objects.
parameters that contained both anonymous tuples and *arg or **arg. Ex:
def f(a, (b, c), *d): pass
Fix the symtable_params() to generate names in the right order for
co_varnames slot of code object. Consider *arg and **arg before the
"complex" names introduced by anonymous tuples.
module__doc__: Document the Warning subclass heirarchy.
make_class(): Added a "goto finally" so that if populate_methods()
fails, the return status will be -1 (failure) instead of 0 (success).
fini_exceptions(): When decref'ing the static pointers to the
exception classes, clear out their dictionaries too. This breaks a
cycle from class->dict->method->class and allows the classes with
unbound methods to be reclaimed. This plugs a large memory leak in a
common Py_Initialize()/dosomething/Py_Finalize() loop.
Jeremy Hylton
Walter Dörwald
Neil Schemenauer
Michael W. Hudson
Martin v. Löwis
Raymond Hettinger
Jason Tishler
Mark Hammond
Fred Drake
Tim Peters
Neal Norwitz
Guido van Rossum
Jack Jansen
Alex Martelli
Thomas Heller
Gustavo Niemeyer
Just van Rossum
Marc-André Lemburg
Andrew MacIntyre
Skip Montanaro
Peter Schneider-Kamp
Barry Warsaw
Greg Ward
Thomas Wouters
Sjoerd Mullender
Moshe Zadka
Fredrik Lundh
Ka-Ping Yee