A new API (only accessible from C) to interrupt a thread by sending it
an exception. This is not always effective, but might help some people.
Requested by Just van Rossum and Alex Martelli. It is intentional
that you have to write your own C extension to call it from Python.
Docs will have to wait.
* Can now test for basic blocks.
* Optimize inverted comparisions.
* Optimize unary_not followed by a conditional jump.
* Added a new opcode, NOP, to keep code size constant.
* Applied NOP to previous transformations where appropriate.
Note, the NOP would not be necessary if other functions were
added to re-target jump addresses and update the co_lnotab mapping.
That would yield slightly faster and cleaner bytecode at the
expense of optimizer simplicity and of keeping it decoupled
from the line-numbering structure.
Added two predictions:
GET_ITER --> FOR_ITER
FOR_ITER --> STORE_FAST or UNPACK_SEQUENCE
Improves timings on pybench and timeit.py. Pystone results are neutral.
Applied to common cases:
COMPARE_OP is often followed by a JUMP_IF.
JUMP_IF is usually followed by POP_TOP.
Shows improved timings on PyStone, PyBench, and specific tests
using timeit.py:
python timeit.py -s "x=1" "if x==1: pass"
python timeit.py -s "x=1" "if x==2: pass"
python timeit.py -s "x=1" "if x: pass"
python timeit.py -s "x=100" "while x!=1: x-=1"
Potential future candidates:
GET_ITER predicts FOR_ITER
FOR_ITER predicts STORE_FAST or UNPACK_SEQUENCE
Also, applied missing goto fast_next_opcode to DUP_TOPX.
My previous patches should have used fast_next_opcode
in a few places instead of continue.
Also, applied one PyInt_AS_LONG macro in a place where
the type had already been checked.
-DCALL_PROFILE: Count the number of function calls executed.
When this symbol is defined, the ceval mainloop and helper functions
count the number of function calls made. It keeps detailed statistics
about what kind of object was called and whether the call hit any of
the special fast paths in the code.
Optimization:
When we take the fast_function() path, which seems to be taken for
most function calls, and there is minimal frame setup to do, avoid
call PyEval_EvalCodeEx(). The eval code ex function does a lot of
work to handle keywords args and star args, free variables,
generators, etc. The inlined version simply allocates the frame and
copies the arguments values into the frame.
The optimization gets a little help from compile.c which adds a
CO_NOFREE flag to code objects that don't have free variables or cell
variables. This change allows fast_function() to get into the fast
path with fewer tests.
I measure a couple of percent speedup in pystone with this change, but
there's surely more that can be done.
Make the code slightly shorter, faster, and easier to
read.
* Eliminate unused DUP_TOPX code for x==1.
compile.c always generates DUP_TOP instead.
* Since only two cases remain for DUP_TOPX, replace
the switch-case with if-elseif.
* The in-lined integer compare does a CheckExact on
both arguments. Since the second is a little more
likely to fail, test it first.
* The switch-case for IS/IS_NOT and IN/NOT_IN can
separate the regular and inverted cases with no
additional work. For all four paths, saves a test and
jump.
The two are semantically equivalent, but the first triggered a compiler
warning about an unused variable. Note, the preceding steps had already
accessed and decreffed the variable so the reference counts were fine.
parameter being either four or five. Currently, compile.c does not
generate calls with a parameter higher than three.
May have to be reverted if the second alpha or beta shakes out some
other tool generating this op code with a parameter of four or five.
Replaced groups of pushes and pops with indexed access to the stack and
a single adjustment (if needed) to the stacklevel.
Avoids scores of unnecessary increments and decrements to the stackpointer.
Removes unnecessary sequential dependencies so that the compiler has more
freedom for optimizations. Frees the processor for more parallel and
pipelined execution by using mostly read-only access and having few pointer
adjustments just prior to a read or write.
all along. Before instr_lb tended to be too high.
I don't think this actually makes any difference, given what the compiler
produces, but it makes me a bit happier.
patch #617312, both on the trunk and the 22-maint branch.
Also added a test case, and ported the test_trace I wrote for HEAD
to 2.2.2 (with all those horrible extra 'line' events ;-).
than when this interval was first established. Checking too frequently just
adds needless overhead because most of the time there is nothing to do and
no other threads ready to run.
globals, _Py_Ticker and _Py_CheckInterval. This also implements Jeremy's
shortcut in Py_AddPendingCall that zeroes out _Py_Ticker. This allows the
test in the main loop to only test a single value.
The gory details are at
http://python.org/sf/602191
Use a slightly different strategy to determine when not to call the line
trace function. This removes the need for the RETURN_NONE opcode, so
that's gone again. Update docs and comments to match.
Thanks to Neal and Armin!
Also add a test suite. This should have come with the original patch...
in LOAD_GLOBAL. Besides saving a C function call, it saves checks
whether f_globals and f_builtins are dicts, and extracting and testing
the string object's hash code is done only once. We bail out of the
inlining if the name is not exactly a string, or when its hash is -1;
because of interning, neither should ever happen. I believe interning
guarantees that the hash code is set, and I believe that the 'names'
tuple of a code object always contains interned strings, but I'm not
assuming that -- I'm simply testing hash != -1.
On my home machine, this makes a pystone variant with new-style
classes and slots run at the same speed as classic pystone! (With
new-style classes but without slots, it is still a lot slower.)
Also, don't handle METH_OLDARGS on the fast path. All the interesting
builtins have been converted to use METH_NOARGS, METH_O, or
METH_VARARGS.
Result is another 1-2% speedup. If I can cobble together 10 of these,
it might make a difference.
This makes the code much easier to ready, because it is at a sane
indentation level. On my box this shows a 1-2% speedup, which means
nothing, except that I'm not going to worry about the performance
effects of the change.
nothing special done if keyword arguments were present, so test for
that earlier and fall through to the normal case if there are any.
This ought to slow down CFunction calls with keyword args, but I don't
care; it's a tiny (1%) improvement for pystone.
[ 587993 ] SET_LINENO killer
Remove SET_LINENO. Tracing is now supported by inspecting co_lnotab.
Many sundry changes to document and adapt to this change.
The staticforward define was needed to support certain broken C
compilers (notably SCO ODT 3.0, perhaps early AIX as well) botched the
static keyword when it was used with a forward declaration of a static
initialized structure. Standard C allows the forward declaration with
static, and we've decided to stop catering to broken C compilers. (In
fact, we expect that the compilers are all fixed eight years later.)
I'm leaving staticforward and statichere defined in object.h as
static. This is only for backwards compatibility with C extensions
that might still use it.
XXX I haven't updated the documentation.
This was a simple typo. Strange that the compiler didn't catch it!
Instead of WHY_CONTINUE, two tests used CONTINUE_LOOP, which isn't a
why_code at all, but an opcode; but even though 'why' is declared as
an enum, comparing it to an int is apparently not even worth a
warning -- not in gcc, and not in VC++. :-(
Will fix in 2.2 too.
[ 558249 ] softspace vs --disable-unicode
And #endif was in the wrong place.
Bugfix candidate, almost surely.
I think I will embark on squashing test failures in --disable-unicode builds --
a Real Bug was hiding under them.
SF bug 535905 (Evil Trashcan and GC interaction).
The SETLOCAL() macro should not DECREF the local variable in-place and
then store the new value; it should copy the old value to a temporary
value, then store the new value, and then DECREF the temporary value.
This is because it is possible that during the DECREF the frame is
accessed by other code (e.g. a __del__ method or gc.collect()) and the
variable would be pointing to already-freed memory.
BUGFIX CANDIDATE!
This fixes the symptom, but PRINT_ITEM has no way to know what (if
anything) PyFile_WriteObject() writes unless the object being printed
is a string. When the object isn't a string, this fix retains the
guess that softspace should be set after PyFile_WriteObject().
We might want to say that it's the job of filelike-object write methods
to leave the file's softspace in the correct state. That would probably
be better -- but everyone relies on PRINT_ITEM to guess for them now.
eval_frame(): Under -Qnew, INPLACE_DIVIDE wasn't getting handed off to
INPLACE_TRUE_DIVIDE (like BINARY_DIVIDE was getting handed off to
BINARY_TRUE_DIVIDE).
Bugfix candidate.
Based on the patch from Danny Yoo. The fix is in exec_statement() in
ceval.c.
There are also changes to introduce use of PyCode_GetNumFree() in
several places.
Had nothing to do with rich comparisons -- some stack cleanup code was
lost as a result of merging in Neil Schemenauer's generators patch.
Reinserted the stack cleanup code, skipping it when yielding.
leak when a class defined a __metaclass__. This fixes the problem
reported on python-dev by Ping; I dunno if it's the same as SF bug
#489669 (since that mentions Unicode).
Big Hammer to implement -Qnew as PEP 238 says it should work (a global
option affecting all instances of "/").
pydebug.h, main.c, pythonrun.c: define a private _Py_QnewFlag flag, true
iff -Qnew is passed on the command line. This should go away (as the
comments say) when true division becomes The Rule. This is
deliberately not exposed to runtime inspection or modification: it's
a one-way one-shot switch to pretend you're using Python 3.
ceval.c: when _Py_QnewFlag is set, treat BINARY_DIVIDE as
BINARY_TRUE_DIVIDE.
test_{descr, generators, zipfile}.py: fiddle so these pass under
-Qnew too. This was just a matter of s!/!//! in test_generators and
test_zipfile. test_descr was trickier, as testbinop() is passed
assumptions that "/" is the same as calling a "__div__" method; put
a temporary hack there to call "__truediv__" instead when the method
name is "__div__" and 1/2 evaluates to 0.5.
Three standard tests still fail under -Qnew (on Windows; somebody
please try the Linux tests with -Qnew too! Linux runs a whole bunch
of tests Windows doesn't):
test_augassign
test_class
test_coercion
I can't stay awake longer to stare at this (be my guest). Offhand
cures weren't obvious, nor was it even obvious that cures are possible
without major hackery.
Question: when -Qnew is in effect, should calls to __div__ magically
change into calls to __truediv__? See "major hackery" at tail end of
last paragraph <wink>.
PyEval_EvalCodeEx(): increment tstate->recursion_depth around the
decref of the frame, because the C stack for this call is still in
use and the decref can lead to __del__ methods getting called.
While this gives tstate->recursion_depth a value proportional to the
depth of the C stack (instead of a small constant no matter how
deeply __del__s recurse), it's not enough to stop the reported crash
when using the default recursion limit on Windows.
Bugfix candidate.
This patch boosts performance for comparing identical string object
by some 20% on my machine while not causing any noticable slow-down
for other operations (according to tests done with pybench).
The new profiler event stream includes a "return" event even when an
exception is being propogated, but the machinery that called the profile
hook did not save & restore the exception. In debug mode, the exception
was detected during the execution of the profile callback, which did not
have the proper internal flags set for the exception. Saving & restoring
the exception state solves the problem.
The profiler does not need to know anything about the exception state,
so we no longer call it when an exception is raised. We do, however,
make sure we *always* call the profiler when we exit a frame. This
ensures that timing events are more easily isolated by a profiler and
finally clauses that do a lot of work don't have their time
mis-allocated.
When an exception is propogated out of the frame, the C callback for
the profiler now receives a PyTrace_RETURN event with an arg of NULL;
the Python-level profile hook function will see a 'return' event with
an arg of None. This means that from Python it is impossible for the
profiler to determine if the frame exited with an exception or if it
returned None, but this doesn't matter for profiling. A C-based
profiler could tell the difference, but this doesn't seem important.
ceval.c:eval_frame(): Simplify the code in two places so that the
profiler is called for every exit from a frame
and not for exceptions.
sysmodule.c:profile_trampoline(): Make sure we don't expose Python
code to NULL; use None instead.
It's possible for PyErr_NormalizeException() to set the traceback
pointer to NULL. I'm not sure how to provoke this directly from
Python, although it may be possible. The error occurs when an
exception is set using PyErr_SetObject() and another exception occurs
while PyErr_NormalizeException() is creating the exception instance.
XXX As a result of this change, it's possible for an exception to
occur but sys.last_traceback to be left undefined. Not sure if this
is a problem.
popped frame-block. What an embarrassing bug! Especially for Jeremy, since
he accepted the patch :-)
This fixes SF bugs #463359 and #462937, and possibly other, *very* obscure
bugs with very deeply nested loops that continue the loop and then break out
of it or raise an exception.
compatibility, this required all places where an array of "struct
memberlist" structures was declared that is referenced from a type's
tp_members slot to change the type of the structure to PyMemberDef;
"struct memberlist" is now only used by old code that still calls
PyMember_Get/Set. The code in PyObject_GenericGetAttr/SetAttr now
calls the new APIs PyMember_GetOne/SetOne, which take a PyMemberDef
argument.
As examples, I added actual docstrings to the attributes of a few
types: file, complex, instance method, super, and xxsubtype.spamlist.
Also converted the symtable to new style getattr.
backwards compatibility. When using the class of the first base as
the metaclass, use its __class__ attribute in preference over its
ob_type slot. This ensures that we can still use classic classes as
metaclasse, as shown in the original "Metaclasses" essay. This also
makes all the examples in Demo/metaclasses/ work again (maybe these
should be turned into a test suite?).
Change to get/set/del slice operations so that if the object doesn't
support slicing, *or* if either of the slice arguments is not an int
or long, we construct a slice object and call the get/set/del item
operation instead. This makes it possible to design classes that
support slice arguments of non-integral types.
CO_FUTURE_DIVISION flag. Redid this to use Jeremy's PyCF_MASK #define
instead, so we dont have to remember to fiddle individual feature names
here again.
pythonrun.h: Also #define a PyCF_MASK_OBSOLETE mask. This isn't used
yet, but will be as part of the PEP 264 implementation (compile() mustn't
raise an error just because old code uses a flag name that's become
obsolete; a warning may be appropriate, but not an error; so compile() has
to know about obsolete flags too, but nobody is going to remember to
update compile() with individual obsolete flag names across releases either
-- i.e., this is the flip side of PyEval_MergeCompilerFlags's oversight).
The descr changes moved the dispatch for calling objects from
call_object() in ceval.c to PyObject_Call() in abstract.c.
call_object() and the many functions it used in ceval.c were no longer
used, but were not removed.
Rename meth_call() as PyCFunction_Call() so that it can be called by
the CALL_FUNCTION opcode in ceval.c.
Also, fix error message that referred to PyEval_EvalCodeEx() by its
old name eval_code2(). (I'll probably refer to it by its old name,
too.)
Replace uses of PyCF_xxx with CO_xxx.
Replace individual feature slots in PyFutureFeatures with single
bitmask ff_features.
When flags must be transfered among the three parts of the interpreter
that care about them -- the pythonrun layer, the compiler, and the
future feature parser -- can simply or (|) the definitions.
This introduces:
- A new operator // that means floor division (the kind of division
where 1/2 is 0).
- The "future division" statement ("from __future__ import division)
which changes the meaning of the / operator to implement "true
division" (where 1/2 is 0.5).
- New overloadable operators __truediv__ and __floordiv__.
- New slots in the PyNumberMethods struct for true and floor division,
new abstract APIs for them, new opcodes, and so on.
I emphasize that without the future division statement, the semantics
of / will remain unchanged until Python 3.0.
Not yet implemented are warnings (default off) when / is used with int
or long arguments.
This has been on display since 7/31 as SF patch #443474.
Flames to /dev/null.
that info to code dynamically compiled *by* code compiled with generators
enabled. Doesn't yet work because there's still no way to tell the parser
that "yield" is OK (unlike nested_scopes, the parser has its fingers in
this too).
Replaced PyEval_GetNestedScopes by a more-general
PyEval_MergeCompilerFlags. Perhaps I should not have? I doubted it was
*intended* to be part of the public API, so just did.
path (with no profile/trace function) through eval_code2() and
eval_frame() avoids several checks.
In the common cases of calls, returns, and exception propogation,
eval_code2() and eval_frame() used to test two values in the
thread-state: the profiling function and the tracing function. With
this change, a flag is set in the thread-state if either of these is
active, allowing a single check to suffice when both are NULL. This
also simplifies the code needed when either function is in use but is
already active (to avoid profiling/tracing the profiler/tracer); the
flag is set to 0 when the profile/trace code is entered, allowing the
same check to suffice for "already in the tracer" for call/return/
exception events.
Python interpreter.
This change adds two new C-level APIs: PyEval_SetProfile() and
PyEval_SetTrace(). These can be used to install profile and trace
functions implemented in C, which can operate at much higher speeds
than Python-based functions. The overhead for calling a C-based
profile function is a very small fraction of a percent of the overhead
involved in calling a Python-based function.
The machinery required to call a Python-based profile or trace
function been moved to sysmodule.c, where sys.setprofile() and
sys.setprofile() simply become users of the new interface.
As a side effect, SF bug #436058 is fixed; there is no longer a
_PyTrace_Init() function to declare.
Iterators list and Python-Dev; e.g., these all pass now:
def g1():
try:
return
except:
yield 1
assert list(g1()) == []
def g2():
try:
return
finally:
yield 1
assert list(g2()) == [1]
def g3():
for i in range(3):
yield None
yield None
assert list(g3()) == [None] * 4
compile.c: compile_funcdef and com_return_stmt: Just van Rossum's patch
to compile the same code for "return" regardless of function type (this
goes back to the previous scheme of returning Py_None).
ceval.c: gen_iternext: take a return (but not a yield) of Py_None as
meaning the generator is exhausted.
the next free valuestack slot, not to the base (in America, stacks push
and pop at the top -- they mutate at the bottom in Australia <winK>).
eval_frame(): assert that f_stacktop isn't NULL upon entry.
frame_delloc(): avoid ordered pointer comparisons involving f_stacktop
when f_stacktop is NULL.
reference to f_back when its really needed. Do a little whitespace
normalization as well. This whole file is a big war between tabs and spaces
but now is probably not the time to reindent everything.
NeilS, please check! This came from staring at your genbug.py, but I'm
not sure it plugs all possible holes. Without this, I caught a
frameobject refcount going negative, and it was also the cause (in debug
build) of _Py_ForgetReference's attempt to forget an object with already-
NULL _ob_prev and _ob_next pointers -- although I'm still not entirely
sure how! Part of the difficulty is that frameobjects are stored on a
free list that gets recycled very quickly, so if there's a stray pointer
to one of them it never looks like an insane frameobject (never goes
trough the free() mangling MS debug forces, etc).
and trace functions lazily, which incurs extra argument pushing and checks
in the C overhead for profiling/tracing, create the strings semi-lazily
when the Python code first registers a profile or trace function. This
simplifies the trampoline into the profile/trace functions.
that should be used to cache an interned version of the event
string passed to the profile/trace function. call_trace() will
create interned strings and cache them in using the storage
specified by this additional parameter, avoiding a lot of string
object creation at runtime when using the profiling or tracing
functions.
All call sites are modified to pass the additional parameter, and four
static PyObject* variables are allocated to cache the interned string
objects.
This closes SF patch #431257.
Keyword arguments passed to builtin functions that don't take them are
ignored.
>>> {}.clear(x=2)
>>>
instead of
>>> {}.clear(x=2)
Traceback (most recent call last):
File "<stdin>", line 1, in ?
TypeError: clear() takes no keyword arguments
If we have a PyCFunction (builtin) and it is METH_VARARGS only, load
the args and dispatch to call_cfunction() directly. This provides a
small speedup for perhaps the most common function calls -- builtins.
sees it (test_iter.py is unchanged).
- Added a tp_iternext slot, which calls the iterator's next() method;
this is much faster for built-in iterators over built-in types
such as lists and dicts, speeding up pybench's ForLoop with about
25% compared to Python 2.1. (Now there's a good argument for
iterators. ;-)
- Renamed the built-in sequence iterator SeqIter, affecting the C API
functions for it. (This frees up the PyIter prefix for generic
iterator operations.)
- Added PyIter_Check(obj), which checks that obj's type has a
tp_iternext slot and that the proper feature flag is set.
- Added PyIter_Next(obj) which calls the tp_iternext slot. It has a
somewhat complex return condition due to the need for speed: when it
returns NULL, it may not have set an exception condition, meaning
the iterator is exhausted; when the exception StopIteration is set
(or a derived exception class), it means the same thing; any other
exception means some other error occurred.
new slot tp_iter in type object, plus new flag Py_TPFLAGS_HAVE_ITER
new C API PyObject_GetIter(), calls tp_iter
new builtin iter(), with two forms: iter(obj), and iter(function, sentinel)
new internal object types iterobject and calliterobject
new exception StopIteration
new opcodes for "for" loops, GET_ITER and FOR_ITER (also supported by dis.py)
new magic number for .pyc files
new special method for instances: __iter__() returns an iterator
iteration over dictionaries: "for x in dict" iterates over the keys
iteration over files: "for x in file" iterates over lines
TODO:
documentation
test suite
decide whether to use a different way to spell iter(function, sentinal)
decide whether "for key in dict" is a good idea
use iterators in map/filter/reduce, min/max, and elsewhere (in/not in?)
speed tuning (make next() a slot tp_next???)
now raises NameError instead of UnboundLocalError, because the var in
question is definitely not local. (This affects test_scope.py)
Also update the recent fix by Ping using get_func_name(). Replace
tests of get_func_name() return value with call to get_func_desc() to
match all the other uses.
Calling an unbound method on a C extension class without providing
an instance can yield a segfault. Try "Exception.__init__()" or
"ValueError.__init__()".
This is a simple fix. The error-reporting bits in call_method
mistakenly treat the misleadingly-named variable "func" as a
function, when in fact it is a method.
If we let get_func_name take care of the work, all is fine.
Fix based on patch #414750 by Michael Hudson.
New functions get_func_name() and get_func_desc() return reasonable
names and descriptions for all objects. XXX Even objects that aren't
actually callable.
If a module has a future statement enabling nested scopes, they are
also enable for the exec statement and the functions compile() and
execfile() if they occur in the module.
If Python is run with the -i option, which enters interactive mode
after executing a script, and the script it runs enables nested
scopes, they are also enabled in interactive mode.
XXX The use of -i with -c "from __future__ import nested_scopes" is
not supported. What's the point?
To support these changes, many function variants have been added to
pythonrun.c. All the variants names end with Flags and they take an
extra PyCompilerFlags * argument. It is possible that this complexity
will be eliminated in a future version of the interpreter in which
nested scopes are not optional.
frees. Note there doesn't seem to be any way to test LocalsToFast(),
because the instructions that trigger it are illegal in nested scopes
with free variables.
Fix allocation strategy for cells that are also formal parameters.
Instead of emitting LOAD_FAST / STORE_DEREF pairs for each parameter,
have the argument handling code in eval_code2() do the right thing.
A side-effect of this change is that cell variables that are also
arguments are listed at the front of co_cellvars in the order they
appear in the argument list.
with free variables. Thanks to Martin v. Loewis for finding two of
the problems. This fixes SF buf 405583.
There is also a C API change: PyFrame_New() is reverting to its
pre-2.1 signature. The change introduced by nested scopes was a
mistake. XXX Is this okay between beta releases?
cell_clear(), the GC helper, must decref its reference to break
cycles.
frame_dealloc() must dealloc all cell vars and free vars in addition
to locals.
eval_code2() setup code must INCREF cells it copies out of the
closure.
The STORE_DEREF opcode implementation must DECREF the object it passes
to PyCell_Set().
create an empty dictionary if it is called without keyword args. Just
pass NULL.
XXX I had believed that this caused weird errors, but the test suite
runs cleanly.
discussion on python-dev. 'from mod import *' is still banned except
at the module level.
Fix value for special NOOPT entry in symtable. Initialze to 0 instead
of None, so that later uses of PyInt_AS_LONG() are valid. (Bug
reported by Donn Cave.)
replace local REPR macros with PyObject_REPR in object.h
This change eliminates an extra malloc/free when a frame with free
variables is created. Any cell vars or free vars are stored in
f_localsplus after the locals and before the stack.
eval_code2() fills in the appropriate values after handling
initialization of locals.
To track the size the frame has an f_size member that tracks the total
size of f_localsplus. It used to be implicitly f_nlocals + f_stacksize.
The majority of the changes are in the compiler. The mainloop changes
primarily to implement the new opcodes and to pass a function's
closure to eval_code2(). Frames and functions got new slots to hold
the closure.
Include/compile.h
Add co_freevars and co_cellvars slots to code objects.
Update PyCode_New() to take freevars and cellvars as arguments
Include/funcobject.h
Add func_closure slot to function objects.
Add GetClosure()/SetClosure() functions (and corresponding
macros) for getting at the closure.
Include/frameobject.h
PyFrame_New() now takes a closure.
Include/opcode.h
Add four new opcodes: MAKE_CLOSURE, LOAD_CLOSURE, LOAD_DEREF,
STORE_DEREF.
Remove comment about old requirement for opcodes to fit in 7
bits.
compile.c
Implement changes to code objects for co_freevars and co_cellvars.
Modify symbol table to use st_cur_name (string object for the name
of the current scope) and st_cur_children (list of nested blocks).
Also define st_nested, which might more properly be called
st_cur_nested. Add several DEF_XXX flags to track def-use
information for free variables.
New or modified functions of note:
com_make_closure(struct compiling *, PyCodeObject *)
Emit LOAD_CLOSURE opcodes as needed to pass cells for free
variables into nested scope.
com_addop_varname(struct compiling *, int, char *)
Emits opcodes for LOAD_DEREF and STORE_DEREF.
get_ref_type(struct compiling *, char *name)
Return NAME_CLOSURE if ref type is FREE or CELL
symtable_load_symbols(struct compiling *)
Decides what variables are cell or free based on def-use info.
Can now raise SyntaxError if nested scopes are mixed with
exec or from blah import *.
make_scope_info(PyObject *, PyObject *, int, int)
Helper functions for symtable scope stack.
symtable_update_free_vars(struct symtable *)
After a code block has been analyzed, it must check each of
its children for free variables that are not defined in the
block. If a variable is free in a child and not defined in
the parent, then it is defined by block the enclosing the
current one or it is a global. This does the right logic.
symtable_add_use() is now a macro for symtable_add_def()
symtable_assign(struct symtable *, node *)
Use goto instead of for (;;)
Fixed bug in symtable where name of keyword argument in function
call was treated as assignment in the scope of the call site. Ex:
def f():
g(a=2) # a was considered a local of f
ceval.c
eval_code2() now take one more argument, a closure.
Implement LOAD_CLOSURE, LOAD_DEREF, STORE_DEREF, MAKE_CLOSURE>
Also: When name error occurs for global variable, report that the
name was global in the error mesage.
Objects/frameobject.c
Initialize f_closure to be a tuple containing space for cellvars
and freevars. f_closure is NULL if neither are present.
Objects/funcobject.c
Add support for func_closure.
Python/import.c
Change the magic number.
Python/marshal.c
Track changes to code objects.
- Use PyObject_RichCompare*() where possible: when comparing
keyword arguments, in _PyEval_SliceIndex(), and of course in
cmp_outcome().
Unrelated stuff:
- Removed all trailing whitespace.
- Folded some long lines.
message, and tries to make the messages more consistent and helpful when
the wrong number of arguments or duplicate keyword arguments are supplied.
Comes with more tests for test_extcall.py and and an update to an error
message in test/output/test_pyexpat.
1) "from M import X" now works even if M is not a real module; it's
basically a getattr() operation with AttributeError exceptions
changed into ImportError.
2) "from M import *" now looks for M.__all__ to decide which names to
import; if M.__all__ doesn't exist, it uses M.__dict__.keys() but
filters out names starting with '_' as before. Whether or not
__all__ exists, there's no restriction on the type of M.
an empty keywords dictionary (via apply() or the extended call syntax),
the keywords dict should be ignored. If the keywords dict is not empty,
TypeError should be raised. (Between the restructuring of the call
machinery and this patch, an empty dict in this situation would trigger
a SystemError via PyErr_BadInternalCall().)
Added regression tests to detect errors for this.
More revision still needed.
Much of the code that was in the mainloop was moved to a series of
helper functions. PyEval_CallObjectWithKeywords was split into two
parts. The first part now only does argument handling. The second
part is now named call_object and delegates the call to a
call_(function,method,etc.) helper.
XXX The call_XXX helper functions should be replaced with tp_call
functions for the respective types.
The CALL_FUNCTION implementation contains three kinds of optimization:
1. fast_cfunction and fast_function are called when the arguments on
the stack can be passed directly to eval_code2() without copying
them into a tuple.
2. PyCFunction objects are dispatched immediately, because they are
presumed to occur more often than anything else.
3. Bound methods are dispatched inline. The method object contains a
pointer to the function object that will be called. The function
is called from within the mainloop, which may allow optimization #1
to be used, too.
The extened call implementation -- f(*args) and f(**kw) -- are
implemented as a separate case in the mainloop. This allows the
common case of normal function calls to execute without wasting time
on checks for extended calls, although it does introduce a small
amount of code duplication.
Also, the unused final argument of eval_code2() was removed. This is
probably the last trace of the access statement :-).
When a method is called with no regular arguments and * args, defer
the first arg is subclass check until after the * args have been
expanded.
N.B. The CALL_FUNCTION implementation is getting really hairy; should
review it to see if it can be simplified.
by making the DUP_TOPX code utterly straightforward. This also gets rid
of all normal-case internal DUP_TOPX if/branches, and allows replacing one
POP() with TOP() in each case, so is a good idea regardless.
Add definitions of INT_MAX and LONG_MAX to pyport.h.
Remove includes of limits.h and conditional definitions of INT_MAX
and LONG_MAX elsewhere.
This closes SourceForge patch #101659 and bug #115323.
which implements the automatic conversion from Unicode to a string
object using the default encoding.
The new API is then put to use to have eval() and exec accept
Unicode objects as code parameter. This closes bugs #110924
and #113890.
As side-effect, the traditional C APIs PyString_Size() and
PyString_AsString() will also accept Unicode objects as
parameters.
ceval.c:
define recurion_limit (static), default value is 2500
define Py_GetRecursionLimit and Py_SetRecursionLimit
raise RuntimeError if limit is exceeded
PC/config.h:
remove plat-specific definition
sysmodule.c:
add sys.(get|set)recursionlimit
how 'import' was called with a compiletime mechanism: create either a tuple
of the import arguments, or None (in the case of a normal import), add it to
the code-block constants, and load it onto the stack before calling
IMPORT_NAME.
Add the EXTENDED_ARG opcode to the virtual machine, allowing 32-bit
arguments to opcodes instead of being forced to stick to the 16-bit
limit. This is especially useful for machine-generated code, which
can be too long for the SET_LINENO parameter to fit into 16 bits.
This closes the implementation portion of SourceForge patch #100893.
eval_code2(): Implement new bytecodes PRINT_ITEM_TO and
PRINT_NEWLINE_TO, as per accepted SF patch #100970.
Also update graminit.c based on related Grammar/Grammar changes.
trying hard enough to find out what the arguments to an import were. There
is no test-case for this bug, yet, but this is what it looked like:
from encodings import cp1006, cp1026
ImportError: cannot import name cp1026
'__import__' was called with only the first name in the 'arguments' list.
name as n'. By doing some twists and turns, "as" is not a reserved word.
There is a slight change in semantics for 'from module import name' (it will
now honour the 'global' keyword) but only in cases that are explicitly
undocumented.
did the same anyway.
I'm not sure what to do with Tools/compiler/compiler/* -- that isn't part of
distutils, is it ? Should it try to be compatible with old bytecode version ?
comments, docstrings or error messages. I fixed two minor things in
test_winreg.py ("didn't" -> "Didn't" and "Didnt" -> "Didn't").
There is a minor style issue involved: Guido seems to have preferred English
grammar (behaviour, honour) in a couple places. This patch changes that to
American, which is the more prominent style in the source. I prefer English
myself, so if English is preferred, I'd be happy to supply a patch myself ;)
For more comments, read the patches@python.org archives.
For documentation read the comments in mymalloc.h and objimpl.h.
(This is not exactly what Vladimir posted to the patches list; I've
made a few changes, and Vladimir sent me a fix in private email for a
problem that only occurs in debug mode. I'm also holding back on his
change to main.c, which seems unnecessary to me.)
"""
Running "test_extcall" repeatedly results in memory leaks.
One of these can't be fixed (at least not easily!), it happens since
this code:
def saboteur(**kw):
kw['x'] = locals()
d = {}
saboteur(a=1, **d)
creates a circular reference - d['x']['d']==d
The others are due to some missing decrefs in ceval.c, fixed by the
patch attached below.
Note: I originally wrote this without the "goto", just adding the
missing decref's where needed. But I think the goto is justified in
keeping the executable code size of ceval as small as possible.
"""
[I think the circular reference is more like kw['x']['kw'] == kw. --GvR]
If a non-tuple sequence is passed as the *arg, convert it to a tuple
before checking its length.
If named keyword arguments are used in combination with **kwargs, make
a copy of kwargs before inserting the new keys.
the return value of PySequence_Length(). If an exception occurred,
the returned length will be -1. Make sure this doesn't get obscurred,
and that the bogus length isn't used.
executive summary:
Instead of typing 'apply(f, args, kwargs)' you can type 'f(*arg, **kwargs)'.
Some file-by-file details follow.
Grammar/Grammar:
simplify varargslist, replacing '*' '*' with '**'
add * & ** options to arglist
Include/opcode.h & Lib/dis.py:
define three new opcodes
CALL_FUNCTION_VAR
CALL_FUNCTION_KW
CALL_FUNCTION_VAR_KW
Python/ceval.c:
extend TypeError "keyword parameter redefined" message to include
the name of the offending keyword
reindent CALL_FUNCTION using four spaces
add handling of sequences and dictionaries using extend calls
fix function import_from to use PyErr_Format
PR#175 -- when exec is passed a code object, it didn't sync the locals
from the dictionary back into their fast representation.
Also took the time to remove some repetitive code there and to do the
syncing even when an exception is raised (since a partial effect
should still be synced).
not as descriptive as what Barry suggests, but this also catches the
(in my opinion important) case where some other C code besides apply()
constructs a kwdict that doesn't have the right format. All the other
possibilities of getting it wrong (non-dict, wrong keywords etc) are
already caught so this makes sense to check here.
tracefunc (or profilefunc -- we're not sure which), zap the global
trace and profile funcs so that we can't get into recursive loop when
instantiating the resulting class based exception.
Introduce a new builtin exception, UnboundLocalError, raised when ceval.c
tries to retrieve or delete a local name that isn't bound to a value.
Currently raises NameError, which makes this behavior a FAQ since the same
error is raised for "missing" global names too: when the user has a global
of the same name as the unbound local, NameError makes no sense to them.
Even in the absence of shadowing, knowing whether a bogus name is local or
global is a real aid to quick understanding.
Example:
D:\src\PCbuild>type local.py
x = 42
def f():
print x
x = 13
return x
f()
D:\src\PCbuild>python local.py
Traceback (innermost last):
File "local.py", line 8, in ?
f()
File "local.py", line 4, in f
print x
UnboundLocalError: x
D:\src\PCbuild>
Note that UnboundLocalError is a subclass of NameError, for compatibility
with existing class-exception code that may be trying to catch this as a
NameError. Unfortunately, I see no way to make this wholly compatible
with -X (see comments in bltinmodule.c): under -X, [UnboundLocalError
is an alias for NameError --GvR].
[The ceval.c patch differs slightly from the second version that Tim
submitted; I decided not to raise UnboundLocalError for DELETE_NAME,
only for DELETE_LOCAL. DELETE_NAME is only generated at the module
level, and since at that level a NameError is raised for referencing
an undefined name, it should also be raised for deleting one.]
swapped arguments].
Also make sure that no use of a function pointer gotten from a
tp_as_sequence or tp_as_mapping structure is made without checking it
for NULL first.
tstate swapping. Only the acquiring and releasing of the lock is
conditional (twice, under ``#ifdef WITH_THREAD'' and inside ``if
(interpreter_lock)'').
This doesn't yet support "import a.b.c" or "from a.b.c import x", but
it does recognize directories. When importing a directory, it
initializes __path__ to a list containing the directory name, and
loads the __init__ module if found.
The (internal) find_module() and load_module() functions are
restructured so that they both also handle built-in and frozen modules
and Mac resources (and directories of course). The imp module's
find_module() and (new) load_module() also have this functionality.
Moreover, imp unconditionally defines constants for all module types,
and has two more new functions: find_module_in_package() and
find_module_in_directory().
There's also a new API function, PyImport_ImportModuleEx(), which
takes all four __import__ arguments (name, globals, locals, fromlist).
The last three may be NULL. This is currently the same as
PyImport_ImportModule() but in the future it will be able to do
relative dotted-path imports.
Other changes:
- bltinmodule.c: in __import__, call PyImport_ImportModuleEx().
- ceval.c: always pass the fromlist to __import__, even if it is a C
function, so PyImport_ImportModuleEx() is useful.
- getmtime.c: the function has a second argument, the FILE*, on which
it applies fstat(). According to Sjoerd this is much faster. The
first (pathname) argument is ignored, but remains for backward
compatibility (so the Mac version still works without changes).
By cleverly combining the new imp functionality, the full support for
dotted names in Python (mini.py, not checked in) is now about 7K,
lavishly commented (vs. 14K for ni plus 11K for ihooks, also lavishly
commented).
Good night!
former rather than the latter, since PyErr_NormalizeException takes
PyObject** and I didn't want to change the interface for set_exc_info
(but I did want the changes propagated to eval_code2!).
UNPACK_LIST byte codes and added a third code path that allows
generalized sequence unpacking. Now both syntaxes:
a, b, c = seq
[a, b, c] = seq
can be used to unpack any sequence with the exact right number of
items.
unpack_sequence(): out-lined implementation of generalized sequence
unpacking. tuple and list unpacking are still inlined.
PyErr_GivenExceptionMatches().
set_exc_info(): make sure to normalize exceptions.
do_raise(): Use PyErr_NormalizeException() if type is a class.
loop_subscript(): Use PyErr_ExceptionMatches() instead of raw pointer
compare for PyExc_IndexError.
PyThreadState pointer instead of a (frame) PyObject pointer. This
makes much more sense. It is backward incompatible, but that's no
problem, because (a) the heaviest users are the Py_{BEGIN,END}_
ALLOW_THREADS macros here, which have been fixed too; (b) there are
very few direct users; (c) those who use it are there will probably
appreciate the change.
Also, added new functions PyEval_AcquireThread() and
PyEval_ReleaseThread() which allows the threads created by the thread
module as well threads created by others (!) to set/reset the current
thread, and at the same time acquire/release the interpreter lock.
Much saner.
int+int, int-int, int <compareop> int, and list[int].
(Unfortunately, int*int is way too much code to inline.)
Also corrected a NULL that should have been a zero.
Jack Jansen
Jeremy Hylton
Armin Rigo
Raymond Hettinger
Neal Norwitz
Guido van Rossum
Neil Schemenauer
Michael W. Hudson
Mark Hammond
Just van Rossum
Skip Montanaro
Fred Drake
Tim Peters
Martin v. Löwis
Marc-André Lemburg
Thomas Wouters
Sjoerd Mullender
Ka-Ping Yee
Moshe Zadka
Charles G. Waldman
Vladimir Marangozov
Paul Prescod
Barry Warsaw
Andrew M. Kuchling