The compiler was reseting the list comprehension tmpname counter for each function, but the symtable was using the same counter for the entire module. Repair by move tmpname into the symtable entry.
Bugfix candidate.
[ 708901 ] Lineno calculation sometimes broken
A one line patch to compile.c and a rather-more-than-one-line patch
to test_dis. Hey ho.
Possibly a backport candidate -- tho' lnotab is less used in 2.2...
* Can now test for basic blocks.
* Optimize inverted comparisions.
* Optimize unary_not followed by a conditional jump.
* Added a new opcode, NOP, to keep code size constant.
* Applied NOP to previous transformations where appropriate.
Note, the NOP would not be necessary if other functions were
added to re-target jump addresses and update the co_lnotab mapping.
That would yield slightly faster and cleaner bytecode at the
expense of optimizer simplicity and of keeping it decoupled
from the line-numbering structure.
Adds a single function to improve generated bytecode. Has a single line
attachment point, so it is completely de-coupled from both the compiler
and ceval.c.
Makes three simple transforms that do not require a basic block analysis
or re-ordering of code. Gives improved timings on pystone, pybench,
and any code using either "while 1" or "x,y=y,x".
"Unsigned" (i.e., positive-looking, but really negative) hex/oct
constants with a leading minus sign are once again properly negated.
The micro-optimization for negated numeric constants did the wrong
thing for such hex/oct constants. The patch avoids the optimization
for all hex/oct constants.
This needs to be backported to Python 2.2!
-DCALL_PROFILE: Count the number of function calls executed.
When this symbol is defined, the ceval mainloop and helper functions
count the number of function calls made. It keeps detailed statistics
about what kind of object was called and whether the call hit any of
the special fast paths in the code.
Optimization:
When we take the fast_function() path, which seems to be taken for
most function calls, and there is minimal frame setup to do, avoid
call PyEval_EvalCodeEx(). The eval code ex function does a lot of
work to handle keywords args and star args, free variables,
generators, etc. The inlined version simply allocates the frame and
copies the arguments values into the frame.
The optimization gets a little help from compile.c which adds a
CO_NOFREE flag to code objects that don't have free variables or cell
variables. This change allows fast_function() to get into the fast
path with fewer tests.
I measure a couple of percent speedup in pystone with this change, but
there's surely more that can be done.
A variety of changes from Michael Hudson to get the compiler working
with 2.3. The primary change is the handling of SET_LINENO:
# The set_lineno() function and the explicit emit() calls for
# SET_LINENO below are only used to generate the line number table.
# As of Python 2.3, the interpreter does not have a SET_LINENO
# instruction. pyassem treats SET_LINENO opcodes as a special case.
A few other small changes:
- Remove unused code from pycodegen and pyassem.
- Fix error handling in parsermodule. When PyParser_SimplerParseString()
fails, it sets an exception with detailed info. The parsermodule
was clobbering that exception and replacing it was a generic
"could not parse string" exception. Keep the original exception.
[#448679] Left to right
* Python/compile.c
(com_dictmaker): Reordered evaluation of dictionaries to follow strict
LTR evaluation.
* Lib/compiler/pycodegen.py
(CodeGenerator.visitDict): Reordered evaluation of dictionaries to
follow strict LTR evaluation.
* Doc/ref/ref5.tex
Documented the general LTR evaluation order idea.
* Misc/NEWS
Documented change in evaluation order of dictionaries.
Use a slightly different strategy to determine when not to call the line
trace function. This removes the need for the RETURN_NONE opcode, so
that's gone again. Update docs and comments to match.
Thanks to Neal and Armin!
Also add a test suite. This should have come with the original patch...
warning for 'global None', but that's either accompanied by an
assignment to None, which will trigger a warning, or not, in which
case it's harmless. :-)
[ 587993 ] SET_LINENO killer
Remove SET_LINENO. Tracing is now supported by inspecting co_lnotab.
Many sundry changes to document and adapt to this change.
currently return inconsistent results for ints and longs; in
particular: hex/oct/%u/%o/%x/%X of negative short ints, and x<<n that
either loses bits or changes sign. (No warnings for repr() of a long,
though that will also change to lose the trailing 'L' eventually.)
This introduces some warnings in the test suite; I'll take care of
those later.
Change the parser and compiler to use PyMalloc.
Only the files implementing processes that will request memory
allocations small enough for PyMalloc to be a win have been
changed, which are:-
- Python/compile.c
- Parser/acceler.c
- Parser/node.c
- Parser/parsetok.c
This augments the aggressive overallocation strategy implemented by
Tim Peters in PyNode_AddChild() [Parser/node.c], in reducing the
impact of platform malloc()/realloc()/free() corner case behaviour.
Such corner cases are known to be triggered by test_longexp and
test_import.
Jeremy Hylton, in accepting this patch, recommended this as a
bugfix candidate for 2.2. While the changes to Python/compile.c
and Parser/node.c backport easily (and could go in), the changes
to Parser/acceler.c and Parser/parsetok.c require other not
insignificant changes as a result of the differences in the memory
APIs between 2.3 and 2.2, which I'm not in a position to work
through at the moment. This is a pity, as the Parser/parsetok.c
changes are the most important after the Parser/node.c changes, due
to the size of the memory requests involved and their frequency.
is slow things down unnecessarily and make tracing much more verbose.
Something like
def f(n):
return [i for i in range(n) if i%2]
should have at most two SET_LINENO instructions, not four. When tracing,
the current line number should be printed once, not 2*n+1 times.
These built-in functions are replaced by their (now callable) type:
slice()
buffer()
and these types can also be called (but have no built-in named
function named after them)
classobj (type name used to be "class")
code
function
instance
instancemethod (type name used to be "instance method")
The module "new" has been replaced with a small backward compatibility
placeholder in Python.
A large portion of the patch simply removes the new module from
various platform-specific build recipes. The following binary Mac
project files still have references to it:
Mac/Build/PythonCore.mcp
Mac/Build/PythonStandSmall.mcp
Mac/Build/PythonStandalone.mcp
[I've tweaked the code layout and the doc strings here and there, and
added a comment to types.py about StringTypes vs. basestring. --Guido]
The old syntax suggested that a trailing comma was OK inside backticks,
but in fact (due to ideosyncrasies of pgen) it was not. Fix the grammar
to avoid the ambiguity. Fred: you may want to update the refman.
+ Redirect PyMem_{Del, DEL} to the object allocator's free() when
pymalloc is enabled. Needed so old extensions can continue to
mix PyObject_New with PyMem_DEL.
+ This implies that pgen needs to be able to see the PyObject_XYZ
declarations too. pgenheaders.h now includes Python.h. An
implication is that I expect obmalloc.o needs to get linked into
pgen on non-Windows boxes.
+ When PYMALLOC_DEBUG is defined, *all* Py memory API functions
now funnel through the debug allocator wrapper around pymalloc.
This is the default in a debug build.
+ That caused compile.c to fail: it indirectly mixed PyMem_Malloc
with raw platform free() in one place. This is verbotten.
type.__module__ problems (again?)
This simply initializes the __module__ local in a class statement from
the __name__ global. I'm not 100% sure that this is the correct fix,
although it usually does the right thing. The problem is that if the
class statement executes in a custom namespace, the __name__ global
may be taken from __builtins__, in which case it would have the value
__builtin__, or it may not exist at all (if the custom namespace also
has a custom __builtins__), in which case the class statement will
fail.
Nevertheless, unless someone finds a better solution, this is a 2.2.1
bugfix too.
Based on the patch from Danny Yoo. The fix is in exec_statement() in
ceval.c.
There are also changes to introduce use of PyCode_GetNumFree() in
several places.
The error for assignment to __debug__ used ste->ste_opt_lineno instead
of n->n_lineno. The latter was at best incorrect; often the slot was
uninitialized. Two fixes here: Use the correct lineno for the error.
Initialize ste_opt_lineno in PySymtable_New(); while there are no
current cases where it is referenced unless it has already been
assigned to, there is no harm in initializing it.
com_arglist(), symtable_check_unoptimized(), symtable_params(),
symtable_global(), symtable_list_comprehension():
Conversion of sprintf() to PyOS_snprintf() for buffer overrun
avoidance.
The symbol table pass didn't have an explicit case for the list_iter
node which is used only for a nested list comprehension. As a result,
the target of the list comprehension was treated as a use instead of
an assignment. Fix is to add a case to symtable_node() to handle
list_iter.
Also, rework and document a couple of the subtler implementation
issues in the symbol table pass. The symtable_node() switch statement
depends on falling through the last several cases, in order to handle
some of the more complicated nodes like atom. Add a comment
explaining the behavior before the first fall through case. Add a
comment /* fall through */ at the end of case so that it is explicitly
marked as such.
Move the for_stmt case out of the fall through logic, which simplifies
both for_stmt and default. (The default used the local variable start
to skip the first three nodes of a for_stmt when it fell through.)
Rename the flag argument to symtable_assign() to def_flag and add a
comment explaining its use:
The third argument to symatble_assign() is a flag to be passed to
symtable_add_def() if it is eventually called. The flag is useful
to specify the particular type of assignment that should be
recorded, e.g. an assignment caused by import.
Also minor tweaks to internal routines.
Use PyCF_MASK instead of explicit list of flags.
For the MAKE_CLOSURE opcode, the number of items popped off the stack
depends on both the oparg and the number of free variables for the
code object. Fix the code so it accounts for the free variables.
In com_classdef(), record an extra pop to account for the STORE call
after the BUILD_CLASS.
Get rid of some commented out debugging code in com_push() and
com_pop().
Factor string resize logic into helper routine com_check_size().
In com_addbyte(), remove redudant if statement after assert. (They
test the same condition.)
In several routines, use string macros instead of string functions.
"for <var> in <testlist> may no longer be a single test followed by
a comma. This solves SF bug #431886. Note that if the testlist
contains more than one test, a trailing comma is still allowed, for
maximum backward compatibility; but this example is not:
[(x, y) for x in range(10), for y in range(10)]
^
The fix involved creating a new nonterminal 'testlist_safe' whose
definition doesn't allow the trailing comma if there's only one test:
testlist_safe: test [(',' test)+ [',']]
compatibility, this required all places where an array of "struct
memberlist" structures was declared that is referenced from a type's
tp_members slot to change the type of the structure to PyMemberDef;
"struct memberlist" is now only used by old code that still calls
PyMember_Get/Set. The code in PyObject_GenericGetAttr/SetAttr now
calls the new APIs PyMember_GetOne/SetOne, which take a PyMemberDef
argument.
As examples, I added actual docstrings to the attributes of a few
types: file, complex, instance method, super, and xxsubtype.spamlist.
Also converted the symtable to new style getattr.
com_factor(): when a unary minus is attached to a float or imaginary zero,
don't optimize the UNARY_MINUS opcode away: the const dict can't
distinguish between +0.0 and -0.0, so ended up treating both like the
first one added to it. Optimizing UNARY_PLUS away isn't a problem.
(BTW, I already uploaded the 2.2a3 Windows installer, and this isn't
important enough to delay the release.)
- Do not compile unicodeobject, unicodectype, and unicodedata if Unicode is disabled
- check for Py_USING_UNICODE in all places that use Unicode functions
- disables unicode literals, and the builtin functions
- add the types.StringTypes list
- remove Unicode literals from most tests.
When code is compiled and compiler flags are passed in, be sure to
update cf_flags with any features defined by future statements in the
compiled code.
Revised version of Fred's patch, including support for ~ operator.
If the unary +, -, or ~ operator is applied to a constant, don't
generate a UNARY_xxx opcode. Just store the approriate value as a
constant. If the value is negative, extend the string containing the
constant and insert a negative in the 0th position.
For ~, compute the inverse of int and longs and use them directly, but
be prepared to generate code for all other possibilities (invalid
numbers, floats, complex).
Replace uses of PyCF_xxx with CO_xxx.
Replace individual feature slots in PyFutureFeatures with single
bitmask ff_features.
When flags must be transfered among the three parts of the interpreter
that care about them -- the pythonrun layer, the compiler, and the
future feature parser -- can simply or (|) the definitions.
This introduces:
- A new operator // that means floor division (the kind of division
where 1/2 is 0).
- The "future division" statement ("from __future__ import division)
which changes the meaning of the / operator to implement "true
division" (where 1/2 is 0.5).
- New overloadable operators __truediv__ and __floordiv__.
- New slots in the PyNumberMethods struct for true and floor division,
new abstract APIs for them, new opcodes, and so on.
I emphasize that without the future division statement, the semantics
of / will remain unchanged until Python 3.0.
Not yet implemented are warnings (default off) when / is used with int
or long arguments.
This has been on display since 7/31 as SF patch #443474.
Flames to /dev/null.
that info to code dynamically compiled *by* code compiled with generators
enabled. Doesn't yet work because there's still no way to tell the parser
that "yield" is OK (unlike nested_scopes, the parser has its fingers in
this too).
Replaced PyEval_GetNestedScopes by a more-general
PyEval_MergeCompilerFlags. Perhaps I should not have? I doubted it was
*intended* to be part of the public API, so just did.
"return expr" instances in generators (which latter may be generators
due to otherwise invisible "yield" stmts hiding in "if 0" blocks).
This was fun the first time, but this has gotten truly ugly now.
Iterators list and Python-Dev; e.g., these all pass now:
def g1():
try:
return
except:
yield 1
assert list(g1()) == []
def g2():
try:
return
finally:
yield 1
assert list(g2()) == [1]
def g3():
for i in range(3):
yield None
yield None
assert list(g3()) == [None] * 4
compile.c: compile_funcdef and com_return_stmt: Just van Rossum's patch
to compile the same code for "return" regardless of function type (this
goes back to the previous scheme of returning Py_None).
ceval.c: gen_iternext: take a return (but not a yield) of Py_None as
meaning the generator is exhausted.
Armin Rigo pointed out that the way the line-# table got built didn't work
for lines generating more than 255 bytes of bytecode. Fixed as he
suggested, plus corresponding changes to pyassem.py, plus added some
long overdue docs about this subtle table to compile.c.
Bugfix candidate (line numbers may be off in tracebacks under -O).
Check for free in class and method only if nested scopes are enabled.
Add assertion to verify that no free variables occur when nested
scopes are disabled.
XXX When should nested scopes by made non-optional on the trunk?
The new test case demonstrates the bug. Be more careful in
symtable_resolve_free() to add a var to cells or frees only if it
won't be added under some other rule.
XXX Add new assertion that will catch this bug.
new slot tp_iter in type object, plus new flag Py_TPFLAGS_HAVE_ITER
new C API PyObject_GetIter(), calls tp_iter
new builtin iter(), with two forms: iter(obj), and iter(function, sentinel)
new internal object types iterobject and calliterobject
new exception StopIteration
new opcodes for "for" loops, GET_ITER and FOR_ITER (also supported by dis.py)
new magic number for .pyc files
new special method for instances: __iter__() returns an iterator
iteration over dictionaries: "for x in dict" iterates over the keys
iteration over files: "for x in file" iterates over lines
TODO:
documentation
test suite
decide whether to use a different way to spell iter(function, sentinal)
decide whether "for key in dict" is a good idea
use iterators in map/filter/reduce, min/max, and elsewhere (in/not in?)
speed tuning (make next() a slot tp_next???)
frees. Note there doesn't seem to be any way to test LocalsToFast(),
because the instructions that trigger it are illegal in nested scopes
with free variables.
Fix allocation strategy for cells that are also formal parameters.
Instead of emitting LOAD_FAST / STORE_DEREF pairs for each parameter,
have the argument handling code in eval_code2() do the right thing.
A side-effect of this change is that cell variables that are also
arguments are listed at the front of co_cellvars in the order they
appear in the argument list.
has a binding for the name. The fix is in two places:
- in symtable_update_free_vars, ignore a global stmt in a class scope
- in symtable_load_symbols, add extra handling for names that are
defined at class scope and free in a method
Closes SF bug 407800
Jeremy Hylton
Tim Peters
Martin v. Löwis
Michael W. Hudson
Raymond Hettinger
Guido van Rossum
Just van Rossum
Neal Norwitz
Gustavo Niemeyer
Walter Dörwald
Barry Warsaw
Andrew MacIntyre
Skip Montanaro
Neil Schemenauer
Marc-André Lemburg
Fred Drake
Sjoerd Mullender