Two different but related problems:
1. PySymtable_Free() must explicitly DECREF(st->st_cur), which should
always point to the global symtable entry. This entry is setup by the
first enter_scope() call, but there is never a corresponding
exit_scope() call.
Since each entry has a reference to scopes defined within it, the
missing DECREF caused all symtable entries to be leaked.
2. The leak here masked a separate problem with
PySymtableEntry_New(). When the requested entry was found in
st->st_symbols, the entry was returned without doing an INCREF.
And problem c) The ste_children slot was getting two copies of each
child entry, because it was populating the slot on the first and
second passes. Now only populate on the first pass.
the symbol table pass. These blocks were already ignored by the code
gen pass. Both passes must visit the same set of blocks in the same
order.
Fixes SF buf 132820
They're actually complaining about something more specific, an assignment
in a lambda as an actual argument, so that Python parses the
lambda as if it were a keyword argument. Like f(lambda x: x[0]=42).
The "lambda x: x[0]" part gets parsed as if it were a keyword, being
bound to 42, and the resulting error msg didn't make much sense.
of nested functions. Either is allowed in a function if it contains
no defs or lambdas or the defs and lambdas it contains have no free
variables. If a function is itself nested and has free variables,
either is illegal.
Revise the symtable to use a PySymtableEntryObject, which holds all
the revelent information for a scope, rather than using a bunch of
st_cur_XXX pointers in the symtable struct. The changes simplify the
internal management of the current symtable scope and of the stack.
Added new C source file: Python/symtable.c. (Does the Windows build
process need to be updated?)
As part of these changes, the initial _symtable module interface
introduced in 2.1a2 is replaced. A dictionary of
PySymtableEntryObjects are returned.
symtable.h, so that they can be used by external module.
Improve error handling in symtable_enter_scope(), which return an
error code that went unchecked by most callers. XXX The error handling
in symtable code is sloppy in general.
Modify symtable to record the line number that begins each scope.
This can help to identify which code block is being referred to when
multiple blocks are bound to the same name.
Add st_scopes dict that is used to preserve scope info when
PyNode_CompileSymtable() is called. Otherwise, this information is
tossed as soon as it is no longer needed.
Add Py_SymtableString() to pythonrun; analogous to Py_CompileString().
discussion on python-dev. 'from mod import *' is still banned except
at the module level.
Fix value for special NOOPT entry in symtable. Initialze to 0 instead
of None, so that later uses of PyInt_AS_LONG() are valid. (Bug
reported by Donn Cave.)
replace local REPR macros with PyObject_REPR in object.h
reference manual but not checked: Names bound by import statemants may
not occur in global statements in the same scope. The from ... import *
form may only occur in a module scope.
I guess these changes could break code, but the reference manual
warned about them.
Several other small changes
If a variable is declared global in the nearest enclosing scope of a
free variable, then treat it is a global in the nested scope too.
Get rid of com_mangle and symtable_mangle functions and call mangle
directly.
If errors occur during symtable table creation, return -1 from
symtable_build().
Do not increment st_errors in assignment to lambda, because exception
is not set.
Add extra argument to symtable_assign(); the argument, flag, is ORed
with DEF_LOCAL for each symtable_add_def() call.
The majority of the changes are in the compiler. The mainloop changes
primarily to implement the new opcodes and to pass a function's
closure to eval_code2(). Frames and functions got new slots to hold
the closure.
Include/compile.h
Add co_freevars and co_cellvars slots to code objects.
Update PyCode_New() to take freevars and cellvars as arguments
Include/funcobject.h
Add func_closure slot to function objects.
Add GetClosure()/SetClosure() functions (and corresponding
macros) for getting at the closure.
Include/frameobject.h
PyFrame_New() now takes a closure.
Include/opcode.h
Add four new opcodes: MAKE_CLOSURE, LOAD_CLOSURE, LOAD_DEREF,
STORE_DEREF.
Remove comment about old requirement for opcodes to fit in 7
bits.
compile.c
Implement changes to code objects for co_freevars and co_cellvars.
Modify symbol table to use st_cur_name (string object for the name
of the current scope) and st_cur_children (list of nested blocks).
Also define st_nested, which might more properly be called
st_cur_nested. Add several DEF_XXX flags to track def-use
information for free variables.
New or modified functions of note:
com_make_closure(struct compiling *, PyCodeObject *)
Emit LOAD_CLOSURE opcodes as needed to pass cells for free
variables into nested scope.
com_addop_varname(struct compiling *, int, char *)
Emits opcodes for LOAD_DEREF and STORE_DEREF.
get_ref_type(struct compiling *, char *name)
Return NAME_CLOSURE if ref type is FREE or CELL
symtable_load_symbols(struct compiling *)
Decides what variables are cell or free based on def-use info.
Can now raise SyntaxError if nested scopes are mixed with
exec or from blah import *.
make_scope_info(PyObject *, PyObject *, int, int)
Helper functions for symtable scope stack.
symtable_update_free_vars(struct symtable *)
After a code block has been analyzed, it must check each of
its children for free variables that are not defined in the
block. If a variable is free in a child and not defined in
the parent, then it is defined by block the enclosing the
current one or it is a global. This does the right logic.
symtable_add_use() is now a macro for symtable_add_def()
symtable_assign(struct symtable *, node *)
Use goto instead of for (;;)
Fixed bug in symtable where name of keyword argument in function
call was treated as assignment in the scope of the call site. Ex:
def f():
g(a=2) # a was considered a local of f
ceval.c
eval_code2() now take one more argument, a closure.
Implement LOAD_CLOSURE, LOAD_DEREF, STORE_DEREF, MAKE_CLOSURE>
Also: When name error occurs for global variable, report that the
name was global in the error mesage.
Objects/frameobject.c
Initialize f_closure to be a tuple containing space for cellvars
and freevars. f_closure is NULL if neither are present.
Objects/funcobject.c
Add support for func_closure.
Python/import.c
Change the magic number.
Python/marshal.c
Track changes to code objects.
parameters that contained both anonymous tuples and *arg or **arg. Ex:
def f(a, (b, c), *d): pass
Fix the symtable_params() to generate names in the right order for
co_varnames slot of code object. Consider *arg and **arg before the
"complex" names introduced by anonymous tuples.
symbol table for each top-level compilation unit. The information in
the symbol table allows the elimination of the later optimize() pass;
the bytecode generation emits the correct opcodes.
The current version passes the complete regression test, but may still
contain some bugs. It's a fairly substantial revision. The current
code adds an assert() and a test that may lead to a Py_FatalError().
I expect to remove these before 2.1 beta 1.
The symbol table (struct symtable) is described in comments in the
code.
The changes affects the several com_XXX() functions that were used to
emit LOAD_NAME and its ilk. The primary interface for this bytecode
is now com_addop_varname() which takes a kind and a name, where kind
is one of VAR_LOAD, VAR_STORE, or VAR_DELETE.
There are many other smaller changes:
- The name mangling code is no longer contained in ifdefs. There are
two functions that expose the mangling logical: com_mangle() and
symtable_mangle().
- The com_error() function can accept NULL for its first argument;
this is useful with is_constant_false() is called during symbol
table generation.
- The loop index names used by list comprehensions have been changed
from __1__ to [1], so that they can not be accessed by Python code.
- in com_funcdef(), com_argdefs() is now called before the body of the
function is compiled. This provides consistency with com_lambdef()
and symtable_funcdef().
- Helpers do_pad(), dump(), and DUMP() are added to aid in debugging
the compiler.
"..." in "from M import ..." was never DECREFed. Leak reported by
James Slaughter and nailed by Barry, who also provided an earlier
version of this patch.
the bug report (for details, look at it), but agree there's no need for Python
to declare atof itself: we #include stdlib.h, and ANSI C sez atof is declared
there already.
Add definitions of INT_MAX and LONG_MAX to pyport.h.
Remove includes of limits.h and conditional definitions of INT_MAX
and LONG_MAX elsewhere.
This closes SourceForge patch #101659 and bug #115323.
how 'import' was called with a compiletime mechanism: create either a tuple
of the import arguments, or None (in the case of a normal import), add it to
the code-block constants, and load it onto the stack before calling
IMPORT_NAME.
(This fix is a bit broken, just as the test already was: the test for
testlist and listmaker are done always, whereas the test for exprlist and
the actual abort() are only done if Py_DEBUG is defined. Suggestions
welcome, I guess ;)
Add the EXTENDED_ARG opcode to the virtual machine, allowing 32-bit
arguments to opcodes instead of being forced to stick to the 16-bit
limit. This is especially useful for machine-generated code, which
can be too long for the SET_LINENO parameter to fit into 16 bits.
This closes the implementation portion of SourceForge patch #100893.
load mod.submod as m, or mod as m ? Both can be achieved differently, and
unambiguously. Also attempt to document this restriction (editor
appreciated!)
Note that this is an artificial check during compile, because incorporating
this in the grammar is hard, and then adjusting the compiler to do the right
thing with the right nodes is harder.
name as n'. By doing some twists and turns, "as" is not a reserved word.
There is a slight change in semantics for 'from module import name' (it will
now honour the 'global' keyword) but only in cases that are explicitly
undocumented.
in binascii.c (only on platforms with signed chars -- although Py_CHARMASK
is documented as returning an int, it only does so on platforms with
signed chars).
filename and lineno attributes, but do not mask the SyntaxError if we
fail.
This is part of what is needed to close SoruceForge bug #110628
(Jitterbug PR#278).
Wrap a long line to fit in under 80 columns.
did the same anyway.
I'm not sure what to do with Tools/compiler/compiler/* -- that isn't part of
distutils, is it ? Should it try to be compatible with old bytecode version ?
The common technique for printing out a pointer has been to cast to a long
and use the "%lx" printf modifier. This is incorrect on Win64 where casting
to a long truncates the pointer. The "%p" formatter should be used instead.
The problem as stated by Tim:
> Unfortunately, the C committee refused to define what %p conversion "looks
> like" -- they explicitly allowed it to be implementation-defined. Older
> versions of Microsoft C even stuck a colon in the middle of the address (in
> the days of segment+offset addressing)!
The result is that the hex value of a pointer will maybe/maybe not have a 0x
prepended to it.
Notes on the patch:
There are two main classes of changes:
- in the various repr() functions that print out pointers
- debugging printf's in the various thread_*.h files (these are why the
patch is large)
Closes SourceForge patch #100505.
For more comments, read the patches@python.org archives.
For documentation read the comments in mymalloc.h and objimpl.h.
(This is not exactly what Vladimir posted to the patches list; I've
made a few changes, and Vladimir sent me a fix in private email for a
problem that only occurs in debug mode. I'm also holding back on his
change to main.c, which seems unnecessary to me.)
Support for the new -U command line option option:
with the option enabled the Python compiler
interprets all "..." strings as u"..." (same with r"..." and
ur"...").
Follow a suggestion in an /*XXX*/ comment [in com_add()] to speed up
compilation by using supplemental dictionaries to keep track of names
and constants, eliminating quadratic behavior. With this patch in
place, the time to import a 5000-line file with lots of constants [at
the global level] is reduced from 20 seconds to under 3 on my system.
comparing code objects. This give sless surprising results in
-Optimized code. It also sorts code objects by name, now.
[I changed the patch to hash() slightly to touch fewer lines.]
executive summary:
Instead of typing 'apply(f, args, kwargs)' you can type 'f(*arg, **kwargs)'.
Some file-by-file details follow.
Grammar/Grammar:
simplify varargslist, replacing '*' '*' with '**'
add * & ** options to arglist
Include/opcode.h & Lib/dis.py:
define three new opcodes
CALL_FUNCTION_VAR
CALL_FUNCTION_KW
CALL_FUNCTION_VAR_KW
Python/ceval.c:
extend TypeError "keyword parameter redefined" message to include
the name of the offending keyword
reindent CALL_FUNCTION using four spaces
add handling of sequences and dictionaries using extend calls
fix function import_from to use PyErr_Format
For a long time I've seen absurd tracebacks under -O (e.g., negative
line numbers), but very rarely. Since I was looking at tracebacks
anyway, thought I'd track it down. Turns out to be Guido's only
predictable blind spot <wink -- "char" is signed on some non-GvR
systems>. Patch follows.
happen when you use a non-keyword argument after a keyword argument,
and in this case you also get a syntax error. I fully suspect that
the underflow is caused by the code that stops generating code when it
detects the syntax error, but I can't find the culprit right now. I
know, I know.)
have a unique name, otherwise they get squished by locals2fast (or
fast2locals, I dunno) when the debugger is invoked before they have
been transferred to real locals.
recognized by the code generator and code generation for the test and
the subsequent suite is suppressed.
One must write *exactly* ``if __debug__:'' or ``elif __debug__:'' --
no parentheses or operators must be present, or the optimization is
not carried through. Whitespace doesn't matter. Other uses of
__debug__ will find __debug__ defined as 0 or 1 in the __builtin__
module.
table which is incorporated in the code object. This way, the runtime
overhead to keep track of line numbers is only incurred when an
exception has to be reported.
to PyCode_New() argument list. Move MAXBLOCKS constant to conpile.h.
Added accurate calculation of the actual stack size needed by the
generated code.
Also commented out all fprintf statements (except for a new one to
diagnose stack underflow, and one in #ifdef'ed out code), and added
some new TO DO suggestions (now that the stacksize is taken of the TO
DO list).
be Ellipsis!).
Bumped the API version because a linker-visible symbol is affected.
Old C code will still compile -- there's a b/w compat macro.
Similarly, old Python code will still run, builtin exports both
Ellipses and Ellipsis.
bltinmodule.c: fixed coerce() nightmare in ternary pow().
modsupport.c (initmodule2): pass METH_FREENAME flag to newmethodobject().
pythonrun.c: move flushline() into and around print_error().
* funcobject.c (func_repr): don't call getstringvalue(None) for anonymous
functions.
* bltinmodule.c: removed lambda (which is now a built-in function);
removed implied lambda for string arg to filter/map/reduce.
* Grammar, graminit.[ch], compile.[ch]: replaced lambda as built-in
function by lambda as grammar entity: instead of "lambda('x: x+1')" you
write "lambda x: x+1".
* Xtmodule.c (checkargdict): return 0, not NULL, for error.
* {tuple,list,mapping,array}object.c: call printobject with 0 for flags
* compile.c (parsestr): use quote instead of '\'' at one crucial point
* arraymodule.c (array_getattr): Added __members__ attribute
* PROTO.h, mymalloc.h: added #ifdefs for TURBOC and GNUC.
* allobjects.h: added #include "rangeobject.h"
* Grammar: added lambda_input; relaxed syntax for exec.
* bltinmodule.c: added bagof, map, reduce, lambda, xrange.
* tupleobject.[ch]: added resizetuple().
* rangeobject.[ch]: new object type to speed up range operations (not
convinced this is needed!!!)
* Grammar: add exec statement; allow testlist in expr statement.
* ceval.c, compile.c, opcode.h: support exec statement;
avoid optimizing locals when it is used
* fileobject.{c,h}: add getfilename() internal function.
* many files: made some functions static; removed "extern int errno;".
* frozenmain.c: fixed bugs introduced on 24 June...
* flmodule.c: remove 1.5 bw compat hacks, add new functions in 2.2a
(and some old functions that were omitted).
* timemodule.c: added MSDOS floatsleep version .
* pgenmain.c: changed exit() to goaway() and added defn of goaway().
* intrcheck.c: add hack (to UNIX only) so interrupting 3 times
will exit from a hanging program. The second interrupt prints
a message explaining this to the user.
yet). The class is now passed to eval_code and stored in the current
frame. It is also stored in instance method objects. An "unbound"
instance method is now returned when a function is retrieved through
"classname.funcname", which when called passes the class to eval_code.
(1) dictionaries/mappings now have attributes values() and items() as
well as keys(); at the C level, use the new function mappinggetnext()
to iterate over a dictionary.
(2) "class C(): ..." is now illegal; you must write "class C: ...".
(3) Class objects now know their own name (finally!); and minor
improvements to the way how classes, functions and methods are
represented as strings.
(4) Added an "access" statement and semantics. (This is still
experimental -- as long as you don't use the keyword 'access' nothing
should be changed.)
lookup (opcode.h, ceval.[ch], compile.c, frameobject.[ch],
pythonrun.c, import.c). The .pyc MAGIC number is changed again.
Added get_menu_text to flmodule.
* Stubs for faster implementation of local variables (not yet finished)
* Added function name to code object. Print it for code and function
objects. THIS MAKES THE .PYC FILE FORMAT INCOMPATIBLE (the version
number has changed accordingly)
* Print address of self for built-in methods
* New internal functions getattro and setattro (getattr/setattr with
string object arg)
* Replaced "dictobject" with more powerful "mappingobject"
* New per-type functio tp_hash to implement arbitrary object hashing,
and hashobject() to interface to it
* Added built-in functions hash(v) and hasattr(v, 'name')
* classobject: made some functions static that accidentally weren't;
added __hash__ special instance method to implement hash()
* Added proper comparison for built-in methods and functions
Jeremy Hylton
Tim Peters
Guido van Rossum
Barry Warsaw
Fred Drake
Fredrik Lundh
Thomas Wouters
Skip Montanaro
Peter Schneider-Kamp
Jack Jansen