Armin Rigo pointed out that the way the line-# table got built didn't work
for lines generating more than 255 bytes of bytecode. Fixed as he
suggested, plus corresponding changes to pyassem.py, plus added some
long overdue docs about this subtle table to compile.c.
Bugfix candidate (line numbers may be off in tracebacks under -O).
Check for free in class and method only if nested scopes are enabled.
Add assertion to verify that no free variables occur when nested
scopes are disabled.
XXX When should nested scopes by made non-optional on the trunk?
The new test case demonstrates the bug. Be more careful in
symtable_resolve_free() to add a var to cells or frees only if it
won't be added under some other rule.
XXX Add new assertion that will catch this bug.
new slot tp_iter in type object, plus new flag Py_TPFLAGS_HAVE_ITER
new C API PyObject_GetIter(), calls tp_iter
new builtin iter(), with two forms: iter(obj), and iter(function, sentinel)
new internal object types iterobject and calliterobject
new exception StopIteration
new opcodes for "for" loops, GET_ITER and FOR_ITER (also supported by dis.py)
new magic number for .pyc files
new special method for instances: __iter__() returns an iterator
iteration over dictionaries: "for x in dict" iterates over the keys
iteration over files: "for x in file" iterates over lines
TODO:
documentation
test suite
decide whether to use a different way to spell iter(function, sentinal)
decide whether "for key in dict" is a good idea
use iterators in map/filter/reduce, min/max, and elsewhere (in/not in?)
speed tuning (make next() a slot tp_next???)
frees. Note there doesn't seem to be any way to test LocalsToFast(),
because the instructions that trigger it are illegal in nested scopes
with free variables.
Fix allocation strategy for cells that are also formal parameters.
Instead of emitting LOAD_FAST / STORE_DEREF pairs for each parameter,
have the argument handling code in eval_code2() do the right thing.
A side-effect of this change is that cell variables that are also
arguments are listed at the front of co_cellvars in the order they
appear in the argument list.
has a binding for the name. The fix is in two places:
- in symtable_update_free_vars, ignore a global stmt in a class scope
- in symtable_load_symbols, add extra handling for names that are
defined at class scope and free in a method
Closes SF bug 407800
Made sure that the warnings issued by symtable_check_unoptimized()
(about import * and exec) contain the proper filename and line number,
and are transformed into SyntaxError exceptions with -Werror.
(Also remove warning about module-level global decl, because we can't
distinguish from code passed to exec.)
Define PyCompilerFlags type contains a single element,
cf_nested_scopes, that is true if a nested scopes future statement has
been entered at the interactive prompt.
New API functions:
PyNode_CompileFlags()
PyRun_InteractiveOneFlags()
-- same as their non Flags counterparts except that the take an
optional PyCompilerFlags pointer
compile.c: In jcompile() use PyCompilerFlags argument. If
cf_nested_scopes is true, compile code with nested scopes. If it
is false, but the code has a valid future nested scopes statement,
set it to true.
pythonrun.c: Create a new PyCompilerFlags object in
PyRun_InteractiveLoop() and thread it through to
PyRun_InteractiveOneFlags().
from __future__ import nested_scopes
x=7
def f():
x=1
def g():
global x
def i():
def h():
return x
return h()
return i()
return g()
print f()
print x
This kind of code didn't work correctly because x was treated as free
in i, leading to an attempt to load x in g to make a closure for i.
Solution is to make global decl apply to nested scopes unless their is
an assignment. Thus, x in h is global.
described in PEP 227.
symtable_check_unoptimized() warns about import * and exec with "in"
when it is used in a function that contains a nested function with
free variables. Warnings are issued unless nested scopes are in
effect, in which case these are SyntaxErrors.
symtable_check_shadow() warns about assignments in a function scope
that shadow free variables defined in a nested scope. This will
always generate a warning -- and will behave differently with nested
scopes than without.
Restore full checking for free vars in children, even when nested
scopes are not enabled. This is needed to support warnings for
shadowing.
Change symtable_warn() to return an int-- the return value of
PyErr_WarnExplicit.
Sundry cleanup: Remove commented out code. Break long lines.
global after assign / use.
Note: I'm not updating the PyErr_Warn() call for import * / exec
combined with a function, because I can't trigger it with an example.
Jeremy, just follow the example of the call to PyErr_WarnExplicit()
that I *did* include.
for errors raised in future.c.
Move some helper functions from compile.c to errors.c and make them
API functions: PyErr_SyntaxLocation() and PyErr_ProgramText().
raised by the compiler.
XXX For now, text entered into the interactive intepreter is not
printed in the traceback.
Inspired by a patch from Roman Sulzhyk
compile.c:
Add helper fetch_program_text() that opens a file and reads until it
finds the specified line number. The code is a near duplicate of
similar code in traceback.c.
Modify com_error() to pass two arguments to SyntaxError constructor,
where the second argument contains the offending text when possible.
Modify set_error_location(), now used only by the symtable pass, to
set the text attribute on existing exceptions.
pythonrun.c:
Change parse_syntax_error() to continue of the offset attribute of a
SyntaxError is None. In this case, it sets offset to -1.
Move code from PyErr_PrintEx() into helper function
print_error_text(). In the helper, only print the caret for a
SyntaxError if offset > 0.
XXX still need to integrate into symtable API
compile.h: Remove ff_n_simple_stmt; obsolete.
Add ff_found_docstring used internally to skip one and only
one string at the beginning of a module.
compile.c: Add check for from __future__ imports to far into the file.
In symtable_global() check for -1 returned from
symtable_lookup(), which signifies name not defined.
Add missing DECERF in symtable_add_def.
Free c->c_future.
future.c: Add special handling for multiple statements joined on a
single line using one or more semicolons; this form can
include an illegal future statement that would otherwise be
hard to detect.
Add support for detecting and skipping doc strings.
Makefile.pre.in: add target future.o
Include/compile.h: define PyFutureFeaters and PyNode_Future()
add c_future slot to struct compiling
Include/symtable.h: add st_future slot to struct symtable
Python/future.c: implementation of PyNode_Future()
Python/compile.c: use PyNode_Future() for nested_scopes support
Python/symtable.c: include compile.h to pick up PyFutureFeatures decl
compile.h: #define NESTED_SCOPES_DEFAULT 0 for Python 2.1
__future__ feature name: "nested_scopes"
symtable.h: Add st_nested_scopes slot. Define flags to track exec and
import star.
Lib/test/test_scope.py: requires nested scopes
compile.c: Fiddle with error messages.
Reverse the sense of ste_optimized flag on
PySymtableEntryObjects. If it is true, there is an optimization
conflict.
Modify get_ref_type to respect st_nested_scopes flags.
Refactor symtable_load_symbols() into several smaller functions,
which use struct symbol_info to share variables. In new function
symtable_update_flags(), raise an error or warning for import * or
bare exec that conflicts with nested scopes. Also, modify handle
for free variables to respect st_nested_scopes flag.
In symtable_init() assign st_nested_scopes flag to
NESTED_SCOPES_DEFAULT (defined in compile.h).
Add preliminary and often incorrect implementation of
symtable_check_future().
Add symtable_lookup() helper for future use.
Two different but related problems:
1. PySymtable_Free() must explicitly DECREF(st->st_cur), which should
always point to the global symtable entry. This entry is setup by the
first enter_scope() call, but there is never a corresponding
exit_scope() call.
Since each entry has a reference to scopes defined within it, the
missing DECREF caused all symtable entries to be leaked.
2. The leak here masked a separate problem with
PySymtableEntry_New(). When the requested entry was found in
st->st_symbols, the entry was returned without doing an INCREF.
And problem c) The ste_children slot was getting two copies of each
child entry, because it was populating the slot on the first and
second passes. Now only populate on the first pass.
the symbol table pass. These blocks were already ignored by the code
gen pass. Both passes must visit the same set of blocks in the same
order.
Fixes SF buf 132820
They're actually complaining about something more specific, an assignment
in a lambda as an actual argument, so that Python parses the
lambda as if it were a keyword argument. Like f(lambda x: x[0]=42).
The "lambda x: x[0]" part gets parsed as if it were a keyword, being
bound to 42, and the resulting error msg didn't make much sense.
of nested functions. Either is allowed in a function if it contains
no defs or lambdas or the defs and lambdas it contains have no free
variables. If a function is itself nested and has free variables,
either is illegal.
Revise the symtable to use a PySymtableEntryObject, which holds all
the revelent information for a scope, rather than using a bunch of
st_cur_XXX pointers in the symtable struct. The changes simplify the
internal management of the current symtable scope and of the stack.
Added new C source file: Python/symtable.c. (Does the Windows build
process need to be updated?)
As part of these changes, the initial _symtable module interface
introduced in 2.1a2 is replaced. A dictionary of
PySymtableEntryObjects are returned.
symtable.h, so that they can be used by external module.
Improve error handling in symtable_enter_scope(), which return an
error code that went unchecked by most callers. XXX The error handling
in symtable code is sloppy in general.
Modify symtable to record the line number that begins each scope.
This can help to identify which code block is being referred to when
multiple blocks are bound to the same name.
Add st_scopes dict that is used to preserve scope info when
PyNode_CompileSymtable() is called. Otherwise, this information is
tossed as soon as it is no longer needed.
Add Py_SymtableString() to pythonrun; analogous to Py_CompileString().
discussion on python-dev. 'from mod import *' is still banned except
at the module level.
Fix value for special NOOPT entry in symtable. Initialze to 0 instead
of None, so that later uses of PyInt_AS_LONG() are valid. (Bug
reported by Donn Cave.)
replace local REPR macros with PyObject_REPR in object.h
reference manual but not checked: Names bound by import statemants may
not occur in global statements in the same scope. The from ... import *
form may only occur in a module scope.
I guess these changes could break code, but the reference manual
warned about them.
Several other small changes
If a variable is declared global in the nearest enclosing scope of a
free variable, then treat it is a global in the nested scope too.
Get rid of com_mangle and symtable_mangle functions and call mangle
directly.
If errors occur during symtable table creation, return -1 from
symtable_build().
Do not increment st_errors in assignment to lambda, because exception
is not set.
Add extra argument to symtable_assign(); the argument, flag, is ORed
with DEF_LOCAL for each symtable_add_def() call.
Tim Peters
Jeremy Hylton
Guido van Rossum
Barry Warsaw