Fix based on patch #414750 by Michael Hudson.
New functions get_func_name() and get_func_desc() return reasonable
names and descriptions for all objects. XXX Even objects that aren't
actually callable.
If a module has a future statement enabling nested scopes, they are
also enable for the exec statement and the functions compile() and
execfile() if they occur in the module.
If Python is run with the -i option, which enters interactive mode
after executing a script, and the script it runs enables nested
scopes, they are also enabled in interactive mode.
XXX The use of -i with -c "from __future__ import nested_scopes" is
not supported. What's the point?
To support these changes, many function variants have been added to
pythonrun.c. All the variants names end with Flags and they take an
extra PyCompilerFlags * argument. It is possible that this complexity
will be eliminated in a future version of the interpreter in which
nested scopes are not optional.
frees. Note there doesn't seem to be any way to test LocalsToFast(),
because the instructions that trigger it are illegal in nested scopes
with free variables.
Fix allocation strategy for cells that are also formal parameters.
Instead of emitting LOAD_FAST / STORE_DEREF pairs for each parameter,
have the argument handling code in eval_code2() do the right thing.
A side-effect of this change is that cell variables that are also
arguments are listed at the front of co_cellvars in the order they
appear in the argument list.
with free variables. Thanks to Martin v. Loewis for finding two of
the problems. This fixes SF buf 405583.
There is also a C API change: PyFrame_New() is reverting to its
pre-2.1 signature. The change introduced by nested scopes was a
mistake. XXX Is this okay between beta releases?
cell_clear(), the GC helper, must decref its reference to break
cycles.
frame_dealloc() must dealloc all cell vars and free vars in addition
to locals.
eval_code2() setup code must INCREF cells it copies out of the
closure.
The STORE_DEREF opcode implementation must DECREF the object it passes
to PyCell_Set().
create an empty dictionary if it is called without keyword args. Just
pass NULL.
XXX I had believed that this caused weird errors, but the test suite
runs cleanly.
discussion on python-dev. 'from mod import *' is still banned except
at the module level.
Fix value for special NOOPT entry in symtable. Initialze to 0 instead
of None, so that later uses of PyInt_AS_LONG() are valid. (Bug
reported by Donn Cave.)
replace local REPR macros with PyObject_REPR in object.h
This change eliminates an extra malloc/free when a frame with free
variables is created. Any cell vars or free vars are stored in
f_localsplus after the locals and before the stack.
eval_code2() fills in the appropriate values after handling
initialization of locals.
To track the size the frame has an f_size member that tracks the total
size of f_localsplus. It used to be implicitly f_nlocals + f_stacksize.
The majority of the changes are in the compiler. The mainloop changes
primarily to implement the new opcodes and to pass a function's
closure to eval_code2(). Frames and functions got new slots to hold
the closure.
Include/compile.h
Add co_freevars and co_cellvars slots to code objects.
Update PyCode_New() to take freevars and cellvars as arguments
Include/funcobject.h
Add func_closure slot to function objects.
Add GetClosure()/SetClosure() functions (and corresponding
macros) for getting at the closure.
Include/frameobject.h
PyFrame_New() now takes a closure.
Include/opcode.h
Add four new opcodes: MAKE_CLOSURE, LOAD_CLOSURE, LOAD_DEREF,
STORE_DEREF.
Remove comment about old requirement for opcodes to fit in 7
bits.
compile.c
Implement changes to code objects for co_freevars and co_cellvars.
Modify symbol table to use st_cur_name (string object for the name
of the current scope) and st_cur_children (list of nested blocks).
Also define st_nested, which might more properly be called
st_cur_nested. Add several DEF_XXX flags to track def-use
information for free variables.
New or modified functions of note:
com_make_closure(struct compiling *, PyCodeObject *)
Emit LOAD_CLOSURE opcodes as needed to pass cells for free
variables into nested scope.
com_addop_varname(struct compiling *, int, char *)
Emits opcodes for LOAD_DEREF and STORE_DEREF.
get_ref_type(struct compiling *, char *name)
Return NAME_CLOSURE if ref type is FREE or CELL
symtable_load_symbols(struct compiling *)
Decides what variables are cell or free based on def-use info.
Can now raise SyntaxError if nested scopes are mixed with
exec or from blah import *.
make_scope_info(PyObject *, PyObject *, int, int)
Helper functions for symtable scope stack.
symtable_update_free_vars(struct symtable *)
After a code block has been analyzed, it must check each of
its children for free variables that are not defined in the
block. If a variable is free in a child and not defined in
the parent, then it is defined by block the enclosing the
current one or it is a global. This does the right logic.
symtable_add_use() is now a macro for symtable_add_def()
symtable_assign(struct symtable *, node *)
Use goto instead of for (;;)
Fixed bug in symtable where name of keyword argument in function
call was treated as assignment in the scope of the call site. Ex:
def f():
g(a=2) # a was considered a local of f
ceval.c
eval_code2() now take one more argument, a closure.
Implement LOAD_CLOSURE, LOAD_DEREF, STORE_DEREF, MAKE_CLOSURE>
Also: When name error occurs for global variable, report that the
name was global in the error mesage.
Objects/frameobject.c
Initialize f_closure to be a tuple containing space for cellvars
and freevars. f_closure is NULL if neither are present.
Objects/funcobject.c
Add support for func_closure.
Python/import.c
Change the magic number.
Python/marshal.c
Track changes to code objects.
- Use PyObject_RichCompare*() where possible: when comparing
keyword arguments, in _PyEval_SliceIndex(), and of course in
cmp_outcome().
Unrelated stuff:
- Removed all trailing whitespace.
- Folded some long lines.
message, and tries to make the messages more consistent and helpful when
the wrong number of arguments or duplicate keyword arguments are supplied.
Comes with more tests for test_extcall.py and and an update to an error
message in test/output/test_pyexpat.
1) "from M import X" now works even if M is not a real module; it's
basically a getattr() operation with AttributeError exceptions
changed into ImportError.
2) "from M import *" now looks for M.__all__ to decide which names to
import; if M.__all__ doesn't exist, it uses M.__dict__.keys() but
filters out names starting with '_' as before. Whether or not
__all__ exists, there's no restriction on the type of M.
an empty keywords dictionary (via apply() or the extended call syntax),
the keywords dict should be ignored. If the keywords dict is not empty,
TypeError should be raised. (Between the restructuring of the call
machinery and this patch, an empty dict in this situation would trigger
a SystemError via PyErr_BadInternalCall().)
Added regression tests to detect errors for this.
More revision still needed.
Much of the code that was in the mainloop was moved to a series of
helper functions. PyEval_CallObjectWithKeywords was split into two
parts. The first part now only does argument handling. The second
part is now named call_object and delegates the call to a
call_(function,method,etc.) helper.
XXX The call_XXX helper functions should be replaced with tp_call
functions for the respective types.
The CALL_FUNCTION implementation contains three kinds of optimization:
1. fast_cfunction and fast_function are called when the arguments on
the stack can be passed directly to eval_code2() without copying
them into a tuple.
2. PyCFunction objects are dispatched immediately, because they are
presumed to occur more often than anything else.
3. Bound methods are dispatched inline. The method object contains a
pointer to the function object that will be called. The function
is called from within the mainloop, which may allow optimization #1
to be used, too.
The extened call implementation -- f(*args) and f(**kw) -- are
implemented as a separate case in the mainloop. This allows the
common case of normal function calls to execute without wasting time
on checks for extended calls, although it does introduce a small
amount of code duplication.
Also, the unused final argument of eval_code2() was removed. This is
probably the last trace of the access statement :-).
When a method is called with no regular arguments and * args, defer
the first arg is subclass check until after the * args have been
expanded.
N.B. The CALL_FUNCTION implementation is getting really hairy; should
review it to see if it can be simplified.
by making the DUP_TOPX code utterly straightforward. This also gets rid
of all normal-case internal DUP_TOPX if/branches, and allows replacing one
POP() with TOP() in each case, so is a good idea regardless.
Add definitions of INT_MAX and LONG_MAX to pyport.h.
Remove includes of limits.h and conditional definitions of INT_MAX
and LONG_MAX elsewhere.
This closes SourceForge patch #101659 and bug #115323.
which implements the automatic conversion from Unicode to a string
object using the default encoding.
The new API is then put to use to have eval() and exec accept
Unicode objects as code parameter. This closes bugs #110924
and #113890.
As side-effect, the traditional C APIs PyString_Size() and
PyString_AsString() will also accept Unicode objects as
parameters.
ceval.c:
define recurion_limit (static), default value is 2500
define Py_GetRecursionLimit and Py_SetRecursionLimit
raise RuntimeError if limit is exceeded
PC/config.h:
remove plat-specific definition
sysmodule.c:
add sys.(get|set)recursionlimit
how 'import' was called with a compiletime mechanism: create either a tuple
of the import arguments, or None (in the case of a normal import), add it to
the code-block constants, and load it onto the stack before calling
IMPORT_NAME.
Add the EXTENDED_ARG opcode to the virtual machine, allowing 32-bit
arguments to opcodes instead of being forced to stick to the 16-bit
limit. This is especially useful for machine-generated code, which
can be too long for the SET_LINENO parameter to fit into 16 bits.
This closes the implementation portion of SourceForge patch #100893.
eval_code2(): Implement new bytecodes PRINT_ITEM_TO and
PRINT_NEWLINE_TO, as per accepted SF patch #100970.
Also update graminit.c based on related Grammar/Grammar changes.
trying hard enough to find out what the arguments to an import were. There
is no test-case for this bug, yet, but this is what it looked like:
from encodings import cp1006, cp1026
ImportError: cannot import name cp1026
'__import__' was called with only the first name in the 'arguments' list.
name as n'. By doing some twists and turns, "as" is not a reserved word.
There is a slight change in semantics for 'from module import name' (it will
now honour the 'global' keyword) but only in cases that are explicitly
undocumented.
did the same anyway.
I'm not sure what to do with Tools/compiler/compiler/* -- that isn't part of
distutils, is it ? Should it try to be compatible with old bytecode version ?
comments, docstrings or error messages. I fixed two minor things in
test_winreg.py ("didn't" -> "Didn't" and "Didnt" -> "Didn't").
There is a minor style issue involved: Guido seems to have preferred English
grammar (behaviour, honour) in a couple places. This patch changes that to
American, which is the more prominent style in the source. I prefer English
myself, so if English is preferred, I'd be happy to supply a patch myself ;)
For more comments, read the patches@python.org archives.
For documentation read the comments in mymalloc.h and objimpl.h.
(This is not exactly what Vladimir posted to the patches list; I've
made a few changes, and Vladimir sent me a fix in private email for a
problem that only occurs in debug mode. I'm also holding back on his
change to main.c, which seems unnecessary to me.)
"""
Running "test_extcall" repeatedly results in memory leaks.
One of these can't be fixed (at least not easily!), it happens since
this code:
def saboteur(**kw):
kw['x'] = locals()
d = {}
saboteur(a=1, **d)
creates a circular reference - d['x']['d']==d
The others are due to some missing decrefs in ceval.c, fixed by the
patch attached below.
Note: I originally wrote this without the "goto", just adding the
missing decref's where needed. But I think the goto is justified in
keeping the executable code size of ceval as small as possible.
"""
[I think the circular reference is more like kw['x']['kw'] == kw. --GvR]
If a non-tuple sequence is passed as the *arg, convert it to a tuple
before checking its length.
If named keyword arguments are used in combination with **kwargs, make
a copy of kwargs before inserting the new keys.
the return value of PySequence_Length(). If an exception occurred,
the returned length will be -1. Make sure this doesn't get obscurred,
and that the bogus length isn't used.
executive summary:
Instead of typing 'apply(f, args, kwargs)' you can type 'f(*arg, **kwargs)'.
Some file-by-file details follow.
Grammar/Grammar:
simplify varargslist, replacing '*' '*' with '**'
add * & ** options to arglist
Include/opcode.h & Lib/dis.py:
define three new opcodes
CALL_FUNCTION_VAR
CALL_FUNCTION_KW
CALL_FUNCTION_VAR_KW
Python/ceval.c:
extend TypeError "keyword parameter redefined" message to include
the name of the offending keyword
reindent CALL_FUNCTION using four spaces
add handling of sequences and dictionaries using extend calls
fix function import_from to use PyErr_Format
PR#175 -- when exec is passed a code object, it didn't sync the locals
from the dictionary back into their fast representation.
Also took the time to remove some repetitive code there and to do the
syncing even when an exception is raised (since a partial effect
should still be synced).
not as descriptive as what Barry suggests, but this also catches the
(in my opinion important) case where some other C code besides apply()
constructs a kwdict that doesn't have the right format. All the other
possibilities of getting it wrong (non-dict, wrong keywords etc) are
already caught so this makes sense to check here.
tracefunc (or profilefunc -- we're not sure which), zap the global
trace and profile funcs so that we can't get into recursive loop when
instantiating the resulting class based exception.
Introduce a new builtin exception, UnboundLocalError, raised when ceval.c
tries to retrieve or delete a local name that isn't bound to a value.
Currently raises NameError, which makes this behavior a FAQ since the same
error is raised for "missing" global names too: when the user has a global
of the same name as the unbound local, NameError makes no sense to them.
Even in the absence of shadowing, knowing whether a bogus name is local or
global is a real aid to quick understanding.
Example:
D:\src\PCbuild>type local.py
x = 42
def f():
print x
x = 13
return x
f()
D:\src\PCbuild>python local.py
Traceback (innermost last):
File "local.py", line 8, in ?
f()
File "local.py", line 4, in f
print x
UnboundLocalError: x
D:\src\PCbuild>
Note that UnboundLocalError is a subclass of NameError, for compatibility
with existing class-exception code that may be trying to catch this as a
NameError. Unfortunately, I see no way to make this wholly compatible
with -X (see comments in bltinmodule.c): under -X, [UnboundLocalError
is an alias for NameError --GvR].
[The ceval.c patch differs slightly from the second version that Tim
submitted; I decided not to raise UnboundLocalError for DELETE_NAME,
only for DELETE_LOCAL. DELETE_NAME is only generated at the module
level, and since at that level a NameError is raised for referencing
an undefined name, it should also be raised for deleting one.]
swapped arguments].
Also make sure that no use of a function pointer gotten from a
tp_as_sequence or tp_as_mapping structure is made without checking it
for NULL first.
tstate swapping. Only the acquiring and releasing of the lock is
conditional (twice, under ``#ifdef WITH_THREAD'' and inside ``if
(interpreter_lock)'').
This doesn't yet support "import a.b.c" or "from a.b.c import x", but
it does recognize directories. When importing a directory, it
initializes __path__ to a list containing the directory name, and
loads the __init__ module if found.
The (internal) find_module() and load_module() functions are
restructured so that they both also handle built-in and frozen modules
and Mac resources (and directories of course). The imp module's
find_module() and (new) load_module() also have this functionality.
Moreover, imp unconditionally defines constants for all module types,
and has two more new functions: find_module_in_package() and
find_module_in_directory().
There's also a new API function, PyImport_ImportModuleEx(), which
takes all four __import__ arguments (name, globals, locals, fromlist).
The last three may be NULL. This is currently the same as
PyImport_ImportModule() but in the future it will be able to do
relative dotted-path imports.
Other changes:
- bltinmodule.c: in __import__, call PyImport_ImportModuleEx().
- ceval.c: always pass the fromlist to __import__, even if it is a C
function, so PyImport_ImportModuleEx() is useful.
- getmtime.c: the function has a second argument, the FILE*, on which
it applies fstat(). According to Sjoerd this is much faster. The
first (pathname) argument is ignored, but remains for backward
compatibility (so the Mac version still works without changes).
By cleverly combining the new imp functionality, the full support for
dotted names in Python (mini.py, not checked in) is now about 7K,
lavishly commented (vs. 14K for ni plus 11K for ihooks, also lavishly
commented).
Good night!
former rather than the latter, since PyErr_NormalizeException takes
PyObject** and I didn't want to change the interface for set_exc_info
(but I did want the changes propagated to eval_code2!).
UNPACK_LIST byte codes and added a third code path that allows
generalized sequence unpacking. Now both syntaxes:
a, b, c = seq
[a, b, c] = seq
can be used to unpack any sequence with the exact right number of
items.
unpack_sequence(): out-lined implementation of generalized sequence
unpacking. tuple and list unpacking are still inlined.
PyErr_GivenExceptionMatches().
set_exc_info(): make sure to normalize exceptions.
do_raise(): Use PyErr_NormalizeException() if type is a class.
loop_subscript(): Use PyErr_ExceptionMatches() instead of raw pointer
compare for PyExc_IndexError.
PyThreadState pointer instead of a (frame) PyObject pointer. This
makes much more sense. It is backward incompatible, but that's no
problem, because (a) the heaviest users are the Py_{BEGIN,END}_
ALLOW_THREADS macros here, which have been fixed too; (b) there are
very few direct users; (c) those who use it are there will probably
appreciate the change.
Also, added new functions PyEval_AcquireThread() and
PyEval_ReleaseThread() which allows the threads created by the thread
module as well threads created by others (!) to set/reset the current
thread, and at the same time acquire/release the interpreter lock.
Much saner.
int+int, int-int, int <compareop> int, and list[int].
(Unfortunately, int*int is way too much code to inline.)
Also corrected a NULL that should have been a zero.
get/set/del item). This removes a pile of duplication. There's no
abstract operator for 'not' but I removed the function call for it
anyway -- it's a little faster in-line.
by the frameobject dealloc when it is time for the locals to go. When
there's still a traceback object referencing this stack frame, we
don't want the local variables to disappear yet.
(Hmm... Shouldn't they be copied to the f_locals dictionary?)
- Use co->... instead of f->f_code->...; save an extra lookup of what
we already have in a local variable).
- Remove test for nlocals > 0 before setting fastlocals to
f->f_localsplus; 0 is a rare case and the assignment is safe even
then.
called with keyword arguments -- the keyword and value were leaked.
This affected for instance with a __call__() method.
Bug reported and fix supplied by Jim Fulton.
i.e., counting opcode frequencies, or (with DXPAIRS defined) opcode
pair frequencies. Define DYNAMIC_EXECUTION_PROFILE on the command
line (for this file and for sysmodule.c) to enable.
This is safe now that both intrcheck() and signalmodule.c schedule a
sigcheck() call via Py_AddPendingCall().
This gives another 7% speedup (never run such a test twice ;-).
The raise logic has one additional feature: if you raise <class>,
<value> where <value> is not an instance, it will construct an
instance using <value> as argument. If <value> is None, <class> is
instantiated without arguments. If <value> is a tuple, it is used as
the argument list.
This feature is intended to make it easier to upgrade code from using
string exceptions to using class exceptions; without this feature,
you'd have to change every raise statement from ``raise X'' to ``raise
X()'' and from ``raise X, y'' to ``raise X(y)''. The latter is still
the recommended form (because it has no ambiguities about the number
of arguments), but this change makes the transition less painful.
Jeremy Hylton
Guido van Rossum
Thomas Wouters
Tim Peters
Ka-Ping Yee
Moshe Zadka
Charles G. Waldman
Fred Drake
Marc-André Lemburg
Vladimir Marangozov
Paul Prescod
Barry Warsaw
Jack Jansen
Andrew M. Kuchling