This was started by Mike Bland and completed by Guido
(with help from Neal).
This still needs a __future__ statement added;
Thomas is working on Michael's patch for that aspect.
There's a small amount of code cleanup and refactoring
in ast.c, compile.c and ceval.c (I fixed the lltrace
behavior when EXT_POP is used -- however I had to make
lltrace a static global).
In C++, it's an error to pass a string literal to a char* function
without a const_cast(). Rather than require every C++ extension
module to put a cast around string literals, fix the API to state the
const-ness.
I focused on parts of the API where people usually pass literals:
PyArg_ParseTuple() and friends, Py_BuildValue(), PyMethodDef, the type
slots, etc. Predictably, there were a large set of functions that
needed to be fixed as a result of these changes. The most pervasive
change was to make the keyword args list passed to
PyArg_ParseTupleAndKewords() to be a const char *kwlist[].
One cast was required as a result of the changes: A type object
mallocs the memory for its tp_doc slot and later frees it.
PyTypeObject says that tp_doc is const char *; but if the type was
created by type_new(), we know it is safe to cast to char *.
This change implements a new bytecode compiler, based on a
transformation of the parse tree to an abstract syntax defined in
Parser/Python.asdl.
The compiler implementation is not complete, but it is in stable
enough shape to run the entire test suite excepting two disabled
tests.
[ 1180995 ] binary formats for marshalling floats
Adds 2 new type codes for marshal (binary floats and binary complexes), a
new marshal version (2), updates MAGIC and fiddles the de-serializing of
code objects to be less likely to clobber the real reason for failing if
it fails.
A problem regarding importing symlinked modules was recently reported on the
Cygwin mailing list:
http://cygwin.com/ml/cygwin/2005-04/msg00257.html
The following test case demonstrates the problem:
$ ls -l
total 1
lrwxrwxrwx 1 jt None 6 Apr 23 13:32 bar.py -> foo.py
-rw-r--r-- 1 jt None 24 Apr 18 20:13 foo.py
$ python -c 'import bar'
Traceback (most recent call last):
File "<string>", line 1, in ?
ImportError: No module named bar
Since Cygwin's case_ok() uses a modified version of the Windows's version, the
symlinked bar module actually resolves to file foo.py instead of bar.py. This
obviously causes the matching code to fail (regardless of case).
The patch fixes this problem by making Cygwin use the Mac OS X case_ok()
instead of a modified Window's version.
* Make a pass to eliminate NOPs. Produce code that is more readable,
more compact, and a tiny bit faster. Makes the peepholer more flexible
in the scope of allowable transformations.
* With Guido's okay, bumped up the magic number so that this patch gets
widely exercised before the alpha goes out.
PyImport_ReloadModule(): restore the module to sys.modules in error cases.
load_package(): semantic-neutral refactoring from an earlier stab at
this patch; giving it a common error exit made the code
easier to follow, so retaining that part.
_RemoveModule(): new little utility to delete a key from sys.modules.
The embed2.diff patch solves the user's problem by exporting the missing
symbols from the Python core so Python can be embedded in another Cygwin
application (well, at lest vim).
mostly from SF patch #683257, but I had to change unlock_import() to
return an error value to avoid fatal error.
Should this be backported? The patch requested this, but it's a new
feature.
- new import hooks in import.c, exposed in the sys module
- new module called 'zipimport'
- various changes to allow bootstrapping from zip files
I hope I didn't break the Windows build (or anything else for that
matter), but then again, it's been sitting on sf long enough...
Regarding the latest discussions on python-dev: zipimport sets
pkg.__path__ as specified in PEP 273, and likewise, sys.path item such as
/path/to/Archive.zip/subdir/ are supported again.
[ 587993 ] SET_LINENO killer
Remove SET_LINENO. Tracing is now supported by inspecting co_lnotab.
Many sundry changes to document and adapt to this change.
This patch enhances Python/import.c/find_module() so
that unicode objects found in sys.path will be treated
as legal directory names (The current code ignores
anything that is not a str). The unicode name is
converted to str using Py_FileSystemDefaultEncoding.
get_file() must convert 'U' to "r" PY_STDIOTEXTMODE before calling
fopen().
imp_load_module() must accept 'r' or 'U' or something with '+'.
Also reflow some long lines.
Highlights: import and friends will understand any of \r, \n and \r\n
as end of line. Python file input will do the same if you use mode 'U'.
Everything can be disabled by configuring with --without-universal-newlines.
See PEP278 for details.
Python/
dynload_shlib.c // EMX port emulates dlopen() etc. for DL extensions
import.c // changes to support 8.3 DLL name limit (VACPP+EMX)
// and case sensitive import semantics
importdl.h
thread_os2.h
Fix for the UTF-8 decoder: it will now accept isolated surrogates
(previously it raised an exception which causes round-trips to
fail).
Added new tests for UTF-8 round-trip safety (we rely on UTF-8 for
marshalling Unicode objects, so we better make sure it works for
all Unicode code points, including isolated surrogates).
Bumped the PYC magic in a non-standard way -- please review. This
was needed because the old PYC format used illegal UTF-8 sequences
for isolated high surrogates which now raise an exception.
to call the corresponding methods. This is not a performance improvement
since the times are still swamped by disk I/O, but cleans up the code just
a little.
This fixes the behavior reported by SF bug #404545, where a file
x.y.py could be imported by the statement "import x.y" when there's a
frozen package x (I believe even if x.y also exists as a frozen
module).
exception in the execution of bar, ensure that foo.bar exists.
(Previously, while sys.modules['foo.bar'] would exist, foo.bar would
only be created upon successful execution of bar. This is
inconvenient; some would say wrong. :-)
Probable fix (the bug report doesn't have enough info to say for sure).
find_init_module(): Insist on a case-sensitive match for __init__ files.
Given __INIT__.PY instead, find_init_module() thought that was fine, but
the later attempt to do find_module("__INIT__.PY") didn't and its caller
silently suppressed the resulting ImportError. Now find_init_module()
refuses to accept __INIT__.PY to begin with.
Bugfix candidate; specific to platforms with case-insensitive filesystems.
Directory containing
Spam.py
spam/__init__.py
Then "import Spam" caused a SystemError, because code checking for
the existence of "Spam/__init__.py" finds it on a case-insensitive
filesystem, but then bails because the directory it finds it in
doesn't match case, and then old code assumed that was still an error
even though it isn't anymore. Changed the code to just continue
looking in this case (instead of calling it an error). So
import Spam
and
import spam
both work now.
new slot tp_iter in type object, plus new flag Py_TPFLAGS_HAVE_ITER
new C API PyObject_GetIter(), calls tp_iter
new builtin iter(), with two forms: iter(obj), and iter(function, sentinel)
new internal object types iterobject and calliterobject
new exception StopIteration
new opcodes for "for" loops, GET_ITER and FOR_ITER (also supported by dis.py)
new magic number for .pyc files
new special method for instances: __iter__() returns an iterator
iteration over dictionaries: "for x in dict" iterates over the keys
iteration over files: "for x in file" iterates over lines
TODO:
documentation
test suite
decide whether to use a different way to spell iter(function, sentinal)
decide whether "for key in dict" is a good idea
use iterators in map/filter/reduce, min/max, and elsewhere (in/not in?)
speed tuning (make next() a slot tp_next???)
the more recent versions of that platform, so we use the value (time_t)(-1)
as the error value. This is the type used in the OpenVMS documentation:
http://www.openvms.compaq.com/commercial/c/5763p048.htm#inde
This closes SF tracker bug #404240.
Also clean up an exception message when detecting overflow of time_t values
beyond 4 bytes.
http://python.sourceforge.net/peps/pep-0235.html
Renamed check_case to case_ok. Substantial code rearrangement to get
this stuff in one place in the file. Innermost loop of find_module()
now much simpler and #ifdef-free, and I want to keep it that way (it's
bad enough that the innermost loop is itself still in an #ifdef!).
Windows semantics tested and are fine.
Jason, Cygwin *should* be fine if and only if what you did before "worked"
for case_ok.
Jack, the semantics on your flavor of Mac have definitely changed (see
the PEP), and need to be tested. The intent is that your flavor of Mac
now work the same as everything else in the "lower left" box, including
respecting PYTHONCASEOK.
Steven, sorry, you did the most work here so far but you got screwed the
worst. Happy to work with you on repairing it, but I don't understand
anything about all your Mac variants. We need to add another branch (or
two, three, ...?) inside case_ok. But we should not need to change
anything else.
save the __builtin__ module in a static variable. But this doesn't
work across Py_Finalise()/Py_Initialize()! It also doesn't work when
using multiple interpreter states created with PyInterpreterState_New().
So I'm ripping out this small optimization.
This was probably broken since PyImport_Import() was introduced in
1997! We really need a better test suite for multiple interpreter
states and repeatedly initializing.
This fixes the problems Barry reported in Demo/embed/loop.c.
hooks to take over the Python import machinery at a very early stage
in the Python startup phase.
If there are still places in the Python interpreter which need to
bypass the __import__ hook, these places must now use
PyImport_ImportModuleEx() instead. So far no other places than in
the import mechanism itself have been identified.
They're named as if public, so I did a Bad Thing by changing
PyMarshal_ReadObjectFromFile() to suck up the remainder of the file in one
gulp: anyone who counted on that leaving the file pointer merely at the
end of the next object would be screwed. So restored
PyMarshal_ReadObjectFromFile() to its earlier state, renamed the new greedy
code to PyMarshal_ReadLastObjectFromFile(), and changed Python internals to
call the latter instead.
The majority of the changes are in the compiler. The mainloop changes
primarily to implement the new opcodes and to pass a function's
closure to eval_code2(). Frames and functions got new slots to hold
the closure.
Include/compile.h
Add co_freevars and co_cellvars slots to code objects.
Update PyCode_New() to take freevars and cellvars as arguments
Include/funcobject.h
Add func_closure slot to function objects.
Add GetClosure()/SetClosure() functions (and corresponding
macros) for getting at the closure.
Include/frameobject.h
PyFrame_New() now takes a closure.
Include/opcode.h
Add four new opcodes: MAKE_CLOSURE, LOAD_CLOSURE, LOAD_DEREF,
STORE_DEREF.
Remove comment about old requirement for opcodes to fit in 7
bits.
compile.c
Implement changes to code objects for co_freevars and co_cellvars.
Modify symbol table to use st_cur_name (string object for the name
of the current scope) and st_cur_children (list of nested blocks).
Also define st_nested, which might more properly be called
st_cur_nested. Add several DEF_XXX flags to track def-use
information for free variables.
New or modified functions of note:
com_make_closure(struct compiling *, PyCodeObject *)
Emit LOAD_CLOSURE opcodes as needed to pass cells for free
variables into nested scope.
com_addop_varname(struct compiling *, int, char *)
Emits opcodes for LOAD_DEREF and STORE_DEREF.
get_ref_type(struct compiling *, char *name)
Return NAME_CLOSURE if ref type is FREE or CELL
symtable_load_symbols(struct compiling *)
Decides what variables are cell or free based on def-use info.
Can now raise SyntaxError if nested scopes are mixed with
exec or from blah import *.
make_scope_info(PyObject *, PyObject *, int, int)
Helper functions for symtable scope stack.
symtable_update_free_vars(struct symtable *)
After a code block has been analyzed, it must check each of
its children for free variables that are not defined in the
block. If a variable is free in a child and not defined in
the parent, then it is defined by block the enclosing the
current one or it is a global. This does the right logic.
symtable_add_use() is now a macro for symtable_add_def()
symtable_assign(struct symtable *, node *)
Use goto instead of for (;;)
Fixed bug in symtable where name of keyword argument in function
call was treated as assignment in the scope of the call site. Ex:
def f():
g(a=2) # a was considered a local of f
ceval.c
eval_code2() now take one more argument, a closure.
Implement LOAD_CLOSURE, LOAD_DEREF, STORE_DEREF, MAKE_CLOSURE>
Also: When name error occurs for global variable, report that the
name was global in the error mesage.
Objects/frameobject.c
Initialize f_closure to be a tuple containing space for cellvars
and freevars. f_closure is NULL if neither are present.
Objects/funcobject.c
Add support for func_closure.
Python/import.c
Change the magic number.
Python/marshal.c
Track changes to code objects.
Guido van Rossum
Martin v. Löwis
Neal Norwitz
Jeremy Hylton
Michael W. Hudson
Georg Brandl
Jason Tishler
Raymond Hettinger
Phillip J. Eby
Tim Peters
Thomas Heller
Fred Drake
Armin Rigo
Nicholas Bastin
Jack Jansen
Walter Dörwald
Neil Schemenauer
Just van Rossum
Andrew MacIntyre
Marc-André Lemburg
Greg Ward
Barry Warsaw