A problem regarding importing symlinked modules was recently reported on the
Cygwin mailing list:
http://cygwin.com/ml/cygwin/2005-04/msg00257.html
The following test case demonstrates the problem:
$ ls -l
total 1
lrwxrwxrwx 1 jt None 6 Apr 23 13:32 bar.py -> foo.py
-rw-r--r-- 1 jt None 24 Apr 18 20:13 foo.py
$ python -c 'import bar'
Traceback (most recent call last):
File "<string>", line 1, in ?
ImportError: No module named bar
Since Cygwin's case_ok() uses a modified version of the Windows's version, the
symlinked bar module actually resolves to file foo.py instead of bar.py. This
obviously causes the matching code to fail (regardless of case).
The patch fixes this problem by making Cygwin use the Mac OS X case_ok()
instead of a modified Window's version.
* Make a pass to eliminate NOPs. Produce code that is more readable,
more compact, and a tiny bit faster. Makes the peepholer more flexible
in the scope of allowable transformations.
* With Guido's okay, bumped up the magic number so that this patch gets
widely exercised before the alpha goes out.
PyImport_ReloadModule(): restore the module to sys.modules in error cases.
load_package(): semantic-neutral refactoring from an earlier stab at
this patch; giving it a common error exit made the code
easier to follow, so retaining that part.
_RemoveModule(): new little utility to delete a key from sys.modules.
The embed2.diff patch solves the user's problem by exporting the missing
symbols from the Python core so Python can be embedded in another Cygwin
application (well, at lest vim).
mostly from SF patch #683257, but I had to change unlock_import() to
return an error value to avoid fatal error.
Should this be backported? The patch requested this, but it's a new
feature.
- new import hooks in import.c, exposed in the sys module
- new module called 'zipimport'
- various changes to allow bootstrapping from zip files
I hope I didn't break the Windows build (or anything else for that
matter), but then again, it's been sitting on sf long enough...
Regarding the latest discussions on python-dev: zipimport sets
pkg.__path__ as specified in PEP 273, and likewise, sys.path item such as
/path/to/Archive.zip/subdir/ are supported again.
[ 587993 ] SET_LINENO killer
Remove SET_LINENO. Tracing is now supported by inspecting co_lnotab.
Many sundry changes to document and adapt to this change.
This patch enhances Python/import.c/find_module() so
that unicode objects found in sys.path will be treated
as legal directory names (The current code ignores
anything that is not a str). The unicode name is
converted to str using Py_FileSystemDefaultEncoding.
get_file() must convert 'U' to "r" PY_STDIOTEXTMODE before calling
fopen().
imp_load_module() must accept 'r' or 'U' or something with '+'.
Also reflow some long lines.
Highlights: import and friends will understand any of \r, \n and \r\n
as end of line. Python file input will do the same if you use mode 'U'.
Everything can be disabled by configuring with --without-universal-newlines.
See PEP278 for details.
Python/
dynload_shlib.c // EMX port emulates dlopen() etc. for DL extensions
import.c // changes to support 8.3 DLL name limit (VACPP+EMX)
// and case sensitive import semantics
importdl.h
thread_os2.h
Fix for the UTF-8 decoder: it will now accept isolated surrogates
(previously it raised an exception which causes round-trips to
fail).
Added new tests for UTF-8 round-trip safety (we rely on UTF-8 for
marshalling Unicode objects, so we better make sure it works for
all Unicode code points, including isolated surrogates).
Bumped the PYC magic in a non-standard way -- please review. This
was needed because the old PYC format used illegal UTF-8 sequences
for isolated high surrogates which now raise an exception.
to call the corresponding methods. This is not a performance improvement
since the times are still swamped by disk I/O, but cleans up the code just
a little.
This fixes the behavior reported by SF bug #404545, where a file
x.y.py could be imported by the statement "import x.y" when there's a
frozen package x (I believe even if x.y also exists as a frozen
module).
exception in the execution of bar, ensure that foo.bar exists.
(Previously, while sys.modules['foo.bar'] would exist, foo.bar would
only be created upon successful execution of bar. This is
inconvenient; some would say wrong. :-)
Probable fix (the bug report doesn't have enough info to say for sure).
find_init_module(): Insist on a case-sensitive match for __init__ files.
Given __INIT__.PY instead, find_init_module() thought that was fine, but
the later attempt to do find_module("__INIT__.PY") didn't and its caller
silently suppressed the resulting ImportError. Now find_init_module()
refuses to accept __INIT__.PY to begin with.
Bugfix candidate; specific to platforms with case-insensitive filesystems.
Directory containing
Spam.py
spam/__init__.py
Then "import Spam" caused a SystemError, because code checking for
the existence of "Spam/__init__.py" finds it on a case-insensitive
filesystem, but then bails because the directory it finds it in
doesn't match case, and then old code assumed that was still an error
even though it isn't anymore. Changed the code to just continue
looking in this case (instead of calling it an error). So
import Spam
and
import spam
both work now.
new slot tp_iter in type object, plus new flag Py_TPFLAGS_HAVE_ITER
new C API PyObject_GetIter(), calls tp_iter
new builtin iter(), with two forms: iter(obj), and iter(function, sentinel)
new internal object types iterobject and calliterobject
new exception StopIteration
new opcodes for "for" loops, GET_ITER and FOR_ITER (also supported by dis.py)
new magic number for .pyc files
new special method for instances: __iter__() returns an iterator
iteration over dictionaries: "for x in dict" iterates over the keys
iteration over files: "for x in file" iterates over lines
TODO:
documentation
test suite
decide whether to use a different way to spell iter(function, sentinal)
decide whether "for key in dict" is a good idea
use iterators in map/filter/reduce, min/max, and elsewhere (in/not in?)
speed tuning (make next() a slot tp_next???)
the more recent versions of that platform, so we use the value (time_t)(-1)
as the error value. This is the type used in the OpenVMS documentation:
http://www.openvms.compaq.com/commercial/c/5763p048.htm#inde
This closes SF tracker bug #404240.
Also clean up an exception message when detecting overflow of time_t values
beyond 4 bytes.
http://python.sourceforge.net/peps/pep-0235.html
Renamed check_case to case_ok. Substantial code rearrangement to get
this stuff in one place in the file. Innermost loop of find_module()
now much simpler and #ifdef-free, and I want to keep it that way (it's
bad enough that the innermost loop is itself still in an #ifdef!).
Windows semantics tested and are fine.
Jason, Cygwin *should* be fine if and only if what you did before "worked"
for case_ok.
Jack, the semantics on your flavor of Mac have definitely changed (see
the PEP), and need to be tested. The intent is that your flavor of Mac
now work the same as everything else in the "lower left" box, including
respecting PYTHONCASEOK.
Steven, sorry, you did the most work here so far but you got screwed the
worst. Happy to work with you on repairing it, but I don't understand
anything about all your Mac variants. We need to add another branch (or
two, three, ...?) inside case_ok. But we should not need to change
anything else.
save the __builtin__ module in a static variable. But this doesn't
work across Py_Finalise()/Py_Initialize()! It also doesn't work when
using multiple interpreter states created with PyInterpreterState_New().
So I'm ripping out this small optimization.
This was probably broken since PyImport_Import() was introduced in
1997! We really need a better test suite for multiple interpreter
states and repeatedly initializing.
This fixes the problems Barry reported in Demo/embed/loop.c.
hooks to take over the Python import machinery at a very early stage
in the Python startup phase.
If there are still places in the Python interpreter which need to
bypass the __import__ hook, these places must now use
PyImport_ImportModuleEx() instead. So far no other places than in
the import mechanism itself have been identified.
They're named as if public, so I did a Bad Thing by changing
PyMarshal_ReadObjectFromFile() to suck up the remainder of the file in one
gulp: anyone who counted on that leaving the file pointer merely at the
end of the next object would be screwed. So restored
PyMarshal_ReadObjectFromFile() to its earlier state, renamed the new greedy
code to PyMarshal_ReadLastObjectFromFile(), and changed Python internals to
call the latter instead.
The majority of the changes are in the compiler. The mainloop changes
primarily to implement the new opcodes and to pass a function's
closure to eval_code2(). Frames and functions got new slots to hold
the closure.
Include/compile.h
Add co_freevars and co_cellvars slots to code objects.
Update PyCode_New() to take freevars and cellvars as arguments
Include/funcobject.h
Add func_closure slot to function objects.
Add GetClosure()/SetClosure() functions (and corresponding
macros) for getting at the closure.
Include/frameobject.h
PyFrame_New() now takes a closure.
Include/opcode.h
Add four new opcodes: MAKE_CLOSURE, LOAD_CLOSURE, LOAD_DEREF,
STORE_DEREF.
Remove comment about old requirement for opcodes to fit in 7
bits.
compile.c
Implement changes to code objects for co_freevars and co_cellvars.
Modify symbol table to use st_cur_name (string object for the name
of the current scope) and st_cur_children (list of nested blocks).
Also define st_nested, which might more properly be called
st_cur_nested. Add several DEF_XXX flags to track def-use
information for free variables.
New or modified functions of note:
com_make_closure(struct compiling *, PyCodeObject *)
Emit LOAD_CLOSURE opcodes as needed to pass cells for free
variables into nested scope.
com_addop_varname(struct compiling *, int, char *)
Emits opcodes for LOAD_DEREF and STORE_DEREF.
get_ref_type(struct compiling *, char *name)
Return NAME_CLOSURE if ref type is FREE or CELL
symtable_load_symbols(struct compiling *)
Decides what variables are cell or free based on def-use info.
Can now raise SyntaxError if nested scopes are mixed with
exec or from blah import *.
make_scope_info(PyObject *, PyObject *, int, int)
Helper functions for symtable scope stack.
symtable_update_free_vars(struct symtable *)
After a code block has been analyzed, it must check each of
its children for free variables that are not defined in the
block. If a variable is free in a child and not defined in
the parent, then it is defined by block the enclosing the
current one or it is a global. This does the right logic.
symtable_add_use() is now a macro for symtable_add_def()
symtable_assign(struct symtable *, node *)
Use goto instead of for (;;)
Fixed bug in symtable where name of keyword argument in function
call was treated as assignment in the scope of the call site. Ex:
def f():
g(a=2) # a was considered a local of f
ceval.c
eval_code2() now take one more argument, a closure.
Implement LOAD_CLOSURE, LOAD_DEREF, STORE_DEREF, MAKE_CLOSURE>
Also: When name error occurs for global variable, report that the
name was global in the error mesage.
Objects/frameobject.c
Initialize f_closure to be a tuple containing space for cellvars
and freevars. f_closure is NULL if neither are present.
Objects/funcobject.c
Add support for func_closure.
Python/import.c
Change the magic number.
Python/marshal.c
Track changes to code objects.
unintentionally caused them to get written in text mode under Windows.
As a result, when .pyc files were later read-- in binary mode --the
magic number was always wrong (note that .pyc magic numbers deliberately
include \r and \n characters, so this was "good" breakage, 100% across
all .pyc files, not random corruption in a subset). Fixed that.
how 'import' was called with a compiletime mechanism: create either a tuple
of the import arguments, or None (in the case of a normal import), add it to
the code-block constants, and load it onto the stack before calling
IMPORT_NAME.
name as n'. By doing some twists and turns, "as" is not a reserved word.
There is a slight change in semantics for 'from module import name' (it will
now honour the 'global' keyword) but only in cases that are explicitly
undocumented.
did the same anyway.
I'm not sure what to do with Tools/compiler/compiler/* -- that isn't part of
distutils, is it ? Should it try to be compatible with old bytecode version ?
MAGIC number. When updating it next time, be sure it's higher than 50715 *
constants. (Shouldn't be a problem if everyone keeps to the proper
algorithm.)
This patch fixes possible overflow in the use of
PyOS_GetLastModificationTime in getmtime.c and Python/import.c.
Currently PyOS_GetLastModificationTime returns a C long. This can
overflow on Win64 where sizeof(time_t) > sizeof(long). Besides it
should logically return a time_t anyway (this patch changes this).
As well, import.c uses PyOS_GetLastModificationTime for .pyc
timestamping. There has been recent discussion about the .pyc header
format on python-dev. This patch adds oveflow checking to import.c so
that an exception will be raised if the modification time
overflows. There are a few other minor 64-bit readiness changes made
to the module as well:
- size_t instead of int or long for function-local buffer and string
length variables
- one buffer overflow check was added (raises an exception on possible
overflow, this overflow chance exists on 32-bit platforms as well), no
other possible buffer overflows existed (from my analysis anyway)
Closes SourceForge patch #100509.
For more comments, read the patches@python.org archives.
For documentation read the comments in mymalloc.h and objimpl.h.
(This is not exactly what Vladimir posted to the patches list; I've
made a few changes, and Vladimir sent me a fix in private email for a
problem that only occurs in debug mode. I'm also holding back on his
change to main.c, which seems unnecessary to me.)
Changed all references to the MAGIC constant to use a global
pyc_magic instead. This global is initially set to MAGIC, but can be
changed by the _PyImport_Init() function to provide for
special features implemented in the compiler which are settable
using command line switches and affect the way PYC files are
generated.
Currently this change is only done for the -U flag.
* in import.c, #ifdef out references to dynamic loading based on
HAVE_DYNAMIC_LOADING
* clean out the platform-specific crud from importdl.c.
[ maybe fold this function into import.c and drop the importdl.c file? Greg.]
* change GetDynLoadFunc's "funcname" parameter to "shortname". change
"name" to "fqname" for clarification.
* each GetDynLoadFunc now creates its own funcname value.
WARNING: as I mentioned previously, we may run into an issue with a
missing "_" on some platforms. Testing will show this pretty quickly,
however.
* move pathname munging into dynload_shlib.c
that file in fact did not exist or at least was not used. Change this
so that __file__ is *only* set to the .pyc/.pyo file when it actually
read the code object from it; otherwise __file__ is set to the .py
file.
frozen packages. (I *think* this means that we can now have a
built-in module bar that's a submodule of a frozen package foo, by
registering the built-in module with a name "foo.bar" in the table of
builtin modules.)
because the path through the code would notice that sys.__path__ did
not exist and it would fall back to the default path (builtins +
sys.path) instead of failing). No longer.
to the table of built-in modules. This should normally be called
*before* Py_Initialize(). When the malloc() or realloc() call fails,
-1 is returned and the existing table is unchanged.
After a similar function by Just van Rossum.
int PyImport_ExtendInittab(struct _inittab *newtab);
int PyImport_AppendInittab(char *name, void (*initfunc)());
Frozen packages are indicated by a negative size (the code string
is the __import__.py file). A frozen package module has its __path__
set to a string, the package name.
time can be in PyImport_ImportModuleEx(). Recursive calls from the
same thread are okay.
Potential problems:
- The lock should really be part of the interpreter state rather than
global, but that would require modifying more files, and I first want
to figure out whether this works at all.
- One could argue that the lock should be per module -- however that
would be complicated to implement. We would have to have a linked
list of locks per module name, *or* invent a new object type to
represent a lock, so we can store the locks in the module or in a
separate dictionary. Both seem unwarranted. The one situation where
this can cause problems is when loading a module takes a long time,
e.g. when the module's initialization code interacts with the user --
during that time, no other threads can run. I say, "too bad."
(modified) and use that.
Some differences in the cleanup algorithm:
- Clear __main__ before the other modules.
- Delete more sys variables: including ps1, ps2, exitfunc, argv, and
even path -- this will prevent new imports!
- Restore stdin, stdout, stderr from __stdin__, __stdout__,
__stderr__, effectively deleting hooks that the user might have
installed -- so their (the hooks') destructors will run.
This is an option for OS-es with case-insensitive but case-preserving
filesystems. It is currently supported for Win32 and MacOS. To
enable it, #define CHECK_IMPORT_CASE in your platform specific
config.h. It is enabled by default on those systems where it is
supported. On Win32, it can be disabled at runtime by setting the
environment variable PYTHONCASEOK (to any value).
When enabled, the feature checks that the case of the requested module
name matches that of the filename found in the filesystem, and raises
a NameError exception when they don't match.
pass it the true file. This is used to set __file__ properly, instead
of believing what the code object carries with it. (If the pointer
is NULL, the code object's co_filename is still used.)
(1) Explicitly clear __builtin__._ and sys.{last,exc}_* before
clearing anything else. These are common places where user values
hide and people complain when their destructors fail. Since the
modules containing them are deleted *last* of all, they would come too
late in the normal destruction order. Sigh.
(2) Add some debugging aid to cleanup (after a suggestion by Marc
Lemburg) -- print the names of the modules being cleaned, and (when
-vv is used) print the names of the variables being cleared.
now implement the following finalization strategy.
1. Whenever this code deletes a module, its directory is cleared
carefully, as follows:
- set all names to None that begin with exactly one underscore
- set all names to None that don't begin with two underscores
- clear the directory
2. Modules are deleted in the following order:
- modules with a reference count of 1, except __builtin__ or __sys__
- repeat until no more are found with a reference count of 1
- __main__ if it's still there
- all remaining modules except __builtin__ or sys
- sys
_ __builtin__
__init__.py (or __init__.pyc/.pyo, whichever applies) is considered a
package. All other subdirectories are left alone. Should make Konrad
Hinsen happy!
dummy entry to sys.modules, marking the absence of a submodule by the
same name.
Thus, if module foo.bar executes the statement "import time",
sys.modules['foo.time'] will be set to None, once the absence of a
module foo.time is confirmed (by looking for it in foo's path).
The next time when foo.bar (or any other submodule of foo) executes
"import time", no I/O is necessary to determine that there is no
module foo.time.
(Justification: It may seem strange to pollute sys.modules. However,
since we're doing the lookup anyway it's definitely the fastest
solution. This is the same convention that 'ni' uses and I haven't
heard any complaints.)
right thing.
Still to do:
- Make reload() of a submodule work.
- Performance tweaks -- currently, a submodule that tries to import a
global module *always* searches the package directory first, even if
the global module was already imported. Not sure how to solve this
one; probably need to record misses per package.
- Documentation!
This doesn't yet support "import a.b.c" or "from a.b.c import x", but
it does recognize directories. When importing a directory, it
initializes __path__ to a list containing the directory name, and
loads the __init__ module if found.
The (internal) find_module() and load_module() functions are
restructured so that they both also handle built-in and frozen modules
and Mac resources (and directories of course). The imp module's
find_module() and (new) load_module() also have this functionality.
Moreover, imp unconditionally defines constants for all module types,
and has two more new functions: find_module_in_package() and
find_module_in_directory().
There's also a new API function, PyImport_ImportModuleEx(), which
takes all four __import__ arguments (name, globals, locals, fromlist).
The last three may be NULL. This is currently the same as
PyImport_ImportModule() but in the future it will be able to do
relative dotted-path imports.
Other changes:
- bltinmodule.c: in __import__, call PyImport_ImportModuleEx().
- ceval.c: always pass the fromlist to __import__, even if it is a C
function, so PyImport_ImportModuleEx() is useful.
- getmtime.c: the function has a second argument, the FILE*, on which
it applies fstat(). According to Sjoerd this is much faster. The
first (pathname) argument is ignored, but remains for backward
compatibility (so the Mac version still works without changes).
By cleverly combining the new imp functionality, the full support for
dotted names in Python (mini.py, not checked in) is now about 7K,
lavishly commented (vs. 14K for ni plus 11K for ihooks, also lavishly
commented).
Good night!
scheme based on object's types, have a simple two-phase scheme based
on object's *names*:
/* To make the execution order of destructors for global
objects a bit more predictable, we first zap all objects
whose name starts with a single underscore, before we clear
the entire dictionary. We zap them by replacing them with
None, rather than deleting them from the dictionary, to
avoid rehashing the dictionary (to some extent). */
importdl.c: the MAXSUFFIXSIZE macro is now defined in importdl.h, and
the modules dictionary is now passed using PyImport_GetModuleDict().
Also undefine USE_SHLIB for AIX -- in AIX 4.2 and up, dlfcn.h exists
but we don't want to use it.
replaces its own entry in sys.module, reference count errors ensue;
even if there is no reference count problem, it would be preferable
for the import to yield the new thing in sys.modules anyway (if only
because that's what later imports will yield). This opens the road to
an official hack to implement a __getattr__ like feature for modules:
stick an instance in sys.modules[__name__].
Jason Tishler
Raymond Hettinger
Phillip J. Eby
Tim Peters
Martin v. Löwis
Thomas Heller
Fred Drake
Jeremy Hylton
Armin Rigo
Nicholas Bastin
Jack Jansen
Walter Dörwald
Neil Schemenauer
Neal Norwitz
Guido van Rossum
Just van Rossum
Michael W. Hudson
Andrew MacIntyre
Marc-André Lemburg
Greg Ward
Barry Warsaw
Thomas Wouters
Sjoerd Mullender