importlib._bootstrap._get_sourcefile().
Thanks to its only use by the C API, it was never properly tested
until now.
Thanks to Neal Norwitz for discovering the bug and Madison May for the patch.
The private attribute was leaking out of importlib and led to at least
one person noticing it. Switch to another hack which won't leak
outside of importlib and is nearly as robust.
The helper function makes it easier to implement
imoprtlib.abc.InspectLoader.get_source() by making that function
require just the raw bytes for source code and handling all other
details.
UnicodeDecodeError as ImportError. That was over-reaching the point of
raising ImportError in get_source() (which is to signal the source
code was not found when it should have). Conflating the two exceptions
with ImportError could lead to masking errors with the source which
should be known outside of whether there was an error simply getting
the source to begin with.
Forgot to raise ModuleNotFoundError when None is found in sys.modules.
This led to introducing the C function PyErr_SetImportErrorSubclass()
to make setting ModuleNotFoundError easier.
Also updated the reference docs to mention ModuleNotFoundError
appropriately. Updated the docs for ModuleNotFoundError to mention the
None in sys.modules case.
Lastly, it was noticed that PyErr_SetImportError() was not setting an
exception when returning None in one case. That issue is now fixed.
ImportError.
The exception is raised by import when a module could not be found.
Technically this is defined as no viable loader could be found for the
specified module. This includes ``from ... import`` statements so that
the module usage is consistent for all situations where import
couldn't find what was requested.
This should allow for the common idiom of::
try:
import something
except ImportError:
pass
to be updated to using ModuleNotFoundError and not accidentally mask
ImportError messages that should propagate (e.g. issues with a
loader).
This work was driven by the fact that the ``from ... import``
statement needed to be able to tell the difference between an
ImportError that simply couldn't find a module (and thus silence the
exception so that ceval can raise it) and an ImportError that
represented an actual problem.
importlib.abc.Loader.init_module_attrs() and implement
importlib.abc.InspectLoader.load_module().
The importlib.abc.Loader.init_module_attrs() method sets the various
attributes on the module being loaded. It is done unconditionally to
support reloading. Typically people used
importlib.util.module_for_loader, but since that's a decorator there
was no way to override it's actions, so init_module_attrs() came into
existence to allow for overriding. This is also why module_for_loader
is now pending deprecation (having its other use replaced by
importlib.util.module_to_load).
All of this allowed for importlib.abc.InspectLoader.load_module() to
be implemented. At this point you can now implement a loader with
nothing more than get_code() (which only requires get_source();
package support requires is_package()). Thanks to init_module_attrs()
the implementation of load_module() is basically a context manager
containing 2 methods calls, a call to exec(), and a return statement.
handle providing (and cleaning up if needed) the module to be loaded.
A future commit will use the context manager in
Lib/importlib/_bootstrap.py and thus why the code is placed there
instead of in Lib/importlib/util.py.
While the previous location was fine, it makes more sense to have the
method higher up in the inheritance chain, especially at a point where
get_source() is defined which is the earliest source_to_code() could
programmatically be used in the inheritance tree in importlib.abc.
attributes to None.
The long-term goal is for people to be able to rely on these
attributes existing and checking for None to see if they have been
set. Since import itself sets these attributes when a loader does not
the only instances when the attributes are None are from someone
overloading __import__() and not using a loader or someone creating a
module from scratch.
This patch also unifies module initialization. Before you could have
different attributes with default values depending on how the module
object was created. Now the only way to not get the same default set
of attributes is to circumvent initialization by calling
ModuleType.__new__() directly.
the default exception/value when called instead of raising/returning
NotimplementedError/NotImplemented (except where appropriate).
This should allow for the ABCs to act as the bottom/end of the MRO with expected
default results.
As part of this work, also make importlib.abc.Loader.module_repr()
optional instead of an abstractmethod.
__loader__ is not set on a module. This brings the exception in line
with when __loader__ is None (which is equivalent to not having the
attribute defined).
First, because the mtime can exceed 4 bytes, make sure to mask it down to 4
bytes before getting its little-endian representation for writing out to a .pyc
file.
Two, cap an rsplit() call to 1 split, else can lead to too many values being
returned for unpacking.
importlib.machinery.FileFinder when the directory has become
unreadable or a file. This brings semantics in line with Python 3.2
import.
Reported and diagnosed by David Pritchard.
fromlist of __import__ propagate.
The problem previously was that if something listed in fromlist didn't
exist then that's okay. The fix for that was too broad in terms of
catching ImportError.
The trick with the solution to this issue is that the proper
refactoring of import thanks to importlib doesn't allow for a way to
distinguish (portably) between an ImportError because finders couldn't
find a loader, or a loader raised the exception. In Python 3.4 the
hope is to introduce a new exception (e.g. ModuleNotFound) to make it
clean to differentiate why ImportError was raised.
When the fromlist argument is specified for __import__() and the
attribute doesn't already exist, an import is attempted. If that fails
(e.g. module doesn't exist), the ImportError will now be silenced (for
backwards-compatibility). This *does not* affect
``from ... import ...`` statements.
Thanks to Eric Snow for the patch and Simon Feltman for reporting the
regression.
state of the import system. Also make importlib.invalidate_caches()
work with sys.meta_path instead of sys.path_importer_cache to
completely separate the path-based import system from the overall
import system.
Patch by Eric Snow.
Lib/imp.py for imp.source_from_cache() instead of its own C version.
Also change PyImport_ExecCodeModuleObject() to not infer the source
path from the bytecode path like
PyImport_ExecCodeModuleWithPathnames() does. This makes the function
less magical.
This also has the side-effect of removing all uses of MAXPATHLEN in
Python/import.c which can cause failures on really long filenames.
statement (e.g. ``from distutils import msvc9compiler``) that triggers
an ImportError of its own (e.g. the non-existence of winreg), let that
exception propagate instead of raising a generic ImportError for the
module being requested (e.g. msvc9compiler).
module name into consideration when determining whether a module is a
package or not. This prevents importing a module's __init__ module
directly and having it considered a package, which can lead to
duplicate sub-modules.
Thanks to Ronan Lamy for reporting the bug.
The long-term goal is to deprecate imp.find_module() in favour of this
API, but it will take some time as some APIs explicitly return/use what
imp.find_module() returns.
importlib.abc.FileLoader.load_module()/get_filename() and
importlib.machinery.ExtensionFileLoader.load_module() have their
single argument be optional as the loader's constructor has all the
ncessary information.
This allows for the deprecation of
imp.load_source()/load_compile()/load_package().
importlib.machinery that provide the suffix details for import.
The attributes were not put on imp so as to compartmentalize
everything importlib needs for setting up imports in
importlib.machinery.
This also led to an indirect deprecation of inspect.getmoduleinfo() as
it directly returned imp.get_suffix's returned tuple which no longer
makes sense.
This introduces a new function, imp.extension_suffixes(), which is
currently undocumented. That is forthcoming once issue #14657 is
resolved and how to expose file suffixes is decided.
importlib.util.module_for_loader also will set __loader__ along with
__package__. This is in conjunction to a forthcoming update to PEP 302
which will make these two attributes required for loaders to set.
be implicit.
Added a warning for when sys.path_hooks is found to be empty. Also
changed the meaning of None in sys.path_importer_cache to represent
trying sys.path_hooks again (an interpretation of previous semantics).
Also added a warning for when None was found.
The long-term goal is for None in sys.path_importer_cache to represent
the same as imp.NullImporter: no finder found for that sys.path entry.
importlib.machinery.(FileFinder, SourceFileLoader,
_SourcelessFileLoader, ExtensionFileLoader).
This exposes all of importlib's mechanisms that will become public on
the sys module.
for performance. While get_magic() could move to Lib/imp.py, having to
support PyImport_GetMagicNumber() would lead to equal, if not more, C
code than sticking with the status quo.
rewriting functionality in pure Python.
To start, imp.new_module() has been rewritten in pure Python, put into
importlib (privately) and then publicly exposed in imp.
of sys.modules when possible.
This is being done for two reasons. One is to gain a little bit of
performance by skipping an unnecessary dict lookup in sys.modules. But
the other (and main) reason is to be a little bit more clear in how
things should work from the perspective of import's interactions with
loaders. Otherwise loaders can easily forget to return the module even
though PEP 302 explicitly states they are expected to return the module
they loaded.
importlib._bootstrap is now frozen into Python/importlib.h and stored
as _frozen_importlib in sys.modules. Py_Initialize() loads the frozen
code along with sys and imp and then uses _frozen_importlib._install()
to set builtins.__import__() w/ _frozen_importlib.__import__().
importation, then respect that injection.
Discovered thanks to Lib/xml/parsers/expat.py injecting
xml.parsers.expat.errors and etree now importing that directly as a
module.
It seems better to cache the finder for the cwd under its full path
insetad of '' in case the cwd changes. Otherwise FileFinder needs to
dynamically change itself based on whether it is given '' instead of
caching a finder for every change to the cwd.
Thanks to os.environ under Windows only updating the dict and not the
environment itself (as exposed by nt.environ), tests using
PYTHONCASEOK always fail. Now the tests are skipped when os.environ
does not do what is expected.
This required updating the code to use posix instead of os. This is
all being done to make bootstrapping easier to removing dependencies
that are kept in importlib.__init__ and thus outside of the single
file to bootstrap from.
paths.
__import__ does a little trick when importing from bytecode by
back-patching the co_filename paths to point to the file location
where the code object was loaded from, *not* where the code object was
originally created. This allows co_filename to point to a valid path.
Problem is that co_filename is immutable from Python, so a private
function -- imp._fix_co_filename() -- had to be introduced in order to
get things working properly. Originally the plan was to add a file
argument to marshal.loads(), but that failed as the algorithm used by
__import__ is not fully recursive as one might expect, so to be fully
backwards-compatible the code used by __import__ needed to be exposed.
This closes issue #6811 by taking a different approach than outlined
in the issue.
Ezio Melotti
Brett Cannon
Benjamin Peterson
Victor Stinner
Serhiy Storchaka
Eric Snow
Andrew Svetlov
Barry Warsaw
Nick Coghlan
Trent Nelson
Jesus Cea
Antoine Pitrou
Philip Jenvey
Martin v. Löwis
Amaury Forgeot d'Arc
Florent Xicluna
Eric V. Smith
Marc-Andre Lemburg
Charles-François Natali
Meador Inge
Vinay Sajip
Éric Araujo
Victor Stinner