Commit Graph

360 Commits

Author SHA1 Message Date
Brett Cannon 5f85b443f7
GH-106176, GH-104702: Fix reference leak when importing across multiple threads (#108497) 2023-08-29 09:17:25 +02:00
Barry Warsaw 326997829d
gh-98040: Remove find_loader, find_module and other deprecated APIs (#98059)
* Remove deprecated classes from pkgutil
* Remove some other PEP 302 obsolescence
* Use find_spec instead of load_module
* Remove more tests of PEP 302 obsolete APIs
* Remove another bunch of tests using obsolete load_modules()
* Remove deleted names from __all__
* Remove obsolete footnote
* imp is removed
* Remove `imp` from generated stdlib names
* What's new and blurb
* Update zipimport documentation for the removed methods
* Fix some Windows tests
* Remove any test (or part of a test) that references `find_module()`.
* Use assertIsNone() / assertIsNotNone() consistently.
* Update Doc/reference/import.rst
* We don't need pkgutil._get_spec() any more either
*  test.test_importlib.fixtures.NullFinder
* ...BadLoaderFinder.find_module
* ...test_api.InvalidatingNullFinder.find_module
* ...test.test_zipimport test of z.find_module
* Suppress cross-references to find_loader and find_module
* Suppress cross-references to Finder
* Suppress cross-references to pkgutil.ImpImporter and pkgutil.ImpLoader

---------

Co-authored-by: Oleg Iarygin <oleg@arhadthedev.net>
Co-authored-by: Adam Turner <9087854+aa-turner@users.noreply.github.com>
2023-05-03 04:55:22 -07:00
Ned Batchelder 1aa376f946
Remove double space in import error message (#103458) 2023-04-14 00:45:14 -07:00
Brett Cannon 5d08c3ff7d
GH-102700: allow built-in modules to be submodules (GH-103162) 2023-04-06 14:19:54 -07:00
Dong-hee Na f482ade4c7
gh-101766: Fix refleak for _BlockingOnManager resources from test suite level (gh-101988) 2023-02-18 00:18:47 +09:00
Dong-hee Na 775f8819e3
gh-101766: Fix refleak for _BlockingOnManager resources (gh-101942) 2023-02-17 19:14:07 +09:00
Jean-Paul Calderone 3325f054e3
gh-91351: Fix some bugs in importlib handling of re-entrant imports (GH-94504)
Co-authored-by: Brett Cannon <brett@python.org>
2023-01-20 16:00:39 -08:00
Anh71me bd221c01dd
gh-98139: enhance namespace package repr 2022-11-06 22:13:40 +00:00
Gregory P. Smith 683ab85955
bpo-38693: Use f-strings instead of str.format() within importlib (#17058)
This is a small performance improvement, especially for one or two hot
places such as _handle_fromlist() that are called a lot and the
.format() method was being used just to join two strings with a dot.

Otherwise it is merely a readability improvement.

We keep `_ERR_MSG` and `_ERR_MSG_PREFIX` as those may be used elsewhere for canonical looking error messages.
2022-10-06 16:43:16 -07:00
Brett Cannon c206e53bb7
gh-65961: Raise `DeprecationWarning` when `__package__` differs from `__spec__.parent` (#97879)
Also remove `importlib.util.set_package()` which was already slated for removal.

Co-authored-by: Eric Snow <ericsnowcurrently@gmail.com>
2022-10-05 15:00:45 -07:00
Barry Warsaw 5dc3599135
gh-97850: Remove all known instances of module_repr() (#97876)
Remove all known instances of module_repr()
2022-10-05 11:42:26 -07:00
Barry Warsaw e1182bc377
gh-94619: Remove long deprecated methods module_repr() and load_module() (#94624)
* gh-94619: Remove long deprecated methods module_repr() and load_module()

Closes #94619

* Update Misc/NEWS.d/next/Library/2022-07-06-14-57-33.gh-issue-94619.PRqKVX.rst

Fix typo

Co-authored-by: Brett Cannon <brett@python.org>

Co-authored-by: Brett Cannon <brett@python.org>
2022-08-04 17:24:26 -07:00
Barry Warsaw 876fc7fcec
bpo-35673: Add a public alias for namespace package __loader__ attribute (#29049)
Rename namespace package __loader__ class to be public.

Make the old name, i.e. _NamespaceLoader, an alias for the public name, for backward compatibility.
2021-10-20 14:05:29 -07:00
Eric Snow 79cf20e48d
bpo-21736: Set __file__ on frozen stdlib modules. (gh-28656)
Currently frozen modules do not have __file__ set.  In their spec, origin is set to "frozen" and they are marked as not having a location.  (Similarly, for frozen packages __path__ is set to an empty list.)  However, for frozen stdlib modules we are able to extrapolate __file__ as long as we can determine the stdlib directory at runtime.  (We now do so since gh-28586.)  Having __file__ set is helpful for a number of reasons.  Likewise, having a non-empty __path__ means we can import submodules of a frozen package from the filesystem (e.g. we could partially freeze the encodings module).

This change sets __file__ (and adds to __path__) for frozen stdlib modules.  It uses sys._stdlibdir (from gh-28586) and the frozen module alias information (from gh-28655).  All that work is done in FrozenImporter (in Lib/importlib/_bootstrap.py). 
 Also, if a frozen module is imported before importlib is bootstrapped (during interpreter initialization) then we fix up that module and its spec during the importlib bootstrapping step (i.e. imporlib._bootstrap._setup()) to match what gets set by FrozenImporter, including setting the file info (if the stdlib dir is known).  To facilitate this, modules imported using PyImport_ImportFrozenModule() have __origname__ set using the frozen module alias info.  __origname__ is popped off during importlib bootstrap.

(To be clear, even with this change the new code to set __file__ during fixups in imporlib._bootstrap._setup() doesn't actually get triggered yet.  This is because sys._stdlibdir hasn't been set yet in interpreter initialization at the point importlib is bootstrapped.  However, we do fix up such modules at that point to otherwise match the result of importing through FrozenImporter, just not the __file__ and __path__ parts.  Doing so will require changes in the order in which things happen during interpreter initialization.  That can be addressed separately.  Once it is, the file-related fixup code from this PR will kick in.)

Here are things this change does not do:

* set __file__ for non-stdlib modules (no way of knowing the parent dir)
* set __file__ if the stdlib dir is not known (nor assume the expense of finding it)
* relatedly, set __file__ if the stdlib is in a zip file
* verify that the filename set to __file__ actually exists (too expensive)
* update __path__ for frozen packages that alias a non-package (since there is no package dir)

Other things this change skips, but we may do later:

* set __file__ on modules imported using PyImport_ImportFrozenModule()
* set co_filename when we unmarshal the frozen code object while importing the module (e.g. in FrozenImporter.exec_module()) -- this would allow tracebacks to show source lines
* implement FrozenImporter.get_filename() and FrozenImporter.get_source()

https://bugs.python.org/issue21736
2021-10-14 15:32:18 -06:00
Eric Snow 08285d563e
bpo-45020: Identify which frozen modules are actually aliases. (gh-28655)
In the list of generated frozen modules at the top of Tools/scripts/freeze_modules.py, you will find that some of the modules have a different name than the module (or .py file) that is actually frozen. Let's call each case an "alias". Aliases do not come into play until we get to the (generated) list of modules in Python/frozen.c. (The tool for freezing modules, Programs/_freeze_module, is only concerned with the source file, not the module it will be used for.)

Knowledge of which frozen modules are aliases (and the identity of the original module) normally isn't important. However, this information is valuable when we go to set __file__ on frozen stdlib modules. This change updates Tools/scripts/freeze_modules.py to map aliases to the original module name (or None if not a stdlib module) in Python/frozen.c. We also add a helper function in Python/import.c to look up a frozen module's alias and add the result of that function to the frozen info returned from find_frozen().

https://bugs.python.org/issue45020
2021-10-05 11:26:37 -06:00
Eric Snow c3d9ac8b34
bpo-45324: Capture data in FrozenImporter.find_spec() to use in exec_module(). (gh-28633)
Before this change we end up duplicating effort and throwing away data in FrozenImporter.find_spec().  Now we do the work once in find_spec() and the only thing we do in FrozenImporter.exec_module() is turn the raw frozen data into a code object and then exec it.

We've added _imp.find_frozen(), add an arg to _imp.get_frozen_object(), and updated FrozenImporter.  We've also moved some code around to reduce duplication, get a little more consistency in outcomes, and be more efficient.

Note that this change is mostly necessary if we want to set __file__ on frozen stdlib modules. (See https://bugs.python.org/issue21736.)

https://bugs.python.org/issue45324
2021-10-05 10:01:27 -06:00
Germán Méndez Bravo 03648a2a91
bpo-43392: Optimize repeated calls to `__import__()` (GH-24735)
Implements a two steps check in `importlib._bootstrap._find_and_load()` to avoid locking when the module has been already imported and it's ready.

---

Using `importlib.__import__()`, after this, does show a big difference:

Before:
```
$ ./python -c 'import timeit; print(timeit.timeit("__import__(\"timeit\")", setup="from importlib import __import__"))'
15.92248619502061
```

After:
```
$ ./python -c 'import timeit; print(timeit.timeit("__import__(\"timeit\")", setup="from importlib import __import__"))'
1.206068897008663
```

---
2021-08-12 11:23:29 -07:00
Filipe Laíns 0a8ae8a50a
bpo-44717: improve AttributeError on circular imports of submodules (GH-27338) 2021-07-24 23:44:46 +01:00
Pablo Galindo Salgado 3eae8f20d7
Revert "bpo-44717: improve AttributeError on circular imports of submodules (GH-27299)" (GH-27331)
This reverts commit 8072a1181d.
2021-07-24 14:33:03 +01:00
Filipe Laíns 8072a1181d
bpo-44717: improve AttributeError on circular imports of submodules (GH-27299)
Signed-off-by: Filipe Laíns <lains@riseup.net>

Co-authored-by: Łukasz Langa <lukasz@langa.pl>
2021-07-24 11:50:17 +02:00
Brett Cannon 57c6cb5100
bpo-42135: Deprecate implementations of find_module() and find_loader() (GH-25169) 2021-04-06 08:56:57 -07:00
Brett Cannon a7ff6df60c
bpo-42134: Raise ImportWarning when calling find_module() in the import system (GH-25044) 2021-03-30 08:43:03 -07:00
Brett Cannon 1899087b21
bpo-42136: Deprecate module_repr() as found in importlib (GH-25022) 2021-03-26 11:55:07 -07:00
Brett Cannon 9cb31d6716
bpo-42137: have ModuleType.__repr__ prefer __spec__ over module_repr() (GH-24953)
This is to work towards the removal of the use of  module_repr() in Python 3.12 (documented as deprecated since 3.4).
2021-03-24 08:26:56 -07:00
Brett Cannon 2de5097ba4
bpo-26131: Deprecate usage of load_module() (GH-23469)
Raise an ImportWarning when the import system falls back on load_module(). As for implementations of load_module(), raise a DeprecationWarning.
2020-12-04 15:39:21 -08:00
Victor Stinner 3be8e220ed
bpo-42403: Use @staticmethod in importlib (GH-23395)
Use @staticmethod on methods using @classmethod but don't use their
cls parameter on the following classes:

* BuiltinImporter
* FrozenImporter
* WindowsRegistryFinder
* PathFinder

Leave methods using @_requires_builtin or @_requires_frozen unchanged,
since this decorator requires the wrapped method to have an extra parameter
(cls or self).
2020-11-20 14:44:02 +01:00
Victor Stinner 3390347aa0
bpo-42403: Simplify importlib external bootstrap (GH-23397)
Simplify the importlib external bootstrap code:
importlib._bootstrap_external now uses regular imports to import
builtin modules. When it is imported, the builtin __import__()
function is already fully working and so can be used to import
builtin modules like sys.
2020-11-19 13:43:43 +01:00
Armin Rigo 6daa37fd42
bpo-38091: Import deadlock detection causes deadlock (GH-17518)
Automerge-Triggered-By: @brettcannon
2020-03-02 17:37:25 -08:00
Dino Viehland 9b6fec4651
bpo-39336: Allow packages to not let their child modules be set on them (#18006)
* bpo-39336: Allow setattr to fail on modules which aren't assignable

When attaching a child module to a package if the object in sys.modules raises an AttributeError (e.g. because it is immutable) it causes the whole import to fail.  This now allows immutable packages to exist and an ImportWarning is reported and the AttributeError exception is ignored.
2020-01-22 16:42:38 -08:00
Dong-hee Na 145cf1f50c bpo-35923: Update the BuiltinImporter to use loader._ORIGIN instead of a hard-coded value (GH-15651) 2019-09-11 17:00:02 +01:00
Serhiy Storchaka 662db125cd
bpo-37685: Fixed __eq__, __lt__ etc implementations in some classes. (GH-14952)
They now return NotImplemented for unsupported type of the other operand.
2019-08-08 08:42:54 +03:00
Ngalim Siregar c5fa44944e bpo-37444: Update differing exception between builtins and importlib (GH-14869)
Imports now raise `TypeError` instead of `ValueError` for relative import failures. This makes things consistent between `builtins.__import__` and `importlib.__import__` as well as using a more natural import for the failure.


https://bugs.python.org/issue37444



Automerge-Triggered-By: @brettcannon
2019-08-02 22:46:02 -07:00
Nina Zakharenko 1dc5cb9cb3 bpo-35887: Add make regen-importlib step to importlib._bootstrap docstring (GH-11777) 2019-02-13 15:21:17 -08:00
Nina Zakharenko 69091cb497 bpo-35321: Set the spec origin to frozen in frozen modules (#11732)
* bpo-35321: Set the spec origin to frozen in frozen modules

This fix correctly sets the spec origin to
"frozen" for the _frozen_importlib module. Note that the
origin was already correctly set in _frozen_importlib_external.

* 📜🤖 Added by blurb_it.
2019-02-04 16:56:26 -08:00
Serhiy Storchaka c93c58b5d5
bpo-33331: Clean modules in the reversed order in PyImport_Cleanup(). (GH-6565)
Modules imported last are now cleared first at interpreter shutdown.

A newly imported module is moved to the end of sys.modules, behind
modules on which it depends.
2018-10-29 19:30:16 +02:00
Serhiy Storchaka 4e2442505c
bpo-32946: Speed up "from ... import ..." from non-packages. (GH-5873) 2018-03-11 10:52:37 +02:00
Barry Warsaw bbbcf8693b
bpo-32303 - Consistency fixes for namespace loaders (#5481)
* Make sure ``__spec__.loader`` matches ``__loader__`` for namespace packages.
* Make sure ``__spec__.origin` matches ``__file__`` for namespace packages.

https://bugs.python.org/issue32303
https://bugs.python.org/issue32305
2018-02-02 15:15:58 -05:00
Serhiy Storchaka 41c56940c6 bpo-21720: Restore the Python 2.7 logic in handling a fromlist. (#4118)
BytesWarning no longer emitted when the fromlist argument of
__import__() or the __all__ attribute of the module contain bytes
instances.
2017-10-26 10:41:59 +03:00
Serhiy Storchaka f07e2b64df bpo-31642: Restore blocking "from" import by setting None in sys.modules. (#3834) 2017-10-08 10:44:10 +03:00
Antoine Pitrou 88c60c9668 Trivial cleanups following bpo-31370 (#3649)
* Trivial cleanups following bpo-31370

* Also cleanup the "importlib._bootstrap_external" module
2017-09-18 23:50:44 +02:00
Serhiy Storchaka 9b0d1d647e bpo-31070: Fix a race condition in importlib _get_module_lock(). (#3033) 2017-08-09 14:29:12 +03:00
Victor Stinner e72b1359f8 bpo-30891: Fix again importlib _find_and_load() (#2665)
Use sys.modules.get() in the "with _ModuleLockManager(name):" block
to protect the dictionary key with the module lock and use an atomic
get to prevent race condition.

Remove also _bootstrap._POPULATE since it was unused
(_bootstrap_external now has its own _POPULATE object), add a new
_SENTINEL object instead.
2017-07-21 13:00:46 +02:00
Serhiy Storchaka 8a9cd20edc bpo-30876: Relative import from unloaded package now reimports the package (#2639)
instead of failing with SystemError.

Relative import from non-package now fails with ImportError rather than
SystemError.
2017-07-12 06:50:03 +03:00
Victor Stinner 4f9a446f3f bpo-30891: Fix importlib _find_and_load() race condition (#2646)
* Rewrite importlib _get_module_lock(): it is now responsible to hold
  the imp lock directly.
* _find_and_load() now holds the module lock to check if name is in
  sys.modules to prevent a race condition
2017-07-10 22:52:32 +02:00
Serhiy Storchaka b4baacee1a bpo-30814: Fixed a race condition when import a submodule from a package. (#2580) 2017-07-06 08:09:03 +03:00
Eric Snow 6b4be195cd bpo-22257: Small changes for PEP 432. (#1728)
PEP 432 specifies a number of large changes to interpreter startup code, including exposing a cleaner C-API. The major changes depend on a number of smaller changes. This patch includes all those smaller changes.
2017-05-22 21:36:03 -07:00
Eric Snow f3fd06a2e4 Issue #28026: Raise ImportError when exec_module() exists but create_module() is missing. 2016-09-08 11:12:31 -07:00
Eric Snow 46f97b85a8 Issue #15767: Use ModuleNotFoundError. 2016-09-07 16:56:15 -07:00
Brett Cannon eaecc692df Issue #27974: Remove importlib._bootstrap._ManageReload.
Class was dead code. Thanks to Xiang Zhang for the patch.
2016-09-06 16:20:46 -07:00
Brett Cannon 3fef2eef2e Issue #26972: Fix some mistakes in importlib-related docstrings.
Thanks to Oren Milman for the patch.
2016-07-08 11:09:35 -07:00