traverseda
/
cpython
1
0
Fork 0
Code Issues Pull Requests Releases Wiki Activity

bpo-42101: allow inheritance of venv 

This add the --inherit option when creating virtual environments.
witch this child environments will inherit all installed package from
all the parents which can be useful to speed up installation; as well as
reduced disk usage.

Example with two environment parent and child.

This acts similarly to Python ChainMap, where the activated environment
is mutable, installing, updating (by shadowing) works by modifying the
child env, uninstalling is not possible when the package is installed in
a parent env.

 $ python -m venv parent
 $ source parent/bin/activate
 $ pip install requests
 $ deactivate

 $ python -m venv child --inherit parent
 $ source child/bin/activate
 $ python -c 'import requests' # ok, pulled from parent.

 $ pip uninstall requests # fails parent is read only.
 $ pip install flask      # works

 $ pip install requests --upgrade # would work and shadow the parent requests

This patch was initially created by the D. E. Shaw group (https://www.deshaw.com/)
and, on their behalf., enhanced/contributed back via
QuanSight Labs (https://www.quansight.com/labs)

The following authors have contributed to the upstream patch:

Co-authored-by: Robert Byrnes <byrnes@deshaw.com>
Co-authored-by: Vitaly Shupak <shchupak@deshaw.com>
This commit is contained in:
Matthias Bussonnier 2020-10-20 14:47:47 -07:00
parent a13b26cac1
commit dfe6c08075
10 changed files with 278 additions and 10 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

6
Doc/library/site.rst
View File

@ -274,6 +274,12 @@ the user site-packages directory is enabled, ``1`` if it was disabled by the
user, ``2`` if it is disabled for security reasons or by an administrator, and a
value greater than 2 if there is an error.
.. cmdoption:: --path
When using inheritied virtual environment, will print a list of path for each
virtual environents. And exit with a 0 exit status. Used in venv
activate script to generate the PATH variable.
.. seealso::
:pep:`370` -- Per user site-packages directory

7
Doc/library/venv.rst
View File

@ -98,7 +98,7 @@ creation according to their needs, the :class:`EnvBuilder` class.
.. class:: EnvBuilder(system_site_packages=False, clear=False, \
symlinks=False, upgrade=False, with_pip=False, \
prompt=None, upgrade_deps=False)
prompt=None, upgrade_deps=False, inherit=None)
The :class:`EnvBuilder` class accepts the following keyword arguments on
instantiation:
@ -136,6 +136,11 @@ creation according to their needs, the :class:`EnvBuilder` class.
.. versionadded:: 3.9
Added the ``upgrade_deps`` parameter
* ``inherit`` -- a List of virtual enviroments this enviroment should
inherit from.
.. versionadded:: 3.10
Creators of third-party virtual environment tools will be free to use the
provided :class:`EnvBuilder` class as a base class.

20
Doc/using/venv-create.inc
View File

@ -14,6 +14,11 @@ used at environment creation time). It also creates an (initially empty)
``Lib\site-packages``). If an existing directory is specified, it will be
re-used.
.. versionadded: 3.10
``pyvenv.cfg`` may now have the key ``inherit`` which value is a list of
absolute path colon separated from which current environment inherit from.
.. deprecated:: 3.6
``pyvenv`` was the recommended tool for creating virtual environments for
Python 3.3 and 3.4, and is `deprecated in Python 3.6
@ -33,6 +38,13 @@ your :ref:`Python installation <using-on-windows>`::
c:\>python -m venv c:\path\to\myenv
Starting at Python 3.10, you can use the ``--inherit`` flag, one or several time
to inherit the packages installed from another virtual environment, and packages
installed on parent venvs will be visible in child one when the child do not
shadow them. This works similarly to ChainMap. Many tools (like pip) will only
let you remove or modify packages in current environment, and will nto allow you
to modify parent directly.
The command, if run with ``-h``, will show the available options::
usage: venv [-h] [--system-site-packages] [--symlinks | --copies] [--clear]
@ -63,10 +75,17 @@ The command, if run with ``-h``, will show the available options::
environment.
--upgrade-deps Upgrade core dependencies: pip setuptools to the
latest version in PyPI
--inherit BASE Inherit packages from another base virtualenv. This
option can be repeated for multiple inheritance.
Once an environment has been created, you may wish to activate it, e.g. by
sourcing an activate script in its bin directory.
.. versionadded:: 3.10
Ass ``--inherit`` option to allow inheritance of virtual environments.
.. versionchanged:: 3.9
Add ``--upgrade-deps`` option to upgrade pip + setuptools to the latest on PyPI
@ -147,3 +166,4 @@ detail (typically a script or shell function will be used).
.. versionadded:: 3.8
PowerShell activation scripts installed under POSIX for PowerShell Core
support.

162
Lib/site.py
View File

@ -75,6 +75,8 @@ import builtins
import _sitebuiltins
import io
# Inheritance information.
INHERIT = {}
# Prefixes for site-packages; add additional prefixes like /usr/local here
PREFIXES = [sys.prefix, sys.exec_prefix]
# Enable per user site-packages directory
@ -573,6 +575,147 @@ def execusercustomize():
(err.__class__.__name__, err))
def _read_inherit(filename):
"""Return a list of directories from an inherit file."""
# Set VIRTUAL_ENV_DISABLE_INHERIT (to any value)
# to disable inheritance.
if "VIRTUAL_ENV_DISABLE_INHERIT" in os.environ:
return []
virtualenvs = []
try:
with open(filename, "r", encoding="utf-8") as f:
for line in f:
if line.startswith("inherit ="):
paths_list_str = line.split('=', maxsplit=1)[1].strip()
paths_list = paths_list_str.split(":")
virtualenvs.extend(paths_list)
break
except IOError as e:
import errno
if e.errno == errno.ENOENT:
return []
else:
raise
return virtualenvs
def _walk_inherit(dir, seen):
"""Walk through the inheritance hierarchy, recursively: depth first.
This sets up a linearized precedence list in INHERIT["sys.prefix"].
"""
# First, visit this directory, and remember that we've been here.
INHERIT["sys.prefix"].append(dir)
seen.add(dir)
# Next, visit each base directory mentioned in an inherit file (if any).
for base in _read_inherit(os.path.join(dir, "pyvenv.cfg")):
# Relative base directories are relative to this directory,
# i.e., the directory containing the inherit file.
absbase = base if os.path.isabs(base) else os.path.join(dir, base)
# Canonicalize the base directory.
canonbase = os.path.realpath(absbase)
# Only visit each directory once.
if not canonbase in seen:
_walk_inherit(canonbase, seen)
def _init_sys_prefix():
"""Initialize INHERIT["sys.prefix"]."""
INHERIT["sys.prefix"] = []
# Initialize the linearized precedence list for all virtualenvs.
_walk_inherit(sys.prefix, set())
def get_venv_base_prefix(base):
"""
Obtain the base python directory of a virtualenv
"""
virtual_conf = os.path.join(base, "pyvenv.cfg")
# the base python installation does not have a pyvenv.cfg file
if not os.path.exists(virtual_conf):
# check that the $base/bin/python symlink exists
pydir = os.path.join(base, "bin", "python")
if not os.path.exists(pydir):
return None
return base, ''
# read the "home" variable from pyvenv.cfg
# and call get_venv_base_prefix on it recursively
with open(virtual_conf, encoding="utf-8") as f:
for line in f:
if "=" in line:
key, _, value = line.partition("=")
key = key.strip().lower()
value = value.strip()
if key == "home":
home = os.path.dirname(value)
base, reason = get_venv_base_prefix(home)
return base
# no lines matching "home = " in the file - bad config
return None
def _check_inherit():
"""Perform sanity checks on the base virtualenvs."""
if "VIRTUAL_ENV_DISABLE_BASE_CHECKS" in os.environ:
return
errmsg = ""
for base in INHERIT["sys.prefix"][1:]:
base_prefix = get_venv_base_prefix(base)
if not base_prefix:
errmsg += "\n%s is not a python3 virtualenv" % base
elif base_prefix != sys.base_prefix:
errmsg += (
"\nbase virtualenv for %s (%s) "
"does not match current python version (%s)"
% (base, base_prefix, sys.base_prefix)
)
if errmsg:
raise RuntimeError("virtualenv inheritance problems: %r" % errmsg)
def _fini_sys_prefix():
"""Finalize INHERIT["sys.prefix"]."""
# Perform sanity checks on the base virtualenvs.
_check_inherit()
# Set up other prefixes, for convenience and consistency.
INHERIT["sys.prefix"].append(USER_BASE)
INHERIT["sys.prefix"].append(sys.base_prefix)
def _init_PATH():
"""Initialize PATH components added by activation scripts."""
# Derive the bin subdirectory from the location of the python executable.
bindir = os.path.basename(os.path.dirname(sys.executable))
# Initialize the list of PATH components by appending the bin subdirectory
# to the top-level directories for all of the virtualenvs.
INHERIT["PATH"] = [
os.path.join(dirpath, bindir) for dirpath in INHERIT["sys.prefix"]
]
def _get_PATH():
"""Return PATH components added by activation scripts, as a string."""
return os.pathsep.join(INHERIT["PATH"])
def main():
"""Add standard site-specific directories to the module search path.
@ -582,6 +725,9 @@ def main():
global ENABLE_USER_SITE
orig_path = sys.path[:]
# No longer populated in py3, but initialized for backwards compatibility
INHERIT["sys.path"] = []
known_paths = removeduppaths()
if orig_path != sys.path:
# removeduppaths() might make sys.path absolute.
@ -589,10 +735,21 @@ def main():
abs_paths()
known_paths = venv(known_paths)
# Once sys.prefix has been set, initialize the linearized precedence list
# for all virtualenvs.
_init_sys_prefix()
# Add all the virtualenvs to site-packages
known_paths = addsitepackages(known_paths, INHERIT["sys.prefix"][1:])
if ENABLE_USER_SITE is None:
ENABLE_USER_SITE = check_enableusersite()
known_paths = addusersitepackages(known_paths)
known_paths = addsitepackages(known_paths)
# Initialize PATH components added by activation scripts
_init_PATH()
# Finalize the linearized precedence list.
# This is done last, after all of the locations have been determined.
_fini_sys_prefix()
setquit()
setcopyright()
sethelper()
@ -609,7 +766,7 @@ if not sys.flags.no_site:
def _script():
help = """\
%s [--user-base] [--user-site]
%s [--user-base] [--user-site] [--path]
Without arguments print some useful information
With arguments print the value of USER_BASE and/or USER_SITE separated
@ -637,6 +794,9 @@ def _script():
print("ENABLE_USER_SITE: %r" % ENABLE_USER_SITE)
sys.exit(0)
if "--path" in args:
print(_get_PATH())
sys.exit(0)
buffer = []
if '--user-base' in args:
buffer.append(USER_BASE)

31
Lib/test/test_venv.py
View File

@ -84,6 +84,37 @@ class BaseTest(unittest.TestCase):
result = f.read()
return result
class VenvInheritTests(BaseTest):
def test_create_inherit(self):
builder1 = venv.EnvBuilder()
with tempfile.TemporaryDirectory() as fake_env_dir:
def pip_cmd_checker(cmd):
self.assertEqual(
cmd,
[
os.path.join(fake_env_dir, bin_path, python_exe),
'-m',
'pip',
'install',
'--upgrade',
'pip',
'setuptools'
]
)
fake_context = builder1.create(fake_env_dir)
with tempfile.TemporaryDirectory() as child_env_dir:
builder2 = venv.EnvBuilder(inherit=[fake_env_dir])
with patch.dict(os.environ, {"VIRTUAL_ENV_DISABLE_BASE_CHECKS": "1"}):
fake_context = builder2.create(child_env_dir)
with open(os.path.join( child_env_dir, 'pyvenv.cfg'), 'r') as f:
self.assertIn(fake_env_dir, f.read())
class BasicTest(BaseTest):
"""Test venv module functionality."""

51
Lib/venv/__init__.py
View File

@ -41,11 +41,12 @@ class EnvBuilder:
environment
:param prompt: Alternative terminal prefix for the environment.
:param upgrade_deps: Update the base venv modules to the latest on PyPI
:param inherit: List of virtualenvs to inherit
"""
def __init__(self, system_site_packages=False, clear=False,
symlinks=False, upgrade=False, with_pip=False, prompt=None,
upgrade_deps=False):
upgrade_deps=False, inherit=None):
self.system_site_packages = system_site_packages
self.clear = clear
self.symlinks = symlinks
@ -55,6 +56,9 @@ class EnvBuilder:
prompt = os.path.basename(os.getcwd())
self.prompt = prompt
self.upgrade_deps = upgrade_deps
if inherit:
assert not isinstance(inherit, str), "inherit must be a iterable of strings, not a string"
self.inherit = inherit
def create(self, env_dir):
"""
@ -63,6 +67,7 @@ class EnvBuilder:
:param env_dir: The target directory to create an environment in.
"""
self.validate_inherit()
env_dir = os.path.abspath(env_dir)
context = self.ensure_directories(env_dir)
# See issue 24875. We need system_site_packages to be False
@ -164,6 +169,37 @@ class EnvBuilder:
f.write('version = %d.%d.%d\n' % sys.version_info[:3])
if self.prompt is not None:
f.write(f'prompt = {self.prompt!r}\n')
if self.inherit:
paths = ':'.join(self.inherit)
f.write(f"inherit = {paths}\n")
def validate_inherit(self):
"""
If inherit was specified, check that it's a python3 virtualenv
with a matching major version
"""
if not self.inherit:
return
if "VIRTUAL_ENV_DISABLE_BASE_CHECKS" in os.environ:
return
import site
for base in self.inherit:
base_prefix = site.get_venv_base_prefix(base)
if not base_prefix:
sys.exit(f"ERROR: {base} is not a python3 virtualenv.")
elif base_prefix != sys.base_prefix:
sys.exit(f"ERROR: {base} does not match current python version {base_prefix}, {sys.base_prefix}")
if os.name == 'nt':
def include_binary(self, f):
if f.endswith(('.pyd', '.dll')):
result = True
else:
result = f.startswith('python') and f.endswith('.exe')
return result
if os.name != 'nt':
def symlink_or_copy(self, src, dst, relative_symlinks_ok=False):
@ -404,11 +440,11 @@ class EnvBuilder:
def create(env_dir, system_site_packages=False, clear=False,
symlinks=False, with_pip=False, prompt=None, upgrade_deps=False):
symlinks=False, with_pip=False, prompt=None, upgrade_deps=False, inherit=None):
"""Create a virtual environment in a directory."""
builder = EnvBuilder(system_site_packages=system_site_packages,
clear=clear, symlinks=symlinks, with_pip=with_pip,
prompt=prompt, upgrade_deps=upgrade_deps)
prompt=prompt, upgrade_deps=upgrade_deps, inherit=inherit)
builder.create(env_dir)
def main(args=None):
@ -476,16 +512,23 @@ def main(args=None):
help='Upgrade core dependencies: {} to the latest '
'version in PyPI'.format(
' '.join(CORE_VENV_DEPS)))
parser.add_argument('--inherit', action="append", metavar="BASE",
default=os.environ.get('VIRTUALENV_INHERIT', '').split(),
help='Inherit packages from another base virtualenv. '
'This option can be repeated for multiple inheritance.')
options = parser.parse_args(args)
if options.upgrade and options.clear:
raise ValueError('you cannot supply --upgrade and --clear together.')
if options.inherit:
options.inherit = [os.path.abspath(path) for path in options.inherit]
builder = EnvBuilder(system_site_packages=options.system_site,
clear=options.clear,
symlinks=options.symlinks,
upgrade=options.upgrade,
with_pip=options.with_pip,
prompt=options.prompt,
upgrade_deps=options.upgrade_deps)
upgrade_deps=options.upgrade_deps,
inherit=options.inherit)
for d in options.dirs:
builder.create(d)

2
Lib/venv/scripts/common/activate
View File

@ -42,7 +42,7 @@ VIRTUAL_ENV="__VENV_DIR__"
export VIRTUAL_ENV
_OLD_VIRTUAL_PATH="$PATH"
PATH="$VIRTUAL_ENV/__VENV_BIN_NAME__:$PATH"
PATH="$($VIRTUAL_ENV/__VENV_BIN_NAME__/python -m site --path):$PATH"
export PATH
# unset PYTHONHOME if set

4
Lib/venv/scripts/posix/activate.csh
View File

@ -3,7 +3,7 @@
# Created by Davide Di Blasi <davidedb@gmail.com>.
# Ported to Python 3.3 venv by Andrew Svetlov <andrew.svetlov@gmail.com>
alias deactivate 'test $?_OLD_VIRTUAL_PATH != 0 && setenv PATH "$_OLD_VIRTUAL_PATH" && unset _OLD_VIRTUAL_PATH; rehash; test $?_OLD_VIRTUAL_PROMPT != 0 && set prompt="$_OLD_VIRTUAL_PROMPT" && unset _OLD_VIRTUAL_PROMPT; unsetenv VIRTUAL_ENV; unsetenv VIRTUAL_ENV_PROMPT; test "\!:*" != "nondestructive" && unalias deactivate'
alias deactivate 'set _old_pev=$?printexitvalue; unset printexitvalue; test $?_OLD_VIRTUAL_PATH != 0 && setenv PATH "$_OLD_VIRTUAL_PATH" && unset _OLD_VIRTUAL_PATH; rehash; test $?_OLD_VIRTUAL_PROMPT != 0 && set prompt="$_OLD_VIRTUAL_PROMPT" && unset _OLD_VIRTUAL_PROMPT; unsetenv VIRTUAL_ENV; unsetenv VIRTUAL_ENV_PROMPT; test "\!:*" != "nondestructive" && unalias deactivate; test $_old_pev != 0 && set printexitvalue; unset _old_pev;'
# Unset irrelevant variables.
deactivate nondestructive
@ -11,7 +11,7 @@ deactivate nondestructive
setenv VIRTUAL_ENV "__VENV_DIR__"
set _OLD_VIRTUAL_PATH="$PATH"
setenv PATH "$VIRTUAL_ENV/__VENV_BIN_NAME__:$PATH"
setenv PATH "`$VIRTUAL_ENV/__VENV_BIN_NAME__/python -m site --path`:$PATH"
set _OLD_VIRTUAL_PROMPT="$prompt"

2
Lib/venv/scripts/posix/activate.fish
View File

@ -33,7 +33,7 @@ deactivate nondestructive
set -gx VIRTUAL_ENV "__VENV_DIR__"
set -gx _OLD_VIRTUAL_PATH $PATH
set -gx PATH "$VIRTUAL_ENV/__VENV_BIN_NAME__" $PATH
set -gx PATH ($VIRTUAL_ENV/__VENV_BIN_NAME__/python -m site --path) $PATH
# Unset PYTHONHOME if set.
if set -q PYTHONHOME

3
Misc/NEWS.d/next/Library/2020-11-24-17-57-35.bpo-42101.09nOry.rst Normal file
View File

@ -0,0 +1,3 @@
Add ability to inherit virtual environment in order to decrease space usage
on disk and speed up creation. This adds the ``--inherit <path>`` option to
the ``venv`` module, as well as ``--path`` option to the ``site`` package.