bpo-38662: ensurepip invokes pip via runpy (GH-18901)

The ensurepip module now invokes pip via the runpy module.
Hence it is no longer tightly coupled with the internal API of the bundled
pip version, allowing easier updates to a newer pip version both
internally and for distributors.

This way, any changes to the internal pip API won't mean ensurepip needs to be
changed as well. Also, distributors can update their pip wheels independent on
CPython release schedule.

Co-Authored-By: Pradyun Gedam <pradyunsg@gmail.com>
Co-Authored-By: Miro Hrončok <miro@hroncok.cz>

Lib/ensurepip/__init__.py

@@ -1,6 +1,7 @@
 import os
 import os.path
 import sys
+import runpy
 import tempfile
 from importlib import resources
 
@@ -26,9 +27,18 @@ def _run_pip(args, additional_paths=None):
     if additional_paths is not None:
         sys.path = additional_paths + sys.path
 
-    # Install the bundled software
-    import pip._internal
-    return pip._internal.main(args)
+    # Invoke pip as if it's the main module, and catch the exit.
+    backup_argv = sys.argv[:]
+    sys.argv[1:] = args
+    try:
+        # run_module() alters sys.modules and sys.argv, but restores them at exit
+        runpy.run_module("pip", run_name="__main__", alter_sys=True)
+    except SystemExit as exc:
+        return exc.code
+    finally:
+        sys.argv[:] = backup_argv
+
+    raise SystemError("pip did not exit, this should never happen")
 
 
 def version():

Misc/NEWS.d/next/Library/2020-03-10-15-32-31.bpo-38662.o1DMXj.rst

@@ -0,0 +1,4 @@
+The ``ensurepip`` module now invokes ``pip`` via the ``runpy`` module.
+Hence it is no longer tightly coupled with the internal API of the bundled
+``pip`` version, allowing easier updates to a newer ``pip`` version both
+internally and for distributors.