bpo-21536: C extensions are no longer linked to libpython (GH-12946)

On Unix, C extensions are no longer linked to libpython.

It is now possible to load a C extension built using a shared library
Python with a statically linked Python.

When Python is embedded, libpython must not be loaded with
RTLD_LOCAL, but RTLD_GLOBAL instead. Previously, using RTLD_LOCAL, it
was already not possible to load C extensions which were not linked
to libpython, like C extensions of the standard library built by the
"*shared*" section of Modules/Setup.

distutils, python-config and python-config.py have been modified.

Doc/distutils/apiref.rst

@@ -277,6 +277,10 @@ the full reference.
    |                        | simply skip the extension.     |                           |
    +------------------------+--------------------------------+---------------------------+
 
+   .. versionchanged:: 3.8
+
+      On Unix, C extensions are no longer linked to libpython.
+
 
 .. class:: Distribution
 

Doc/whatsnew/3.8.rst

@@ -851,16 +851,19 @@ Changes in the Python API
 Changes in the C API
 --------------------
 
+* On Unix, C extensions are no longer linked to libpython. When Python is
+  embedded, ``libpython`` must not be loaded with ``RTLD_LOCAL``, but
+  ``RTLD_GLOBAL`` instead. Previously, using ``RTLD_LOCAL``, it was already not
+  possible to load C extensions which were not linked to ``libpython``, like C
+  extensions of the standard library built by the ``*shared*`` section of
+  ``Modules/Setup``.
+
 * Use of ``#`` variants of formats in parsing or building value (e.g.
   :c:func:`PyArg_ParseTuple`, :c:func:`Py_BuildValue`, :c:func:`PyObject_CallFunction`,
   etc.) without ``PY_SSIZE_T_CLEAN`` defined raises ``DeprecationWarning`` now.
   It will be removed in 3.10 or 4.0.  Read :ref:`arg-parsing` for detail.
   (Contributed by Inada Naoki in :issue:`36381`.)
 
-
-Changes in the C API
---------------------------
-
 * Instances of heap-allocated types (such as those created with
   :c:func:`PyType_FromSpec`) hold a reference to their type object.
   Increasing the reference count of these type objects has been moved from

Lib/distutils/command/build_ext.py

@@ -714,20 +714,5 @@ class build_ext(Command):
                 # don't extend ext.libraries, it may be shared with other
                 # extensions, it is a reference to the original list
                 return ext.libraries + [pythonlib]
-            else:
+
-                return ext.libraries
+        return ext.libraries
-        elif sys.platform == 'darwin':
-            # Don't use the default code below
-            return ext.libraries
-        elif sys.platform[:3] == 'aix':
-            # Don't use the default code below
-            return ext.libraries
-        else:
-            from distutils import sysconfig
-            if sysconfig.get_config_var('Py_ENABLE_SHARED'):
-                pythonlib = 'python{}.{}{}'.format(
-                    sys.hexversion >> 24, (sys.hexversion >> 16) & 0xff,
-                    sysconfig.get_config_var('ABIFLAGS'))
-                return ext.libraries + [pythonlib]
-            else:
-                return ext.libraries

Makefile.pre.in

@@ -1460,7 +1460,7 @@ libinstall:	build_all $(srcdir)/Modules/xxmodule.c
 	-PYTHONPATH=$(DESTDIR)$(LIBDEST) $(RUNSHARED) \
 		$(PYTHON_FOR_BUILD) -m lib2to3.pgen2.driver $(DESTDIR)$(LIBDEST)/lib2to3/PatternGrammar.txt
 
-python-config: $(srcdir)/Misc/python-config.in Misc/python-config.sh
+python-config: $(srcdir)/Misc/python-config.in $(srcdir)/Misc/python-config.sh
 	@ # Substitution happens here, as the completely-expanded BINDIR
 	@ # is not available in configure
 	sed -e "s,@EXENAME@,$(BINDIR)/python$(LDVERSION)$(EXE)," < $(srcdir)/Misc/python-config.in >python-config.py

Misc/NEWS.d/next/Build/2019-04-25-01-51-52.bpo-21536.ACQkiC.rst

@@ -0,0 +1,12 @@
+On Unix, C extensions are no longer linked to libpython.
+
+It is now possible to load a C extension built using a shared library Python
+with a statically linked Python.
+
+When Python is embedded, ``libpython`` must not be loaded with ``RTLD_LOCAL``,
+but ``RTLD_GLOBAL`` instead. Previously, using ``RTLD_LOCAL``, it was already
+not possible to load C extensions which were not linked to ``libpython``, like
+C extensions of the standard library built by the ``*shared*`` section of
+``Modules/Setup``.
+
+distutils, python-config and python-config.py have been modified.

Misc/python-config.in

@@ -47,9 +47,7 @@ for opt in opt_flags:
         print(' '.join(flags))
 
     elif opt in ('--libs', '--ldflags'):
-        libs = ['-lpython' + pyver + sys.abiflags]
+        libs = getvar('LIBS').split() + getvar('SYSLIBS').split()
-        libs += getvar('LIBS').split()
-        libs += getvar('SYSLIBS').split()
         # add the prefix/lib/pythonX.Y/config dir, but only if there is no
         # shared library in prefix/lib/.
         if opt == '--ldflags':

Misc/python-config.sh.in

@@ -41,7 +41,7 @@ LIBM="@LIBM@"
 LIBC="@LIBC@"
 SYSLIBS="$LIBM $LIBC"
 ABIFLAGS="@ABIFLAGS@"
-LIBS="-lpython${VERSION}${ABIFLAGS} @LIBS@ $SYSLIBS"
+LIBS="@LIBS@ $SYSLIBS"
 BASECFLAGS="@BASECFLAGS@"
 LDLIBRARY="@LDLIBRARY@"
 OPT="@OPT@"