bpo-34206: Improve docs and test coverage for pre-init functions (#8023)

- move the Py_Main documentation from the very high level API section
  to the initialization and finalization section
- make it clear that it encapsulates a full Py_Initialize/Finalize
  cycle of its own
- point out that exactly which settings will be read and applied
  correctly when Py_Main is called after a separate runtime
  initialization call is version dependent
- be explicit that Py_IsInitialized can be called prior to
  initialization
- actually test that Py_IsInitialized can be called prior to
  initialization
- flush stdout in the embedding tests that run code so it appears
  in the expected order when running with "-vv"
- make "-vv" on the subinterpreter embedding tests less spammy

---------

Co-authored-by: Carol Willing <carolcode@willingconsulting.com>

Doc/c-api/init.rst

@@ -7,7 +7,8 @@
 Initialization, Finalization, and Threads
 *****************************************
 
-See also the :ref:`Python Initialization Configuration <init-config>`.
+See :ref:`Python Initialization Configuration <init-config>` for details
+on how to configure the interpreter prior to initialization.
 
 .. _pre-init-safe:
 
@@ -21,6 +22,15 @@ a few functions and the :ref:`global configuration variables
 
 The following functions can be safely called before Python is initialized:
 
+* Functions that initialize the interpreter:
+
+  * :c:func:`Py_Initialize`
+  * :c:func:`Py_InitializeEx`
+  * :c:func:`Py_InitializeFromConfig`
+  * :c:func:`Py_BytesMain`
+  * :c:func:`Py_Main`
+  * the runtime pre-initialization functions covered in :ref:`init-config`
+
 * Configuration functions:
 
   * :c:func:`PyImport_AppendInittab`
@@ -32,6 +42,7 @@ The following functions can be safely called before Python is initialized:
   * :c:func:`Py_SetProgramName`
   * :c:func:`Py_SetPythonHome`
   * :c:func:`PySys_ResetWarnOptions`
+  * the configuration functions covered in :ref:`init-config`
 
 * Informative functions:
 
@@ -43,10 +54,12 @@ The following functions can be safely called before Python is initialized:
   * :c:func:`Py_GetCopyright`
   * :c:func:`Py_GetPlatform`
   * :c:func:`Py_GetVersion`
+  * :c:func:`Py_IsInitialized`
 
 * Utilities:
 
   * :c:func:`Py_DecodeLocale`
+  * the status reporting and utility functions covered in :ref:`init-config`
 
 * Memory allocators:
 
@@ -62,11 +75,13 @@ The following functions can be safely called before Python is initialized:
 
 .. note::
 
-   The following functions **should not be called** before
+   Despite their apparent similarity to some of the functions listed above,
-   :c:func:`Py_Initialize`: :c:func:`Py_EncodeLocale`, :c:func:`Py_GetPath`,
+   the following functions **should not be called** before the interpreter has
+   been initialized: :c:func:`Py_EncodeLocale`, :c:func:`Py_GetPath`,
    :c:func:`Py_GetPrefix`, :c:func:`Py_GetExecPrefix`,
    :c:func:`Py_GetProgramFullPath`, :c:func:`Py_GetPythonHome`,
-   :c:func:`Py_GetProgramName` and :c:func:`PyEval_InitThreads`.
+   :c:func:`Py_GetProgramName`, :c:func:`PyEval_InitThreads`, and
+   :c:func:`Py_RunMain`.
 
 
 .. _global-conf-vars:
@@ -346,34 +361,42 @@ Initializing and finalizing the interpreter
    this should be called before using any other Python/C API functions; see
    :ref:`Before Python Initialization <pre-init-safe>` for the few exceptions.
 
-   This initializes
+   This initializes the table of loaded modules (``sys.modules``), and creates
-   the table of loaded modules (``sys.modules``), and creates the fundamental
+   the fundamental modules :mod:`builtins`, :mod:`__main__` and :mod:`sys`.
-   modules :mod:`builtins`, :mod:`__main__` and :mod:`sys`.  It also initializes
+   It also initializes the module search path (``sys.path``). It does not set
-   the module search path (``sys.path``). It does not set ``sys.argv``; use
+   ``sys.argv``; use the :ref:`Python Initialization Configuration <init-config>`
-   the new :c:type:`PyConfig` API of the :ref:`Python Initialization
+   API for that. This is a no-op when called for a second time (without calling
-   Configuration <init-config>` for that.  This is a no-op when called for a
+   :c:func:`Py_FinalizeEx` first).  There is no return value; it is a fatal
-   second time
+   error if the initialization fails.
-   (without calling :c:func:`Py_FinalizeEx` first).  There is no return value; it is a
-   fatal error if the initialization fails.
 
-   Use the :c:func:`Py_InitializeFromConfig` function to customize the
+   Use :c:func:`Py_InitializeFromConfig` to customize the
    :ref:`Python Initialization Configuration <init-config>`.
 
    .. note::
-      On Windows, changes the console mode from ``O_TEXT`` to ``O_BINARY``, which will
+      On Windows, changes the console mode from ``O_TEXT`` to ``O_BINARY``,
-      also affect non-Python uses of the console using the C Runtime.
+      which will also affect non-Python uses of the console using the C Runtime.
 
 
 .. c:function:: void Py_InitializeEx(int initsigs)
 
    This function works like :c:func:`Py_Initialize` if *initsigs* is ``1``. If
-   *initsigs* is ``0``, it skips initialization registration of signal handlers, which
+   *initsigs* is ``0``, it skips initialization registration of signal handlers,
-   might be useful when Python is embedded.
+   which may be useful when CPython is embedded as part of a larger application.
 
-   Use the :c:func:`Py_InitializeFromConfig` function to customize the
+   Use :c:func:`Py_InitializeFromConfig` to customize the
    :ref:`Python Initialization Configuration <init-config>`.
 
 
+.. c:function:: PyStatus Py_InitializeFromConfig(const PyConfig *config)
+
+   Initialize Python from *config* configuration, as described in
+   :ref:`init-from-config`.
+
+   See the :ref:`init-config` section for details on pre-initializing the
+   interpreter, populating the runtime configuration structure, and querying
+   the returned status structure.
+
+
 .. c:function:: int Py_IsInitialized()
 
    Return true (nonzero) when the Python interpreter has been initialized, false
@@ -440,12 +463,111 @@ Initializing and finalizing the interpreter
 
    .. versionadded:: 3.6
 
+
 .. c:function:: void Py_Finalize()
 
    This is a backwards-compatible version of :c:func:`Py_FinalizeEx` that
    disregards the return value.
 
 
+.. c:function:: int Py_BytesMain(int argc, char **argv)
+
+   Similar to :c:func:`Py_Main` but *argv* is an array of bytes strings,
+   allowing the calling application to delegate the text decoding step to
+   the CPython runtime.
+
+   .. versionadded:: 3.8
+
+
+.. c:function:: int Py_Main(int argc, wchar_t **argv)
+
+   The main program for the standard interpreter, encapsulating a full
+   initialization/finalization cycle, as well as additional
+   behaviour to implement reading configurations settings from the environment
+   and command line, and then executing ``__main__`` in accordance with
+   :ref:`using-on-cmdline`.
+
+   This is made available for programs which wish to support the full CPython
+   command line interface, rather than just embedding a Python runtime in a
+   larger application.
+
+   The *argc* and *argv* parameters are similar to those which are passed to a
+   C program's :c:func:`main` function, except that the *argv* entries are first
+   converted to ``wchar_t`` using :c:func:`Py_DecodeLocale`. It is also
+   important to note that the argument list entries may be modified to point to
+   strings other than those passed in (however, the contents of the strings
+   pointed to by the argument list are not modified).
+
+   The return value will be ``0`` if the interpreter exits normally (i.e.,
+   without an exception), ``1`` if the interpreter exits due to an exception,
+   or ``2`` if the argument list does not represent a valid Python command
+   line.
+
+   Note that if an otherwise unhandled :exc:`SystemExit` is raised, this
+   function will not return ``1``, but exit the process, as long as
+   ``Py_InspectFlag`` is not set. If ``Py_InspectFlag`` is set, execution will
+   drop into the interactive Python prompt, at which point a second otherwise
+   unhandled :exc:`SystemExit` will still exit the process, while any other
+   means of exiting will set the return value as described above.
+
+   In terms of the CPython runtime configuration APIs documented in the
+   :ref:`runtime configuration <init-config>` section (and without accounting
+   for error handling), ``Py_Main`` is approximately equivalent to::
+
+      PyConfig config;
+      PyConfig_InitPythonConfig(&config);
+      PyConfig_SetArgv(&config, argc, argv);
+      Py_InitializeFromConfig(&config);
+      PyConfig_Clear(&config);
+
+      Py_RunMain();
+
+   In normal usage, an embedding application will call this function
+   *instead* of calling :c:func:`Py_Initialize`, :c:func:`Py_InitializeEx` or
+   :c:func:`Py_InitializeFromConfig` directly, and all settings will be applied
+   as described elsewhere in this documentation. If this function is instead
+   called *after* a preceding runtime initialization API call, then exactly
+   which environmental and command line configuration settings will be updated
+   is version dependent (as it depends on which settings correctly support
+   being modified after they have already been set once when the runtime was
+   first initialized).
+
+
+.. c:function:: int Py_RunMain(void)
+
+   Executes the main module in a fully configured CPython runtime.
+
+   Executes the command (:c:member:`PyConfig.run_command`), the script
+   (:c:member:`PyConfig.run_filename`) or the module
+   (:c:member:`PyConfig.run_module`) specified on the command line or in the
+   configuration. If none of these values are set, runs the interactive Python
+   prompt (REPL) using the ``__main__`` module's global namespace.
+
+   If :c:member:`PyConfig.inspect` is not set (the default), the return value
+   will be ``0`` if the interpreter exits normally (that is, without raising
+   an exception), or ``1`` if the interpreter exits due to an exception. If an
+   otherwise unhandled :exc:`SystemExit` is raised, the function will immediately
+   exit the process instead of returning ``1``.
+
+   If :c:member:`PyConfig.inspect` is set (such as when the :option:`-i` option
+   is used), rather than returning when the interpreter exits, execution will
+   instead resume in an interactive Python prompt (REPL) using the ``__main__``
+   module's global namespace. If the interpreter exited with an exception, it
+   is immediately raised in the REPL session. The function return value is
+   then determined by the way the *REPL session* terminates: returning ``0``
+   if the session terminates without raising an unhandled exception, exiting
+   immediately for an unhandled :exc:`SystemExit`, and returning ``1`` for
+   any other unhandled exception.
+
+   This function always finalizes the Python interpreter regardless of whether
+   it returns a value or immediately exits the process due to an unhandled
+   :exc:`SystemExit` exception.
+
+   See :ref:`Python Configuration <init-python-config>` for an example of a
+   customized Python that always runs in isolated mode using
+   :c:func:`Py_RunMain`.
+
+
 Process-wide parameters
 =======================
 

Doc/c-api/init_config.rst

@@ -1356,14 +1356,13 @@ the :option:`-X` command line option.
    The ``show_alloc_count`` field has been removed.
 
 
+.. _init-from-config:
+
 Initialization with PyConfig
 ----------------------------
 
-Function to initialize Python:
+Initializing the interpreter from a populated configuration struct is handled
-
+by calling :c:func:`Py_InitializeFromConfig`.
-.. c:function:: PyStatus Py_InitializeFromConfig(const PyConfig *config)
-
-   Initialize Python from *config* configuration.
 
 The caller is responsible to handle exceptions (error or exit) using
 :c:func:`PyStatus_Exception` and :c:func:`Py_ExitStatusException`.
@@ -1835,26 +1834,6 @@ return ``-1`` on error:
     }
 
 
-Py_RunMain()
-============
-
-.. c:function:: int Py_RunMain(void)
-
-   Execute the command (:c:member:`PyConfig.run_command`), the script
-   (:c:member:`PyConfig.run_filename`) or the module
-   (:c:member:`PyConfig.run_module`) specified on the command line or in the
-   configuration.
-
-   By default and when if :option:`-i` option is used, run the REPL.
-
-   Finally, finalizes Python and returns an exit status that can be passed to
-   the ``exit()`` function.
-
-See :ref:`Python Configuration <init-python-config>` for an example of
-customized Python always running in isolated mode using
-:c:func:`Py_RunMain`.
-
-
 Runtime Python configuration API
 ================================
 

Doc/c-api/veryhigh.rst

@@ -25,30 +25,6 @@ are only passed to these functions if it is certain that they were created by
 the same library that the Python runtime is using.
 
 
-.. c:function:: int Py_Main(int argc, wchar_t **argv)
-
-   The main program for the standard interpreter.  This is made available for
-   programs which embed Python.  The *argc* and *argv* parameters should be
-   prepared exactly as those which are passed to a C program's :c:func:`main`
-   function (converted to wchar_t according to the user's locale).  It is
-   important to note that the argument list may be modified (but the contents of
-   the strings pointed to by the argument list are not). The return value will
-   be ``0`` if the interpreter exits normally (i.e., without an exception),
-   ``1`` if the interpreter exits due to an exception, or ``2`` if the parameter
-   list does not represent a valid Python command line.
-
-   Note that if an otherwise unhandled :exc:`SystemExit` is raised, this
-   function will not return ``1``, but exit the process, as long as
-   :c:member:`PyConfig.inspect` is zero.
-
-
-.. c:function:: int Py_BytesMain(int argc, char **argv)
-
-   Similar to :c:func:`Py_Main` but *argv* is an array of bytes strings.
-
-   .. versionadded:: 3.8
-
-
 .. c:function:: int PyRun_AnyFile(FILE *fp, const char *filename)
 
    This is a simplified interface to :c:func:`PyRun_AnyFileExFlags` below, leaving

Lib/test/test_embed.py

@@ -168,7 +168,8 @@ class EmbeddingTestsMixin:
             # Parse the line from the loop.  The first line is the main
             # interpreter and the 3 afterward are subinterpreters.
             interp = Interp(*match.groups())
-            if support.verbose > 1:
+            if support.verbose > 2:
+                # 5 lines per pass is super-spammy, so limit that to -vvv
                 print(interp)
             self.assertTrue(interp.interp)
             self.assertTrue(interp.tstate)
@@ -279,6 +280,10 @@ class EmbeddingTests(EmbeddingTestsMixin, unittest.TestCase):
         """
         env = dict(os.environ, PYTHONPATH=os.pathsep.join(sys.path))
         out, err = self.run_embedded_interpreter("test_pre_initialization_api", env=env)
+        if support.verbose > 1:
+            print()
+            print(out)
+            print(err)
         if MS_WINDOWS:
             expected_path = self.test_exe
         else:
@@ -296,6 +301,10 @@ class EmbeddingTests(EmbeddingTestsMixin, unittest.TestCase):
         env['PYTHONPATH'] = os.pathsep.join(sys.path)
         out, err = self.run_embedded_interpreter(
                         "test_pre_initialization_sys_options", env=env)
+        if support.verbose > 1:
+            print()
+            print(out)
+            print(err)
         expected_output = (
             "sys.warnoptions: ['once', 'module', 'default']\n"
             "sys._xoptions: {'not_an_option': '1', 'also_not_an_option': '2'}\n"

Misc/NEWS.d/next/C API/2018-06-30-21-48-16.bpo-34008.2Wjtm0.rst

@@ -0,0 +1,2 @@
+Added ``Py_IsInitialized`` to the list of APIs that are safe to call before
+the interpreter is initialized, and updated the embedding tests to cover it.

Misc/NEWS.d/next/Documentation/2018-07-04-20-35-25.bpo-34008.bqecIb.rst

@@ -0,0 +1,8 @@
+The :c:func:`Py_Main` documentation moved from the "Very High Level API" section to the
+"Initialization and Finalization" section.
+
+Also make it explicit that we expect ``Py_Main`` to typically be called instead
+of ``Py_Initialize`` rather than after it (since ``Py_Main`` makes its own
+call to ``Py_Initialize``). Document that calling both is
+supported but is version dependent on which settings
+will be applied correctly.

Programs/_testembed.c

@@ -311,14 +311,36 @@ static int test_pre_initialization_api(void)
     _Py_EMBED_PREINIT_CHECK("Checking Py_SetProgramName\n");
     Py_SetProgramName(program);
 
+    _Py_EMBED_PREINIT_CHECK("Checking !Py_IsInitialized pre-initialization\n");
+    if (Py_IsInitialized()) {
+        fprintf(stderr, "Fatal error: initialized before initialization!\n");
+        return 1;
+    }
+
     _Py_EMBED_PREINIT_CHECK("Initializing interpreter\n");
     Py_Initialize();
+
+    _Py_EMBED_PREINIT_CHECK("Checking Py_IsInitialized post-initialization\n");
+    if (!Py_IsInitialized()) {
+        fprintf(stderr, "Fatal error: not initialized after initialization!\n");
+        return 1;
+    }
+
     _Py_EMBED_PREINIT_CHECK("Check sys module contents\n");
-    PyRun_SimpleString("import sys; "
+    PyRun_SimpleString(
-                       "print('sys.executable:', sys.executable)");
+        "import sys; "
+        "print('sys.executable:', sys.executable); "
+        "sys.stdout.flush(); "
+    );
     _Py_EMBED_PREINIT_CHECK("Finalizing interpreter\n");
     Py_Finalize();
 
+    _Py_EMBED_PREINIT_CHECK("Checking !Py_IsInitialized post-finalization\n");
+    if (Py_IsInitialized()) {
+        fprintf(stderr, "Fatal error: still initialized after finalization!\n");
+        return 1;
+    }
+
     _Py_EMBED_PREINIT_CHECK("Freeing memory allocated by Py_DecodeLocale\n");
     PyMem_RawFree(program);
     return 0;
@@ -364,12 +386,15 @@ static int test_pre_initialization_sys_options(void)
     _Py_EMBED_PREINIT_CHECK("Initializing interpreter\n");
     _testembed_Py_InitializeFromConfig();
     _Py_EMBED_PREINIT_CHECK("Check sys module contents\n");
-    PyRun_SimpleString("import sys; "
+    PyRun_SimpleString(
-                       "print('sys.warnoptions:', sys.warnoptions); "
+        "import sys; "
-                       "print('sys._xoptions:', sys._xoptions); "
+        "print('sys.warnoptions:', sys.warnoptions); "
-                       "warnings = sys.modules['warnings']; "
+        "print('sys._xoptions:', sys._xoptions); "
-                       "latest_filters = [f[0] for f in warnings.filters[:3]]; "
+        "warnings = sys.modules['warnings']; "
-                       "print('warnings.filters[:3]:', latest_filters)");
+        "latest_filters = [f[0] for f in warnings.filters[:3]]; "
+        "print('warnings.filters[:3]:', latest_filters); "
+        "sys.stdout.flush(); "
+    );
     _Py_EMBED_PREINIT_CHECK("Finalizing interpreter\n");
     Py_Finalize();