bpo-33649: Fix asyncio-dev (GH-9324)

Doc/library/asyncio-api-index.rst

@@ -15,7 +15,7 @@ Utilities to run asyncio programs, create Tasks, and
 await on multiple things with timeouts.
 
 .. list-table::
-    :widths: 30 70
+    :widths: 50 50
 
     * - :func:`run`
       - Create event loop, run a coroutine, close the loop.
@@ -71,7 +71,7 @@ implement connection pools, and pub/sub patterns.
 
 
 .. list-table::
-    :widths: 30 70
+    :widths: 50 50
 
     * - :class:`Queue`
       - A FIFO queue.
@@ -97,7 +97,7 @@ Subprocesses
 Utilities to spawn subprocesses and run shell commands.
 
 .. list-table::
-    :widths: 30 70
+    :widths: 50 50
 
     * - ``await`` :func:`create_subprocess_exec`
       - Create a subprocess.
@@ -120,7 +120,7 @@ Streams
 High-level APIs to work with network IO.
 
 .. list-table::
-    :widths: 30 70
+    :widths: 50 50
 
     * - ``await`` :func:`open_connection`
       -  Establish a TCP connection.
@@ -155,7 +155,7 @@ Synchronization
 Threading-like synchronization primitives that can be used in Tasks.
 
 .. list-table::
-    :widths: 30 70
+    :widths: 50 50
 
     * - :class:`Lock`
       - A mutex lock.
@@ -185,7 +185,7 @@ Exceptions
 ==========
 
 .. list-table::
-    :widths: 30 70
+    :widths: 50 50
 
 
     * - :exc:`asyncio.TimeoutError`

Doc/library/asyncio-dev.rst

@@ -2,415 +2,237 @@
 
 .. _asyncio-dev:
 
-Develop with asyncio
+=======================
-====================
+Developing with asyncio
+=======================
 
-Asynchronous programming is different than classical "sequential" programming.
+Asynchronous programming is different from classical "sequential"
-This page lists common traps and explains how to avoid them.
+programming.
+
+This page lists common mistakes and traps and explains how
+to avoid them.
 
 
 .. _asyncio-debug-mode:
 
-Debug mode of asyncio
+Debug Mode
----------------------
+==========
 
-The implementation of :mod:`asyncio` has been written for performance.
+By default asyncio runs in production mode.  In order to ease
-In order to ease the development of asynchronous code, you may wish to
+the development asyncio has a *debug mode*.
-enable *debug mode*.
 
-To enable all debug checks for an application:
+To enable debugging for an application:
 
-* Enable the asyncio debug mode globally by setting the environment variable
+* Enable the debug mode globally by setting the environment variable
-  :envvar:`PYTHONASYNCIODEBUG` to ``1``, using ``-X dev`` command line option
+  :envvar:`PYTHONASYNCIODEBUG` to ``1``.
-  (see the :option:`-X` option), or by calling
-  :meth:`loop.set_debug`.
-* Set the log level of the :ref:`asyncio logger <asyncio-logger>` to
-  :py:data:`logging.DEBUG`. For example, call
-  ``logging.basicConfig(level=logging.DEBUG)`` at startup.
-* Configure the :mod:`warnings` module to display :exc:`ResourceWarning`
-  warnings. For example, use the ``-Wdefault`` command line option of Python to
-  display them.
 
-Examples debug checks:
+* Alternatively, the debug mode can be enabled by using the ``-X dev``
+  command line option for Python (see the :option:`-X` option).
 
-* Log :ref:`coroutines defined but never "yielded from"
+* Yet another way to enable the debug mode is by calling
-  <asyncio-coroutine-not-scheduled>`
+  :meth:`loop.set_debug` or by passing ``debug=True`` to
-* :meth:`loop.call_soon` and :meth:`loop.call_at` methods
+  :func:`asyncio.run`.
-  raise an exception if they are called from the wrong thread.
+
-* Log the execution time of the selector
+In addition to enabling debug mode, consider also:
-* Log callbacks taking more than 100 ms to be executed. The
+
+* setting the log level of the :ref:`asyncio logger <asyncio-logger>` to
+  :py:data:`logging.DEBUG`, for example the following snippet of code
+  can be run at startup of the application::
+
+    logging.basicConfig(level=logging.DEBUG)
+
+* configuring the :mod:`warnings` module to display
+  :exc:`ResourceWarning` warnings.  One way of doing that is by
+  using the ``-Wdefault`` command line option.
+
+
+In asyncio debug mode the following checks are performed:
+
+* Log :ref:`coroutines that were not awaited
+  <asyncio-coroutine-not-scheduled>`; this mitigates the "forgotten
+  await" pitfall.
+
+* Many non-treadsafe asyncio APIs (such as :meth:`loop.call_soon` and
+  :meth:`loop.call_at` methods) raise an exception if they are called
+  from a wrong thread.
+
+* Log the execution time of the IO selector if it takes too long to
+  perform an IO operation.
+
+* Log callbacks taking longer than 100 ms to be executed.  The
   :attr:`loop.slow_callback_duration` attribute is the minimum
   duration in seconds of "slow" callbacks.
-* :exc:`ResourceWarning` warnings are emitted when transports and event loops
-  are :ref:`not closed explicitly <asyncio-close-transports>`.
-
-.. versionchanged:: 3.7
-
-   The new ``-X dev`` command line option can now also be used to enable
-   the debug mode.
-
-.. seealso::
-
-   The :meth:`loop.set_debug` method and the :ref:`asyncio logger
-   <asyncio-logger>`.
-
-
-Cancellation
-------------
-
-Cancellation of tasks is not common in classic programming. In asynchronous
-programming, not only is it something common, but you have to prepare your
-code to handle it.
-
-Futures and tasks can be cancelled explicitly with their :meth:`Future.cancel`
-method. The :func:`wait_for` function cancels the waited task when the timeout
-occurs. There are many other cases where a task can be cancelled indirectly.
-
-Don't call :meth:`~Future.set_result` or :meth:`~Future.set_exception` method
-of :class:`Future` if the future is cancelled: it would fail with an exception.
-For example, write::
-
-    if not fut.cancelled():
-        fut.set_result('done')
-
-Don't schedule directly a call to the :meth:`~Future.set_result` or the
-:meth:`~Future.set_exception` method of a future with
-:meth:`loop.call_soon`: the future can be cancelled before its method
-is called.
-
-If you wait for a future, you should check early if the future was cancelled to
-avoid useless operations. Example::
-
-    async def slow_operation(fut):
-        if fut.cancelled():
-            return
-        # ... slow computation ...
-        await fut
-        # ...
-
-The :func:`shield` function can also be used to ignore cancellation.
 
 
 .. _asyncio-multithreading:
 
-Concurrency and multithreading
+Concurrency and Multithreading
-------------------------------
+==============================
 
 An event loop runs in a thread (typically the main thread) and executes
-all callbacks and tasks in its thread.  While a task is running in the
+all callbacks and Tasks in its thread.  While a Task is running in the
-event loop, no other tasks may run in the same thread. When a task
+event loop, no other Tasks can run in the same thread.  When a Task
-executes an ``await`` expression, the running task gets suspended, and the
+executes an ``await`` expression, the running Task gets suspended, and
-event loop executes the next task. Prior to suspending the task, the awaiting
+the event loop executes the next Task.
-chain is checked, and if the chain ends with a future, the running task is
-not suspended.
 
-To schedule a callback from a different thread, the
+To schedule a callback from a different OS thread, the
 :meth:`loop.call_soon_threadsafe` method should be used. Example::
 
     loop.call_soon_threadsafe(callback, *args)
 
-Most asyncio objects are not thread safe. You should only worry if you access
+Almost all asyncio objects are not thread safe, which is typically
-objects outside the event loop. For example, to cancel a future, don't call
+not a problem unless there is code that works with them from outside
-directly its :meth:`Future.cancel` method, but::
+of a Task or a callback.  If there's a need for such code to call a
+low-level asyncio API, the :meth:`loop.call_soon_threadsafe` method
+should be used, e.g.::
 
     loop.call_soon_threadsafe(fut.cancel)
 
-To handle signals and to execute subprocesses, the event loop must be run in
+To schedule a coroutine object from a different OS thread, the
-the main thread.
-
-To schedule a coroutine object from a different thread, the
 :func:`run_coroutine_threadsafe` function should be used. It returns a
 :class:`concurrent.futures.Future` to access the result::
 
+     async def coro_func():
+          return await asyncio.sleep(1, 42)
+
+     # Later in another OS thread:
+
      future = asyncio.run_coroutine_threadsafe(coro_func(), loop)
-     result = future.result(timeout)  # Wait for the result with a timeout
+     # Wait for the result:
+     result = future.result()
+
+To handle signals and to execute subprocesses, the event loop must be
+run in the main thread.
 
 The :meth:`loop.run_in_executor` method can be used with a
-:class:`concurrent.futures.ThreadPoolExecutor` to execute a callback in
+:class:`concurrent.futures.ThreadPoolExecutor` to execute
-different thread so as not to block the event loop's main thread.
+blocking code in a different OS thread without blocking the OS thread
-
+that the event loop runs in.
-.. seealso::
-
-   The :ref:`Synchronization primitives <asyncio-sync>` section describes ways
-   to synchronize tasks.
-
-   The :ref:`Subprocess and threads <asyncio-subprocess-threads>` section lists
-   asyncio limitations to run subprocesses from different threads.
-
-
 
 
 .. _asyncio-handle-blocking:
 
-Handle blocking functions correctly
+Running Blocking Code
------------------------------------
+=====================
 
-Blocking functions should not be called directly. For example, if a function
+Blocking (CPU-bound) code should not be called directly.  For example,
-blocks for 1 second, other tasks are delayed by 1 second which can have an
+if a function performs a CPU-intensive calculation for 1 second,
-important impact on reactivity.
+all concurrent asyncio Tasks and IO operations would be delayed
+by 1 second.
 
-For networking and subprocesses, the :mod:`asyncio` module provides high-level
+An executor can be used to run a task in a different thread or even in
-APIs like :ref:`protocols <asyncio-protocol>`.
+a different process to avoid blocking block the OS thread with the
-
+event loop.  See the :meth:`loop.run_in_executor` method for more
-An executor can be used to run a task in a different thread or even in a
+details.
-different process, to not block the thread of the event loop. See the
-:meth:`loop.run_in_executor` method.
-
-.. seealso::
-
-   The :ref:`Delayed calls <asyncio-delayed-calls>` section details how the
-   event loop handles time.
 
 
 .. _asyncio-logger:
 
 Logging
--------
+=======
 
-The :mod:`asyncio` module logs information with the :mod:`logging` module in
+asyncio uses the :mod:`logging` module and all logging is performed
-the logger ``'asyncio'``.
+via the ``"asyncio"`` logger.
 
-The default log level for the :mod:`asyncio` module is :py:data:`logging.INFO`.
+The default log level is :py:data:`logging.INFO`, which can easily be
-For those not wanting such verbosity from :mod:`asyncio` the log level can
+adjusted::
-be changed.  For example, to change the level to :py:data:`logging.WARNING`:
 
-.. code-block:: none
+   logging.getLogger("asyncio").setLevel(logging.WARNING)
-
-   logging.getLogger('asyncio').setLevel(logging.WARNING)
 
 
 .. _asyncio-coroutine-not-scheduled:
 
-Detect coroutine objects never scheduled
+Detect never awaited coroutines
-----------------------------------------
+===============================
 
-When a coroutine function is called and its result is not passed to
+When a coroutine is called (e.g. ``coro()`` instead of ``await coro()``)
-:meth:`asyncio.create_task` the execution of the coroutine object will
+the call is not wrapped with :meth:`asyncio.create_task`, the execution
-never be scheduled which is probably a bug. Using ``asyncio.create_task`` is
+of the coroutine object will never be scheduled.  For example::
-preferred to the low level :func:`ensure_future` and :meth:`loop.create_task`
-methods. :ref:`Enable the debug mode of asyncio <asyncio-debug-mode>`
-to :ref:`log a warning <asyncio-logger>` to detect it.
-
-Example with the bug::
 
     import asyncio
 
     async def test():
         print("never scheduled")
 
+    async def main():
+        test()
+
+    asyncio.run(main())
+
+Output::
+
+  test.py:7: RuntimeWarning: coroutine 'test' was never awaited
     test()
 
 Output in debug mode::
 
-    Coroutine test() at test.py:3 was never yielded from
+  test.py:7: RuntimeWarning: coroutine 'test' was never awaited
-    Coroutine object created at (most recent call last):
+  Coroutine created at (most recent call last)
-      File "test.py", line 7, in <module>
+    File "../t.py", line 9, in <module>
-        test()
+      asyncio.run(main(), debug=True)
 
-The fix is to call the :meth:`asyncio.create_task` function. Using
+    < .. >
-``asyncio.create_task`` is preferred to the low level :func:`ensure_future` and
-:meth:`loop.create_task` methods.
 
-.. seealso::
+    File "../t.py", line 7, in main
+      test()
+    test()
 
-   :ref:`Pending task destroyed <asyncio-pending-task-destroyed>`.
+The usual fix is to either await the coroutine or call the
+:meth:`asyncio.create_task` function::
+
+    async def main():
+        await test()
 
 
-Detect exceptions never consumed
+Detect never consumed exceptions
---------------------------------
+================================
 
-Python usually calls :func:`sys.excepthook` on unhandled exceptions. If
+If a :meth:`Future.set_exception` is called but the Future object is
-:meth:`Future.set_exception` is called, but the exception is never consumed,
+never awaited on, the exception would never be propagated to the
-:func:`sys.excepthook` is not called. Instead, :ref:`a log is emitted
+user code.  In this case, asyncio would emit a log message when the
-<asyncio-logger>` when the future is deleted by the garbage collector, with the
+Future object is garbage collected.
-traceback where the exception was raised.
 
-Example of unhandled exception::
+Example of an unhandled exception::
 
     import asyncio
 
-    @asyncio.coroutine
+    async def bug():
-    def bug():
         raise Exception("not consumed")
 
-    loop = asyncio.get_event_loop()
+    async def main():
-    asyncio.ensure_future(bug())
+        asyncio.create_task(bug())
-    loop.run_forever()
+
-    loop.close()
+    asyncio.run(main())
 
 Output::
 
     Task exception was never retrieved
-    future: <Task finished coro=<coro() done, defined at asyncio/coroutines.py:139> exception=Exception('not consumed',)>
+    future: <Task finished coro=<bug() done, defined at test.py:3>
+      exception=Exception('not consumed')>
+
     Traceback (most recent call last):
-      File "asyncio/tasks.py", line 237, in _step
+      File "test.py", line 4, in bug
-        result = next(coro)
-      File "asyncio/coroutines.py", line 141, in coro
-        res = func(*args, **kw)
-      File "test.py", line 5, in bug
         raise Exception("not consumed")
     Exception: not consumed
 
-:ref:`Enable the debug mode of asyncio <asyncio-debug-mode>` to get the
+:ref:`Enable the debug mode <asyncio-debug-mode>` to get the
-traceback where the task was created. Output in debug mode::
+traceback where the task was created::
+
+    asyncio.run(main(), debug=True)
+
+Output in debug mode::
 
     Task exception was never retrieved
-    future: <Task finished coro=<bug() done, defined at test.py:3> exception=Exception('not consumed',) created at test.py:8>
+    future: <Task finished coro=<bug() done, defined at test.py:3>
+        exception=Exception('not consumed') created at asyncio/tasks.py:321>
+
     source_traceback: Object created at (most recent call last):
-      File "test.py", line 8, in <module>
+      File "../t.py", line 9, in <module>
-        asyncio.ensure_future(bug())
+        asyncio.run(main(), debug=True)
+
+    < .. >
+
     Traceback (most recent call last):
-      File "asyncio/tasks.py", line 237, in _step
+      File "../t.py", line 4, in bug
-        result = next(coro)
-      File "asyncio/coroutines.py", line 79, in __next__
-        return next(self.gen)
-      File "asyncio/coroutines.py", line 141, in coro
-        res = func(*args, **kw)
-      File "test.py", line 5, in bug
         raise Exception("not consumed")
     Exception: not consumed
-
-There are different options to fix this issue. The first option is to chain the
-coroutine in another coroutine and use classic try/except::
-
-    async def handle_exception():
-        try:
-            await bug()
-        except Exception:
-            print("exception consumed")
-
-    loop = asyncio.get_event_loop()
-    asyncio.ensure_future(handle_exception())
-    loop.run_forever()
-    loop.close()
-
-Another option is to use the :meth:`asyncio.run` function::
-
-    asyncio.run(bug())
-
-.. seealso::
-
-   The :meth:`Future.exception` method.
-
-
-Chain coroutines correctly
---------------------------
-
-When a coroutine function calls other coroutine functions and tasks, they
-should be chained explicitly with ``await``. Otherwise, the execution is
-not guaranteed to be sequential.
-
-Example with different bugs using :func:`asyncio.sleep` to simulate slow
-operations::
-
-    import asyncio
-
-    async def create():
-        await asyncio.sleep(3.0)
-        print("(1) create file")
-
-    async def write():
-        await asyncio.sleep(1.0)
-        print("(2) write into file")
-
-    async def close():
-        print("(3) close file")
-
-    async def test():
-        asyncio.ensure_future(create())
-        asyncio.ensure_future(write())
-        asyncio.ensure_future(close())
-        await asyncio.sleep(2.0)
-        loop.stop()
-
-    loop = asyncio.get_event_loop()
-    asyncio.ensure_future(test())
-    loop.run_forever()
-    print("Pending tasks at exit: %s" % asyncio.Task.all_tasks(loop))
-    loop.close()
-
-Expected output:
-
-.. code-block:: none
-
-    (1) create file
-    (2) write into file
-    (3) close file
-    Pending tasks at exit: set()
-
-Actual output:
-
-.. code-block:: none
-
-    (3) close file
-    (2) write into file
-    Pending tasks at exit: {<Task pending create() at test.py:7 wait_for=<Future pending cb=[Task._wakeup()]>>}
-    Task was destroyed but it is pending!
-    task: <Task pending create() done at test.py:5 wait_for=<Future pending cb=[Task._wakeup()]>>
-
-The loop stopped before the ``create()`` finished, ``close()`` has been called
-before ``write()``, whereas coroutine functions were called in this order:
-``create()``, ``write()``, ``close()``.
-
-To fix the example, tasks must be marked with ``await``::
-
-    async def test():
-        await asyncio.ensure_future(create())
-        await asyncio.ensure_future(write())
-        await asyncio.ensure_future(close())
-        await asyncio.sleep(2.0)
-        loop.stop()
-
-Or without ``asyncio.ensure_future()``::
-
-    async def test():
-        await create()
-        await write()
-        await close()
-        await asyncio.sleep(2.0)
-        loop.stop()
-
-
-.. _asyncio-pending-task-destroyed:
-
-Pending task destroyed
-----------------------
-
-If a pending task is destroyed, the execution of its wrapped :ref:`coroutine
-<coroutine>` did not complete. It is probably a bug and so a warning is logged.
-
-Example of log:
-
-.. code-block:: none
-
-    Task was destroyed but it is pending!
-    task: <Task pending coro=<kill_me() done, defined at test.py:5> wait_for=<Future pending cb=[Task._wakeup()]>>
-
-:ref:`Enable the debug mode of asyncio <asyncio-debug-mode>` to get the
-traceback where the task was created. Example of log in debug mode:
-
-.. code-block:: none
-
-    Task was destroyed but it is pending!
-    source_traceback: Object created at (most recent call last):
-      File "test.py", line 15, in <module>
-        task = asyncio.ensure_future(coro, loop=loop)
-    task: <Task pending coro=<kill_me() done, defined at test.py:5> wait_for=<Future pending cb=[Task._wakeup()] created at test.py:7> created at test.py:15>
-
-
-.. seealso::
-
-   :ref:`Detect coroutine objects never scheduled <asyncio-coroutine-not-scheduled>`.
-
-.. _asyncio-close-transports:
-
-Close transports and event loops
---------------------------------
-
-When a transport is no longer needed, call its ``close()`` method to release
-resources. Event loops must also be closed explicitly.
-
-If a transport or an event loop is not closed explicitly, a
-:exc:`ResourceWarning` warning will be emitted in its destructor. By default,
-:exc:`ResourceWarning` warnings are ignored. The :ref:`Debug mode of asyncio
-<asyncio-debug-mode>` section explains how to display them.

Doc/library/asyncio-eventloop.rst

@@ -1075,6 +1075,11 @@ Enabling debug mode
 
    Set the debug mode of the event loop.
 
+   .. versionchanged:: 3.7
+
+      The new ``-X dev`` command line option can now also be used
+      to enable the debug mode.
+
 .. seealso::
 
    The :ref:`debug mode of asyncio <asyncio-debug-mode>`.

Doc/library/asyncio-future.rst

@@ -121,6 +121,16 @@ Future Object
       or an exception set with :meth:`set_result` or
       :meth:`set_exception` calls.
 
+   .. method:: cancelled()
+
+      Return ``True`` if the Future was *cancelled*.
+
+      The method is usually used to check if a Future is not
+      *cancelled* before setting a result or an exception for it::
+
+          if not fut.cancelled():
+              fut.set_result(42)
+
    .. method:: add_done_callback(callback, *, context=None)
 
       Add a callback to be run when the Future is *done*.
@@ -180,10 +190,6 @@ Future Object
 
       .. versionadded:: 3.7
 
-   .. method:: cancelled()
-
-      Return ``True`` if the Future was *cancelled*.
-
 
 This example creates a Future object, creates and schedules an
 asynchronous Task to set result for the Future, and waits until

Doc/library/asyncio.rst

@@ -32,8 +32,9 @@ asyncio provides a set of **high-level** APIs to:
 as well as **low-level** APIs for *library and framework developers* to:
 
 * create and manage :ref:`event loops <asyncio-event-loop>`, which
-  provide asynchronous APIs for networking, subprocesses, OS signals,
+  provide asynchronous APIs for :meth:`networking <loop.create_server>`,
-  etc;
+  running :meth:`subprocesses <loop.subprocess_exec>`,
+  handling :meth:`OS signals <loop.add_signal_handler>`, etc;
 
 * implement efficient protocols using
   :ref:`transports <asyncio-transports-protocols>`;
@@ -75,11 +76,3 @@ Reference
 
    asyncio-api-index.rst
    asyncio-dev.rst
-
-
-.. seealso::
-
-   The :mod:`asyncio` module was proposed in :PEP:`3156`.
-   Since the acceptance of the PEP many new APIs were added and many
-   original APIs were altered.  The PEP should be treated as a
-   historical document.

Doc/library/ipc.rst

@@ -1,11 +1,11 @@
 .. _ipc:
 
 *****************************************
-Interprocess Communication and Networking
+Networking and Interprocess Communication
 *****************************************
 
-The modules described in this chapter provide mechanisms for different processes
+The modules described in this chapter provide mechanisms for
-to communicate.
+networking and inter-processes communication.
 
 Some modules only work for two processes that are on the same machine, e.g.
 :mod:`signal` and :mod:`mmap`.  Other modules support networking protocols
@@ -15,6 +15,7 @@ The list of modules described in this chapter is:
 
 
 .. toctree::
+   :maxdepth: 1
 
    asyncio.rst
    socket.rst