diff --git a/Doc/library/asyncio-api-index.rst b/Doc/library/asyncio-api-index.rst index a4d0e1ba32e..dc691000471 100644 --- a/Doc/library/asyncio-api-index.rst +++ b/Doc/library/asyncio-api-index.rst @@ -15,7 +15,7 @@ Utilities to run asyncio programs, create Tasks, and await on multiple things with timeouts. .. list-table:: - :widths: 30 70 + :widths: 50 50 * - :func:`run` - Create event loop, run a coroutine, close the loop. @@ -71,7 +71,7 @@ implement connection pools, and pub/sub patterns. .. list-table:: - :widths: 30 70 + :widths: 50 50 * - :class:`Queue` - A FIFO queue. @@ -97,7 +97,7 @@ Subprocesses Utilities to spawn subprocesses and run shell commands. .. list-table:: - :widths: 30 70 + :widths: 50 50 * - ``await`` :func:`create_subprocess_exec` - Create a subprocess. @@ -120,7 +120,7 @@ Streams High-level APIs to work with network IO. .. list-table:: - :widths: 30 70 + :widths: 50 50 * - ``await`` :func:`open_connection` - Establish a TCP connection. @@ -155,7 +155,7 @@ Synchronization Threading-like synchronization primitives that can be used in Tasks. .. list-table:: - :widths: 30 70 + :widths: 50 50 * - :class:`Lock` - A mutex lock. @@ -185,7 +185,7 @@ Exceptions ========== .. list-table:: - :widths: 30 70 + :widths: 50 50 * - :exc:`asyncio.TimeoutError` diff --git a/Doc/library/asyncio-dev.rst b/Doc/library/asyncio-dev.rst index 87abe92c294..e8153186760 100644 --- a/Doc/library/asyncio-dev.rst +++ b/Doc/library/asyncio-dev.rst @@ -2,415 +2,237 @@ .. _asyncio-dev: -Develop with asyncio -==================== +======================= +Developing with asyncio +======================= -Asynchronous programming is different than classical "sequential" programming. -This page lists common traps and explains how to avoid them. +Asynchronous programming is different from classical "sequential" +programming. + +This page lists common mistakes and traps and explains how +to avoid them. .. _asyncio-debug-mode: -Debug mode of asyncio ---------------------- +Debug Mode +========== -The implementation of :mod:`asyncio` has been written for performance. -In order to ease the development of asynchronous code, you may wish to -enable *debug mode*. +By default asyncio runs in production mode. In order to ease +the development asyncio has a *debug mode*. -To enable all debug checks for an application: +To enable debugging for an application: -* Enable the asyncio debug mode globally by setting the environment variable - :envvar:`PYTHONASYNCIODEBUG` to ``1``, using ``-X dev`` command line option - (see the :option:`-X` option), or by calling - :meth:`loop.set_debug`. -* Set the log level of the :ref:`asyncio logger ` to - :py:data:`logging.DEBUG`. For example, call - ``logging.basicConfig(level=logging.DEBUG)`` at startup. -* Configure the :mod:`warnings` module to display :exc:`ResourceWarning` - warnings. For example, use the ``-Wdefault`` command line option of Python to - display them. +* Enable the debug mode globally by setting the environment variable + :envvar:`PYTHONASYNCIODEBUG` to ``1``. -Examples debug checks: +* Alternatively, the debug mode can be enabled by using the ``-X dev`` + command line option for Python (see the :option:`-X` option). -* Log :ref:`coroutines defined but never "yielded from" - ` -* :meth:`loop.call_soon` and :meth:`loop.call_at` methods - raise an exception if they are called from the wrong thread. -* Log the execution time of the selector -* Log callbacks taking more than 100 ms to be executed. The +* Yet another way to enable the debug mode is by calling + :meth:`loop.set_debug` or by passing ``debug=True`` to + :func:`asyncio.run`. + +In addition to enabling debug mode, consider also: + +* setting the log level of the :ref:`asyncio logger ` to + :py:data:`logging.DEBUG`, for example the following snippet of code + can be run at startup of the application:: + + logging.basicConfig(level=logging.DEBUG) + +* configuring the :mod:`warnings` module to display + :exc:`ResourceWarning` warnings. One way of doing that is by + using the ``-Wdefault`` command line option. + + +In asyncio debug mode the following checks are performed: + +* Log :ref:`coroutines that were not awaited + `; this mitigates the "forgotten + await" pitfall. + +* Many non-treadsafe asyncio APIs (such as :meth:`loop.call_soon` and + :meth:`loop.call_at` methods) raise an exception if they are called + from a wrong thread. + +* Log the execution time of the IO selector if it takes too long to + perform an IO operation. + +* Log callbacks taking longer than 100 ms to be executed. The :attr:`loop.slow_callback_duration` attribute is the minimum duration in seconds of "slow" callbacks. -* :exc:`ResourceWarning` warnings are emitted when transports and event loops - are :ref:`not closed explicitly `. - -.. versionchanged:: 3.7 - - The new ``-X dev`` command line option can now also be used to enable - the debug mode. - -.. seealso:: - - The :meth:`loop.set_debug` method and the :ref:`asyncio logger - `. - - -Cancellation ------------- - -Cancellation of tasks is not common in classic programming. In asynchronous -programming, not only is it something common, but you have to prepare your -code to handle it. - -Futures and tasks can be cancelled explicitly with their :meth:`Future.cancel` -method. The :func:`wait_for` function cancels the waited task when the timeout -occurs. There are many other cases where a task can be cancelled indirectly. - -Don't call :meth:`~Future.set_result` or :meth:`~Future.set_exception` method -of :class:`Future` if the future is cancelled: it would fail with an exception. -For example, write:: - - if not fut.cancelled(): - fut.set_result('done') - -Don't schedule directly a call to the :meth:`~Future.set_result` or the -:meth:`~Future.set_exception` method of a future with -:meth:`loop.call_soon`: the future can be cancelled before its method -is called. - -If you wait for a future, you should check early if the future was cancelled to -avoid useless operations. Example:: - - async def slow_operation(fut): - if fut.cancelled(): - return - # ... slow computation ... - await fut - # ... - -The :func:`shield` function can also be used to ignore cancellation. .. _asyncio-multithreading: -Concurrency and multithreading ------------------------------- +Concurrency and Multithreading +============================== An event loop runs in a thread (typically the main thread) and executes -all callbacks and tasks in its thread. While a task is running in the -event loop, no other tasks may run in the same thread. When a task -executes an ``await`` expression, the running task gets suspended, and the -event loop executes the next task. Prior to suspending the task, the awaiting -chain is checked, and if the chain ends with a future, the running task is -not suspended. +all callbacks and Tasks in its thread. While a Task is running in the +event loop, no other Tasks can run in the same thread. When a Task +executes an ``await`` expression, the running Task gets suspended, and +the event loop executes the next Task. -To schedule a callback from a different thread, the +To schedule a callback from a different OS thread, the :meth:`loop.call_soon_threadsafe` method should be used. Example:: loop.call_soon_threadsafe(callback, *args) -Most asyncio objects are not thread safe. You should only worry if you access -objects outside the event loop. For example, to cancel a future, don't call -directly its :meth:`Future.cancel` method, but:: +Almost all asyncio objects are not thread safe, which is typically +not a problem unless there is code that works with them from outside +of a Task or a callback. If there's a need for such code to call a +low-level asyncio API, the :meth:`loop.call_soon_threadsafe` method +should be used, e.g.:: loop.call_soon_threadsafe(fut.cancel) -To handle signals and to execute subprocesses, the event loop must be run in -the main thread. - -To schedule a coroutine object from a different thread, the +To schedule a coroutine object from a different OS thread, the :func:`run_coroutine_threadsafe` function should be used. It returns a :class:`concurrent.futures.Future` to access the result:: + async def coro_func(): + return await asyncio.sleep(1, 42) + + # Later in another OS thread: + future = asyncio.run_coroutine_threadsafe(coro_func(), loop) - result = future.result(timeout) # Wait for the result with a timeout + # Wait for the result: + result = future.result() + +To handle signals and to execute subprocesses, the event loop must be +run in the main thread. The :meth:`loop.run_in_executor` method can be used with a -:class:`concurrent.futures.ThreadPoolExecutor` to execute a callback in -different thread so as not to block the event loop's main thread. - -.. seealso:: - - The :ref:`Synchronization primitives ` section describes ways - to synchronize tasks. - - The :ref:`Subprocess and threads ` section lists - asyncio limitations to run subprocesses from different threads. - - +:class:`concurrent.futures.ThreadPoolExecutor` to execute +blocking code in a different OS thread without blocking the OS thread +that the event loop runs in. .. _asyncio-handle-blocking: -Handle blocking functions correctly ------------------------------------ +Running Blocking Code +===================== -Blocking functions should not be called directly. For example, if a function -blocks for 1 second, other tasks are delayed by 1 second which can have an -important impact on reactivity. +Blocking (CPU-bound) code should not be called directly. For example, +if a function performs a CPU-intensive calculation for 1 second, +all concurrent asyncio Tasks and IO operations would be delayed +by 1 second. -For networking and subprocesses, the :mod:`asyncio` module provides high-level -APIs like :ref:`protocols `. - -An executor can be used to run a task in a different thread or even in a -different process, to not block the thread of the event loop. See the -:meth:`loop.run_in_executor` method. - -.. seealso:: - - The :ref:`Delayed calls ` section details how the - event loop handles time. +An executor can be used to run a task in a different thread or even in +a different process to avoid blocking block the OS thread with the +event loop. See the :meth:`loop.run_in_executor` method for more +details. .. _asyncio-logger: Logging -------- +======= -The :mod:`asyncio` module logs information with the :mod:`logging` module in -the logger ``'asyncio'``. +asyncio uses the :mod:`logging` module and all logging is performed +via the ``"asyncio"`` logger. -The default log level for the :mod:`asyncio` module is :py:data:`logging.INFO`. -For those not wanting such verbosity from :mod:`asyncio` the log level can -be changed. For example, to change the level to :py:data:`logging.WARNING`: +The default log level is :py:data:`logging.INFO`, which can easily be +adjusted:: -.. code-block:: none - - logging.getLogger('asyncio').setLevel(logging.WARNING) + logging.getLogger("asyncio").setLevel(logging.WARNING) .. _asyncio-coroutine-not-scheduled: -Detect coroutine objects never scheduled ----------------------------------------- +Detect never awaited coroutines +=============================== -When a coroutine function is called and its result is not passed to -:meth:`asyncio.create_task` the execution of the coroutine object will -never be scheduled which is probably a bug. Using ``asyncio.create_task`` is -preferred to the low level :func:`ensure_future` and :meth:`loop.create_task` -methods. :ref:`Enable the debug mode of asyncio ` -to :ref:`log a warning ` to detect it. - -Example with the bug:: +When a coroutine is called (e.g. ``coro()`` instead of ``await coro()``) +the call is not wrapped with :meth:`asyncio.create_task`, the execution +of the coroutine object will never be scheduled. For example:: import asyncio async def test(): print("never scheduled") + async def main(): + test() + + asyncio.run(main()) + +Output:: + + test.py:7: RuntimeWarning: coroutine 'test' was never awaited test() Output in debug mode:: - Coroutine test() at test.py:3 was never yielded from - Coroutine object created at (most recent call last): - File "test.py", line 7, in - test() + test.py:7: RuntimeWarning: coroutine 'test' was never awaited + Coroutine created at (most recent call last) + File "../t.py", line 9, in + asyncio.run(main(), debug=True) -The fix is to call the :meth:`asyncio.create_task` function. Using -``asyncio.create_task`` is preferred to the low level :func:`ensure_future` and -:meth:`loop.create_task` methods. + < .. > -.. seealso:: + File "../t.py", line 7, in main + test() + test() - :ref:`Pending task destroyed `. +The usual fix is to either await the coroutine or call the +:meth:`asyncio.create_task` function:: + + async def main(): + await test() -Detect exceptions never consumed --------------------------------- +Detect never consumed exceptions +================================ -Python usually calls :func:`sys.excepthook` on unhandled exceptions. If -:meth:`Future.set_exception` is called, but the exception is never consumed, -:func:`sys.excepthook` is not called. Instead, :ref:`a log is emitted -` when the future is deleted by the garbage collector, with the -traceback where the exception was raised. +If a :meth:`Future.set_exception` is called but the Future object is +never awaited on, the exception would never be propagated to the +user code. In this case, asyncio would emit a log message when the +Future object is garbage collected. -Example of unhandled exception:: +Example of an unhandled exception:: import asyncio - @asyncio.coroutine - def bug(): + async def bug(): raise Exception("not consumed") - loop = asyncio.get_event_loop() - asyncio.ensure_future(bug()) - loop.run_forever() - loop.close() + async def main(): + asyncio.create_task(bug()) + + asyncio.run(main()) Output:: Task exception was never retrieved - future: exception=Exception('not consumed',)> + future: + exception=Exception('not consumed')> + Traceback (most recent call last): - File "asyncio/tasks.py", line 237, in _step - result = next(coro) - File "asyncio/coroutines.py", line 141, in coro - res = func(*args, **kw) - File "test.py", line 5, in bug + File "test.py", line 4, in bug raise Exception("not consumed") Exception: not consumed -:ref:`Enable the debug mode of asyncio ` to get the -traceback where the task was created. Output in debug mode:: +:ref:`Enable the debug mode ` to get the +traceback where the task was created:: + + asyncio.run(main(), debug=True) + +Output in debug mode:: Task exception was never retrieved - future: exception=Exception('not consumed',) created at test.py:8> + future: + exception=Exception('not consumed') created at asyncio/tasks.py:321> + source_traceback: Object created at (most recent call last): - File "test.py", line 8, in - asyncio.ensure_future(bug()) + File "../t.py", line 9, in + asyncio.run(main(), debug=True) + + < .. > + Traceback (most recent call last): - File "asyncio/tasks.py", line 237, in _step - result = next(coro) - File "asyncio/coroutines.py", line 79, in __next__ - return next(self.gen) - File "asyncio/coroutines.py", line 141, in coro - res = func(*args, **kw) - File "test.py", line 5, in bug + File "../t.py", line 4, in bug raise Exception("not consumed") Exception: not consumed - -There are different options to fix this issue. The first option is to chain the -coroutine in another coroutine and use classic try/except:: - - async def handle_exception(): - try: - await bug() - except Exception: - print("exception consumed") - - loop = asyncio.get_event_loop() - asyncio.ensure_future(handle_exception()) - loop.run_forever() - loop.close() - -Another option is to use the :meth:`asyncio.run` function:: - - asyncio.run(bug()) - -.. seealso:: - - The :meth:`Future.exception` method. - - -Chain coroutines correctly --------------------------- - -When a coroutine function calls other coroutine functions and tasks, they -should be chained explicitly with ``await``. Otherwise, the execution is -not guaranteed to be sequential. - -Example with different bugs using :func:`asyncio.sleep` to simulate slow -operations:: - - import asyncio - - async def create(): - await asyncio.sleep(3.0) - print("(1) create file") - - async def write(): - await asyncio.sleep(1.0) - print("(2) write into file") - - async def close(): - print("(3) close file") - - async def test(): - asyncio.ensure_future(create()) - asyncio.ensure_future(write()) - asyncio.ensure_future(close()) - await asyncio.sleep(2.0) - loop.stop() - - loop = asyncio.get_event_loop() - asyncio.ensure_future(test()) - loop.run_forever() - print("Pending tasks at exit: %s" % asyncio.Task.all_tasks(loop)) - loop.close() - -Expected output: - -.. code-block:: none - - (1) create file - (2) write into file - (3) close file - Pending tasks at exit: set() - -Actual output: - -.. code-block:: none - - (3) close file - (2) write into file - Pending tasks at exit: {>} - Task was destroyed but it is pending! - task: > - -The loop stopped before the ``create()`` finished, ``close()`` has been called -before ``write()``, whereas coroutine functions were called in this order: -``create()``, ``write()``, ``close()``. - -To fix the example, tasks must be marked with ``await``:: - - async def test(): - await asyncio.ensure_future(create()) - await asyncio.ensure_future(write()) - await asyncio.ensure_future(close()) - await asyncio.sleep(2.0) - loop.stop() - -Or without ``asyncio.ensure_future()``:: - - async def test(): - await create() - await write() - await close() - await asyncio.sleep(2.0) - loop.stop() - - -.. _asyncio-pending-task-destroyed: - -Pending task destroyed ----------------------- - -If a pending task is destroyed, the execution of its wrapped :ref:`coroutine -` did not complete. It is probably a bug and so a warning is logged. - -Example of log: - -.. code-block:: none - - Task was destroyed but it is pending! - task: wait_for=> - -:ref:`Enable the debug mode of asyncio ` to get the -traceback where the task was created. Example of log in debug mode: - -.. code-block:: none - - Task was destroyed but it is pending! - source_traceback: Object created at (most recent call last): - File "test.py", line 15, in - task = asyncio.ensure_future(coro, loop=loop) - task: wait_for= created at test.py:15> - - -.. seealso:: - - :ref:`Detect coroutine objects never scheduled `. - -.. _asyncio-close-transports: - -Close transports and event loops --------------------------------- - -When a transport is no longer needed, call its ``close()`` method to release -resources. Event loops must also be closed explicitly. - -If a transport or an event loop is not closed explicitly, a -:exc:`ResourceWarning` warning will be emitted in its destructor. By default, -:exc:`ResourceWarning` warnings are ignored. The :ref:`Debug mode of asyncio -` section explains how to display them. diff --git a/Doc/library/asyncio-eventloop.rst b/Doc/library/asyncio-eventloop.rst index 19a0acdfd55..718c32277c2 100644 --- a/Doc/library/asyncio-eventloop.rst +++ b/Doc/library/asyncio-eventloop.rst @@ -1075,6 +1075,11 @@ Enabling debug mode Set the debug mode of the event loop. + .. versionchanged:: 3.7 + + The new ``-X dev`` command line option can now also be used + to enable the debug mode. + .. seealso:: The :ref:`debug mode of asyncio `. diff --git a/Doc/library/asyncio-future.rst b/Doc/library/asyncio-future.rst index ff04339d858..19bf8a6e058 100644 --- a/Doc/library/asyncio-future.rst +++ b/Doc/library/asyncio-future.rst @@ -121,6 +121,16 @@ Future Object or an exception set with :meth:`set_result` or :meth:`set_exception` calls. + .. method:: cancelled() + + Return ``True`` if the Future was *cancelled*. + + The method is usually used to check if a Future is not + *cancelled* before setting a result or an exception for it:: + + if not fut.cancelled(): + fut.set_result(42) + .. method:: add_done_callback(callback, *, context=None) Add a callback to be run when the Future is *done*. @@ -180,10 +190,6 @@ Future Object .. versionadded:: 3.7 - .. method:: cancelled() - - Return ``True`` if the Future was *cancelled*. - This example creates a Future object, creates and schedules an asynchronous Task to set result for the Future, and waits until diff --git a/Doc/library/asyncio.rst b/Doc/library/asyncio.rst index 25daeb6b853..9f45f7458c3 100644 --- a/Doc/library/asyncio.rst +++ b/Doc/library/asyncio.rst @@ -32,8 +32,9 @@ asyncio provides a set of **high-level** APIs to: as well as **low-level** APIs for *library and framework developers* to: * create and manage :ref:`event loops `, which - provide asynchronous APIs for networking, subprocesses, OS signals, - etc; + provide asynchronous APIs for :meth:`networking `, + running :meth:`subprocesses `, + handling :meth:`OS signals `, etc; * implement efficient protocols using :ref:`transports `; @@ -75,11 +76,3 @@ Reference asyncio-api-index.rst asyncio-dev.rst - - -.. seealso:: - - The :mod:`asyncio` module was proposed in :PEP:`3156`. - Since the acceptance of the PEP many new APIs were added and many - original APIs were altered. The PEP should be treated as a - historical document. diff --git a/Doc/library/ipc.rst b/Doc/library/ipc.rst index 8f5b3b216c7..b88a174eb97 100644 --- a/Doc/library/ipc.rst +++ b/Doc/library/ipc.rst @@ -1,11 +1,11 @@ .. _ipc: ***************************************** -Interprocess Communication and Networking +Networking and Interprocess Communication ***************************************** -The modules described in this chapter provide mechanisms for different processes -to communicate. +The modules described in this chapter provide mechanisms for +networking and inter-processes communication. Some modules only work for two processes that are on the same machine, e.g. :mod:`signal` and :mod:`mmap`. Other modules support networking protocols @@ -15,6 +15,7 @@ The list of modules described in this chapter is: .. toctree:: + :maxdepth: 1 asyncio.rst socket.rst