From a8efb6b080d1066fc3cc4bf186ab410ee33cb652 Mon Sep 17 00:00:00 2001 From: Antoine Pitrou Date: Sun, 11 Jan 2015 15:09:27 +0100 Subject: [PATCH] Issue #22952: improve multiprocessing doc introduction and defer notes until appropriate. Patch by Davin Potts. --- Doc/library/multiprocessing.rst | 83 +++++++++++++++++++++------------ 1 file changed, 52 insertions(+), 31 deletions(-) diff --git a/Doc/library/multiprocessing.rst b/Doc/library/multiprocessing.rst index fb8ddfb07d5..77cb877c92d 100644 --- a/Doc/library/multiprocessing.rst +++ b/Doc/library/multiprocessing.rst @@ -18,41 +18,27 @@ to this, the :mod:`multiprocessing` module allows the programmer to fully leverage multiple processors on a given machine. It runs on both Unix and Windows. -.. warning:: +The :mod:`multiprocessing` module also introduces APIs which do not have +analogs in the :mod:`threading` module. A prime example of this is the +:class:`Pool` object which offers a convenient means of parallelizing the +execution of a function across multiple input values, distributing the +input data across processes (data parallelism). The following example +demonstrates the common practice of defining such functions in a module so +that child processes can successfully import that module. This basic example +of data parallelism using :class:`Pool`, :: - Some of this package's functionality requires a functioning shared semaphore - implementation on the host operating system. Without one, the - :mod:`multiprocessing.synchronize` module will be disabled, and attempts to - import it will result in an :exc:`ImportError`. See - :issue:`3770` for additional information. + from multiprocessing import Pool -.. note:: + def f(x): + return x*x - Functionality within this package requires that the ``__main__`` module be - importable by the children. This is covered in :ref:`multiprocessing-programming` - however it is worth pointing out here. This means that some examples, such - as the :class:`multiprocessing.Pool` examples will not work in the - interactive interpreter. For example:: + if __name__ == '__main__': + p = Pool(5) + print(p.map(f, [1, 2, 3])) - >>> from multiprocessing import Pool - >>> p = Pool(5) - >>> def f(x): - ... return x*x - ... - >>> p.map(f, [1,2,3]) - Process PoolWorker-1: - Process PoolWorker-2: - Process PoolWorker-3: - Traceback (most recent call last): - Traceback (most recent call last): - Traceback (most recent call last): - AttributeError: 'module' object has no attribute 'f' - AttributeError: 'module' object has no attribute 'f' - AttributeError: 'module' object has no attribute 'f' +will print to standard output :: - (If you try this it will actually output three full tracebacks - interleaved in a semi-random fashion, and then you may have to - stop the master process somehow.) + [1, 4, 9] The :class:`Process` class @@ -99,7 +85,6 @@ For an explanation of why (on Windows) the ``if __name__ == '__main__'`` part is necessary, see :ref:`multiprocessing-programming`. - Exchanging objects between processes ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ @@ -175,6 +160,14 @@ that only one process prints to standard output at a time:: Without using the lock output from the different processes is liable to get all mixed up. +.. warning:: + + Some of this package's functionality requires a functioning shared semaphore + implementation on the host operating system. Without one, the + :mod:`multiprocessing.synchronize` module will be disabled, and attempts to + import it will result in an :exc:`ImportError`. See + :issue:`3770` for additional information. + Sharing state between processes ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ @@ -290,6 +283,34 @@ For example:: Note that the methods of a pool should only ever be used by the process which created it. +.. note:: + + Functionality within this package requires that the ``__main__`` module be + importable by the children. This is covered in :ref:`multiprocessing-programming` + however it is worth pointing out here. This means that some examples, such + as the :class:`Pool` examples will not work in the interactive interpreter. + For example:: + + >>> from multiprocessing import Pool + >>> p = Pool(5) + >>> def f(x): + ... return x*x + ... + >>> p.map(f, [1,2,3]) + Process PoolWorker-1: + Process PoolWorker-2: + Process PoolWorker-3: + Traceback (most recent call last): + Traceback (most recent call last): + Traceback (most recent call last): + AttributeError: 'module' object has no attribute 'f' + AttributeError: 'module' object has no attribute 'f' + AttributeError: 'module' object has no attribute 'f' + + (If you try this it will actually output three full tracebacks + interleaved in a semi-random fashion, and then you may have to + stop the master process somehow.) + Reference ---------