gh-114756: Update FAQ section on removing the GIL (#114957)

Update FAQ section on removing the GIL to reflect recent progress on PEP 703 and PEP 684.

Co-authored-by: AN Long <aisk@users.noreply.github.com>
This commit is contained in:
da-woods 2024-02-06 15:55:44 +00:00 committed by GitHub
parent d7334e2c20
commit 0e2ab73dc3
No known key found for this signature in database
GPG Key ID: B5690EEEBB952194
1 changed files with 32 additions and 26 deletions

View File

@ -405,22 +405,37 @@ lists. When in doubt, use a mutex!
Can't we get rid of the Global Interpreter Lock?
------------------------------------------------
.. XXX link to dbeazley's talk about GIL?
The :term:`global interpreter lock` (GIL) is often seen as a hindrance to Python's
deployment on high-end multiprocessor server machines, because a multi-threaded
Python program effectively only uses one CPU, due to the insistence that
(almost) all Python code can only run while the GIL is held.
Back in the days of Python 1.5, Greg Stein actually implemented a comprehensive
patch set (the "free threading" patches) that removed the GIL and replaced it
with fine-grained locking. Adam Olsen recently did a similar experiment
in his `python-safethread <https://code.google.com/archive/p/python-safethread>`_
project. Unfortunately, both experiments exhibited a sharp drop in single-thread
performance (at least 30% slower), due to the amount of fine-grained locking
necessary to compensate for the removal of the GIL.
With the approval of :pep:`703` work is now underway to remove the GIL from the
CPython implementation of Python. Initially it will be implemented as an
optional compiler flag when building the interpreter, and so separate
builds will be available with and without the GIL. Long-term, the hope is
to settle on a single build, once the performance implications of removing the
GIL are fully understood. Python 3.13 is likely to be the first release
containing this work, although it may not be completely functional in this
release.
This doesn't mean that you can't make good use of Python on multi-CPU machines!
The current work to remove the GIL is based on a
`fork of Python 3.9 with the GIL removed <https://github.com/colesbury/nogil>`_
by Sam Gross.
Prior to that,
in the days of Python 1.5, Greg Stein actually implemented a comprehensive
patch set (the "free threading" patches) that removed the GIL and replaced it
with fine-grained locking. Adam Olsen did a similar experiment
in his `python-safethread <https://code.google.com/archive/p/python-safethread>`_
project. Unfortunately, both of these earlier experiments exhibited a sharp
drop in single-thread
performance (at least 30% slower), due to the amount of fine-grained locking
necessary to compensate for the removal of the GIL. The Python 3.9 fork
is the first attempt at removing the GIL with an acceptable performance
impact.
The presence of the GIL in current Python releases
doesn't mean that you can't make good use of Python on multi-CPU machines!
You just have to be creative with dividing the work up between multiple
*processes* rather than multiple *threads*. The
:class:`~concurrent.futures.ProcessPoolExecutor` class in the new
@ -434,22 +449,13 @@ thread of execution is in the C code and allow other threads to get some work
done. Some standard library modules such as :mod:`zlib` and :mod:`hashlib`
already do this.
It has been suggested that the GIL should be a per-interpreter-state lock rather
than truly global; interpreters then wouldn't be able to share objects.
Unfortunately, this isn't likely to happen either. It would be a tremendous
amount of work, because many object implementations currently have global state.
For example, small integers and short strings are cached; these caches would
have to be moved to the interpreter state. Other object types have their own
free list; these free lists would have to be moved to the interpreter state.
And so on.
And I doubt that it can even be done in finite time, because the same problem
exists for 3rd party extensions. It is likely that 3rd party extensions are
being written at a faster rate than you can convert them to store all their
global state in the interpreter state.
And finally, once you have multiple interpreters not sharing any state, what
have you gained over running each interpreter in a separate process?
An alternative approach to reducing the impact of the GIL is
to make the GIL a per-interpreter-state lock rather than truly global.
This was :ref:`first implemented in Python 3.12 <whatsnew312-pep684>` and is
available in the C API. A Python interface to it is expected in Python 3.13.
The main limitation to it at the moment is likely to be 3rd party extension
modules, since these must be written with multiple interpreters in mind in
order to be usable, so many older extension modules will not be usable.
Input and Output