traverseda
/
cpython
1
0
Fork 0
Code Issues Pull Requests Releases Wiki Activity

bpo-39429: Add a new "Python Development Mode" doc page (GH-18132)

This commit is contained in:
Victor Stinner 2020-01-24 10:22:18 +01:00 committed by GitHub
parent e131c9720d
commit b9783d2e03
No known key found for this signature in database
GPG Key ID: 4AEE18F83AFDEB23
13 changed files with 268 additions and 49 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

2
Doc/c-api/init_config.rst
View File

@ -466,7 +466,7 @@ PyConfig
.. c:member:: int dev_mode
Development mode: see :option:`-X dev <-X>`.
If non-zero, enable the :ref:`Python Development Mode <devmode>`.
.. c:member:: int dump_refs

2
Doc/library/asyncio-dev.rst
View File

@ -25,7 +25,7 @@ There are several ways to enable asyncio debug mode:
* Setting the :envvar:`PYTHONASYNCIODEBUG` environment variable to ``1``.
* Using the :option:`-X` ``dev`` Python command line option.
* Using the :ref:`Python Development Mode <devmode>`.
* Passing ``debug=True`` to :func:`asyncio.run`.

2
Doc/library/asyncio-eventloop.rst
View File

@ -1200,7 +1200,7 @@ Enabling debug mode
.. versionchanged:: 3.7
The new ``-X dev`` command line option can now also be used
The new :ref:`Python Development Mode <devmode>` can now also be used
to enable the debug mode.
.. seealso::

4
Doc/library/development.rst
View File

@ -18,12 +18,10 @@ The list of modules described in this chapter is:
typing.rst
pydoc.rst
devmode.rst
doctest.rst
unittest.rst
unittest.mock.rst
unittest.mock-examples.rst
2to3.rst
test.rst
See also the Python development mode: the :option:`-X` ``dev`` option and
:envvar:`PYTHONDEVMODE` environment variable.

214
Doc/library/devmode.rst Normal file
View File

@ -0,0 +1,214 @@
.. _devmode:
Python Development Mode
=======================
.. versionadded:: 3.7
The Python Development Mode introduces additional runtime checks that are too
expensive to be enabled by default. It should not be more verbose than the
default if the code is correct; new warnings are only emitted when an issue is
detected.
It can be enabled using the :option:`-X dev <-X>` command line option or by
setting the :envvar:`PYTHONDEVMODE` environment variable to ``1``.
Effects of the Python Development Mode
======================================
Enabling the Python Development Mode is similar to the following command, but
with additional effects described below::
PYTHONMALLOC=debug PYTHONASYNCIODEBUG=1 python3 -W default -X faulthandler
Effects of the Python Development Mode:
* Add ``default`` :ref:`warning filter <describing-warning-filters>`. The
following warnings are shown:
* :exc:`DeprecationWarning`
* :exc:`ImportWarning`
* :exc:`PendingDeprecationWarning`
* :exc:`ResourceWarning`
Normally, the above warnings are filtered by the default :ref:`warning
filters <describing-warning-filters>`.
It behaves as if the :option:`-W default <-W>` command line option is used.
Use the :option:`-W error <-W>` command line option or set the
:envvar:`PYTHONWARNINGS` environment variable to ``error`` to treat warnings
as errors.
* Install debug hooks on memory allocators to check for:
* Buffer underflow
* Buffer overflow
* Memory allocator API violation
* Unsafe usage of the GIL
See the :c:func:`PyMem_SetupDebugHooks` C function.
It behaves as if the :envvar:`PYTHONMALLOC` environment variable is set to
``debug``.
To enable the Python Development Mode without installing debug hooks on
memory allocators, set the :envvar:`PYTHONMALLOC` environment variable to
``default``.
* Call :func:`faulthandler.enable` at Python startup to install handlers for
the :const:`SIGSEGV`, :const:`SIGFPE`, :const:`SIGABRT`, :const:`SIGBUS` and
:const:`SIGILL` signals to dump the Python traceback on a crash.
It behaves as if the :option:`-X faulthandler <-X>` command line option is
used or if the :envvar:`PYTHONFAULTHANDLER` environment variable is set to
``1``.
* Enable :ref:`asyncio debug mode <asyncio-debug-mode>`. For example,
:mod:`asyncio` checks for coroutines that were not awaited and logs them.
It behaves as if the :envvar:`PYTHONASYNCIODEBUG` environment variable is set
to ``1``.
* Check the *encoding* and *errors* arguments for string encoding and decoding
operations. Examples: :func:`open`, :meth:`str.encode` and
:meth:`bytes.decode`.
By default, for best performance, the *errors* argument is only checked at
the first encoding/decoding error and the *encoding* argument is sometimes
ignored for empty strings.
* The :class:`io.IOBase` destructor logs ``close()`` exceptions.
* Set the :attr:`~sys.flags.dev_mode` attribute of :attr:`sys.flags` to
``True``.
The Python Development Mode does not enable the :mod:`tracemalloc` module by
default, because the overhead cost (to performance and memory) would be too
large. Enabling the :mod:`tracemalloc` module provides additional information
on the origin of some errors. For example, :exc:`ResourceWarning` logs the
traceback where the resource was allocated, and a buffer overflow error logs
the traceback where the memory block was allocated.
The Python Development Mode does not prevent the :option:`-O` command line
option from removing :keyword:`assert` statements nor from setting
:const:`__debug__` to ``False``.
.. versionchanged:: 3.8
The :class:`io.IOBase` destructor now logs ``close()`` exceptions.
.. versionchanged:: 3.9
The *encoding* and *errors* arguments are now checked for string encoding
and decoding operations.
ResourceWarning Example
=======================
Example of a script counting the number of lines of the text file specified in
the command line::
import sys
def main():
fp = open(sys.argv[1])
nlines = len(fp.readlines())
print(nlines)
# The file is closed implicitly
if __name__ == "__main__":
main()
The script does not close the file explicitly. By default, Python does not emit
any warning. Example using README.txt, which has 269 lines:
.. code-block:: shell-session
$ python3 script.py README.txt
269
Enabling the Python Development Mode displays a :exc:`ResourceWarning` warning:
.. code-block:: shell-session
$ python3 -X dev script.py README.txt
269
script.py:10: ResourceWarning: unclosed file <_io.TextIOWrapper name='README.rst' mode='r' encoding='UTF-8'>
main()
ResourceWarning: Enable tracemalloc to get the object allocation traceback
In addition, enabling :mod:`tracemalloc` shows the line where the file was
opened:
.. code-block:: shell-session
$ python3 -X dev -X tracemalloc=5 script.py README.rst
269
script.py:10: ResourceWarning: unclosed file <_io.TextIOWrapper name='README.rst' mode='r' encoding='UTF-8'>
main()
Object allocated at (most recent call last):
File "script.py", lineno 10
main()
File "script.py", lineno 4
fp = open(sys.argv[1])
The fix is to close explicitly the file. Example using a context manager::
def main():
# Close the file explicitly when exiting the with block
with open(sys.argv[1]) as fp:
nlines = len(fp.readlines())
print(nlines)
Not closing a resource explicitly can leave a resource open for way longer than
expected; it can cause severe issues upon exiting Python. It is bad in
CPython, but it is even worse in PyPy. Closing resources explicitly makes an
application more deterministic and more reliable.
Bad file descriptor error example
=================================
Script displaying the first line of itself::
import os
def main():
fp = open(__file__)
firstline = fp.readline()
print(firstline.rstrip())
os.close(fp.fileno())
# The file is closed implicitly
main()
By default, Python does not emit any warning:
.. code-block:: shell-session
$ python3 script.py
import os
The Python Development Mode shows a :exc:`ResourceWarning` and logs a "Bad file
descriptor" error when finalizing the file object:
.. code-block:: shell-session
$ python3 script.py
import os
script.py:10: ResourceWarning: unclosed file <_io.TextIOWrapper name='script.py' mode='r' encoding='UTF-8'>
main()
ResourceWarning: Enable tracemalloc to get the object allocation traceback
Exception ignored in: <_io.TextIOWrapper name='script.py' mode='r' encoding='UTF-8'>
Traceback (most recent call last):
File "script.py", line 10, in <module>
main()
OSError: [Errno 9] Bad file descriptor
``os.close(fp.fileno())`` closes the file descriptor. When the file object
finalizer tries to close the file descriptor again, it fails with the ``Bad
file descriptor`` error. A file descriptor must be closed only once. In the
worst case scenario, closing it twice can lead to a crash (see :issue:`18748`
for an example).
The fix is to remove the ``os.close(fp.fileno())`` line, or open the file with
``closefd=False``.

16
Doc/library/exceptions.rst
View File

@ -688,6 +688,10 @@ The following exceptions are used as warning categories; see the
Base class for warnings about deprecated features when those warnings are
intended for other Python developers.
Ignored by the default warning filters, except in the ``__main__`` module
(:pep:`565`). Enabling the :ref:`Python Development Mode <devmode>` shows
this warning.
.. exception:: PendingDeprecationWarning
@ -699,6 +703,9 @@ The following exceptions are used as warning categories; see the
upcoming deprecation is unusual, and :exc:`DeprecationWarning`
is preferred for already active deprecations.
Ignored by the default warning filters. Enabling the :ref:`Python
Development Mode <devmode>` shows this warning.
.. exception:: SyntaxWarning
@ -720,6 +727,9 @@ The following exceptions are used as warning categories; see the
Base class for warnings about probable mistakes in module imports.
Ignored by the default warning filters. Enabling the :ref:`Python
Development Mode <devmode>` shows this warning.
.. exception:: UnicodeWarning
@ -733,8 +743,10 @@ The following exceptions are used as warning categories; see the
.. exception:: ResourceWarning
Base class for warnings related to resource usage. Ignored by the default
warning filters.
Base class for warnings related to resource usage.
Ignored by the default warning filters. Enabling the :ref:`Python
Development Mode <devmode>` shows this warning.
.. versionadded:: 3.2

3
Doc/library/faulthandler.rst
View File

@ -40,6 +40,9 @@ alternatively be passed to :func:`faulthandler.enable`.
The module is implemented in C, so tracebacks can be dumped on a crash or when
Python is deadlocked.
The :ref:`Python Development Mode <devmode>` calls :func:`faulthandler.enable`
at Python startup.
Dumping the traceback
---------------------

8
Doc/library/stdtypes.rst
View File

@ -1559,8 +1559,8 @@ expression support in the :mod:`re` module).
list of possible encodings, see section :ref:`standard-encodings`.
By default, the *errors* argument is not checked for best performances, but
only used at the first encoding error. Enable the development mode
(:option:`-X` ``dev`` option), or use a debug build, to check *errors*.
only used at the first encoding error. Enable the :ref:`Python Development
Mode <devmode>`, or use a debug build to check *errors*.
.. versionchanged:: 3.1
Support for keyword arguments added.
@ -2596,8 +2596,8 @@ arbitrary binary data.
list of possible encodings, see section :ref:`standard-encodings`.
By default, the *errors* argument is not checked for best performances, but
only used at the first decoding error. Enable the development mode
(:option:`-X` ``dev`` option), or use a debug build, to check *errors*.
only used at the first decoding error. Enable the :ref:`Python Development
Mode <devmode>`, or use a debug build to check *errors*.
.. note::

15
Doc/library/sys.rst
View File

@ -428,9 +428,9 @@ always available.
The :term:`named tuple` *flags* exposes the status of command line
flags. The attributes are read only.
============================= =============================
============================= ================================================================
attribute flag
============================= =============================
============================= ================================================================
:const:`debug` :option:`-d`
:const:`inspect` :option:`-i`
:const:`interactive` :option:`-i`
@ -444,9 +444,9 @@ always available.
:const:`bytes_warning` :option:`-b`
:const:`quiet` :option:`-q`
:const:`hash_randomization` :option:`-R`
:const:`dev_mode` :option:`-X` ``dev``
:const:`utf8_mode` :option:`-X` ``utf8``
============================= =============================
:const:`dev_mode` :option:`-X dev <-X>` (:ref:`Python Development Mode <devmode>`)
:const:`utf8_mode` :option:`-X utf8 <-X>`
============================= ================================================================
.. versionchanged:: 3.2
Added ``quiet`` attribute for the new :option:`-q` flag.
@ -461,8 +461,9 @@ always available.
Added ``isolated`` attribute for :option:`-I` ``isolated`` flag.
.. versionchanged:: 3.7
Added ``dev_mode`` attribute for the new :option:`-X` ``dev`` flag
and ``utf8_mode`` attribute for the new :option:`-X` ``utf8`` flag.
Added the ``dev_mode`` attribute for the new :ref:`Python Development
Mode <devmode>` and the ``utf8_mode`` attribute for the new :option:`-X`
``utf8`` flag.
.. data:: float_info

26
Doc/using/cmdline.rst
View File

@ -442,24 +442,9 @@ Miscellaneous options
nested imports). Note that its output may be broken in multi-threaded
application. Typical usage is ``python3 -X importtime -c 'import
asyncio'``. See also :envvar:`PYTHONPROFILEIMPORTTIME`.
* ``-X dev``: enable CPython's "development mode", introducing additional
runtime checks which are too expensive to be enabled by default. It should
not be more verbose than the default if the code is correct: new warnings
are only emitted when an issue is detected. Effect of the developer mode:
* Check *encoding* and *errors* arguments on string encoding and decoding
operations. Examples: :func:`open`, :meth:`str.encode` and
:meth:`bytes.decode`.
* Add ``default`` warning filter, as :option:`-W` ``default``.
* Install debug hooks on memory allocators: see the
:c:func:`PyMem_SetupDebugHooks` C function.
* Enable the :mod:`faulthandler` module to dump the Python traceback
on a crash.
* Enable :ref:`asyncio debug mode <asyncio-debug-mode>`.
* Set the :attr:`~sys.flags.dev_mode` attribute of :attr:`sys.flags` to
``True``.
* :class:`io.IOBase` destructor logs ``close()`` exceptions.
* ``-X dev``: enable :ref:`Python Development Mode <devmode>`, introducing
additional runtime checks that are too expensive to be enabled by
default.
* ``-X utf8`` enables UTF-8 mode for operating system interfaces, overriding
the default locale-aware mode. ``-X utf8=0`` explicitly disables UTF-8
mode (even when it would otherwise activate automatically).
@ -890,8 +875,9 @@ conflict.
.. envvar:: PYTHONDEVMODE
If this environment variable is set to a non-empty string, enable the
CPython "development mode". See the :option:`-X` ``dev`` option.
If this environment variable is set to a non-empty string, enable
:ref:`Python Development Mode <devmode>`, introducing additional runtime
checks that are too expensive to be enabled by default.
.. versionadded:: 3.7

10
Doc/whatsnew/3.7.rst
View File

@ -102,7 +102,7 @@ CPython implementation improvements:
* :ref:`PEP 538 <whatsnew37-pep538>`, legacy C locale coercion
* :ref:`PEP 540 <whatsnew37-pep540>`, forced UTF-8 runtime mode
* :ref:`PEP 552 <whatsnew37-pep552>`, deterministic .pycs
* :ref:`the new development runtime mode <whatsnew37-devmode>`
* :ref:`New Python Development Mode <whatsnew37-devmode>`
* :ref:`PEP 565 <whatsnew37-pep565>`, improved :exc:`DeprecationWarning`
handling
@ -479,15 +479,15 @@ Three new translations have been added:
.. _whatsnew37-devmode:
Development Runtime Mode: -X dev
Python Development Mode (-X dev)
--------------------------------
The new :option:`-X` ``dev`` command line option or the new
:envvar:`PYTHONDEVMODE` environment variable can be used to enable
CPython's *development mode*. When in development mode, CPython performs
:ref:`Python Development Mode <devmode>`. When in development mode, Python performs
additional runtime checks that are too expensive to be enabled by default.
See :option:`-X` ``dev`` documentation for the full description of the effects
of this mode.
See :ref:`Python Development Mode <devmode>` documentation for the full
description.
Other Language Changes

7
Doc/whatsnew/3.9.rst
View File

@ -90,9 +90,10 @@ Other Language Changes
in this case.
(Contributed by Victor Stinner in :issue:`20443`.)
* In development mode and in debug build, *encoding* and *errors* arguments are
now checked on string encoding and decoding operations. Examples:
:func:`open`, :meth:`str.encode` and :meth:`bytes.decode`.
* In the :ref:`Python Development Mode <devmode>` and in debug build, the
*encoding* and *errors* arguments are now checked for string encoding and
decoding operations. Examples: :func:`open`, :meth:`str.encode` and
:meth:`bytes.decode`.
By default, for best performance, the *errors* argument is only checked at
the first encoding/decoding error and the *encoding* argument is sometimes

8
Include/cpython/initconfig.h
View File

@ -113,7 +113,11 @@ typedef struct {
"POSIX", otherwise it is set to 0. Inherit Py_UTF8Mode value value. */
int utf8_mode;
int dev_mode; /* Development mode. PYTHONDEVMODE, -X dev */
/* If non-zero, enable the Python Development Mode.
Set to 1 by the -X dev command line option. Set by the PYTHONDEVMODE
environment variable. */
int dev_mode;
/* Memory allocator: PYTHONMALLOC env var.
See PyMemAllocatorName for valid values. */
@ -131,7 +135,7 @@ typedef struct {
int isolated; /* Isolated mode? see PyPreConfig.isolated */
int use_environment; /* Use environment variables? see PyPreConfig.use_environment */
int dev_mode; /* Development mode? See PyPreConfig.dev_mode */
int dev_mode; /* Python Development Mode? See PyPreConfig.dev_mode */
/* Install signal handlers? Yes by default. */
int install_signal_handlers;