**************************** What's New In Python 3.9 **************************** :Release: |release| :Date: |today| .. Rules for maintenance: * Anyone can add text to this document. Do not spend very much time on the wording of your changes, because your text will probably get rewritten to some degree. * The maintainer will go through Misc/NEWS periodically and add changes; it's therefore more important to add your changes to Misc/NEWS than to this file. * This is not a complete list of every single change; completeness is the purpose of Misc/NEWS. Some changes I consider too small or esoteric to include. If such a change is added to the text, I'll just remove it. (This is another reason you shouldn't spend too much time on writing your addition.) * If you want to draw your new text to the attention of the maintainer, add 'XXX' to the beginning of the paragraph or section. * It's OK to just add a fragmentary note about a change. For example: "XXX Describe the transmogrify() function added to the socket module." The maintainer will research the change and write the necessary text. * You can comment out your additions if you like, but it's not necessary (especially when a final release is some months away). * Credit the author of a patch or bugfix. Just the name is sufficient; the e-mail address isn't necessary. * It's helpful to add the bug/patch number as a comment: XXX Describe the transmogrify() function added to the socket module. (Contributed by P.Y. Developer in :issue:`12345`.) This saves the maintainer the effort of going through the Mercurial log when researching a change. This article explains the new features in Python 3.9, compared to 3.8. For full details, see the :ref:`changelog `. .. note:: Prerelease users should be aware that this document is currently in draft form. It will be updated substantially as Python 3.9 moves towards release, so it's worth checking back even after reading earlier versions. Summary -- Release highlights ============================= .. This section singles out the most important changes in Python 3.9. Brevity is key. .. PEP-sized items next. You should check for DeprecationWarning in your code ==================================================== When Python 2.7 was still supported, many functions were kept for backward compatibility with Python 2.7. With the end of Python 2.7 support, these backward compatibility layers have been removed, or will be removed soon. Most of them emitted a :exc:`DeprecationWarning` warning for several years. For example, using ``collections.Mapping`` instead of ``collections.abc.Mapping`` emits a :exc:`DeprecationWarning` since Python 3.3, released in 2012. Test your application with the :option:`-W` ``default`` command-line option to see :exc:`DeprecationWarning` and :exc:`PendingDeprecationWarning`, or even with :option:`-W` ``error`` to treat them as errors. :ref:`Warnings Filter ` can be used to ignore warnings from third-party code. It has been decided to keep a few backward compatibility layers for one last release, to give more time to Python projects maintainers to organize the removal of the Python 2 support and add support for Python 3.9. Aliases to :ref:`Abstract Base Classes ` in the :mod:`collections` module, like ``collections.Mapping`` alias to :class:`collections.abc.Mapping`, are kept for one last release for backward compatibility. They will be removed from Python 3.10. More generally, try to run your tests in the :ref:`Python Development Mode ` which helps to prepare your code to make it compatible with the next Python version. New Features ============ Dictionary Merge & Update Operators ----------------------------------- Merge (``|``) and update (``|=``) operators have been added to the built-in :class:`dict` class. See :pep:`584` for a full description. (Contributed by Brandt Bucher in :issue:`36144`.) PEP 616: New removeprefix() and removesuffix() string methods ------------------------------------------------------------- :meth:`str.removeprefix(prefix)` and :meth:`str.removesuffix(suffix)` have been added to easily remove an unneeded prefix or a suffix from a string. Corresponding ``bytes``, ``bytearray``, and ``collections.UserString`` methods have also been added. See :pep:`616` for a full description. (Contributed by Dennis Sweeney in :issue:`18939`.) PEP 585: Builtin Generic Types ------------------------------ In type annotations you can now use built-in collection types such as ``list`` and ``dict`` as generic types instead of importing the corresponding capitalized types (e.g. ``List`` or ``Dict``) from ``typing``. Some other types in the standard library are also now generic, for example ``queue.Queue``. Example: .. code-block:: python def greet_all(names: list[str]) -> None: for name in names: print("Hello", name) See :pep:`585` for more details. (Contributed by Guido van Rossum, Ethan Smith, and Batuhan Taşkaya in :issue:`39481`.) PEP 617: New Parser ------------------- Python 3.9 uses a new parser, based on `PEG `_ instead of `LL(1) `_. The new parser's performance is roughly comparable to that of the old parser, but the PEG formalism is more flexible than LL(1) when it comes to designing new language features. We'll start using this flexibility in Python 3.10 and later. The :mod:`ast` module uses the new parser and produces the same AST as the old parser. In Python 3.10, the old parser will be deleted and so will all functionality that depends on it (primarily the :mod:`parser` module, which has long been deprecated). In Python 3.9 *only*, you can switch back to the LL(1) parser using a command line switch (``-X oldparser``) or an environment variable (``PYTHONOLDPARSER=1``). See :pep:`617` for more details. (Contributed by Guido van Rossum, Pablo Galindo and Lysandros Nikolau in :issue:`40334`.) Other Language Changes ====================== * :func:`__import__` now raises :exc:`ImportError` instead of :exc:`ValueError`, which used to occur when a relative import went past its top-level package. (Contributed by Ngalim Siregar in :issue:`37444`.) * Python now gets the absolute path of the script filename specified on the command line (ex: ``python3 script.py``): the ``__file__`` attribute of the :mod:`__main__` module became an absolute path, rather than a relative path. These paths now remain valid after the current directory is changed by :func:`os.chdir`. As a side effect, the traceback also displays the absolute path for :mod:`__main__` module frames in this case. (Contributed by Victor Stinner in :issue:`20443`.) * In the :ref:`Python Development Mode ` and in debug build, the *encoding* and *errors* arguments are now checked for string encoding and decoding operations. Examples: :func:`open`, :meth:`str.encode` and :meth:`bytes.decode`. By default, for best performance, the *errors* argument is only checked at the first encoding/decoding error and the *encoding* argument is sometimes ignored for empty strings. (Contributed by Victor Stinner in :issue:`37388`.) * ``"".replace("", s, n)`` now returns ``s`` instead of an empty string for all non-zero ``n``. It is now consistent with ``"".replace("", s)``. There are similar changes for :class:`bytes` and :class:`bytearray` objects. (Contributed by Serhiy Storchaka in :issue:`28029`.) * Any valid expression can now be used as a :term:`decorator`. Previously, the grammar was much more restrictive. See :pep:`614` for details. (Contributed by Brandt Bucher in :issue:`39702`.) * Improved help for the :mod:`typing` module. Docstrings are now shown for all special forms and special generic aliases (like ``Union`` and ``List``). Using :func:`help` with generic alias like ``List[int]`` will show the help for the correspondent concrete type (``list`` in this case). (Contributed by Serhiy Storchaka in :issue:`40257`.) New Modules =========== zoneinfo -------- The :mod:`zoneinfo` module brings support for the IANA time zone database to the standard library. It adds :class:`zoneinfo.ZoneInfo`, a concrete :class:`datetime.tzinfo` implementation backed by the system's time zone data. Example:: >>> from zoneinfo import ZoneInfo >>> from datetime import datetime, timedelta >>> # Daylight saving time >>> dt = datetime(2020, 10, 31, 12, tzinfo=ZoneInfo("America/Los_Angeles")) >>> print(dt) 2020-10-31 12:00:00-07:00 >>> dt.tzname() 'PDT' >>> # Standard time >>> dt += timedelta(days=7) >>> print(dt) 2020-11-07 12:00:00-08:00 >>> print(dt.tzname()) PST As a fall-back source of data for platforms that don't ship the IANA database, the |tzdata|_ module was released as a first-party package -- distributed via PyPI and maintained by the CPython core team. .. |tzdata| replace:: ``tzdata`` .. _tzdata: https://pypi.org/project/tzdata/ .. seealso:: :pep:`615` -- Support for the IANA Time Zone Database in the Standard Library PEP written and implemented by Paul Ganssle Improved Modules ================ ast --- Added the *indent* option to :func:`~ast.dump` which allows it to produce a multiline indented output. (Contributed by Serhiy Storchaka in :issue:`37995`.) Added :func:`ast.unparse` as a function in the :mod:`ast` module that can be used to unparse an :class:`ast.AST` object and produce a string with code that would produce an equivalent :class:`ast.AST` object when parsed. (Contributed by Pablo Galindo and Batuhan Taskaya in :issue:`38870`.) Added docstrings to AST nodes that contains the ASDL signature used to construct that node. (Contributed by Batuhan Taskaya in :issue:`39638`.) asyncio ------- Due to significant security concerns, the *reuse_address* parameter of :meth:`asyncio.loop.create_datagram_endpoint` is no longer supported. This is because of the behavior of the socket option ``SO_REUSEADDR`` in UDP. For more details, see the documentation for ``loop.create_datagram_endpoint()``. (Contributed by Kyle Stanley, Antoine Pitrou, and Yury Selivanov in :issue:`37228`.) Added a new :term:`coroutine` :meth:`~asyncio.loop.shutdown_default_executor` that schedules a shutdown for the default executor that waits on the :class:`~concurrent.futures.ThreadPoolExecutor` to finish closing. Also, :func:`asyncio.run` has been updated to use the new :term:`coroutine`. (Contributed by Kyle Stanley in :issue:`34037`.) Added :class:`asyncio.PidfdChildWatcher`, a Linux-specific child watcher implementation that polls process file descriptors. (:issue:`38692`) Added a new :term:`coroutine` :func:`asyncio.to_thread`. It is mainly used for running IO-bound functions in a separate thread to avoid blocking the event loop, and essentially works as a high-level version of :meth:`~asyncio.loop.run_in_executor` that can directly take keyword arguments. (Contributed by Kyle Stanley and Yury Selivanov in :issue:`32309`.) compileall ---------- Added new possibility to use hardlinks for duplicated ``.pyc`` files: *hardlink_dupes* parameter and --hardlink-dupes command line option. (Contributed by Lumír 'Frenzy' Balhar in :issue:`40495`.) Added new options for path manipulation in resulting ``.pyc`` files: *stripdir*, *prependdir*, *limit_sl_dest* parameters and -s, -p, -e command line options. Added the possibility to specify the option for an optimization level multiple times. (Contributed by Lumír 'Frenzy' Balhar in :issue:`38112`.) concurrent.futures ------------------ Added a new *cancel_futures* parameter to :meth:`concurrent.futures.Executor.shutdown` that cancels all pending futures which have not started running, instead of waiting for them to complete before shutting down the executor. (Contributed by Kyle Stanley in :issue:`39349`.) Removed daemon threads from :class:`~concurrent.futures.ThreadPoolExecutor` and :class:`~concurrent.futures.ProcessPoolExecutor`. This improves compatibility with subinterpreters and predictability in their shutdown processes. (Contributed by Kyle Stanley in :issue:`39812`.) Workers in :class:`~concurrent.futures.ProcessPoolExecutor` are now spawned on demand, only when there are no available idle workers to reuse. This optimizes startup overhead and reduces the amount of lost CPU time to idle workers. (Contributed by Kyle Stanley in :issue:`39207`.) curses ------ Add :func:`curses.get_escdelay`, :func:`curses.set_escdelay`, :func:`curses.get_tabsize`, and :func:`curses.set_tabsize` functions. (Contributed by Anthony Sottile in :issue:`38312`.) datetime -------- The :meth:`~datetime.date.isocalendar()` of :class:`datetime.date` and :meth:`~datetime.datetime.isocalendar()` of :class:`datetime.datetime` methods now returns a :func:`~collections.namedtuple` instead of a :class:`tuple`. (Contributed by Dong-hee Na in :issue:`24416`.) distutils --------- The :command:`upload` command now creates SHA2-256 and Blake2b-256 hash digests. It skips MD5 on platforms that block MD5 digest. (Contributed by Christian Heimes in :issue:`40698`.) fcntl ----- Added constants :data:`~fcntl.F_OFD_GETLK`, :data:`~fcntl.F_OFD_SETLK` and :data:`~fcntl.F_OFD_SETLKW`. (Contributed by Dong-hee Na in :issue:`38602`.) ftplib ------- :class:`~ftplib.FTP` and :class:`~ftplib.FTP_TLS` now raise a :class:`ValueError` if the given timeout for their constructor is zero to prevent the creation of a non-blocking socket. (Contributed by Dong-hee Na in :issue:`39259`.) functools --------- Add the :class:`functools.TopologicalSorter` class to offer functionality to perform topological sorting of graphs. (Contributed by Pablo Galindo, Tim Peters and Larry Hastings in :issue:`17005`.) gc -- When the garbage collector makes a collection in which some objects resurrect (they are reachable from outside the isolated cycles after the finalizers have been executed), do not block the collection of all objects that are still unreachable. (Contributed by Pablo Galindo and Tim Peters in :issue:`38379`.) Added a new function :func:`gc.is_finalized` to check if an object has been finalized by the garbage collector. (Contributed by Pablo Galindo in :issue:`39322`.) hashlib ------- Builtin hash modules can now be disabled with ``./configure --without-builtin-hashlib-hashes`` or selectively enabled with e.g. ``./configure --with-builtin-hashlib-hashes=sha3,blake2`` to force use of OpenSSL based implementation. (Contributed by Christian Heimes in :issue:`40479`) http ---- HTTP status codes ``103 EARLY_HINTS``, ``418 IM_A_TEAPOT`` and ``425 TOO_EARLY`` are added to :class:`http.HTTPStatus`. (Contributed by Dong-hee Na in :issue:`39509` and Ross Rhodes in :issue:`39507`.) imaplib ------- :class:`~imaplib.IMAP4` and :class:`~imaplib.IMAP4_SSL` now have an optional *timeout* parameter for their constructors. Also, the :meth:`~imaplib.IMAP4.open` method now has an optional *timeout* parameter with this change. The overridden methods of :class:`~imaplib.IMAP4_SSL` and :class:`~imaplib.IMAP4_stream` were applied to this change. (Contributed by Dong-hee Na in :issue:`38615`.) :meth:`imaplib.IMAP4.unselect` is added. :meth:`imaplib.IMAP4.unselect` frees server's resources associated with the selected mailbox and returns the server to the authenticated state. This command performs the same actions as :meth:`imaplib.IMAP4.close`, except that no messages are permanently removed from the currently selected mailbox. (Contributed by Dong-hee Na in :issue:`40375`.) importlib --------- To improve consistency with import statements, :func:`importlib.util.resolve_name` now raises :exc:`ImportError` instead of :exc:`ValueError` for invalid relative import attempts. (Contributed by Ngalim Siregar in :issue:`37444`.) inspect ------- :attr:`inspect.BoundArguments.arguments` is changed from ``OrderedDict`` to regular dict. (Contributed by Inada Naoki in :issue:`36350` and :issue:`39775`.) ipaddress --------- :mod:`ipaddress` now supports IPv6 Scoped Addresses (IPv6 address with suffix ``%``). Scoped IPv6 addresses can be parsed using :class:`ipaddress.IPv6Address`. If present, scope zone ID is available through the :attr:`~ipaddress.IPv6Address.scope_id` attribute. (Contributed by Oleksandr Pavliuk in :issue:`34788`.) math ---- Expanded the :func:`math.gcd` function to handle multiple arguments. Formerly, it only supported two arguments. (Contributed by Serhiy Storchaka in :issue:`39648`.) Add :func:`math.lcm`: return the least common multiple of specified arguments. (Contributed by Mark Dickinson, Ananthakrishnan and Serhiy Storchaka in :issue:`39479` and :issue:`39648`.) Add :func:`math.nextafter`: return the next floating-point value after *x* towards *y*. (Contributed by Victor Stinner in :issue:`39288`.) Add :func:`math.ulp`: return the value of the least significant bit of a float. (Contributed by Victor Stinner in :issue:`39310`.) multiprocessing --------------- The :class:`multiprocessing.SimpleQueue` class has a new :meth:`~multiprocessing.SimpleQueue.close` method to explicitly close the queue. (Contributed by Victor Stinner in :issue:`30966`.) nntplib ------- :class:`~nntplib.NNTP` and :class:`~nntplib.NNTP_SSL` now raise a :class:`ValueError` if the given timeout for their constructor is zero to prevent the creation of a non-blocking socket. (Contributed by Dong-hee Na in :issue:`39259`.) os -- Added :data:`~os.CLD_KILLED` and :data:`~os.CLD_STOPPED` for :attr:`si_code`. (Contributed by Dong-hee Na in :issue:`38493`.) Exposed the Linux-specific :func:`os.pidfd_open` (:issue:`38692`) and :data:`os.P_PIDFD` (:issue:`38713`) for process management with file descriptors. The :func:`os.unsetenv` function is now also available on Windows. (Contributed by Victor Stinner in :issue:`39413`.) The :func:`os.putenv` and :func:`os.unsetenv` functions are now always available. (Contributed by Victor Stinner in :issue:`39395`.) Add :func:`os.waitstatus_to_exitcode` function: convert a wait status to an exit code. (Contributed by Victor Stinner in :issue:`40094`.) pathlib ------- Added :meth:`pathlib.Path.readlink()` which acts similarly to :func:`os.readlink`. (Contributed by Girts Folkmanis in :issue:`30618`) poplib ------ :class:`~poplib.POP3` and :class:`~poplib.POP3_SSL` now raise a :class:`ValueError` if the given timeout for their constructor is zero to prevent the creation of a non-blocking socket. (Contributed by Dong-hee Na in :issue:`39259`.) pprint ------ :mod:`pprint` can now pretty-print :class:`types.SimpleNamespace`. (Contributed by Carl Bordum Hansen in :issue:`37376`.) pydoc ----- The documentation string is now shown not only for class, function, method etc, but for any object that has its own ``__doc__`` attribute. (Contributed by Serhiy Storchaka in :issue:`40257`.) random ------ Add a new :attr:`random.Random.randbytes` method: generate random bytes. (Contributed by Victor Stinner in :issue:`40286`.) signal ------ Exposed the Linux-specific :func:`signal.pidfd_send_signal` for sending to signals to a process using a file descriptor instead of a pid. (:issue:`38712`) smtplib ------- :class:`~smtplib.SMTP` and :class:`~smtplib.SMTP_SSL` now raise a :class:`ValueError` if the given timeout for their constructor is zero to prevent the creation of a non-blocking socket. (Contributed by Dong-hee Na in :issue:`39259`.) :class:`~smtplib.LMTP` constructor now has an optional *timeout* parameter. (Contributed by Dong-hee Na in :issue:`39329`.) socket ------ The :mod:`socket` module now exports the :data:`~socket.CAN_RAW_JOIN_FILTERS` constant on Linux 4.1 and greater. (Contributed by Stefan Tatschner and Zackery Spytz in :issue:`25780`.) The socket module now supports the :data:`~socket.CAN_J1939` protocol on platforms that support it. (Contributed by Karl Ding in :issue:`40291`.) time ---- On AIX, :func:`~time.thread_time` is now implemented with ``thread_cputime()`` which has nanosecond resolution, rather than ``clock_gettime(CLOCK_THREAD_CPUTIME_ID)`` which has a resolution of 10 ms. (Contributed by Batuhan Taskaya in :issue:`40192`) sys --- Add a new :attr:`sys.platlibdir` attribute: name of the platform-specific library directory. It is used to build the path of platform-specific dynamic libraries and the path of the standard library. It is equal to ``"lib"`` on most platforms. On Fedora and SuSE, it is equal to ``"lib64"`` on 64-bit platforms. (Contributed by Jan Matějek, Matěj Cepl, Charalampos Stratakis and Victor Stinner in :issue:`1294959`.) Previously, :attr:`sys.stderr` was block-buffered when non-interactive. Now ``stderr`` defaults to always being line-buffered. (Contributed by Jendrik Seipp in :issue:`13601`.) typing ------ :pep:`593` introduced an :data:`typing.Annotated` type to decorate existing types with context-specific metadata and new ``include_extras`` parameter to :func:`typing.get_type_hints` to access the metadata at runtime. (Contributed by Till Varoquaux and Konstantin Kashin.) unicodedata ----------- The Unicode database has been updated to version 13.0.0. (:issue:`39926`). venv ---- The activation scripts provided by :mod:`venv` now all specify their prompt customization consistently by always using the value specified by ``__VENV_PROMPT__``. Previously some scripts unconditionally used ``__VENV_PROMPT__``, others only if it happened to be set (which was the default case), and one used ``__VENV_NAME__`` instead. (Contributed by Brett Cannon in :issue:`37663`.) xml --- White space characters within attributes are now preserved when serializing :mod:`xml.etree.ElementTree` to XML file. EOLNs are no longer normalized to "\n". This is the result of discussion about how to interpret section 2.11 of XML spec. (Contributed by Mefistotelis in :issue:`39011`.) Optimizations ============= * Optimized the idiom for assignment a temporary variable in comprehensions. Now ``for y in [expr]`` in comprehensions is as fast as a simple assignment ``y = expr``. For example: sums = [s for s in [0] for x in data for s in [s + x]] Unlike to the ``:=`` operator this idiom does not leak a variable to the outer scope. (Contributed by Serhiy Storchaka in :issue:`32856`.) * Optimize signal handling in multithreaded applications. If a thread different than the main thread gets a signal, the bytecode evaluation loop is no longer interrupted at each bytecode instruction to check for pending signals which cannot be handled. Only the main thread of the main interpreter can handle signals. Previously, the bytecode evaluation loop was interrupted at each instruction until the main thread handles signals. (Contributed by Victor Stinner in :issue:`40010`.) * Optimize the :mod:`subprocess` module on FreeBSD using ``closefrom()``. (Contributed by Ed Maste, Conrad Meyer, Kyle Evans, Kubilay Kocak and Victor Stinner in :issue:`38061`.) Here's a summary of performance improvements from Python 3.4 through Python 3.9: .. code-block:: none Python version 3.4 3.5 3.6 3.7 3.8 3.9 -------------- --- --- --- --- --- --- Variable and attribute read access: read_local 7.1 7.1 5.4 5.1 3.9 4.0 read_nonlocal 7.1 8.1 5.8 5.4 4.4 4.8 read_global 15.5 19.0 14.3 13.6 7.6 7.7 read_builtin 21.1 21.6 18.5 19.0 7.5 7.7 read_classvar_from_class 25.6 26.5 20.7 19.5 18.4 18.6 read_classvar_from_instance 22.8 23.5 18.8 17.1 16.4 20.1 read_instancevar 32.4 33.1 28.0 26.3 25.4 27.7 read_instancevar_slots 27.8 31.3 20.8 20.8 20.2 24.5 read_namedtuple 73.8 57.5 45.0 46.8 18.4 23.2 read_boundmethod 37.6 37.9 29.6 26.9 27.7 45.9 Variable and attribute write access: write_local 8.7 9.3 5.5 5.3 4.3 4.2 write_nonlocal 10.5 11.1 5.6 5.5 4.7 4.9 write_global 19.7 21.2 18.0 18.0 15.8 17.2 write_classvar 92.9 96.0 104.6 102.1 39.2 43.2 write_instancevar 44.6 45.8 40.0 38.9 35.5 40.7 write_instancevar_slots 35.6 36.1 27.3 26.6 25.7 27.7 Data structure read access: read_list 24.2 24.5 20.8 20.8 19.0 21.1 read_deque 24.7 25.5 20.2 20.6 19.8 21.6 read_dict 24.3 25.7 22.3 23.0 21.0 22.5 read_strdict 22.6 24.3 19.5 21.2 18.9 21.6 Data structure write access: write_list 27.1 28.5 22.5 21.6 20.0 21.6 write_deque 28.7 30.1 22.7 21.8 23.5 23.2 write_dict 31.4 33.3 29.3 29.2 24.7 27.8 write_strdict 28.4 29.9 27.5 25.2 23.1 29.8 Stack (or queue) operations: list_append_pop 93.4 112.7 75.4 74.2 50.8 53.9 deque_append_pop 43.5 57.0 49.4 49.2 42.5 45.5 deque_append_popleft 43.7 57.3 49.7 49.7 42.8 45.5 Timing loop: loop_overhead 0.5 0.6 0.4 0.3 0.3 0.3 These results were generated from the variable access benchmark script at: ``Tools/scripts/var_access_benchmark.py``. The benchmark script displays timings in nanoseconds. The benchmarks were measured on an `Intel® Core™ i7-4960HQ processor `_ running the macOS 64-bit builds found at `python.org `_. Deprecated ========== * The distutils ``bdist_msi`` command is now deprecated, use ``bdist_wheel`` (wheel packages) instead. (Contributed by Hugo van Kemenade in :issue:`39586`.) * Currently :func:`math.factorial` accepts :class:`float` instances with non-negative integer values (like ``5.0``). It raises a :exc:`ValueError` for non-integral and negative floats. It is now deprecated. In future Python versions it will raise a :exc:`TypeError` for all floats. (Contributed by Serhiy Storchaka in :issue:`37315`.) * The :mod:`parser` module is deprecated and will be removed in future versions of Python. For the majority of use cases, users can leverage the Abstract Syntax Tree (AST) generation and compilation stage, using the :mod:`ast` module. * Using :data:`NotImplemented` in a boolean context has been deprecated, as it is almost exclusively the result of incorrect rich comparator implementations. It will be made a :exc:`TypeError` in a future version of Python. (Contributed by Josh Rosenberg in :issue:`35712`.) * The :mod:`random` module currently accepts any hashable type as a possible seed value. Unfortunately, some of those types are not guaranteed to have a deterministic hash value. After Python 3.9, the module will restrict its seeds to :const:`None`, :class:`int`, :class:`float`, :class:`str`, :class:`bytes`, and :class:`bytearray`. * Opening the :class:`~gzip.GzipFile` file for writing without specifying the *mode* argument is deprecated. In future Python versions it will always be opened for reading by default. Specify the *mode* argument for opening it for writing and silencing a warning. (Contributed by Serhiy Storchaka in :issue:`28286`.) * Deprecated the ``split()`` method of :class:`_tkinter.TkappType` in favour of the ``splitlist()`` method which has more consistent and predicable behavior. (Contributed by Serhiy Storchaka in :issue:`38371`.) * The explicit passing of coroutine objects to :func:`asyncio.wait` has been deprecated and will be removed in version 3.11. (Contributed by Yury Selivanov and Kyle Stanley in :issue:`34790`.) * binhex4 and hexbin4 standards are now deprecated. The :mod:`binhex` module and the following :mod:`binascii` functions are now deprecated: * :func:`~binascii.b2a_hqx`, :func:`~binascii.a2b_hqx` * :func:`~binascii.rlecode_hqx`, :func:`~binascii.rledecode_hqx` (Contributed by Victor Stinner in :issue:`39353`.) * :mod:`ast` classes ``slice``, ``Index`` and ``ExtSlice`` are considered deprecated and will be removed in future Python versions. ``value`` itself should be used instead of ``Index(value)``. ``Tuple(slices, Load())`` should be used instead of ``ExtSlice(slices)``. (Contributed by Serhiy Storchaka in :issue:`32892`.) * :mod:`ast` classes ``Suite``, ``Param``, ``AugLoad`` and ``AugStore`` are considered deprecated and will be removed in future Python versions. They were not generated by the parser and not accepted by the code generator in Python 3. (Contributed by Batuhan Taskaya in :issue:`39639` and :issue:`39969` and Serhiy Storchaka in :issue:`39988`.) * The :c:func:`PyEval_InitThreads` and :c:func:`PyEval_ThreadsInitialized` functions are now deprecated and will be removed in Python 3.11. Calling :c:func:`PyEval_InitThreads` now does nothing. The :term:`GIL` is initialized by :c:func:`Py_Initialize()` since Python 3.7. (Contributed by Victor Stinner in :issue:`39877`.) * Passing ``None`` as the first argument to the :func:`shlex.split` function has been deprecated. (Contributed by Zackery Spytz in :issue:`33262`.) * The :mod:`lib2to3` module now emits a :exc:`PendingDeprecationWarning`. Python 3.9 switched to a PEG parser (see :pep:`617`), and Python 3.10 may include new language syntax that is not parsable by lib2to3's LL(1) parser. The ``lib2to3`` module may be removed from the standard library in a future Python version. Consider third-party alternatives such as `LibCST`_ or `parso`_. (Contributed by Carl Meyer in :issue:`40360`.) .. _LibCST: https://libcst.readthedocs.io/ .. _parso: https://parso.readthedocs.io/ Removed ======= * The erroneous version at :data:`unittest.mock.__version__` has been removed. * :class:`nntplib.NNTP`: ``xpath()`` and ``xgtitle()`` methods have been removed. These methods are deprecated since Python 3.3. Generally, these extensions are not supported or not enabled by NNTP server administrators. For ``xgtitle()``, please use :meth:`nntplib.NNTP.descriptions` or :meth:`nntplib.NNTP.description` instead. (Contributed by Dong-hee Na in :issue:`39366`.) * :class:`array.array`: ``tostring()`` and ``fromstring()`` methods have been removed. They were aliases to ``tobytes()`` and ``frombytes()``, deprecated since Python 3.2. (Contributed by Victor Stinner in :issue:`38916`.) * The undocumented ``sys.callstats()`` function has been removed. Since Python 3.7, it was deprecated and always returned :const:`None`. It required a special build option ``CALL_PROFILE`` which was already removed in Python 3.7. (Contributed by Victor Stinner in :issue:`37414`.) * The ``sys.getcheckinterval()`` and ``sys.setcheckinterval()`` functions have been removed. They were deprecated since Python 3.2. Use :func:`sys.getswitchinterval` and :func:`sys.setswitchinterval` instead. (Contributed by Victor Stinner in :issue:`37392`.) * The C function ``PyImport_Cleanup()`` has been removed. It was documented as: "Empty the module table. For internal use only." (Contributed by Victor Stinner in :issue:`36710`.) * ``_dummy_thread`` and ``dummy_threading`` modules have been removed. These modules were deprecated since Python 3.7 which requires threading support. (Contributed by Victor Stinner in :issue:`37312`.) * ``aifc.openfp()`` alias to ``aifc.open()``, ``sunau.openfp()`` alias to ``sunau.open()``, and ``wave.openfp()`` alias to :func:`wave.open()` have been removed. They were deprecated since Python 3.7. (Contributed by Victor Stinner in :issue:`37320`.) * The :meth:`~threading.Thread.isAlive()` method of :class:`threading.Thread` has been removed. It was deprecated since Python 3.8. Use :meth:`~threading.Thread.is_alive()` instead. (Contributed by Dong-hee Na in :issue:`37804`.) * Methods ``getchildren()`` and ``getiterator()`` of classes :class:`~xml.etree.ElementTree.ElementTree` and :class:`~xml.etree.ElementTree.Element` in the :mod:`~xml.etree.ElementTree` module have been removed. They were deprecated in Python 3.2. Use ``iter(x)`` or ``list(x)`` instead of ``x.getchildren()`` and ``x.iter()`` or ``list(x.iter())`` instead of ``x.getiterator()``. The ``xml.etree.cElementTree`` module has been removed, use the :mod:`xml.etree.ElementTree` module instead. Since Python 3.3 the ``xml.etree.cElementTree`` module has been deprecated, the ``xml.etree.ElementTree`` module uses a fast implementation whenever available. (Contributed by Serhiy Storchaka in :issue:`36543`.) * The old :mod:`plistlib` API has been removed, it was deprecated since Python 3.4. Use the :func:`~plistlib.load`, :func:`~plistlib.loads`, :func:`~plistlib.dump`, and :func:`~plistlib.dumps` functions. Additionally, the *use_builtin_types* parameter was removed, standard :class:`bytes` objects are always used instead. (Contributed by Jon Janzen in :issue:`36409`.) * The C function ``PyThreadState_DeleteCurrent()`` has been removed. It was not documented. (Contributed by Joannah Nanjekye in :issue:`37878`.) * The C function ``PyGen_NeedsFinalizing`` has been removed. It was not documented, tested, or used anywhere within CPython after the implementation of :pep:`442`. Patch by Joannah Nanjekye. (Contributed by Joannah Nanjekye in :issue:`15088`) * ``base64.encodestring()`` and ``base64.decodestring()``, aliases deprecated since Python 3.1, have been removed: use :func:`base64.encodebytes` and :func:`base64.decodebytes` instead. (Contributed by Victor Stinner in :issue:`39351`.) * ``fractions.gcd()`` function has been removed, it was deprecated since Python 3.5 (:issue:`22486`): use :func:`math.gcd` instead. (Contributed by Victor Stinner in :issue:`39350`.) * The *buffering* parameter of :class:`bz2.BZ2File` has been removed. Since Python 3.0, it was ignored and using it emitted a :exc:`DeprecationWarning`. Pass an open file object to control how the file is opened. (Contributed by Victor Stinner in :issue:`39357`.) * The *encoding* parameter of :func:`json.loads` has been removed. As of Python 3.1, it was deprecated and ignored; using it has emitted a :exc:`DeprecationWarning` since Python 3.8. (Contributed by Inada Naoki in :issue:`39377`) * ``with (await asyncio.lock):`` and ``with (yield from asyncio.lock):`` statements are not longer supported, use ``async with lock`` instead. The same is correct for ``asyncio.Condition`` and ``asyncio.Semaphore``. (Contributed by Andrew Svetlov in :issue:`34793`.) * The :func:`sys.getcounts` function, the ``-X showalloccount`` command line option and the ``show_alloc_count`` field of the C structure :c:type:`PyConfig` have been removed. They required a special Python build by defining ``COUNT_ALLOCS`` macro. (Contributed by Victor Stinner in :issue:`39489`.) * The ``_field_types`` attribute of the :class:`typing.NamedTuple` class has been removed. It was deprecated deprecated since Python 3.8. Use the ``__annotations__`` attribute instead. (Contributed by Serhiy Storchaka in :issue:`40182`.) * The :meth:`symtable.SymbolTable.has_exec` method has been removed. It was deprecated since 2006, and only returning ``False`` when it's called. (Contributed by Batuhan Taskaya in :issue:`40208`) Porting to Python 3.9 ===================== This section lists previously described changes and other bugfixes that may require changes to your code. Changes in the Python API ------------------------- * :func:`__import__` and :func:`importlib.util.resolve_name` now raise :exc:`ImportError` where it previously raised :exc:`ValueError`. Callers catching the specific exception type and supporting both Python 3.9 and earlier versions will need to catch both using ``except (ImportError, ValueError):``. * The :mod:`venv` activation scripts no longer special-case when ``__VENV_PROMPT__`` is set to ``""``. * The :meth:`select.epoll.unregister` method no longer ignores the :data:`~errno.EBADF` error. (Contributed by Victor Stinner in :issue:`39239`.) * The *compresslevel* parameter of :class:`bz2.BZ2File` became keyword-only, since the *buffering* parameter has been removed. (Contributed by Victor Stinner in :issue:`39357`.) * Simplified AST for subscription. Simple indices will be represented by their value, extended slices will be represented as tuples. ``Index(value)`` will return a ``value`` itself, ``ExtSlice(slices)`` will return ``Tuple(slices, Load())``. (Contributed by Serhiy Storchaka in :issue:`34822`.) * The :mod:`importlib` module now ignores the :envvar:`PYTHONCASEOK` environment variable when the :option:`-E` or :option:`-I` command line options are being used. * The *encoding* parameter has been added to the classes :class:`ftplib.FTP` and :class:`ftplib.FTP_TLS` as a keyword-only parameter, and the default encoding is changed from Latin-1 to UTF-8 to follow :rfc:`2640`. * :meth:`asyncio.loop.shutdown_default_executor` has been added to :class:`~asyncio.AbstractEventLoop`, meaning alternative event loops that inherit from it should have this method defined. (Contributed by Kyle Stanley in :issue:`34037`.) * The constant values of future flags in the :mod:`__future__` module is updated in order to prevent collision with compiler flags. Previously ``PyCF_ALLOW_TOP_LEVEL_AWAIT`` was clashing with ``CO_FUTURE_DIVISION``. (Contributed by Batuhan Taskaya in :issue:`39562`) * ``array('u')`` now uses ``wchar_t`` as C type instead of ``Py_UNICODE``. This change doesn't affect to its behavior because ``Py_UNICODE`` is alias of ``wchar_t`` since Python 3.3. (Contributed by Inada Naoki in :issue:`34538`.) CPython bytecode changes ------------------------ * The :opcode:`LOAD_ASSERTION_ERROR` opcode was added for handling the :keyword:`assert` statement. Previously, the assert statement would not work correctly if the :exc:`AssertionError` exception was being shadowed. (Contributed by Zackery Spytz in :issue:`34880`.) Build Changes ============= * Add ``--with-platlibdir`` option to the ``configure`` script: name of the platform-specific library directory, stored in the new :attr:`sys.platlibdir` attribute. See :attr:`sys.platlibdir` attribute for more information. (Contributed by Jan Matějek, Matěj Cepl, Charalampos Stratakis and Victor Stinner in :issue:`1294959`.) * The ``COUNT_ALLOCS`` special build macro has been removed. (Contributed by Victor Stinner in :issue:`39489`.) * On non-Windows platforms, the :c:func:`setenv` and :c:func:`unsetenv` functions are now required to build Python. (Contributed by Victor Stinner in :issue:`39395`.) C API Changes ============= New Features ------------ * Add :c:func:`PyFrame_GetCode` function: get a frame code. Add :c:func:`PyFrame_GetBack` function: get the frame next outer frame. (Contributed by Victor Stinner in :issue:`40421`.) * Add :c:func:`PyFrame_GetLineNumber` to the limited C API. (Contributed by Victor Stinner in :issue:`40421`.) * Add :c:func:`PyThreadState_GetInterpreter` and :c:func:`PyInterpreterState_Get` functions to get the interpreter. Add :c:func:`PyThreadState_GetFrame` function to get the current frame of a Python thread state. Add :c:func:`PyThreadState_GetID` function: get the unique identifier of a Python thread state. (Contributed by Victor Stinner in :issue:`39947`.) * Add a new public :c:func:`PyObject_CallNoArgs` function to the C API, which calls a callable Python object without any arguments. It is the most efficient way to call a callable Python object without any argument. (Contributed by Victor Stinner in :issue:`37194`.) * Changes in the limited C API (if ``Py_LIMITED_API`` macro is defined): * Provide :c:func:`Py_EnterRecursiveCall` and :c:func:`Py_LeaveRecursiveCall` as regular functions for the limited API. Previously, there were defined as macros, but these macros didn't compile with the limited C API which cannot access ``PyThreadState.recursion_depth`` field (the structure is opaque in the limited C API). * ``PyObject_INIT()`` and ``PyObject_INIT_VAR()`` become regular "opaque" function to hide implementation details. (Contributed by Victor Stinner in :issue:`38644` and :issue:`39542`.) * The :c:func:`PyModule_AddType` function is added to help adding a type to a module. (Contributed by Dong-hee Na in :issue:`40024`.) * Add the functions :c:func:`PyObject_GC_IsTracked` and :c:func:`PyObject_GC_IsFinalized` to the public API to allow to query if Python objects are being currently tracked or have been already finalized by the garbage collector respectively. (Contributed by Pablo Galindo in :issue:`40241`.) Porting to Python 3.9 --------------------- * ``PyInterpreterState.eval_frame`` (:pep:`523`) now requires a new mandatory *tstate* parameter (``PyThreadState*``). (Contributed by Victor Stinner in :issue:`38500`.) * Extension modules: :c:member:`~PyModuleDef.m_traverse`, :c:member:`~PyModuleDef.m_clear` and :c:member:`~PyModuleDef.m_free` functions of :c:type:`PyModuleDef` are no longer called if the module state was requested but is not allocated yet. This is the case immediately after the module is created and before the module is executed (:c:data:`Py_mod_exec` function). More precisely, these functions are not called if :c:member:`~PyModuleDef.m_size` is greater than 0 and the module state (as returned by :c:func:`PyModule_GetState`) is ``NULL``. Extension modules without module state (``m_size <= 0``) are not affected. * If :c:func:`Py_AddPendingCall` is called in a subinterpreter, the function is now scheduled to be called from the subinterpreter, rather than being called from the main interpreter. Each subinterpreter now has its own list of scheduled calls. (Contributed by Victor Stinner in :issue:`39984`.) * The Windows registry is no longer used to initialize :data:`sys.path` when the ``-E`` option is used (if :c:member:`PyConfig.use_environment` is set to ``0``). This is significant when embedding Python on Windows. (Contributed by Zackery Spytz in :issue:`8901`.) * The global variable :c:data:`PyStructSequence_UnnamedField` is now a constant and refers to a constant string. (Contributed by Serhiy Storchaka in :issue:`38650`.) Removed ------- * Exclude ``PyFPE_START_PROTECT()`` and ``PyFPE_END_PROTECT()`` macros of ``pyfpe.h`` from the limited C API. (Contributed by Victor Stinner in :issue:`38835`.) * The ``tp_print`` slot of :ref:`PyTypeObject ` has been removed. It was used for printing objects to files in Python 2.7 and before. Since Python 3.0, it has been ignored and unused. (Contributed by Jeroen Demeyer in :issue:`36974`.) * Changes in the limited C API (if ``Py_LIMITED_API`` macro is defined): * Exclude the following functions from the limited C API: * ``_Py_CheckRecursionLimit`` * ``_Py_NewReference()`` * ``_Py_ForgetReference()`` * ``_PyTraceMalloc_NewReference()`` * ``_Py_GetRefTotal()`` * The trashcan mechanism which never worked in the limited C API. * ``PyTrash_UNWIND_LEVEL`` * ``Py_TRASHCAN_BEGIN_CONDITION`` * ``Py_TRASHCAN_BEGIN`` * ``Py_TRASHCAN_END`` * ``Py_TRASHCAN_SAFE_BEGIN`` * ``Py_TRASHCAN_SAFE_END`` * Move following functions and definitions to the internal C API: * ``_PyDebug_PrintTotalRefs()`` * ``_Py_PrintReferences()`` * ``_Py_PrintReferenceAddresses()`` * ``_Py_tracemalloc_config`` * ``_Py_AddToAllObjects()`` (specific to ``Py_TRACE_REFS`` build) (Contributed by Victor Stinner in :issue:`38644` and :issue:`39542`.) * Remove ``_PyRuntime.getframe`` hook and remove ``_PyThreadState_GetFrame`` macro which was an alias to ``_PyRuntime.getframe``. They were only exposed by the internal C API. Remove also ``PyThreadFrameGetter`` type. (Contributed by Victor Stinner in :issue:`39946`.) * Remove the following functions from the C API. Call :c:func:`PyGC_Collect` explicitly to clear all free lists. (Contributed by Inada Naoki and Victor Stinner in :issue:`37340`, :issue:`38896` and :issue:`40428`.) * ``PyAsyncGen_ClearFreeLists()`` * ``PyContext_ClearFreeList()`` * ``PyDict_ClearFreeList()`` * ``PyFloat_ClearFreeList()`` * ``PyFrame_ClearFreeList()`` * ``PyList_ClearFreeList()`` * ``PyMethod_ClearFreeList()`` and ``PyCFunction_ClearFreeList()``: the free lists of bound method objects have been removed. * ``PySet_ClearFreeList()``: the set free list has been removed in Python 3.4. * ``PyTuple_ClearFreeList()`` * ``PyUnicode_ClearFreeList()``: the Unicode free list has been removed in Python 3.3. * Remove ``_PyUnicode_ClearStaticStrings()`` function. (Contributed by Victor Stinner in :issue:`39465`.)