mirror of https://github.com/python/cpython
1911 lines
77 KiB
ReStructuredText
1911 lines
77 KiB
ReStructuredText
****************************
|
|
What's New In Python 3.11
|
|
****************************
|
|
|
|
:Release: |release|
|
|
:Date: |today|
|
|
|
|
.. Rules for maintenance:
|
|
|
|
* Anyone can add text to this document. Do not spend very much time
|
|
on the wording of your changes, because your text will probably
|
|
get rewritten to some degree.
|
|
|
|
* The maintainer will go through Misc/NEWS periodically and add
|
|
changes; it's therefore more important to add your changes to
|
|
Misc/NEWS than to this file.
|
|
|
|
* This is not a complete list of every single change; completeness
|
|
is the purpose of Misc/NEWS. Some changes I consider too small
|
|
or esoteric to include. If such a change is added to the text,
|
|
I'll just remove it. (This is another reason you shouldn't spend
|
|
too much time on writing your addition.)
|
|
|
|
* If you want to draw your new text to the attention of the
|
|
maintainer, add 'XXX' to the beginning of the paragraph or
|
|
section.
|
|
|
|
* It's OK to just add a fragmentary note about a change. For
|
|
example: "XXX Describe the transmogrify() function added to the
|
|
socket module." The maintainer will research the change and
|
|
write the necessary text.
|
|
|
|
* You can comment out your additions if you like, but it's not
|
|
necessary (especially when a final release is some months away).
|
|
|
|
* Credit the author of a patch or bugfix. Just the name is
|
|
sufficient; the e-mail address isn't necessary.
|
|
|
|
* It's helpful to add the bug/patch number as a comment:
|
|
|
|
XXX Describe the transmogrify() function added to the socket
|
|
module.
|
|
(Contributed by P.Y. Developer in :issue:`12345`.)
|
|
|
|
This saves the maintainer the effort of going through the Mercurial log
|
|
when researching a change.
|
|
|
|
This article explains the new features in Python 3.11, compared to 3.10.
|
|
|
|
For full details, see the :ref:`changelog <changelog>`.
|
|
|
|
.. note::
|
|
|
|
Prerelease users should be aware that this document is currently in draft
|
|
form. It will be updated substantially as Python 3.11 moves towards release,
|
|
so it's worth checking back even after reading earlier versions.
|
|
|
|
|
|
Summary -- Release highlights
|
|
=============================
|
|
|
|
.. This section singles out the most important changes in Python 3.11.
|
|
Brevity is key.
|
|
|
|
- Python 3.11 is up to 10-60% faster than Python 3.10. On average, we measured a
|
|
1.25x speedup on the standard benchmark suite. See `Faster CPython`_ for details.
|
|
|
|
.. PEP-sized items next.
|
|
|
|
New syntax features:
|
|
|
|
* :pep:`654`: Exception Groups and ``except*``.
|
|
(Contributed by Irit Katriel in :issue:`45292`.)
|
|
|
|
New typing features:
|
|
|
|
* :pep:`646`: Variadic generics.
|
|
* :pep:`655`: Marking individual TypedDict items as required or potentially-missing.
|
|
* :pep:`673`: ``Self`` type.
|
|
* :pep:`675`: Arbitrary literal string type.
|
|
|
|
Security improvements:
|
|
|
|
* New :option:`-P` command line option and :envvar:`PYTHONSAFEPATH` environment
|
|
variable to not prepend a potentially unsafe path to :data:`sys.path` such as
|
|
the current directory, the script's directory or an empty string.
|
|
|
|
|
|
New Features
|
|
============
|
|
|
|
.. _whatsnew311-pep657:
|
|
|
|
Enhanced error locations in tracebacks
|
|
--------------------------------------
|
|
|
|
When printing tracebacks, the interpreter will now point to the exact expression
|
|
that caused the error instead of just the line. For example:
|
|
|
|
.. code-block:: python
|
|
|
|
Traceback (most recent call last):
|
|
File "distance.py", line 11, in <module>
|
|
print(manhattan_distance(p1, p2))
|
|
^^^^^^^^^^^^^^^^^^^^^^^^^^
|
|
File "distance.py", line 6, in manhattan_distance
|
|
return abs(point_1.x - point_2.x) + abs(point_1.y - point_2.y)
|
|
^^^^^^^^^
|
|
AttributeError: 'NoneType' object has no attribute 'x'
|
|
|
|
Previous versions of the interpreter would point to just the line making it
|
|
ambiguous which object was ``None``. These enhanced errors can also be helpful
|
|
when dealing with deeply nested dictionary objects and multiple function calls,
|
|
|
|
.. code-block:: python
|
|
|
|
Traceback (most recent call last):
|
|
File "query.py", line 37, in <module>
|
|
magic_arithmetic('foo')
|
|
^^^^^^^^^^^^^^^^^^^^^^^
|
|
File "query.py", line 18, in magic_arithmetic
|
|
return add_counts(x) / 25
|
|
^^^^^^^^^^^^^
|
|
File "query.py", line 24, in add_counts
|
|
return 25 + query_user(user1) + query_user(user2)
|
|
^^^^^^^^^^^^^^^^^
|
|
File "query.py", line 32, in query_user
|
|
return 1 + query_count(db, response['a']['b']['c']['user'], retry=True)
|
|
~~~~~~~~~~~~~~~~~~^^^^^
|
|
TypeError: 'NoneType' object is not subscriptable
|
|
|
|
as well as complex arithmetic expressions:
|
|
|
|
.. code-block:: python
|
|
|
|
Traceback (most recent call last):
|
|
File "calculation.py", line 54, in <module>
|
|
result = (x / y / z) * (a / b / c)
|
|
~~~~~~^~~
|
|
ZeroDivisionError: division by zero
|
|
|
|
See :pep:`657` for more details. (Contributed by Pablo Galindo, Batuhan Taskaya
|
|
and Ammar Askar in :issue:`43950`.)
|
|
|
|
.. note::
|
|
This feature requires storing column positions in code objects which may
|
|
result in a small increase of disk usage of compiled Python files or
|
|
interpreter memory usage. To avoid storing the extra information and/or
|
|
deactivate printing the extra traceback information, the
|
|
:option:`-X` ``no_debug_ranges`` command line flag or the :envvar:`PYTHONNODEBUGRANGES`
|
|
environment variable can be used.
|
|
|
|
Column information for code objects
|
|
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
|
|
|
|
The information used by the enhanced traceback feature is made available as a
|
|
general API that can be used to correlate bytecode instructions with source
|
|
code. This information can be retrieved using:
|
|
|
|
- The :meth:`codeobject.co_positions` method in Python.
|
|
- The :c:func:`PyCode_Addr2Location` function in the C-API.
|
|
|
|
The :option:`-X` ``no_debug_ranges`` option and the environment variable
|
|
:envvar:`PYTHONNODEBUGRANGES` can be used to disable this feature.
|
|
|
|
See :pep:`657` for more details. (Contributed by Pablo Galindo, Batuhan Taskaya
|
|
and Ammar Askar in :issue:`43950`.)
|
|
|
|
Exceptions can be enriched with notes (PEP 678)
|
|
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
|
|
|
|
The :meth:`add_note` method was added to :exc:`BaseException`. It can be
|
|
used to enrich exceptions with context information which is not available
|
|
at the time when the exception is raised. The notes added appear in the
|
|
default traceback. See :pep:`678` for more details. (Contributed by
|
|
Irit Katriel in :issue:`45607`.)
|
|
|
|
|
|
.. _new-feat-related-type-hints-311:
|
|
|
|
New Features Related to Type Hints
|
|
==================================
|
|
|
|
This section covers major changes affecting :pep:`484` type hints and
|
|
the :mod:`typing` module.
|
|
|
|
PEP 646: Variadic generics
|
|
--------------------------
|
|
|
|
:pep:`484` introduced :data:`~typing.TypeVar`, enabling creation
|
|
of generics parameterised with a single type. :pep:`646` introduces
|
|
:data:`~typing.TypeVarTuple`, enabling parameterisation
|
|
with an *arbitrary* number of types. In other words,
|
|
a :data:`~typing.TypeVarTuple` is a *variadic* type variable,
|
|
enabling *variadic* generics. This enables a wide variety of use cases.
|
|
In particular, it allows the type of array-like structures
|
|
in numerical computing libraries such as NumPy and TensorFlow to be
|
|
parameterised with the array *shape*. Static type checkers will now
|
|
be able to catch shape-related bugs in code that uses these libraries.
|
|
|
|
See :pep:`646` for more details.
|
|
|
|
(Contributed by Matthew Rahtz in :issue:`43224`, with contributions by
|
|
Serhiy Storchaka and Jelle Zijlstra. PEP written by Mark Mendoza, Matthew
|
|
Rahtz, Pradeep Kumar Srinivasan, and Vincent Siles.)
|
|
|
|
PEP 655: Marking individual ``TypedDict`` items as required or not-required
|
|
---------------------------------------------------------------------------
|
|
|
|
:data:`~typing.Required` and :data:`~typing.NotRequired` provide a
|
|
straightforward way to mark whether individual items in a
|
|
:data:`~typing.TypedDict` must be present. Previously this was only possible
|
|
using inheritance.
|
|
|
|
Fields are still required by default, unless the ``total=False``
|
|
parameter is set.
|
|
For example, the following specifies a dictionary with one required and
|
|
one not-required key::
|
|
|
|
class Movie(TypedDict):
|
|
title: str
|
|
year: NotRequired[int]
|
|
|
|
m1: Movie = {"title": "Black Panther", "year": 2018} # ok
|
|
m2: Movie = {"title": "Star Wars"} # ok (year is not required)
|
|
m3: Movie = {"year": 2022} # error (missing required field title)
|
|
|
|
The following definition is equivalent::
|
|
|
|
class Movie(TypedDict, total=False):
|
|
title: Required[str]
|
|
year: int
|
|
|
|
See :pep:`655` for more details.
|
|
|
|
(Contributed by David Foster and Jelle Zijlstra in :issue:`47087`. PEP
|
|
written by David Foster.)
|
|
|
|
PEP 673: ``Self`` type
|
|
----------------------
|
|
|
|
The new :data:`~typing.Self` annotation provides a simple and intuitive
|
|
way to annotate methods that return an instance of their class. This
|
|
behaves the same as the :data:`~typing.TypeVar`-based approach specified
|
|
in :pep:`484` but is more concise and easier to follow.
|
|
|
|
Common use cases include alternative constructors provided as classmethods
|
|
and :meth:`~object.__enter__` methods that return ``self``::
|
|
|
|
class MyLock:
|
|
def __enter__(self) -> Self:
|
|
self.lock()
|
|
return self
|
|
|
|
...
|
|
|
|
class MyInt:
|
|
@classmethod
|
|
def fromhex(cls, s: str) -> Self:
|
|
return cls(int(s, 16))
|
|
|
|
...
|
|
|
|
:data:`~typing.Self` can also be used to annotate method parameters
|
|
or attributes of the same type as their enclosing class.
|
|
|
|
See :pep:`673` for more details.
|
|
|
|
(Contributed by James Hilton-Balfe in :issue:`46534`. PEP written by
|
|
Pradeep Kumar Srinivasan and James Hilton-Balfe.)
|
|
|
|
PEP 675: Arbitrary literal string type
|
|
--------------------------------------
|
|
|
|
The new :data:`~typing.LiteralString` annotation may be used to indicate
|
|
that a function parameter can be of any literal string type. This allows
|
|
a function to accept arbitrary literal string types, as well as strings
|
|
created from other literal strings. Type checkers can then
|
|
enforce that sensitive functions, such as those that execute SQL
|
|
statements or shell commands, are called only with static arguments,
|
|
providing protection against injection attacks.
|
|
|
|
For example, a SQL query function could be annotated as follows::
|
|
|
|
def run_query(sql: LiteralString) -> ...
|
|
...
|
|
|
|
def caller(
|
|
arbitrary_string: str,
|
|
query_string: LiteralString,
|
|
table_name: LiteralString,
|
|
) -> None:
|
|
run_query("SELECT * FROM students") # ok
|
|
run_query(query_string) # ok
|
|
run_query("SELECT * FROM " + table_name) # ok
|
|
run_query(arbitrary_string) # type checker error
|
|
run_query( # type checker error
|
|
f"SELECT * FROM students WHERE name = {arbitrary_string}"
|
|
)
|
|
|
|
See :pep:`675` for more details.
|
|
|
|
(Contributed by Jelle Zijlstra in :issue:`47088`. PEP written by Pradeep
|
|
Kumar Srinivasan and Graham Bleaney.)
|
|
|
|
PEP 681: Data Class Transforms
|
|
------------------------------
|
|
|
|
The new :data:`~typing.dataclass_transform` annotation may be used to
|
|
decorate a function that is itself a decorator, a class, or a metaclass.
|
|
The presence of ``@dataclass_transform()`` tells a static type checker that the
|
|
decorated function, class, or metaclass performs runtime "magic" that
|
|
transforms a class, endowing it with dataclass-like behaviors.
|
|
|
|
For example::
|
|
|
|
# The ``create_model`` decorator is defined by a library.
|
|
@typing.dataclass_transform()
|
|
def create_model(cls: Type[_T]) -> Type[_T]:
|
|
cls.__init__ = ...
|
|
cls.__eq__ = ...
|
|
cls.__ne__ = ...
|
|
return cls
|
|
|
|
# The ``create_model`` decorator can now be used to create new model
|
|
# classes, like this:
|
|
@create_model
|
|
class CustomerModel:
|
|
id: int
|
|
name: str
|
|
|
|
c = CustomerModel(id=327, name="John Smith")
|
|
|
|
See :pep:`681` for more details.
|
|
|
|
(Contributed by Jelle Zijlstra in :gh:`91860`. PEP written by
|
|
Erik De Bonte and Eric Traut.)
|
|
|
|
Other Language Changes
|
|
======================
|
|
|
|
* Starred expressions can be used in :ref:`for statements<for>`. (See
|
|
:issue:`46725` for more details.)
|
|
|
|
* Asynchronous comprehensions are now allowed inside comprehensions in
|
|
asynchronous functions. Outer comprehensions implicitly become
|
|
asynchronous. (Contributed by Serhiy Storchaka in :issue:`33346`.)
|
|
|
|
* A :exc:`TypeError` is now raised instead of an :exc:`AttributeError` in
|
|
:meth:`contextlib.ExitStack.enter_context` and
|
|
:meth:`contextlib.AsyncExitStack.enter_async_context` for objects which do not
|
|
support the :term:`context manager` or :term:`asynchronous context manager`
|
|
protocols correspondingly.
|
|
(Contributed by Serhiy Storchaka in :issue:`44471`.)
|
|
|
|
* A :exc:`TypeError` is now raised instead of an :exc:`AttributeError` in
|
|
:keyword:`with` and :keyword:`async with` statements for objects which do not
|
|
support the :term:`context manager` or :term:`asynchronous context manager`
|
|
protocols correspondingly.
|
|
(Contributed by Serhiy Storchaka in :issue:`12022`.)
|
|
|
|
* Added :meth:`object.__getstate__` which provides the default
|
|
implementation of the ``__getstate__()`` method. :mod:`Copying <copy>`
|
|
and :mod:`pickling <pickle>` instances of subclasses of builtin types
|
|
:class:`bytearray`, :class:`set`, :class:`frozenset`,
|
|
:class:`collections.OrderedDict`, :class:`collections.deque`,
|
|
:class:`weakref.WeakSet`, and :class:`datetime.tzinfo` now copies and
|
|
pickles instance attributes implemented as :term:`slots <__slots__>`.
|
|
(Contributed by Serhiy Storchaka in :issue:`26579`.)
|
|
|
|
* Add :option:`-P` command line option and :envvar:`PYTHONSAFEPATH` environment
|
|
variable to not prepend a potentially unsafe path to :data:`sys.path` such as
|
|
the current directory, the script's directory or an empty string.
|
|
(Contributed by Victor Stinner in :gh:`57684`.)
|
|
|
|
|
|
Other CPython Implementation Changes
|
|
====================================
|
|
|
|
* Special methods :meth:`complex.__complex__` and :meth:`bytes.__bytes__` are implemented to
|
|
support :class:`typing.SupportsComplex` and :class:`typing.SupportsBytes` protocols.
|
|
(Contributed by Mark Dickinson and Dong-hee Na in :issue:`24234`.)
|
|
|
|
* ``siphash13`` is added as a new internal hashing algorithms. It has similar security
|
|
properties as ``siphash24`` but it is slightly faster for long inputs. ``str``, ``bytes``,
|
|
and some other types now use it as default algorithm for :func:`hash`. :pep:`552`
|
|
hash-based pyc files now use ``siphash13``, too.
|
|
(Contributed by Inada Naoki in :issue:`29410`.)
|
|
|
|
* When an active exception is re-raised by a :keyword:`raise` statement with no parameters,
|
|
the traceback attached to this exception is now always ``sys.exc_info()[1].__traceback__``.
|
|
This means that changes made to the traceback in the current :keyword:`except` clause are
|
|
reflected in the re-raised exception.
|
|
(Contributed by Irit Katriel in :issue:`45711`.)
|
|
|
|
* The interpreter state's representation of handled exceptions (a.k.a exc_info, or
|
|
_PyErr_StackItem) now has only the ``exc_value`` field, ``exc_type`` and ``exc_traceback``
|
|
have been removed as their values can be derived from ``exc_value``.
|
|
(Contributed by Irit Katriel in :issue:`45711`.)
|
|
|
|
* A new command line option for the Windows installer ``AppendPath`` has been added.
|
|
It behaves similiar to ``PrependPath`` but appends the install and scripts directories
|
|
instead of prepending them.
|
|
(Contributed by Bastian Neuburger in :issue:`44934`.)
|
|
|
|
|
|
New Modules
|
|
===========
|
|
|
|
* A new module, :mod:`tomllib`, was added for parsing TOML.
|
|
(Contributed by Taneli Hukkinen in :issue:`40059`.)
|
|
|
|
* :mod:`wsgiref.types`, containing WSGI-specific types for static type
|
|
checking, was added.
|
|
(Contributed by Sebastian Rittau in :issue:`42012`.)
|
|
|
|
|
|
Improved Modules
|
|
================
|
|
|
|
asyncio
|
|
-------
|
|
|
|
* Add raw datagram socket functions to the event loop:
|
|
:meth:`~asyncio.AbstractEventLoop.sock_sendto`,
|
|
:meth:`~asyncio.AbstractEventLoop.sock_recvfrom` and
|
|
:meth:`~asyncio.AbstractEventLoop.sock_recvfrom_into`.
|
|
(Contributed by Alex Grönholm in :issue:`46805`.)
|
|
|
|
* Add :meth:`~asyncio.streams.StreamWriter.start_tls` method for upgrading
|
|
existing stream-based connections to TLS. (Contributed by Ian Good in
|
|
:issue:`34975`.)
|
|
|
|
datetime
|
|
--------
|
|
|
|
* :meth:`datetime.date.fromisoformat`, :meth:`datetime.time.fromisoformat` and
|
|
:meth:`datetime.datetime.fromisoformat` can now be used to parse most ISO 8601
|
|
formats (barring only those that support fractional hours and minutes).
|
|
(Contributed by Paul Ganssle in :gh:`80010`.)
|
|
|
|
fractions
|
|
---------
|
|
|
|
* Support :PEP:`515`-style initialization of :class:`~fractions.Fraction` from
|
|
string. (Contributed by Sergey B Kirpichev in :issue:`44258`.)
|
|
|
|
* :class:`~fractions.Fraction` now implements an ``__int__`` method, so
|
|
that an ``isinstance(some_fraction, typing.SupportsInt)`` check passes.
|
|
(Contributed by Mark Dickinson in :issue:`44547`.)
|
|
|
|
functools
|
|
---------
|
|
|
|
* :func:`functools.singledispatch` now supports :data:`types.UnionType`
|
|
and :data:`typing.Union` as annotations to the dispatch argument.::
|
|
|
|
>>> from functools import singledispatch
|
|
>>> @singledispatch
|
|
... def fun(arg, verbose=False):
|
|
... if verbose:
|
|
... print("Let me just say,", end=" ")
|
|
... print(arg)
|
|
...
|
|
>>> @fun.register
|
|
... def _(arg: int | float, verbose=False):
|
|
... if verbose:
|
|
... print("Strength in numbers, eh?", end=" ")
|
|
... print(arg)
|
|
...
|
|
>>> from typing import Union
|
|
>>> @fun.register
|
|
... def _(arg: Union[list, set], verbose=False):
|
|
... if verbose:
|
|
... print("Enumerate this:")
|
|
... for i, elem in enumerate(arg):
|
|
... print(i, elem)
|
|
...
|
|
|
|
(Contributed by Yurii Karabas in :issue:`46014`.)
|
|
|
|
hashlib
|
|
-------
|
|
|
|
* :func:`hashlib.blake2b` and :func:`hashlib.blake2s` now prefer `libb2`_
|
|
over Python's vendored copy.
|
|
(Contributed by Christian Heimes in :issue:`47095`.)
|
|
|
|
* The internal ``_sha3`` module with SHA3 and SHAKE algorithms now uses
|
|
*tiny_sha3* instead of the *Keccak Code Package* to reduce code and binary
|
|
size. The :mod:`hashlib` module prefers optimized SHA3 and SHAKE
|
|
implementations from OpenSSL. The change affects only installations without
|
|
OpenSSL support.
|
|
(Contributed by Christian Heimes in :issue:`47098`.)
|
|
|
|
IDLE and idlelib
|
|
----------------
|
|
|
|
* Apply syntax highlighting to `.pyi` files. (Contributed by Alex
|
|
Waygood and Terry Jan Reedy in :issue:`45447`.)
|
|
|
|
inspect
|
|
-------
|
|
* Add :func:`inspect.getmembers_static`: return all members without
|
|
triggering dynamic lookup via the descriptor protocol. (Contributed by
|
|
Weipeng Hong in :issue:`30533`.)
|
|
|
|
* Add :func:`inspect.ismethodwrapper` for checking if the type of an object is a
|
|
:class:`~types.MethodWrapperType`. (Contributed by Hakan Çelik in :issue:`29418`.)
|
|
|
|
* Change the frame-related functions in the :mod:`inspect` module to return a
|
|
regular object (that is backwards compatible with the old tuple-like
|
|
interface) that include the extended :pep:`657` position information (end
|
|
line number, column and end column). The affected functions are:
|
|
:func:`inspect.getframeinfo`, :func:`inspect.getouterframes`, :func:`inspect.getinnerframes`,
|
|
:func:`inspect.stack` and :func:`inspect.trace`. (Contributed by Pablo Galindo in
|
|
:gh:`88116`)
|
|
|
|
locale
|
|
------
|
|
|
|
* Add :func:`locale.getencoding` to get the current locale encoding. It is similar to
|
|
``locale.getpreferredencoding(False)`` but ignores the
|
|
:ref:`Python UTF-8 Mode <utf8-mode>`.
|
|
|
|
math
|
|
----
|
|
|
|
* Add :func:`math.exp2`: return 2 raised to the power of x.
|
|
(Contributed by Gideon Mitchell in :issue:`45917`.)
|
|
|
|
* Add :func:`math.cbrt`: return the cube root of x.
|
|
(Contributed by Ajith Ramachandran in :issue:`44357`.)
|
|
|
|
* The behaviour of two :func:`math.pow` corner cases was changed, for
|
|
consistency with the IEEE 754 specification. The operations
|
|
``math.pow(0.0, -math.inf)`` and ``math.pow(-0.0, -math.inf)`` now return
|
|
``inf``. Previously they raised :exc:`ValueError`. (Contributed by Mark
|
|
Dickinson in :issue:`44339`.)
|
|
|
|
* The :data:`math.nan` value is now always available.
|
|
(Contributed by Victor Stinner in :issue:`46917`.)
|
|
|
|
|
|
operator
|
|
--------
|
|
|
|
* A new function ``operator.call`` has been added, such that
|
|
``operator.call(obj, *args, **kwargs) == obj(*args, **kwargs)``.
|
|
(Contributed by Antony Lee in :issue:`44019`.)
|
|
|
|
|
|
os
|
|
--
|
|
|
|
* On Windows, :func:`os.urandom` now uses ``BCryptGenRandom()``,
|
|
instead of ``CryptGenRandom()`` which is deprecated.
|
|
(Contributed by Dong-hee Na in :issue:`44611`.)
|
|
|
|
|
|
pathlib
|
|
-------
|
|
|
|
* :meth:`~pathlib.Path.glob` and :meth:`~pathlib.Path.rglob` return only
|
|
directories if *pattern* ends with a pathname components separator:
|
|
:data:`~os.sep` or :data:`~os.altsep`.
|
|
(Contributed by Eisuke Kawasima in :issue:`22276` and :issue:`33392`.)
|
|
|
|
re
|
|
--
|
|
|
|
* Atomic grouping (``(?>...)``) and possessive quantifiers (``*+``, ``++``,
|
|
``?+``, ``{m,n}+``) are now supported in regular expressions.
|
|
(Contributed by Jeffrey C. Jacobs and Serhiy Storchaka in :issue:`433030`.)
|
|
|
|
shutil
|
|
------
|
|
|
|
* Add optional parameter *dir_fd* in :func:`shutil.rmtree`.
|
|
(Contributed by Serhiy Storchaka in :issue:`46245`.)
|
|
|
|
|
|
socket
|
|
------
|
|
|
|
* Add CAN Socket support for NetBSD.
|
|
(Contributed by Thomas Klausner in :issue:`30512`.)
|
|
|
|
* :meth:`~socket.create_connection` has an option to raise, in case of
|
|
failure to connect, an :exc:`ExceptionGroup` containing all errors
|
|
instead of only raising the last error.
|
|
(Contributed by Irit Katriel in :issue:`29980`).
|
|
|
|
sqlite3
|
|
-------
|
|
|
|
* You can now disable the authorizer by passing :const:`None` to
|
|
:meth:`~sqlite3.Connection.set_authorizer`.
|
|
(Contributed by Erlend E. Aasland in :issue:`44491`.)
|
|
|
|
* Collation name :meth:`~sqlite3.Connection.create_collation` can now
|
|
contain any Unicode character. Collation names with invalid characters
|
|
now raise :exc:`UnicodeEncodeError` instead of :exc:`sqlite3.ProgrammingError`.
|
|
(Contributed by Erlend E. Aasland in :issue:`44688`.)
|
|
|
|
* :mod:`sqlite3` exceptions now include the SQLite extended error code as
|
|
:attr:`~sqlite3.Error.sqlite_errorcode` and the SQLite error name as
|
|
:attr:`~sqlite3.Error.sqlite_errorname`.
|
|
(Contributed by Aviv Palivoda, Daniel Shahaf, and Erlend E. Aasland in
|
|
:issue:`16379` and :issue:`24139`.)
|
|
|
|
* Add :meth:`~sqlite3.Connection.setlimit` and
|
|
:meth:`~sqlite3.Connection.getlimit` to :class:`sqlite3.Connection` for
|
|
setting and getting SQLite limits by connection basis.
|
|
(Contributed by Erlend E. Aasland in :issue:`45243`.)
|
|
|
|
* :mod:`sqlite3` now sets :attr:`sqlite3.threadsafety` based on the default
|
|
threading mode the underlying SQLite library has been compiled with.
|
|
(Contributed by Erlend E. Aasland in :issue:`45613`.)
|
|
|
|
* :mod:`sqlite3` C callbacks now use unraisable exceptions if callback
|
|
tracebacks are enabled. Users can now register an
|
|
:func:`unraisable hook handler <sys.unraisablehook>` to improve their debug
|
|
experience.
|
|
(Contributed by Erlend E. Aasland in :issue:`45828`.)
|
|
|
|
* Fetch across rollback no longer raises :exc:`~sqlite3.InterfaceError`.
|
|
Instead we leave it to the SQLite library to handle these cases.
|
|
(Contributed by Erlend E. Aasland in :issue:`44092`.)
|
|
|
|
* Add :meth:`~sqlite3.Connection.serialize` and
|
|
:meth:`~sqlite3.Connection.deserialize` to :class:`sqlite3.Connection` for
|
|
serializing and deserializing databases.
|
|
(Contributed by Erlend E. Aasland in :issue:`41930`.)
|
|
|
|
* Add :meth:`~sqlite3.Connection.create_window_function` to
|
|
:class:`sqlite3.Connection` for creating aggregate window functions.
|
|
(Contributed by Erlend E. Aasland in :issue:`34916`.)
|
|
|
|
* Add :meth:`~sqlite3.Connection.blobopen` to :class:`sqlite3.Connection`.
|
|
:class:`sqlite3.Blob` allows incremental I/O operations on blobs.
|
|
(Contributed by Aviv Palivoda and Erlend E. Aasland in :issue:`24905`)
|
|
|
|
|
|
sys
|
|
---
|
|
|
|
* :func:`sys.exc_info` now derives the ``type`` and ``traceback`` fields
|
|
from the ``value`` (the exception instance), so when an exception is
|
|
modified while it is being handled, the changes are reflected in
|
|
the results of subsequent calls to :func:`exc_info`.
|
|
(Contributed by Irit Katriel in :issue:`45711`.)
|
|
|
|
* Add :func:`sys.exception` which returns the active exception instance
|
|
(equivalent to ``sys.exc_info()[1]``).
|
|
(Contributed by Irit Katriel in :issue:`46328`.)
|
|
|
|
* Add the :data:`sys.flags.safe_path <sys.flags>` flag.
|
|
(Contributed by Victor Stinner in :gh:`57684`.)
|
|
|
|
|
|
sysconfig
|
|
---------
|
|
|
|
* Two new :ref:`installation schemes <installation_paths>`
|
|
(*posix_venv*, *nt_venv* and *venv*) were added and are used when Python
|
|
creates new virtual environments or when it is running from a virtual
|
|
environment.
|
|
The first two schemes (*posix_venv* and *nt_venv*) are OS-specific
|
|
for non-Windows and Windows, the *venv* is essentially an alias to one of
|
|
them according to the OS Python runs on.
|
|
This is useful for downstream distributors who modify
|
|
:func:`sysconfig.get_preferred_scheme`.
|
|
Third party code that creates new virtual environments should use the new
|
|
*venv* installation scheme to determine the paths, as does :mod:`venv`.
|
|
(Contributed by Miro Hrončok in :issue:`45413`.)
|
|
|
|
|
|
threading
|
|
---------
|
|
|
|
* On Unix, if the ``sem_clockwait()`` function is available in the C library
|
|
(glibc 2.30 and newer), the :meth:`threading.Lock.acquire` method now uses
|
|
the monotonic clock (:data:`time.CLOCK_MONOTONIC`) for the timeout, rather
|
|
than using the system clock (:data:`time.CLOCK_REALTIME`), to not be affected
|
|
by system clock changes.
|
|
(Contributed by Victor Stinner in :issue:`41710`.)
|
|
|
|
|
|
time
|
|
----
|
|
|
|
* On Unix, :func:`time.sleep` now uses the ``clock_nanosleep()`` or
|
|
``nanosleep()`` function, if available, which has a resolution of 1 nanosecond
|
|
(10\ :sup:`-9` seconds), rather than using ``select()`` which has a resolution
|
|
of 1 microsecond (10\ :sup:`-6` seconds).
|
|
(Contributed by Benjamin Szőke and Victor Stinner in :issue:`21302`.)
|
|
|
|
* On Windows 8.1 and newer, :func:`time.sleep` now uses a waitable timer based
|
|
on `high-resolution timers
|
|
<https://docs.microsoft.com/en-us/windows-hardware/drivers/kernel/high-resolution-timers>`_
|
|
which has a resolution of 100 nanoseconds (10\ :sup:`-7` seconds). Previously,
|
|
it had a resolution of 1 millisecond (10\ :sup:`-3` seconds).
|
|
(Contributed by Benjamin Szőke, Dong-hee Na, Eryk Sun and Victor Stinner in :issue:`21302` and :issue:`45429`.)
|
|
|
|
typing
|
|
------
|
|
|
|
For major changes, see :ref:`new-feat-related-type-hints-311`.
|
|
|
|
* Add :func:`typing.assert_never` and :class:`typing.Never`.
|
|
:func:`typing.assert_never` is useful for asking a type checker to confirm
|
|
that a line of code is not reachable. At runtime, it raises an
|
|
:exc:`AssertionError`.
|
|
(Contributed by Jelle Zijlstra in :gh:`90633`.)
|
|
|
|
* Add :func:`typing.reveal_type`. This is useful for asking a type checker
|
|
what type it has inferred for a given expression. At runtime it prints
|
|
the type of the received value.
|
|
(Contributed by Jelle Zijlstra in :gh:`90572`.)
|
|
|
|
* Add :func:`typing.assert_type`. This is useful for asking a type checker
|
|
to confirm that the type it has inferred for a given expression matches
|
|
the given type. At runtime it simply returns the received value.
|
|
(Contributed by Jelle Zijlstra in :gh:`90638`.)
|
|
|
|
* Allow subclassing of :class:`typing.Any`. This is useful for avoiding
|
|
type checker errors related to highly dynamic class, such as mocks.
|
|
(Contributed by Shantanu Jain in :gh:`91154`.)
|
|
|
|
* The :func:`typing.final` decorator now sets the ``__final__`` attributed on
|
|
the decorated object.
|
|
(Contributed by Jelle Zijlstra in :gh:`90500`.)
|
|
|
|
* The :func:`typing.get_overloads` function can be used for introspecting
|
|
the overloads of a function. :func:`typing.clear_overloads` can be used
|
|
to clear all registered overloads of a function.
|
|
(Contributed by Jelle Zijlstra in :gh:`89263`.)
|
|
|
|
* :data:`typing.TypedDict` subclasses can now be generic. (Contributed by
|
|
Samodya Abey in :gh:`89026`.)
|
|
|
|
* :class:`~typing.NamedTuple` subclasses can now be generic.
|
|
(Contributed by Serhiy Storchaka in :issue:`43923`.)
|
|
|
|
|
|
tkinter
|
|
-------
|
|
|
|
* Added method ``info_patchlevel()`` which returns the exact version of
|
|
the Tcl library as a named tuple similar to :data:`sys.version_info`.
|
|
(Contributed by Serhiy Storchaka in :gh:`91827`.)
|
|
|
|
|
|
unicodedata
|
|
-----------
|
|
|
|
* The Unicode database has been updated to version 14.0.0. (:issue:`45190`).
|
|
|
|
|
|
unittest
|
|
--------
|
|
|
|
* Added methods :meth:`~unittest.TestCase.enterContext` and
|
|
:meth:`~unittest.TestCase.enterClassContext` of class
|
|
:class:`~unittest.TestCase`, method
|
|
:meth:`~unittest.IsolatedAsyncioTestCase.enterAsyncContext` of
|
|
class :class:`~unittest.IsolatedAsyncioTestCase` and function
|
|
:func:`unittest.enterModuleContext`.
|
|
(Contributed by Serhiy Storchaka in :issue:`45046`.)
|
|
|
|
|
|
venv
|
|
----
|
|
|
|
* When new Python virtual environments are created, the *venv*
|
|
:ref:`sysconfig installation scheme <installation_paths>` is used
|
|
to determine the paths inside the environment.
|
|
When Python runs in a virtual environment, the same installation scheme
|
|
is the default.
|
|
That means that downstream distributors can change the default sysconfig install
|
|
scheme without changing behavior of virtual environments.
|
|
Third party code that also creates new virtual environments should do the same.
|
|
(Contributed by Miro Hrončok in :issue:`45413`.)
|
|
|
|
warnings
|
|
--------
|
|
|
|
* :func:`warnings.catch_warnings` now accepts arguments for :func:`warnings.simplefilter`,
|
|
providing a more concise way to locally ignore warnings or convert them to errors.
|
|
(Contributed by Zac Hatfield-Dodds in :issue:`47074`.)
|
|
|
|
zipfile
|
|
-------
|
|
|
|
* Added support for specifying member name encoding for reading
|
|
metadata in the zipfile's directory and file headers.
|
|
(Contributed by Stephen J. Turnbull and Serhiy Storchaka in :issue:`28080`.)
|
|
|
|
fcntl
|
|
-----
|
|
|
|
* On FreeBSD, the :attr:`F_DUP2FD` and :attr:`F_DUP2FD_CLOEXEC` flags respectively
|
|
are supported, the former equals to ``dup2`` usage while the latter set
|
|
the ``FD_CLOEXEC`` flag in addition.
|
|
|
|
|
|
Optimizations
|
|
=============
|
|
|
|
* Compiler now optimizes simple C-style formatting with literal format
|
|
containing only format codes ``%s``, ``%r`` and ``%a`` and makes it as
|
|
fast as corresponding f-string expression.
|
|
(Contributed by Serhiy Storchaka in :issue:`28307`.)
|
|
|
|
* "Zero-cost" exceptions are implemented. The cost of ``try`` statements is
|
|
almost eliminated when no exception is raised.
|
|
(Contributed by Mark Shannon in :issue:`40222`.)
|
|
|
|
* Pure ASCII strings are now normalized in constant time by :func:`unicodedata.normalize`.
|
|
(Contributed by Dong-hee Na in :issue:`44987`.)
|
|
|
|
* :mod:`math` functions :func:`~math.comb` and :func:`~math.perm` are now up
|
|
to 10 times or more faster for large arguments (the speed up is larger for
|
|
larger *k*).
|
|
(Contributed by Serhiy Storchaka in :issue:`37295`.)
|
|
|
|
* Dict don't store hash value when all inserted keys are Unicode objects.
|
|
This reduces dict size. For example, ``sys.getsizeof(dict.fromkeys("abcdefg"))``
|
|
becomes 272 bytes from 352 bytes on 64bit platform.
|
|
(Contributed by Inada Naoki in :issue:`46845`.)
|
|
|
|
* :mod:`re`'s regular expression matching engine has been partially refactored,
|
|
and now uses computed gotos (or "threaded code") on supported platforms. As a
|
|
result, Python 3.11 executes the `pyperformance regular expression benchmarks
|
|
<https://pyperformance.readthedocs.io/benchmarks.html#regex-dna>`_ up to 10%
|
|
faster than Python 3.10.
|
|
|
|
|
|
Faster CPython
|
|
==============
|
|
|
|
CPython 3.11 is on average `25% faster <https://github.com/faster-cpython/ideas/blob/main/main-vs-310.rst>`_
|
|
than CPython 3.10 when measured with the
|
|
`pyperformance <https://github.com/python/pyperformance>`_ benchmark suite,
|
|
and compiled with GCC on Ubuntu Linux. Depending on your workload, the speedup
|
|
could be up to 10-60% faster.
|
|
|
|
This project focuses on two major areas in Python: faster startup and faster
|
|
runtime. Other optimizations not under this project are listed in `Optimizations`_.
|
|
|
|
Faster Startup
|
|
--------------
|
|
|
|
Frozen imports / Static code objects
|
|
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
|
|
|
|
Python caches bytecode in the :ref:`__pycache__<tut-pycache>` directory to
|
|
speed up module loading.
|
|
|
|
Previously in 3.10, Python module execution looked like this:
|
|
|
|
.. code-block:: text
|
|
|
|
Read __pycache__ -> Unmarshal -> Heap allocated code object -> Evaluate
|
|
|
|
In Python 3.11, the core modules essential for Python startup are "frozen".
|
|
This means that their code objects (and bytecode) are statically allocated
|
|
by the interpreter. This reduces the steps in module execution process to this:
|
|
|
|
.. code-block:: text
|
|
|
|
Statically allocated code object -> Evaluate
|
|
|
|
Interpreter startup is now 10-15% faster in Python 3.11. This has a big
|
|
impact for short-running programs using Python.
|
|
|
|
(Contributed by Eric Snow, Guido van Rossum and Kumar Aditya in numerous issues.)
|
|
|
|
|
|
Faster Runtime
|
|
--------------
|
|
|
|
Cheaper, lazy Python frames
|
|
~~~~~~~~~~~~~~~~~~~~~~~~~~~
|
|
Python frames are created whenever Python calls a Python function. This frame
|
|
holds execution information. The following are new frame optimizations:
|
|
|
|
- Streamlined the frame creation process.
|
|
- Avoided memory allocation by generously re-using frame space on the C stack.
|
|
- Streamlined the internal frame struct to contain only essential information.
|
|
Frames previously held extra debugging and memory management information.
|
|
|
|
Old-style frame objects are now created only when requested by debuggers or
|
|
by Python introspection functions such as ``sys._getframe`` or
|
|
``inspect.currentframe``. For most user code, no frame objects are
|
|
created at all. As a result, nearly all Python functions calls have sped
|
|
up significantly. We measured a 3-7% speedup in pyperformance.
|
|
|
|
(Contributed by Mark Shannon in :issue:`44590`.)
|
|
|
|
.. _inline-calls:
|
|
|
|
Inlined Python function calls
|
|
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
|
|
During a Python function call, Python will call an evaluating C function to
|
|
interpret that function's code. This effectively limits pure Python recursion to
|
|
what's safe for the C stack.
|
|
|
|
In 3.11, when CPython detects Python code calling another Python function,
|
|
it sets up a new frame, and "jumps" to the new code inside the new frame. This
|
|
avoids calling the C interpreting function altogether.
|
|
|
|
Most Python function calls now consume no C stack space. This speeds up
|
|
most of such calls. In simple recursive functions like fibonacci or
|
|
factorial, a 1.7x speedup was observed. This also means recursive functions
|
|
can recurse significantly deeper (if the user increases the recursion limit).
|
|
We measured a 1-3% improvement in pyperformance.
|
|
|
|
(Contributed by Pablo Galindo and Mark Shannon in :issue:`45256`.)
|
|
|
|
PEP 659: Specializing Adaptive Interpreter
|
|
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
|
|
:pep:`659` is one of the key parts of the faster CPython project. The general
|
|
idea is that while Python is a dynamic language, most code has regions where
|
|
objects and types rarely change. This concept is known as *type stability*.
|
|
|
|
At runtime, Python will try to look for common patterns and type stability
|
|
in the executing code. Python will then replace the current operation with a
|
|
more specialized one. This specialized operation uses fast paths available only
|
|
to those use cases/types, which generally outperform their generic
|
|
counterparts. This also brings in another concept called *inline caching*, where
|
|
Python caches the results of expensive operations directly in the bytecode.
|
|
|
|
The specializer will also combine certain common instruction pairs into one
|
|
superinstruction. This reduces the overhead during execution.
|
|
|
|
Python will only specialize
|
|
when it sees code that is "hot" (executed multiple times). This prevents Python
|
|
from wasting time for run-once code. Python can also de-specialize when code is
|
|
too dynamic or when the use changes. Specialization is attempted periodically,
|
|
and specialization attempts are not too expensive. This allows specialization
|
|
to adapt to new circumstances.
|
|
|
|
(PEP written by Mark Shannon, with ideas inspired by Stefan Brunthaler.
|
|
See :pep:`659` for more information. Implementation by Mark Shannon and Brandt
|
|
Bucher, with additional help from Irit Katriel and Dennis Sweeney.)
|
|
|
|
..
|
|
If I missed out anyone, please add them.
|
|
|
|
+---------------+--------------------+-------------------------------------------------------+-------------------+-------------------+
|
|
| Operation | Form | Specialization | Operation speedup | Contributor(s) |
|
|
| | | | (up to) | |
|
|
+===============+====================+=======================================================+===================+===================+
|
|
| Binary | ``x+x; x*x; x-x;`` | Binary add, multiply and subtract for common types | 10% | Mark Shannon, |
|
|
| operations | | such as ``int``, ``float``, and ``str`` take custom | | Dong-hee Na, |
|
|
| | | fast paths for their underlying types. | | Brandt Bucher, |
|
|
| | | | | Dennis Sweeney |
|
|
+---------------+--------------------+-------------------------------------------------------+-------------------+-------------------+
|
|
| Subscript | ``a[i]`` | Subscripting container types such as ``list``, | 10-25% | Irit Katriel, |
|
|
| | | ``tuple`` and ``dict`` directly index the underlying | | Mark Shannon |
|
|
| | | data structures. | | |
|
|
| | | | | |
|
|
| | | Subscripting custom ``__getitem__`` | | |
|
|
| | | is also inlined similar to :ref:`inline-calls`. | | |
|
|
+---------------+--------------------+-------------------------------------------------------+-------------------+-------------------+
|
|
| Store | ``a[i] = z`` | Similar to subscripting specialization above. | 10-25% | Dennis Sweeney |
|
|
| subscript | | | | |
|
|
+---------------+--------------------+-------------------------------------------------------+-------------------+-------------------+
|
|
| Calls | ``f(arg)`` | Calls to common builtin (C) functions and types such | 20% | Mark Shannon, |
|
|
| | ``C(arg)`` | as ``len`` and ``str`` directly call their underlying | | Ken Jin |
|
|
| | | C version. This avoids going through the internal | | |
|
|
| | | calling convention. | | |
|
|
| | | | | |
|
|
+---------------+--------------------+-------------------------------------------------------+-------------------+-------------------+
|
|
| Load | ``print`` | The object's index in the globals/builtins namespace | [1]_ | Mark Shannon |
|
|
| global | ``len`` | is cached. Loading globals and builtins require | | |
|
|
| variable | | zero namespace lookups. | | |
|
|
+---------------+--------------------+-------------------------------------------------------+-------------------+-------------------+
|
|
| Load | ``o.attr`` | Similar to loading global variables. The attribute's | [2]_ | Mark Shannon |
|
|
| attribute | | index inside the class/object's namespace is cached. | | |
|
|
| | | In most cases, attribute loading will require zero | | |
|
|
| | | namespace lookups. | | |
|
|
+---------------+--------------------+-------------------------------------------------------+-------------------+-------------------+
|
|
| Load | ``o.meth()`` | The actual address of the method is cached. Method | 10-20% | Ken Jin, |
|
|
| methods for | | loading now has no namespace lookups -- even for | | Mark Shannon |
|
|
| call | | classes with long inheritance chains. | | |
|
|
+---------------+--------------------+-------------------------------------------------------+-------------------+-------------------+
|
|
| Store | ``o.attr = z`` | Similar to load attribute optimization. | 2% | Mark Shannon |
|
|
| attribute | | | in pyperformance | |
|
|
+---------------+--------------------+-------------------------------------------------------+-------------------+-------------------+
|
|
| Unpack | ``*seq`` | Specialized for common containers such as ``list`` | 8% | Brandt Bucher |
|
|
| Sequence | | and ``tuple``. Avoids internal calling convention. | | |
|
|
+---------------+--------------------+-------------------------------------------------------+-------------------+-------------------+
|
|
|
|
.. [1] A similar optimization already existed since Python 3.8. 3.11
|
|
specializes for more forms and reduces some overhead.
|
|
|
|
.. [2] A similar optimization already existed since Python 3.10.
|
|
3.11 specializes for more forms. Furthermore, all attribute loads should
|
|
be sped up by :issue:`45947`.
|
|
|
|
|
|
Misc
|
|
----
|
|
|
|
* Objects now require less memory due to lazily created object namespaces. Their
|
|
namespace dictionaries now also share keys more freely.
|
|
(Contributed Mark Shannon in :issue:`45340` and :issue:`40116`.)
|
|
|
|
* A more concise representation of exceptions in the interpreter reduced the
|
|
time required for catching an exception by about 10%.
|
|
(Contributed by Irit Katriel in :issue:`45711`.)
|
|
|
|
FAQ
|
|
---
|
|
|
|
| Q: How should I write my code to utilize these speedups?
|
|
|
|
|
| A: You don't have to change your code. Write Pythonic code that follows common
|
|
best practices. The Faster CPython project optimizes for common code
|
|
patterns we observe.
|
|
|
|
|
|
|
|
| Q: Will CPython 3.11 use more memory?
|
|
|
|
|
| A: Maybe not. We don't expect memory use to exceed 20% more than 3.10.
|
|
This is offset by memory optimizations for frame objects and object
|
|
dictionaries as mentioned above.
|
|
|
|
|
|
|
|
| Q: I don't see any speedups in my workload. Why?
|
|
|
|
|
| A: Certain code won't have noticeable benefits. If your code spends most of
|
|
its time on I/O operations, or already does most of its
|
|
computation in a C extension library like numpy, there won't be significant
|
|
speedup. This project currently benefits pure-Python workloads the most.
|
|
|
|
|
| Furthermore, the pyperformance figures are a geometric mean. Even within the
|
|
pyperformance benchmarks, certain benchmarks have slowed down slightly, while
|
|
others have sped up by nearly 2x!
|
|
|
|
|
|
|
|
| Q: Is there a JIT compiler?
|
|
|
|
|
| A: No. We're still exploring other optimizations.
|
|
|
|
|
|
About
|
|
-----
|
|
|
|
Faster CPython explores optimizations for :term:`CPython`. The main team is
|
|
funded by Microsoft to work on this full-time. Pablo Galindo Salgado is also
|
|
funded by Bloomberg LP to work on the project part-time. Finally, many
|
|
contributors are volunteers from the community.
|
|
|
|
|
|
CPython bytecode changes
|
|
========================
|
|
|
|
* Replaced all numeric ``BINARY_*`` and ``INPLACE_*`` instructions with a single
|
|
:opcode:`BINARY_OP` implementation.
|
|
|
|
* Replaced the three call instructions: :opcode:`CALL_FUNCTION`,
|
|
:opcode:`CALL_FUNCTION_KW` and :opcode:`CALL_METHOD` with
|
|
:opcode:`PUSH_NULL`, :opcode:`PRECALL`, :opcode:`CALL`,
|
|
and :opcode:`KW_NAMES`.
|
|
This decouples the argument shifting for methods from the handling of
|
|
keyword arguments and allows better specialization of calls.
|
|
|
|
* Removed ``COPY_DICT_WITHOUT_KEYS`` and ``GEN_START``.
|
|
|
|
* :opcode:`MATCH_CLASS` and :opcode:`MATCH_KEYS` no longer push an additional
|
|
boolean value indicating whether the match succeeded or failed. Instead, they
|
|
indicate failure with :const:`None` (where a tuple of extracted values would
|
|
otherwise be).
|
|
|
|
* Replace several stack manipulation instructions (``DUP_TOP``, ``DUP_TOP_TWO``,
|
|
``ROT_TWO``, ``ROT_THREE``, ``ROT_FOUR``, and ``ROT_N``) with new
|
|
:opcode:`COPY` and :opcode:`SWAP` instructions.
|
|
|
|
* Replaced :opcode:`JUMP_IF_NOT_EXC_MATCH` by :opcode:`CHECK_EXC_MATCH` which
|
|
performs the check but does not jump.
|
|
|
|
* Replaced :opcode:`JUMP_IF_NOT_EG_MATCH` by :opcode:`CHECK_EG_MATCH` which
|
|
performs the check but does not jump.
|
|
|
|
* Replaced :opcode:`JUMP_ABSOLUTE` by the relative :opcode:`JUMP_BACKWARD`.
|
|
|
|
* Added :opcode:`JUMP_BACKWARD_NO_INTERRUPT`, which is used in certain loops where it
|
|
is undesirable to handle interrupts.
|
|
|
|
* Replaced :opcode:`POP_JUMP_IF_TRUE` and :opcode:`POP_JUMP_IF_FALSE` by
|
|
the relative :opcode:`POP_JUMP_FORWARD_IF_TRUE`, :opcode:`POP_JUMP_BACKWARD_IF_TRUE`,
|
|
:opcode:`POP_JUMP_FORWARD_IF_FALSE` and :opcode:`POP_JUMP_BACKWARD_IF_FALSE`.
|
|
|
|
* Added :opcode:`POP_JUMP_FORWARD_IF_NOT_NONE`, :opcode:`POP_JUMP_BACKWARD_IF_NOT_NONE`,
|
|
:opcode:`POP_JUMP_FORWARD_IF_NONE` and :opcode:`POP_JUMP_BACKWARD_IF_NONE`
|
|
opcodes to speed up conditional jumps.
|
|
|
|
* :opcode:`JUMP_IF_TRUE_OR_POP` and :opcode:`JUMP_IF_FALSE_OR_POP` are now
|
|
relative rather than absolute.
|
|
|
|
|
|
Deprecated
|
|
==========
|
|
|
|
* Chaining :class:`classmethod` descriptors (introduced in :issue:`19072`)
|
|
is now deprecated. It can no longer be used to wrap other descriptors
|
|
such as :class:`property`. The core design of this feature was flawed
|
|
and caused a number of downstream problems. To "pass-through" a
|
|
:class:`classmethod`, consider using the ``__wrapped__`` attribute
|
|
that was added in Python 3.10.
|
|
(Contributed by Raymond Hettinger in :gh:`89519`.)
|
|
|
|
* Octal escapes with value larger than ``0o377`` now produce
|
|
a :exc:`DeprecationWarning`.
|
|
In a future Python version they will be a :exc:`SyntaxWarning` and
|
|
eventually a :exc:`SyntaxError`.
|
|
(Contributed by Serhiy Storchaka in :gh:`81548`.)
|
|
|
|
* The :mod:`lib2to3` package and ``2to3`` tool are now deprecated and may not
|
|
be able to parse Python 3.10 or newer. See the :pep:`617` (New PEG parser for
|
|
CPython). (Contributed by Victor Stinner in :issue:`40360`.)
|
|
|
|
* Undocumented modules ``sre_compile``, ``sre_constants`` and ``sre_parse``
|
|
are now deprecated.
|
|
(Contributed by Serhiy Storchaka in :issue:`47152`.)
|
|
|
|
* :class:`webbrowser.MacOSX` is deprecated and will be removed in Python 3.13.
|
|
It is untested and undocumented and also not used by webbrowser itself.
|
|
(Contributed by Dong-hee Na in :issue:`42255`.)
|
|
|
|
* The behavior of returning a value from a :class:`~unittest.TestCase` and
|
|
:class:`~unittest.IsolatedAsyncioTestCase` test methods (other than the
|
|
default ``None`` value), is now deprecated.
|
|
|
|
* Deprecated the following :mod:`unittest` functions, scheduled for removal in
|
|
Python 3.13:
|
|
|
|
* :func:`unittest.findTestCases`
|
|
* :func:`unittest.makeSuite`
|
|
* :func:`unittest.getTestCaseNames`
|
|
|
|
Use :class:`~unittest.TestLoader` method instead:
|
|
|
|
* :meth:`unittest.TestLoader.loadTestsFromModule`
|
|
* :meth:`unittest.TestLoader.loadTestsFromTestCase`
|
|
* :meth:`unittest.TestLoader.getTestCaseNames`
|
|
|
|
(Contributed by Erlend E. Aasland in :issue:`5846`.)
|
|
|
|
* The :meth:`turtle.RawTurtle.settiltangle` is deprecated since Python 3.1,
|
|
it now emits a deprecation warning and will be removed in Python 3.13. Use
|
|
:meth:`turtle.RawTurtle.tiltangle` instead (it was earlier incorrectly marked
|
|
as deprecated, its docstring is now corrected).
|
|
(Contributed by Hugo van Kemenade in :issue:`45837`.)
|
|
|
|
* The delegation of :func:`int` to :meth:`__trunc__` is now deprecated. Calling
|
|
``int(a)`` when ``type(a)`` implements :meth:`__trunc__` but not
|
|
:meth:`__int__` or :meth:`__index__` now raises a :exc:`DeprecationWarning`.
|
|
(Contributed by Zackery Spytz in :issue:`44977`.)
|
|
|
|
* The following have been deprecated in :mod:`configparser` since Python 3.2.
|
|
Their deprecation warnings have now been updated to note they will removed in
|
|
Python 3.12:
|
|
|
|
* the :class:`configparser.SafeConfigParser` class
|
|
* the :attr:`configparser.ParsingError.filename` property
|
|
* the :meth:`configparser.ParsingError.readfp` method
|
|
|
|
(Contributed by Hugo van Kemenade in :issue:`45173`.)
|
|
|
|
* :class:`configparser.LegacyInterpolation` has been deprecated in the docstring
|
|
since Python 3.2. It now emits a :exc:`DeprecationWarning` and will be removed
|
|
in Python 3.13. Use :class:`configparser.BasicInterpolation` or
|
|
:class:`configparser.ExtendedInterpolation` instead.
|
|
(Contributed by Hugo van Kemenade in :issue:`46607`.)
|
|
|
|
* The :func:`locale.getdefaultlocale` function is deprecated and will be
|
|
removed in Python 3.13. Use :func:`locale.setlocale`,
|
|
:func:`locale.getpreferredencoding(False) <locale.getpreferredencoding>` and
|
|
:func:`locale.getlocale` functions instead.
|
|
(Contributed by Victor Stinner in :issue:`46659`.)
|
|
|
|
* The :mod:`asynchat`, :mod:`asyncore` and :mod:`smtpd` modules have been
|
|
deprecated since at least Python 3.6. Their documentation and deprecation
|
|
warnings have now been updated to note they will removed in Python 3.12
|
|
(:pep:`594`).
|
|
(Contributed by Hugo van Kemenade in :issue:`47022`.)
|
|
|
|
* :pep:`594` led to the deprecations of the following modules which are
|
|
slated for removal in Python 3.13:
|
|
|
|
* :mod:`aifc`
|
|
* :mod:`audioop`
|
|
* :mod:`cgi`
|
|
* :mod:`cgitb`
|
|
* :mod:`chunk`
|
|
* :mod:`crypt`
|
|
* :mod:`imghdr`
|
|
* :mod:`mailcap`
|
|
* :mod:`msilib`
|
|
* :mod:`nis`
|
|
* :mod:`nntplib`
|
|
* :mod:`ossaudiodev`
|
|
* :mod:`pipes`
|
|
* :mod:`sndhdr`
|
|
* :mod:`spwd`
|
|
* :mod:`sunau`
|
|
* :mod:`telnetlib`
|
|
* :mod:`uu`
|
|
* :mod:`xdrlib`
|
|
|
|
(Contributed by Brett Cannon in :issue:`47061` and Victor Stinner in
|
|
:gh:`68966`.)
|
|
|
|
* More strict rules will be applied now applied for numerical group references
|
|
and group names in regular expressions in future Python versions.
|
|
Only sequence of ASCII digits will be now accepted as a numerical reference.
|
|
The group name in bytes patterns and replacement strings could only
|
|
contain ASCII letters and digits and underscore.
|
|
For now, a deprecation warning is raised for such syntax.
|
|
(Contributed by Serhiy Storchaka in :gh:`91760`.)
|
|
|
|
* :class:`typing.Text`, which exists solely to provide compatibility support
|
|
between Python 2 and Python 3 code, is now deprecated. Its removal is
|
|
currently unplanned, but users are encouraged to use :class:`str` instead
|
|
wherever possible.
|
|
(Contributed by Alex Waygood in :gh:`92332`.)
|
|
|
|
|
|
Pending Removal in Python 3.12
|
|
==============================
|
|
|
|
The following APIs have been deprecated in earlier Python releases,
|
|
and will be removed in Python 3.12.
|
|
|
|
Python API:
|
|
|
|
* :class:`pkgutil.ImpImporter`
|
|
* :class:`pkgutil.ImpLoader`
|
|
* :envvar:`PYTHONTHREADDEBUG`
|
|
* :func:`importlib.find_loader`
|
|
* :func:`importlib.util.module_for_loader`
|
|
* :func:`importlib.util.set_loader_wrapper`
|
|
* :func:`importlib.util.set_package_wrapper`
|
|
* :meth:`importlib.abc.Loader.module_repr`
|
|
* :meth:`importlib.abc.Loadermodule_repr`
|
|
* :meth:`importlib.abc.MetaPathFinder.find_module`
|
|
* :meth:`importlib.abc.MetaPathFinder.find_module`
|
|
* :meth:`importlib.abc.PathEntryFinder.find_loader`
|
|
* :meth:`importlib.abc.PathEntryFinder.find_module`
|
|
* :meth:`importlib.machinery.BuiltinImporter.find_module`
|
|
* :meth:`importlib.machinery.BuiltinLoader.module_repr`
|
|
* :meth:`importlib.machinery.FileFinder.find_loader`
|
|
* :meth:`importlib.machinery.FileFinder.find_module`
|
|
* :meth:`importlib.machinery.FrozenImporter.find_module`
|
|
* :meth:`importlib.machinery.FrozenLoader.module_repr`
|
|
* :meth:`importlib.machinery.PathFinder.find_module`
|
|
* :meth:`importlib.machinery.WindowsRegistryFinder.find_module`
|
|
* :meth:`pathlib.Path.link_to`
|
|
* The entire :ref:`distutils namespace <distutils-deprecated>`
|
|
* :func:`cgi.log`
|
|
* :func:`sqlite3.OptimizedUnicode`
|
|
* :func:`sqlite3.enable_shared_cache`
|
|
|
|
C API:
|
|
|
|
* :c:func:`PyUnicode_AS_DATA`
|
|
* :c:func:`PyUnicode_AS_UNICODE`
|
|
* :c:func:`PyUnicode_AsUnicodeAndSize`
|
|
* :c:func:`PyUnicode_AsUnicode`
|
|
* :c:func:`PyUnicode_FromUnicode`
|
|
* :c:func:`PyUnicode_GET_DATA_SIZE`
|
|
* :c:func:`PyUnicode_GET_SIZE`
|
|
* :c:func:`PyUnicode_GetSize`
|
|
* :c:func:`PyUnicode_IS_COMPACT`
|
|
* :c:func:`PyUnicode_IS_READY`
|
|
* :c:func:`PyUnicode_READY`
|
|
* :c:func:`Py_UNICODE_WSTR_LENGTH`
|
|
* :c:func:`_PyUnicode_AsUnicode`
|
|
* :c:macro:`PyUnicode_WCHAR_KIND`
|
|
* :c:type:`PyUnicodeObject`
|
|
* :c:func:`PyUnicode_InternImmortal()`
|
|
|
|
|
|
Removed
|
|
=======
|
|
|
|
* :class:`smtpd.MailmanProxy` is now removed as it is unusable without
|
|
an external module, ``mailman``. (Contributed by Dong-hee Na in :issue:`35800`.)
|
|
|
|
* The ``binhex`` module, deprecated in Python 3.9, is now removed.
|
|
The following :mod:`binascii` functions, deprecated in Python 3.9, are now
|
|
also removed:
|
|
|
|
* ``a2b_hqx()``, ``b2a_hqx()``;
|
|
* ``rlecode_hqx()``, ``rledecode_hqx()``.
|
|
|
|
The :func:`binascii.crc_hqx` function remains available.
|
|
|
|
(Contributed by Victor Stinner in :issue:`45085`.)
|
|
|
|
* The distutils ``bdist_msi`` command, deprecated in Python 3.9, is now removed.
|
|
Use ``bdist_wheel`` (wheel packages) instead.
|
|
(Contributed by Hugo van Kemenade in :issue:`45124`.)
|
|
|
|
* Due to significant security concerns, the *reuse_address* parameter of
|
|
:meth:`asyncio.loop.create_datagram_endpoint`, disabled in Python 3.9, is
|
|
now entirely removed. This is because of the behavior of the socket option
|
|
``SO_REUSEADDR`` in UDP.
|
|
(Contributed by Hugo van Kemenade in :issue:`45129`.)
|
|
|
|
* Removed :meth:`__getitem__` methods of
|
|
:class:`xml.dom.pulldom.DOMEventStream`, :class:`wsgiref.util.FileWrapper`
|
|
and :class:`fileinput.FileInput`, deprecated since Python 3.9.
|
|
(Contributed by Hugo van Kemenade in :issue:`45132`.)
|
|
|
|
* The following deprecated functions and methods are removed in the :mod:`gettext`
|
|
module: :func:`~gettext.lgettext`, :func:`~gettext.ldgettext`,
|
|
:func:`~gettext.lngettext` and :func:`~gettext.ldngettext`.
|
|
|
|
Function :func:`~gettext.bind_textdomain_codeset`, methods
|
|
:meth:`~gettext.NullTranslations.output_charset` and
|
|
:meth:`~gettext.NullTranslations.set_output_charset`, and the *codeset*
|
|
parameter of functions :func:`~gettext.translation` and
|
|
:func:`~gettext.install` are also removed, since they are only used for
|
|
the ``l*gettext()`` functions.
|
|
(Contributed by Dong-hee Na and Serhiy Storchaka in :issue:`44235`.)
|
|
|
|
* The :func:`@asyncio.coroutine <asyncio.coroutine>` :term:`decorator` enabling
|
|
legacy generator-based coroutines to be compatible with async/await code.
|
|
The function has been deprecated since Python 3.8 and the removal was
|
|
initially scheduled for Python 3.10. Use :keyword:`async def` instead.
|
|
(Contributed by Illia Volochii in :issue:`43216`.)
|
|
|
|
* :class:`asyncio.coroutines.CoroWrapper` used for wrapping legacy
|
|
generator-based coroutine objects in the debug mode.
|
|
(Contributed by Illia Volochii in :issue:`43216`.)
|
|
|
|
* Removed the deprecated ``split()`` method of :class:`_tkinter.TkappType`.
|
|
(Contributed by Erlend E. Aasland in :issue:`38371`.)
|
|
|
|
* Removed from the :mod:`inspect` module:
|
|
|
|
* the ``getargspec`` function, deprecated since Python 3.0;
|
|
use :func:`inspect.signature` or :func:`inspect.getfullargspec` instead.
|
|
|
|
* the ``formatargspec`` function, deprecated since Python 3.5;
|
|
use the :func:`inspect.signature` function and :class:`Signature` object
|
|
directly.
|
|
|
|
* the undocumented ``Signature.from_builtin`` and ``Signature.from_function``
|
|
functions, deprecated since Python 3.5; use the
|
|
:meth:`Signature.from_callable() <inspect.Signature.from_callable>` method
|
|
instead.
|
|
|
|
(Contributed by Hugo van Kemenade in :issue:`45320`.)
|
|
|
|
* Remove namespace package support from unittest discovery. It was introduced in
|
|
Python 3.4 but has been broken since Python 3.7.
|
|
(Contributed by Inada Naoki in :issue:`23882`.)
|
|
|
|
* Remove ``__class_getitem__`` method from :class:`pathlib.PurePath`,
|
|
because it was not used and added by mistake in previous versions.
|
|
(Contributed by Nikita Sobolev in :issue:`46483`.)
|
|
|
|
* Remove the undocumented private ``float.__set_format__()`` method, previously
|
|
known as ``float.__setformat__()`` in Python 3.7. Its docstring said: "You
|
|
probably don't want to use this function. It exists mainly to be used in
|
|
Python's test suite."
|
|
(Contributed by Victor Stinner in :issue:`46852`.)
|
|
|
|
Porting to Python 3.11
|
|
======================
|
|
|
|
This section lists previously described changes and other bugfixes
|
|
that may require changes to your code.
|
|
|
|
|
|
Changes in the Python API
|
|
-------------------------
|
|
|
|
* Prohibited passing non-:class:`concurrent.futures.ThreadPoolExecutor`
|
|
executors to :meth:`loop.set_default_executor` following a deprecation in
|
|
Python 3.8.
|
|
(Contributed by Illia Volochii in :issue:`43234`.)
|
|
|
|
* :func:`open`, :func:`io.open`, :func:`codecs.open` and
|
|
:class:`fileinput.FileInput` no longer accept ``'U'`` ("universal newline")
|
|
in the file mode. This flag was deprecated since Python 3.3. In Python 3, the
|
|
"universal newline" is used by default when a file is open in text mode. The
|
|
:ref:`newline parameter <open-newline-parameter>` of :func:`open` controls
|
|
how universal newlines works.
|
|
(Contributed by Victor Stinner in :issue:`37330`.)
|
|
|
|
* The :mod:`pdb` module now reads the :file:`.pdbrc` configuration file with
|
|
the ``'utf-8'`` encoding.
|
|
(Contributed by Srinivas Reddy Thatiparthy (శ్రీనివాస్ రెడ్డి తాటిపర్తి) in :issue:`41137`.)
|
|
|
|
* When sorting using tuples as keys, the order of the result may differ
|
|
from earlier releases if the tuple elements don't define a total
|
|
ordering (see :ref:`expressions-value-comparisons` for
|
|
information on total ordering). It's generally true that the result
|
|
of sorting simply isn't well-defined in the absence of a total ordering
|
|
on list elements.
|
|
|
|
* :mod:`calendar`: The :class:`calendar.LocaleTextCalendar` and
|
|
:class:`calendar.LocaleHTMLCalendar` classes now use
|
|
:func:`locale.getlocale`, instead of using :func:`locale.getdefaultlocale`,
|
|
if no locale is specified.
|
|
(Contributed by Victor Stinner in :issue:`46659`.)
|
|
|
|
* Global inline flags (e.g. ``(?i)``) can now only be used at the start of
|
|
the regular expressions. Using them not at the start of expression was
|
|
deprecated since Python 3.6.
|
|
(Contributed by Serhiy Storchaka in :issue:`47066`.)
|
|
|
|
* :mod:`re` module: Fix a few long-standing bugs where, in rare cases,
|
|
capturing group could get wrong result. So the result may be different than
|
|
before.
|
|
(Contributed by Ma Lin in :issue:`35859`.)
|
|
|
|
* The *population* parameter of :func:`random.sample` must be a sequence.
|
|
Automatic conversion of sets to lists is no longer supported. If the sample size
|
|
is larger than the population size, a :exc:`ValueError` is raised.
|
|
(Contributed by Raymond Hettinger in :issue:`40465`.)
|
|
|
|
|
|
Build Changes
|
|
=============
|
|
|
|
* Building Python now requires a C11 compiler without optional C11 features.
|
|
(Contributed by Victor Stinner in :issue:`46656`.)
|
|
|
|
* Building Python now requires support of IEEE 754 floating point numbers.
|
|
(Contributed by Victor Stinner in :issue:`46917`.)
|
|
|
|
* CPython can now be built with the ThinLTO option via ``--with-lto=thin``.
|
|
(Contributed by Dong-hee Na and Brett Holman in :issue:`44340`.)
|
|
|
|
* libpython is no longer linked against libcrypt.
|
|
(Contributed by Mike Gilbert in :issue:`45433`.)
|
|
|
|
* Building Python now requires a C99 ``<math.h>`` header file providing
|
|
the following functions: ``copysign()``, ``hypot()``, ``isfinite()``,
|
|
``isinf()``, ``isnan()``, ``round()``.
|
|
(Contributed by Victor Stinner in :issue:`45440`.)
|
|
|
|
* Building Python now requires a C99 ``<math.h>`` header file providing
|
|
a ``NAN`` constant, or the ``__builtin_nan()`` built-in function.
|
|
(Contributed by Victor Stinner in :issue:`46640`.)
|
|
|
|
* Building Python now requires support for floating point Not-a-Number (NaN):
|
|
remove the ``Py_NO_NAN`` macro.
|
|
(Contributed by Victor Stinner in :issue:`46656`.)
|
|
|
|
* Freelists for object structs can now be disabled. A new :program:`configure`
|
|
option :option:`!--without-freelists` can be used to disable all freelists
|
|
except empty tuple singleton.
|
|
(Contributed by Christian Heimes in :issue:`45522`)
|
|
|
|
* ``Modules/Setup`` and ``Modules/makesetup`` have been improved and tied up.
|
|
Extension modules can now be built through ``makesetup``. All except some
|
|
test modules can be linked statically into main binary or library.
|
|
(Contributed by Brett Cannon and Christian Heimes in :issue:`45548`,
|
|
:issue:`45570`, :issue:`45571`, and :issue:`43974`.)
|
|
|
|
* Build dependencies, compiler flags, and linker flags for most stdlib
|
|
extension modules are now detected by :program:`configure`. libffi, libnsl,
|
|
libsqlite3, zlib, bzip2, liblzma, libcrypt, Tcl/Tk libs, and uuid flags
|
|
are detected by ``pkg-config`` (when available).
|
|
(Contributed by Christian Heimes and Erlend Egeberg Aasland in
|
|
:issue:`45847`, :issue:`45747`, and :issue:`45763`.)
|
|
|
|
.. note::
|
|
Use the environment variables ``TCLTK_CFLAGS`` and ``TCLTK_LIBS`` to
|
|
manually specify the location of Tcl/Tk headers and libraries.
|
|
The :program:`configure` options ``--with-tcltk-includes`` and
|
|
``--with-tcltk-libs`` have been removed.
|
|
|
|
* CPython now has experimental support for cross compiling to WebAssembly
|
|
platform ``wasm32-emscripten``. The effort is inspired by previous work
|
|
like Pyodide.
|
|
(Contributed by Christian Heimes and Ethan Smith in :issue:`40280`.)
|
|
|
|
* CPython will now use 30-bit digits by default for the Python :class:`int`
|
|
implementation. Previously, the default was to use 30-bit digits on platforms
|
|
with ``SIZEOF_VOID_P >= 8``, and 15-bit digits otherwise. It's still possible
|
|
to explicitly request use of 15-bit digits via either the
|
|
``--enable-big-digits`` option to the configure script or (for Windows) the
|
|
``PYLONG_BITS_IN_DIGIT`` variable in ``PC/pyconfig.h``, but this option may
|
|
be removed at some point in the future. (Contributed by Mark Dickinson in
|
|
:issue:`45569`.)
|
|
|
|
* The :mod:`tkinter` package now requires Tcl/Tk version 8.5.12 or newer.
|
|
(Contributed by Serhiy Storchaka in :issue:`46996`.)
|
|
|
|
|
|
C API Changes
|
|
=============
|
|
|
|
New Features
|
|
------------
|
|
|
|
* Add a new :c:func:`PyType_GetName` function to get type's short name.
|
|
(Contributed by Hai Shi in :issue:`42035`.)
|
|
|
|
* Add a new :c:func:`PyType_GetQualName` function to get type's qualified name.
|
|
(Contributed by Hai Shi in :issue:`42035`.)
|
|
|
|
* Add new :c:func:`PyThreadState_EnterTracing` and
|
|
:c:func:`PyThreadState_LeaveTracing` functions to the limited C API to
|
|
suspend and resume tracing and profiling.
|
|
(Contributed by Victor Stinner in :issue:`43760`.)
|
|
|
|
* Added the :c:data:`Py_Version` constant which bears the same value as
|
|
:c:macro:`PY_VERSION_HEX`.
|
|
(Contributed by Gabriele N. Tornetta in :issue:`43931`.)
|
|
|
|
* :c:type:`Py_buffer` and APIs are now part of the limited API and the stable
|
|
ABI:
|
|
|
|
* :c:func:`PyObject_CheckBuffer`
|
|
* :c:func:`PyObject_GetBuffer`
|
|
* :c:func:`PyBuffer_GetPointer`
|
|
* :c:func:`PyBuffer_SizeFromFormat`
|
|
* :c:func:`PyBuffer_ToContiguous`
|
|
* :c:func:`PyBuffer_FromContiguous`
|
|
* :c:func:`PyBuffer_CopyData`
|
|
* :c:func:`PyBuffer_IsContiguous`
|
|
* :c:func:`PyBuffer_FillContiguousStrides`
|
|
* :c:func:`PyBuffer_FillInfo`
|
|
* :c:func:`PyBuffer_Release`
|
|
* :c:func:`PyMemoryView_FromBuffer`
|
|
* :c:member:`~PyBufferProcs.bf_getbuffer` and
|
|
:c:member:`~PyBufferProcs.bf_releasebuffer` type slots
|
|
|
|
(Contributed by Christian Heimes in :issue:`45459`.)
|
|
|
|
* Added the :c:data:`PyType_GetModuleByDef` function, used to get the module
|
|
in which a method was defined, in cases where this information is not
|
|
available directly (via :c:type:`PyCMethod`).
|
|
(Contributed by Petr Viktorin in :issue:`46613`.)
|
|
|
|
* Add new functions to pack and unpack C double (serialize and deserialize):
|
|
:c:func:`PyFloat_Pack2`, :c:func:`PyFloat_Pack4`, :c:func:`PyFloat_Pack8`,
|
|
:c:func:`PyFloat_Unpack2`, :c:func:`PyFloat_Unpack4` and
|
|
:c:func:`PyFloat_Unpack8`.
|
|
(Contributed by Victor Stinner in :issue:`46906`.)
|
|
|
|
* Add new functions to get frame object attributes:
|
|
:c:func:`PyFrame_GetBuiltins`, :c:func:`PyFrame_GetGenerator`,
|
|
:c:func:`PyFrame_GetGlobals`, :c:func:`PyFrame_GetLasti`.
|
|
|
|
* Added two new functions to get and set the active exception instance:
|
|
:c:func:`PyErr_GetHandledException` and :c:func:`PyErr_SetHandledException`.
|
|
These are alternatives to :c:func:`PyErr_SetExcInfo()` and
|
|
:c:func:`PyErr_GetExcInfo()` which work with the legacy 3-tuple
|
|
representation of exceptions.
|
|
(Contributed by Irit Katriel in :issue:`46343`.)
|
|
|
|
* Added the :c:member:`PyConfig.safe_path` member.
|
|
(Contributed by Victor Stinner in :gh:`57684`.)
|
|
|
|
Porting to Python 3.11
|
|
----------------------
|
|
|
|
* :c:func:`PyErr_SetExcInfo()` no longer uses the ``type`` and ``traceback``
|
|
arguments, the interpreter now derives those values from the exception
|
|
instance (the ``value`` argument). The function still steals references
|
|
of all three arguments.
|
|
(Contributed by Irit Katriel in :issue:`45711`.)
|
|
|
|
* :c:func:`PyErr_GetExcInfo()` now derives the ``type`` and ``traceback``
|
|
fields of the result from the exception instance (the ``value`` field).
|
|
(Contributed by Irit Katriel in :issue:`45711`.)
|
|
|
|
* :c:type:`_frozen` has a new ``is_package`` field to indicate whether
|
|
or not the frozen module is a package. Previously, a negative value
|
|
in the ``size`` field was the indicator. Now only non-negative values
|
|
be used for ``size``.
|
|
(Contributed by Kumar Aditya in :issue:`46608`.)
|
|
|
|
* :c:func:`_PyFrameEvalFunction` now takes ``_PyInterpreterFrame*``
|
|
as its second parameter, instead of ``PyFrameObject*``.
|
|
See :pep:`523` for more details of how to use this function pointer type.
|
|
|
|
* :c:func:`PyCode_New` and :c:func:`PyCode_NewWithPosOnlyArgs` now take
|
|
an additional ``exception_table`` argument.
|
|
Using these functions should be avoided, if at all possible.
|
|
To get a custom code object: create a code object using the compiler,
|
|
then get a modified version with the ``replace`` method.
|
|
|
|
* :c:type:`PyCodeObject` no longer has a ``co_code`` field. Instead,
|
|
use ``PyObject_GetAttrString(code_object, "co_code")`` or
|
|
:c:func:`PyCode_GetCode` to get the underlying bytes object.
|
|
(Contributed by Brandt Bucher in :issue:`46841` and Ken Jin in :gh:`92154`.)
|
|
|
|
* The old trashcan macros (``Py_TRASHCAN_SAFE_BEGIN``/``Py_TRASHCAN_SAFE_END``)
|
|
are now deprecated. They should be replaced by the new macros
|
|
``Py_TRASHCAN_BEGIN`` and ``Py_TRASHCAN_END``.
|
|
|
|
A tp_dealloc function that has the old macros, such as::
|
|
|
|
static void
|
|
mytype_dealloc(mytype *p)
|
|
{
|
|
PyObject_GC_UnTrack(p);
|
|
Py_TRASHCAN_SAFE_BEGIN(p);
|
|
...
|
|
Py_TRASHCAN_SAFE_END
|
|
}
|
|
|
|
should migrate to the new macros as follows::
|
|
|
|
static void
|
|
mytype_dealloc(mytype *p)
|
|
{
|
|
PyObject_GC_UnTrack(p);
|
|
Py_TRASHCAN_BEGIN(p, mytype_dealloc)
|
|
...
|
|
Py_TRASHCAN_END
|
|
}
|
|
|
|
Note that ``Py_TRASHCAN_BEGIN`` has a second argument which
|
|
should be the deallocation function it is in.
|
|
|
|
To support older Python versions in the same codebase, you
|
|
can define the following macros and use them throughout
|
|
the code (credit: these were copied from the ``mypy`` codebase)::
|
|
|
|
#if PY_MAJOR_VERSION >= 3 && PY_MINOR_VERSION >= 8
|
|
# define CPy_TRASHCAN_BEGIN(op, dealloc) Py_TRASHCAN_BEGIN(op, dealloc)
|
|
# define CPy_TRASHCAN_END(op) Py_TRASHCAN_END
|
|
#else
|
|
# define CPy_TRASHCAN_BEGIN(op, dealloc) Py_TRASHCAN_SAFE_BEGIN(op)
|
|
# define CPy_TRASHCAN_END(op) Py_TRASHCAN_SAFE_END(op)
|
|
#endif
|
|
|
|
* The :c:func:`PyType_Ready` function now raises an error if a type is defined
|
|
with the :const:`Py_TPFLAGS_HAVE_GC` flag set but has no traverse function
|
|
(:c:member:`PyTypeObject.tp_traverse`).
|
|
(Contributed by Victor Stinner in :issue:`44263`.)
|
|
|
|
* Heap types with the :const:`Py_TPFLAGS_IMMUTABLETYPE` flag can now inherit
|
|
the :pep:`590` vectorcall protocol. Previously, this was only possible for
|
|
:ref:`static types <static-types>`.
|
|
(Contributed by Erlend E. Aasland in :issue:`43908`)
|
|
|
|
* Since :c:func:`Py_TYPE()` is changed to a inline static function,
|
|
``Py_TYPE(obj) = new_type`` must be replaced with
|
|
``Py_SET_TYPE(obj, new_type)``: see the :c:func:`Py_SET_TYPE()` function
|
|
(available since Python 3.9). For backward compatibility, this macro can be
|
|
used::
|
|
|
|
#if PY_VERSION_HEX < 0x030900A4 && !defined(Py_SET_TYPE)
|
|
static inline void _Py_SET_TYPE(PyObject *ob, PyTypeObject *type)
|
|
{ ob->ob_type = type; }
|
|
#define Py_SET_TYPE(ob, type) _Py_SET_TYPE((PyObject*)(ob), type)
|
|
#endif
|
|
|
|
(Contributed by Victor Stinner in :issue:`39573`.)
|
|
|
|
* Since :c:func:`Py_SIZE()` is changed to a inline static function,
|
|
``Py_SIZE(obj) = new_size`` must be replaced with
|
|
``Py_SET_SIZE(obj, new_size)``: see the :c:func:`Py_SET_SIZE()` function
|
|
(available since Python 3.9). For backward compatibility, this macro can be
|
|
used::
|
|
|
|
#if PY_VERSION_HEX < 0x030900A4 && !defined(Py_SET_SIZE)
|
|
static inline void _Py_SET_SIZE(PyVarObject *ob, Py_ssize_t size)
|
|
{ ob->ob_size = size; }
|
|
#define Py_SET_SIZE(ob, size) _Py_SET_SIZE((PyVarObject*)(ob), size)
|
|
#endif
|
|
|
|
(Contributed by Victor Stinner in :issue:`39573`.)
|
|
|
|
* ``<Python.h>`` no longer includes the header files ``<stdlib.h>``,
|
|
``<stdio.h>``, ``<errno.h>`` and ``<string.h>`` when the ``Py_LIMITED_API``
|
|
macro is set to ``0x030b0000`` (Python 3.11) or higher. C extensions should
|
|
explicitly include the header files after ``#include <Python.h>``.
|
|
(Contributed by Victor Stinner in :issue:`45434`.)
|
|
|
|
* The non-limited API files ``cellobject.h``, ``classobject.h``, ``code.h``, ``context.h``,
|
|
``funcobject.h``, ``genobject.h`` and ``longintrepr.h`` have been moved to
|
|
the ``Include/cpython`` directory. Moreover, the ``eval.h`` header file was
|
|
removed. These files must not be included directly, as they are already
|
|
included in ``Python.h``: :ref:`Include Files <api-includes>`. If they have
|
|
been included directly, consider including ``Python.h`` instead.
|
|
(Contributed by Victor Stinner in :issue:`35134`.)
|
|
|
|
* The :c:func:`PyUnicode_CHECK_INTERNED` macro has been excluded from the
|
|
limited C API. It was never usable there, because it used internal structures
|
|
which are not available in the limited C API.
|
|
(Contributed by Victor Stinner in :issue:`46007`.)
|
|
|
|
.. _pyframeobject-3.11-hiding:
|
|
|
|
* The :c:type:`PyFrameObject` structure members have been removed from the
|
|
public C API.
|
|
|
|
While the documentation notes that the :c:type:`PyFrameObject` fields are
|
|
subject to change at any time, they have been stable for a long time and were
|
|
used in several popular extensions.
|
|
|
|
In Python 3.11, the frame struct was reorganized to allow performance
|
|
optimizations. Some fields were removed entirely, as they were details of the
|
|
old implementation.
|
|
|
|
:c:type:`PyFrameObject` fields:
|
|
|
|
* ``f_back``: use :c:func:`PyFrame_GetBack`.
|
|
* ``f_blockstack``: removed.
|
|
* ``f_builtins``: use :c:func:`PyFrame_GetBuiltins`.
|
|
* ``f_code``: use :c:func:`PyFrame_GetCode`.
|
|
* ``f_gen``: use :c:func:`PyFrame_GetGenerator`.
|
|
* ``f_globals``: use :c:func:`PyFrame_GetGlobals`.
|
|
* ``f_iblock``: removed.
|
|
* ``f_lasti``: use :c:func:`PyFrame_GetLasti`.
|
|
Code using ``f_lasti`` with ``PyCode_Addr2Line()`` should use
|
|
:c:func:`PyFrame_GetLineNumber` instead; it may be faster.
|
|
* ``f_lineno``: use :c:func:`PyFrame_GetLineNumber`
|
|
* ``f_locals``: use :c:func:`PyFrame_GetLocals`.
|
|
* ``f_stackdepth``: removed.
|
|
* ``f_state``: no public API (renamed to ``f_frame.f_state``).
|
|
* ``f_trace``: no public API.
|
|
* ``f_trace_lines``: use ``PyObject_GetAttrString((PyObject*)frame, "f_trace_lines")``.
|
|
* ``f_trace_opcodes``: use ``PyObject_GetAttrString((PyObject*)frame, "f_trace_opcodes")``.
|
|
* ``f_localsplus``: no public API (renamed to ``f_frame.localsplus``).
|
|
* ``f_valuestack``: removed.
|
|
|
|
The Python frame object is now created lazily. A side effect is that the
|
|
``f_back`` member must not be accessed directly, since its value is now also
|
|
computed lazily. The :c:func:`PyFrame_GetBack` function must be called
|
|
instead.
|
|
|
|
Debuggers that accessed the ``f_locals`` directly *must* call
|
|
:c:func:`PyFrame_GetLocals` instead. They no longer need to call
|
|
:c:func:`PyFrame_FastToLocalsWithError` or :c:func:`PyFrame_LocalsToFast`,
|
|
in fact they should not call those functions. The necessary updating of the
|
|
frame is now managed by the virtual machine.
|
|
|
|
Code defining ``PyFrame_GetCode()`` on Python 3.8 and older::
|
|
|
|
#if PY_VERSION_HEX < 0x030900B1
|
|
static inline PyCodeObject* PyFrame_GetCode(PyFrameObject *frame)
|
|
{
|
|
Py_INCREF(frame->f_code);
|
|
return frame->f_code;
|
|
}
|
|
#endif
|
|
|
|
Code defining ``PyFrame_GetBack()`` on Python 3.8 and older::
|
|
|
|
#if PY_VERSION_HEX < 0x030900B1
|
|
static inline PyFrameObject* PyFrame_GetBack(PyFrameObject *frame)
|
|
{
|
|
Py_XINCREF(frame->f_back);
|
|
return frame->f_back;
|
|
}
|
|
#endif
|
|
|
|
Or use the `pythoncapi_compat project
|
|
<https://github.com/python/pythoncapi_compat>`__ to get these two
|
|
functions on older Python versions.
|
|
|
|
* Changes of the :c:type:`PyThreadState` structure members:
|
|
|
|
* ``frame``: removed, use :c:func:`PyThreadState_GetFrame` (function added
|
|
to Python 3.9 by :issue:`40429`).
|
|
Warning: the function returns a :term:`strong reference`, need to call
|
|
:c:func:`Py_XDECREF`.
|
|
* ``tracing``: changed, use :c:func:`PyThreadState_EnterTracing`
|
|
and :c:func:`PyThreadState_LeaveTracing`
|
|
(functions added to Python 3.11 by :issue:`43760`).
|
|
* ``recursion_depth``: removed,
|
|
use ``(tstate->recursion_limit - tstate->recursion_remaining)`` instead.
|
|
* ``stackcheck_counter``: removed.
|
|
|
|
Code defining ``PyThreadState_GetFrame()`` on Python 3.8 and older::
|
|
|
|
#if PY_VERSION_HEX < 0x030900B1
|
|
static inline PyFrameObject* PyThreadState_GetFrame(PyThreadState *tstate)
|
|
{
|
|
Py_XINCREF(tstate->frame);
|
|
return tstate->frame;
|
|
}
|
|
#endif
|
|
|
|
Code defining ``PyThreadState_EnterTracing()`` and
|
|
``PyThreadState_LeaveTracing()`` on Python 3.10 and older::
|
|
|
|
#if PY_VERSION_HEX < 0x030B00A2
|
|
static inline void PyThreadState_EnterTracing(PyThreadState *tstate)
|
|
{
|
|
tstate->tracing++;
|
|
#if PY_VERSION_HEX >= 0x030A00A1
|
|
tstate->cframe->use_tracing = 0;
|
|
#else
|
|
tstate->use_tracing = 0;
|
|
#endif
|
|
}
|
|
|
|
static inline void PyThreadState_LeaveTracing(PyThreadState *tstate)
|
|
{
|
|
int use_tracing = (tstate->c_tracefunc != NULL || tstate->c_profilefunc != NULL);
|
|
tstate->tracing--;
|
|
#if PY_VERSION_HEX >= 0x030A00A1
|
|
tstate->cframe->use_tracing = use_tracing;
|
|
#else
|
|
tstate->use_tracing = use_tracing;
|
|
#endif
|
|
}
|
|
#endif
|
|
|
|
Or use `the pythoncapi_compat project
|
|
<https://github.com/python/pythoncapi_compat>`__ to get these functions
|
|
on old Python functions.
|
|
|
|
* Distributors are encouraged to build Python with the optimized Blake2
|
|
library `libb2`_.
|
|
|
|
|
|
Deprecated
|
|
----------
|
|
|
|
* Deprecate the following functions to configure the Python initialization:
|
|
|
|
* :c:func:`PySys_AddWarnOptionUnicode`
|
|
* :c:func:`PySys_AddWarnOption`
|
|
* :c:func:`PySys_AddXOption`
|
|
* :c:func:`PySys_HasWarnOptions`
|
|
* :c:func:`PySys_SetArgvEx`
|
|
* :c:func:`PySys_SetArgv`
|
|
* :c:func:`PySys_SetPath`
|
|
* :c:func:`Py_SetPath`
|
|
* :c:func:`Py_SetProgramName`
|
|
* :c:func:`Py_SetPythonHome`
|
|
* :c:func:`Py_SetStandardStreamEncoding`
|
|
* :c:func:`_Py_SetProgramFullPath`
|
|
|
|
Use the new :c:type:`PyConfig` API of the :ref:`Python Initialization Configuration
|
|
<init-config>` instead (:pep:`587`).
|
|
(Contributed by Victor Stinner in :gh:`88279`.)
|
|
|
|
* Deprecate the ``ob_shash`` member of the :c:type:`PyBytesObject`. Use :c:func:`PyObject_Hash` instead.
|
|
(Contributed by Inada Naoki in :issue:`46864`.)
|
|
|
|
Removed
|
|
-------
|
|
|
|
* :c:func:`PyFrame_BlockSetup` and :c:func:`PyFrame_BlockPop` have been
|
|
removed.
|
|
(Contributed by Mark Shannon in :issue:`40222`.)
|
|
|
|
* Remove the following math macros using the ``errno`` variable:
|
|
|
|
* ``Py_ADJUST_ERANGE1()``
|
|
* ``Py_ADJUST_ERANGE2()``
|
|
* ``Py_OVERFLOWED()``
|
|
* ``Py_SET_ERANGE_IF_OVERFLOW()``
|
|
* ``Py_SET_ERRNO_ON_MATH_ERROR()``
|
|
|
|
(Contributed by Victor Stinner in :issue:`45412`.)
|
|
|
|
* Remove ``Py_UNICODE_COPY()`` and ``Py_UNICODE_FILL()`` macros, deprecated
|
|
since Python 3.3. Use ``PyUnicode_CopyCharacters()`` or ``memcpy()``
|
|
(``wchar_t*`` string), and ``PyUnicode_Fill()`` functions instead.
|
|
(Contributed by Victor Stinner in :issue:`41123`.)
|
|
|
|
* Remove the ``pystrhex.h`` header file. It only contains private functions.
|
|
C extensions should only include the main ``<Python.h>`` header file.
|
|
(Contributed by Victor Stinner in :issue:`45434`.)
|
|
|
|
* Remove the ``Py_FORCE_DOUBLE()`` macro. It was used by the
|
|
``Py_IS_INFINITY()`` macro.
|
|
(Contributed by Victor Stinner in :issue:`45440`.)
|
|
|
|
* The following items are no longer available when :c:macro:`Py_LIMITED_API`
|
|
is defined:
|
|
|
|
* :c:func:`PyMarshal_WriteLongToFile`
|
|
* :c:func:`PyMarshal_WriteObjectToFile`
|
|
* :c:func:`PyMarshal_ReadObjectFromString`
|
|
* :c:func:`PyMarshal_WriteObjectToString`
|
|
* the ``Py_MARSHAL_VERSION`` macro
|
|
|
|
These are not part of the :ref:`limited API <stable-abi-list>`.
|
|
|
|
(Contributed by Victor Stinner in :issue:`45474`.)
|
|
|
|
* Exclude :c:func:`PyWeakref_GET_OBJECT` from the limited C API. It never
|
|
worked since the :c:type:`PyWeakReference` structure is opaque in the
|
|
limited C API.
|
|
(Contributed by Victor Stinner in :issue:`35134`.)
|
|
|
|
* Remove the ``PyHeapType_GET_MEMBERS()`` macro. It was exposed in the
|
|
public C API by mistake, it must only be used by Python internally.
|
|
Use the ``PyTypeObject.tp_members`` member instead.
|
|
(Contributed by Victor Stinner in :issue:`40170`.)
|
|
|
|
* Remove the ``HAVE_PY_SET_53BIT_PRECISION`` macro (moved to the internal C
|
|
API).
|
|
(Contributed by Victor Stinner in :issue:`45412`.)
|
|
|
|
|
|
.. _libb2: https://www.blake2.net/
|