**************************** What's New In Python 3.2 **************************** :Author: Raymond Hettinger :Release: |release| :Date: |today| .. $Id$ Rules for maintenance: * Anyone can add text to this document. Do not spend very much time on the wording of your changes, because your text will probably get rewritten. * The maintainer will go through Misc/NEWS periodically and add changes; it's therefore more important to add your changes to Misc/NEWS than to this file. * This is not a complete list of every single change; completeness is the purpose of Misc/NEWS. Some changes I consider too small or esoteric to include. If such a change is added to the text, I'll just remove it. (This is another reason you shouldn't spend too much time on writing your addition.) * If you want to draw your new text to the attention of the maintainer, add 'XXX' to the beginning of the paragraph or section. * It's OK to just add a fragmentary note about a change. For example: "XXX Describe the transmogrify() function added to the socket module." The maintainer will research the change and write the necessary text. * You can comment out your additions if you like, but it's not necessary (especially when a final release is some months away). * Credit the author of a patch or bugfix. Just the name is sufficient; the e-mail address isn't necessary. It's helpful to add the issue number: XXX Describe the transmogrify() function added to the socket module. (Contributed by P.Y. Developer; :issue:`12345`.) This saves the maintainer the effort of going through the SVN log when researching a change. This article explains the new features in Python 3.2, compared to 3.1. PEP 384: Defining a Stable ABI ============================== In the past, extension modules built for one Python version were often not usable with other Python versions. Particularly on Windows, every feature release of Python required rebuilding all extension modules that one wanted to use. This requirement was the result of the free access to Python interpreter internals that extension modules could use. With Python 3.2, an alternative approach becomes available: extension modules which restrict themselves to a limited API (by defining Py_LIMITED_API) cannot use many of the internals, but are constrained to a set of API functions that are promised to be stable for several releases. As a consequence, extension modules built for 3.2 in that mode will also work with 3.3, 3.4, and so on. Extension modules that make use of details of memory structures can still be built, but will need to be recompiled for every feature release. .. seealso:: :pep:`384` - Define a Stable ABI PEP written by Martin von Loewis. PEP 389: Argparse Command Line Parsing Module ============================================= A new module for command line parsing, :mod:`argparse`, was introduced to overcome the limitations of :mod:`optparse` which did not provide support for positional arguments (not just option), subcommands, required options and other common patterns of specifying and validatig options. This module has already has wide-spread success in the community as a third-party module. Being more fully featured than its predecessor, :mod:`argparse`, is now the preferred module for command-line processing. The older module is still being kept available because of the substantial amount of legacy code that depends on it. .. XXX add examples that highlight the new features .. seealso:: :pep:`389` - New Command Line Parsing Module PEP written by Steven Bethard. PEP 391: Dictionary Based Configuration for Logging ==================================================== The :mod:`logging` module provided two kinds of configuration, one style with function calls for each option or another style driven by an external file saved in a :mod:`ConfigParser` format. Those options did not provide the flexibility to create configurations from JSON or YAML files, nor did they support incremental configuration, which is needed for specifying logger options from a command line. To support a more flexible style, the module now offers :func:`logging.config.dictConfig` for specifying logging configuration with plain Python dictionaries. The configuration options include formatters, handlers, filters, and loggers. Here's a working example of a configuration dictionary:: {"version": 1, "formatters": {"brief": {"format": "%(levelname)-8s: %(name)-15s: %(message)s"}, "full": {"format": "%(asctime)s %(name)-15s %(levelname)-8s %(message)s"}, }, "handlers": {"console": { "class": "logging.StreamHandler", "formatter": "brief", "level": "INFO", "stream": "ext://sys.stdout"}, "console_priority": { "class": "logging.StreamHandler", "formatter": "full", "level": "ERROR", "stream": "ext://sys.stderr"}, }, "root": {"level": "DEBUG", "handlers": ["console", "console_priority"]}} If that dictionary is stored in a file called :file:`conf.json`, it can loaded and called with code like this:: >>> import logging.config >>> logging.config.dictConfig(json.load(open('conf.json', 'rb'))) >>> logging.info("Transaction completed normally") >>> logging.critical("Abnormal termination") .. seealso:: :pep:`391` - Dictionary Based Configuration for Logging PEP written by Vinay Sajip. PEP 3148: The ``concurrent.futures`` module ============================================ Code for creating and managing concurrency is being collected in a new toplevel namespace, *concurrent*. Its first member is a *futures* package which provides a uniform high level interface for managing threads and processes. The design for :mod:`concurrent.futures` was inspired by *java.util.concurrent.package*. In that model, a running call and its result are represented by a :class:`~concurrent.futures.Future` object which abstracts features common to threads, processes, and remote procedure calls. That object supports status checks (running or done), timeouts, cancellations, adding callbacks, and access to results or exceptions.XS The primary offering of the new module is a pair of executor classes for launching and managing calls. The goal of the executors is to make it easier to use existing tools for making parallel calls. They save the effort needed to setup a pool of resources, launch the calls, create a results queue, add time-out handling, and limit the total number of threads, processes, or remote procedure calls. Ideally, each application should share a single executor across multiple components so that process and thread limits can be centrally managed. This solves the design challenge that arises when each component has its own competing strategy for resource management. For an example of :class:`~concurrent.futures.ThreadPoolExecutor`, see :ref:`code for threaded parallel URL reads`. For an example of :class:`~concurrent.futures.ProcessPoolExecutor`, see :ref:`code for computing prime numbers in parallel`. .. seealso:: :pep:`3148` - Futures -- Execute Computations Asynchronously PEP written by Brain Quinlan. PEP 3147: PYC Repository Directories ===================================== Python's scheme for caching bytecode in *.pyc* files did not wosrk well in environments with multiple python interpreters. If one interpreter encountered a cached file created by another interpreter, it would recompile the source and overwrite the cached file, thus losing the benefits of caching. The issue of "pyc fights" has become more pronounced as it has become commonplace for Linux distributions to ship with multiple versions of Python. These conflicts also arise with CPython alternatives such as Unladen Swallow. To solve this problem, Python's import machinery has been extended to use distinct filenames for each interpreter. Instead of Python 3.2 and Python 3.3 and Unladen Swallow each competing for a file called "mymodule.pyc", they will now look for "mymodule.cpython-32.pyc", "mymodule.cpython-33.pyc", and "mymodule.unladen10.pyc". And to prevent all of these new files from cluttering source directories, the *pyc* files are now collected in a "__pycache__" directory stored under the package directory. Aside from the filenames and target directories, the new scheme has a few aspects that are visible to the programmer: * Imported modules now have a :attr:`__cached__` attribute which stores the name of the actual file that was imported: >>> import collections >>> collections.__cached__ 'c:/py32/lib/__pycache__/collections.cpython-32.pyc' * The tag that is unique to each interpreter is accessible from the :mod:`imp` module: >>> import imp >>> imp.get_tag() 'cpython-32' * Scripts that try to deduce source filename from the imported file now need to be smarter. It is no longer sufficient to simply strip the "c" from a ".pyc" filename. Instead, use the new functions in the :mod:`imp` module: >>> imp.source_from_cache('c:/py32/lib/__pycache__/collections.cpython-32.pyc') 'c:/py32/lib/collections.py' >>> imp.cache_from_source('c:/py32/lib/collections.py') 'c:/py32/lib/__pycache__/collections.cpython-32.pyc' * The :mod:`py_compile` and :mod:`compileall` modules have been updated to reflect the new naming convention and target directory. .. seealso:: :pep:`3147` - PYC Repository Directories PEP written by Barry Warsaw. PEP 3149 ABI Version Tagged .so Files ===================================== The PYC repository directory allows multiple bytecode cache files to be co-located. This PEP implements a similar mechanism for shared object files by giving them a common directory and distinct names for each version. The common directory is "pyshared" and the file names are made distinct by identifying the Python implementation (such as CPython, PyPy, Jython, etc.), the major and minor version numbers, and optional build flags (such as "d" for debug, "m" for pymalloc, "u" for wide-unicode). For an arbitrary package "foo", you may see these files when the distribution package is installed:: /usr/share/pyshared/foo.cpython-32m.so /usr/share/pyshared/foo.cpython-33md.so In Python itself, the tags are accessible from functions in the :mod:`sysconfig` module:: >>> import sysconfig >>> sysconfig.get_config_var('SOABI') # find the version tag 'cpython-32mu' >>> sysconfig.get_config_var('SO') # find the full filename extension 'cpython-32mu.so' .. seealso:: :pep:`3149` - ABI Version Tagged .so Files PEP written by Barry Warsaw. Email 5.1 ========= The email package is extended to be able to parse and generate email messages in bytes format. * New functions :func:`~email.message_from_bytes` and :func:`~email.message_from_binary_file`, and new classes :class:`~email.parser.BytesFeedParser` and :class:`~email.parser.BytesParser` allow binary message data to be parsed into model objects. * Given bytes input to the model, :meth:`~email.message.Message.get_payload` will by default decode a message body that has a :mailheader:`Content-Transfer-Encoding` of ``8bit`` using the charset specified in the MIME headers and return the resulting string. * Given bytes input to the model, :class:`~email.generator.Generator` will convert message bodies that have a :mailheader:`Content-Transfer-Encoding` of 8bit to instead have a 7bit Content-Transfer-Encoding. * New class :class:`~email.generator.BytesGenerator` produces bytes as output, preserving any unchanged non-ASCII data that was present in the input used to build the model, including message bodies with a :mailheader:`Content-Transfer-Encoding` of 8bit. (Proposed and implemented by R. David Murray, :issue:`4661`.) Other Language Changes ====================== Some smaller changes made to the core Python language are: * The interpreter can now be started with a quiet option, ``-q``, to suppress the copyright and version information in an interactive mode. (Contributed by Marcin Wojdyr in issue:`1772833`). * The :func:`hasattr` function used to catch and suppress any Exception. Now, it only catches :exc:`AttributeError`. Under the hood, :func:`hasattr` works by calling :func:`getattr` and throwing away the results. This is necessary because dynamic attribute creation is possible using :meth:`__getattribute__` or :meth:`__getattr__`. If :func:`hasattr` were to just scan instance and class dictionaries it would miss the dynamic methods and make it difficult to implement proxy objects. (Discovered by Yury Selivanov and fixed by Benjamin Peterson; :issue:`9666`.) * The :func:`str` of a float or complex number is now the same as its :func:`repr`. Previously, the :func:`str` form was shorter but that just caused confusion and is no longer needed now that the shortest possible :func:`repr` is displayed by default: >>> repr(math.pi) '3.141592653589793' >>> str(math.pi) '3.141592653589793' (Proposed and implemented by Mark Dickinson; :issue:`9337`.) * :class:`memoryview` objects now have a :meth:`release()` method and support the context manager protocol. This allows timely release of any resources that were acquired when requesting a buffer from the original object. (Added by Antoine Pitrou; :issue:`9757`.) * Mark Dickinson crafted an elegant and efficient scheme for assuring that different numeric datatypes will have the same hash value whenever their actual values are equal:: >>> assert hash(Fraction(3, 2)) == hash(1.5) == \ hash(Decimal("1.5")) == hash(complex(1.5, 0)) (See :issue:`8188`.) * Previously it was illegal to delete a name from the local namespace if it occurs as a free variable in a nested block:: >>> def outer(x): ... def inner(): ... return x ... inner() ... del x This is now allowed. Remember that the target of an :keyword:`except` clause is cleared, so this code which used to work with Python 2.6, raised a :exc:`SyntaxError` with Python 3.1 and now works again:: >>> def f(): ... def print_error(): ... print(e) ... try: ... something ... except Exception as e: ... print_error() ... # implicit "del e" here (See :issue:`4617`.) * A new warning category, :exc:`ResourceWarning`, has been added. It is emitted when potential issues with resource consumption or cleanup are detected. It is silenced by default in normal release builds, but can be enabled through the means provided by the :mod:`warnings` module, or on the command line. :exc:`ResourceWarning` is issued at interpreter shutdown if the :data:`gc.garbage` list isn't empty. This is meant to make the programmer aware that their code contains object finalization issues. (Added by Antoine Pitrou and Georg Brandl; :issue:`477863`.) :exc:`ResourceWarning` is also issued when a :term:`file object` is destroyed without having been explicitly closed. While the deallocator for such object ensures it closes the underlying operating system resource (usually, a file descriptor), the delay in deallocating the object could produce various issues, especially under Windows. Here is an example of enabling the warning from the command line:: $ ./python -Wdefault Python 3.2a3+ (py3k, Nov 5 2010, 22:58:04) [GCC 4.4.3] on linux2 Type "help", "copyright", "credits" or "license" for more information. >>> f = open("foo", "wb") >>> del f __main__:1: ResourceWarning: unclosed file <_io.BufferedWriter name='foo'> >>> (Added by Antoine Pitrou, :issue:`10093`.) .. XXX: Issues #9213 and #2690 make the objects returned by range() more sequence like in accordance with their registration as implementing the Sequence ABC New, Improved, and Deprecated Modules ===================================== * The :mod:`functools` module includes a new decorator for caching function calls. :func:`functools.lru_cache` can save repeated queries to an external resource whenever the results are expected to be the same. For example, adding a caching decorator to a database query function can save database accesses for popular searches:: @functools.lru_cache(maxsize=300) def get_phone_number(name): c = conn.cursor() c.execute('SELECT phonenumber FROM phonelist WHERE name=?', (name,)) return c.fetchone()[0] >>> for name in user_requests: ... get_phone_number(name) # cached lookup To help with choosing an effective cache size, the wrapped function is instrumented for tracking cache statistics: >>> get_phone_number.cache_info() CacheInfo(hits=4805, misses=980, maxsize=300, currsize=300) If the phonelist table gets updated, the outdated contents of the cache can be cleared with: >>> get_phone_number.cache_clear() (Contributed by Raymond Hettinger and incorporating design ideas from Jim Baker, Miki Tebeka, and Nick Coglan.) * The :func:`functools.wraps` decorator now adds a :attr:`__wrapped__` attribute pointing to the original callable function. This allows wrapped functions to be introspected. It also copies :attr:`__annotations__` if defined. And now it also gracefully skips over missing attributes such as :attr:`__doc__` which might not be defined for the wrapped callable. (By Nick Coghlan and Terrence Cole; :issue:`9567`, :issue:`3445`, and :issue:`8814`.) * The :mod:`itertools` module has a new function, :func:`~itertools.accumulate` modeled on APL's *scan* operator and on Numpy's *accumulate* function: >>> list(accumulate(8, 2, 50)) [8, 10, 60] >>> prob_dist = [0.1, 0.4, 0.2, 0.3] >>> list(accumulate(prob_dist)) # cumulative probability distribution [0.1, 0.5, 0.7, 1.0] For an example using :func:`~itertools.accumulate`, see the :ref:`examples for the random module `. (Contributed by Raymond Hettinger and incorporating design suggestions from Mark Dickinson.) * The :mod:`nntplib` module gets a revamped implementation with better bytes / unicode semantics as well as more practical APIs. These improvements break compatibility with the nntplib version in Python 3.1, which was partly dysfunctional in itself. (Contributed by Antoine Pitrou in :issue:`9360`) * The :mod:`abc` module now supports :func:`~abc.abstractclassmethod` and :func:`~abc.abstractstaticmethod`. These tools make it possible to define an :term:`Abstract Base Class` that requires a particular :func:`classmethod` or :func:`staticmethod` to be implemented. (Patch submitted by Daniel Urban; :issue:`5867`.) * The previously deprecated :func:`contextlib.nested` function has been removed in favor of a plain :keyword:`with` statement which can accept multiple context managers. The latter technique is faster (because it is built-in), and it does a better job finalizing multiple context managers when one of them raises an exception. (Contributed by Georg Brandl and Mattias Brändström; `appspot issue 53094 `_.) * The :class:`ftplib.FTP` class now supports the context manager protocol to unconditionally consume :exc:`socket.error` exceptions and to close the FTP connection when done:: >>> from ftplib import FTP >>> with FTP("ftp1.at.proftpd.org") as ftp: ... ftp.login() ... ftp.dir() ... '230 Anonymous login ok, restrictions apply.' dr-xr-xr-x 9 ftp ftp 154 May 6 10:43 . dr-xr-xr-x 9 ftp ftp 154 May 6 10:43 .. dr-xr-xr-x 5 ftp ftp 4096 May 6 10:43 CentOS dr-xr-xr-x 3 ftp ftp 18 Jul 10 2008 Fedora Other file-like objects such as :class:`mmap.mmap` and :func:`fileinput.input` also grew auto-closing context managers:: with fileinput.input(files=('log1.txt', 'log2.txt')) as f: for line in f: process(line) (Contributed by Tarek Ziadé and Giampaolo Rodolà in :issue:`4972`, and by Georg Brandl in :issue:`8046` and :issue:`1286`.) * :class:`gzip.GzipFile` now implements the :class:`io.BufferedIOBase` ABC (except for ``truncate()``), has a :meth:`~gzip.GzipFile.peek` method, and supports unseekable as well as zero-padded file objects. (Contributed by Antoine Pitrou, Nir Aides and Brian Curtin in :issue:`9962`, :issue:`1675951`, :issue:`7471` and :issue:`2846`.) The :mod:`gzip` module also gains the :func:`~gzip.compress` and :func:`~gzip.decompress` functions for easier in-memory compression and decompression. (Contributed by Anand B. Pillai in :issue:`3488`.) * The :mod:`os` module now has the :const:`ST_RDONLY` and :const:`ST_NOSUID` constants, for use with the :func:`~os.statvfs` function. (Patch by Adam Jackson; :issue:`7647`.) * :func:`os.getppid` is now supported on Windows. Note that it will continue to return the same pid even after the parent process has exited. (Patch by Jon Anglin; :issue:`6394`.) * The :func:`shutil.copytree` function has two new options: * *ignore_dangling_symlinks*: when ``symlinks=False`` so that the function copies the file pointed to by the symlink, not the symlink itself. This option will silence the error raised if the file doesn't exist. * *copy_function*: is a callable that will be used to copy files. :func:`shutil.copy2` is used by default. (Contributed by Tarek Ziadé.) * Socket objects now have a :meth:`~socket.socket.detach()` method which puts the socket into closed state without actually closing the underlying file descriptor. The latter can then be reused for other purposes. (Added by Antoine Pitrou; :issue:`8524`.) * The :mod:`sqlite3` module has two new capabilities. The :attr:`Connection.in_transit` attribute is true if there is an active transaction for uncommitted changes. The :meth:`Connection.enable_load_extension` and :meth:`Connection.load_extension` methods allows you to load SQLite extensions from ".so" files. One well-known extension is the fulltext-search extension distributed with SQLite. (Contributed by R. David Murray and Shashwat Anand; :issue:`8845`.) * The :mod:`ssl` module has a new class, :class:`~ssl.SSLContext` which serves as a container for various persistent SSL data, such as protocol settings, certificates, private keys, and various other options. The :meth:`~ssl.SSLContext.wrap_socket` method allows to create an SSL socket from such an SSL context. (Added by Antoine Pitrou; :issue:`8550`.) A new function, :func:`ssl.match_hostname`, helps implement server identity verification for higher-level protocols by implementing the rules of HTTPS (from :rfc:`2818`), which are also suitable for other protocols. (Added by Antoine Pitrou, :issue:`1589`). The :func:`ssl.wrap_socket` constructor function now takes a *ciphers* argument that's a string listing the encryption algorithms to be allowed; the format of the string is described `in the OpenSSL documentation `__. (Added by Antoine Pitrou; :issue:`8322`.) When linked against a recent enough version of OpenSSL, the :mod:`ssl` module now supports the Server Name Indication extension to the TLS protocol, allowing for several "virtual hosts" using different certificates on a single IP/port. This extension is only supported in client mode, and is activated by passing the *server_hostname* argument to :meth:`SSLContext.wrap_socket`. (Added by Antoine Pitrou, :issue:`5639`.) Various options have been added to the :mod:`ssl` module, such as :data:`~ssl.OP_NO_SSLv2` which allows to force disabling of the insecure and obsolete SSLv2 protocol. (Added by Antoine Pitrou; :issue:`4870`.) Another change makes the extension load all of OpenSSL's ciphers and digest algorithms so that they're all available. Some SSL certificates couldn't be verified, reporting an "unknown algorithm" error. (Reported by Beda Kosata, and fixed by Antoine Pitrou; :issue:`8484`.) The version of OpenSSL being used is now available as the module attributes :data:`ssl.OPENSSL_VERSION` (a string), :data:`ssl.OPENSSL_VERSION_INFO` (a 5-tuple), and :data:`ssl.OPENSSL_VERSION_NUMBER` (an integer). (Added by Antoine Pitrou; :issue:`8321`.) * :class:`http.client.HTTPSConnection`, :class:`urllib.request.HTTPSHandler` and :func:`urllib.request.urlopen` now take optional arguments to allow for server certificate checking against a set of Certificate Authorities, as recommended in public uses of HTTPS. (Added by Antoine Pitrou, :issue:`9003`.) * Instances of :class:`unittest.TestCase` have two new methods :meth:`~unittest.TestCase.assertWarns` and :meth:`~unittest.TestCase.assertWarnsRegexp` to check that a given warning type was triggered by the code under test:: with self.assertWarns(DeprecationWarning): legacy_function('XYZ') * The following :class:`unittest.TestCase` methods are now deprecated: * :meth:`assert_` (use :meth:`.assertTrue` instead); * :meth:`assertEquals` (use :meth:`.assertEqual` instead); * :meth:`assertNotEquals` (use :meth:`.assertNotEqual` instead); * :meth:`assertAlmostEquals` (use :meth:`.assertAlmostEqual` instead); * :meth:`assertNotAlmostEquals` (use :meth:`.assertNotAlmostEqual` instead); The ``TestCase.fail*`` methods deprecated in Python 3.1 will be removed in Python 3.3. See also the :ref:`deprecated-aliases` section in the :mod:`unittest` documentation. (Contributed by Ezio Melotti; :issue:`9424`.) * The previously deprecated :func:`string.maketrans` function has been removed in favor of the static methods, :meth:`bytes.maketrans` and :meth:`bytearray.maketrans`. This change solves the confusion around which types were supported by the :mod:`string` module. Now, :class:`str`, :class:`bytes`, and :class:`bytearray` each have their own **maketrans** and **translate** methods with intermediate translation tables of the appropriate type. (Contributed by Georg Brandl; :issue:`5675`.) * :class:`~poplib.POP3_SSL` class now accepts a *context* parameter, which is a :class:`ssl.SSLContext` object allowing bundling SSL configuration options, certificates and private keys into a single (potentially long-lived) structure. (Contributed by Giampaolo Rodolà; :issue:`8807`.) * :func:`socket.create_connection` now supports the context manager protocol to unconditionally consume :exc:`socket.error` exceptions and to close the socket when done. (Contributed by Giampaolo Rodolà; :issue:`9794`.) * :class:`asyncore.dispatcher` now provides a :meth:`~asyncore.dispatcher.handle_accepted()` method returning a `(sock, addr)` pair which is called when a connection has actually been established with a new remote endpoint. This is supposed to be used as a replacement for old :meth:`~asyncore.dispatcher.handle_accept()` and avoids the user to call :meth:`~asyncore.dispatcher.accept()` directly. (Contributed by Giampaolo Rodolà; :issue:`6706`.) * The :mod:`tempfile` module has a new context manager, :class:`~tempfile.TemporaryDirectory` which provides easy deterministic cleanup of temporary directories. (Contributed by Neil Schemenauer and Nick Coghlan; :issue:`5178`.) * The :mod:`smtplib` :class:`~smtplib.SMTP` class now accepts a byte string for the *msg* argument to the :meth:`~smtplib.SMTP.sendmail` method, and a new method, :meth:`~smtplib.SMTP.send_message` accepts a :class:`~email.message.Message` object and can optionally obtain the *from_addr* and *to_addrs* addresses directly from the object. (Contributed by R. David Murray, :issue:`10321`.) * The :mod:`inspect` module has a new function :func:`getgenatorstate` to easily identify the current state of a generator as one of ``GEN_CREATED``, ``GEN_RUNNING``, ``GEN_SUSPENDED`` or ``GEN_CLOSED``. (Contributed by Rodolpho Eckhardt and Nick Coghlan, :issue:`10220`.) .. XXX: Mention inspect.getattr_static (Michael Foord) .. XXX: Mention urllib.parse changes Issue 9873 (Nick Coghlan): - ASCII byte sequence support in URL parsing - named tuple for urldefrag return value Issue 5468 (Dan Mahn) for urlencode: - bytes input support - non-UTF8 percent encoding of non-ASCII characters Issue 2987 for IPv6 (RFC2732) support in urlparse * The :mod:`pydoc` module now provides a much improved Web server interface, as well as a new command-line option to automatically open a browser window to display that server. (Contributed by Ron Adam; :issue:`2001`.) Multi-threading =============== * The mechanism for serializing execution of concurrently running Python threads (generally known as the GIL or Global Interpreter Lock) has been rewritten. Among the objectives were more predictable switching intervals and reduced overhead due to lock contention and the number of ensuing system calls. The notion of a "check interval" to allow thread switches has been abandoned and replaced by an absolute duration expressed in seconds. This parameter is tunable through :func:`sys.setswitchinterval()`. It currently defaults to 5 milliseconds. Additional details about the implementation can be read from a `python-dev mailing-list message `_ (however, "priority requests" as exposed in this message have not been kept for inclusion). (Contributed by Antoine Pitrou.) * Recursive locks (created with the :func:`threading.RLock` API) now benefit from a C implementation which makes them as fast as regular locks, and between 10x and 15x faster than their previous pure Python implementation. (Contributed by Antoine Pitrou; :issue:`3001`.) * Regular and recursive locks now accept an optional *timeout* argument to their :meth:`acquire` method. (Contributed by Antoine Pitrou; :issue:`7316`.) Similarly, :meth:`threading.Semaphore.acquire` also gains a *timeout* argument. (Contributed by Torsten Landschoff; :issue:`850728`.) Optimizations ============= A number of small performance enhancements have been added: * JSON decoding performance is improved and memory consumption is reduced whenever the same string is repeated for multiple keys. (Contributed by Antoine Pitrou; :issue:`7451`.) * JSON encoding now uses the C speedups also when the ``sort_keys`` argument is true. (Contributed by Raymond Hettinger and Antoine Pitrou, :issue:`10314`.) * Python's peephole optimizer now recognizes patterns such ``x in {1, 2, 3}`` as being a test for membership in a set of constants. The optimizer recasts the :class:`set` as a :class:`frozenset` and stores the pre-built constant. Now that the speed penalty is gone, it is practical to start writing membership tests using set-notation. This style is both semantically clear and operationally fast:: extension = name.rpartition('.')[2] if extension in {'xml', 'html', 'xhtml', 'css'}: handle(name) (Patch and additional tests by Dave Malcolm; :issue:`6690`). * The fast-search algorithm in stringlib is now used by the :meth:`split`, :meth:`rsplit`, :meth:`splitlines` and :meth:`replace` methods on :class:`bytes`, :class:`bytearray` and :class:`str` objects. Likewise, the algorithm is also used by :meth:`rfind`, :meth:`rindex`, :meth:`rsplit` and :meth:`rpartition`. (Patch by Florent Xicluna in :issue:`7622` and :issue:`7462`.) * Serializing and unserializing data using the :mod:`pickle` module is now several times faster. (Contributed by Alexandre Vassalotti, Antoine Pitrou and the Unladen Swallow team in :issue:`9410` and :issue:`3873`.) * The `Timsort algorithm `_ used in :meth:`list.sort` and :func:`sorted` now runs faster and used less memory when called with a :term:`key function`. Previously, every element of a list was wrapped with a temporary object that remembered the key value associated with each element. Now, an array of keys and values are sorted in parallel. This save the memory consumed by the sort wrappers, and it saves time lost from during comparisons which where delegated by the sort wrappers. (Patch by Daniel Stuzback in :issue:`9915`.) Unicode ======= Python has been updated to Unicode 6.0.0. The new features of the Unicode Standard that will affect Python users include: * adds 2,088 characters, including over 1,000 additional symbols—chief among them the additional emoji symbols, which are especially important for mobile phones; * corrects character properties for existing characters including - a general category change to two Kannada characters (U+0CF1, U+0CF2), which has the effect of making them newly eligible for inclusion in identifiers; - a general category change to one New Tai Lue numeric character (U+19DA), which would have the effect of disqualifying it from inclusion in identifiers unless grandfathering measures are in place for the defining identifier syntax. The :mod:`os` module has two new functions: :func:`~os.fsencode` and :func:`~os.fsdecode`. Add :data:`os.environb`: bytes version of :data:`os.environ`, :func:`os.getenvb` function and :data:`os.supports_bytes_environ` constant. ``'mbcs'`` encoding doesn't ignore the error handler argument any more. By default (strict mode), it raises an UnicodeDecodeError on undecodable byte sequence and UnicodeEncodeError on unencodable character. To get the ``'mbcs'`` encoding of Python 3.1, use ``'ignore'`` error handler to decode and ``'replace'`` error handler to encode. ``'mbcs'`` supports ``'strict'`` and ``'ignore'`` error handlers for decoding, and ``'strict'`` and ``'replace'`` for encoding. On Mac OS X, Python uses ``'utf-8'`` to decode the command line arguments, instead of the locale encoding (which is ISO-8859-1 if the ``LANG`` environment variable is not set). By default, tarfile uses ``'utf-8'`` encoding on Windows (instead of ``'mbcs'``), and the ``'surrogateescape'`` error handler on all operating systems. .. IDLE ==== * Stub Build and C API Changes ======================= Changes to Python's build process and to the C API include: * The C functions that access the Unicode Database now accept and return characters from the full Unicode range, even on narrow unicode builds (Py_UNICODE_TOLOWER, Py_UNICODE_ISDECIMAL, and others). A visible difference in Python is that :func:`unicodedata.numeric` now returns the correct value for large code points, and :func:`repr` may consider more characters as printable. (Reported by Bupjoe Lee and fixed by Amaury Forgeot D'Arc; :issue:`5127`.) * Computed gotos are now enabled by default on supported compilers (which are detected by the configure script). They can still be disabled selectively by specifying ``--without-computed-gotos``. (Contributed by Antoine Pitrou; :issue:`9203`.) * The option ``--with-wctype-functions`` was removed. The built-in unicode database is now used for all functions. (Contributed by Amaury Forgeot D'Arc; :issue:`9210`.) * Hash values are now values of a new type, Py_hash_t, which is defined to be the same size as a pointer. Previously they were of type long, which on some 64-bit operating systems is still only 32 bits long. (Contributed by Benjamin Peterson; :issue:`9778`.) Porting to Python 3.2 ===================== This section lists previously described changes and other bugfixes that may require changes to your code: * The :mod:`nntplib` module was reworked extensively, meaning that its APIs are often incompatible with the 3.1 APIs. * :class:`bytearray` objects cannot be used any more as filenames: convert them to :class:`bytes`. * PyArg_Parse*() functions: * "t#" format has been removed: use "s#" or "s*" instead * "w" and "w#" formats has been removed: use "w*" instead * The :c:type:`PyCObject` type, deprecated in 3.1, has been removed. To wrap opaque C pointers in Python objects, the :c:type:`PyCapsule` API should be used instead; the new type has a well-defined interface for passing typing safety information and a less complicated signature for calling a destructor. * The :func:`sys.setfilesystemencoding` function was removed because it has a flawed design.