**************************** What's New In Python 3.2 **************************** :Author: Raymond Hettinger :Release: |release| :Date: |today| .. $Id$ Rules for maintenance: * Anyone can add text to this document. Do not spend very much time on the wording of your changes, because your text will probably get rewritten to some degree. * The maintainer will go through Misc/NEWS periodically and add changes; it's therefore more important to add your changes to Misc/NEWS than to this file. * This is not a complete list of every single change; completeness is the purpose of Misc/NEWS. Some changes I consider too small or esoteric to include. If such a change is added to the text, I'll just remove it. (This is another reason you shouldn't spend too much time on writing your addition.) * If you want to draw your new text to the attention of the maintainer, add 'XXX' to the beginning of the paragraph or section. * It's OK to just add a fragmentary note about a change. For example: "XXX Describe the transmogrify() function added to the socket module." The maintainer will research the change and write the necessary text. * You can comment out your additions if you like, but it's not necessary (especially when a final release is some months away). * Credit the author of a patch or bugfix. Just the name is sufficient; the e-mail address isn't necessary. It's helpful to add the issue number: XXX Describe the transmogrify() function added to the socket module. (Contributed by P.Y. Developer; :issue:`12345`.) This saves the maintainer the effort of going through the SVN log when researching a change. This article explains the new features in Python 3.2, compared to 3.1. PEP 391: Dictionary Based Configuration for Logging ==================================================== The :mod:`logging` module provided two kinds of configuration, one style with function calls for each option or another style driven by an external file saved in a :mod:`ConfigParser` format. Those options did not provide the flexibility to create configurations from JSON or YAML files, nor did they support incremental configuration, which is needed for specifying logger options from a command line. To support a more flexible style, the module now offers :func:`logging.config.dictConfig` for specifying logging configuration with plain Python dictionaries. The configuration options include formatters, handlers, filters, and loggers. Here's a working example of a configuration dictionary:: {"version": 1, "formatters": {"brief": {"format": "%(levelname)-8s: %(name)-15s: %(message)s"}, "full": {"format": "%(asctime)s %(name)-15s %(levelname)-8s %(message)s"}, }, "handlers": {"console": { "class": "logging.StreamHandler", "formatter": "brief", "level": "INFO", "stream": "ext://sys.stdout"}, "console_priority": { "class": "logging.StreamHandler", "formatter": "full", "level": "ERROR", "stream": "ext://sys.stderr"}, }, "root": {"level": "DEBUG", "handlers": ["console", "console_priority"]}} If that dictionary is stored in a file called "conf.json", it can loaded and called with code like this:: >>> import logging.config >>> logging.config.dictConfig(json.load(open('conf.json', 'rb'))) >>> logging.info("Transaction completed normally") >>> logging.critical("Abnormal termination") .. seealso:: :pep:`391` - Dictionary Based Configuration for Logging PEP written by Vinay Sajip. PEP 3147: PYC Repository Directories ===================================== Python's scheme for caching bytecode in *.pyc* files did not work well in environments with multiple python interpreters. If one interpreter encountered a cached file created by another interpreter, it would recompile the source and overwrite the cached file, thus losing the benefits of caching. The issue of "pyc fights" has become more pronounced as it has become commonplace for Linux distributions to ship with multiple versions of Python. These conflicts also arise with CPython alternatives such as Unladen Swallow. To solve this problem, Python's import machinery has been extended to use distinct filenames for each interpreter. Instead of Python 3.2 and Python 3.3 and Unladen Swallow each competing for a file called "mymodule.pyc", they will now look for "mymodule.cpython-32.pyc", "mymodule.cpython-33.pyc", and "mymodule.unladen10.pyc". And to prevent all of these new files from cluttering source directories, the *pyc* files are now collected in a "__pycache__" directory stored under the package directory. Aside from the filenames and target directories, the new scheme has a few aspects that are visible to the programmer: * Imported modules now have a :attr:`__cached__` attribute which stores the name of the actual file that was imported: >>> import collections >>> collections.__cached__ 'c:/py32/lib/__pycache__/collections.cpython-32.pyc' * The tag that is unique to each interpreter is accessible from the :mod:`imp` module: >>> import imp >>> imp.get_tag() 'cpython-32' * Scripts that try to deduce source filename from the imported file now need to be smarter. It is no longer sufficient to simply strip the "c" from a ".pyc" filename. Instead, use the new functions in the :mod:`imp` module: >>> imp.source_from_cache('c:/py32/lib/__pycache__/collections.cpython-32.pyc') 'c:/py32/lib/collections.py' >>> imp.cache_from_source('c:/py32/lib/collections.py') 'c:/py32/lib/__pycache__/collections.cpython-32.pyc' * The :mod:`py_compile` and :mod:`compileall` modules have been updated to reflect the new naming convention and target directory. .. seealso:: :pep:`3147` - PYC Repository Directories PEP written by Barry Warsaw. PEP 3149 ABI Version Tagged .so Files ===================================== The PYC repository directory allows multiple bytecode cache files to be co-located. This PEP implements a similar mechanism for shared object files by giving them a common directory and distinct names for each version. The common directory is "pyshared" and the file names are made distinct by identifying the Python implementation (such as CPython, PyPy, Jython, etc.), the major and minor version numbers, and optional build flags (such as "d" for debug, "m" for pymalloc, "u" for wide-unicode). For an arbitrary package "foo", you may see these files when the distribution package is installed:: /usr/share/pyshared/foo.cpython-32m.so /usr/share/pyshared/foo.cpython-33md.so In Python itself, the tags are accessible from functions in the :mod:`sysconfig` module:: >>> import sysconfig >>> sysconfig.get_config_var('SOABI') # find the version tag 'cpython-32mu' >>> sysconfig.get_config_var('SO') # find the full filename extension 'cpython-32mu.so' .. seealso:: :pep:`3149` - ABI Version Tagged .so Files PEP written by Barry Warsaw. Email 5.1 ========= The email package is extended to be able to parse and generate email messages in bytes format. * New functions :func:`~email.message_from_bytes` and :func:`~email.message_from_binary_file`, and new classes :class:`~email.parser.BytesFeedParser` and :class:`~email.parser.BytesParser` allow binary message data to be parsed into model objects. * Given bytes input to the model, :meth:`~email.message.Message.get_payload` will by default decode a message body that has a :mailheader:`Content-Transfer-Encoding` of ``8bit`` using the charset specified in the MIME headers and return the resulting string. * Given bytes input to the model, :class:`~email.generator.Generator` will convert message bodies that have a :mailheader:`Content-Transfer-Encoding` of 8bit to instead have a 7bit Content-Transfer-Encoding. * New class :class:`~email.generator.BytesGenerator` produces bytes as output, preserving any unchanged non-ASCII data that was present in the input used to build the model, including message bodies with a :mailheader:`Content-Transfer-Encoding` of 8bit. (Proposed and implemented by R. David Murray, :issue:`4661`.) Other Language Changes ====================== Some smaller changes made to the core Python language are: * The :func:`hasattr` function used to catch and suppress any Exception. Now, it only catches :exc:`AttributeError`. Under the hood, :func:`hasattr` works by calling :func:`getattr` and throwing away the results. This is necessary because dynamic attribute creation is possible using :meth:`__getattribute__` or :meth:`__getattr__`. If :func:`hasattr` were to just scan instance and class dictionaries it would miss the dynamic methods and make it difficult to implement proxy objects. (Discovered by Yury Selivanov and fixed by Benjamin Peterson; :issue:`9666`.) * The :func:`str` of a float or complex number is now the same as its :func:`repr`. Previously, the :func:`str` form was shorter but that just caused confusion and is no longer needed now that the shortest possible :func:`repr` is displayed by default: >>> repr(math.pi) '3.141592653589793' >>> str(math.pi) '3.141592653589793' (Proposed and implemented by Mark Dickinson; :issue:`9337`.) * :class:`memoryview` objects now have a :meth:`release()` method and support the context manager protocol. This allows timely release of any resources that were acquired when requesting a buffer from the original object. (Added by Antoine Pitrou; :issue:`9757`.) * Mark Dickinson crafted an elegant and efficient scheme for assuring that different numeric datatypes will have the same hash value whenever their actual values are equal:: >>> assert hash(Fraction(3, 2)) == hash(1.5) == \ hash(Decimal("1.5")) == hash(complex(1.5, 0)) (See :issue:`8188`.) * Previously it was illegal to delete a name from the local namespace if it occurs as a free variable in a nested block:: >>> def outer(x): ... def inner(): ... return x ... inner() ... del x This is now allowed. Remember that the target of an :keyword:`except` clause is cleared, so this code which used to work with Python 2.6, raised a :exc:`SyntaxError` with Python 3.1 and now works again:: >>> def f(): ... def print_error(): ... print(e) ... try: ... something ... except Exception as e: ... print_error() ... # implicit "del e" here (See :issue:`4617`.) * A new warning category, :exc:`ResourceWarning`, has been added. It is emitted when certain potential issues with resource consumption or cleanup are detected. It is silenced by default in normal release builds, but can be easily enabled through the means provided by the :mod:`warnings` module, or on the command line. :exc:`ResourceWarning` is issued at interpreter shutdown if the :data:`gc.garbage` list isn't empty. This is meant to make the programmer aware that their code contains object finalization issues. (Added by Antoine Pitrou and Georg Brandl; :issue:`477863`.) :exc:`ResourceWarning` is also issued when a :term:`file object` is destroyed without having been explicitly closed. While the deallocator for such object ensures it closes the underlying operating system resource (usually, a file descriptor), the delay in deallocating the object could produce various issues, especially under Windows. Here is an example of enabling the warning from the command line:: $ ./python -Wdefault Python 3.2a3+ (py3k, Nov 5 2010, 22:58:04) [GCC 4.4.3] on linux2 Type "help", "copyright", "credits" or "license" for more information. >>> f = open("foo", "wb") >>> del f __main__:1: ResourceWarning: unclosed file <_io.BufferedWriter name='foo'> >>> (Added by Antoine Pitrou, :issue:`10093`.) New, Improved, and Deprecated Modules ===================================== * XXX mention :mod:`argparse`. * The :mod:`functools` module includes a new decorator for caching function calls. :func:`functools.lru_cache` can save repeated queries to an external resource whenever the results are expected to be the same. For example, adding a caching decorator to a database query function can save database accesses for popular searches:: @functools.lru_cache(maxsize=300) def get_phone_number(name): c = conn.cursor() c.execute('SELECT phonenumber FROM phonelist WHERE name=?', (name,)) return c.fetchone()[0] To help with choosing an effective cache size, the wrapped function is instrumented with two attributes *cache_hits* and *cache_misses*: >>> for name in user_requests: ... get_phone_number(name) >>> print(get_phone_number.cache_hits, get_phone_number.cache_misses) 4805 980 If the phonelist table gets updated, the outdated contents of the cache can be cleared with: >>> get_phone_number.cache_clear() (Contributed by Raymond Hettinger.) * The :func:`functools.wraps` decorator now adds a :attr:`__wrapped__` attribute pointing to the original callable function. This allows wrapped functions to be introspected. It also copies :attr:`__annotations__` if defined. And now it also gracefully skips over missing attributes such as :attr:`__doc__` which might not be defined for the wrapped callable. (By Nick Coghlan and Terrence Cole; :issue:`9567`, :issue:`3445`, and :issue:`8814`.) * The :mod:`nntplib` module gets a revamped implementation with better bytes / unicode semantics as well as more practical APIs. These improvements break compatibility with the nntplib version in Python 3.1, which was partly dysfunctional in itself. (Contributed by Antoine Pitrou in :issue:`9360`) * The :mod:`abc` module now supports :func:`~abc.abstractclassmethod` and :func:`~abc.abstractstaticmethod`. (Patch submitted by Daniel Urban; :issue:`5867`.) * The previously deprecated :func:`contextlib.nested` function has been removed in favor of a plain :keyword:`with` statement which can accept multiple context managers. The latter technique is faster (because it is built-in), and it does a better job finalizing multiple context managers when one of them raises an exception. (Contributed by Georg Brandl and Mattias Brändström; `appspot issue 53094 `_.) * The :class:`ftplib.FTP` class now supports the context manager protocol to unconditionally consume :exc:`socket.error` exceptions and to close the FTP connection when done:: >>> from ftplib import FTP >>> with FTP("ftp1.at.proftpd.org") as ftp: ... ftp.login() ... ftp.dir() ... '230 Anonymous login ok, restrictions apply.' dr-xr-xr-x 9 ftp ftp 154 May 6 10:43 . dr-xr-xr-x 9 ftp ftp 154 May 6 10:43 .. dr-xr-xr-x 5 ftp ftp 4096 May 6 10:43 CentOS dr-xr-xr-x 3 ftp ftp 18 Jul 10 2008 Fedora Other file-like objects such as :class:`mmap.mmap` and :func:`fileinput.input` also grew auto-closing context managers:: with fileinput.input(files=('log1.txt', 'log2.txt')) as f: for line in f: process(line) (Contributed by Tarek Ziadé and Giampaolo Rodolà in :issue:`4972`, and by Georg Brandl in :issue:`8046` and :issue:`1286`.) * :class:`gzip.GzipFile` now implements the :class:`io.BufferedIOBase` ABC (except for ``truncate()``), has a :meth:`~gzip.GzipFile.peek` method, and supports unseekable as well as zero-padded file objects. (Contributed by Antoine Pitrou, Nir Aides and Brian Curtin in :issue:`9962`, :issue:`1675951`, :issue:`7471` and :issue:`2846`.) The :mod:`gzip` module also gains the :func:`~gzip.compress` and :func:`~gzip.decompress` functions for easier in-memory compression and decompression. (Contributed by Anand B. Pillai in :issue:`3488`.) * The :mod:`os` module now has the :const:`ST_RDONLY` and :const:`ST_NOSUID` constants, for use with the :func:`~os.statvfs` function. (Patch by Adam Jackson; :issue:`7647`.) * :func:`os.getppid` is now supported on Windows. Note that it will continue to return the same pid even after the parent process has exited. (Patch by Jon Anglin; :issue:`6394`.) * The :func:`shutil.copytree` function has two new options: * *ignore_dangling_symlinks*: when ``symlinks=False`` so that the function copies the file pointed to by the symlink, not the symlink itself. This option will silence the error raised if the file doesn't exist. * *copy_function*: is a callable that will be used to copy files. :func:`shutil.copy2` is used by default. (Contributed by Tarek Ziadé.) * Socket objects now have a :meth:`~socket.socket.detach()` method which puts the socket into closed state without actually closing the underlying file descriptor. The latter can then be reused for other purposes. (Added by Antoine Pitrou; :issue:`8524`.) * The :mod:`sqlite3` module has two new capabilities. The :attr:`Connection.in_transit` attribute is true if there is an active transaction for uncommitted changes. The :meth:`Connection.enable_load_extension` and :meth:`Connection.load_extension` methods allows you to load SQLite extensions from ".so" files. One well-known extension is the fulltext-search extension distributed with SQLite. (Contributed by R. David Murray and Shashwat Anand; :issue:`8845`.) * The :mod:`ssl` module has a new class, :class:`~ssl.SSLContext` which serves as a container for various persistent SSL data, such as protocol settings, certificates, private keys, and various other options. The :meth:`~ssl.SSLContext.wrap_socket` method allows to create an SSL socket from such an SSL context. (Added by Antoine Pitrou; :issue:`8550`.) A new function, :func:`ssl.match_hostname`, helps implement server identity verification for higher-level protocols by implementing the rules of HTTPS (from :rfc:`2818`), which are also suitable for other protocols. (Added by Antoine Pitrou, :issue:`1589`). The :func:`ssl.wrap_socket` constructor function now takes a *ciphers* argument that's a string listing the encryption algorithms to be allowed; the format of the string is described `in the OpenSSL documentation `__. (Added by Antoine Pitrou; :issue:`8322`.) When linked against a recent enough version of OpenSSL, the :mod:`ssl` module now supports the Server Name Indication extension to the TLS protocol, allowing for several "virtual hosts" using different certificates on a single IP/port. This extension is only supported in client mode, and is activated by passing the *server_hostname* argument to :meth:`SSLContext.wrap_socket`. (Added by Antoine Pitrou, :issue:`5639`.) Various options have been added to the :mod:`ssl` module, such as :data:`~ssl.OP_NO_SSLv2` which allows to force disabling of the insecure and obsolete SSLv2 protocol. (Added by Antoine Pitrou; :issue:`4870`.) Another change makes the extension load all of OpenSSL's ciphers and digest algorithms so that they're all available. Some SSL certificates couldn't be verified, reporting an "unknown algorithm" error. (Reported by Beda Kosata, and fixed by Antoine Pitrou; :issue:`8484`.) The version of OpenSSL being used is now available as the module attributes :data:`ssl.OPENSSL_VERSION` (a string), :data:`ssl.OPENSSL_VERSION_INFO` (a 5-tuple), and :data:`ssl.OPENSSL_VERSION_NUMBER` (an integer). (Added by Antoine Pitrou; :issue:`8321`.) * :class:`http.client.HTTPSConnection`, :class:`urllib.request.HTTPSHandler` and :func:`urllib.request.urlopen` now take optional arguments to allow for server certificate checking against a set of Certificate Authorities, as recommended in public uses of HTTPS. (Added by Antoine Pitrou, :issue:`9003`.) * Instances of :class:`unittest.TestCase` have two new methods :meth:`~unittest.TestCase.assertWarns` and :meth:`~unittest.TestCase.assertWarnsRegexp` to check that a given warning type was triggered by the code under test:: with self.assertWarns(DeprecationWarning): legacy_function('XYZ') * The previously deprecated :func:`string.maketrans` function has been removed in favor of the static methods, :meth:`bytes.maketrans` and :meth:`bytearray.maketrans`. This change solves the confusion around which types were supported by the :mod:`string` module. Now, :class:`str`, :class:`bytes`, and :class:`bytearray` each have their own **maketrans** and **translate** methods with intermediate translation tables of the appropriate type. (Contributed by Georg Brandl; :issue:`5675`.) * :class:`~poplib.POP3_SSL` class now accepts a *context* parameter, which is a :class:`ssl.SSLContext` object allowing bundling SSL configuration options, certificates and private keys into a single (potentially long-lived) structure. (Contributed by Giampaolo Rodolà; :issue:`8807`.) * :func:`socket.create_connection` now supports the context manager protocol to unconditionally consume :exc:`socket.error` exceptions and to close the socket when done. (Contributed by Giampaolo Rodolà; :issue:`9794`.) * :class:`asyncore.dispatcher` now provides a :meth:`~asyncore.dispatcher.handle_accepted()` method returning a `(sock, addr)` pair which is called when a connection has actually been established with a new remote endpoint. This is supposed to be used as a replacement for old :meth:`~asyncore.dispatcher.handle_accept()` and avoids the user to call :meth:`~asyncore.dispatcher.accept()` directly. (Contributed by Giampaolo Rodolà; :issue:`6706`.) * The :mod:`tempfile` module has a new context manager, :class:`~tempfile.TemporaryDirectory` which provides easy deterministic cleanup of temporary directories. (Contributed by Neil Schemenauer and Nick Coghlan; :issue:`5178`.) Multi-threading =============== * The mechanism for serializing execution of concurrently running Python threads (generally known as the GIL or Global Interpreter Lock) has been rewritten. Among the objectives were more predictable switching intervals and reduced overhead due to lock contention and the number of ensuing system calls. The notion of a "check interval" to allow thread switches has been abandoned and replaced by an absolute duration expressed in seconds. This parameter is tunable through :func:`sys.setswitchinterval()`. It currently defaults to 5 milliseconds. Additional details about the implementation can be read from a `python-dev mailing-list message `_ (however, "priority requests" as exposed in this message have not been kept for inclusion). (Contributed by Antoine Pitrou.) * Recursive locks (created with the :func:`threading.RLock` API) now benefit from a C implementation which makes them as fast as regular locks, and between 10x and 15x faster than their previous pure Python implementation. (Contributed by Antoine Pitrou; :issue:`3001`.) * Regular and recursive locks now accept an optional *timeout* argument to their :meth:`acquire` method. (Contributed by Antoine Pitrou; :issue:`7316`.) Similarly, :meth:`threading.Semaphore.acquire` also gains a *timeout* argument. (Contributed by Torsten Landschoff; :issue:`850728`.) Optimizations ============= A number of small performance enhancements have been added: * JSON decoding performance is improved and memory consumption is reduced whenever the same string is repeated for multiple keys. (Contributed by Antoine Pitrou; :issue:`7451`.) * JSON encoding now uses the C speedups also when the ``sort_keys`` argument is true. (Contributed by Raymond Hettinger and Antoine Pitrou, :issue:`10314`.) * Python's peephole optimizer now recognizes patterns such ``x in {1, 2, 3}`` as being a test for membership in a set of constants. The optimizer recasts the :class:`set` as a :class:`frozenset` and stores the pre-built constant. Now that the speed penalty is gone, it is practical to start writing membership tests using set-notation. This style is both semantically clear and operationally fast:: extension = name.rpartition('.')[2] if extension in {'xml', 'html', 'xhtml', 'css'}: handle(name) (Patch and additional tests by Dave Malcolm; :issue:`6690`). * The fast-search algorithm in stringlib is now used by the :meth:`split`, :meth:`rsplit`, :meth:`splitlines` and :meth:`replace` methods on :class:`bytes`, :class:`bytearray` and :class:`str` objects. Likewise, the algorithm is also used by :meth:`rfind`, :meth:`rindex`, :meth:`rsplit` and :meth:`rpartition`. (Patch by Florent Xicluna in :issue:`7622` and :issue:`7462`.) * Serializing and unserializing data using the :mod:`pickle` module is now several times faster. (Contributed by Alexandre Vassalotti, Antoine Pitrou and the Unladen Swallow team in :issue:`9410` and :issue:`3873`.) Unicode ======= The :mod:`os` module has two new functions: :func:`~os.fsencode` and :func:`~os.fsdecode`. Add :data:`os.environb`: bytes version of :data:`os.environ`, :func:`os.getenvb` function and :data:`os.supports_bytes_environ` constant. ``'mbcs'`` encoding doesn't ignore the error handler argument anymore. By default (strict mode), it raises an UnicodeDecodeError on undecodable byte sequence and UnicodeEncodeError on unencodable character. To get the ``'mbcs'`` encoding of Python 3.1, use ``'ignore'`` error handler to decode and ``'replace'`` error handler to encode. ``'mbcs'`` supports ``'strict'`` and ``'ignore'`` error handlers for decoding, and ``'strict'`` and ``'replace'`` for encoding. On Mac OS X, Python uses ``'utf-8'`` to decode the command line arguments, instead of the locale encoding (which is ISO-8859-1 if the ``LANG`` environment variable is not set). By default, tarfile uses ``'utf-8'`` encoding on Windows (instead of ``'mbcs'``), and the ``'surrogateescape'`` error handler on all operating systems. .. IDLE ==== * Stub Build and C API Changes ======================= Changes to Python's build process and to the C API include: * The C functions that access the Unicode Database now accept and return characters from the full Unicode range, even on narrow unicode builds (Py_UNICODE_TOLOWER, Py_UNICODE_ISDECIMAL, and others). A visible difference in Python is that :func:`unicodedata.numeric` now returns the correct value for large code points, and :func:`repr` may consider more characters as printable. (Reported by Bupjoe Lee and fixed by Amaury Forgeot D'Arc; :issue:`5127`.) * Computed gotos are now enabled by default on supported compilers (which are detected by the configure script). They can still be disabled selectively by specifying ``--without-computed-gotos``. (Contributed by Antoine Pitrou; :issue:`9203`.) * The option ``--with-wctype-functions`` was removed. The built-in unicode database is now used for all functions. (Contributed by Amaury Forgeot D'Arc; :issue:`9210`.) * Hash values are now values of a new type, Py_hash_t, which is defined to be the same size as a pointer. Previously they were of type long, which on some 64-bit operating systems is still only 32 bits long. (Contributed by Benjamin Peterson; :issue:`9778`.) Porting to Python 3.2 ===================== This section lists previously described changes and other bugfixes that may require changes to your code: * The :mod:`nntplib` module was reworked extensively, meaning that its APIs are often incompatible with the 3.1 APIs. * :class:`bytearray` objects cannot be used anymore as filenames: convert them to :class:`bytes`. * PyArg_Parse*() functions: * "t#" format has been removed: use "s#" or "s*" instead * "w" and "w#" formats has been removed: use "w*" instead * The :c:type:`PyCObject` type, deprecated in 3.1, has been removed. To wrap opaque C pointers in Python objects, the :c:type:`PyCapsule` API should be used instead; the new type has a well-defined interface for passing typing safety information and a less complicated signature for calling a destructor. * Remove sys.setfilesystemencoding() function: it was broken by design.