**************************** What's New In Python 3.2 **************************** :Author: Raymond Hettinger :Release: |release| :Date: |today| .. $Id$ Rules for maintenance: * Anyone can add text to this document. Do not spend very much time on the wording of your changes, because your text will probably get rewritten to some degree. * The maintainer will go through Misc/NEWS periodically and add changes; it's therefore more important to add your changes to Misc/NEWS than to this file. * This is not a complete list of every single change; completeness is the purpose of Misc/NEWS. Some changes I consider too small or esoteric to include. If such a change is added to the text, I'll just remove it. (This is another reason you shouldn't spend too much time on writing your addition.) * If you want to draw your new text to the attention of the maintainer, add 'XXX' to the beginning of the paragraph or section. * It's OK to just add a fragmentary note about a change. For example: "XXX Describe the transmogrify() function added to the socket module." The maintainer will research the change and write the necessary text. * You can comment out your additions if you like, but it's not necessary (especially when a final release is some months away). * Credit the author of a patch or bugfix. Just the name is sufficient; the e-mail address isn't necessary. * It's helpful to add the bug/patch number as a comment: % Patch 12345 XXX Describe the transmogrify() function added to the socket module. (Contributed by P.Y. Developer.) This saves the maintainer the effort of going through the SVN log when researching a change. This article explains the new features in Python 3.2, compared to 3.1. PEP 3147: PYC Repository Directories ===================================== Python's scheme for caching bytecode in *.pyc* files did not work well in environments with multiple python interpreters. If one interpreter encountered a cached file created by another interpreter, it would recompile the source and overwrite the cached file, thus losing the benefits of caching. The issue of "pyc fights" has become more pronounced as it has become common-place for Linux distributions to ship with multiple versions of Python. These conflicts also arise with CPython alternatives such as Unladen Swallow. To solve this problem, Python's import machinery has been extended to use distinct filenames for each interpreter. Instead of Python3.2 and Python3.3 and UnladenSwallow each competing for a file called "mymodule.pyc", they will now look for "mymodule.cpython-32.pyc", "mymodule.cpython-33.pyc", and "mymodule.unladen10.pyc". And to keep prevent all of these new files from cluttering source directories, the *pyc* files are now collected in a "__pycache__" directory stored under the package directory. Aside from the filenames and target directories, the new scheme has a few aspects that are visible to the programmer: * Imported modules now have a :attr:`__cached__` attribute which stores the name of the actual file that was imported:: >>> import collections >>> collections.__cached__ 'c:/py32/lib/__pycache__/collections.cpython-32.pyc' * The tag that is unique to each interpreter is accessible from the :mod:`imp` module:: >>> import imp >>> imp.get_tag() 'cpython-32' * Scripts that try to deduce source filename from the imported file now need to be smarter. It is no longer sufficient to simply strip the "c" from a ".pyc" filename. Instead, use the new functions in the :mod:`imp` module: >>> imp.source_from_cache('c:/py32/lib/__pycache__/collections.cpython-32.pyc') 'c:/py32/lib/collections.py' >>> imp.cache_from_source('c:/py32/lib/collections.py') 'c:/py32/lib/__pycache__/collections.cpython-32.pyc' * The :mod:`py_compile` and :mod:`compileall` modules have been updated to reflect the new naming convention and target directory. .. seealso:: :pep:`3147` - PYC Repository Directories PEP written by Barry Warsaw. PEP 3149 ABI Version Tagged .so Files ===================================== The PYC repository directory allows multiple bytecode cache files to be co-located. This PEP implements a similar mechanism for shared object files by giving them a common directory and distinct names for each version. The common directory is "pyshared" and the file names are made distinct by identifying the Python implementation (such as CPython, PyPy, Jython, etc.), the major and minor version numbers, and optional build flags (such as "d" for debug, "m" for pymalloc, "u" for wide-unicode). For an arbtrary package, "foo", you may see these files when the distribution package is installed:: /usr/share/pyshared/foo.cpython-32m.so /usr/share/pyshared/foo.cpython-33md.so In Python itself, the tags are accessible from functions in the :mod:`sysconfig` module:: >>> import sysconfig >>> sysconfig.get_config_var('SOABI') # find the version tag 'cpython-32mu' >>> sysconfig.get_config_var('SO') # find the full filename extension 'cpython-32mu.so' .. seealso:: :pep:`3149` - ABI Version Tagged .so Files PEP written by Barry Warsaw. Other Language Changes ====================== Some smaller changes made to the core Python language are: * Stub New, Improved, and Deprecated Modules ===================================== * The :mod:`functools` module now includes a new decorator for caching function calls. :func:`functools.lru_cache` can save repeated queries to an external resource whenever the results are expected to be the same. For example, adding a caching decorator to a database query function can save database accesses for popular searches:: @functools.lru_cache(maxsize=300) def get_phone_number(name): c = conn.cursor() c.execute('SELECT phonenumber FROM phonelist WHERE name=?', (name,)) return c.fetchone()[0] To help with choosing an effective cache size, the wrapped function is instrumented with two attributes *cache_hits* and *cache_misses*:: >>> for name in user_requests: ... get_phone_number(name) >>> print(get_phone_number.cache_hits, get_phone_number.cache_misses) 4805 980 If the phonelist table gets updated, the outdated contents of the cache can be cleared with:: >>> get_phone_number.cache_clear() (Contributed by Raymond Hettinger) * The previously deprecated :func:`contextlib.nested` function has been removed in favor of a plain :keyword:`with` statement which can accept multiple context managers. The latter technique is faster (because it is built-in), and it does a better job finalizing multiple context managers when one of them raises an exception. (Contributed by Georg Brandl and Mattias Brändström; `appspot issue 53094 `_.) * The :class:`ftplib.FTP` class now supports the context manager protocol (Contributed by Tarek Ziadé and Giampaolo Rodolà; :issue:`4972`.) * A warning message will now get printed at interpreter shutdown if the :data:`gc.garbage` list isn't empty. This is meant to make the programmer aware that his code contains object finalization issues. (Added by Antoine Pitrou; :issue:`477863`.) * The :mod:`os` module now has the :const:`ST_RDONLY` and :const:`ST_NOSUID` constants, for use with the :func:`~os.statvfs` function. (Patch by Adam Jackson; :issue:`7647`.) * The :func:`shutil.copytree` function has two new options: * *ignore_dangling_symlinks*: when ``symlinks=False`` (meaning that the function copies the file pointed to by the symlink, not the symlink itself) this option will silence the error raised if the file doesn't exist. * *copy_function*: a callable that will be used to copy files. :func:`shutil.copy2` is used by default. (Contributed by Tarek Ziadé.) * Socket objects now have a :meth:`~socket.socket.detach()` method which puts the socket into closed state without actually closing the underlying file descriptor. The latter can then be reused for other purposes. (Added by Antoine Pitrou; :issue:`8524`.) * The :mod:`sqlite3` module has some new features: * XXX *enable_load_extension* * XXX *load_extension* * New :class:`~sqlite3.Connection` attribute :attr:`~sqlite3.Connection.in_transaction` is :const:`True` when there are uncommitted changes, and :const:`False` otherwise. (Contributed by R. David Murray and Shashwat Anand, :issue:`8845`.) * The :mod:`ssl` module has a new class, :class:`~ssl.SSLContext` which serves as a container for various persistent SSL data, such as protocol settings, certificates, private keys, and various other options. The :meth:`~ssl.SSLContext.wrap_socket` method allows to create an SSL socket from such an SSL context. (Added by Antoine Pitrou; :issue:`8550`.) The :func:`ssl.wrap_socket` constructor function now takes a *ciphers* argument that's a string listing the encryption algorithms to be allowed; the format of the string is described `in the OpenSSL documentation `__. (Added by Antoine Pitrou; :issue:`8322`.) Various options have been added to the :mod:`ssl` module, such as :data:`~ssl.OP_NO_SSLv2` which allows to force disabling of the insecure and obsolete SSLv2 protocol. (Added by Antoine Pitrou; :issue:`4870`.) Another change makes the extension load all of OpenSSL's ciphers and digest algorithms so that they're all available. Some SSL certificates couldn't be verified, reporting an "unknown algorithm" error. (Reported by Beda Kosata, and fixed by Antoine Pitrou; :issue:`8484`.) The version of OpenSSL being used is now available as the module attributes :data:`ssl.OPENSSL_VERSION` (a string), :data:`ssl.OPENSSL_VERSION_INFO` (a 5-tuple), and :data:`ssl.OPENSSL_VERSION_NUMBER` (an integer). (Added by Antoine Pitrou; :issue:`8321`.) * The previously deprecated :func:`string.maketrans` function has been removed in favor of the static methods, :meth:`bytes.maketrans` and :meth:`bytearray.maketrans`. This change solves the confusion around which types were supported by the :mod:`string` module. Now, :class:`str`, :class:`bytes`, and :class:`bytearray` each have their own **maketrans** and **translate** methods with intermediate translation tables of the appropriate type. (Contributed by Georg Brandl; :issue:`5675`.) * Parameters passed to :func:`socket.getaddrinfo()` function can now be specified as single keyword arguments. (Contributed by Giampaolo Rodolà; :issue:`8866`.) * :class:`~poplib.POP3_SSL` class now accepts a *context* parameter, which is a :class:`ssl.SSLContext` object allowing bundling SSL configuration options, certificates and private keys into a single (potentially long-lived) structure. (Contributed by Giampaolo Rodolà; :issue:`8807`.) Multi-threading =============== * The mechanism for serializing execution of concurrently running Python threads (generally known as the GIL or Global Interpreter Lock) has been rewritten. Among the objectives were more predictable switching intervals and reduced overhead due to lock contention and the number of ensuing system calls. The notion of a "check interval" to allow thread switches has been abandoned and replaced by an absolute duration expressed in seconds. This parameter is tunable through :func:`sys.setswitchinterval()`. It currently defaults to 5 milliseconds. Additional details about the implementation can be read from a `python-dev mailing-list message `_ (however, "priority requests" as exposed in this message have not been kept for inclusion). (Contributed by Antoine Pitrou.) * Recursive locks (created with the :func:`threading.RLock` API) now benefit from a C implementation which makes them as fast as regular locks, and between 10x and 15x faster than their previous pure Python implementation. (Contributed by Antoine Pitrou; :issue:`3001`.) * Regular and recursive locks now accept an optional *timeout* argument to their ``acquire`` method. (Contributed by Antoine Pitrou; :issue:`7316`) Similarly, :meth:`threading.Semaphore.acquire` also gains a *timeout* argument. (Contributed by Torsten Landschoff; :issue:`850728`.) Optimizations ============= Major performance enhancements have been added: * Stub Filenames and unicode ===================== The filesystem encoding can be specified by setting the :envvar:`PYTHONFSENCODING` environment variable before running the interpreter. The value is an encoding name, e.g. ``iso-8859-1``. This variable is not available (ignored) on Windows and Mac OS X: the filesystem encoding is pinned to ``'mbcs'`` on Windows and ``'utf-8'`` on Mac OS X. The :mod:`os` module has two new functions: :func:`os.fsencode` and :func:`os.fsdecode`. IDLE ==== * Stub Build and C API Changes ======================= Changes to Python's build process and to the C API include: * Stub Porting to Python 3.2 ===================== This section lists previously described changes and other bugfixes that may require changes to your code: * bytearray objects cannot be used anymore as filenames: convert them to bytes * PyArg_Parse*() functions: * "t#" format has been removed: use "s#" or "s*" instead * "w" and "w#" formats has been removed: use "w*" instead * The :ctype:`PyCObject` type, deprecated in 3.1, has been removed. To wrap opaque C pointers in Python objects, the :ctype:`PyCapsule` API should be used instead; the new type has a well defined interface for passing typing safety information and a less complicated signature for calling a destructor.