cpython/Doc/whatsnew/whatsnew25.tex

\documentclass{howto}
\usepackage{distutils}
% $Id$


\title{What's New in Python 2.5}
\release{0.0}
\author{A.M. Kuchling}
\authoraddress{\email{amk@amk.ca}}

\begin{document}
\maketitle
\tableofcontents

This article explains the new features in Python 2.5.  No release date
for Python 2.5 has been set; it will probably be released in the
autumn of 2006.

% XXX Compare with previous release in 2 - 3 sentences here.

This article doesn't attempt to provide a complete specification of
the new features, but instead provides a convenient overview.  For
full details, you should refer to the documentation for Python 2.5.
% XXX add hyperlink when the documentation becomes available online.
If you want to understand the complete implementation and design
rationale, refer to the PEP for a particular new feature.


%======================================================================
\section{PEP 308: Conditional Expressions}

For a long time, people have been requesting a way to write
conditional expressions, expressions that return value A or value B
depending on whether a Boolean value is true or false.  A conditional
expression lets you write a single assignment statement that has the
same effect as the following:

\begin{verbatim}
if condition:
    x = true_value
else:
    x = false_value
\end{verbatim}

There have been endless tedious discussions of syntax on both
python-dev and comp.lang.python, and even a vote that found the
majority of voters wanted some way to write conditional expressions,
but there was no syntax that was clearly preferred by a majority.
Candidates include C's \code{cond ? true_v : false_v},
\code{if cond then true_v else false_v}, and 16 other variations.

GvR eventually chose a surprising syntax:

\begin{verbatim}
x = true_value if condition else false_value
\end{verbatim}

Evaluation is still lazy as in existing Boolean expression, so the
evaluation jumps around a bit.  The \var{condition} expression is
evaluated first, and the \var{true_value} expression is evaluated only
if the condition was true.  Similarly, the \var{false_value}
expression is only evaluated when the condition is false.

This syntax may seem strange and backwards; why does the condition go
in the \emph{middle} of the expression, and not in the front as in C's
\code{c ? x : y}?  The decision was checked by applying the new syntax
to the modules in the standard library and seeing how the resulting
code read.  In many cases where a conditional expression is used, one
value seems to be the 'common case' and one value is an 'exceptional
case', used only on rarer occasions when the condition isn't met.  The
conditional syntax makes this pattern a bit more obvious:

\begin{verbatim}
contents = ((doc + '\n') if doc else '')
\end{verbatim}

I read the above statement as meaning ``here \var{contents} is 
usually assigned a value of \code{doc+'\e n'}; sometimes 
\var{doc} is empty, in which special case an empty string is returned.''  
I doubt I will use conditional expressions very often where there 
isn't a clear common and uncommon case.

There was some discussion of whether the language should require
surrounding conditional expressions with parentheses.  The decision
was made to \emph{not} require parentheses in the Python language's
grammar, but as a matter of style I think you should always use them.
Consider these two statements:

\begin{verbatim}
# First version -- no parens
level = 1 if logging else 0

# Second version -- with parens
level = (1 if logging else 0)
\end{verbatim}

In the first version, I think a reader's eye might group the statement
into 'level = 1', 'if logging', 'else 0', and think that the condition
decides whether the assignment to \var{level} is performed.  The
second version reads better, in my opinion, because it makes it clear
that the assignment is always performed and the choice is being made
between two values.

Another reason for including the brackets: a few odd combinations of
list comprehensions and lambdas could look like incorrect conditional
expressions. See \pep{308} for some examples.  If you put parentheses
around your conditional expressions, you won't run into this case.


\begin{seealso}

\seepep{308}{Conditional Expressions}{PEP written by
Guido van Rossum and Raymond D. Hettinger; implemented by Thomas
Wouters.}

\end{seealso}


%======================================================================
\section{PEP 309: Partial Function Application}

The \module{functional} module is intended to contain tools for
functional-style programming.  Currently it only contains a
\class{partial()} function, but new functions will probably be added
in future versions of Python.

For programs written in a functional style, it can be useful to
construct variants of existing functions that have some of the
parameters filled in.  Consider a Python function \code{f(a, b, c)};
you could create a new function \code{g(b, c)} that was equivalent to
\code{f(1, b, c)}.  This is called ``partial function application'',
and is provided by the \class{partial} class in the new
\module{functional} module.

The constructor for \class{partial} takes the arguments
\code{(\var{function}, \var{arg1}, \var{arg2}, ...
\var{kwarg1}=\var{value1}, \var{kwarg2}=\var{value2})}.  The resulting
object is callable, so you can just call it to invoke \var{function}
with the filled-in arguments.

Here's a small but realistic example:

\begin{verbatim}
import functional

def log (message, subsystem):
    "Write the contents of 'message' to the specified subsystem."
    print '%s: %s' % (subsystem, message)
    ...

server_log = functional.partial(log, subsystem='server')
server_log('Unable to open socket')
\end{verbatim}

Here's another example, from a program that uses PyGTk.  Here a
context-sensitive pop-up menu is being constructed dynamically.  The
callback provided for the menu option is a partially applied version
of the \method{open_item()} method, where the first argument has been
provided.

\begin{verbatim}
...
class Application:
    def open_item(self, path):
       ...
    def init (self):
        open_func = functional.partial(self.open_item, item_path)
        popup_menu.append( ("Open", open_func, 1) )
\end{verbatim}


\begin{seealso}

\seepep{309}{Partial Function Application}{PEP proposed and written by
Peter Harris; implemented by Hye-Shik Chang, with adaptations by
Raymond Hettinger.}

\end{seealso}


%======================================================================
\section{PEP 314: Metadata for Python Software Packages v1.1}

Some simple dependency support was added to Distutils.  The
\function{setup()} function now has \code{requires}, \code{provides},
and \code{obsoletes} keyword parameters.  When you build a source
distribution using the \code{sdist} command, the dependency
information will be recorded in the \file{PKG-INFO} file.

Another new keyword parameter is \code{download_url}, which should be
set to a URL for the package's source code.  This means it's now
possible to look up an entry in the package index, determine the
dependencies for a package, and download the required packages.

% XXX put example here
 
\begin{seealso}

\seepep{314}{Metadata for Python Software Packages v1.1}{PEP proposed
and written by A.M. Kuchling, Richard Jones, and Fred Drake; 
implemented by Richard Jones and Fred Drake.}

\end{seealso}


%======================================================================
\section{PEP 328: Absolute and Relative Imports}

% XXX write this


%======================================================================
\section{PEP 338: Executing Modules as Scripts}

% XXX write this


%======================================================================
\section{PEP 341: Unified try/except/finally}

% XXX write this


%======================================================================
\section{PEP 342: New Generator Features}

As introduced in Python 2.3, generators only produce output; once a
generator's code is invoked to create an iterator, there's no way to
pass any new information into the function when its execution is
resumed.  Hackish solutions to this include making the generator's
code look at a global variable and then changing the global variable's
value, or passing in some mutable object that callers then modify.
Python 2.5 adds the ability to pass values \emph{into} a generator.

To refresh your memory of basic generators, here's a simple example:

\begin{verbatim}
def counter (maximum):
    i = 0
    while i < maximum:
        yield i
	i += 1
\end{verbatim}

When you call \code{counter(10)}, the result is an iterator that
returns the values from 0 up to 9.  On encountering the
\keyword{yield} statement, the iterator returns the provided value and
suspends the function's execution, preserving the local variables.
Execution resumes on the following call to the iterator's 
\method{next()} method, picking up after the \keyword{yield} statement.

In Python 2.3, \keyword{yield} was a statement; it didn't return any
value.  In 2.5, \keyword{yield} is now an expression, returning a
value that can be assigned to a variable or otherwise operated on:

\begin{verbatim}
val = (yield i)
\end{verbatim}

I recommend that you always put parentheses around a \keyword{yield}
expression when you're doing something with the returned value, as in
the above example.  The parentheses aren't always necessary, but it's
easier to always add them instead of having to remember when they're
needed.\footnote{The exact rules are that a \keyword{yield}-expression must
always be parenthesized except when it occurs at the top-level
expression on the right-hand side of an assignment, meaning you can
write \code{val = yield i} but have to use parentheses when there's an
operation, as in \code{val = (yield i) + 12}.}

Values are sent into a generator by calling its
\method{send(\var{value})} method.  The generator's code is then
resumed and the \keyword{yield} expression returns the specified
\var{value}.  If the regular \method{next()} method is called, the
\keyword{yield} returns \constant{None}.

Here's the previous example, modified to allow changing the value of
the internal counter.

\begin{verbatim}
def counter (maximum):
    i = 0
    while i < maximum:
        val = (yield i)
	# If value provided, change counter
        if val is not None:
            i = val
	else:
  	    i += 1
\end{verbatim}

And here's an example of changing the counter:

\begin{verbatim}
>>> it = counter(10)
>>> print it.next()
0
>>> print it.next()
1
>>> print it.send(8)
8
>>> print it.next()
9
>>> print it.next()
Traceback (most recent call last):
  File ``t.py'', line 15, in ?
    print it.next()
StopIteration
\end{verbatim}

Because \keyword{yield} will often be returning \constant{None}, you
should always check for this case.  Don't just use its value in
expressions unless you're sure that the \method{send()} method
will be the only method used resume your generator function.

In addition to \method{send()}, there are two other new methods on
generators:

\begin{itemize}

  \item \method{throw(\var{type}, \var{value}=None,
  \var{traceback}=None)} is used to raise an exception inside the
  generator; the exception is raised by the \keyword{yield} expression
  where the generator's execution is paused.

  \item \method{close()} raises a new \exception{GeneratorExit}
  exception inside the generator to terminate the iteration.  
  On receiving this
  exception, the generator's code must either raise
  \exception{GeneratorExit} or \exception{StopIteration}; catching the 
  exception and doing anything else is illegal and will trigger
  a \exception{RuntimeError}.  \method{close()} will also be called by 
  Python's garbage collection when the generator is garbage-collected.

  If you need to run cleanup code in case of a \exception{GeneratorExit},
  I suggest using a \code{try: ... finally:} suite instead of 
  catching \exception{GeneratorExit}.

\end{itemize}

The cumulative effect of these changes is to turn generators from
one-way producers of information into both producers and consumers.

Generators also become \emph{coroutines}, a more generalized form of
subroutines.  Subroutines are entered at one point and exited at
another point (the top of the function, and a \keyword{return
statement}), but coroutines can be entered, exited, and resumed at
many different points (the \keyword{yield} statements).


\begin{seealso}

\seepep{342}{Coroutines via Enhanced Generators}{PEP written by 
Guido van Rossum and Phillip J. Eby;
implemented by Phillip J. Eby.  Includes examples of 
some fancier uses of generators as coroutines.}

\seeurl{http://en.wikipedia.org/wiki/Coroutine}{The Wikipedia entry for 
coroutines.}

\seeurl{http://www.sidhe.org/\~{}dan/blog/archives/000178.html}{An
explanation of coroutines from a Perl point of view, written by Dan
Sugalski.}

\end{seealso}


%======================================================================
\section{PEP 343: The 'with' statement}

% XXX write this


%======================================================================
\section{PEP 352: Exceptions as New-Style Classes}

Exception classes can now be new-style classes, not just classic classes,
and the built-in \exception{Exception} class and all

The inheritance hierarchy for exceptions has been rearranged a bit.
In 2.5, the inheritance relationships are:

\begin{verbatim}
BaseException       # New in Python 2.5
|- KeyboardInterrupt
|- SystemExit
|- Exception
   |- (all other current built-in exceptions)
\end{verbatim}

This rearrangement was done because people often want to catch all
exceptions that indicate program errors.  \exception{KeyboardInterrupt} and
\exception{SystemExit} aren't errors, though, and usually represent an explicit
action such as the user hitting Control-C or code calling
\function{sys.exit()}.  A bare \code{except:} will catch all exceptions,
so you commonly need to list \exception{KeyboardInterrupt} and
\exception{SystemExit} in order to re-raise them.  The usual pattern is:

\begin{verbatim}
try:
    ...
except (KeyboardInterrupt, SystemExit):
    raise
except: 
    # Log error...  
    # Continue running program...
\end{verbatim}

In Python 2.5, you can now write \code{except Exception} to achieve
the same result, catching all the exceptions that usually indicate errors 
but leaving \exception{KeyboardInterrupt} and
\exception{SystemExit} alone.  As in previous versions,
a bare \code{except:} still catches all exceptions.

The goal for Python 3.0 is to require any class raised as an exception
to derive from \exception{BaseException} or some descendant of
\exception{BaseException}, and future releases in the
Python 2.x series may begin to enforce this constraint.  Therefore, I
suggest you begin making all your exception classes derive from
\exception{Exception} now.  It's been suggested that the bare
\code{except:} form should be removed in Python 3.0, but Guido van~Rossum
hasn't decided whether to do this or not.

Raising of strings as exceptions, as in the statement \code{raise
"Error occurred"}, is deprecated in Python 2.5 and will trigger a
warning.  The aim is to be able to remove the string-exception feature
in a few releases.


\begin{seealso}

\seepep{352}{}{PEP written by 
Brett Cannon and Guido van Rossum; implemented by Brett Cannon.}

\end{seealso}


%======================================================================
\section{PEP 357: The '__index__' method}

% XXX write this


%======================================================================
\section{Other Language Changes}

Here are all of the changes that Python 2.5 makes to the core Python
language.

\begin{itemize}

\item The \function{min()} and \function{max()} built-in functions
gained a \code{key} keyword argument analogous to the \code{key}
argument for \method{sort()}.  This argument supplies a function
that takes a single argument and is called for every value in the list; 
\function{min()}/\function{max()} will return the element with the 
smallest/largest return value from this function.
For example, to find the longest string in a list, you can do:

\begin{verbatim}
L = ['medium', 'longest', 'short']
# Prints 'longest'
print max(L, key=len)		   
# Prints 'short', because lexicographically 'short' has the largest value
print max(L)         
\end{verbatim}

(Contributed by Steven Bethard and Raymond Hettinger.)

\item Two new built-in functions, \function{any()} and
\function{all()}, evaluate whether an iterator contains any true or
false values.  \function{any()} returns \constant{True} if any value
returned by the iterator is true; otherwise it will return
\constant{False}.  \function{all()} returns \constant{True} only if
all of the values returned by the iterator evaluate as being true.

% XXX who added?


\item The list of base classes in a class definition can now be empty.  
As an example, this is now legal:

\begin{verbatim}
class C():
    pass
\end{verbatim}
(Implemented by Brett Cannon.)

\end{itemize}


%======================================================================
\subsection{Optimizations}

\begin{itemize}

\item When they were introduced 
in Python 2.4, the built-in \class{set} and \class{frozenset} types
were built on top of Python's dictionary type.  
In 2.5 the internal data structure has been customized for implementing sets,
and as a result sets will use a third less memory and are somewhat faster.
(Implemented by Raymond Hettinger.)

\end{itemize}

The net result of the 2.5 optimizations is that Python 2.5 runs the
pystone benchmark around XX\% faster than Python 2.4.


%======================================================================
\section{New, Improved, and Deprecated Modules}

As usual, Python's standard library received a number of enhancements and
bug fixes.  Here's a partial list of the most notable changes, sorted
alphabetically by module name. Consult the
\file{Misc/NEWS} file in the source tree for a more
complete list of changes, or look through the SVN logs for all the
details.

\begin{itemize}

% ctypes added

% collections.deque now has .remove()

% the cPickle module no longer accepts the deprecated None option in the
% args tuple returned by __reduce__().

% csv module improvements

% datetime.datetime() now has a strptime class method which can be used to
% create datetime object using a string and format.

\item A new \module{hashlib} module has been added to replace the
\module{md5} and \module{sha} modules.  \module{hashlib} adds support
for additional secure hashes (SHA-224, SHA-256, SHA-384, and SHA-512).
When available, the module uses OpenSSL for fast platform optimized
implementations of algorithms.  The old \module{md5} and \module{sha}
modules still exist as wrappers around hashlib to preserve backwards
compatibility.  (Contributed by Gregory P. Smith.)

\item The \function{nsmallest()} and 
\function{nlargest()} functions in the \module{heapq} module 
now support a \code{key} keyword argument similar to the one
provided by the \function{min()}/\function{max()} functions
and the \method{sort()} methods.  For example:
Example:

\begin{verbatim}
>>> import heapq
>>> L = ["short", 'medium', 'longest', 'longer still']
>>> heapq.nsmallest(2, L)  # Return two lowest elements, lexicographically
['longer still', 'longest']
>>> heapq.nsmallest(2, L, key=len)   # Return two shortest elements
['short', 'medium']
\end{verbatim}

(Contributed by Raymond Hettinger.)

\item The \function{itertools.islice()} function now accepts
\code{None} for the start and step arguments.  This makes it more
compatible with the attributes of slice objects, so that you can now write
the following:

\begin{verbatim}
s = slice(5)     # Create slice object
itertools.islice(iterable, s.start, s.stop, s.step)
\end{verbatim}

(Contributed by Raymond Hettinger.)

\item The \module{operator} module's \function{itemgetter()} 
and \function{attrgetter()} functions now support multiple fields.  
A call such as \code{operator.attrgetter('a', 'b')}
will return a function 
that retrieves the \member{a} and \member{b} attributes.  Combining 
this new feature with the \method{sort()} method's \code{key} parameter 
lets you easily sort lists using multiple fields.

% XXX who added?


\item The \module{os} module underwent a number of changes.  The
\member{stat_float_times} variable now defaults to true, meaning that
\function{os.stat()} will now return time values as floats.  (This
doesn't necessarily mean that \function{os.stat()} will return times
that are precise to fractions of a second; not all systems support
such precision.)

Constants named \member{os.SEEK_SET}, \member{os.SEEK_CUR}, and
\member{os.SEEK_END} have been added; these are the parameters to the
\function{os.lseek()} function.  Two new constants for locking are
\member{os.O_SHLOCK} and \member{os.O_EXLOCK}.

On FreeBSD, the \function{os.stat()} function now returns 
times with nanosecond resolution, and the returned object
now has \member{st_gen} and \member{st_birthtime}.
The \member{st_flags} member is also available, if the platform supports it.
% XXX patch 1180695, 1212117

\item The old \module{regex} and \module{regsub} modules, which have been 
deprecated ever since Python 2.0, have finally been deleted.  
Other deleted modules: \module{statcache}, \module{tzparse},
\module{whrandom}.

\item The \file{lib-old} directory,
which includes ancient modules such as \module{dircmp} and
\module{ni}, was also deleted.  \file{lib-old} wasn't on the default
\code{sys.path}, so unless your programs explicitly added the directory to 
\code{sys.path}, this removal shouldn't affect your code.

\item The \module{socket} module now supports \constant{AF_NETLINK}
sockets on Linux, thanks to a patch from Philippe Biondi.  
Netlink sockets are a Linux-specific mechanism for communications
between a user-space process and kernel code; an introductory 
article about them is at \url{http://www.linuxjournal.com/article/7356}.
In Python code, netlink addresses are represented as a tuple of 2 integers, 
\code{(\var{pid}, \var{group_mask})}.

\item New module: \module{spwd} provides functions for accessing the
shadow password database on systems that support it.  
% XXX give example

\item The \class{TarFile} class in the \module{tarfile} module now has
an \method{extractall()} method that extracts all members from the
archive into the current working directory.  It's also possible to set
a different directory as the extraction target, and to unpack only a
subset of the archive's members.  

A tarfile's compression can be autodetected by 
using the mode \code{'r|*'}.
% patch 918101
(Contributed by Lars Gust\"abel.)

\item The \module{unicodedata} module has been updated to use version 4.1.0
of the Unicode character database.  Version 3.2.0 is required 
by some specifications, so it's still available as 
\member{unicodedata.db_3_2_0}.

\item A new package \module{xml.etree} has been added, which contains
a subset of the ElementTree XML library.  Available modules are
\module{ElementTree}, \module{ElementPath}, and
\module{ElementInclude}, from ElementTree 1.2.6. (Contributed by
Fredrik Lundh.)

\item The \module{xmlrpclib} module now supports returning 
      \class{datetime} objects for the XML-RPC date type.  Supply 
      \code{use_datetime=True} to the \function{loads()} function
      or the \class{Unmarshaller} class to enable this feature.
% XXX patch 1120353


\end{itemize}



%======================================================================
% whole new modules get described in \subsections here

% XXX new distutils features: upload

% XXX should hashlib perhaps be described here instead?
% XXX should xml.etree perhaps be described here instead?



% ======================================================================
\section{Build and C API Changes}

Changes to Python's build process and to the C API include:

\begin{itemize}

\item The design of the bytecode compiler has changed a great deal, no
longer generating bytecode by traversing the parse tree.  Instead
the parse tree is converted to an abstract syntax tree (or AST), and it is 
the abstract syntax tree that's traversed to produce the bytecode.

No documentation has been written for the AST code yet.  To start
learning about it, read the definition of the various AST nodes in
\file{Parser/Python.asdl}.  A Python script reads this file and
generates a set of C structure definitions in
\file{Include/Python-ast.h}.  The \cfunction{PyParser_ASTFromString()}
and \cfunction{PyParser_ASTFromFile()}, defined in
\file{Include/pythonrun.h}, take Python source as input and return the
root of an AST representing the contents.  This AST can then be turned
into a code object by \cfunction{PyAST_Compile()}.  For more
information, read the source code, and then ask questions on
python-dev.

% List of names taken from Jeremy's python-dev post at 
% http://mail.python.org/pipermail/python-dev/2005-October/057500.html
The AST code was developed under Jeremy Hylton's management, and
implemented by (in alphabetical order) Brett Cannon, Nick Coghlan,
Grant Edwards, John Ehresman, Kurt Kaiser, Neal Norwitz, Tim Peters,
Armin Rigo, and Neil Schemenauer, plus the participants in a number of
AST sprints at conferences such as PyCon.
 
\item The built-in set types now have an official C API.  Call
\cfunction{PySet_New()} and \cfunction{PyFrozenSet_New()} to create a
new set, \cfunction{PySet_Add()} and \cfunction{PySet_Discard()} to
add and remove elements, and \cfunction{PySet_Contains} and
\cfunction{PySet_Size} to examine the set's state.

\item The \cfunction{PyRange_New()} function was removed.  It was
never documented, never used in the core code, and had dangerously lax
error checking.

\end{itemize}


%======================================================================
\subsection{Port-Specific Changes}

Platform-specific changes go here.


%======================================================================
\section{Other Changes and Fixes \label{section-other}}

As usual, there were a bunch of other improvements and bugfixes
scattered throughout the source tree.  A search through the SVN change
logs finds there were XXX patches applied and YYY bugs fixed between
Python 2.4 and 2.5.  Both figures are likely to be underestimates.

Some of the more notable changes are:

\begin{itemize}

\item Evan Jones's patch to obmalloc, first described in a talk
at PyCon DC 2005, was applied.  Python 2.4 allocated small objects in
256K-sized arenas, but never freed arenas.  With this patch, Python
will free arenas when they're empty.  The net effect is that on some
platforms, when you allocate many objects, Python's memory usage may
actually drop when you delete them, and the memory may be returned to
the operating system.  (Implemented by Evan Jones, and reworked by Tim
Peters.)

\end{itemize}


%======================================================================
\section{Porting to Python 2.5}

This section lists previously described changes that may require
changes to your code:

\begin{itemize}

% the pickle module no longer uses the deprecated bin parameter.

\end{itemize}


%======================================================================
\section{Acknowledgements \label{acks}}

The author would like to thank the following people for offering
suggestions, corrections and assistance with various drafts of this
article: .

\end{document}