\documentclass{howto}
\usepackage{distutils}
% $Id$

\title{What's New in Python 2.4}
\release{0.0}
\author{A.M.\ Kuchling}
\authoraddress{
	\strong{Python Software Foundation}\\
	Email: \email{amk@amk.ca}
}

\begin{document}
\maketitle
\tableofcontents

This article explains the new features in Python 2.4.  No release date
for Python 2.4 has been set; expect that this will happen mid-2004.

While Python 2.3 was primarily a library development release, Python
2.4 may extend the core language and interpreter in
as-yet-undetermined ways.

This article doesn't attempt to provide a complete specification of
the new features, but instead provides a convenient overview.  For
full details, you should refer to the documentation for Python 2.4,
such as the \citetitle[../lib/lib.html]{Python Library Reference} and
the \citetitle[../ref/ref.html]{Python Reference Manual}.  
If you want to understand the complete implementation and design
rationale, refer to the PEP for a particular new feature.


%======================================================================
\section{PEP 218: Built-In Set Objects}

Two new built-in types, \function{set(\var{iterable})} and
\function{frozenset(\var{iterable})} provide high speed data types for
membership testing, for eliminating duplicates from sequences, and
for mathematical operations like unions, intersections, differences,
and symmetric differences.  

\begin{verbatim}
>>> a = set('abracadabra')              # form a set from a string
>>> 'z' in a                            # fast membership testing
False
>>> a                                   # unique letters in a
set(['a', 'r', 'b', 'c', 'd'])
>>> ''.join(a)                          # convert back into a string
'arbcd'

>>> b = set('alacazam')                 # form a second set
>>> a - b                               # letters in a but not in b
set(['r', 'd', 'b'])
>>> a | b                               # letters in either a or b
set(['a', 'c', 'r', 'd', 'b', 'm', 'z', 'l'])
>>> a & b                               # letters in both a and b
set(['a', 'c'])
>>> a ^ b                               # letters in a or b but not both
set(['r', 'd', 'b', 'm', 'z', 'l'])

>>> a.add('z')                          # add a new element
>>> a.update('wxy')                     # add multiple new elements
>>> a
set(['a', 'c', 'b', 'd', 'r', 'w', 'y', 'x', 'z'])       
>>> a.remove('x')                       # take one element out
>>> a
set(['a', 'c', 'b', 'd', 'r', 'w', 'y', 'z'])       
\end{verbatim}

The type \function{frozenset()} is an immutable version of \function{set()}.
Since it is immutable and hashable, it may be used as a dictionary key or
as a member of another set.  Accordingly, it does not have methods
like \method{add()} and \method{remove()} which could alter its contents.

% XXX what happens to the sets module?
% The current thinking is that the sets module will be left alone.
% That way, existing code will continue to run without alteration.
% Also, the module provides an autoconversion feature not supported by set()
% and frozenset().

\begin{seealso}
\seepep{218}{Adding a Built-In Set Object Type}{Originally proposed by
Greg Wilson and ultimately implemented by Raymond Hettinger.}
\end{seealso}

%======================================================================
\section{PEP 237: Unifying Long Integers and Integers}

XXX write this.

%======================================================================
\section{PEP 229: Generator Expressions}

Generator expressions create in-line generators using a syntax similar
to list comprehensions but with parenthesis instead of the surrounding
brackets.

Genexps allow simple generators to be constructed without a separate function
definition.  Writing:
       
\begin{verbatim}
       g = (tgtexp  for var1 in exp1  for var2 in exp2 if exp3)
\end{verbatim}
       
is equivalent to:

\begin{verbatim}
    def _generator(exp):
        for var1 in exp:
            for var2 in exp2:
                if exp3:
                    yield tgtexp
    g = _generator(exp1)
    del _generator
\end{verbatim}

The advantage over full generator definitions is in economy of
expression.  Their advantage over list comprehensions is in saving
memory by creating data only when it is needed rather than forming
a whole list is memory all at once.  Applications using memory
friendly generator expressions may scale-up to high volumes of data
more readily than with list comprehensions.

Generator expressions are intended to be used inside functions
such as \function{sum()}, \function{min()}, \function{set()}, and
\function{dict()}.  These functions consume their data all at once
and would not benefit from having a full list instead of a generator
as an input:

\begin{verbatim}
>>> sum(i*i for i in range(10))
285

>>> sorted(set(i*i for i in xrange(-20, 20) if i%2==1)) # odd squares
[1, 9, 25, 49, 81, 121, 169, 225, 289, 361]

>>> from itertools import izip
>>> xvec = [10, 20, 30]
>>> yvec = [7, 5, 3]
>>> sum(x*y for x,y in izip(xvec, yvec))                # dot product
260

>>> from math import pi, sin
>>> sine_table = dict((x, sin(x*pi/180)) for x in xrange(0, 91))

>>> unique_words = set(word  for line in page  for word in line.split())

>>> valedictorian = max((student.gpa, student.name) for student in graduates)

\end{verbatim}     

These examples show the intended use for generator expressions
in situations where the values get consumed immediately after the
generator is created.  In these situations, they operate like
memory efficient versions of list comprehensions.

For more complex uses of generators, it is strongly recommended that
the traditional full generator definitions be used instead.  In a
generator expression, the first for-loop expression is evaluated
as soon as the expression is defined while the other expressions do
not get evaluated until the generator is run.  This nuance is never
an issue when the generator is used immediately.  If it is not used
right away, then it is better to write a full generator definition
which more clearly reveals when the expressions are evaluated and is
more obvious about the visibility and lifetime of its looping variables.

\begin{seealso}
\seepep{289}{Generator Expressions}{Proposed by Raymond Hettinger and
implemented by Jiwon Seo with early efforts steered by Hye-Shik Chang.}
\end{seealso}

%======================================================================
\section{PEP 322: Reverse Iteration}

A new built-in function, \function{reversed(\var{seq})}, takes a sequence
and returns an iterator that returns the elements of the sequence 
in reverse order.  

\begin{verbatim}
>>> for i in reversed(xrange(1,4)):
...    print i
... 
3
2
1
\end{verbatim}

Compared to extended slicing, \code{range(1,4)[::-1]}, \function{reversed()}
is easier to read, runs faster, and uses substantially less memory.

Note that \function{reversed()} only accepts sequences, not arbitrary
iterators.  If you want to reverse an iterator, first convert it to 
a list with \function{list()}.

\begin{verbatim}
>>> input= open('/etc/passwd', 'r')
>>> for line in reversed(list(input)):
...   print line
... 
root:*:0:0:System Administrator:/var/root:/bin/tcsh
  ...
\end{verbatim}

\begin{seealso}
\seepep{322}{Reverse Iteration}{Written and implemented by Raymond Hettinger.}

\end{seealso}


%======================================================================
\section{Other Language Changes}

Here are all of the changes that Python 2.4 makes to the core Python
language.

\begin{itemize}

\item The \method{dict.update()} method now accepts the same
argument forms as the \class{dict} constructor.  This includes any
mapping, any iterable of key/value pairs, and/or keyword arguments.       

\item The string methods, \method{ljust()}, \method{rjust()}, and
\method{center()} now take an optional argument for specifying a
fill character other than a space.

\item Strings also gained an \method{rsplit()} method that
works like the \method{split()} method but splits from the end of
the string.  

\begin{verbatim}
>>> 'www.python.org'.split('.', 1)
['www', 'python.org']
'www.python.org'.rsplit('.', 1)
['www.python', 'org']        
\end{verbatim}      

\item The \method{sort()} method of lists gained three keyword
arguments, \var{cmp}, \var{key}, and \var{reverse}.  These arguments
make some common usages of \method{sort()} simpler.  All are optional.

\var{cmp} is the same as the previous single argument to
\method{sort()}; if provided, the value should be a comparison
function that takes two arguments and returns -1, 0, or +1 depending
on how the arguments compare.  

\var{key} should be a single-argument function that takes a list
element and returns a comparison key for the element.  The list is
then sorted using the comparison keys.  The following example sorts a
list case-insensitively:

\begin{verbatim}
>>> L = ['A', 'b', 'c', 'D']
>>> L.sort()                 # Case-sensitive sort
>>> L
['A', 'D', 'b', 'c']
>>> L.sort(key=lambda x: x.lower())
>>> L
['A', 'b', 'c', 'D']
>>> L.sort(cmp=lambda x,y: cmp(x.lower(), y.lower()))
>>> L
['A', 'b', 'c', 'D']
\end{verbatim}

The last example, which uses the \var{cmp} parameter, is the old way
to perform a case-insensitive sort.  It works but is slower than
using a \var{key} parameter.  Using \var{key} results in calling the
\method{lower()} method once for each element in the list while using
\var{cmp} will call the method twice for each comparison.

For simple key functions and comparison functions, it is often
possible to avoid a \keyword{lambda} expression by using an unbound
method instead.  For example, the above case-insensitive sort is best
coded as:

\begin{verbatim}
>>> L.sort(key=str.lower)
>>> L
['A', 'b', 'c', 'D']
\end{verbatim}       

The \var{reverse} parameter should have a Boolean value.  If the value is
\constant{True}, the list will be sorted into reverse order.  Instead
of \code{L.sort(lambda x,y: cmp(y.score, x.score))}, you can now write:
\code{L.sort(key = lambda x: x.score, reverse=True)}.

The results of sorting are now guaranteed to be stable.  This means
that two entries with equal keys will be returned in the same order as
they were input.  For example, you can sort a list of people by name,
and then sort the list by age, resulting in a list sorted by age where
people with the same age are in name-sorted order.

\item There is a new built-in function
\function{sorted(\var{iterable})} that works like the in-place
\method{list.sort()} method but has been made suitable for use in
expressions.  The differences are:
  \begin{itemize}
  \item the input may be any iterable;
  \item a newly formed copy is sorted, leaving the original intact; and
  \item the expression returns the new sorted copy
  \end{itemize}

\begin{verbatim}
>>> L = [9,7,8,3,2,4,1,6,5]
>>> [10+i for i in sorted(L)]       # usable in a list comprehension
[11, 12, 13, 14, 15, 16, 17, 18, 19]
>>> L = [9,7,8,3,2,4,1,6,5]         # original is left unchanged
[9,7,8,3,2,4,1,6,5]   

>>> sorted('Monte Python')          # any iterable may be an input
[' ', 'M', 'P', 'e', 'h', 'n', 'n', 'o', 'o', 't', 't', 'y']

>>> # List the contents of a dict sorted by key values
>>> colormap = dict(red=1, blue=2, green=3, black=4, yellow=5)
>>> for k, v in sorted(colormap.iteritems()):
...     print k, v
...
black 4
blue 2
green 3
red 1
yellow 5

\end{verbatim}

\item The \function{zip()} built-in function and \function{itertools.izip()}
  now return an empty list instead of raising a \exception{TypeError}
  exception if called with no arguments.  This makes them more
  suitable for use with variable length argument lists:

\begin{verbatim}
>>> def transpose(array):
...    return zip(*array)
...
>>> transpose([(1,2,3), (4,5,6)])
[(1, 4), (2, 5), (3, 6)]
>>> transpose([])
[]
\end{verbatim}

\end{itemize}


%======================================================================
\subsection{Optimizations}

\begin{itemize}

\item The inner loops for \class{list} and \class{tuple} slicing
 were optimized and now run about one-third faster.  The inner
 loops were also optimized for \class{dict} with performance
 boosts to \method{keys()}, \method{values()}, \method{items()},
\method{iterkeys()}, \method{itervalues()}, and \method{iteritems()}.

\item The machinery for growing and shrinking lists was optimized
 for speed and for space efficiency.  Small lists (under eight elements)
 never over-allocate by more than three elements.  Large lists do not
 over-allocate by more than 1/8th.  Appending and popping from lists
 now runs faster due to more efficient code paths and less frequent
 use of the underlying system realloc().  List comprehensions also
 benefit.  The amount of improvement varies between systems and shows
 the greatest improvement on systems with poor realloc() implementations.
 \method{list.extend()} was also optimized and no longer converts its
 argument into a temporary list prior to extending the base list.

\item \function{list()}, \function{tuple()}, \function{map()},
  \function{filter()}, and \function{zip()} now run several times
  faster with non-sequence arguments that supply a \method{__len__()}
  method.  Previously, the pre-sizing optimization only applied to
  sequence arguments.

\item The methods \method{list.__getitem__()},
  \method{dict.__getitem__()}, and \method{dict.__contains__()} are
  are now implemented as \class{method_descriptor} objects rather
  than \class{wrapper_descriptor} objects.  This form of optimized
  access doubles their performance and makes them more suitable for
  use as arguments to functionals:
  \samp{map(mydict.__getitem__, keylist)}.

\item Added an newcode opcode, \code{LIST_APPEND}, that simplifies
  the generated bytecode for list comprehensions and speeds them up
  by about a third.

\end{itemize}

The net result of the 2.4 optimizations is that Python 2.4 runs the
pystone benchmark around XX\% faster than Python 2.3 and YY\% faster
than Python 2.2.


%======================================================================
\section{New, Improved, and Deprecated Modules}

As usual, Python's standard library received a number of enhancements and
bug fixes.  Here's a partial list of the most notable changes, sorted
alphabetically by module name. Consult the
\file{Misc/NEWS} file in the source tree for a more
complete list of changes, or look through the CVS logs for all the
details.

\begin{itemize}

\item The \module{curses} modules now supports the ncurses extension 
   \function{use_default_colors()}.   On platforms where the terminal 
   supports transparency, this makes it possible to use a transparent background.
   (Contributed by J\"org Lehmann.)

\item The \module{bisect} module now has an underlying C implementation
   for improved performance.
   (Contributed by Dmitry Vasiliev.)

\item The CJKCodecs collections of East Asian codecs, maintained
by Hye-Shik Chang, was integrated into 2.4.  
The new encodings are:

\begin{itemize}
 \item Chinese (PRC): gb2312, gbk, gb18030, hz
 \item Chinese (ROC): big5, cp950
 \item Japanese: cp932, shift-jis, shift-jisx0213, euc-jp,
euc-jisx0213, iso-2022-jp, iso-2022-jp-1, iso-2022-jp-2,
 iso-2022-jp-3, iso-2022-jp-ext
 \item Korean: cp949, euc-kr, johab, iso-2022-kr
\end{itemize} 

\item There is a new \module{collections} module for 
   various specialized collection datatypes.  
   Currently it contains just one type, \class{deque}, 
   a double-ended queue that supports efficiently adding and removing
   elements from either end.

\begin{verbatim}
>>> from collections import deque
>>> d = deque('ghi')        # make a new deque with three items
>>> d.append('j')           # add a new entry to the right side
>>> d.appendleft('f')       # add a new entry to the left side
>>> d                       # show the representation of the deque
deque(['f', 'g', 'h', 'i', 'j'])
>>> d.pop()                 # return and remove the rightmost item
'j'
>>> d.popleft()             # return and remove the leftmost item
'f'
>>> list(d)                 # list the contents of the deque
['g', 'h', 'i']
>>> 'h' in d                # search the deque
True  
\end{verbatim}

Several modules now take advantage of \class{collections.deque} for
improved performance:  \module{Queue}, \module{mutex}, \module{shlex}
\module{threading}, and \module{pydoc}.

\item The \module{ConfigParser} classes have been enhanced slightly.
   The \method{read()} method now returns a list of the files that
   were successfully parsed, and the \method{set()} method raises
   \exception{TypeError} if passed a \var{value} argument that isn't a
   string.

\item The \module{heapq} module has been converted to C.  The resulting
   tenfold improvement in speed makes the module suitable for handling
   high volumes of data.

\item The \module{imaplib} module now supports IMAP's THREAD command.
(Contributed by Yves Dionne.)

\item The \module{itertools} module gained a
  \function{groupby(\var{iterable}\optional{, \var{func}})} function,
  inspired by the GROUP BY clause from SQL.
  \var{iterable} returns a succession of elements, and the optional
  \var{func} is a function that takes an element and returns a key
  value; if omitted, the key is simply the element itself.
  \function{groupby()} then groups the elements into subsequences
  which have matching values of the key, and returns a series of 2-tuples
  containing the key value and an iterator over the subsequence.
 
Here's an example.  The \var{key} function simply returns whether a
number is even or odd, so the result of \function{groupby()} is to
return consecutive runs of odd or even numbers.

\begin{verbatim}
>>> import itertools
>>> L = [2,4,6, 7,8,9,11, 12, 14]
>>> for key_val, it in itertools.groupby(L, lambda x: x % 2):
...    print key_val, list(it)
... 
0 [2, 4, 6]
1 [7]
0 [8]
1 [9, 11]
0 [12, 14]
>>> 
\end{verbatim}

Like its SQL counterpart, \function{groupby()} is typically used with
sorted input.  The logic for \function{groupby()} is similar to the
\UNIX{} \code{uniq} filter which makes it handy for eliminating,
counting, or identifying duplicate elements:

\begin{verbatim}
>>> word = 'abracadabra'
>>> letters = sorted(word)   # Turn string into a sorted list of letters
>>> letters 
['a', 'a', 'a', 'a', 'a', 'b', 'b', 'c', 'd', 'r', 'r']
>>> [k for k, g in groupby(letters)]                     # List unique letters
['a', 'b', 'c', 'd', 'r']
>>> [(k, len(list(g))) for k, g in groupby(letters)]     # Count letter occurences
[('a', 5), ('b', 2), ('c', 1), ('d', 1), ('r', 2)]
>>> [k for k, g in groupby(letters) if len(list(g)) > 1] # List duplicated letters
['a', 'b', 'r']
\end{verbatim}

\item \module{itertools} also gained a function named
\function{tee(\var{iterator}, \var{N})} that returns \var{N} independent
iterators that replicate \var{iterator}.  If \var{N} is omitted, the
default is 2.

\begin{verbatim}
>>> L = [1,2,3]
>>> i1, i2 = itertools.tee(L)
>>> i1,i2
(<itertools.tee object at 0x402c2080>, <itertools.tee object at 0x402c2090>)
>>> list(i1)               # Run the first iterator to exhaustion
[1, 2, 3]
>>> list(i2)               # Run the second iterator to exhaustion
[1, 2, 3]
>\end{verbatim}

Note that \function{tee()} has to keep copies of the values returned 
by the iterator; in the worst case, it may need to keep all of them.  
This should therefore be used carefully if the leading iterator
can run far ahead of the trailing iterator in a long stream of inputs.
If the separation is large, then it becomes preferable to use
\function{list()} instead.  When the iterators track closely with one
another, \function{tee()} is ideal.  Possible applications include
bookmarking, windowing, or lookahead iterators.

\item A new \function{getsid()} function was added to the
\module{posix} module that underlies the \module{os} module.
(Contributed by J. Raynor.)

\item The \module{operator} module gained two new functions, 
\function{attrgetter(\var{attr})} and \function{itemgetter(\var{index})}.
Both functions return callables that take a single argument and return
the corresponding attribute or item; these callables make excellent
data extractors when used with \function{map()} or \function{sorted()}.
For example:

\begin{verbatim}
>>> L = [('c', 2), ('d', 1), ('a', 4), ('b', 3)]
>>> map(operator.itemgetter(0), L)
['c', 'd', 'a', 'b']
>>> map(operator.itemgetter(1), L)
[2, 1, 4, 3]
>>> sorted(L, key=operator.itemgetter(1)) # Sort list by second tuple item
[('d', 1), ('c', 2), ('b', 3), ('a', 4)]
\end{verbatim}

\item The \module{random} module has a new method called \method{getrandbits(N)} 
   which returns an N-bit long integer.  This method supports the existing
   \method{randrange()} method, making it possible to efficiently generate
   arbitrarily large random numbers.

\item The regular expression language accepted by the \module{re} module
   was extended with simple conditional expressions, written as
   \code{(?(\var{group})\var{A}|\var{B})}.  \var{group} is either a
   numeric group ID or a group name defined with \code{(?P<group>...)} 
   earlier in the expression.  If the specified group matched, the
   regular expression pattern \var{A} will be tested against the string; if
   the group didn't match, the pattern \var{B} will be used instead.
   
\end{itemize}


%======================================================================
% whole new modules get described in \subsections here


% ======================================================================
\section{Build and C API Changes}

Changes to Python's build process and to the C API include:

\begin{itemize}

  \item Three new convenience macros were added for common return
  values from extension functions: \csimplemacro{Py_RETURN_NONE},
  \csimplemacro{Py_RETURN_TRUE}, and \csimplemacro{Py_RETURN_FALSE}.

  \item A new function, \cfunction{PyTuple_Pack(\var{N}, \var{obj1},
  \var{obj2}, ..., \var{objN})}, constructs tuples from a variable
  length argument list of Python objects.

  \item A new function, \cfunction{PyDict_Contains(\var{d}, \var{k})},
  implements fast dictionary lookups without masking exceptions raised
  during the look-up process.

  \item A new method flag, \constant{METH_COEXISTS}, allows a function
  defined in slots to co-exist with a PyCFunction having the same name.
  This can halve the access to time to a method such as
  \method{set.__contains__()}

\end{itemize}


%======================================================================
\subsection{Port-Specific Changes}

\begin{itemize}

\item The Windows port now builds under MSVC++ 7.1 as well as version 6.

\end{itemize}


%======================================================================
\section{Other Changes and Fixes \label{section-other}}

As usual, there were a bunch of other improvements and bugfixes
scattered throughout the source tree.  A search through the CVS change
logs finds there were XXX patches applied and YYY bugs fixed between
Python 2.3 and 2.4.  Both figures are likely to be underestimates.

Some of the more notable changes are:

\begin{itemize}

\item The \module{timeit} module now automatically disables periodic
  garbarge collection during the timing loop.  This change makes
  consecutive timings more comparable.

\item The \module{base64} module now has more complete RFC 3548 support
  for Base64, Base32, and Base16 encoding and decoding, including
  optional case folding and optional alternative alphabets.
  (Contributed by Barry Warsaw.)

\end{itemize}


%======================================================================
\section{Porting to Python 2.4}

This section lists previously described changes that may require
changes to your code:

\begin{itemize}

\item The \function{zip()} built-in function and \function{itertools.izip()}
  now return  an empty list instead of raising a \exception{TypeError}
  exception if called with no arguments.

\item \function{dircache.listdir()} now passes exceptions to the caller
      instead of returning empty lists.

\item \function{LexicalHandler.startDTD()} used to receive public and
  system ID in the wrong order.  This has been corrected; applications
  relying on the wrong order need to be fixed.

\end{itemize}


%======================================================================
\section{Acknowledgements \label{acks}}

The author would like to thank the following people for offering
suggestions, corrections and assistance with various drafts of this
article: Raymond Hettinger.

\end{document}