cpython/Misc/NEWS

What's new in this release?
===========================

Below is a list of all relevant changes since the release 1.4, up till
the release of 1.5a3.  At the end is a list of changes made since
1.5a3 that will be in 1.5a4; this list is not yet complete and will be
merged with the main list later.

A note on attributions: while I have sprinkled some names throughout
here, I'm grateful to many more people who remain anonymous.  You may
find your name in the ACKS file.  If you believe you deserve more
credit, let me know and I'll add you to the list!


Security
--------

- If you are using the setuid script C wrapper (Misc/setuid-prog.c),
please use the new version.  The old version has a huge security leak.

Miscellaneous
-------------

- Because of various (small) incompatible changes in the Python
bytecode interpreter, the magic number for .pyc files has changed
again.

- The default module search path is now much saner.  Both on Unix and
Windows, it is essentially derived from the path to the executable
(which can be overridden by setting the environment variable
$PYTHONHOME).  The value of $PYTHONPATH on Windows is now inserted in
front of the default path, like in Unix (instead of overriding the
default path).  On Windows, the directory containing the executable is
added to the end of the path.

- A new version of python-mode.el for Emacs has been included.  Also,
a new file ccpy-style.el has been added to configure Emacs cc-mode for
the preferred style in Python C sources.

- On Unix, when using sys.argv[0] to insert the script directory in
front of sys.path, expand a symbolic link.  You can now install a
program in a private directory and have a symbolic link to it in a
public bin directory, and it will put the private directory in the
module search path.  Note that the symlink is expanded in sys.path[0]
but not in sys.argv[0], so you can still tell the name by which you
were invoked.

- It is now recommended to use ``#!/usr/bin/env python'' instead of
``#!/usr/local/bin/python'' at the start of executable scripts, except
for CGI scripts.  It has been determined that the use of /usr/bin/env
is more portable than that of /usr/local/bin/python -- scripts almost
never have to be edited when the Python interpreter lives in a
non-standard place.  Note that this doesn't work for CGI scripts since
the python executable often doesn't live in the HTTP server's default
search path.

- The silly -s command line option and the corresponding
PYTHONSUPPRESS environment variable (and the Py_SuppressPrint global
flag in the Python/C API) are gone.

- Most problems on 64-bit platforms should now be fixed.  Andrew
Kuchling helped.  Some uncommon extension modules are still not
clean (image and audio ops?).

- Fixed a bug where multiple anonymous tuple arguments would be mixed up
when using the debugger or profiler (reported by Just van Rossum).
The simplest example is ``def f((a,b),(c,d)): print a,b,c,d''; this
would print the wrong value when run under the debugger or profiler.

- The hacks that the dictionary implementation used to speed up
repeated lookups of the same C string were removed; these were a
source of subtle problems and don't seem to serve much of a purpose
any longer.

- All traces of support for the long dead access statement have been
removed from the sources.

- Plugged the two-byte memory leak in the tokenizer when reading an
interactive EOF.

- There's a -O option to the interpreter that removes SET_LINENO
instructions and assert statements (see below); it uses and produces
.pyo files instead of .pyc files.  The speedup is only a few percent
in most cases.  The line numbers are still available in the .pyo file,
as a separate table (which is also available in .pyc files).  However,
the removal of the SET_LINENO instructions means that the debugger
(pdb) can't set breakpoints on lines in -O mode.  The traceback module
contains a function to extract a line number from the code object
referenced in a traceback object.  In the future it should be possible
to write external bytecode optimizers that create better optimized
.pyo files, and there should be more control over optimization;
consider the -O option a "teaser".  Without -O, the assert statement
actually generates code that first checks __debug__; if this variable
is false, the assertion is not checked.  __debug__ is a built-in
variable whose value is initialized to track the -O flag (it's true
iff -O is not specified).  With -O, no code is generated for assert
statements, nor for code of the form ``if __debug__: <something>''.
Sorry, no further constant folding happens.


Performance
-----------

- It's much faster (almost twice for pystone.py -- see
Tools/scripts).  See the entry on string interning below.

- Some speedup by using separate free lists for method objects (both
the C and the Python variety) and for floating point numbers.

- Big speedup by allocating frame objects with a single malloc() call.
The Python/C API for frames is changed (you shouldn't be using this
anyway).

- Significant speedup by inlining some common opcodes for common operand 
types (e.g.  i+i, i-i, and list[i]).  Fredrik Lundh.

- Small speedup by reordering the method tables of some common
objects (e.g. list.append is now first).

- Big optimization to the read() method of file objects.  A read()
without arguments now attempts to use fstat to allocate a buffer of
the right size; for pipes and sockets, it will fall back to doubling
the buffer size.  While that the improvement is real on all systems,
it is most dramatic on Windows.


Documentation
-------------

- Many new pieces of library documentation were contributed, mostly by
Andrew Kuchling.  Even cmath is now documented!  There's also a
chapter of the library manual, "libundoc.tex", which provides a
listing of all undocumented modules, plus their status (e.g. internal,
obsolete, or in need of documentation).  Also contributions by Sue
Williams, Skip Montanaro, and some module authors who succumbed to
pressure to document their own contributed modules :-).  Note that
printing the documentation now kills fewer trees -- the margins have
been reduced.

- I have started documenting the Python/C API. Unfortunately this project 
hasn't been completed yet.  It will be complete before the final release of 
Python 1.5, though.  At the moment, it's better to read the LaTeX source 
than to attempt to run it through LaTeX and print the resulting dvi file.

- The posix module (and hence os.py) now has doc strings!  Thanks to Neil 
Schemenauer.  I received a few other contributions of doc strings.  In most 
other places, doc strings are still wishful thinking...


Language changes
----------------

- Private variables with leading double underscore are now a permanent 
feature of the language.  (These were experimental in release 1.4.  I have 
favorable experience using them; I can't label them "experimental" 
forever.)

- There's new string literal syntax for "raw strings".  Prefixing a string 
literal with the letter r (or R) disables all escape processing in the 
string; for example, r'\n' is a two-character string consisting of a 
backslash followed by the letter n.  This combines with all forms of string 
quotes; it is actually useful for triple quoted doc strings which might 
contain references to \n or \t.  An embedded quote prefixed with a 
backslash does not terminate the string, but the backslash is still 
included in the string; for example, r'\'' is a two-character string 
consisting of a backslash and a quote.  (Raw strings are also 
affectionately known as Robin strings, after their inventor, Robin 
Friedrich.)

- There's a simple assert statement, and a new exception
AssertionError.  For example, ``assert foo > 0'' is equivalent to ``if
not foo > 0: raise AssertionError''.  Sorry, the text of the asserted
condition is not available; it would be too complicated to generate
code for this (since the code is generated from a parse tree).
However, the text is displayed as part of the traceback!

- The raise statement has a new feature: when using "raise SomeClass,
somevalue" where somevalue is not an instance of SomeClass, it
instantiates SomeClass(somevalue).  In 1.5a4, if somevalue is an
instance of a *derived* class of SomeClass, the exception class raised
is set to somevalue.__class__, and SomeClass is ignored after that.

- Duplicate keyword arguments are now detected at compile time;
f(a=1,a=2) is now a syntax error.


Changes to builtin features
---------------------------

- There's a new exception FloatingPointError (used only by Lee Busby's
patches to catch floating point exceptions, at the moment).

- The obsolete exception ConflictError (presumably used by the long
obsolete access statement) has been deleted.

- There's a new function sys.exc_info() which returns the tuple 
(sys.exc_type, sys.exc_value, sys.exc_traceback) in a thread-safe way.

- There's a new variable sys.executable, pointing to the executable file 
for the Python interpreter.

- The sort() methods for lists no longer uses the C library qsort(); I
wrote my own quicksort implementation, with lots of help (in the form
of a kind of competition) from Tim Peters.  This solves a bug in
dictionary comparisons on some Solaris versions when Python is built
with threads, and makes sorting lists even faster.

- The semantics of comparing two dictionaries have changed, to make
comparison of unequal dictionaries faster.  A shorter dictionary is
always considered smaller than a larger dictionary.  For dictionaries
of the same size, the smallest differing element determines the
outcome (which yields the same results as before in this case, without
explicit sorting).  Thanks to Aaron Watters for suggesting something
like this.

- The semantics of try-except have changed subtly so that calling a
function in an exception handler that itself raises and catches an
exception no longer overwrites the sys.exc_* variables.  This also
alleviates the problem that objects referenced in a stack frame that
caught an exception are kept alive until another exception is caught
-- the sys.exc_* variables are restored to their previous value when
returning from a function that caught an exception.

- There's a new "buffer" interface.  Certain objects (e.g. strings and
arrays) now support the "buffer" protocol.  Buffer objects are acceptable 
whenever formerly a string was required for a write operation; mutable 
buffer objects can be the target of a read operation using the call
f.readinto(buffer).  A cool feature is that regular expression matching now 
also work on array objects.  Contribution by Jack Jansen.  (Needs 
documentation.)

- String interning: dictionary lookups are faster when the lookup
string object is the same object as the key in the dictionary, not
just a string with the same value.  This is done by having a pool of
"interned" strings.  Most names generated by the interpreter are now
automatically interned, and there's a new built-in function intern(s)
that returns the interned version of a string.  Interned strings are
not a different object type, and interning is totally optional, but by
interning most keys a speedup of about 15% was obtained for the
pystone benchmark.

- Dictionary objects have several new methods; clear() and copy() have
the obvious semantics, while update(d) merges the contents of another
dictionary d into this one, overriding existing keys.  The dictionary
implementation file is now called dictobject.c rather than the
confusing mappingobject.c.

- The intrinsic function dir() is much smarter; it looks in __dict__,
__members__ and __methods__.

- The intrinsic functions int(), long() and float() can now take a
string argument and then do the same thing as string.atoi(),
string.atol(), and string.atof().  No second 'base' argument is
allowed, and complex() does not take a string (nobody cared enough).

- When a module is deleted, its globals are now deleted in two phases.
In the first phase, all variables whose name begins with exactly one
underscore are replaced by None; in the second phase, all variables
are deleted.  This makes it possible to have global objects whose
destructors depend on other globals.  The deletion order within each
phase is still random.

- It is no longer an error for a function to be called without a
global variable __builtins__ -- an empty directory will be provided
by default.

- Guido's corollary to the "Don Beaudry hook": it is now possible to
do metaprogramming by using an instance as a base class.  Not for the
faint of heart; and undocumented as yet, but basically if a base class
is an instance, its class will be instantiated to create the new
class.  Jim Fulton will love it -- it also works with instances of his
"extension classes", since it is triggered by the presence of a
__class__ attribute on the purported base class.  See
Demo/metaclasses/index.html for an explanation and see that directory
for examples.

- Another change is that the Don Beaudry hook is now invoked when
*any* base class is special.  (Up to 1.5a3, the *last* special base
class is used; in 1.5a4, the more rational choice of the *first*
special base class is used.)

- New optional parameter to the readlines() method of file objects.
This indicates the number of bytes to read (the actual number of bytes
read will be somewhat larger due to buffering reading until the end of
the line).  Some optimizations have also been made to speed it up (but
not as much as read()).

- Complex numbers no longer have the ".conj" pseudo attribute; use
z.conjugate() instead, or complex(z.real, -z.imag).  Complex numbers
now *do* support the __members__ and __methods__ special attributes.

- The complex() function now looks for a __complex__() method on class
instances before giving up.

- Long integers now support arbitrary shift counts, so you can now
write 1L<<1000000, memory permitting.  (Python 1.4 reports "outrageous
shift count for this.)

- The hex() and oct() functions have been changed so that for regular
integers, they never emit a minus sign.  For example, on a 32-bit
machine, oct(-1) now returns '037777777777' and hex(-1) returns
'0xffffffff'.  While this may seem inconsistent, it is much more
useful.  (For long integers, a minus sign is used as before, to fit
the result in memory :-)

- The hash() function computes better hashes for several data types,
including strings, floating point numbers, and complex numbers.


New extension modules
---------------------

- New extension modules cStringIO.c and cPickle.c, written by Jim
Fulton and other folks at Digital Creations.  These are much more
efficient than their Python counterparts StringIO.py and pickle.py,
but don't support subclassing.  cPickle.c clocks up to 1000 times
faster than pickle.py; cStringIO.c's improvement is less dramatic but
still significant.

- New extension module zlibmodule.c, interfacing to the free zlib
library (gzip compatible compression).  There's also a module gzip.py
which provides a higher level interface.  Written by Andrew Kuchling
and Jeremy Hylton.

- New module readline; see the "miscellaneous" section above.

- New Unix extension module resource.c, by Jeremy Hylton, provides
access to getrlimit(), getrusage(), setrusage(), getpagesize(), and
related symbolic constants.

- New extension puremodule.c, by Barry Warsaw, which interfaces to the
Purify(TM) C API.  See also the file Misc/PURIFY.README.  It is also
possible to enable Purify by simply setting the PURIFY Makefile
variable in the Modules/Setup file.


Changes in extension modules
----------------------------

- The struct extension module has several new features to control byte
order and word size.  It supports reading and writing IEEE floats even
on platforms where this is not the native format.  It uses uppercase
format codes for unsigned integers of various sizes (always using
Python long ints for 'I' and 'L'), 's' with a size prefix for strings,
and 'p' for "Pascal strings" (with a leading length byte, included in
the size; blame Hannu Krosing).  A prefix '>' forces big-endian data
and '<' forces little-endian data; these also select standard data
sizes and disable automatic alignment (use pad bytes as needed).

- The array module supports uppercase format codes for unsigned data
formats (like the struct module).

- The fcntl extension module now exports the needed symbolic
constants.  (Formerly these were in FCNTL.py which was not available
or correct for all platforms.)

- The extension modules dbm, gdbm and bsddb now check that the
database is still open before making any new calls.

- The dbhash module is no more.  Use bsddb instead.  (There's a third
party interface for the BSD 2.x code somewhere on the web; support for
bsddb will be deprecated.)

- The gdbm module now supports a sync() method.

- The socket module now has some new functions: getprotobyname(), and
the set {ntoh,hton}{s,l}().

- Various modules now export their type object: socket.SocketType,
array.ArrayType.

- The socket module's accept() method now returns unknown addresses as
a tuple rather than raising an exception.  (This can happen in
promiscuous mode.)  Theres' also a new function getprotobyname().

- The pthread support for the thread module now works on most platforms.

- STDWIN is now officially obsolete.  Support for it will eventually
be removed from the distribution.

- The binascii extension module is now hopefully fully debugged.
(XXX Oops -- Fredrik Lundh promised me a uuencode fix that I never
received.)

- audioop.c: added a ratecv() function; better handling of overflow in
add().

- posixmodule.c: now exports the O_* flags (O_APPEND etc.).  On
Windows, also O_TEXT and O_BINARY.  The 'error' variable (the
exception is raises) is renamed -- its string value is now "os.error",
so newbies don't believe they have to import posix (or nt) to catch
it when they see os.error reported as posix.error.  The execve()
function now accepts any mapping object for the environment.

- A new version of the al (audio library) module for SGI was
contributed by Sjoerd Mullender.

- The regex module has a new function get_syntax() which retrieves the
syntax setting set by set_syntax().  The code was also sanitized,
removing worries about unclean error handling.  See also below for its
successor, re.py.

- The "new" module (which creates new objects of various types) once
again has a fully functioning new.function() method.  Dangerous as
ever!  Also, new.code() has several new arguments.

- A problem has been fixed in the rotor module: on systems with signed
characters, rotor-encoded data was not portable when the key contained
8-bit characters.  Also, setkey() now requires its argument rather
than having broken code to default it.

- The sys.builtin_module_names variable is now a tuple.  Another new
variables in sys is sys.executable (the full path to the Python
binary, if known).

- The specs for time.strftime() have undergone some revisions.  It
appears that not all format characters are supported in the same way
on all platforms.  Rather than reimplement it, we note these
differences in the documentation, and emphasize the shared set of
features.  There's also a thorough test set (that occasionally finds
problems in the C library implementation, e.g. on some Linuxes),
thanks to Skip Montanaro.

- The nis module seems broken when used with NIS+; unfortunately
nobody knows how to fix it.  It should still work with old NIS.


New library modules
-------------------

- New (still experimental) Perl-style regular expression module,
re.py, which uses a new interface for matching as well as a new
syntax; the new interface avoids the thread-unsafety of the regex
interface.  This comes with a helper extension reopmodule.c and vastly
rewritten regexpr.c.  Most work on this was done by Jeffrey Ollie, Tim
Peters, and Andrew Kuchling.  See the documentation libre.tex.  In
1.5, the old regex module is still fully supported; in the future, it
will become obsolete.

- New module gzip.py; see zlib above.

- New module keyword.py exports knowledge about Python's built-in
keywords.  (New version by Ka-Ping Yee.)

- New module pprint.py (with documentation) which supports
pretty-printing of lists, tuples, & dictionaries recursively.  By Fred
Drake.

- New module code.py.  The function code.compile_command() can
determine whether an interactively entered command is complete or not,
distinguishing incomplete from invalid input.  (XXX Unfortunately,
this seems broken at this moment, and I don't have the time to fix
it.  It's probably better to add an explicit interface to the parser
for this.)

- There is now a library module xdrlib.py which can read and write the
XDR data format as used by Sun RPC, for example.  It uses the struct
module.


Changes in library modules
--------------------------

- Module codehack.py is now completely obsolete.

- The pickle.py module has been updated to make it compatible with the
new binary format that cPickle.c produces.  By default it produces the
old all-ASCII format compatible with the old pickle.py, still much
faster than pickle.py; it will read both formats automatically.  A few
other updates have been made.

- A new helper module, copy_reg.py, is provided to register extensions
to the pickling code.

- Revamped module tokenize.py is much more accurate and has an
interface that makes it a breeze to write code to colorize Python
source code.  Contributed by Ka-Ping Yee.

- In ihooks.py, ModuleLoader.load_module() now closes the file under
all circumstances.

- The tempfile.py module has a new class, TemporaryFile, which creates
an open temporary file that will be deleted automatically when
closed.  This works on Windows and MacOS as well as on Unix.  (Jim
Fulton.)

- Changes to the cgi.py module: Most imports are now done at the
top of the module, which provides a speedup when using ni (Jim
Fulton).  The problem with file upload to a Windows platform is solved
by using the new tempfile.TemporaryFile class; temporary files are now
always opened in binary mode (Jim Fulton).  The cgi.escape() function
now takes an optional flag argument that quotes '"' to '&quot;'.  It
is now possible to invoke cgi.py from a command line script, to test
cgi scripts more easily outside an http server.  There's an optional
limit to the size of uploads to POST (Skip Montanaro).  Added a
'strict_parsing' option to all parsing functions (Jim Fulton).  The
function parse_qs() now uses urllib.unquote() on the name as well as
the value of fields (Clarence Gardner).  The FieldStorage class now
has a __len__() method.

- httplib.py: the socket object is no longer closed; all HTTP/1.*
responses are now accepted; and it is now thread-safe (by not using
the regex module).

- BaseHTTPModule.py: treat all HTTP/1.* versions the same.

- The popen2.py module is now rewritten using a class, which makes
access to the standard error stream and the process id of the
subprocess possible.

- Added timezone support to the rfc822.py module, in the form of a
getdate_tz() method and a parsedate_tz() function; also a mktime_tz().
Also added recognition of some non-standard date formats, by Lars
Wirzenius, and RFC 850 dates (Chris Lawrence).

- mhlib.py: various enhancements, including almost compatible parsing
of message sequence specifiers without invoking a subprocess.  Also
added a createmessage() method by Lars Wirzenius.

- The StringIO.StringIO class now supports readline(nbytes).  (Lars 
Wirzenius.)  (Of course, you should be using cStringIO for performance.)

- UserDict.py supports the new dictionary methods as well.

- Improvements for whrandom.py by Tim Peters: use 32-bit arithmetic to
speed it up, and replace 0 seed values by 1 to avoid degeneration.
A bug was fixed in the test for invalid arguments.

- Module ftplib.py: added support for parsing a .netrc file (Fred
Drake).  Also added an ntransfercmd() method to the FTP class, which
allows access to the expected size of a transfer when available, and a
parse150() function to the module which parses the corresponding 150
response.

- urllib.py: the ftp cache is now limited to 10 entries.  Added
quote_plus() and unquote_plus() functions which are like quote() and
unquote() but also replace spaces with '+' or vice versa, for
encoding/decoding CGI form arguments.  Catch all errors from the ftp
module.  HTTP requests now add the Host: header line.  The proxy
variable names are now mapped to lower case, for Windows.  The
spliturl() function no longer erroneously throws away all data past
the first newline.  The basejoin() function now intereprets "../"
correctly.  I *believe* that the problems with "exception raised in
__del__" under certain circumstances have been fixed (mostly by
changes elsewher in the interpreter).

- In urlparse.py, there is a cache for results in urlparse.urlparse();
its size limit is set to 20.  Also, new URL schemes shttp, https, and
snews are "supported".

- shelve.py: use cPickle and cStringIO when available.  Also added
a sync() method, which calls the database's sync() method if there is
one.

- The mimetools.py module now uses the available Python modules for
decoding quoted-printable, uuencode and base64 formats, rather than
creating a subprocess.

- The python debugger (pdb.py, and its base class bdb.py) now support
conditional breakpoints.  See the docs.

- The modules base64.py, uu.py and quopri.py can now be used as simple
command line utilities.

- Various small fixes to the nntplib.py module that I can't bother to
document in detail.

- Sjoerd Mullender's mimify.py module now supports base64 encoding and 
includes functions to handle the funny encoding you sometimes see in mail 
headers.  It is now documented.

- mailbox.py: Added BabylMailbox.  Improved the way the mailbox is
gotten from the environment.

- Many more modules now correctly open files in binary mode when this
is necessary on non-Unix platforms.

- The copying functions in the undocumented module shutil.py are
smarter.

- The Writer classes in the formatter.py module now have a flush()
method.

- The sgmllib.py module accepts hyphens and periods in the middle of
attribute names.  While this is against the SGML standard, there is
some HTML out there that uses this...

- The interface for the Python bytecode disassembler module, dis.py,
has been enhanced quite a bit.  There's now one main function,
dis.dis(), which takes almost any kind of object (function, module,
class, instance, method, code object) and disassembles it; without
arguments it disassembles the last frame of the last traceback.  The
other functions have changed slightly, too.

- The imghdr.py module recognizes new image types: BMP, PNG.

- The string.py module has a new function replace(str, old, new,
[maxsplit]) which does substring replacements.  It is actually
implemented in C in the strop module.  The functions [r]find() an
[r]index() have an optional 4th argument indicating the end of the
substring to search, alsoo implemented by their strop counterparts.
(Remember, never import strop -- import string uses strop when
available with zero overhead.)

- The string.join() function now accepts any sequence argument, not
just lists and tuples.

- The string.maketrans() requires its first two arguments to be
present.  The old version didn't require them, but there's not much
point without them, and the documentation suggests that they are
required, so we fixed the code to match the documentation.

- The regsub.py module has a function clear_cache(), which clears its
internal cache of compiled regular expressions.  Also, the cache now
takes the current syntax setting into account.  (However, this module
is now obsolete -- use the sub() or subn() functions or methods in the
re module.)

- The undocumented module Complex.py has been removed, now that Python
has built-in complex numbers.  A similar module remains as
Demo/classes/Complex.py, as an example.


Changes to the build process
----------------------------

- The way GNU readline is configured is totally different.  The
--with-readline configure option is gone.  It is now an extension
module, which may be loaded dynamically.  You must enable it (and
specify the correct linraries to link with) in the Modules/Setup file.
Importing the module installs some hooks which enable command line
editing.  When the interpreter shell is invoked interactively, it
attempts to import the readline module; when this fails, the default
input mechanism is used.  The hook variables are PyOS_InputHook and
PyOS_ReadlineFunctionPointer.  (Code contributed by Lee Busby, with
ideas from William Magro.)

- New build procedure: a single library, libpython1.5.a, is now built,
which contains absolutely everything except for a one-line main()
program (which calls Py_Main(argc, argv) to start the interpreter
shell).  This makes life much simpler for applications that need to
embed Python.  The serial number of the build is now included in the
version string (sys.version).

- As far as I can tell, neither gcc -Wall nor the Microsoft compiler
emits a single warning any more when compiling Python.

- A number of new Makefile variables have been added for special
situations, e.g. LDLAST is appended to the link command.  These are
used by editing the Makefile or passing them on the make command
line.

- A set of patches from Lee Busby has been integrated that make it
possible to catch floating point exceptions.  Use the configure option
--with-fpectl to enable the patches; the extension modules fpectl and
fpetest provide control to enable/disable and test the feature,
respectively.

- The support for shared libraries under AIX is now simpler and more
robust.  Thanks to Vladimir Marangozov for revamping his own patches!

- The Modules/makesetup script now reads a file Setup.local as well as
a file Setup.  Most changes to the Setup script can be done by editing
Setup.local instead, which makes it easier to carry a particular setup
over from one release to the next.

- The Modules/makesetup script now copies any "include" lines it
encounters verbatim into the output Makefile.  It also recognizes .cxx
and .cpp as C++ source files.

- The configure script is smarter about C compiler options; e.g. with
gcc it uses -O2 and -g when possible, and on some other platforms it
uses -Olimit 1500 to avoid a warning from the optimizer about the main
loop in ceval.c (which has more than 1000 basic blocks).

- The configure script now detects whether malloc(0) returns a NULL
pointer or a valid block (of length zero).  This avoids the nonsense
of always adding one byte to all malloc() arguments on most platforms.

- The configure script has a new option, --with-dec-threads, to enable
DEC threads on DEC Alpha platforms.  Also, --with-threads is now an
alias for --with-thread (this was the Most Common Typo in configure
arguments).

- Many changes in Doc/Makefile; amongst others, latex2html is now used
to generate HTML from all latex documents.


Change to the Python/C API
--------------------------

- Because some interfaces have changed, the PYTHON_API macro has been
bumped.  Most extensions built for the old API version will still run,
but I can't guarantee this.  Python prints a warning message on
version mismatches; it dumps core when the version mismatch causes a
serious problem :-)

- I've completed the Grand Renaming, with the help of Roger Masse and
Barry Warsaw.  This makes reading or debugging the code much easier.
Many other unrelated code reorganizations have also been carried out.
The allobjects.h header file is gone; instead, you would have to
include Python.h followed by rename2.h.  But you're better off running
Tools/scripts/fixcid.py -s Misc/RENAME on your source, so you can omit
the rename2.h; it will disappear in the next release.

- Various and sundry small bugs in the "abstract" interfaces have been
fixed.  Thanks to all the (involuntary) testers of the Python 1.4
version!  Some new functions have been added, e.g. PySequence_List(o),
equivalent to list(o) in Python.

- New API functions PyLong_FromUnsignedLong() and
PyLong_AsUnsignedLong().

- The API functions in the file cgensupport.c are no longer
supported.  This file has been moved to Modules and is only ever
compiled when the SGI specific 'gl' module is built.

- PyObject_Compare() can now raise an exception.  Check with
PyErr_Occurred().  The comparison function in an object type may also
raise an exception.

- The slice interface uses an upper bound of INT_MAX when no explicit
upper bound is given (e.x. for a[1:]).  It used to ask the object for
its length and do the calculations.

- Support for multiple independent interpreters.  See Doc/api.tex,
functions Py_NewInterpreter() and Py_EndInterpreter().  Since the
documentation is incomplete, also see the new Demo/pysvr example
(which shows how to use these in a threaded application) and the
source code.

- There is now a Py_Finalize() function which "de-initializes"
Python.  It is possible to completely restart the interpreter
repeatedly by calling Py_Finalize() followed by Py_Initialize().  A
change of functionality in Py_Initialize() means that it is now a
fatal error to call it while the interpreter is already initialized.
The old, half-hearted Py_Cleanup() routine is gone.  Use of Py_Exit()
is deprecated (it is nothing more than Py_Finalize() followed by
exit()).

- There are no known memory leaks left.  While Py_Finalize() doesn't
free *all* allocated memory (some of it is hard to track down),
repeated calls to Py_Finalize() and Py_Initialize() do not create
unaccessible heap blocks.

- There is now explicit per-thread state.  (Inspired by, but not the
same as, Greg Stein's free threading patches.)

- There is now better support for threading C applications.  There are
now explicit APIs to manipulate the interpreter lock.  Read the source
or the Demo/pysvr example; the new functions are
PyEval_{Acquire,Release}{Lock,Thread}().

- The test macro DEBUG has changed to Py_DEBUG, to avoid interference
with other libraries' DEBUG macros.  Likewise for any other test
macros that didn't yet start with Py_.

- New wrappers around malloc() and friends: Py_Malloc() etc. call
malloc() and call PyErr_NoMemory() when it fails; PyMem_Malloc() call
just malloc().  Use of these wrappers could be essential if multiple
memory allocators exist (e.g. when using certain DLL setups under
Windows).  (Idea by Jim Fulton.)

- New C API PyImport_Import() which uses whatever __import__() hook
that is installed for the current execution environment.  By Jim
Fulton.

- It is now possible for an extension module's init function to fail
non-fatally, by calling one of the PyErr_* functions and returning.

- The PyInt_AS_LONG() and PyFloat_AS_DOUBLE() macros now cast their
argument to the proper type, like the similar PyString macros already
did.  (Suggestion by Marc-Andre Lemburg.)  Similar for PyList_GET_SIZE
and PyList_GET_ITEM.

- Some of the Py_Get* function, like Py_GetVersion() (but not yet
Py_GetPath()) are now declared as returning a const char *.  (More
should follow.)

- Changed the run-time library to check for exceptions after object
comparisons.  PyObject_Compare() can now return an exception; use
PyErr_Occurred() to check (there is *no* special return value).

- PyFile_WriteString() and Py_Flushline() now return error indicators
instead of clearing exceptions.  This fixes an obscure bug where using
these would clear a pending exception, discovered by Just van Rossum.

- There's a new function, PyArg_ParseTupleAndKeywords(), which parses
an argument list including keyword arguments.  Contributed by Geoff
Philbrick.

- PyArg_GetInt() is gone.

- It's no longer necessary to include graminit.h when calling one of
the extended parser API functions.  The three public grammar start
symbols are now in Python.h as Py_single_input, Py_file_input, and
Py_eval_input.

- The CObject interface has a new function,
PyCObject_Import(module, name).  It calls PyCObject_AsVoidPtr()
on the object referenced by "module.name".


Tkinter
-------

- On popular demand, _tkinter once again installs a hook for readline
that processes certain Tk events while waiting for the user to type
(using PyOS_InputHook).

- A patch by Craig McPheeters plugs the most obnoxious memory leaks,
caused by command definitions referencing widget objects beyond their
lifetime.

- New standard dialog modules: tkColorChooser.py, tkCommonDialog.py,
tkMessageBox.py, tkFileDialog.py, tkSimpleDialog.py These interface
with the new Tk dialog scripts, and provide more "native platform"
style file selection dialog boxes on some platforms.  Contributed by
Fredrik Lundh.

- Tkinter.py: when the first Tk object is destroyed, it sets the
hiddel global _default_root to None, so that when another Tk object is
created it becomes the new default root.  Other miscellaneous
changes and fixes.

- The Image class now has a configure method.

- Added a bunch of new winfo options to Tkinter.py; we should now be
up to date with Tk 4.2.  The new winfo options supported are:
mananger, pointerx, pointerxy, pointery, server, viewable, visualid,
visualsavailable.

- The broken bind() method on Canvas objects defined in the Canvas.py
module has been fixed.  The CanvasItem and Group classes now also have
an unbind() method.

- The problem with Tkinter.py falling back to trying to import
"tkinter" when "_tkinter" is not found has been fixed -- it no longer
tries "tkinter", ever.  This makes diagnosing the problem "_tkinter
not configured" much easier and will hopefully reduce the newsgroup
traffic on this topic.

- The ScrolledText module once again supports the 'cnf' parameter, to
be compatible with the examples in Mark Lutz' book (I know, I know,
too late...)

- The _tkinter.c extension module has been revamped.  It now support
Tk versions 4.1 through 8.0; support for 4.0 has been dropped.  It
works well under Windows and Mac (with the latest Tk ports to those
platforms).  It also supports threading -- it is safe for one
(Python-created) thread to be blocked in _tkinter.mainloop() while
other threads modify widgets.  To make the changes visible, those
threads must use update_idletasks()method.  (The patch for threading
in 1.5a3 was broken; in 1.5a4, it is back in a different version,
which requires access to the Tcl sources to get it to work -- hence it
is disabled by default.)

- A bug in _tkinter.c has been fixed, where Split() with a string
containing an unmatched '"' could cause an exception or core dump.

- Unfortunately, on Windows and Mac, Tk 8.0 no longer supports
CreateFileHandler, so _tkinter.createfilehandler is not available on
those platforms when using Tk 8.0 or later.  I will have to rethink
how to interface with Tcl's lower-level event mechanism, or with its
channels (which are like Python's file-like objects).  Jack Jansen has
provided a fix for the Mac, so createfilehandler *is* actually
supported there; maybe I can adapt his fix for Windows.


Tools and Demos
---------------

- A new regression test suite is provided, which tests most of the
standard and built-in modules.  The regression test is run by invoking
the script Lib/test/regrtest.py.  Barry Warsaw wrote the test harnass;
he and Roger Masse contributed most of the new tests.

- New tool: faqwiz -- the CGI script that is used to maintain the
Python FAQ (http://grail.cnri.reston.va.us/cgi-bin/faqw.py).  In
Tools/faqwiz.

- New tool: webchecker -- a simple extensible web robot that, when
aimed at a web server, checks that server for dead links.  Available
are a command line utility as well as a Tkinter based GUI version.  In
Tools/webchecker.  A simplified version of this program is dissected
in my article in O'Reilly's WWW Journal, the issue on Scripting
Languages (Vol 2, No 2); Scripting the Web with Python (pp 97-120).
Includes a parser for robots.txt files by Skip Montanaro.

- New small tools: cvsfiles.py (prints a list of all files under CVS
n a particular directory tree), treesync.py (a rather Guido-specific
script to synchronize two source trees, one on Windows NT, the other
one on Unix under CVS but accessible from the NT box), and logmerge.py
(sort a collection of RCS or CVS logs by date).  In Tools/scripts.

- The freeze script now also works under Windows (NT).  Another
feature allows the -p option to be pointed at the Python source tree
instead of the installation prefix.  This was loosely based on part of
xfreeze by Sam Rushing and Bill Tutt.

- New examples (Demo/extend) that show how to use the generic
extension makefile (Misc/Makefile.pre.in).

- Tools/scripts/h2py.py now supports C++ comments.

- Tools/scripts/pystone.py script is upgraded to version 1.1; there
was a bug in version 1.0 (distributed with Python 1.4) that leaked
memory.  Also, in 1.1, the LOOPS variable is incremented to 10000.

- Demo/classes/Rat.py completely rewritten by Sjoerd Mullender.


Windows (NT and 95)
-------------------

- New project files for Developer Studio (Visual C++) 5.0 for Windows
NT (the old VC++ 4.2 Makefile is also still supported, but will
eventually be withdrawn due to its bulkiness).

- See the note on the new module search path in the "Miscellaneous" section 
above.

- Support for Win32s (the 32-bit Windows API under Windows 3.1) is
basically withdrawn.  If it still works for you, you're lucky.

- There's a new extension module, msvcrt.c, which provides various 
low-level operations defined in the Microsoft Visual C++ Runtime Library.  
These include locking(), setmode(), get_osfhandle(), set_osfhandle(), and 
console I/O functions like kbhit(), getch() and putch().

- The -u option not only sets the standard I/O streams to unbuffered
status, but also sets them in binary mode.  (This can also be done
using msvcrt.setmode(), by the way.)

- The, sys.prefix and sys.exec_prefix variables point to the directory 
where Python is installed, or to the top of the source tree, if it was run 
from there.

- The various os.path modules (posixpath, ntpath, macpath) now support
passing more than two arguments to the join() function, so
os.path.join(a, b, c) is the same as os.path.join(a, os.path.join(b,
c)).

- The ntpath module (normally used as os.path) supports ~ to $HOME 
expansion in expanduser().

- The freeze tool now works on Windows.

- See also the Tkinter category for a sad note on
_tkinter.createfilehandler().

- The truncate() method for file objects now works on Windows.

- Py_Initialize() is no longer called when the DLL is loaded.  You
must call it yourself.  (And you can't call it twice -- it's a fatal
error to call it when Python is already initialized.)

- The time module's clock() function now has good precision through
the use of the Win32 API QueryPerformanceCounter().

- Mark Hammond will release Python 1.5 versions of PythonWin and his
other Windows specific code: the win32api extensions, COM/ActiveX
support, and the MFC interface.


Mac
---

- As always, the Macintosh port will be done by Jack Jansen.  He will
make a separate announcement for the Mac specific source code and the
binary distribution(s) when these are ready.


Fixed after 1.5a3 was released
------------------------------

The following changes have been made to the source base after the
release of 1.5a3.  These need to be merged into their respective
categories for the next release.

- faqwiz.py: version 0.8; Recognize https:// as URL; <html>...</html>
feature; better install instructions; removed faqmain.py (which was an
older version).

- nntplib.py: Fixed some bugs reported by Lars Wirzenius (to Debian)
about the treatment of lines starting with '.'.  Added a minimal test
function.

- struct module: ignore most whitespace in format strings.

- urllib.py: close the socket and temp file in URLopener.retrieve() so
that multiple retrievals using the same connection work.

- Three new C API functions:

  - int PyErr_GivenExceptionMatches(obj1, obj2)

    Returns 1 if obj1 and obj2 are the same object, or if obj1 is an
    instance of type obj2, or of a class derived from obj2

  - int PyErr_ExceptionMatches(obj)

    Higher level wrapper around PyErr_GivenExceptionMatches() which uses
    PyErr_Occurred() as obj1.  This will be the more commonly called
    function.

  - void PyErr_NormalizeException(typeptr, valptr, tbptr)

    Normalizes exceptions, and places the normalized values in the
    arguments.  If type is not a class, this does nothing.  If type is a
    class, then it makes sure that value is an instance of the class by:

    1. if instance is of the type, or a class derived from type, it does
       nothing.

    2. otherwise it instantiates the class, using the value as an
       argument.  If value is None, it uses an empty arg tuple, and if
       the value is a tuple, it uses just that.

- Demo/metaclasses: new demo subdir explains metaclasses (read
index.html in a browser).

- core interpreter: remove the distinction between tuple and list
unpacking; allow an arbitrary sequence on the right hand side of any
unpack instruction.  (UNPACK_LIST and UNPACK_TUPLE now do the same
thing, which should really be called UNPACK_SEQUENCE.)

- classes: Allow assignments to an instance's __dict__ or __class__,
so you can change ivars (including shared ivars -- shock horror) and
change classes dynamically.  Also make the check on read-only
attributes of classes less draconic -- only the specials names
__dict__, __bases__, __name__ and __{get,set,del}attr__ can't be
assigned.

- Two new built-in functions: issubclass() and isinstance().  Both
take classes as their second arguments.  The former takes a class as
the first argument and returns true iff first is second, or is a
subclass of second.  The latter takes any object as the first argument
and returns true iff first is an instance of the second, or any
subclass of second.

- configure: Added configuration tests for presence of alarm(),
pause(), and getpwent().

- Doc/Makefile: changed latex2html targets.

- classes: Reverse the search order for the Don Beaudry hook so that
the first class with an applicable hook wins.  Makes more sense.

- Changed the checks made in Py_Initialize() and Py_Finalize().  It is
now legal to call these more than once.  The first call to
Py_Initialize() initializes, the first call to Py_Finalize()
finalizes.  There's also a new API, Py_IsInitalized() which checks
whether we are already initialized (in case you want to leave things
as they were).

- Completely disable the declarations for malloc(), realloc() and
free().  Any 90's C compiler has these in header files, and the tests
to decide whether to suppress the declarations kept failing on some
platforms.

- *Before* (instead of after) signalmodule.o is added, remove both
intrcheck.o and sigcheck.o.  This should get rid of warnings in ar or
ld on various systems.

- Added reop to PC/config.c

- configure: Decided to use -Aa -D_HPUX_SOURCE on HP-UX platforms.
Removed outdated HP-UX comments from README.  Added Cray T3E comments.

- Various renames of statically defined functions that had name
conflicts on some systems, e.g. strndup (GNU libc), join (Cray),
roundup (sys/types.h).

- urllib.py: Interpret three slashes in file: URL as local file (for
Netscape on Windows/Mac).

- copy.py: Make sure the objects returned by __getinitargs__() are
kept alive (in the memo) to avoid a certain kind of nasty crash.  (Not
easily reproducable because it requires a later call to
__getinitargs__() to return a tuple that happens to be allocated at
the same address.)

- Added definition of AR to toplevel Makefile.  Renamed @buildno temp
file to buildno1.

- Moved Include/assert.h to Parser/assert.h, which seems to be the
only place where it's needed.

- Alas, the thread support for _tkinter didn't work.  Withdrew it.

- Tweaked the dictionary lookup code again for some more speed
(Vladimir Marangozov).

- NT build: Changed the way python15.lib is included in the other
projects.  Per Mark Hammond's suggestion, add it to the extra libs in
Settings instead of to the project's source files.

- regrtest.py: Change default verbosity so that there are only three
levels left: -q, default and -v.  In default mode, the name of each
test is now printed.  -v is the same as the old -vv.  -q is more quiet
than the old default mode.

- Removed the old FAQ from the distribution.  You now have to get it
from the web!

- Removed the PC/make_nt.in file from the distribution; it is no
longer needed.

- Changed the build sequence so that shared modules are built last.
This fixes things for AIX and doesn't hurt elsewhere.

- Improved test for GNU MP v1 in mpzmodule.c

- fileobject.c: ftell() on Linux discards all buffered data; changed
read() code to use lseek() instead to get the same effect

- configure.in, configure, importdl.c: NeXT sharedlib fixes

- tupleobject.c: PyTuple_SetItem asserts refcnt==1

- resource.c: Different strategy regarding whether to declare
getrusage() and getpagesize() -- #ifdef doesn't work, Linux has
conflicting decls in its headers.  Choice: only declare the return
type, not the argument prototype, and not on Linux.

- importdl.c, configure*: set sharedlib extensions properly for NeXT

- configure*, Makefile.in, Modules/Makefile.pre.in: AIX shared libraries
fixed; moved addition of PURIFY to LINKCC to configure

- reopmodule.c, regexmodule.c, regexpr.c, zlibmodule.c: needed casts
added to shup up various compilers.

- _tkinter.c: removed buggy mac #ifndef

- Doc: various Mac documentation changes, added docs for 'ic' module

- PC/make_nt.in: deleted

- test_time.py, test_strftime.py: tweaks to catch %Z (which may return
"")

- test_rotor.py: print b -> print `b`

- Tkinter.py: (tagOrId) -> (tagOrId,)

- Tkinter.py: the Tk class now also has a configure() method and
friends (they have been moved to the Misc class to accomplish this).

- dict.get(key[, default]) returns dict[key] if it exists, or default
if it doesn't.  The default defaults to None.  This is quicker for
some applications than using either has_key() or try:...except
KeyError:....