303 lines
12 KiB
TeX
303 lines
12 KiB
TeX
\section{\module{pickle} ---
|
|
Python object serialization.}
|
|
\declaremodule{standard}{pickle}
|
|
|
|
\modulesynopsis{Convert Python objects to streams of bytes and back.}
|
|
|
|
\index{persistency}
|
|
\indexii{persistent}{objects}
|
|
\indexii{serializing}{objects}
|
|
\indexii{marshalling}{objects}
|
|
\indexii{flattening}{objects}
|
|
\indexii{pickling}{objects}
|
|
|
|
|
|
The \module{pickle} module implements a basic but powerful algorithm for
|
|
``pickling'' (a.k.a.\ serializing, marshalling or flattening) nearly
|
|
arbitrary Python objects. This is the act of converting objects to a
|
|
stream of bytes (and back: ``unpickling'').
|
|
This is a more primitive notion than
|
|
persistency --- although \module{pickle} reads and writes file objects,
|
|
it does not handle the issue of naming persistent objects, nor the
|
|
(even more complicated) area of concurrent access to persistent
|
|
objects. The \module{pickle} module can transform a complex object into
|
|
a byte stream and it can transform the byte stream into an object with
|
|
the same internal structure. The most obvious thing to do with these
|
|
byte streams is to write them onto a file, but it is also conceivable
|
|
to send them across a network or store them in a database. The module
|
|
\module{shelve} provides a simple interface to pickle and unpickle
|
|
objects on ``dbm''-style database files.
|
|
\refstmodindex{shelve}
|
|
|
|
\strong{Note:} The \module{pickle} module is rather slow. A
|
|
reimplementation of the same algorithm in \C{}, which is up to 1000 times
|
|
faster, is available as the \module{cPickle}\refbimodindex{cPickle}
|
|
module. This has the same interface except that \code{Pickler} and
|
|
\code{Unpickler} are factory functions, not classes (so they cannot be
|
|
used as base classes for inheritance).
|
|
|
|
Unlike the built-in module \module{marshal}, \module{pickle} handles
|
|
the following correctly:
|
|
\refbimodindex{marshal}
|
|
|
|
\begin{itemize}
|
|
|
|
\item recursive objects (objects containing references to themselves)
|
|
|
|
\item object sharing (references to the same object in different places)
|
|
|
|
\item user-defined classes and their instances
|
|
|
|
\end{itemize}
|
|
|
|
The data format used by \module{pickle} is Python-specific. This has
|
|
the advantage that there are no restrictions imposed by external
|
|
standards such as XDR%
|
|
\index{XDR}
|
|
\index{External Data Representation}
|
|
(which can't represent pointer sharing); however
|
|
it means that non-Python programs may not be able to reconstruct
|
|
pickled Python objects.
|
|
|
|
By default, the \module{pickle} data format uses a printable \ASCII{}
|
|
representation. This is slightly more voluminous than a binary
|
|
representation. The big advantage of using printable \ASCII{} (and of
|
|
some other characteristics of \module{pickle}'s representation) is that
|
|
for debugging or recovery purposes it is possible for a human to read
|
|
the pickled file with a standard text editor.
|
|
|
|
A binary format, which is slightly more efficient, can be chosen by
|
|
specifying a nonzero (true) value for the \var{bin} argument to the
|
|
\class{Pickler} constructor or the \function{dump()} and \function{dumps()}
|
|
functions. The binary format is not the default because of backwards
|
|
compatibility with the Python 1.4 pickle module. In a future version,
|
|
the default may change to binary.
|
|
|
|
The \module{pickle} module doesn't handle code objects, which the
|
|
\module{marshal} module does. I suppose \module{pickle} could, and maybe
|
|
it should, but there's probably no great need for it right now (as
|
|
long as \module{marshal} continues to be used for reading and writing
|
|
code objects), and at least this avoids the possibility of smuggling
|
|
Trojan horses into a program.
|
|
\refbimodindex{marshal}
|
|
|
|
For the benefit of persistency modules written using \module{pickle}, it
|
|
supports the notion of a reference to an object outside the pickled
|
|
data stream. Such objects are referenced by a name, which is an
|
|
arbitrary string of printable \ASCII{} characters. The resolution of
|
|
such names is not defined by the \module{pickle} module --- the
|
|
persistent object module will have to implement a method
|
|
\method{persistent_load()}. To write references to persistent objects,
|
|
the persistent module must define a method \method{persistent_id()} which
|
|
returns either \code{None} or the persistent ID of the object.
|
|
|
|
There are some restrictions on the pickling of class instances.
|
|
|
|
First of all, the class must be defined at the top level in a module.
|
|
Furthermore, all its instance variables must be picklable.
|
|
|
|
\setindexsubitem{(pickle protocol)}
|
|
|
|
When a pickled class instance is unpickled, its \method{__init__()} method
|
|
is normally \emph{not} invoked. \strong{Note:} This is a deviation
|
|
from previous versions of this module; the change was introduced in
|
|
Python 1.5b2. The reason for the change is that in many cases it is
|
|
desirable to have a constructor that requires arguments; it is a
|
|
(minor) nuisance to have to provide a \method{__getinitargs__()} method.
|
|
|
|
If it is desirable that the \method{__init__()} method be called on
|
|
unpickling, a class can define a method \method{__getinitargs__()},
|
|
which should return a \emph{tuple} containing the arguments to be
|
|
passed to the class constructor (\method{__init__()}). This method is
|
|
called at pickle time; the tuple it returns is incorporated in the
|
|
pickle for the instance.
|
|
\ttindex{__getinitargs__()}
|
|
\ttindex{__init__()}
|
|
|
|
Classes can further influence how their instances are pickled --- if the class
|
|
defines the method \method{__getstate__()}, it is called and the return
|
|
state is pickled as the contents for the instance, and if the class
|
|
defines the method \method{__setstate__()}, it is called with the
|
|
unpickled state. (Note that these methods can also be used to
|
|
implement copying class instances.) If there is no
|
|
\method{__getstate__()} method, the instance's \member{__dict__} is
|
|
pickled. If there is no \method{__setstate__()} method, the pickled
|
|
object must be a dictionary and its items are assigned to the new
|
|
instance's dictionary. (If a class defines both \method{__getstate__()}
|
|
and \method{__setstate__()}, the state object needn't be a dictionary
|
|
--- these methods can do what they want.) This protocol is also used
|
|
by the shallow and deep copying operations defined in the \module{copy}
|
|
module.\refstmodindex{copy}
|
|
\ttindex{__getstate__()}
|
|
\ttindex{__setstate__()}
|
|
\ttindex{__dict__}
|
|
|
|
Note that when class instances are pickled, their class's code and
|
|
data are not pickled along with them. Only the instance data are
|
|
pickled. This is done on purpose, so you can fix bugs in a class or
|
|
add methods and still load objects that were created with an earlier
|
|
version of the class. If you plan to have long-lived objects that
|
|
will see many versions of a class, it may be worthwhile to put a version
|
|
number in the objects so that suitable conversions can be made by the
|
|
class's \method{__setstate__()} method.
|
|
|
|
When a class itself is pickled, only its name is pickled --- the class
|
|
definition is not pickled, but re-imported by the unpickling process.
|
|
Therefore, the restriction that the class must be defined at the top
|
|
level in a module applies to pickled classes as well.
|
|
|
|
\setindexsubitem{(in module pickle)}
|
|
|
|
The interface can be summarized as follows.
|
|
|
|
To pickle an object \code{x} onto a file \code{f}, open for writing:
|
|
|
|
\begin{verbatim}
|
|
p = pickle.Pickler(f)
|
|
p.dump(x)
|
|
\end{verbatim}
|
|
|
|
A shorthand for this is:
|
|
|
|
\begin{verbatim}
|
|
pickle.dump(x, f)
|
|
\end{verbatim}
|
|
|
|
To unpickle an object \code{x} from a file \code{f}, open for reading:
|
|
|
|
\begin{verbatim}
|
|
u = pickle.Unpickler(f)
|
|
x = u.load()
|
|
\end{verbatim}
|
|
|
|
A shorthand is:
|
|
|
|
\begin{verbatim}
|
|
x = pickle.load(f)
|
|
\end{verbatim}
|
|
|
|
The \class{Pickler} class only calls the method \code{f.write()} with a
|
|
string argument. The \class{Unpickler} calls the methods \code{f.read()}
|
|
(with an integer argument) and \code{f.readline()} (without argument),
|
|
both returning a string. It is explicitly allowed to pass non-file
|
|
objects here, as long as they have the right methods.
|
|
\ttindex{Unpickler}
|
|
\ttindex{Pickler}
|
|
|
|
The constructor for the \class{Pickler} class has an optional second
|
|
argument, \var{bin}. If this is present and nonzero, the binary
|
|
pickle format is used; if it is zero or absent, the (less efficient,
|
|
but backwards compatible) text pickle format is used. The
|
|
\class{Unpickler} class does not have an argument to distinguish
|
|
between binary and text pickle formats; it accepts either format.
|
|
|
|
The following types can be pickled:
|
|
\begin{itemize}
|
|
|
|
\item \code{None}
|
|
|
|
\item integers, long integers, floating point numbers
|
|
|
|
\item strings
|
|
|
|
\item tuples, lists and dictionaries containing only picklable objects
|
|
|
|
\item classes that are defined at the top level in a module
|
|
|
|
\item instances of such classes whose \member{__dict__} or
|
|
\method{__setstate__()} is picklable
|
|
|
|
\end{itemize}
|
|
|
|
Attempts to pickle unpicklable objects will raise the
|
|
\exception{PicklingError} exception; when this happens, an unspecified
|
|
number of bytes may have been written to the file.
|
|
|
|
It is possible to make multiple calls to the \method{dump()} method of
|
|
the same \class{Pickler} instance. These must then be matched to the
|
|
same number of calls to the \method{load()} method of the
|
|
corresponding \class{Unpickler} instance. If the same object is
|
|
pickled by multiple \method{dump()} calls, the \method{load()} will all
|
|
yield references to the same object. \emph{Warning}: this is intended
|
|
for pickling multiple objects without intervening modifications to the
|
|
objects or their parts. If you modify an object and then pickle it
|
|
again using the same \class{Pickler} instance, the object is not
|
|
pickled again --- a reference to it is pickled and the
|
|
\class{Unpickler} will return the old value, not the modified one.
|
|
(There are two problems here: (a) detecting changes, and (b)
|
|
marshalling a minimal set of changes. I have no answers. Garbage
|
|
Collection may also become a problem here.)
|
|
|
|
Apart from the \class{Pickler} and \class{Unpickler} classes, the
|
|
module defines the following functions, and an exception:
|
|
|
|
\begin{funcdesc}{dump}{object, file\optional{, bin}}
|
|
Write a pickled representation of \var{obect} to the open file object
|
|
\var{file}. This is equivalent to
|
|
\samp{Pickler(\var{file}, \var{bin}).dump(\var{object})}.
|
|
If the optional \var{bin} argument is present and nonzero, the binary
|
|
pickle format is used; if it is zero or absent, the (less efficient)
|
|
text pickle format is used.
|
|
\end{funcdesc}
|
|
|
|
\begin{funcdesc}{load}{file}
|
|
Read a pickled object from the open file object \var{file}. This is
|
|
equivalent to \samp{Unpickler(\var{file}).load()}.
|
|
\end{funcdesc}
|
|
|
|
\begin{funcdesc}{dumps}{object\optional{, bin}}
|
|
Return the pickled representation of the object as a string, instead
|
|
of writing it to a file. If the optional \var{bin} argument is
|
|
present and nonzero, the binary pickle format is used; if it is zero
|
|
or absent, the (less efficient) text pickle format is used.
|
|
\end{funcdesc}
|
|
|
|
\begin{funcdesc}{loads}{string}
|
|
Read a pickled object from a string instead of a file. Characters in
|
|
the string past the pickled object's representation are ignored.
|
|
\end{funcdesc}
|
|
|
|
\begin{excdesc}{PicklingError}
|
|
This exception is raised when an unpicklable object is passed to
|
|
\code{Pickler.dump()}.
|
|
\end{excdesc}
|
|
|
|
|
|
\begin{seealso}
|
|
\seemodule[copyreg]{copy_reg}{pickle interface constructor
|
|
registration}
|
|
|
|
\seemodule{shelve}{indexed databases of objects; uses \module{pickle}}
|
|
|
|
\seemodule{copy}{shallow and deep object copying}
|
|
|
|
\seemodule{marshal}{high-performance serialization of built-in types}
|
|
\end{seealso}
|
|
|
|
|
|
\section{\module{cPickle} ---
|
|
Alternate implementation of \module{pickle}.}
|
|
\declaremodule{builtin}{cPickle}
|
|
|
|
\modulesynopsis{Faster version of \module{pickle}, but not subclassable.}
|
|
|
|
|
|
% This section was written by Fred L. Drake, Jr. <fdrake@acm.org>
|
|
|
|
The \module{cPickle} module provides a similar interface and identical
|
|
functionality as the \module{pickle} module, but can be up to 1000
|
|
times faster since it is implemented in \C{}. The only other
|
|
important difference to note is that \function{Pickler()} and
|
|
\function{Unpickler()} are functions and not classes, and so cannot be
|
|
subclassed. This should not be an issue in most cases.
|
|
|
|
The format of the pickle data is identical to that produced using the
|
|
\module{pickle} module, so it is possible to use \module{pickle} and
|
|
\module{cPickle} interchangably with existing pickles.
|
|
|
|
(Since the pickle data format is actually a tiny stack-oriented
|
|
programming language, and there are some freedoms in the encodings of
|
|
certain objects, it's possible that the two modules produce different
|
|
pickled data for the same input objects; however they will always be
|
|
able to read each others pickles back in.)
|