2007-08-15 11:28:22 -03:00
|
|
|
:mod:`pickle` --- Python object serialization
|
|
|
|
=============================================
|
|
|
|
|
|
|
|
.. index::
|
|
|
|
single: persistence
|
|
|
|
pair: persistent; objects
|
|
|
|
pair: serializing; objects
|
|
|
|
pair: marshalling; objects
|
|
|
|
pair: flattening; objects
|
|
|
|
pair: pickling; objects
|
|
|
|
|
|
|
|
.. module:: pickle
|
|
|
|
:synopsis: Convert Python objects to streams of bytes and back.
|
Merged revisions 59605-59624 via svnmerge from
svn+ssh://pythondev@svn.python.org/python/trunk
........
r59606 | georg.brandl | 2007-12-29 11:57:00 +0100 (Sat, 29 Dec 2007) | 2 lines
Some cleanup in the docs.
........
r59611 | martin.v.loewis | 2007-12-29 19:49:21 +0100 (Sat, 29 Dec 2007) | 2 lines
Bug #1699: Define _BSD_SOURCE only on OpenBSD.
........
r59612 | raymond.hettinger | 2007-12-29 23:09:34 +0100 (Sat, 29 Dec 2007) | 1 line
Simpler documentation for itertools.tee(). Should be backported.
........
r59613 | raymond.hettinger | 2007-12-29 23:16:24 +0100 (Sat, 29 Dec 2007) | 1 line
Improve docs for itertools.groupby(). The use of xrange(0) to create a unique object is less obvious than object().
........
r59620 | christian.heimes | 2007-12-31 15:47:07 +0100 (Mon, 31 Dec 2007) | 3 lines
Added wininst-9.0.exe executable for VS 2008
Integrated bdist_wininst into PCBuild9 directory
........
r59621 | christian.heimes | 2007-12-31 15:51:18 +0100 (Mon, 31 Dec 2007) | 1 line
Moved PCbuild directory to PC/VS7.1
........
r59622 | christian.heimes | 2007-12-31 15:59:26 +0100 (Mon, 31 Dec 2007) | 1 line
Fix paths for build bot
........
r59623 | christian.heimes | 2007-12-31 16:02:41 +0100 (Mon, 31 Dec 2007) | 1 line
Fix paths for build bot, part 2
........
r59624 | christian.heimes | 2007-12-31 16:18:55 +0100 (Mon, 31 Dec 2007) | 1 line
Renamed PCBuild9 directory to PCBuild
........
2007-12-31 12:14:33 -04:00
|
|
|
.. sectionauthor:: Jim Kerr <jbkerr@sr.hp.com>.
|
|
|
|
.. sectionauthor:: Barry Warsaw <barry@zope.com>
|
2007-08-15 11:28:22 -03:00
|
|
|
|
|
|
|
The :mod:`pickle` module implements a fundamental, but powerful algorithm for
|
|
|
|
serializing and de-serializing a Python object structure. "Pickling" is the
|
|
|
|
process whereby a Python object hierarchy is converted into a byte stream, and
|
|
|
|
"unpickling" is the inverse operation, whereby a byte stream is converted back
|
|
|
|
into an object hierarchy. Pickling (and unpickling) is alternatively known as
|
|
|
|
"serialization", "marshalling," [#]_ or "flattening", however, to avoid
|
2008-06-20 18:03:22 -03:00
|
|
|
confusion, the terms used here are "pickling" and "unpickling"..
|
2007-08-15 11:28:22 -03:00
|
|
|
|
|
|
|
|
|
|
|
Relationship to other Python modules
|
|
|
|
------------------------------------
|
|
|
|
|
2008-06-20 18:03:22 -03:00
|
|
|
The :mod:`pickle` module has an transparent optimizer (:mod:`_pickle`) written
|
2008-10-18 17:47:58 -03:00
|
|
|
in C. It is used whenever available. Otherwise the pure Python implementation is
|
2008-06-20 18:03:22 -03:00
|
|
|
used.
|
2007-08-15 11:28:22 -03:00
|
|
|
|
|
|
|
Python has a more primitive serialization module called :mod:`marshal`, but in
|
|
|
|
general :mod:`pickle` should always be the preferred way to serialize Python
|
|
|
|
objects. :mod:`marshal` exists primarily to support Python's :file:`.pyc`
|
|
|
|
files.
|
|
|
|
|
|
|
|
The :mod:`pickle` module differs from :mod:`marshal` several significant ways:
|
|
|
|
|
|
|
|
* The :mod:`pickle` module keeps track of the objects it has already serialized,
|
|
|
|
so that later references to the same object won't be serialized again.
|
|
|
|
:mod:`marshal` doesn't do this.
|
|
|
|
|
|
|
|
This has implications both for recursive objects and object sharing. Recursive
|
|
|
|
objects are objects that contain references to themselves. These are not
|
|
|
|
handled by marshal, and in fact, attempting to marshal recursive objects will
|
|
|
|
crash your Python interpreter. Object sharing happens when there are multiple
|
|
|
|
references to the same object in different places in the object hierarchy being
|
|
|
|
serialized. :mod:`pickle` stores such objects only once, and ensures that all
|
|
|
|
other references point to the master copy. Shared objects remain shared, which
|
|
|
|
can be very important for mutable objects.
|
|
|
|
|
|
|
|
* :mod:`marshal` cannot be used to serialize user-defined classes and their
|
|
|
|
instances. :mod:`pickle` can save and restore class instances transparently,
|
|
|
|
however the class definition must be importable and live in the same module as
|
|
|
|
when the object was stored.
|
|
|
|
|
|
|
|
* The :mod:`marshal` serialization format is not guaranteed to be portable
|
|
|
|
across Python versions. Because its primary job in life is to support
|
|
|
|
:file:`.pyc` files, the Python implementers reserve the right to change the
|
|
|
|
serialization format in non-backwards compatible ways should the need arise.
|
|
|
|
The :mod:`pickle` serialization format is guaranteed to be backwards compatible
|
|
|
|
across Python releases.
|
|
|
|
|
|
|
|
.. warning::
|
|
|
|
|
|
|
|
The :mod:`pickle` module is not intended to be secure against erroneous or
|
|
|
|
maliciously constructed data. Never unpickle data received from an untrusted or
|
|
|
|
unauthenticated source.
|
|
|
|
|
|
|
|
Note that serialization is a more primitive notion than persistence; although
|
|
|
|
:mod:`pickle` reads and writes file objects, it does not handle the issue of
|
|
|
|
naming persistent objects, nor the (even more complicated) issue of concurrent
|
|
|
|
access to persistent objects. The :mod:`pickle` module can transform a complex
|
|
|
|
object into a byte stream and it can transform the byte stream into an object
|
|
|
|
with the same internal structure. Perhaps the most obvious thing to do with
|
|
|
|
these byte streams is to write them onto a file, but it is also conceivable to
|
|
|
|
send them across a network or store them in a database. The module
|
|
|
|
:mod:`shelve` provides a simple interface to pickle and unpickle objects on
|
|
|
|
DBM-style database files.
|
|
|
|
|
|
|
|
|
|
|
|
Data stream format
|
|
|
|
------------------
|
|
|
|
|
|
|
|
.. index::
|
|
|
|
single: XDR
|
|
|
|
single: External Data Representation
|
|
|
|
|
|
|
|
The data format used by :mod:`pickle` is Python-specific. This has the
|
|
|
|
advantage that there are no restrictions imposed by external standards such as
|
|
|
|
XDR (which can't represent pointer sharing); however it means that non-Python
|
|
|
|
programs may not be able to reconstruct pickled Python objects.
|
|
|
|
|
2008-10-18 16:25:07 -03:00
|
|
|
By default, the :mod:`pickle` data format uses a compact binary representation.
|
|
|
|
The module :mod:`pickletools` contains tools for analyzing data streams
|
|
|
|
generated by :mod:`pickle`.
|
2007-08-15 11:28:22 -03:00
|
|
|
|
2008-04-06 05:39:37 -03:00
|
|
|
There are currently 4 different protocols which can be used for pickling.
|
2007-08-15 11:28:22 -03:00
|
|
|
|
2009-01-23 00:50:05 -04:00
|
|
|
* Protocol version 0 is the original human-readable protocol and is
|
|
|
|
backwards compatible with earlier versions of Python.
|
2007-08-15 11:28:22 -03:00
|
|
|
|
|
|
|
* Protocol version 1 is the old binary format which is also compatible with
|
|
|
|
earlier versions of Python.
|
|
|
|
|
|
|
|
* Protocol version 2 was introduced in Python 2.3. It provides much more
|
#1370: Finish the merge r58749, log below, by resolving all conflicts in Doc/.
Merged revisions 58221-58741 via svnmerge from
svn+ssh://pythondev@svn.python.org/python/trunk
........
r58221 | georg.brandl | 2007-09-20 10:57:59 -0700 (Thu, 20 Sep 2007) | 2 lines
Patch #1181: add os.environ.clear() method.
........
r58225 | sean.reifschneider | 2007-09-20 23:33:28 -0700 (Thu, 20 Sep 2007) | 3 lines
Issue1704287: "make install" fails unless you do "make" first. Make
oldsharedmods and sharedmods in "libinstall".
........
r58232 | guido.van.rossum | 2007-09-22 13:18:03 -0700 (Sat, 22 Sep 2007) | 4 lines
Patch # 188 by Philip Jenvey.
Make tell() mark CRLF as a newline.
With unit test.
........
r58242 | georg.brandl | 2007-09-24 10:55:47 -0700 (Mon, 24 Sep 2007) | 2 lines
Fix typo and double word.
........
r58245 | georg.brandl | 2007-09-24 10:59:28 -0700 (Mon, 24 Sep 2007) | 2 lines
#1196: document default radix for int().
........
r58247 | georg.brandl | 2007-09-24 11:08:24 -0700 (Mon, 24 Sep 2007) | 2 lines
#1177: accept 2xx responses for https too, not only http.
........
r58249 | andrew.kuchling | 2007-09-24 16:45:51 -0700 (Mon, 24 Sep 2007) | 1 line
Remove stray odd character; grammar fix
........
r58250 | andrew.kuchling | 2007-09-24 16:46:28 -0700 (Mon, 24 Sep 2007) | 1 line
Typo fix
........
r58251 | andrew.kuchling | 2007-09-24 17:09:42 -0700 (Mon, 24 Sep 2007) | 1 line
Add various items
........
r58268 | vinay.sajip | 2007-09-26 22:34:45 -0700 (Wed, 26 Sep 2007) | 1 line
Change to flush and close logic to fix #1760556.
........
r58269 | vinay.sajip | 2007-09-26 22:38:51 -0700 (Wed, 26 Sep 2007) | 1 line
Change to basicConfig() to fix #1021.
........
r58270 | georg.brandl | 2007-09-26 23:26:58 -0700 (Wed, 26 Sep 2007) | 2 lines
#1208: document match object's boolean value.
........
r58271 | vinay.sajip | 2007-09-26 23:56:13 -0700 (Wed, 26 Sep 2007) | 1 line
Minor date change.
........
r58272 | vinay.sajip | 2007-09-27 00:35:10 -0700 (Thu, 27 Sep 2007) | 1 line
Change to LogRecord.__init__() to fix #1206. Note that archaic use of type(x) == types.DictType is because of keeping 1.5.2 compatibility. While this is much less relevant these days, there probably needs to be a separate commit for removing all archaic constructs at the same time.
........
r58288 | brett.cannon | 2007-09-30 12:45:10 -0700 (Sun, 30 Sep 2007) | 9 lines
tuple.__repr__ did not consider a reference loop as it is not possible from
Python code; but it is possible from C. object.__str__ had the issue of not
expecting a type to doing something within it's tp_str implementation that
could trigger an infinite recursion, but it could in C code.. Both found
thanks to BaseException and how it handles its repr.
Closes issue #1686386. Thanks to Thomas Herve for taking an initial stab at
coming up with a solution.
........
r58289 | brett.cannon | 2007-09-30 13:37:19 -0700 (Sun, 30 Sep 2007) | 3 lines
Fix error introduced by r58288; if a tuple is length 0 return its repr and
don't worry about any self-referring tuples.
........
r58294 | facundo.batista | 2007-10-02 10:01:24 -0700 (Tue, 02 Oct 2007) | 11 lines
Made the various is_* operations return booleans. This was discussed
with Cawlishaw by mail, and he basically confirmed that to these is_*
operations, there's no need to return Decimal(0) and Decimal(1) if
the language supports the False and True booleans.
Also added a few tests for the these functions in extra.decTest, since
they are mostly untested (apart from the doctests).
Thanks Mark Dickinson
........
r58295 | facundo.batista | 2007-10-02 11:21:18 -0700 (Tue, 02 Oct 2007) | 4 lines
Added a class to store the digits of log(10), so that they can be made
available when necessary without recomputing. Thanks Mark Dickinson
........
r58299 | mark.summerfield | 2007-10-03 01:53:21 -0700 (Wed, 03 Oct 2007) | 4 lines
Added note in footnote about string comparisons about
unicodedata.normalize().
........
r58304 | raymond.hettinger | 2007-10-03 14:18:11 -0700 (Wed, 03 Oct 2007) | 1 line
enumerate() is no longer bounded to using sequences shorter than LONG_MAX. The possibility of overflow was sending some newsgroup posters into a tizzy.
........
r58305 | raymond.hettinger | 2007-10-03 17:20:27 -0700 (Wed, 03 Oct 2007) | 1 line
itertools.count() no longer limited to sys.maxint.
........
r58306 | kurt.kaiser | 2007-10-03 18:49:54 -0700 (Wed, 03 Oct 2007) | 3 lines
Assume that the user knows when he wants to end the line; don't insert
something he didn't select or complete.
........
r58307 | kurt.kaiser | 2007-10-03 19:07:50 -0700 (Wed, 03 Oct 2007) | 2 lines
Remove unused theme that was causing a fault in p3k.
........
r58308 | kurt.kaiser | 2007-10-03 19:09:17 -0700 (Wed, 03 Oct 2007) | 2 lines
Clean up EditorWindow close.
........
r58309 | kurt.kaiser | 2007-10-03 19:53:07 -0700 (Wed, 03 Oct 2007) | 7 lines
textView cleanup. Patch 1718043 Tal Einat.
M idlelib/EditorWindow.py
M idlelib/aboutDialog.py
M idlelib/textView.py
M idlelib/NEWS.txt
........
r58310 | kurt.kaiser | 2007-10-03 20:11:12 -0700 (Wed, 03 Oct 2007) | 3 lines
configDialog cleanup. Patch 1730217 Tal Einat.
........
r58311 | neal.norwitz | 2007-10-03 23:00:48 -0700 (Wed, 03 Oct 2007) | 4 lines
Coverity #151: Remove deadcode.
All this code already exists above starting at line 653.
........
r58325 | fred.drake | 2007-10-04 19:46:12 -0700 (Thu, 04 Oct 2007) | 1 line
wrap lines to <80 characters before fixing errors
........
r58326 | raymond.hettinger | 2007-10-04 19:47:07 -0700 (Thu, 04 Oct 2007) | 6 lines
Add __asdict__() to NamedTuple and refine the docs.
Add maxlen support to deque() and fixup docs.
Partially fix __reduce__(). The None as a third arg was no longer supported.
Still needs work on __reduce__() to handle recursive inputs.
........
r58327 | fred.drake | 2007-10-04 19:48:32 -0700 (Thu, 04 Oct 2007) | 3 lines
move descriptions of ac_(in|out)_buffer_size to the right place
http://bugs.python.org/issue1053
........
r58329 | neal.norwitz | 2007-10-04 20:39:17 -0700 (Thu, 04 Oct 2007) | 3 lines
dict could be NULL, so we need to XDECREF.
Fix a compiler warning about passing a PyTypeObject* instead of PyObject*.
........
r58330 | neal.norwitz | 2007-10-04 20:41:19 -0700 (Thu, 04 Oct 2007) | 2 lines
Fix Coverity #158: Check the correct variable.
........
r58332 | neal.norwitz | 2007-10-04 22:01:38 -0700 (Thu, 04 Oct 2007) | 7 lines
Fix Coverity #159.
This code was broken if save() returned a negative number since i contained
a boolean value and then we compared i < 0 which should never be true.
Will backport (assuming it's necessary)
........
r58334 | neal.norwitz | 2007-10-04 22:29:17 -0700 (Thu, 04 Oct 2007) | 1 line
Add a note about fixing some more warnings found by Coverity.
........
r58338 | raymond.hettinger | 2007-10-05 12:07:31 -0700 (Fri, 05 Oct 2007) | 1 line
Restore BEGIN/END THREADS macros which were squashed in the previous checkin
........
r58343 | gregory.p.smith | 2007-10-06 00:48:10 -0700 (Sat, 06 Oct 2007) | 3 lines
Stab in the dark attempt to fix the test_bsddb3 failure on sparc and S-390
ubuntu buildbots.
........
r58344 | gregory.p.smith | 2007-10-06 00:51:59 -0700 (Sat, 06 Oct 2007) | 2 lines
Allows BerkeleyDB 4.6.x >= 4.6.21 for the bsddb module.
........
r58348 | gregory.p.smith | 2007-10-06 08:47:37 -0700 (Sat, 06 Oct 2007) | 3 lines
Use the host the author likely meant in the first place. pop.gmail.com is
reliable. gmail.org is someones personal domain.
........
r58351 | neal.norwitz | 2007-10-06 12:16:28 -0700 (Sat, 06 Oct 2007) | 3 lines
Ensure that this test will pass even if another test left an unwritable TESTFN.
Also use the safe unlink in test_support instead of rolling our own here.
........
r58368 | georg.brandl | 2007-10-08 00:50:24 -0700 (Mon, 08 Oct 2007) | 3 lines
#1123: fix the docs for the str.split(None, sep) case.
Also expand a few other methods' docs, which had more info in the deprecated string module docs.
........
r58369 | georg.brandl | 2007-10-08 01:06:05 -0700 (Mon, 08 Oct 2007) | 2 lines
Update docstring of sched, also remove an unused assignment.
........
r58370 | raymond.hettinger | 2007-10-08 02:14:28 -0700 (Mon, 08 Oct 2007) | 5 lines
Add comments to NamedTuple code.
Let the field spec be either a string or a non-string sequence (suggested by Martin Blais with use cases).
Improve the error message in the case of a SyntaxError (caused by a duplicate field name).
........
r58371 | raymond.hettinger | 2007-10-08 02:56:29 -0700 (Mon, 08 Oct 2007) | 1 line
Missed a line in the docs
........
r58372 | raymond.hettinger | 2007-10-08 03:11:51 -0700 (Mon, 08 Oct 2007) | 1 line
Better variable names
........
r58376 | georg.brandl | 2007-10-08 07:12:47 -0700 (Mon, 08 Oct 2007) | 3 lines
#1199: docs for tp_as_{number,sequence,mapping}, by Amaury Forgeot d'Arc.
No need to merge this to py3k!
........
r58380 | raymond.hettinger | 2007-10-08 14:26:58 -0700 (Mon, 08 Oct 2007) | 1 line
Eliminate camelcase function name
........
r58381 | andrew.kuchling | 2007-10-08 16:23:03 -0700 (Mon, 08 Oct 2007) | 1 line
Eliminate camelcase function name
........
r58382 | raymond.hettinger | 2007-10-08 18:36:23 -0700 (Mon, 08 Oct 2007) | 1 line
Make the error messages more specific
........
r58384 | gregory.p.smith | 2007-10-08 23:02:21 -0700 (Mon, 08 Oct 2007) | 10 lines
Splits Modules/_bsddb.c up into bsddb.h and _bsddb.c and adds a C API
object available as bsddb.db.api. This is based on the patch submitted
by Duncan Grisby here:
http://sourceforge.net/tracker/index.php?func=detail&aid=1551895&group_id=13900&atid=313900
See this thread for additional info:
http://sourceforge.net/mailarchive/forum.php?thread_name=E1GAVDK-0002rk-Iw%40apasphere.com&forum_name=pybsddb-users
It also cleans up the code a little by removing some ifdef/endifs for
python prior to 2.1 and for unsupported Berkeley DB <= 3.2.
........
r58385 | gregory.p.smith | 2007-10-08 23:50:43 -0700 (Mon, 08 Oct 2007) | 5 lines
Fix a double free when positioning a database cursor to a non-existant
string key (and probably a few other situations with string keys).
This was reported with a patch as pybsddb sourceforge bug 1708868 by
jjjhhhlll at gmail.
........
r58386 | gregory.p.smith | 2007-10-09 00:19:11 -0700 (Tue, 09 Oct 2007) | 3 lines
Use the highest cPickle protocol in bsddb.dbshelve. This comes from
sourceforge pybsddb patch 1551443 by w_barnes.
........
r58394 | gregory.p.smith | 2007-10-09 11:26:02 -0700 (Tue, 09 Oct 2007) | 2 lines
remove another sleepycat reference
........
r58396 | kurt.kaiser | 2007-10-09 12:31:30 -0700 (Tue, 09 Oct 2007) | 3 lines
Allow interrupt only when executing user code in subprocess
Patch 1225 Tal Einat modified from IDLE-Spoon.
........
r58399 | brett.cannon | 2007-10-09 17:07:50 -0700 (Tue, 09 Oct 2007) | 5 lines
Remove file-level typedefs that were inconsistently used throughout the file.
Just move over to the public API names.
Closes issue1238.
........
r58401 | raymond.hettinger | 2007-10-09 17:26:46 -0700 (Tue, 09 Oct 2007) | 1 line
Accept Jim Jewett's api suggestion to use None instead of -1 to indicate unbounded deques.
........
r58403 | kurt.kaiser | 2007-10-09 17:55:40 -0700 (Tue, 09 Oct 2007) | 2 lines
Allow cursor color change w/o restart. Patch 1725576 Tal Einat.
........
r58404 | kurt.kaiser | 2007-10-09 18:06:47 -0700 (Tue, 09 Oct 2007) | 2 lines
show paste if > 80 columns. Patch 1659326 Tal Einat.
........
r58415 | thomas.heller | 2007-10-11 12:51:32 -0700 (Thu, 11 Oct 2007) | 5 lines
On OS X, use os.uname() instead of gestalt.sysv(...) to get the
operating system version. This allows to use ctypes when Python
was configured with --disable-toolbox-glue.
........
r58419 | neal.norwitz | 2007-10-11 20:01:01 -0700 (Thu, 11 Oct 2007) | 1 line
Get rid of warning about not being able to create an existing directory.
........
r58420 | neal.norwitz | 2007-10-11 20:01:30 -0700 (Thu, 11 Oct 2007) | 1 line
Get rid of warnings on a bunch of platforms by using a proper prototype.
........
r58421 | neal.norwitz | 2007-10-11 20:01:54 -0700 (Thu, 11 Oct 2007) | 4 lines
Get rid of compiler warning about retval being used (returned) without
being initialized. (gcc warning and Coverity 202)
........
r58422 | neal.norwitz | 2007-10-11 20:03:23 -0700 (Thu, 11 Oct 2007) | 1 line
Fix Coverity 168: Close the file before returning (exiting).
........
r58423 | neal.norwitz | 2007-10-11 20:04:18 -0700 (Thu, 11 Oct 2007) | 4 lines
Fix Coverity 180: Don't overallocate. We don't need structs, but pointers.
Also fix a memory leak.
........
r58424 | neal.norwitz | 2007-10-11 20:05:19 -0700 (Thu, 11 Oct 2007) | 5 lines
Fix Coverity 185-186: If the passed in FILE is NULL, uninitialized memory
would be accessed.
Will backport.
........
r58425 | neal.norwitz | 2007-10-11 20:52:34 -0700 (Thu, 11 Oct 2007) | 1 line
Get this module to compile with bsddb versions prior to 4.3
........
r58430 | martin.v.loewis | 2007-10-12 01:56:52 -0700 (Fri, 12 Oct 2007) | 3 lines
Bug #1216: Restore support for Visual Studio 2002.
Will backport to 2.5.
........
r58433 | raymond.hettinger | 2007-10-12 10:53:11 -0700 (Fri, 12 Oct 2007) | 1 line
Fix test of count.__repr__() to ignore the 'L' if the count is a long
........
r58434 | gregory.p.smith | 2007-10-12 11:44:06 -0700 (Fri, 12 Oct 2007) | 4 lines
Fixes http://bugs.python.org/issue1233 - bsddb.dbshelve.DBShelf.append
was useless due to inverted logic. Also adds a test case for RECNO dbs
to test_dbshelve.
........
r58445 | georg.brandl | 2007-10-13 06:20:03 -0700 (Sat, 13 Oct 2007) | 2 lines
Fix email example.
........
r58450 | gregory.p.smith | 2007-10-13 16:02:05 -0700 (Sat, 13 Oct 2007) | 2 lines
Fix an uncollectable reference leak in bsddb.db.DBShelf.append
........
r58453 | neal.norwitz | 2007-10-13 17:18:40 -0700 (Sat, 13 Oct 2007) | 8 lines
Let the O/S supply a port if none of the default ports can be used.
This should make the tests more robust at the expense of allowing
tests to be sloppier by not requiring them to cleanup after themselves.
(It will legitamitely help when running two test suites simultaneously
or if another process is already using one of the predefined ports.)
Also simplifies (slightLy) the exception handling elsewhere.
........
r58459 | neal.norwitz | 2007-10-14 11:30:21 -0700 (Sun, 14 Oct 2007) | 2 lines
Don't raise a string exception, they don't work anymore.
........
r58460 | neal.norwitz | 2007-10-14 11:40:37 -0700 (Sun, 14 Oct 2007) | 1 line
Use unittest for assertions
........
r58468 | armin.rigo | 2007-10-15 00:48:35 -0700 (Mon, 15 Oct 2007) | 2 lines
test_bigbits was not testing what it seemed to.
........
r58471 | guido.van.rossum | 2007-10-15 08:54:11 -0700 (Mon, 15 Oct 2007) | 3 lines
Change a PyErr_Print() into a PyErr_Clear(),
per discussion in issue 1031213.
........
r58500 | raymond.hettinger | 2007-10-16 12:18:30 -0700 (Tue, 16 Oct 2007) | 1 line
Improve error messages
........
r58506 | raymond.hettinger | 2007-10-16 14:28:32 -0700 (Tue, 16 Oct 2007) | 1 line
More docs, error messages, and tests
........
r58507 | andrew.kuchling | 2007-10-16 15:58:03 -0700 (Tue, 16 Oct 2007) | 1 line
Add items
........
r58508 | brett.cannon | 2007-10-16 16:24:06 -0700 (Tue, 16 Oct 2007) | 3 lines
Remove ``:const:`` notation on None in parameter list. Since the markup is not
rendered for parameters it just showed up as ``:const:`None` `` in the output.
........
r58509 | brett.cannon | 2007-10-16 16:26:45 -0700 (Tue, 16 Oct 2007) | 3 lines
Re-order some functions whose parameters differ between PyObject and const char
* so that they are next to each other.
........
r58522 | armin.rigo | 2007-10-17 11:46:37 -0700 (Wed, 17 Oct 2007) | 5 lines
Fix the overflow checking of list_repeat.
Introduce overflow checking into list_inplace_repeat.
Backport candidate, possibly.
........
r58530 | facundo.batista | 2007-10-17 20:16:03 -0700 (Wed, 17 Oct 2007) | 7 lines
Issue #1580738. When HTTPConnection reads the whole stream with read(),
it closes itself. When the stream is read in several calls to read(n),
it should behave in the same way if HTTPConnection knows where the end
of the stream is (through self.length). Added a test case for this
behaviour.
........
r58531 | facundo.batista | 2007-10-17 20:44:48 -0700 (Wed, 17 Oct 2007) | 3 lines
Issue 1289, just a typo.
........
r58532 | gregory.p.smith | 2007-10-18 00:56:54 -0700 (Thu, 18 Oct 2007) | 4 lines
cleanup test_dbtables to use mkdtemp. cleanup dbtables to pass txn as a
keyword argument whenever possible to avoid bugs and confusion. (dbtables.py
line 447 self.db.get using txn as a non-keyword was an actual bug due to this)
........
r58533 | gregory.p.smith | 2007-10-18 01:34:20 -0700 (Thu, 18 Oct 2007) | 4 lines
Fix a weird bug in dbtables: if it chose a random rowid string that contained
NULL bytes it would cause the database all sorts of problems in the future
leading to very strange random failures and corrupt dbtables.bsdTableDb dbs.
........
r58534 | gregory.p.smith | 2007-10-18 09:32:02 -0700 (Thu, 18 Oct 2007) | 3 lines
A cleaner fix than the one committed last night. Generate random rowids that
do not contain null bytes.
........
r58537 | gregory.p.smith | 2007-10-18 10:17:57 -0700 (Thu, 18 Oct 2007) | 2 lines
mention bsddb fixes.
........
r58538 | raymond.hettinger | 2007-10-18 14:13:06 -0700 (Thu, 18 Oct 2007) | 1 line
Remove useless warning
........
r58539 | gregory.p.smith | 2007-10-19 00:31:20 -0700 (Fri, 19 Oct 2007) | 2 lines
squelch the warning that this test is supposed to trigger.
........
r58542 | georg.brandl | 2007-10-19 05:32:39 -0700 (Fri, 19 Oct 2007) | 2 lines
Clarify wording for apply().
........
r58544 | mark.summerfield | 2007-10-19 05:48:17 -0700 (Fri, 19 Oct 2007) | 3 lines
Added a cross-ref to each other.
........
r58545 | georg.brandl | 2007-10-19 10:38:49 -0700 (Fri, 19 Oct 2007) | 2 lines
#1284: "S" means "seen", not unread.
........
r58548 | thomas.heller | 2007-10-19 11:11:41 -0700 (Fri, 19 Oct 2007) | 4 lines
Fix ctypes on 32-bit systems when Python is configured --with-system-ffi.
See also https://bugs.launchpad.net/bugs/72505.
Ported from release25-maint branch.
........
r58550 | facundo.batista | 2007-10-19 12:25:57 -0700 (Fri, 19 Oct 2007) | 8 lines
The constructor from tuple was way too permissive: it allowed bad
coefficient numbers, floats in the sign, and other details that
generated directly the wrong number in the best case, or triggered
misfunctionality in the alorithms.
Test cases added for these issues. Thanks Mark Dickinson.
........
r58559 | georg.brandl | 2007-10-20 06:22:53 -0700 (Sat, 20 Oct 2007) | 2 lines
Fix code being interpreted as a target.
........
r58561 | georg.brandl | 2007-10-20 06:36:24 -0700 (Sat, 20 Oct 2007) | 2 lines
Document new "cmdoption" directive.
........
r58562 | georg.brandl | 2007-10-20 08:21:22 -0700 (Sat, 20 Oct 2007) | 2 lines
Make a path more Unix-standardy.
........
r58564 | georg.brandl | 2007-10-20 10:51:39 -0700 (Sat, 20 Oct 2007) | 2 lines
Document new directive "envvar".
........
r58567 | georg.brandl | 2007-10-20 11:08:14 -0700 (Sat, 20 Oct 2007) | 6 lines
* Add new toplevel chapter, "Using Python." (how to install,
configure and setup python on different platforms -- at least
in theory.)
* Move the Python on Mac docs in that chapter.
* Add a new chapter about the command line invocation, by stargaming.
........
r58568 | georg.brandl | 2007-10-20 11:33:20 -0700 (Sat, 20 Oct 2007) | 2 lines
Change title, for now.
........
r58569 | georg.brandl | 2007-10-20 11:39:25 -0700 (Sat, 20 Oct 2007) | 2 lines
Add entry to ACKS.
........
r58570 | georg.brandl | 2007-10-20 12:05:45 -0700 (Sat, 20 Oct 2007) | 2 lines
Clarify -E docs.
........
r58571 | georg.brandl | 2007-10-20 12:08:36 -0700 (Sat, 20 Oct 2007) | 2 lines
Even more clarification.
........
r58572 | andrew.kuchling | 2007-10-20 12:25:37 -0700 (Sat, 20 Oct 2007) | 1 line
Fix protocol name
........
r58573 | andrew.kuchling | 2007-10-20 12:35:18 -0700 (Sat, 20 Oct 2007) | 1 line
Various items
........
r58574 | andrew.kuchling | 2007-10-20 12:39:35 -0700 (Sat, 20 Oct 2007) | 1 line
Use correct header line
........
r58576 | armin.rigo | 2007-10-21 02:14:15 -0700 (Sun, 21 Oct 2007) | 3 lines
Add a crasher for the long-standing issue with closing a file
while another thread uses it.
........
r58577 | georg.brandl | 2007-10-21 03:01:56 -0700 (Sun, 21 Oct 2007) | 2 lines
Remove duplicate crasher.
........
r58578 | georg.brandl | 2007-10-21 03:24:20 -0700 (Sun, 21 Oct 2007) | 2 lines
Unify "byte code" to "bytecode". Also sprinkle :term: markup for it.
........
r58579 | georg.brandl | 2007-10-21 03:32:54 -0700 (Sun, 21 Oct 2007) | 2 lines
Add markup to new function descriptions.
........
r58580 | georg.brandl | 2007-10-21 03:45:46 -0700 (Sun, 21 Oct 2007) | 2 lines
Add :term:s for descriptors.
........
r58581 | georg.brandl | 2007-10-21 03:46:24 -0700 (Sun, 21 Oct 2007) | 2 lines
Unify "file-descriptor" to "file descriptor".
........
r58582 | georg.brandl | 2007-10-21 03:52:38 -0700 (Sun, 21 Oct 2007) | 2 lines
Add :term: for generators.
........
r58583 | georg.brandl | 2007-10-21 05:10:28 -0700 (Sun, 21 Oct 2007) | 2 lines
Add :term:s for iterator.
........
r58584 | georg.brandl | 2007-10-21 05:15:05 -0700 (Sun, 21 Oct 2007) | 2 lines
Add :term:s for "new-style class".
........
r58588 | neal.norwitz | 2007-10-21 21:47:54 -0700 (Sun, 21 Oct 2007) | 1 line
Add Chris Monson so he can edit PEPs.
........
r58594 | guido.van.rossum | 2007-10-22 09:27:19 -0700 (Mon, 22 Oct 2007) | 4 lines
Issue #1307, patch by Derek Shockey.
When "MAIL" is received without args, an exception happens instead of
sending a 501 syntax error response.
........
r58598 | travis.oliphant | 2007-10-22 19:40:56 -0700 (Mon, 22 Oct 2007) | 1 line
Add phuang patch from Issue 708374 which adds offset parameter to mmap module.
........
r58601 | neal.norwitz | 2007-10-22 22:44:27 -0700 (Mon, 22 Oct 2007) | 2 lines
Bug #1313, fix typo (wrong variable name) in example.
........
r58609 | georg.brandl | 2007-10-23 11:21:35 -0700 (Tue, 23 Oct 2007) | 2 lines
Update Pygments version from externals.
........
r58618 | guido.van.rossum | 2007-10-23 12:25:41 -0700 (Tue, 23 Oct 2007) | 3 lines
Issue 1307 by Derek Shockey, fox the same bug for RCPT.
Neal: please backport!
........
r58620 | raymond.hettinger | 2007-10-23 13:37:41 -0700 (Tue, 23 Oct 2007) | 1 line
Shorter name for namedtuple()
........
r58621 | andrew.kuchling | 2007-10-23 13:55:47 -0700 (Tue, 23 Oct 2007) | 1 line
Update name
........
r58622 | raymond.hettinger | 2007-10-23 14:23:07 -0700 (Tue, 23 Oct 2007) | 1 line
Fixup news entry
........
r58623 | raymond.hettinger | 2007-10-23 18:28:33 -0700 (Tue, 23 Oct 2007) | 1 line
Optimize sum() for integer and float inputs.
........
r58624 | raymond.hettinger | 2007-10-23 19:05:51 -0700 (Tue, 23 Oct 2007) | 1 line
Fixup error return and add support for intermixed ints and floats/
........
r58628 | vinay.sajip | 2007-10-24 03:47:06 -0700 (Wed, 24 Oct 2007) | 1 line
Bug #1321: Fixed logic error in TimedRotatingFileHandler.__init__()
........
r58641 | facundo.batista | 2007-10-24 12:11:08 -0700 (Wed, 24 Oct 2007) | 4 lines
Issue 1290. CharacterData.__repr__ was constructing a string
in response that keeped having a non-ascii character.
........
r58643 | thomas.heller | 2007-10-24 12:50:45 -0700 (Wed, 24 Oct 2007) | 1 line
Added unittest for calling a function with paramflags (backport from py3k branch).
........
r58645 | matthias.klose | 2007-10-24 13:00:44 -0700 (Wed, 24 Oct 2007) | 2 lines
- Build using system ffi library on arm*-linux*.
........
r58651 | georg.brandl | 2007-10-24 14:40:38 -0700 (Wed, 24 Oct 2007) | 2 lines
Bug #1287: make os.environ.pop() work as expected.
........
r58652 | raymond.hettinger | 2007-10-24 19:26:58 -0700 (Wed, 24 Oct 2007) | 1 line
Missing DECREFs
........
r58653 | matthias.klose | 2007-10-24 23:37:24 -0700 (Wed, 24 Oct 2007) | 2 lines
- Build using system ffi library on arm*-linux*, pass --with-system-ffi to CONFIG_ARGS
........
r58655 | thomas.heller | 2007-10-25 12:47:32 -0700 (Thu, 25 Oct 2007) | 2 lines
ffi_type_longdouble may be already #defined.
See issue 1324.
........
r58656 | kurt.kaiser | 2007-10-25 15:43:45 -0700 (Thu, 25 Oct 2007) | 3 lines
Correct an ancient bug in an unused path by removing that path: register() is
now idempotent.
........
r58660 | kurt.kaiser | 2007-10-25 17:10:09 -0700 (Thu, 25 Oct 2007) | 4 lines
1. Add comments to provide top-level documentation.
2. Refactor to use more descriptive names.
3. Enhance tests in main().
........
r58675 | georg.brandl | 2007-10-26 11:30:41 -0700 (Fri, 26 Oct 2007) | 2 lines
Fix new pop() method on os.environ on ignorecase-platforms.
........
r58696 | neal.norwitz | 2007-10-27 15:32:21 -0700 (Sat, 27 Oct 2007) | 1 line
Update URL for Pygments. 0.8.1 is no longer available
........
r58697 | hyeshik.chang | 2007-10-28 04:19:02 -0700 (Sun, 28 Oct 2007) | 3 lines
- Add support for FreeBSD 8 which is recently forked from FreeBSD 7.
- Regenerate IN module for most recent maintenance tree of FreeBSD 6 and 7.
........
r58698 | hyeshik.chang | 2007-10-28 05:38:09 -0700 (Sun, 28 Oct 2007) | 2 lines
Enable platform-specific tweaks for FreeBSD 8 (exactly same to FreeBSD 7's yet)
........
r58700 | kurt.kaiser | 2007-10-28 12:03:59 -0700 (Sun, 28 Oct 2007) | 2 lines
Add confirmation dialog before printing. Patch 1717170 Tal Einat.
........
r58706 | guido.van.rossum | 2007-10-29 13:52:45 -0700 (Mon, 29 Oct 2007) | 3 lines
Patch 1353 by Jacob Winther.
Add mp4 mapping to mimetypes.py.
........
r58709 | guido.van.rossum | 2007-10-29 15:15:05 -0700 (Mon, 29 Oct 2007) | 6 lines
Backport fixes for the code that decodes octal escapes (and for PyString
also hex escapes) -- this was reaching beyond the end of the input string
buffer, even though it is not supposed to be \0-terminated.
This has no visible effect but is clearly the correct thing to do.
(In 3.0 it had a visible effect after removing ob_sstate from PyString.)
........
r58710 | kurt.kaiser | 2007-10-29 19:38:54 -0700 (Mon, 29 Oct 2007) | 7 lines
check in Tal Einat's update to tabpage.py
Patch 1612746
M configDialog.py
M NEWS.txt
AM tabbedpages.py
........
r58715 | georg.brandl | 2007-10-30 10:51:18 -0700 (Tue, 30 Oct 2007) | 2 lines
Use correct markup.
........
r58716 | georg.brandl | 2007-10-30 10:57:12 -0700 (Tue, 30 Oct 2007) | 2 lines
Make example about hiding None return values at the prompt clearer.
........
r58728 | neal.norwitz | 2007-10-30 23:33:20 -0700 (Tue, 30 Oct 2007) | 1 line
Fix some compiler warnings for signed comparisons on Unix and Windows.
........
r58731 | martin.v.loewis | 2007-10-31 10:19:33 -0700 (Wed, 31 Oct 2007) | 2 lines
Adding Christian Heimes.
........
r58737 | raymond.hettinger | 2007-10-31 14:57:58 -0700 (Wed, 31 Oct 2007) | 1 line
Clarify the reasons why pickle is almost always better than marshal
........
r58739 | raymond.hettinger | 2007-10-31 15:15:49 -0700 (Wed, 31 Oct 2007) | 1 line
Sets are marshalable.
........
2007-11-01 17:32:30 -03:00
|
|
|
efficient pickling of :term:`new-style class`\es.
|
2007-08-15 11:28:22 -03:00
|
|
|
|
2008-04-06 05:39:37 -03:00
|
|
|
* Protocol version 3 was added in Python 3.0. It has explicit support for
|
2008-10-18 17:47:58 -03:00
|
|
|
bytes and cannot be unpickled by Python 2.x pickle modules. This is
|
2008-10-18 16:25:07 -03:00
|
|
|
the current recommended protocol, use it whenever it is possible.
|
2008-04-06 05:39:37 -03:00
|
|
|
|
2008-10-23 22:32:40 -03:00
|
|
|
Refer to :pep:`307` for information about improvements brought by
|
|
|
|
protocol 2. See :mod:`pickletools`'s source code for extensive
|
|
|
|
comments about opcodes used by pickle protocols.
|
2007-08-15 11:28:22 -03:00
|
|
|
|
|
|
|
|
2008-10-23 22:32:40 -03:00
|
|
|
Module Interface
|
|
|
|
----------------
|
2007-08-15 11:28:22 -03:00
|
|
|
|
|
|
|
To serialize an object hierarchy, you first create a pickler, then you call the
|
|
|
|
pickler's :meth:`dump` method. To de-serialize a data stream, you first create
|
|
|
|
an unpickler, then you call the unpickler's :meth:`load` method. The
|
|
|
|
:mod:`pickle` module provides the following constant:
|
|
|
|
|
|
|
|
|
|
|
|
.. data:: HIGHEST_PROTOCOL
|
|
|
|
|
|
|
|
The highest protocol version available. This value can be passed as a
|
|
|
|
*protocol* value.
|
|
|
|
|
2008-10-18 16:25:07 -03:00
|
|
|
.. data:: DEFAULT_PROTOCOL
|
|
|
|
|
|
|
|
The default protocol used for pickling. May be less than HIGHEST_PROTOCOL.
|
|
|
|
Currently the default protocol is 3; a backward-incompatible protocol
|
|
|
|
designed for Python 3.0.
|
|
|
|
|
|
|
|
|
2007-08-15 11:28:22 -03:00
|
|
|
The :mod:`pickle` module provides the following functions to make the pickling
|
|
|
|
process more convenient:
|
|
|
|
|
|
|
|
.. function:: dump(obj, file[, protocol])
|
|
|
|
|
2008-10-18 16:25:07 -03:00
|
|
|
Write a pickled representation of *obj* to the open file object *file*. This
|
|
|
|
is equivalent to ``Pickler(file, protocol).dump(obj)``.
|
2007-08-15 11:28:22 -03:00
|
|
|
|
2008-10-18 16:25:07 -03:00
|
|
|
The optional *protocol* argument tells the pickler to use the given protocol;
|
|
|
|
supported protocols are 0, 1, 2, 3. The default protocol is 3; a
|
|
|
|
backward-incompatible protocol designed for Python 3.0.
|
2007-08-15 11:28:22 -03:00
|
|
|
|
2008-10-18 16:25:07 -03:00
|
|
|
Specifying a negative protocol version selects the highest protocol version
|
|
|
|
supported. The higher the protocol used, the more recent the version of
|
|
|
|
Python needed to read the pickle produced.
|
2007-08-15 11:28:22 -03:00
|
|
|
|
2008-10-18 16:25:07 -03:00
|
|
|
The *file* argument must have a write() method that accepts a single bytes
|
2008-10-18 17:47:58 -03:00
|
|
|
argument. It can thus be a file object opened for binary writing, a
|
2008-10-18 16:25:07 -03:00
|
|
|
io.BytesIO instance, or any other custom object that meets this interface.
|
2007-08-15 11:28:22 -03:00
|
|
|
|
2008-10-18 16:25:07 -03:00
|
|
|
.. function:: dumps(obj[, protocol])
|
2007-08-15 11:28:22 -03:00
|
|
|
|
2008-10-18 16:25:07 -03:00
|
|
|
Return the pickled representation of the object as a :class:`bytes`
|
|
|
|
object, instead of writing it to a file.
|
2007-08-15 11:28:22 -03:00
|
|
|
|
2008-10-18 16:25:07 -03:00
|
|
|
The optional *protocol* argument tells the pickler to use the given protocol;
|
|
|
|
supported protocols are 0, 1, 2, 3. The default protocol is 3; a
|
|
|
|
backward-incompatible protocol designed for Python 3.0.
|
2007-08-15 11:28:22 -03:00
|
|
|
|
2008-10-18 16:25:07 -03:00
|
|
|
Specifying a negative protocol version selects the highest protocol version
|
|
|
|
supported. The higher the protocol used, the more recent the version of
|
|
|
|
Python needed to read the pickle produced.
|
2007-08-15 11:28:22 -03:00
|
|
|
|
2008-10-18 16:25:07 -03:00
|
|
|
.. function:: load(file, [\*, encoding="ASCII", errors="strict"])
|
2007-08-15 11:28:22 -03:00
|
|
|
|
2008-10-18 16:25:07 -03:00
|
|
|
Read a pickled object representation from the open file object *file* and
|
|
|
|
return the reconstituted object hierarchy specified therein. This is
|
|
|
|
equivalent to ``Unpickler(file).load()``.
|
2007-08-15 11:28:22 -03:00
|
|
|
|
2008-10-18 16:25:07 -03:00
|
|
|
The protocol version of the pickle is detected automatically, so no protocol
|
|
|
|
argument is needed. Bytes past the pickled object's representation are
|
|
|
|
ignored.
|
2007-08-15 11:28:22 -03:00
|
|
|
|
2008-10-18 16:25:07 -03:00
|
|
|
The argument *file* must have two methods, a read() method that takes an
|
|
|
|
integer argument, and a readline() method that requires no arguments. Both
|
|
|
|
methods should return bytes. Thus *file* can be a binary file object opened
|
|
|
|
for reading, a BytesIO object, or any other custom object that meets this
|
|
|
|
interface.
|
2007-08-15 11:28:22 -03:00
|
|
|
|
2008-10-18 16:25:07 -03:00
|
|
|
Optional keyword arguments are encoding and errors, which are used to decode
|
|
|
|
8-bit string instances pickled by Python 2.x. These default to 'ASCII' and
|
|
|
|
'strict', respectively.
|
2007-08-15 11:28:22 -03:00
|
|
|
|
2008-10-18 16:25:07 -03:00
|
|
|
.. function:: loads(bytes_object, [\*, encoding="ASCII", errors="strict"])
|
2007-08-15 11:28:22 -03:00
|
|
|
|
2008-10-18 16:25:07 -03:00
|
|
|
Read a pickled object hierarchy from a :class:`bytes` object and return the
|
|
|
|
reconstituted object hierarchy specified therein
|
2007-08-15 11:28:22 -03:00
|
|
|
|
2008-10-18 16:25:07 -03:00
|
|
|
The protocol version of the pickle is detected automatically, so no protocol
|
|
|
|
argument is needed. Bytes past the pickled object's representation are
|
|
|
|
ignored.
|
2007-08-15 11:28:22 -03:00
|
|
|
|
2008-10-18 16:25:07 -03:00
|
|
|
Optional keyword arguments are encoding and errors, which are used to decode
|
|
|
|
8-bit string instances pickled by Python 2.x. These default to 'ASCII' and
|
|
|
|
'strict', respectively.
|
|
|
|
|
|
|
|
|
|
|
|
The :mod:`pickle` module defines three exceptions:
|
2007-08-15 11:28:22 -03:00
|
|
|
|
|
|
|
.. exception:: PickleError
|
|
|
|
|
2008-10-18 17:47:58 -03:00
|
|
|
Common base class for the other pickling exceptions. It inherits
|
2007-08-15 11:28:22 -03:00
|
|
|
:exc:`Exception`.
|
|
|
|
|
|
|
|
.. exception:: PicklingError
|
|
|
|
|
2008-10-18 16:25:07 -03:00
|
|
|
Error raised when an unpicklable object is encountered by :class:`Pickler`.
|
|
|
|
It inherits :exc:`PickleError`.
|
2007-08-15 11:28:22 -03:00
|
|
|
|
2008-10-18 17:47:58 -03:00
|
|
|
Refer to :ref:`pickle-picklable` to learn what kinds of objects can be
|
|
|
|
pickled.
|
|
|
|
|
2007-08-15 11:28:22 -03:00
|
|
|
.. exception:: UnpicklingError
|
|
|
|
|
2008-10-18 16:25:07 -03:00
|
|
|
Error raised when there a problem unpickling an object, such as a data
|
2008-10-18 17:47:58 -03:00
|
|
|
corruption or a security violation. It inherits :exc:`PickleError`.
|
2007-08-15 11:28:22 -03:00
|
|
|
|
2008-10-18 16:25:07 -03:00
|
|
|
Note that other exceptions may also be raised during unpickling, including
|
|
|
|
(but not necessarily limited to) AttributeError, EOFError, ImportError, and
|
|
|
|
IndexError.
|
2007-08-15 11:28:22 -03:00
|
|
|
|
|
|
|
|
2008-10-18 16:25:07 -03:00
|
|
|
The :mod:`pickle` module exports two classes, :class:`Pickler` and
|
|
|
|
:class:`Unpickler`:
|
2007-08-15 11:28:22 -03:00
|
|
|
|
2008-10-18 16:25:07 -03:00
|
|
|
.. class:: Pickler(file[, protocol])
|
2007-08-15 11:28:22 -03:00
|
|
|
|
2008-10-18 16:25:07 -03:00
|
|
|
This takes a binary file for writing a pickle data stream.
|
2007-08-15 11:28:22 -03:00
|
|
|
|
2008-10-18 16:25:07 -03:00
|
|
|
The optional *protocol* argument tells the pickler to use the given protocol;
|
|
|
|
supported protocols are 0, 1, 2, 3. The default protocol is 3; a
|
|
|
|
backward-incompatible protocol designed for Python 3.0.
|
2007-08-15 11:28:22 -03:00
|
|
|
|
2008-10-18 16:25:07 -03:00
|
|
|
Specifying a negative protocol version selects the highest protocol version
|
|
|
|
supported. The higher the protocol used, the more recent the version of
|
|
|
|
Python needed to read the pickle produced.
|
2007-08-15 11:28:22 -03:00
|
|
|
|
2008-10-18 16:25:07 -03:00
|
|
|
The *file* argument must have a write() method that accepts a single bytes
|
2008-10-18 17:47:58 -03:00
|
|
|
argument. It can thus be a file object opened for binary writing, a
|
2008-10-18 16:25:07 -03:00
|
|
|
io.BytesIO instance, or any other custom object that meets this interface.
|
2007-08-15 11:28:22 -03:00
|
|
|
|
2008-04-24 22:59:09 -03:00
|
|
|
.. method:: dump(obj)
|
2007-08-15 11:28:22 -03:00
|
|
|
|
2008-10-18 16:25:07 -03:00
|
|
|
Write a pickled representation of *obj* to the open file object given in
|
|
|
|
the constructor.
|
|
|
|
|
|
|
|
.. method:: persistent_id(obj)
|
2007-08-15 11:28:22 -03:00
|
|
|
|
2008-10-18 16:25:07 -03:00
|
|
|
Do nothing by default. This exists so a subclass can override it.
|
|
|
|
|
|
|
|
If :meth:`persistent_id` returns ``None``, *obj* is pickled as usual. Any
|
|
|
|
other value causes :class:`Pickler` to emit the returned value as a
|
|
|
|
persistent ID for *obj*. The meaning of this persistent ID should be
|
|
|
|
defined by :meth:`Unpickler.persistent_load`. Note that the value
|
|
|
|
returned by :meth:`persistent_id` cannot itself have a persistent ID.
|
|
|
|
|
|
|
|
See :ref:`pickle-persistent` for details and examples of uses.
|
2007-08-15 11:28:22 -03:00
|
|
|
|
2008-04-24 22:59:09 -03:00
|
|
|
.. method:: clear_memo()
|
2007-08-15 11:28:22 -03:00
|
|
|
|
2008-10-18 17:47:58 -03:00
|
|
|
Deprecated. Use the :meth:`clear` method on :attr:`memo`, instead.
|
|
|
|
Clear the pickler's memo, useful when reusing picklers.
|
2008-10-18 16:25:07 -03:00
|
|
|
|
|
|
|
.. attribute:: fast
|
|
|
|
|
2008-10-29 20:32:33 -03:00
|
|
|
Deprecated. Enable fast mode if set to a true value. The fast mode
|
|
|
|
disables the usage of memo, therefore speeding the pickling process by not
|
|
|
|
generating superfluous PUT opcodes. It should not be used with
|
|
|
|
self-referential objects, doing otherwise will cause :class:`Pickler` to
|
|
|
|
recurse infinitely.
|
2008-10-18 16:25:07 -03:00
|
|
|
|
|
|
|
Use :func:`pickletools.optimize` if you need more compact pickles.
|
|
|
|
|
|
|
|
.. attribute:: memo
|
|
|
|
|
|
|
|
Dictionary holding previously pickled objects to allow shared or
|
|
|
|
recursive objects to pickled by reference as opposed to by value.
|
2007-08-15 11:28:22 -03:00
|
|
|
|
|
|
|
|
2008-10-29 20:32:33 -03:00
|
|
|
.. XXX Move these comments to somewhere more appropriate.
|
|
|
|
|
2007-08-15 11:28:22 -03:00
|
|
|
It is possible to make multiple calls to the :meth:`dump` method of the same
|
|
|
|
:class:`Pickler` instance. These must then be matched to the same number of
|
|
|
|
calls to the :meth:`load` method of the corresponding :class:`Unpickler`
|
|
|
|
instance. If the same object is pickled by multiple :meth:`dump` calls, the
|
2008-10-18 16:25:07 -03:00
|
|
|
:meth:`load` will all yield references to the same object.
|
2007-08-15 11:28:22 -03:00
|
|
|
|
2008-10-18 16:25:07 -03:00
|
|
|
Please note, this is intended for pickling multiple objects without intervening
|
|
|
|
modifications to the objects or their parts. If you modify an object and then
|
|
|
|
pickle it again using the same :class:`Pickler` instance, the object is not
|
|
|
|
pickled again --- a reference to it is pickled and the :class:`Unpickler` will
|
|
|
|
return the old value, not the modified one.
|
2007-08-15 11:28:22 -03:00
|
|
|
|
|
|
|
|
2008-10-18 16:25:07 -03:00
|
|
|
.. class:: Unpickler(file, [\*, encoding="ASCII", errors="strict"])
|
2007-08-15 11:28:22 -03:00
|
|
|
|
2008-10-18 16:25:07 -03:00
|
|
|
This takes a binary file for reading a pickle data stream.
|
2007-08-15 11:28:22 -03:00
|
|
|
|
2008-10-18 16:25:07 -03:00
|
|
|
The protocol version of the pickle is detected automatically, so no
|
|
|
|
protocol argument is needed.
|
2007-08-15 11:28:22 -03:00
|
|
|
|
2008-10-18 16:25:07 -03:00
|
|
|
The argument *file* must have two methods, a read() method that takes an
|
|
|
|
integer argument, and a readline() method that requires no arguments. Both
|
|
|
|
methods should return bytes. Thus *file* can be a binary file object opened
|
|
|
|
for reading, a BytesIO object, or any other custom object that meets this
|
|
|
|
interface.
|
2007-08-15 11:28:22 -03:00
|
|
|
|
2008-10-18 16:25:07 -03:00
|
|
|
Optional keyword arguments are encoding and errors, which are used to decode
|
|
|
|
8-bit string instances pickled by Python 2.x. These default to 'ASCII' and
|
|
|
|
'strict', respectively.
|
2007-08-15 11:28:22 -03:00
|
|
|
|
2008-04-24 22:59:09 -03:00
|
|
|
.. method:: load()
|
2007-08-15 11:28:22 -03:00
|
|
|
|
2008-04-24 22:59:09 -03:00
|
|
|
Read a pickled object representation from the open file object given in
|
|
|
|
the constructor, and return the reconstituted object hierarchy specified
|
2008-10-18 17:47:58 -03:00
|
|
|
therein. Bytes past the pickled object's representation are ignored.
|
2007-08-15 11:28:22 -03:00
|
|
|
|
2008-10-18 16:25:07 -03:00
|
|
|
.. method:: persistent_load(pid)
|
2007-08-15 11:28:22 -03:00
|
|
|
|
2008-10-18 16:25:07 -03:00
|
|
|
Raise an :exc:`UnpickingError` by default.
|
2007-08-15 11:28:22 -03:00
|
|
|
|
2008-10-18 16:25:07 -03:00
|
|
|
If defined, :meth:`persistent_load` should return the object specified by
|
2008-10-18 17:47:58 -03:00
|
|
|
the persistent ID *pid*. If an invalid persistent ID is encountered, an
|
|
|
|
:exc:`UnpickingError` should be raised.
|
2007-08-15 11:28:22 -03:00
|
|
|
|
2008-10-18 16:25:07 -03:00
|
|
|
See :ref:`pickle-persistent` for details and examples of uses.
|
|
|
|
|
|
|
|
.. method:: find_class(module, name)
|
|
|
|
|
2008-10-18 17:47:58 -03:00
|
|
|
Import *module* if necessary and return the object called *name* from it,
|
2008-10-23 22:32:40 -03:00
|
|
|
where the *module* and *name* arguments are :class:`str` objects. Note,
|
|
|
|
unlike its name suggests, :meth:`find_class` is also used for finding
|
|
|
|
functions.
|
2008-10-18 17:47:58 -03:00
|
|
|
|
|
|
|
Subclasses may override this to gain control over what type of objects and
|
2008-10-23 22:32:40 -03:00
|
|
|
how they can be loaded, potentially reducing security risks. Refer to
|
|
|
|
:ref:`pickle-restrict` for details.
|
2008-10-18 17:47:58 -03:00
|
|
|
|
2007-08-15 11:28:22 -03:00
|
|
|
|
2008-10-18 17:47:58 -03:00
|
|
|
.. _pickle-picklable:
|
2007-08-15 11:28:22 -03:00
|
|
|
|
|
|
|
What can be pickled and unpickled?
|
|
|
|
----------------------------------
|
|
|
|
|
|
|
|
The following types can be pickled:
|
|
|
|
|
|
|
|
* ``None``, ``True``, and ``False``
|
|
|
|
|
2007-11-29 13:24:34 -04:00
|
|
|
* integers, floating point numbers, complex numbers
|
2007-08-15 11:28:22 -03:00
|
|
|
|
2008-02-01 07:56:49 -04:00
|
|
|
* strings, bytes, bytearrays
|
2007-08-15 11:28:22 -03:00
|
|
|
|
|
|
|
* tuples, lists, sets, and dictionaries containing only picklable objects
|
|
|
|
|
|
|
|
* functions defined at the top level of a module
|
|
|
|
|
|
|
|
* built-in functions defined at the top level of a module
|
|
|
|
|
|
|
|
* classes that are defined at the top level of a module
|
|
|
|
|
|
|
|
* instances of such classes whose :attr:`__dict__` or :meth:`__setstate__` is
|
2008-10-29 20:32:33 -03:00
|
|
|
picklable (see section :ref:`pickle-inst` for details)
|
2007-08-15 11:28:22 -03:00
|
|
|
|
|
|
|
Attempts to pickle unpicklable objects will raise the :exc:`PicklingError`
|
|
|
|
exception; when this happens, an unspecified number of bytes may have already
|
2008-10-18 17:47:58 -03:00
|
|
|
been written to the underlying file. Trying to pickle a highly recursive data
|
2007-08-15 11:28:22 -03:00
|
|
|
structure may exceed the maximum recursion depth, a :exc:`RuntimeError` will be
|
2008-10-18 17:47:58 -03:00
|
|
|
raised in this case. You can carefully raise this limit with
|
2007-08-15 11:28:22 -03:00
|
|
|
:func:`sys.setrecursionlimit`.
|
|
|
|
|
|
|
|
Note that functions (built-in and user-defined) are pickled by "fully qualified"
|
|
|
|
name reference, not by value. This means that only the function name is
|
|
|
|
pickled, along with the name of module the function is defined in. Neither the
|
|
|
|
function's code, nor any of its function attributes are pickled. Thus the
|
|
|
|
defining module must be importable in the unpickling environment, and the module
|
|
|
|
must contain the named object, otherwise an exception will be raised. [#]_
|
|
|
|
|
|
|
|
Similarly, classes are pickled by named reference, so the same restrictions in
|
|
|
|
the unpickling environment apply. Note that none of the class's code or data is
|
|
|
|
pickled, so in the following example the class attribute ``attr`` is not
|
|
|
|
restored in the unpickling environment::
|
|
|
|
|
|
|
|
class Foo:
|
2008-10-18 17:47:58 -03:00
|
|
|
attr = 'A class attribute'
|
2007-08-15 11:28:22 -03:00
|
|
|
|
|
|
|
picklestring = pickle.dumps(Foo)
|
|
|
|
|
|
|
|
These restrictions are why picklable functions and classes must be defined in
|
|
|
|
the top level of a module.
|
|
|
|
|
|
|
|
Similarly, when class instances are pickled, their class's code and data are not
|
|
|
|
pickled along with them. Only the instance data are pickled. This is done on
|
|
|
|
purpose, so you can fix bugs in a class or add methods to the class and still
|
|
|
|
load objects that were created with an earlier version of the class. If you
|
|
|
|
plan to have long-lived objects that will see many versions of a class, it may
|
|
|
|
be worthwhile to put a version number in the objects so that suitable
|
|
|
|
conversions can be made by the class's :meth:`__setstate__` method.
|
|
|
|
|
|
|
|
|
2008-10-29 20:32:33 -03:00
|
|
|
.. _pickle-inst:
|
2007-08-15 11:28:22 -03:00
|
|
|
|
2008-10-29 20:32:33 -03:00
|
|
|
Pickling Class Instances
|
|
|
|
------------------------
|
2007-08-15 11:28:22 -03:00
|
|
|
|
2008-10-29 20:32:33 -03:00
|
|
|
In this section, we describe the general mechanisms available to you to define,
|
|
|
|
customize, and control how class instances are pickled and unpickled.
|
2007-08-15 11:28:22 -03:00
|
|
|
|
2008-10-29 20:32:33 -03:00
|
|
|
In most cases, no additional code is needed to make instances picklable. By
|
|
|
|
default, pickle will retrieve the class and the attributes of an instance via
|
|
|
|
introspection. When a class instance is unpickled, its :meth:`__init__` method
|
|
|
|
is usually *not* invoked. The default behaviour first creates an uninitialized
|
|
|
|
instance and then restores the saved attributes. The following code shows an
|
|
|
|
implementation of this behaviour::
|
2007-08-15 11:28:22 -03:00
|
|
|
|
2008-10-29 20:32:33 -03:00
|
|
|
def save(obj):
|
|
|
|
return (obj.__class__, obj.__dict__)
|
2007-08-15 11:28:22 -03:00
|
|
|
|
2008-10-29 20:32:33 -03:00
|
|
|
def load(cls, attributes):
|
|
|
|
obj = cls.__new__(cls)
|
|
|
|
obj.__dict__.update(attributes)
|
|
|
|
return obj
|
2007-08-15 11:28:22 -03:00
|
|
|
|
2008-10-29 20:32:33 -03:00
|
|
|
.. index:: single: __getnewargs__() (copy protocol)
|
2007-08-15 11:28:22 -03:00
|
|
|
|
2008-10-29 20:32:33 -03:00
|
|
|
Classes can alter the default behaviour by providing one or severals special
|
|
|
|
methods. In protocol 2 and newer, classes that implements the
|
|
|
|
:meth:`__getnewargs__` method can dictate the values passed to the
|
|
|
|
:meth:`__new__` method upon unpickling. This is often needed for classes
|
|
|
|
whose :meth:`__new__` method requires arguments.
|
2007-08-31 13:33:38 -03:00
|
|
|
|
2008-10-29 20:32:33 -03:00
|
|
|
.. index:: single: __getstate__() (copy protocol)
|
2007-08-15 11:28:22 -03:00
|
|
|
|
2008-10-29 20:32:33 -03:00
|
|
|
Classes can further influence how their instances are pickled; if the class
|
|
|
|
defines the method :meth:`__getstate__`, it is called and the returned object is
|
|
|
|
pickled as the contents for the instance, instead of the contents of the
|
|
|
|
instance's dictionary. If the :meth:`__getstate__` method is absent, the
|
|
|
|
instance's :attr:`__dict__` is pickled as usual.
|
2007-08-15 11:28:22 -03:00
|
|
|
|
2008-10-29 20:32:33 -03:00
|
|
|
.. index:: single: __setstate__() (copy protocol)
|
2007-08-15 11:28:22 -03:00
|
|
|
|
2008-10-29 20:32:33 -03:00
|
|
|
Upon unpickling, if the class defines :meth:`__setstate__`, it is called with
|
|
|
|
the unpickled state. In that case, there is no requirement for the state object
|
|
|
|
to be a dictionary. Otherwise, the pickled state must be a dictionary and its
|
|
|
|
items are assigned to the new instance's dictionary.
|
2007-08-15 11:28:22 -03:00
|
|
|
|
2008-10-29 20:32:33 -03:00
|
|
|
.. note::
|
2007-08-15 11:28:22 -03:00
|
|
|
|
2008-10-29 20:32:33 -03:00
|
|
|
If :meth:`__getstate__` returns a false value, the :meth:`__setstate__`
|
|
|
|
method will not be called.
|
2007-08-15 11:28:22 -03:00
|
|
|
|
2008-10-29 20:32:33 -03:00
|
|
|
Refer to the section :ref:`pickle-state` for more information about how to use
|
|
|
|
the methods :meth:`__getstate__` and :meth:`__setstate__`.
|
2007-08-15 11:28:22 -03:00
|
|
|
|
2008-10-29 20:32:33 -03:00
|
|
|
.. index::
|
|
|
|
pair: copy; protocol
|
|
|
|
single: __reduce__() (copy protocol)
|
2007-08-15 11:28:22 -03:00
|
|
|
|
2008-10-29 20:32:33 -03:00
|
|
|
As we shall see, pickle does not use directly the methods described above. In
|
|
|
|
fact, these methods are part of the copy protocol which implements the
|
|
|
|
:meth:`__reduce__` special method. The copy protocol provides a unified
|
|
|
|
interface for retrieving the data necessary for pickling and copying
|
2009-01-03 17:18:54 -04:00
|
|
|
objects. [#]_
|
2007-08-15 11:28:22 -03:00
|
|
|
|
2008-10-29 20:32:33 -03:00
|
|
|
Although powerful, implementing :meth:`__reduce__` directly in your classes is
|
|
|
|
error prone. For this reason, class designers should use the high-level
|
|
|
|
interface (i.e., :meth:`__getnewargs__`, :meth:`__getstate__` and
|
|
|
|
:meth:`__setstate__`) whenever possible. We will show however cases where using
|
|
|
|
:meth:`__reduce__` is the only option or leads to more efficient pickling or
|
|
|
|
both.
|
2007-08-15 11:28:22 -03:00
|
|
|
|
2008-10-29 20:32:33 -03:00
|
|
|
The interface is currently defined as follow. The :meth:`__reduce__` method
|
|
|
|
takes no argument and shall return either a string or preferably a tuple (the
|
|
|
|
returned object is often refered as the "reduce value").
|
2007-08-15 11:28:22 -03:00
|
|
|
|
2008-10-29 20:32:33 -03:00
|
|
|
If a string is returned, the string should be interpreted as the name of a
|
|
|
|
global variable. It should be the object's local name relative to its module;
|
|
|
|
the pickle module searches the module namespace to determine the object's
|
|
|
|
module. This behaviour is typically useful for singletons.
|
2007-08-15 11:28:22 -03:00
|
|
|
|
2008-10-29 20:32:33 -03:00
|
|
|
When a tuple is returned, it must be between two and five items long. Optional
|
|
|
|
items can either be omitted, or ``None`` can be provided as their value. The
|
|
|
|
semantics of each item are in order:
|
2007-08-15 11:28:22 -03:00
|
|
|
|
2008-10-29 20:32:33 -03:00
|
|
|
.. XXX Mention __newobj__ special-case?
|
2007-08-15 11:28:22 -03:00
|
|
|
|
|
|
|
* A callable object that will be called to create the initial version of the
|
2008-10-29 20:32:33 -03:00
|
|
|
object.
|
2007-08-15 11:28:22 -03:00
|
|
|
|
2008-10-29 20:32:33 -03:00
|
|
|
* A tuple of arguments for the callable object. An empty tuple must be given if
|
|
|
|
the callable does not accept any argument.
|
2007-08-15 11:28:22 -03:00
|
|
|
|
|
|
|
* Optionally, the object's state, which will be passed to the object's
|
2008-10-29 20:32:33 -03:00
|
|
|
:meth:`__setstate__` method as previously described. If the object has no
|
|
|
|
such method then, the value must be a dictionary and it will be added to the
|
|
|
|
object's :attr:`__dict__` attribute.
|
|
|
|
|
|
|
|
* Optionally, an iterator (and not a sequence) yielding successive items. These
|
|
|
|
items will be appended to the object either using ``obj.append(item)`` or, in
|
|
|
|
batch, using ``obj.extend(list_of_items)``. This is primarily used for list
|
|
|
|
subclasses, but may be used by other classes as long as they have
|
2007-08-15 11:28:22 -03:00
|
|
|
:meth:`append` and :meth:`extend` methods with the appropriate signature.
|
|
|
|
(Whether :meth:`append` or :meth:`extend` is used depends on which pickle
|
2008-10-29 20:32:33 -03:00
|
|
|
protocol version is used as well as the number of items to append, so both
|
|
|
|
must be supported.)
|
|
|
|
|
|
|
|
* Optionally, an iterator (not a sequence) yielding successive key-value pairs.
|
|
|
|
These items will be stored to the object using ``obj[key] = value``. This is
|
|
|
|
primarily used for dictionary subclasses, but may be used by other classes as
|
|
|
|
long as they implement :meth:`__setitem__`.
|
|
|
|
|
|
|
|
.. index:: single: __reduce_ex__() (copy protocol)
|
2007-08-15 11:28:22 -03:00
|
|
|
|
2008-10-29 20:32:33 -03:00
|
|
|
Alternatively, a :meth:`__reduce_ex__` method may be defined. The only
|
|
|
|
difference is this method should take a single integer argument, the protocol
|
|
|
|
version. When defined, pickle will prefer it over the :meth:`__reduce__`
|
|
|
|
method. In addition, :meth:`__reduce__` automatically becomes a synonym for the
|
|
|
|
extended version. The main use for this method is to provide
|
|
|
|
backwards-compatible reduce values for older Python releases.
|
2007-08-15 11:28:22 -03:00
|
|
|
|
2008-10-18 16:25:07 -03:00
|
|
|
.. _pickle-persistent:
|
|
|
|
|
2008-10-29 20:32:33 -03:00
|
|
|
Persistence of External Objects
|
|
|
|
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
|
2007-08-15 11:28:22 -03:00
|
|
|
|
2008-02-23 14:30:17 -04:00
|
|
|
.. index::
|
|
|
|
single: persistent_id (pickle protocol)
|
|
|
|
single: persistent_load (pickle protocol)
|
|
|
|
|
2007-08-15 11:28:22 -03:00
|
|
|
For the benefit of object persistence, the :mod:`pickle` module supports the
|
|
|
|
notion of a reference to an object outside the pickled data stream. Such
|
2008-10-18 17:47:58 -03:00
|
|
|
objects are referenced by a persistent ID, which should be either a string of
|
|
|
|
alphanumeric characters (for protocol 0) [#]_ or just an arbitrary object (for
|
|
|
|
any newer protocol).
|
2007-08-15 11:28:22 -03:00
|
|
|
|
2008-10-18 17:47:58 -03:00
|
|
|
The resolution of such persistent IDs is not defined by the :mod:`pickle`
|
|
|
|
module; it will delegate this resolution to the user defined methods on the
|
|
|
|
pickler and unpickler, :meth:`persistent_id` and :meth:`persistent_load`
|
|
|
|
respectively.
|
2007-08-15 11:28:22 -03:00
|
|
|
|
|
|
|
To pickle objects that have an external persistent id, the pickler must have a
|
2008-10-18 17:47:58 -03:00
|
|
|
custom :meth:`persistent_id` method that takes an object as an argument and
|
2007-08-15 11:28:22 -03:00
|
|
|
returns either ``None`` or the persistent id for that object. When ``None`` is
|
2008-10-18 17:47:58 -03:00
|
|
|
returned, the pickler simply pickles the object as normal. When a persistent ID
|
|
|
|
string is returned, the pickler will pickle that object, along with a marker so
|
|
|
|
that the unpickler will recognize it as a persistent ID.
|
2007-08-15 11:28:22 -03:00
|
|
|
|
|
|
|
To unpickle external objects, the unpickler must have a custom
|
2008-10-18 17:47:58 -03:00
|
|
|
:meth:`persistent_load` method that takes a persistent ID object and returns the
|
|
|
|
referenced object.
|
2007-08-15 11:28:22 -03:00
|
|
|
|
2008-10-29 20:32:33 -03:00
|
|
|
Here is a comprehensive example presenting how persistent ID can be used to
|
|
|
|
pickle external objects by reference.
|
2007-08-15 11:28:22 -03:00
|
|
|
|
2008-10-18 17:47:58 -03:00
|
|
|
.. literalinclude:: ../includes/dbpickle.py
|
2009-01-23 01:28:16 -04:00
|
|
|
|
2007-08-15 11:28:22 -03:00
|
|
|
|
2008-10-29 20:32:33 -03:00
|
|
|
.. _pickle-state:
|
|
|
|
|
|
|
|
Handling Stateful Objects
|
|
|
|
^^^^^^^^^^^^^^^^^^^^^^^^^
|
|
|
|
|
|
|
|
.. index::
|
|
|
|
single: __getstate__() (copy protocol)
|
|
|
|
single: __setstate__() (copy protocol)
|
|
|
|
|
|
|
|
Here's an example that shows how to modify pickling behavior for a class.
|
|
|
|
The :class:`TextReader` class opens a text file, and returns the line number and
|
|
|
|
line contents each time its :meth:`readline` method is called. If a
|
|
|
|
:class:`TextReader` instance is pickled, all attributes *except* the file object
|
|
|
|
member are saved. When the instance is unpickled, the file is reopened, and
|
|
|
|
reading resumes from the last location. The :meth:`__setstate__` and
|
|
|
|
:meth:`__getstate__` methods are used to implement this behavior. ::
|
|
|
|
|
|
|
|
class TextReader:
|
|
|
|
"""Print and number lines in a text file."""
|
|
|
|
|
|
|
|
def __init__(self, filename):
|
|
|
|
self.filename = filename
|
|
|
|
self.file = open(filename)
|
|
|
|
self.lineno = 0
|
|
|
|
|
|
|
|
def readline(self):
|
|
|
|
self.lineno += 1
|
|
|
|
line = self.file.readline()
|
|
|
|
if not line:
|
|
|
|
return None
|
|
|
|
if line.endswith("\n"):
|
|
|
|
line = line[:-1]
|
|
|
|
return "%i: %s" % (self.lineno, line)
|
|
|
|
|
|
|
|
def __getstate__(self):
|
|
|
|
# Copy the object's state from self.__dict__ which contains
|
|
|
|
# all our instance attributes. Always use the dict.copy()
|
|
|
|
# method to avoid modifying the original state.
|
|
|
|
state = self.__dict__.copy()
|
|
|
|
# Remove the unpicklable entries.
|
|
|
|
del state['file']
|
|
|
|
return state
|
|
|
|
|
|
|
|
def __setstate__(self, state):
|
|
|
|
# Restore instance attributes (i.e., filename and lineno).
|
|
|
|
self.__dict__.update(state)
|
|
|
|
# Restore the previously opened file's state. To do so, we need to
|
|
|
|
# reopen it and read from it until the line count is restored.
|
|
|
|
file = open(self.filename)
|
|
|
|
for _ in range(self.lineno):
|
|
|
|
file.readline()
|
|
|
|
# Finally, save the file.
|
|
|
|
self.file = file
|
|
|
|
|
|
|
|
|
|
|
|
A sample usage might be something like this::
|
|
|
|
|
|
|
|
>>> reader = TextReader("hello.txt")
|
|
|
|
>>> reader.readline()
|
|
|
|
'1: Hello world!'
|
|
|
|
>>> reader.readline()
|
|
|
|
'2: I am line number two.'
|
|
|
|
>>> new_reader = pickle.loads(pickle.dumps(reader))
|
|
|
|
>>> new_reader.readline()
|
|
|
|
'3: Goodbye!'
|
|
|
|
|
|
|
|
|
2008-10-23 22:32:40 -03:00
|
|
|
.. _pickle-restrict:
|
2007-08-15 11:28:22 -03:00
|
|
|
|
2008-10-23 22:32:40 -03:00
|
|
|
Restricting Globals
|
2008-10-29 20:32:33 -03:00
|
|
|
-------------------
|
2007-08-15 11:28:22 -03:00
|
|
|
|
2008-02-23 14:30:17 -04:00
|
|
|
.. index::
|
2008-10-23 22:32:40 -03:00
|
|
|
single: find_class() (pickle protocol)
|
|
|
|
|
|
|
|
By default, unpickling will import any class or function that it finds in the
|
|
|
|
pickle data. For many applications, this behaviour is unacceptable as it
|
|
|
|
permits the unpickler to import and invoke arbitrary code. Just consider what
|
|
|
|
this hand-crafted pickle data stream does when loaded::
|
|
|
|
|
|
|
|
>>> import pickle
|
|
|
|
>>> pickle.loads(b"cos\nsystem\n(S'echo hello world'\ntR.")
|
|
|
|
hello world
|
|
|
|
0
|
|
|
|
|
|
|
|
In this example, the unpickler imports the :func:`os.system` function and then
|
|
|
|
apply the string argument "echo hello world". Although this example is
|
|
|
|
inoffensive, it is not difficult to imagine one that could damage your system.
|
|
|
|
|
|
|
|
For this reason, you may want to control what gets unpickled by customizing
|
|
|
|
:meth:`Unpickler.find_class`. Unlike its name suggests, :meth:`find_class` is
|
|
|
|
called whenever a global (i.e., a class or a function) is requested. Thus it is
|
|
|
|
possible to either forbid completely globals or restrict them to a safe subset.
|
|
|
|
|
|
|
|
Here is an example of an unpickler allowing only few safe classes from the
|
|
|
|
:mod:`builtins` module to be loaded::
|
|
|
|
|
|
|
|
import builtins
|
|
|
|
import io
|
|
|
|
import pickle
|
2007-08-15 11:28:22 -03:00
|
|
|
|
2008-10-23 22:32:40 -03:00
|
|
|
safe_builtins = {
|
|
|
|
'range',
|
|
|
|
'complex',
|
|
|
|
'set',
|
|
|
|
'frozenset',
|
|
|
|
'slice',
|
|
|
|
}
|
|
|
|
|
|
|
|
class RestrictedUnpickler(pickle.Unpickler):
|
2008-10-29 20:32:33 -03:00
|
|
|
|
2008-10-23 22:32:40 -03:00
|
|
|
def find_class(self, module, name):
|
|
|
|
# Only allow safe classes from builtins.
|
|
|
|
if module == "builtins" and name in safe_builtins:
|
|
|
|
return getattr(builtins, name)
|
|
|
|
# Forbid everything else.
|
|
|
|
raise pickle.UnpicklingError("global '%s.%s' is forbidden" %
|
|
|
|
(module, name))
|
|
|
|
|
|
|
|
def restricted_loads(s):
|
|
|
|
"""Helper function analogous to pickle.loads()."""
|
|
|
|
return RestrictedUnpickler(io.BytesIO(s)).load()
|
|
|
|
|
|
|
|
A sample usage of our unpickler working has intended::
|
|
|
|
|
|
|
|
>>> restricted_loads(pickle.dumps([1, 2, range(15)]))
|
|
|
|
[1, 2, range(0, 15)]
|
|
|
|
>>> restricted_loads(b"cos\nsystem\n(S'echo hello world'\ntR.")
|
|
|
|
Traceback (most recent call last):
|
|
|
|
...
|
|
|
|
pickle.UnpicklingError: global 'os.system' is forbidden
|
|
|
|
>>> restricted_loads(b'cbuiltins\neval\n'
|
|
|
|
... b'(S\'getattr(__import__("os"), "system")'
|
|
|
|
... b'("echo hello world")\'\ntR.')
|
|
|
|
Traceback (most recent call last):
|
|
|
|
...
|
|
|
|
pickle.UnpicklingError: global 'builtins.eval' is forbidden
|
|
|
|
|
2008-10-29 20:32:33 -03:00
|
|
|
|
|
|
|
.. XXX Add note about how extension codes could evade our protection
|
2009-01-03 17:18:54 -04:00
|
|
|
mechanism (e.g. cached classes do not invokes find_class()).
|
2008-10-29 20:32:33 -03:00
|
|
|
|
|
|
|
As our examples shows, you have to be careful with what you allow to be
|
|
|
|
unpickled. Therefore if security is a concern, you may want to consider
|
|
|
|
alternatives such as the marshalling API in :mod:`xmlrpc.client` or third-party
|
|
|
|
solutions.
|
|
|
|
|
2007-08-15 11:28:22 -03:00
|
|
|
|
|
|
|
.. _pickle-example:
|
|
|
|
|
2009-01-23 01:28:16 -04:00
|
|
|
Usage Examples
|
|
|
|
--------------
|
2007-08-15 11:28:22 -03:00
|
|
|
|
|
|
|
For the simplest code, use the :func:`dump` and :func:`load` functions. Note
|
|
|
|
that a self-referencing list is pickled and restored correctly. ::
|
|
|
|
|
|
|
|
import pickle
|
|
|
|
|
2009-01-23 01:28:16 -04:00
|
|
|
# An arbitrary collection of objects supported by pickle.
|
|
|
|
data = {
|
|
|
|
'a': [1, 2.0, 3, 4+6j],
|
|
|
|
'b': ("character string", b"byte string"),
|
|
|
|
'c': set([None, True, False])
|
|
|
|
}
|
2007-08-15 11:28:22 -03:00
|
|
|
|
2009-01-23 01:28:16 -04:00
|
|
|
with open('data.pickle', 'wb') as f:
|
|
|
|
# Pickle the 'data' dictionary using the highest protocol available.
|
|
|
|
pickle.dump(data, f, pickle.HIGHEST_PROTOCOL)
|
2007-08-15 11:28:22 -03:00
|
|
|
|
|
|
|
|
2009-01-23 01:28:16 -04:00
|
|
|
The following example reads the resulting pickled data. ::
|
2007-08-15 11:28:22 -03:00
|
|
|
|
2009-01-23 01:28:16 -04:00
|
|
|
import pickle
|
2007-08-15 11:28:22 -03:00
|
|
|
|
2009-01-23 01:28:16 -04:00
|
|
|
with open('data.pickle', 'rb') as f:
|
|
|
|
# The protocol version used is detected automatically, so we do not
|
|
|
|
# have to specify it.
|
|
|
|
data = pickle.load(f)
|
2007-08-15 11:28:22 -03:00
|
|
|
|
|
|
|
|
|
|
|
.. seealso::
|
|
|
|
|
2008-05-11 05:55:36 -03:00
|
|
|
Module :mod:`copyreg`
|
2007-08-15 11:28:22 -03:00
|
|
|
Pickle interface constructor registration for extension types.
|
|
|
|
|
|
|
|
Module :mod:`shelve`
|
|
|
|
Indexed databases of objects; uses :mod:`pickle`.
|
|
|
|
|
|
|
|
Module :mod:`copy`
|
|
|
|
Shallow and deep object copying.
|
|
|
|
|
|
|
|
Module :mod:`marshal`
|
|
|
|
High-performance serialization of built-in types.
|
|
|
|
|
|
|
|
|
|
|
|
.. rubric:: Footnotes
|
|
|
|
|
|
|
|
.. [#] Don't confuse this with the :mod:`marshal` module
|
|
|
|
|
|
|
|
.. [#] The exception raised will likely be an :exc:`ImportError` or an
|
|
|
|
:exc:`AttributeError` but it could be something else.
|
|
|
|
|
2008-10-29 20:32:33 -03:00
|
|
|
.. [#] The :mod:`copy` module uses this protocol for shallow and deep copying
|
|
|
|
operations.
|
2008-10-18 17:47:58 -03:00
|
|
|
|
2008-10-23 22:32:40 -03:00
|
|
|
.. [#] The limitation on alphanumeric characters is due to the fact
|
|
|
|
the persistent IDs, in protocol 0, are delimited by the newline
|
|
|
|
character. Therefore if any kind of newline characters occurs in
|
2008-10-18 17:47:58 -03:00
|
|
|
persistent IDs, the resulting pickle will become unreadable.
|