cpython/Doc/library/pickle.rst

775 lines
30 KiB
ReStructuredText
Raw Normal View History

2007-08-15 11:28:22 -03:00
:mod:`pickle` --- Python object serialization
=============================================
.. index::
single: persistence
pair: persistent; objects
pair: serializing; objects
pair: marshalling; objects
pair: flattening; objects
pair: pickling; objects
.. module:: pickle
:synopsis: Convert Python objects to streams of bytes and back.
.. sectionauthor:: Jim Kerr <jbkerr@sr.hp.com>.
.. sectionauthor:: Barry Warsaw <barry@zope.com>
2007-08-15 11:28:22 -03:00
The :mod:`pickle` module implements a fundamental, but powerful algorithm for
serializing and de-serializing a Python object structure. "Pickling" is the
process whereby a Python object hierarchy is converted into a byte stream, and
"unpickling" is the inverse operation, whereby a byte stream is converted back
into an object hierarchy. Pickling (and unpickling) is alternatively known as
"serialization", "marshalling," [#]_ or "flattening", however, to avoid
confusion, the terms used here are "pickling" and "unpickling"..
2007-08-15 11:28:22 -03:00
Relationship to other Python modules
------------------------------------
The :mod:`pickle` module has an transparent optimizer (:mod:`_pickle`) written
in C. It is used whenever available. Otherwise the pure Python implementation is
used.
2007-08-15 11:28:22 -03:00
Python has a more primitive serialization module called :mod:`marshal`, but in
general :mod:`pickle` should always be the preferred way to serialize Python
objects. :mod:`marshal` exists primarily to support Python's :file:`.pyc`
files.
The :mod:`pickle` module differs from :mod:`marshal` several significant ways:
* The :mod:`pickle` module keeps track of the objects it has already serialized,
so that later references to the same object won't be serialized again.
:mod:`marshal` doesn't do this.
This has implications both for recursive objects and object sharing. Recursive
objects are objects that contain references to themselves. These are not
handled by marshal, and in fact, attempting to marshal recursive objects will
crash your Python interpreter. Object sharing happens when there are multiple
references to the same object in different places in the object hierarchy being
serialized. :mod:`pickle` stores such objects only once, and ensures that all
other references point to the master copy. Shared objects remain shared, which
can be very important for mutable objects.
* :mod:`marshal` cannot be used to serialize user-defined classes and their
instances. :mod:`pickle` can save and restore class instances transparently,
however the class definition must be importable and live in the same module as
when the object was stored.
* The :mod:`marshal` serialization format is not guaranteed to be portable
across Python versions. Because its primary job in life is to support
:file:`.pyc` files, the Python implementers reserve the right to change the
serialization format in non-backwards compatible ways should the need arise.
The :mod:`pickle` serialization format is guaranteed to be backwards compatible
across Python releases.
.. warning::
The :mod:`pickle` module is not intended to be secure against erroneous or
maliciously constructed data. Never unpickle data received from an untrusted or
unauthenticated source.
Note that serialization is a more primitive notion than persistence; although
:mod:`pickle` reads and writes file objects, it does not handle the issue of
naming persistent objects, nor the (even more complicated) issue of concurrent
access to persistent objects. The :mod:`pickle` module can transform a complex
object into a byte stream and it can transform the byte stream into an object
with the same internal structure. Perhaps the most obvious thing to do with
these byte streams is to write them onto a file, but it is also conceivable to
send them across a network or store them in a database. The module
:mod:`shelve` provides a simple interface to pickle and unpickle objects on
DBM-style database files.
Data stream format
------------------
.. index::
single: XDR
single: External Data Representation
The data format used by :mod:`pickle` is Python-specific. This has the
advantage that there are no restrictions imposed by external standards such as
XDR (which can't represent pointer sharing); however it means that non-Python
programs may not be able to reconstruct pickled Python objects.
By default, the :mod:`pickle` data format uses a compact binary representation.
The module :mod:`pickletools` contains tools for analyzing data streams
generated by :mod:`pickle`.
2007-08-15 11:28:22 -03:00
There are currently 4 different protocols which can be used for pickling.
2007-08-15 11:28:22 -03:00
* Protocol version 0 is the original human-readable protocol and is
backwards compatible with earlier versions of Python.
2007-08-15 11:28:22 -03:00
* Protocol version 1 is the old binary format which is also compatible with
earlier versions of Python.
* Protocol version 2 was introduced in Python 2.3. It provides much more
#1370: Finish the merge r58749, log below, by resolving all conflicts in Doc/. Merged revisions 58221-58741 via svnmerge from svn+ssh://pythondev@svn.python.org/python/trunk ........ r58221 | georg.brandl | 2007-09-20 10:57:59 -0700 (Thu, 20 Sep 2007) | 2 lines Patch #1181: add os.environ.clear() method. ........ r58225 | sean.reifschneider | 2007-09-20 23:33:28 -0700 (Thu, 20 Sep 2007) | 3 lines Issue1704287: "make install" fails unless you do "make" first. Make oldsharedmods and sharedmods in "libinstall". ........ r58232 | guido.van.rossum | 2007-09-22 13:18:03 -0700 (Sat, 22 Sep 2007) | 4 lines Patch # 188 by Philip Jenvey. Make tell() mark CRLF as a newline. With unit test. ........ r58242 | georg.brandl | 2007-09-24 10:55:47 -0700 (Mon, 24 Sep 2007) | 2 lines Fix typo and double word. ........ r58245 | georg.brandl | 2007-09-24 10:59:28 -0700 (Mon, 24 Sep 2007) | 2 lines #1196: document default radix for int(). ........ r58247 | georg.brandl | 2007-09-24 11:08:24 -0700 (Mon, 24 Sep 2007) | 2 lines #1177: accept 2xx responses for https too, not only http. ........ r58249 | andrew.kuchling | 2007-09-24 16:45:51 -0700 (Mon, 24 Sep 2007) | 1 line Remove stray odd character; grammar fix ........ r58250 | andrew.kuchling | 2007-09-24 16:46:28 -0700 (Mon, 24 Sep 2007) | 1 line Typo fix ........ r58251 | andrew.kuchling | 2007-09-24 17:09:42 -0700 (Mon, 24 Sep 2007) | 1 line Add various items ........ r58268 | vinay.sajip | 2007-09-26 22:34:45 -0700 (Wed, 26 Sep 2007) | 1 line Change to flush and close logic to fix #1760556. ........ r58269 | vinay.sajip | 2007-09-26 22:38:51 -0700 (Wed, 26 Sep 2007) | 1 line Change to basicConfig() to fix #1021. ........ r58270 | georg.brandl | 2007-09-26 23:26:58 -0700 (Wed, 26 Sep 2007) | 2 lines #1208: document match object's boolean value. ........ r58271 | vinay.sajip | 2007-09-26 23:56:13 -0700 (Wed, 26 Sep 2007) | 1 line Minor date change. ........ r58272 | vinay.sajip | 2007-09-27 00:35:10 -0700 (Thu, 27 Sep 2007) | 1 line Change to LogRecord.__init__() to fix #1206. Note that archaic use of type(x) == types.DictType is because of keeping 1.5.2 compatibility. While this is much less relevant these days, there probably needs to be a separate commit for removing all archaic constructs at the same time. ........ r58288 | brett.cannon | 2007-09-30 12:45:10 -0700 (Sun, 30 Sep 2007) | 9 lines tuple.__repr__ did not consider a reference loop as it is not possible from Python code; but it is possible from C. object.__str__ had the issue of not expecting a type to doing something within it's tp_str implementation that could trigger an infinite recursion, but it could in C code.. Both found thanks to BaseException and how it handles its repr. Closes issue #1686386. Thanks to Thomas Herve for taking an initial stab at coming up with a solution. ........ r58289 | brett.cannon | 2007-09-30 13:37:19 -0700 (Sun, 30 Sep 2007) | 3 lines Fix error introduced by r58288; if a tuple is length 0 return its repr and don't worry about any self-referring tuples. ........ r58294 | facundo.batista | 2007-10-02 10:01:24 -0700 (Tue, 02 Oct 2007) | 11 lines Made the various is_* operations return booleans. This was discussed with Cawlishaw by mail, and he basically confirmed that to these is_* operations, there's no need to return Decimal(0) and Decimal(1) if the language supports the False and True booleans. Also added a few tests for the these functions in extra.decTest, since they are mostly untested (apart from the doctests). Thanks Mark Dickinson ........ r58295 | facundo.batista | 2007-10-02 11:21:18 -0700 (Tue, 02 Oct 2007) | 4 lines Added a class to store the digits of log(10), so that they can be made available when necessary without recomputing. Thanks Mark Dickinson ........ r58299 | mark.summerfield | 2007-10-03 01:53:21 -0700 (Wed, 03 Oct 2007) | 4 lines Added note in footnote about string comparisons about unicodedata.normalize(). ........ r58304 | raymond.hettinger | 2007-10-03 14:18:11 -0700 (Wed, 03 Oct 2007) | 1 line enumerate() is no longer bounded to using sequences shorter than LONG_MAX. The possibility of overflow was sending some newsgroup posters into a tizzy. ........ r58305 | raymond.hettinger | 2007-10-03 17:20:27 -0700 (Wed, 03 Oct 2007) | 1 line itertools.count() no longer limited to sys.maxint. ........ r58306 | kurt.kaiser | 2007-10-03 18:49:54 -0700 (Wed, 03 Oct 2007) | 3 lines Assume that the user knows when he wants to end the line; don't insert something he didn't select or complete. ........ r58307 | kurt.kaiser | 2007-10-03 19:07:50 -0700 (Wed, 03 Oct 2007) | 2 lines Remove unused theme that was causing a fault in p3k. ........ r58308 | kurt.kaiser | 2007-10-03 19:09:17 -0700 (Wed, 03 Oct 2007) | 2 lines Clean up EditorWindow close. ........ r58309 | kurt.kaiser | 2007-10-03 19:53:07 -0700 (Wed, 03 Oct 2007) | 7 lines textView cleanup. Patch 1718043 Tal Einat. M idlelib/EditorWindow.py M idlelib/aboutDialog.py M idlelib/textView.py M idlelib/NEWS.txt ........ r58310 | kurt.kaiser | 2007-10-03 20:11:12 -0700 (Wed, 03 Oct 2007) | 3 lines configDialog cleanup. Patch 1730217 Tal Einat. ........ r58311 | neal.norwitz | 2007-10-03 23:00:48 -0700 (Wed, 03 Oct 2007) | 4 lines Coverity #151: Remove deadcode. All this code already exists above starting at line 653. ........ r58325 | fred.drake | 2007-10-04 19:46:12 -0700 (Thu, 04 Oct 2007) | 1 line wrap lines to <80 characters before fixing errors ........ r58326 | raymond.hettinger | 2007-10-04 19:47:07 -0700 (Thu, 04 Oct 2007) | 6 lines Add __asdict__() to NamedTuple and refine the docs. Add maxlen support to deque() and fixup docs. Partially fix __reduce__(). The None as a third arg was no longer supported. Still needs work on __reduce__() to handle recursive inputs. ........ r58327 | fred.drake | 2007-10-04 19:48:32 -0700 (Thu, 04 Oct 2007) | 3 lines move descriptions of ac_(in|out)_buffer_size to the right place http://bugs.python.org/issue1053 ........ r58329 | neal.norwitz | 2007-10-04 20:39:17 -0700 (Thu, 04 Oct 2007) | 3 lines dict could be NULL, so we need to XDECREF. Fix a compiler warning about passing a PyTypeObject* instead of PyObject*. ........ r58330 | neal.norwitz | 2007-10-04 20:41:19 -0700 (Thu, 04 Oct 2007) | 2 lines Fix Coverity #158: Check the correct variable. ........ r58332 | neal.norwitz | 2007-10-04 22:01:38 -0700 (Thu, 04 Oct 2007) | 7 lines Fix Coverity #159. This code was broken if save() returned a negative number since i contained a boolean value and then we compared i < 0 which should never be true. Will backport (assuming it's necessary) ........ r58334 | neal.norwitz | 2007-10-04 22:29:17 -0700 (Thu, 04 Oct 2007) | 1 line Add a note about fixing some more warnings found by Coverity. ........ r58338 | raymond.hettinger | 2007-10-05 12:07:31 -0700 (Fri, 05 Oct 2007) | 1 line Restore BEGIN/END THREADS macros which were squashed in the previous checkin ........ r58343 | gregory.p.smith | 2007-10-06 00:48:10 -0700 (Sat, 06 Oct 2007) | 3 lines Stab in the dark attempt to fix the test_bsddb3 failure on sparc and S-390 ubuntu buildbots. ........ r58344 | gregory.p.smith | 2007-10-06 00:51:59 -0700 (Sat, 06 Oct 2007) | 2 lines Allows BerkeleyDB 4.6.x >= 4.6.21 for the bsddb module. ........ r58348 | gregory.p.smith | 2007-10-06 08:47:37 -0700 (Sat, 06 Oct 2007) | 3 lines Use the host the author likely meant in the first place. pop.gmail.com is reliable. gmail.org is someones personal domain. ........ r58351 | neal.norwitz | 2007-10-06 12:16:28 -0700 (Sat, 06 Oct 2007) | 3 lines Ensure that this test will pass even if another test left an unwritable TESTFN. Also use the safe unlink in test_support instead of rolling our own here. ........ r58368 | georg.brandl | 2007-10-08 00:50:24 -0700 (Mon, 08 Oct 2007) | 3 lines #1123: fix the docs for the str.split(None, sep) case. Also expand a few other methods' docs, which had more info in the deprecated string module docs. ........ r58369 | georg.brandl | 2007-10-08 01:06:05 -0700 (Mon, 08 Oct 2007) | 2 lines Update docstring of sched, also remove an unused assignment. ........ r58370 | raymond.hettinger | 2007-10-08 02:14:28 -0700 (Mon, 08 Oct 2007) | 5 lines Add comments to NamedTuple code. Let the field spec be either a string or a non-string sequence (suggested by Martin Blais with use cases). Improve the error message in the case of a SyntaxError (caused by a duplicate field name). ........ r58371 | raymond.hettinger | 2007-10-08 02:56:29 -0700 (Mon, 08 Oct 2007) | 1 line Missed a line in the docs ........ r58372 | raymond.hettinger | 2007-10-08 03:11:51 -0700 (Mon, 08 Oct 2007) | 1 line Better variable names ........ r58376 | georg.brandl | 2007-10-08 07:12:47 -0700 (Mon, 08 Oct 2007) | 3 lines #1199: docs for tp_as_{number,sequence,mapping}, by Amaury Forgeot d'Arc. No need to merge this to py3k! ........ r58380 | raymond.hettinger | 2007-10-08 14:26:58 -0700 (Mon, 08 Oct 2007) | 1 line Eliminate camelcase function name ........ r58381 | andrew.kuchling | 2007-10-08 16:23:03 -0700 (Mon, 08 Oct 2007) | 1 line Eliminate camelcase function name ........ r58382 | raymond.hettinger | 2007-10-08 18:36:23 -0700 (Mon, 08 Oct 2007) | 1 line Make the error messages more specific ........ r58384 | gregory.p.smith | 2007-10-08 23:02:21 -0700 (Mon, 08 Oct 2007) | 10 lines Splits Modules/_bsddb.c up into bsddb.h and _bsddb.c and adds a C API object available as bsddb.db.api. This is based on the patch submitted by Duncan Grisby here: http://sourceforge.net/tracker/index.php?func=detail&aid=1551895&group_id=13900&atid=313900 See this thread for additional info: http://sourceforge.net/mailarchive/forum.php?thread_name=E1GAVDK-0002rk-Iw%40apasphere.com&forum_name=pybsddb-users It also cleans up the code a little by removing some ifdef/endifs for python prior to 2.1 and for unsupported Berkeley DB <= 3.2. ........ r58385 | gregory.p.smith | 2007-10-08 23:50:43 -0700 (Mon, 08 Oct 2007) | 5 lines Fix a double free when positioning a database cursor to a non-existant string key (and probably a few other situations with string keys). This was reported with a patch as pybsddb sourceforge bug 1708868 by jjjhhhlll at gmail. ........ r58386 | gregory.p.smith | 2007-10-09 00:19:11 -0700 (Tue, 09 Oct 2007) | 3 lines Use the highest cPickle protocol in bsddb.dbshelve. This comes from sourceforge pybsddb patch 1551443 by w_barnes. ........ r58394 | gregory.p.smith | 2007-10-09 11:26:02 -0700 (Tue, 09 Oct 2007) | 2 lines remove another sleepycat reference ........ r58396 | kurt.kaiser | 2007-10-09 12:31:30 -0700 (Tue, 09 Oct 2007) | 3 lines Allow interrupt only when executing user code in subprocess Patch 1225 Tal Einat modified from IDLE-Spoon. ........ r58399 | brett.cannon | 2007-10-09 17:07:50 -0700 (Tue, 09 Oct 2007) | 5 lines Remove file-level typedefs that were inconsistently used throughout the file. Just move over to the public API names. Closes issue1238. ........ r58401 | raymond.hettinger | 2007-10-09 17:26:46 -0700 (Tue, 09 Oct 2007) | 1 line Accept Jim Jewett's api suggestion to use None instead of -1 to indicate unbounded deques. ........ r58403 | kurt.kaiser | 2007-10-09 17:55:40 -0700 (Tue, 09 Oct 2007) | 2 lines Allow cursor color change w/o restart. Patch 1725576 Tal Einat. ........ r58404 | kurt.kaiser | 2007-10-09 18:06:47 -0700 (Tue, 09 Oct 2007) | 2 lines show paste if > 80 columns. Patch 1659326 Tal Einat. ........ r58415 | thomas.heller | 2007-10-11 12:51:32 -0700 (Thu, 11 Oct 2007) | 5 lines On OS X, use os.uname() instead of gestalt.sysv(...) to get the operating system version. This allows to use ctypes when Python was configured with --disable-toolbox-glue. ........ r58419 | neal.norwitz | 2007-10-11 20:01:01 -0700 (Thu, 11 Oct 2007) | 1 line Get rid of warning about not being able to create an existing directory. ........ r58420 | neal.norwitz | 2007-10-11 20:01:30 -0700 (Thu, 11 Oct 2007) | 1 line Get rid of warnings on a bunch of platforms by using a proper prototype. ........ r58421 | neal.norwitz | 2007-10-11 20:01:54 -0700 (Thu, 11 Oct 2007) | 4 lines Get rid of compiler warning about retval being used (returned) without being initialized. (gcc warning and Coverity 202) ........ r58422 | neal.norwitz | 2007-10-11 20:03:23 -0700 (Thu, 11 Oct 2007) | 1 line Fix Coverity 168: Close the file before returning (exiting). ........ r58423 | neal.norwitz | 2007-10-11 20:04:18 -0700 (Thu, 11 Oct 2007) | 4 lines Fix Coverity 180: Don't overallocate. We don't need structs, but pointers. Also fix a memory leak. ........ r58424 | neal.norwitz | 2007-10-11 20:05:19 -0700 (Thu, 11 Oct 2007) | 5 lines Fix Coverity 185-186: If the passed in FILE is NULL, uninitialized memory would be accessed. Will backport. ........ r58425 | neal.norwitz | 2007-10-11 20:52:34 -0700 (Thu, 11 Oct 2007) | 1 line Get this module to compile with bsddb versions prior to 4.3 ........ r58430 | martin.v.loewis | 2007-10-12 01:56:52 -0700 (Fri, 12 Oct 2007) | 3 lines Bug #1216: Restore support for Visual Studio 2002. Will backport to 2.5. ........ r58433 | raymond.hettinger | 2007-10-12 10:53:11 -0700 (Fri, 12 Oct 2007) | 1 line Fix test of count.__repr__() to ignore the 'L' if the count is a long ........ r58434 | gregory.p.smith | 2007-10-12 11:44:06 -0700 (Fri, 12 Oct 2007) | 4 lines Fixes http://bugs.python.org/issue1233 - bsddb.dbshelve.DBShelf.append was useless due to inverted logic. Also adds a test case for RECNO dbs to test_dbshelve. ........ r58445 | georg.brandl | 2007-10-13 06:20:03 -0700 (Sat, 13 Oct 2007) | 2 lines Fix email example. ........ r58450 | gregory.p.smith | 2007-10-13 16:02:05 -0700 (Sat, 13 Oct 2007) | 2 lines Fix an uncollectable reference leak in bsddb.db.DBShelf.append ........ r58453 | neal.norwitz | 2007-10-13 17:18:40 -0700 (Sat, 13 Oct 2007) | 8 lines Let the O/S supply a port if none of the default ports can be used. This should make the tests more robust at the expense of allowing tests to be sloppier by not requiring them to cleanup after themselves. (It will legitamitely help when running two test suites simultaneously or if another process is already using one of the predefined ports.) Also simplifies (slightLy) the exception handling elsewhere. ........ r58459 | neal.norwitz | 2007-10-14 11:30:21 -0700 (Sun, 14 Oct 2007) | 2 lines Don't raise a string exception, they don't work anymore. ........ r58460 | neal.norwitz | 2007-10-14 11:40:37 -0700 (Sun, 14 Oct 2007) | 1 line Use unittest for assertions ........ r58468 | armin.rigo | 2007-10-15 00:48:35 -0700 (Mon, 15 Oct 2007) | 2 lines test_bigbits was not testing what it seemed to. ........ r58471 | guido.van.rossum | 2007-10-15 08:54:11 -0700 (Mon, 15 Oct 2007) | 3 lines Change a PyErr_Print() into a PyErr_Clear(), per discussion in issue 1031213. ........ r58500 | raymond.hettinger | 2007-10-16 12:18:30 -0700 (Tue, 16 Oct 2007) | 1 line Improve error messages ........ r58506 | raymond.hettinger | 2007-10-16 14:28:32 -0700 (Tue, 16 Oct 2007) | 1 line More docs, error messages, and tests ........ r58507 | andrew.kuchling | 2007-10-16 15:58:03 -0700 (Tue, 16 Oct 2007) | 1 line Add items ........ r58508 | brett.cannon | 2007-10-16 16:24:06 -0700 (Tue, 16 Oct 2007) | 3 lines Remove ``:const:`` notation on None in parameter list. Since the markup is not rendered for parameters it just showed up as ``:const:`None` `` in the output. ........ r58509 | brett.cannon | 2007-10-16 16:26:45 -0700 (Tue, 16 Oct 2007) | 3 lines Re-order some functions whose parameters differ between PyObject and const char * so that they are next to each other. ........ r58522 | armin.rigo | 2007-10-17 11:46:37 -0700 (Wed, 17 Oct 2007) | 5 lines Fix the overflow checking of list_repeat. Introduce overflow checking into list_inplace_repeat. Backport candidate, possibly. ........ r58530 | facundo.batista | 2007-10-17 20:16:03 -0700 (Wed, 17 Oct 2007) | 7 lines Issue #1580738. When HTTPConnection reads the whole stream with read(), it closes itself. When the stream is read in several calls to read(n), it should behave in the same way if HTTPConnection knows where the end of the stream is (through self.length). Added a test case for this behaviour. ........ r58531 | facundo.batista | 2007-10-17 20:44:48 -0700 (Wed, 17 Oct 2007) | 3 lines Issue 1289, just a typo. ........ r58532 | gregory.p.smith | 2007-10-18 00:56:54 -0700 (Thu, 18 Oct 2007) | 4 lines cleanup test_dbtables to use mkdtemp. cleanup dbtables to pass txn as a keyword argument whenever possible to avoid bugs and confusion. (dbtables.py line 447 self.db.get using txn as a non-keyword was an actual bug due to this) ........ r58533 | gregory.p.smith | 2007-10-18 01:34:20 -0700 (Thu, 18 Oct 2007) | 4 lines Fix a weird bug in dbtables: if it chose a random rowid string that contained NULL bytes it would cause the database all sorts of problems in the future leading to very strange random failures and corrupt dbtables.bsdTableDb dbs. ........ r58534 | gregory.p.smith | 2007-10-18 09:32:02 -0700 (Thu, 18 Oct 2007) | 3 lines A cleaner fix than the one committed last night. Generate random rowids that do not contain null bytes. ........ r58537 | gregory.p.smith | 2007-10-18 10:17:57 -0700 (Thu, 18 Oct 2007) | 2 lines mention bsddb fixes. ........ r58538 | raymond.hettinger | 2007-10-18 14:13:06 -0700 (Thu, 18 Oct 2007) | 1 line Remove useless warning ........ r58539 | gregory.p.smith | 2007-10-19 00:31:20 -0700 (Fri, 19 Oct 2007) | 2 lines squelch the warning that this test is supposed to trigger. ........ r58542 | georg.brandl | 2007-10-19 05:32:39 -0700 (Fri, 19 Oct 2007) | 2 lines Clarify wording for apply(). ........ r58544 | mark.summerfield | 2007-10-19 05:48:17 -0700 (Fri, 19 Oct 2007) | 3 lines Added a cross-ref to each other. ........ r58545 | georg.brandl | 2007-10-19 10:38:49 -0700 (Fri, 19 Oct 2007) | 2 lines #1284: "S" means "seen", not unread. ........ r58548 | thomas.heller | 2007-10-19 11:11:41 -0700 (Fri, 19 Oct 2007) | 4 lines Fix ctypes on 32-bit systems when Python is configured --with-system-ffi. See also https://bugs.launchpad.net/bugs/72505. Ported from release25-maint branch. ........ r58550 | facundo.batista | 2007-10-19 12:25:57 -0700 (Fri, 19 Oct 2007) | 8 lines The constructor from tuple was way too permissive: it allowed bad coefficient numbers, floats in the sign, and other details that generated directly the wrong number in the best case, or triggered misfunctionality in the alorithms. Test cases added for these issues. Thanks Mark Dickinson. ........ r58559 | georg.brandl | 2007-10-20 06:22:53 -0700 (Sat, 20 Oct 2007) | 2 lines Fix code being interpreted as a target. ........ r58561 | georg.brandl | 2007-10-20 06:36:24 -0700 (Sat, 20 Oct 2007) | 2 lines Document new "cmdoption" directive. ........ r58562 | georg.brandl | 2007-10-20 08:21:22 -0700 (Sat, 20 Oct 2007) | 2 lines Make a path more Unix-standardy. ........ r58564 | georg.brandl | 2007-10-20 10:51:39 -0700 (Sat, 20 Oct 2007) | 2 lines Document new directive "envvar". ........ r58567 | georg.brandl | 2007-10-20 11:08:14 -0700 (Sat, 20 Oct 2007) | 6 lines * Add new toplevel chapter, "Using Python." (how to install, configure and setup python on different platforms -- at least in theory.) * Move the Python on Mac docs in that chapter. * Add a new chapter about the command line invocation, by stargaming. ........ r58568 | georg.brandl | 2007-10-20 11:33:20 -0700 (Sat, 20 Oct 2007) | 2 lines Change title, for now. ........ r58569 | georg.brandl | 2007-10-20 11:39:25 -0700 (Sat, 20 Oct 2007) | 2 lines Add entry to ACKS. ........ r58570 | georg.brandl | 2007-10-20 12:05:45 -0700 (Sat, 20 Oct 2007) | 2 lines Clarify -E docs. ........ r58571 | georg.brandl | 2007-10-20 12:08:36 -0700 (Sat, 20 Oct 2007) | 2 lines Even more clarification. ........ r58572 | andrew.kuchling | 2007-10-20 12:25:37 -0700 (Sat, 20 Oct 2007) | 1 line Fix protocol name ........ r58573 | andrew.kuchling | 2007-10-20 12:35:18 -0700 (Sat, 20 Oct 2007) | 1 line Various items ........ r58574 | andrew.kuchling | 2007-10-20 12:39:35 -0700 (Sat, 20 Oct 2007) | 1 line Use correct header line ........ r58576 | armin.rigo | 2007-10-21 02:14:15 -0700 (Sun, 21 Oct 2007) | 3 lines Add a crasher for the long-standing issue with closing a file while another thread uses it. ........ r58577 | georg.brandl | 2007-10-21 03:01:56 -0700 (Sun, 21 Oct 2007) | 2 lines Remove duplicate crasher. ........ r58578 | georg.brandl | 2007-10-21 03:24:20 -0700 (Sun, 21 Oct 2007) | 2 lines Unify "byte code" to "bytecode". Also sprinkle :term: markup for it. ........ r58579 | georg.brandl | 2007-10-21 03:32:54 -0700 (Sun, 21 Oct 2007) | 2 lines Add markup to new function descriptions. ........ r58580 | georg.brandl | 2007-10-21 03:45:46 -0700 (Sun, 21 Oct 2007) | 2 lines Add :term:s for descriptors. ........ r58581 | georg.brandl | 2007-10-21 03:46:24 -0700 (Sun, 21 Oct 2007) | 2 lines Unify "file-descriptor" to "file descriptor". ........ r58582 | georg.brandl | 2007-10-21 03:52:38 -0700 (Sun, 21 Oct 2007) | 2 lines Add :term: for generators. ........ r58583 | georg.brandl | 2007-10-21 05:10:28 -0700 (Sun, 21 Oct 2007) | 2 lines Add :term:s for iterator. ........ r58584 | georg.brandl | 2007-10-21 05:15:05 -0700 (Sun, 21 Oct 2007) | 2 lines Add :term:s for "new-style class". ........ r58588 | neal.norwitz | 2007-10-21 21:47:54 -0700 (Sun, 21 Oct 2007) | 1 line Add Chris Monson so he can edit PEPs. ........ r58594 | guido.van.rossum | 2007-10-22 09:27:19 -0700 (Mon, 22 Oct 2007) | 4 lines Issue #1307, patch by Derek Shockey. When "MAIL" is received without args, an exception happens instead of sending a 501 syntax error response. ........ r58598 | travis.oliphant | 2007-10-22 19:40:56 -0700 (Mon, 22 Oct 2007) | 1 line Add phuang patch from Issue 708374 which adds offset parameter to mmap module. ........ r58601 | neal.norwitz | 2007-10-22 22:44:27 -0700 (Mon, 22 Oct 2007) | 2 lines Bug #1313, fix typo (wrong variable name) in example. ........ r58609 | georg.brandl | 2007-10-23 11:21:35 -0700 (Tue, 23 Oct 2007) | 2 lines Update Pygments version from externals. ........ r58618 | guido.van.rossum | 2007-10-23 12:25:41 -0700 (Tue, 23 Oct 2007) | 3 lines Issue 1307 by Derek Shockey, fox the same bug for RCPT. Neal: please backport! ........ r58620 | raymond.hettinger | 2007-10-23 13:37:41 -0700 (Tue, 23 Oct 2007) | 1 line Shorter name for namedtuple() ........ r58621 | andrew.kuchling | 2007-10-23 13:55:47 -0700 (Tue, 23 Oct 2007) | 1 line Update name ........ r58622 | raymond.hettinger | 2007-10-23 14:23:07 -0700 (Tue, 23 Oct 2007) | 1 line Fixup news entry ........ r58623 | raymond.hettinger | 2007-10-23 18:28:33 -0700 (Tue, 23 Oct 2007) | 1 line Optimize sum() for integer and float inputs. ........ r58624 | raymond.hettinger | 2007-10-23 19:05:51 -0700 (Tue, 23 Oct 2007) | 1 line Fixup error return and add support for intermixed ints and floats/ ........ r58628 | vinay.sajip | 2007-10-24 03:47:06 -0700 (Wed, 24 Oct 2007) | 1 line Bug #1321: Fixed logic error in TimedRotatingFileHandler.__init__() ........ r58641 | facundo.batista | 2007-10-24 12:11:08 -0700 (Wed, 24 Oct 2007) | 4 lines Issue 1290. CharacterData.__repr__ was constructing a string in response that keeped having a non-ascii character. ........ r58643 | thomas.heller | 2007-10-24 12:50:45 -0700 (Wed, 24 Oct 2007) | 1 line Added unittest for calling a function with paramflags (backport from py3k branch). ........ r58645 | matthias.klose | 2007-10-24 13:00:44 -0700 (Wed, 24 Oct 2007) | 2 lines - Build using system ffi library on arm*-linux*. ........ r58651 | georg.brandl | 2007-10-24 14:40:38 -0700 (Wed, 24 Oct 2007) | 2 lines Bug #1287: make os.environ.pop() work as expected. ........ r58652 | raymond.hettinger | 2007-10-24 19:26:58 -0700 (Wed, 24 Oct 2007) | 1 line Missing DECREFs ........ r58653 | matthias.klose | 2007-10-24 23:37:24 -0700 (Wed, 24 Oct 2007) | 2 lines - Build using system ffi library on arm*-linux*, pass --with-system-ffi to CONFIG_ARGS ........ r58655 | thomas.heller | 2007-10-25 12:47:32 -0700 (Thu, 25 Oct 2007) | 2 lines ffi_type_longdouble may be already #defined. See issue 1324. ........ r58656 | kurt.kaiser | 2007-10-25 15:43:45 -0700 (Thu, 25 Oct 2007) | 3 lines Correct an ancient bug in an unused path by removing that path: register() is now idempotent. ........ r58660 | kurt.kaiser | 2007-10-25 17:10:09 -0700 (Thu, 25 Oct 2007) | 4 lines 1. Add comments to provide top-level documentation. 2. Refactor to use more descriptive names. 3. Enhance tests in main(). ........ r58675 | georg.brandl | 2007-10-26 11:30:41 -0700 (Fri, 26 Oct 2007) | 2 lines Fix new pop() method on os.environ on ignorecase-platforms. ........ r58696 | neal.norwitz | 2007-10-27 15:32:21 -0700 (Sat, 27 Oct 2007) | 1 line Update URL for Pygments. 0.8.1 is no longer available ........ r58697 | hyeshik.chang | 2007-10-28 04:19:02 -0700 (Sun, 28 Oct 2007) | 3 lines - Add support for FreeBSD 8 which is recently forked from FreeBSD 7. - Regenerate IN module for most recent maintenance tree of FreeBSD 6 and 7. ........ r58698 | hyeshik.chang | 2007-10-28 05:38:09 -0700 (Sun, 28 Oct 2007) | 2 lines Enable platform-specific tweaks for FreeBSD 8 (exactly same to FreeBSD 7's yet) ........ r58700 | kurt.kaiser | 2007-10-28 12:03:59 -0700 (Sun, 28 Oct 2007) | 2 lines Add confirmation dialog before printing. Patch 1717170 Tal Einat. ........ r58706 | guido.van.rossum | 2007-10-29 13:52:45 -0700 (Mon, 29 Oct 2007) | 3 lines Patch 1353 by Jacob Winther. Add mp4 mapping to mimetypes.py. ........ r58709 | guido.van.rossum | 2007-10-29 15:15:05 -0700 (Mon, 29 Oct 2007) | 6 lines Backport fixes for the code that decodes octal escapes (and for PyString also hex escapes) -- this was reaching beyond the end of the input string buffer, even though it is not supposed to be \0-terminated. This has no visible effect but is clearly the correct thing to do. (In 3.0 it had a visible effect after removing ob_sstate from PyString.) ........ r58710 | kurt.kaiser | 2007-10-29 19:38:54 -0700 (Mon, 29 Oct 2007) | 7 lines check in Tal Einat's update to tabpage.py Patch 1612746 M configDialog.py M NEWS.txt AM tabbedpages.py ........ r58715 | georg.brandl | 2007-10-30 10:51:18 -0700 (Tue, 30 Oct 2007) | 2 lines Use correct markup. ........ r58716 | georg.brandl | 2007-10-30 10:57:12 -0700 (Tue, 30 Oct 2007) | 2 lines Make example about hiding None return values at the prompt clearer. ........ r58728 | neal.norwitz | 2007-10-30 23:33:20 -0700 (Tue, 30 Oct 2007) | 1 line Fix some compiler warnings for signed comparisons on Unix and Windows. ........ r58731 | martin.v.loewis | 2007-10-31 10:19:33 -0700 (Wed, 31 Oct 2007) | 2 lines Adding Christian Heimes. ........ r58737 | raymond.hettinger | 2007-10-31 14:57:58 -0700 (Wed, 31 Oct 2007) | 1 line Clarify the reasons why pickle is almost always better than marshal ........ r58739 | raymond.hettinger | 2007-10-31 15:15:49 -0700 (Wed, 31 Oct 2007) | 1 line Sets are marshalable. ........
2007-11-01 17:32:30 -03:00
efficient pickling of :term:`new-style class`\es.
2007-08-15 11:28:22 -03:00
* Protocol version 3 was added in Python 3.0. It has explicit support for
bytes and cannot be unpickled by Python 2.x pickle modules. This is
the current recommended protocol, use it whenever it is possible.
Refer to :pep:`307` for information about improvements brought by
protocol 2. See :mod:`pickletools`'s source code for extensive
comments about opcodes used by pickle protocols.
2007-08-15 11:28:22 -03:00
Module Interface
----------------
2007-08-15 11:28:22 -03:00
To serialize an object hierarchy, you first create a pickler, then you call the
pickler's :meth:`dump` method. To de-serialize a data stream, you first create
an unpickler, then you call the unpickler's :meth:`load` method. The
:mod:`pickle` module provides the following constant:
.. data:: HIGHEST_PROTOCOL
The highest protocol version available. This value can be passed as a
*protocol* value.
.. data:: DEFAULT_PROTOCOL
The default protocol used for pickling. May be less than HIGHEST_PROTOCOL.
Currently the default protocol is 3; a backward-incompatible protocol
designed for Python 3.0.
2007-08-15 11:28:22 -03:00
The :mod:`pickle` module provides the following functions to make the pickling
process more convenient:
.. function:: dump(obj, file[, protocol])
Write a pickled representation of *obj* to the open file object *file*. This
is equivalent to ``Pickler(file, protocol).dump(obj)``.
2007-08-15 11:28:22 -03:00
The optional *protocol* argument tells the pickler to use the given protocol;
supported protocols are 0, 1, 2, 3. The default protocol is 3; a
backward-incompatible protocol designed for Python 3.0.
2007-08-15 11:28:22 -03:00
Specifying a negative protocol version selects the highest protocol version
supported. The higher the protocol used, the more recent the version of
Python needed to read the pickle produced.
2007-08-15 11:28:22 -03:00
The *file* argument must have a write() method that accepts a single bytes
argument. It can thus be a file object opened for binary writing, a
io.BytesIO instance, or any other custom object that meets this interface.
2007-08-15 11:28:22 -03:00
.. function:: dumps(obj[, protocol])
2007-08-15 11:28:22 -03:00
Return the pickled representation of the object as a :class:`bytes`
object, instead of writing it to a file.
2007-08-15 11:28:22 -03:00
The optional *protocol* argument tells the pickler to use the given protocol;
supported protocols are 0, 1, 2, 3. The default protocol is 3; a
backward-incompatible protocol designed for Python 3.0.
2007-08-15 11:28:22 -03:00
Specifying a negative protocol version selects the highest protocol version
supported. The higher the protocol used, the more recent the version of
Python needed to read the pickle produced.
2007-08-15 11:28:22 -03:00
.. function:: load(file, [\*, encoding="ASCII", errors="strict"])
2007-08-15 11:28:22 -03:00
Read a pickled object representation from the open file object *file* and
return the reconstituted object hierarchy specified therein. This is
equivalent to ``Unpickler(file).load()``.
2007-08-15 11:28:22 -03:00
The protocol version of the pickle is detected automatically, so no protocol
argument is needed. Bytes past the pickled object's representation are
ignored.
2007-08-15 11:28:22 -03:00
The argument *file* must have two methods, a read() method that takes an
integer argument, and a readline() method that requires no arguments. Both
methods should return bytes. Thus *file* can be a binary file object opened
for reading, a BytesIO object, or any other custom object that meets this
interface.
2007-08-15 11:28:22 -03:00
Optional keyword arguments are encoding and errors, which are used to decode
8-bit string instances pickled by Python 2.x. These default to 'ASCII' and
'strict', respectively.
2007-08-15 11:28:22 -03:00
.. function:: loads(bytes_object, [\*, encoding="ASCII", errors="strict"])
2007-08-15 11:28:22 -03:00
Read a pickled object hierarchy from a :class:`bytes` object and return the
reconstituted object hierarchy specified therein
2007-08-15 11:28:22 -03:00
The protocol version of the pickle is detected automatically, so no protocol
argument is needed. Bytes past the pickled object's representation are
ignored.
2007-08-15 11:28:22 -03:00
Optional keyword arguments are encoding and errors, which are used to decode
8-bit string instances pickled by Python 2.x. These default to 'ASCII' and
'strict', respectively.
The :mod:`pickle` module defines three exceptions:
2007-08-15 11:28:22 -03:00
.. exception:: PickleError
Common base class for the other pickling exceptions. It inherits
2007-08-15 11:28:22 -03:00
:exc:`Exception`.
.. exception:: PicklingError
Error raised when an unpicklable object is encountered by :class:`Pickler`.
It inherits :exc:`PickleError`.
2007-08-15 11:28:22 -03:00
Refer to :ref:`pickle-picklable` to learn what kinds of objects can be
pickled.
2007-08-15 11:28:22 -03:00
.. exception:: UnpicklingError
Error raised when there a problem unpickling an object, such as a data
corruption or a security violation. It inherits :exc:`PickleError`.
2007-08-15 11:28:22 -03:00
Note that other exceptions may also be raised during unpickling, including
(but not necessarily limited to) AttributeError, EOFError, ImportError, and
IndexError.
2007-08-15 11:28:22 -03:00
The :mod:`pickle` module exports two classes, :class:`Pickler` and
:class:`Unpickler`:
2007-08-15 11:28:22 -03:00
.. class:: Pickler(file[, protocol])
2007-08-15 11:28:22 -03:00
This takes a binary file for writing a pickle data stream.
2007-08-15 11:28:22 -03:00
The optional *protocol* argument tells the pickler to use the given protocol;
supported protocols are 0, 1, 2, 3. The default protocol is 3; a
backward-incompatible protocol designed for Python 3.0.
2007-08-15 11:28:22 -03:00
Specifying a negative protocol version selects the highest protocol version
supported. The higher the protocol used, the more recent the version of
Python needed to read the pickle produced.
2007-08-15 11:28:22 -03:00
The *file* argument must have a write() method that accepts a single bytes
argument. It can thus be a file object opened for binary writing, a
io.BytesIO instance, or any other custom object that meets this interface.
2007-08-15 11:28:22 -03:00
.. method:: dump(obj)
2007-08-15 11:28:22 -03:00
Write a pickled representation of *obj* to the open file object given in
the constructor.
.. method:: persistent_id(obj)
2007-08-15 11:28:22 -03:00
Do nothing by default. This exists so a subclass can override it.
If :meth:`persistent_id` returns ``None``, *obj* is pickled as usual. Any
other value causes :class:`Pickler` to emit the returned value as a
persistent ID for *obj*. The meaning of this persistent ID should be
defined by :meth:`Unpickler.persistent_load`. Note that the value
returned by :meth:`persistent_id` cannot itself have a persistent ID.
See :ref:`pickle-persistent` for details and examples of uses.
2007-08-15 11:28:22 -03:00
.. method:: clear_memo()
2007-08-15 11:28:22 -03:00
Deprecated. Use the :meth:`clear` method on :attr:`memo`, instead.
Clear the pickler's memo, useful when reusing picklers.
.. attribute:: fast
Deprecated. Enable fast mode if set to a true value. The fast mode
disables the usage of memo, therefore speeding the pickling process by not
generating superfluous PUT opcodes. It should not be used with
self-referential objects, doing otherwise will cause :class:`Pickler` to
recurse infinitely.
Use :func:`pickletools.optimize` if you need more compact pickles.
.. attribute:: memo
Dictionary holding previously pickled objects to allow shared or
recursive objects to pickled by reference as opposed to by value.
2007-08-15 11:28:22 -03:00
.. XXX Move these comments to somewhere more appropriate.
2007-08-15 11:28:22 -03:00
It is possible to make multiple calls to the :meth:`dump` method of the same
:class:`Pickler` instance. These must then be matched to the same number of
calls to the :meth:`load` method of the corresponding :class:`Unpickler`
instance. If the same object is pickled by multiple :meth:`dump` calls, the
:meth:`load` will all yield references to the same object.
2007-08-15 11:28:22 -03:00
Please note, this is intended for pickling multiple objects without intervening
modifications to the objects or their parts. If you modify an object and then
pickle it again using the same :class:`Pickler` instance, the object is not
pickled again --- a reference to it is pickled and the :class:`Unpickler` will
return the old value, not the modified one.
2007-08-15 11:28:22 -03:00
.. class:: Unpickler(file, [\*, encoding="ASCII", errors="strict"])
2007-08-15 11:28:22 -03:00
This takes a binary file for reading a pickle data stream.
2007-08-15 11:28:22 -03:00
The protocol version of the pickle is detected automatically, so no
protocol argument is needed.
2007-08-15 11:28:22 -03:00
The argument *file* must have two methods, a read() method that takes an
integer argument, and a readline() method that requires no arguments. Both
methods should return bytes. Thus *file* can be a binary file object opened
for reading, a BytesIO object, or any other custom object that meets this
interface.
2007-08-15 11:28:22 -03:00
Optional keyword arguments are encoding and errors, which are used to decode
8-bit string instances pickled by Python 2.x. These default to 'ASCII' and
'strict', respectively.
2007-08-15 11:28:22 -03:00
.. method:: load()
2007-08-15 11:28:22 -03:00
Read a pickled object representation from the open file object given in
the constructor, and return the reconstituted object hierarchy specified
therein. Bytes past the pickled object's representation are ignored.
2007-08-15 11:28:22 -03:00
.. method:: persistent_load(pid)
2007-08-15 11:28:22 -03:00
Raise an :exc:`UnpickingError` by default.
2007-08-15 11:28:22 -03:00
If defined, :meth:`persistent_load` should return the object specified by
the persistent ID *pid*. If an invalid persistent ID is encountered, an
:exc:`UnpickingError` should be raised.
2007-08-15 11:28:22 -03:00
See :ref:`pickle-persistent` for details and examples of uses.
.. method:: find_class(module, name)
Import *module* if necessary and return the object called *name* from it,
where the *module* and *name* arguments are :class:`str` objects. Note,
unlike its name suggests, :meth:`find_class` is also used for finding
functions.
Subclasses may override this to gain control over what type of objects and
how they can be loaded, potentially reducing security risks. Refer to
:ref:`pickle-restrict` for details.
2007-08-15 11:28:22 -03:00
.. _pickle-picklable:
2007-08-15 11:28:22 -03:00
What can be pickled and unpickled?
----------------------------------
The following types can be pickled:
* ``None``, ``True``, and ``False``
* integers, floating point numbers, complex numbers
2007-08-15 11:28:22 -03:00
* strings, bytes, bytearrays
2007-08-15 11:28:22 -03:00
* tuples, lists, sets, and dictionaries containing only picklable objects
* functions defined at the top level of a module
* built-in functions defined at the top level of a module
* classes that are defined at the top level of a module
* instances of such classes whose :attr:`__dict__` or :meth:`__setstate__` is
picklable (see section :ref:`pickle-inst` for details)
2007-08-15 11:28:22 -03:00
Attempts to pickle unpicklable objects will raise the :exc:`PicklingError`
exception; when this happens, an unspecified number of bytes may have already
been written to the underlying file. Trying to pickle a highly recursive data
2007-08-15 11:28:22 -03:00
structure may exceed the maximum recursion depth, a :exc:`RuntimeError` will be
raised in this case. You can carefully raise this limit with
2007-08-15 11:28:22 -03:00
:func:`sys.setrecursionlimit`.
Note that functions (built-in and user-defined) are pickled by "fully qualified"
name reference, not by value. This means that only the function name is
pickled, along with the name of module the function is defined in. Neither the
function's code, nor any of its function attributes are pickled. Thus the
defining module must be importable in the unpickling environment, and the module
must contain the named object, otherwise an exception will be raised. [#]_
Similarly, classes are pickled by named reference, so the same restrictions in
the unpickling environment apply. Note that none of the class's code or data is
pickled, so in the following example the class attribute ``attr`` is not
restored in the unpickling environment::
class Foo:
attr = 'A class attribute'
2007-08-15 11:28:22 -03:00
picklestring = pickle.dumps(Foo)
These restrictions are why picklable functions and classes must be defined in
the top level of a module.
Similarly, when class instances are pickled, their class's code and data are not
pickled along with them. Only the instance data are pickled. This is done on
purpose, so you can fix bugs in a class or add methods to the class and still
load objects that were created with an earlier version of the class. If you
plan to have long-lived objects that will see many versions of a class, it may
be worthwhile to put a version number in the objects so that suitable
conversions can be made by the class's :meth:`__setstate__` method.
.. _pickle-inst:
2007-08-15 11:28:22 -03:00
Pickling Class Instances
------------------------
2007-08-15 11:28:22 -03:00
In this section, we describe the general mechanisms available to you to define,
customize, and control how class instances are pickled and unpickled.
2007-08-15 11:28:22 -03:00
In most cases, no additional code is needed to make instances picklable. By
default, pickle will retrieve the class and the attributes of an instance via
introspection. When a class instance is unpickled, its :meth:`__init__` method
is usually *not* invoked. The default behaviour first creates an uninitialized
instance and then restores the saved attributes. The following code shows an
implementation of this behaviour::
2007-08-15 11:28:22 -03:00
def save(obj):
return (obj.__class__, obj.__dict__)
2007-08-15 11:28:22 -03:00
def load(cls, attributes):
obj = cls.__new__(cls)
obj.__dict__.update(attributes)
return obj
2007-08-15 11:28:22 -03:00
.. index:: single: __getnewargs__() (copy protocol)
2007-08-15 11:28:22 -03:00
Classes can alter the default behaviour by providing one or severals special
methods. In protocol 2 and newer, classes that implements the
:meth:`__getnewargs__` method can dictate the values passed to the
:meth:`__new__` method upon unpickling. This is often needed for classes
whose :meth:`__new__` method requires arguments.
.. index:: single: __getstate__() (copy protocol)
2007-08-15 11:28:22 -03:00
Classes can further influence how their instances are pickled; if the class
defines the method :meth:`__getstate__`, it is called and the returned object is
pickled as the contents for the instance, instead of the contents of the
instance's dictionary. If the :meth:`__getstate__` method is absent, the
instance's :attr:`__dict__` is pickled as usual.
2007-08-15 11:28:22 -03:00
.. index:: single: __setstate__() (copy protocol)
2007-08-15 11:28:22 -03:00
Upon unpickling, if the class defines :meth:`__setstate__`, it is called with
the unpickled state. In that case, there is no requirement for the state object
to be a dictionary. Otherwise, the pickled state must be a dictionary and its
items are assigned to the new instance's dictionary.
2007-08-15 11:28:22 -03:00
.. note::
2007-08-15 11:28:22 -03:00
If :meth:`__getstate__` returns a false value, the :meth:`__setstate__`
method will not be called.
2007-08-15 11:28:22 -03:00
Refer to the section :ref:`pickle-state` for more information about how to use
the methods :meth:`__getstate__` and :meth:`__setstate__`.
2007-08-15 11:28:22 -03:00
.. index::
pair: copy; protocol
single: __reduce__() (copy protocol)
2007-08-15 11:28:22 -03:00
As we shall see, pickle does not use directly the methods described above. In
fact, these methods are part of the copy protocol which implements the
:meth:`__reduce__` special method. The copy protocol provides a unified
interface for retrieving the data necessary for pickling and copying
2009-01-03 17:18:54 -04:00
objects. [#]_
2007-08-15 11:28:22 -03:00
Although powerful, implementing :meth:`__reduce__` directly in your classes is
error prone. For this reason, class designers should use the high-level
interface (i.e., :meth:`__getnewargs__`, :meth:`__getstate__` and
:meth:`__setstate__`) whenever possible. We will show however cases where using
:meth:`__reduce__` is the only option or leads to more efficient pickling or
both.
2007-08-15 11:28:22 -03:00
The interface is currently defined as follow. The :meth:`__reduce__` method
takes no argument and shall return either a string or preferably a tuple (the
returned object is often refered as the "reduce value").
2007-08-15 11:28:22 -03:00
If a string is returned, the string should be interpreted as the name of a
global variable. It should be the object's local name relative to its module;
the pickle module searches the module namespace to determine the object's
module. This behaviour is typically useful for singletons.
2007-08-15 11:28:22 -03:00
When a tuple is returned, it must be between two and five items long. Optional
items can either be omitted, or ``None`` can be provided as their value. The
semantics of each item are in order:
2007-08-15 11:28:22 -03:00
.. XXX Mention __newobj__ special-case?
2007-08-15 11:28:22 -03:00
* A callable object that will be called to create the initial version of the
object.
2007-08-15 11:28:22 -03:00
* A tuple of arguments for the callable object. An empty tuple must be given if
the callable does not accept any argument.
2007-08-15 11:28:22 -03:00
* Optionally, the object's state, which will be passed to the object's
:meth:`__setstate__` method as previously described. If the object has no
such method then, the value must be a dictionary and it will be added to the
object's :attr:`__dict__` attribute.
* Optionally, an iterator (and not a sequence) yielding successive items. These
items will be appended to the object either using ``obj.append(item)`` or, in
batch, using ``obj.extend(list_of_items)``. This is primarily used for list
subclasses, but may be used by other classes as long as they have
2007-08-15 11:28:22 -03:00
:meth:`append` and :meth:`extend` methods with the appropriate signature.
(Whether :meth:`append` or :meth:`extend` is used depends on which pickle
protocol version is used as well as the number of items to append, so both
must be supported.)
* Optionally, an iterator (not a sequence) yielding successive key-value pairs.
These items will be stored to the object using ``obj[key] = value``. This is
primarily used for dictionary subclasses, but may be used by other classes as
long as they implement :meth:`__setitem__`.
.. index:: single: __reduce_ex__() (copy protocol)
2007-08-15 11:28:22 -03:00
Alternatively, a :meth:`__reduce_ex__` method may be defined. The only
difference is this method should take a single integer argument, the protocol
version. When defined, pickle will prefer it over the :meth:`__reduce__`
method. In addition, :meth:`__reduce__` automatically becomes a synonym for the
extended version. The main use for this method is to provide
backwards-compatible reduce values for older Python releases.
2007-08-15 11:28:22 -03:00
.. _pickle-persistent:
Persistence of External Objects
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
2007-08-15 11:28:22 -03:00
2008-02-23 14:30:17 -04:00
.. index::
single: persistent_id (pickle protocol)
single: persistent_load (pickle protocol)
2007-08-15 11:28:22 -03:00
For the benefit of object persistence, the :mod:`pickle` module supports the
notion of a reference to an object outside the pickled data stream. Such
objects are referenced by a persistent ID, which should be either a string of
alphanumeric characters (for protocol 0) [#]_ or just an arbitrary object (for
any newer protocol).
2007-08-15 11:28:22 -03:00
The resolution of such persistent IDs is not defined by the :mod:`pickle`
module; it will delegate this resolution to the user defined methods on the
pickler and unpickler, :meth:`persistent_id` and :meth:`persistent_load`
respectively.
2007-08-15 11:28:22 -03:00
To pickle objects that have an external persistent id, the pickler must have a
custom :meth:`persistent_id` method that takes an object as an argument and
2007-08-15 11:28:22 -03:00
returns either ``None`` or the persistent id for that object. When ``None`` is
returned, the pickler simply pickles the object as normal. When a persistent ID
string is returned, the pickler will pickle that object, along with a marker so
that the unpickler will recognize it as a persistent ID.
2007-08-15 11:28:22 -03:00
To unpickle external objects, the unpickler must have a custom
:meth:`persistent_load` method that takes a persistent ID object and returns the
referenced object.
2007-08-15 11:28:22 -03:00
Here is a comprehensive example presenting how persistent ID can be used to
pickle external objects by reference.
2007-08-15 11:28:22 -03:00
.. literalinclude:: ../includes/dbpickle.py
2009-01-23 01:28:16 -04:00
2007-08-15 11:28:22 -03:00
.. _pickle-state:
Handling Stateful Objects
^^^^^^^^^^^^^^^^^^^^^^^^^
.. index::
single: __getstate__() (copy protocol)
single: __setstate__() (copy protocol)
Here's an example that shows how to modify pickling behavior for a class.
The :class:`TextReader` class opens a text file, and returns the line number and
line contents each time its :meth:`readline` method is called. If a
:class:`TextReader` instance is pickled, all attributes *except* the file object
member are saved. When the instance is unpickled, the file is reopened, and
reading resumes from the last location. The :meth:`__setstate__` and
:meth:`__getstate__` methods are used to implement this behavior. ::
class TextReader:
"""Print and number lines in a text file."""
def __init__(self, filename):
self.filename = filename
self.file = open(filename)
self.lineno = 0
def readline(self):
self.lineno += 1
line = self.file.readline()
if not line:
return None
if line.endswith("\n"):
line = line[:-1]
return "%i: %s" % (self.lineno, line)
def __getstate__(self):
# Copy the object's state from self.__dict__ which contains
# all our instance attributes. Always use the dict.copy()
# method to avoid modifying the original state.
state = self.__dict__.copy()
# Remove the unpicklable entries.
del state['file']
return state
def __setstate__(self, state):
# Restore instance attributes (i.e., filename and lineno).
self.__dict__.update(state)
# Restore the previously opened file's state. To do so, we need to
# reopen it and read from it until the line count is restored.
file = open(self.filename)
for _ in range(self.lineno):
file.readline()
# Finally, save the file.
self.file = file
A sample usage might be something like this::
>>> reader = TextReader("hello.txt")
>>> reader.readline()
'1: Hello world!'
>>> reader.readline()
'2: I am line number two.'
>>> new_reader = pickle.loads(pickle.dumps(reader))
>>> new_reader.readline()
'3: Goodbye!'
.. _pickle-restrict:
2007-08-15 11:28:22 -03:00
Restricting Globals
-------------------
2007-08-15 11:28:22 -03:00
2008-02-23 14:30:17 -04:00
.. index::
single: find_class() (pickle protocol)
By default, unpickling will import any class or function that it finds in the
pickle data. For many applications, this behaviour is unacceptable as it
permits the unpickler to import and invoke arbitrary code. Just consider what
this hand-crafted pickle data stream does when loaded::
>>> import pickle
>>> pickle.loads(b"cos\nsystem\n(S'echo hello world'\ntR.")
hello world
0
In this example, the unpickler imports the :func:`os.system` function and then
apply the string argument "echo hello world". Although this example is
inoffensive, it is not difficult to imagine one that could damage your system.
For this reason, you may want to control what gets unpickled by customizing
:meth:`Unpickler.find_class`. Unlike its name suggests, :meth:`find_class` is
called whenever a global (i.e., a class or a function) is requested. Thus it is
possible to either forbid completely globals or restrict them to a safe subset.
Here is an example of an unpickler allowing only few safe classes from the
:mod:`builtins` module to be loaded::
import builtins
import io
import pickle
2007-08-15 11:28:22 -03:00
safe_builtins = {
'range',
'complex',
'set',
'frozenset',
'slice',
}
class RestrictedUnpickler(pickle.Unpickler):
def find_class(self, module, name):
# Only allow safe classes from builtins.
if module == "builtins" and name in safe_builtins:
return getattr(builtins, name)
# Forbid everything else.
raise pickle.UnpicklingError("global '%s.%s' is forbidden" %
(module, name))
def restricted_loads(s):
"""Helper function analogous to pickle.loads()."""
return RestrictedUnpickler(io.BytesIO(s)).load()
A sample usage of our unpickler working has intended::
>>> restricted_loads(pickle.dumps([1, 2, range(15)]))
[1, 2, range(0, 15)]
>>> restricted_loads(b"cos\nsystem\n(S'echo hello world'\ntR.")
Traceback (most recent call last):
...
pickle.UnpicklingError: global 'os.system' is forbidden
>>> restricted_loads(b'cbuiltins\neval\n'
... b'(S\'getattr(__import__("os"), "system")'
... b'("echo hello world")\'\ntR.')
Traceback (most recent call last):
...
pickle.UnpicklingError: global 'builtins.eval' is forbidden
.. XXX Add note about how extension codes could evade our protection
2009-01-03 17:18:54 -04:00
mechanism (e.g. cached classes do not invokes find_class()).
As our examples shows, you have to be careful with what you allow to be
unpickled. Therefore if security is a concern, you may want to consider
alternatives such as the marshalling API in :mod:`xmlrpc.client` or third-party
solutions.
2007-08-15 11:28:22 -03:00
.. _pickle-example:
2009-01-23 01:28:16 -04:00
Usage Examples
--------------
2007-08-15 11:28:22 -03:00
For the simplest code, use the :func:`dump` and :func:`load` functions. Note
that a self-referencing list is pickled and restored correctly. ::
import pickle
2009-01-23 01:28:16 -04:00
# An arbitrary collection of objects supported by pickle.
data = {
'a': [1, 2.0, 3, 4+6j],
'b': ("character string", b"byte string"),
'c': set([None, True, False])
}
2007-08-15 11:28:22 -03:00
2009-01-23 01:28:16 -04:00
with open('data.pickle', 'wb') as f:
# Pickle the 'data' dictionary using the highest protocol available.
pickle.dump(data, f, pickle.HIGHEST_PROTOCOL)
2007-08-15 11:28:22 -03:00
2009-01-23 01:28:16 -04:00
The following example reads the resulting pickled data. ::
2007-08-15 11:28:22 -03:00
2009-01-23 01:28:16 -04:00
import pickle
2007-08-15 11:28:22 -03:00
2009-01-23 01:28:16 -04:00
with open('data.pickle', 'rb') as f:
# The protocol version used is detected automatically, so we do not
# have to specify it.
data = pickle.load(f)
2007-08-15 11:28:22 -03:00
.. seealso::
Module :mod:`copyreg`
2007-08-15 11:28:22 -03:00
Pickle interface constructor registration for extension types.
Module :mod:`shelve`
Indexed databases of objects; uses :mod:`pickle`.
Module :mod:`copy`
Shallow and deep object copying.
Module :mod:`marshal`
High-performance serialization of built-in types.
.. rubric:: Footnotes
.. [#] Don't confuse this with the :mod:`marshal` module
.. [#] The exception raised will likely be an :exc:`ImportError` or an
:exc:`AttributeError` but it could be something else.
.. [#] The :mod:`copy` module uses this protocol for shallow and deep copying
operations.
.. [#] The limitation on alphanumeric characters is due to the fact
the persistent IDs, in protocol 0, are delimited by the newline
character. Therefore if any kind of newline characters occurs in
persistent IDs, the resulting pickle will become unreadable.