cpython/Doc/library/shelve.rst

202 lines
7.8 KiB
ReStructuredText
Raw Normal View History

2007-08-15 11:28:01 -03:00
:mod:`shelve` --- Python object persistence
===========================================
.. module:: shelve
:synopsis: Python object persistence.
.. index:: module: pickle
A "shelf" is a persistent, dictionary-like object. The difference with "dbm"
databases is that the values (not the keys!) in a shelf can be essentially
arbitrary Python objects --- anything that the :mod:`pickle` module can handle.
This includes most class instances, recursive data types, and objects containing
lots of shared sub-objects. The keys are ordinary strings.
.. function:: open(filename[, flag='c'[, protocol=None[, writeback=False]]])
Open a persistent dictionary. The filename specified is the base filename for
the underlying database. As a side-effect, an extension may be added to the
filename and more than one file may be created. By default, the underlying
database file is opened for reading and writing. The optional *flag* parameter
has the same interpretation as the *flag* parameter of :func:`anydbm.open`.
By default, version 0 pickles are used to serialize values. The version of the
pickle protocol can be specified with the *protocol* parameter.
.. versionchanged:: 2.3
The *protocol* parameter was added.
Because of Python semantics, a shelf cannot know when a mutable
persistent-dictionary entry is modified. By default modified objects are
Merged revisions 74861-74863,74876,74896,74930,74933,74952-74953,75015,75019,75260-75263,75265-75266,75289 via svnmerge from svn+ssh://pythondev@svn.python.org/python/trunk ........ r74861 | benjamin.peterson | 2009-09-17 05:18:28 +0200 (Do, 17 Sep 2009) | 1 line pep 8 defaults ........ r74862 | brett.cannon | 2009-09-17 05:24:45 +0200 (Do, 17 Sep 2009) | 1 line Note in the intro to Extending... that ctypes can be a simpler, more portable solution than custom C code. ........ r74863 | benjamin.peterson | 2009-09-17 05:27:33 +0200 (Do, 17 Sep 2009) | 1 line rationalize a bit ........ r74876 | georg.brandl | 2009-09-17 18:15:53 +0200 (Do, 17 Sep 2009) | 1 line #6932: remove paragraph that advises relying on __del__ being called. ........ r74896 | georg.brandl | 2009-09-18 09:22:41 +0200 (Fr, 18 Sep 2009) | 1 line #6936: for interactive use, quit() is just fine. ........ r74930 | georg.brandl | 2009-09-18 23:21:41 +0200 (Fr, 18 Sep 2009) | 1 line #6925: rewrite docs for locals() and vars() a bit. ........ r74933 | georg.brandl | 2009-09-18 23:35:59 +0200 (Fr, 18 Sep 2009) | 1 line #6930: clarify description about byteorder handling in UTF decoder routines. ........ r74952 | georg.brandl | 2009-09-19 12:42:34 +0200 (Sa, 19 Sep 2009) | 1 line #6946: fix duplicate index entries for datetime classes. ........ r74953 | georg.brandl | 2009-09-19 14:04:16 +0200 (Sa, 19 Sep 2009) | 1 line Fix references to threading.enumerate(). ........ r75015 | georg.brandl | 2009-09-22 12:55:08 +0200 (Di, 22 Sep 2009) | 1 line Fix encoding name. ........ r75019 | vinay.sajip | 2009-09-22 19:23:41 +0200 (Di, 22 Sep 2009) | 1 line Fixed a typo, and added sections on optimization and using arbitrary objects as messages. ........ r75260 | andrew.kuchling | 2009-10-05 23:24:20 +0200 (Mo, 05 Okt 2009) | 1 line Wording fix ........ r75261 | andrew.kuchling | 2009-10-05 23:24:35 +0200 (Mo, 05 Okt 2009) | 1 line Fix narkup ........ r75262 | andrew.kuchling | 2009-10-05 23:25:03 +0200 (Mo, 05 Okt 2009) | 1 line Document 'skip' parameter to constructor ........ r75263 | andrew.kuchling | 2009-10-05 23:25:35 +0200 (Mo, 05 Okt 2009) | 1 line Note side benefit of socket.create_connection() ........ r75265 | andrew.kuchling | 2009-10-06 00:31:11 +0200 (Di, 06 Okt 2009) | 1 line Reword sentence ........ r75266 | andrew.kuchling | 2009-10-06 00:32:48 +0200 (Di, 06 Okt 2009) | 1 line Use standard comma punctuation; reword some sentences in the docs ........ r75289 | mark.dickinson | 2009-10-08 22:02:25 +0200 (Do, 08 Okt 2009) | 2 lines Issue #7051: Clarify behaviour of 'g' and 'G'-style formatting. ........
2009-10-27 11:59:26 -03:00
written only when assigned to the shelf (see :ref:`shelve-example`). If the
optional *writeback* parameter is set to *True*, all entries accessed are
cached in memory, and written back on :meth:`sync` and :meth:`close`; this
can make it handier to mutate mutable entries in the persistent dictionary,
but, if many entries are accessed, it can consume vast amounts of memory for
the cache, and it can make the close operation very slow since all accessed
entries are written back (there is no way to determine which accessed entries
are mutable, nor which ones were actually mutated).
.. note::
Do not rely on the shelf being closed automatically; always call
:meth:`close` explicitly when you don't need it any more, or use a
:keyword:`with` statement with :func:`contextlib.closing`.
2007-08-15 11:28:01 -03:00
Merged revisions 72319-72320,72467,72661,72675-72679,72703,72708,72710,72712,72801-72802,72820,72822,72824,72826-72828,72830 via svnmerge from svn+ssh://pythondev@svn.python.org/python/trunk ........ r72319 | georg.brandl | 2009-05-05 10:28:49 +0200 (Di, 05 Mai 2009) | 1 line #1309567: fix linecache behavior of stripping subdirectories from paths when looking for relative filename matches. Also add a linecache test suite. ........ r72320 | georg.brandl | 2009-05-05 10:30:28 +0200 (Di, 05 Mai 2009) | 1 line Add a news entry for r72319. ........ r72467 | georg.brandl | 2009-05-08 14:17:34 +0200 (Fr, 08 Mai 2009) | 1 line Fix name. ........ r72661 | georg.brandl | 2009-05-15 10:03:03 +0200 (Fr, 15 Mai 2009) | 1 line Fix example output for doctest-like demos. ........ r72675 | georg.brandl | 2009-05-16 13:13:21 +0200 (Sa, 16 Mai 2009) | 1 line #6034: clarify __reversed__ doc. ........ r72676 | georg.brandl | 2009-05-16 13:14:46 +0200 (Sa, 16 Mai 2009) | 1 line #6025: fix signature of parse(). ........ r72677 | georg.brandl | 2009-05-16 13:18:55 +0200 (Sa, 16 Mai 2009) | 1 line #6009: undocument default argument of Option as deprecated. ........ r72678 | georg.brandl | 2009-05-16 13:21:29 +0200 (Sa, 16 Mai 2009) | 1 line #2856: document 2.x os.listdir() behavior for undecodable filenames. ........ r72679 | georg.brandl | 2009-05-16 13:24:41 +0200 (Sa, 16 Mai 2009) | 1 line Fix about and bugs pages to match real workflow. ........ r72703 | georg.brandl | 2009-05-17 10:10:27 +0200 (So, 17 Mai 2009) | 1 line part of #4144: fix exception message in console session. ........ r72708 | georg.brandl | 2009-05-17 10:24:29 +0200 (So, 17 Mai 2009) | 1 line #6017: better document behavior of dictiterators when the dict is changed. ........ r72710 | georg.brandl | 2009-05-17 10:36:04 +0200 (So, 17 Mai 2009) | 1 line #5942: Copy over flag table from dbm.rst which is clearer. ........ r72712 | georg.brandl | 2009-05-17 10:55:00 +0200 (So, 17 Mai 2009) | 1 line #5935: mention that BROWSER is looked for in PATH. ........ r72801 | georg.brandl | 2009-05-20 20:31:14 +0200 (Mi, 20 Mai 2009) | 1 line #6055: refer to "sqlite3" consistently. ........ r72802 | georg.brandl | 2009-05-20 20:35:27 +0200 (Mi, 20 Mai 2009) | 1 line #6051: refer to email examples for better way to construct email messages. ........ r72820 | georg.brandl | 2009-05-22 09:23:32 +0200 (Fr, 22 Mai 2009) | 1 line Use raise X(y). ........ r72822 | georg.brandl | 2009-05-22 11:33:25 +0200 (Fr, 22 Mai 2009) | 1 line #6084: fix example. ........ r72824 | georg.brandl | 2009-05-22 11:43:17 +0200 (Fr, 22 Mai 2009) | 1 line Fix references to file-related functions and methods (os.* vs file.*). ........ r72826 | georg.brandl | 2009-05-22 11:49:42 +0200 (Fr, 22 Mai 2009) | 1 line Fix confusing wording. ........ r72827 | georg.brandl | 2009-05-22 11:50:30 +0200 (Fr, 22 Mai 2009) | 1 line s/use/call/ ........ r72828 | georg.brandl | 2009-05-22 11:58:48 +0200 (Fr, 22 Mai 2009) | 1 line Correction in softspace behavior description. ........ r72830 | georg.brandl | 2009-05-22 12:40:00 +0200 (Fr, 22 Mai 2009) | 1 line #6086: fix spelling and use a better exception to catch. ........
2009-05-26 06:04:23 -03:00
Shelf objects support all methods supported by dictionaries. This eases the
2007-08-15 11:28:01 -03:00
transition from dictionary based scripts to those requiring persistent storage.
Merged revisions 74861-74863,74876,74896,74930,74933,74952-74953,75015,75019,75260-75263,75265-75266,75289 via svnmerge from svn+ssh://pythondev@svn.python.org/python/trunk ........ r74861 | benjamin.peterson | 2009-09-17 05:18:28 +0200 (Do, 17 Sep 2009) | 1 line pep 8 defaults ........ r74862 | brett.cannon | 2009-09-17 05:24:45 +0200 (Do, 17 Sep 2009) | 1 line Note in the intro to Extending... that ctypes can be a simpler, more portable solution than custom C code. ........ r74863 | benjamin.peterson | 2009-09-17 05:27:33 +0200 (Do, 17 Sep 2009) | 1 line rationalize a bit ........ r74876 | georg.brandl | 2009-09-17 18:15:53 +0200 (Do, 17 Sep 2009) | 1 line #6932: remove paragraph that advises relying on __del__ being called. ........ r74896 | georg.brandl | 2009-09-18 09:22:41 +0200 (Fr, 18 Sep 2009) | 1 line #6936: for interactive use, quit() is just fine. ........ r74930 | georg.brandl | 2009-09-18 23:21:41 +0200 (Fr, 18 Sep 2009) | 1 line #6925: rewrite docs for locals() and vars() a bit. ........ r74933 | georg.brandl | 2009-09-18 23:35:59 +0200 (Fr, 18 Sep 2009) | 1 line #6930: clarify description about byteorder handling in UTF decoder routines. ........ r74952 | georg.brandl | 2009-09-19 12:42:34 +0200 (Sa, 19 Sep 2009) | 1 line #6946: fix duplicate index entries for datetime classes. ........ r74953 | georg.brandl | 2009-09-19 14:04:16 +0200 (Sa, 19 Sep 2009) | 1 line Fix references to threading.enumerate(). ........ r75015 | georg.brandl | 2009-09-22 12:55:08 +0200 (Di, 22 Sep 2009) | 1 line Fix encoding name. ........ r75019 | vinay.sajip | 2009-09-22 19:23:41 +0200 (Di, 22 Sep 2009) | 1 line Fixed a typo, and added sections on optimization and using arbitrary objects as messages. ........ r75260 | andrew.kuchling | 2009-10-05 23:24:20 +0200 (Mo, 05 Okt 2009) | 1 line Wording fix ........ r75261 | andrew.kuchling | 2009-10-05 23:24:35 +0200 (Mo, 05 Okt 2009) | 1 line Fix narkup ........ r75262 | andrew.kuchling | 2009-10-05 23:25:03 +0200 (Mo, 05 Okt 2009) | 1 line Document 'skip' parameter to constructor ........ r75263 | andrew.kuchling | 2009-10-05 23:25:35 +0200 (Mo, 05 Okt 2009) | 1 line Note side benefit of socket.create_connection() ........ r75265 | andrew.kuchling | 2009-10-06 00:31:11 +0200 (Di, 06 Okt 2009) | 1 line Reword sentence ........ r75266 | andrew.kuchling | 2009-10-06 00:32:48 +0200 (Di, 06 Okt 2009) | 1 line Use standard comma punctuation; reword some sentences in the docs ........ r75289 | mark.dickinson | 2009-10-08 22:02:25 +0200 (Do, 08 Okt 2009) | 2 lines Issue #7051: Clarify behaviour of 'g' and 'G'-style formatting. ........
2009-10-27 11:59:26 -03:00
Two additional methods are supported:
2007-08-15 11:28:01 -03:00
.. method:: Shelf.sync()
Merged revisions 74861-74863,74876,74896,74930,74933,74952-74953,75015,75019,75260-75263,75265-75266,75289 via svnmerge from svn+ssh://pythondev@svn.python.org/python/trunk ........ r74861 | benjamin.peterson | 2009-09-17 05:18:28 +0200 (Do, 17 Sep 2009) | 1 line pep 8 defaults ........ r74862 | brett.cannon | 2009-09-17 05:24:45 +0200 (Do, 17 Sep 2009) | 1 line Note in the intro to Extending... that ctypes can be a simpler, more portable solution than custom C code. ........ r74863 | benjamin.peterson | 2009-09-17 05:27:33 +0200 (Do, 17 Sep 2009) | 1 line rationalize a bit ........ r74876 | georg.brandl | 2009-09-17 18:15:53 +0200 (Do, 17 Sep 2009) | 1 line #6932: remove paragraph that advises relying on __del__ being called. ........ r74896 | georg.brandl | 2009-09-18 09:22:41 +0200 (Fr, 18 Sep 2009) | 1 line #6936: for interactive use, quit() is just fine. ........ r74930 | georg.brandl | 2009-09-18 23:21:41 +0200 (Fr, 18 Sep 2009) | 1 line #6925: rewrite docs for locals() and vars() a bit. ........ r74933 | georg.brandl | 2009-09-18 23:35:59 +0200 (Fr, 18 Sep 2009) | 1 line #6930: clarify description about byteorder handling in UTF decoder routines. ........ r74952 | georg.brandl | 2009-09-19 12:42:34 +0200 (Sa, 19 Sep 2009) | 1 line #6946: fix duplicate index entries for datetime classes. ........ r74953 | georg.brandl | 2009-09-19 14:04:16 +0200 (Sa, 19 Sep 2009) | 1 line Fix references to threading.enumerate(). ........ r75015 | georg.brandl | 2009-09-22 12:55:08 +0200 (Di, 22 Sep 2009) | 1 line Fix encoding name. ........ r75019 | vinay.sajip | 2009-09-22 19:23:41 +0200 (Di, 22 Sep 2009) | 1 line Fixed a typo, and added sections on optimization and using arbitrary objects as messages. ........ r75260 | andrew.kuchling | 2009-10-05 23:24:20 +0200 (Mo, 05 Okt 2009) | 1 line Wording fix ........ r75261 | andrew.kuchling | 2009-10-05 23:24:35 +0200 (Mo, 05 Okt 2009) | 1 line Fix narkup ........ r75262 | andrew.kuchling | 2009-10-05 23:25:03 +0200 (Mo, 05 Okt 2009) | 1 line Document 'skip' parameter to constructor ........ r75263 | andrew.kuchling | 2009-10-05 23:25:35 +0200 (Mo, 05 Okt 2009) | 1 line Note side benefit of socket.create_connection() ........ r75265 | andrew.kuchling | 2009-10-06 00:31:11 +0200 (Di, 06 Okt 2009) | 1 line Reword sentence ........ r75266 | andrew.kuchling | 2009-10-06 00:32:48 +0200 (Di, 06 Okt 2009) | 1 line Use standard comma punctuation; reword some sentences in the docs ........ r75289 | mark.dickinson | 2009-10-08 22:02:25 +0200 (Do, 08 Okt 2009) | 2 lines Issue #7051: Clarify behaviour of 'g' and 'G'-style formatting. ........
2009-10-27 11:59:26 -03:00
Write back all entries in the cache if the shelf was opened with *writeback*
set to :const:`True`. Also empty the cache and synchronize the persistent
dictionary on disk, if feasible. This is called automatically when the shelf
is closed with :meth:`close`.
.. method:: Shelf.close()
Synchronize and close the persistent *dict* object. Operations on a closed
shelf will fail with a :exc:`ValueError`.
2007-08-15 11:28:01 -03:00
2009-04-04 02:39:34 -03:00
.. seealso::
`Persistent dictionary recipe <http://code.activestate.com/recipes/576642/>`_
with widely supported storage formats and having the speed of native
dictionaries.
2007-08-15 11:28:01 -03:00
Restrictions
------------
.. index::
module: dbm
module: gdbm
module: bsddb
* The choice of which database package will be used (such as :mod:`dbm`,
:mod:`gdbm` or :mod:`bsddb`) depends on which interface is available. Therefore
it is not safe to open the database directly using :mod:`dbm`. The database is
also (unfortunately) subject to the limitations of :mod:`dbm`, if it is used ---
this means that (the pickled representation of) the objects stored in the
database should be fairly small, and in rare cases key collisions may cause the
database to refuse updates.
* The :mod:`shelve` module does not support *concurrent* read/write access to
shelved objects. (Multiple simultaneous read accesses are safe.) When a
program has a shelf open for writing, no other program should have it open for
reading or writing. Unix file locking can be used to solve this, but this
differs across Unix versions and requires knowledge about the database
implementation used.
.. class:: Shelf(dict[, protocol=None[, writeback=False]])
A subclass of :class:`UserDict.DictMixin` which stores pickled values in the
*dict* object.
By default, version 0 pickles are used to serialize values. The version of the
pickle protocol can be specified with the *protocol* parameter. See the
:mod:`pickle` documentation for a discussion of the pickle protocols.
.. versionchanged:: 2.3
The *protocol* parameter was added.
If the *writeback* parameter is ``True``, the object will hold a cache of all
entries accessed and write them back to the *dict* at sync and close times.
This allows natural operations on mutable entries, but can consume much more
memory and make sync and close take a long time.
.. class:: BsdDbShelf(dict[, protocol=None[, writeback=False]])
A subclass of :class:`Shelf` which exposes :meth:`first`, :meth:`!next`,
2007-08-15 11:28:01 -03:00
:meth:`previous`, :meth:`last` and :meth:`set_location` which are available in
the :mod:`bsddb` module but not in other database modules. The *dict* object
passed to the constructor must support those methods. This is generally
accomplished by calling one of :func:`bsddb.hashopen`, :func:`bsddb.btopen` or
:func:`bsddb.rnopen`. The optional *protocol* and *writeback* parameters have
the same interpretation as for the :class:`Shelf` class.
.. class:: DbfilenameShelf(filename[, flag='c'[, protocol=None[, writeback=False]]])
A subclass of :class:`Shelf` which accepts a *filename* instead of a dict-like
object. The underlying file will be opened using :func:`anydbm.open`. By
default, the file will be created and opened for both read and write. The
optional *flag* parameter has the same interpretation as for the :func:`.open`
2007-08-15 11:28:01 -03:00
function. The optional *protocol* and *writeback* parameters have the same
interpretation as for the :class:`Shelf` class.
.. _shelve-example:
2007-08-15 11:28:01 -03:00
Example
-------
To summarize the interface (``key`` is a string, ``data`` is an arbitrary
object)::
import shelve
d = shelve.open(filename) # open -- file may get suffix added by low-level
# library
d[key] = data # store data at key (overwrites old data if
# using an existing key)
data = d[key] # retrieve a COPY of data at key (raise KeyError if no
# such key)
del d[key] # delete data stored at key (raises KeyError
# if no such key)
flag = d.has_key(key) # true if the key exists
klist = d.keys() # a list of all existing keys (slow!)
# as d was opened WITHOUT writeback=True, beware:
d['xx'] = range(4) # this works as expected, but...
Merged revisions 71058,71149-71150,71212,71214-71216,71222,71225,71234,71237-71238,71240-71241,71243,71249,71251 via svnmerge from svn+ssh://pythondev@svn.python.org/python/trunk ........ r71058 | georg.brandl | 2009-04-02 20:09:04 +0200 (Do, 02 Apr 2009) | 3 lines PyErr_NormalizeException may not set an error, so convert the PyErr_SetObject call on hitting the recursion limit into just assigning it to the arguments provided. ........ r71149 | georg.brandl | 2009-04-04 15:42:39 +0200 (Sa, 04 Apr 2009) | 1 line #5642: clarify map() compatibility to the builtin. ........ r71150 | georg.brandl | 2009-04-04 15:45:49 +0200 (Sa, 04 Apr 2009) | 1 line #5601: clarify that webbrowser is not meant for file names. ........ r71212 | georg.brandl | 2009-04-05 12:24:20 +0200 (So, 05 Apr 2009) | 1 line #1742837: expand HTTP server docs, and fix SocketServer ones to document methods as methods, not functions. ........ r71214 | georg.brandl | 2009-04-05 12:29:57 +0200 (So, 05 Apr 2009) | 1 line Normalize spelling of Mac OS X. ........ r71215 | georg.brandl | 2009-04-05 12:32:26 +0200 (So, 05 Apr 2009) | 1 line Avoid sure signs of a diseased mind. ........ r71216 | georg.brandl | 2009-04-05 12:41:02 +0200 (So, 05 Apr 2009) | 1 line #1718017: document the relation of os.path and the posixpath, ntpath etc. modules better. ........ r71222 | georg.brandl | 2009-04-05 13:07:14 +0200 (So, 05 Apr 2009) | 1 line #5615: make it possible to configure --without-threads again. ........ r71225 | georg.brandl | 2009-04-05 13:54:07 +0200 (So, 05 Apr 2009) | 1 line #5580: no need to use parentheses when converterr() argument is actually a type description. ........ r71234 | georg.brandl | 2009-04-05 15:16:35 +0200 (So, 05 Apr 2009) | 1 line Whitespace normalization. ........ r71237 | georg.brandl | 2009-04-05 16:24:52 +0200 (So, 05 Apr 2009) | 1 line #1326077: fix traceback formatting of SyntaxErrors. This fixes two differences with formatting coming from Python: a) the reproduction of location details in the error message if no line text is given, b) the prefixing of the last line by one space. ........ r71238 | georg.brandl | 2009-04-05 16:25:41 +0200 (So, 05 Apr 2009) | 1 line Add NEWS entry for r71237. ........ r71240 | georg.brandl | 2009-04-05 16:40:06 +0200 (So, 05 Apr 2009) | 1 line #5370: doc update about unpickling objects with custom __getattr__ etc. methods. ........ r71241 | georg.brandl | 2009-04-05 16:48:49 +0200 (So, 05 Apr 2009) | 1 line #5471: fix expanduser() for $HOME set to "/". ........ r71243 | georg.brandl | 2009-04-05 17:14:29 +0200 (So, 05 Apr 2009) | 1 line #5432: make plistlib docstring a raw string, since it contains examples with backslash escapes. ........ r71249 | georg.brandl | 2009-04-05 18:30:43 +0200 (So, 05 Apr 2009) | 1 line #5444: adapt make.bat to new htmlhelp output file name. ........ r71251 | georg.brandl | 2009-04-05 19:17:42 +0200 (So, 05 Apr 2009) | 1 line #5298: clarify docs about GIL by using more consistent wording. ........
2009-04-05 18:26:31 -03:00
d['xx'].append(5) # *this doesn't!* -- d['xx'] is STILL range(4)!
2007-08-15 11:28:01 -03:00
# having opened d without writeback=True, you need to code carefully:
temp = d['xx'] # extracts the copy
temp.append(5) # mutates the copy
d['xx'] = temp # stores the copy right back, to persist it
# or, d=shelve.open(filename,writeback=True) would let you just code
# d['xx'].append(5) and have it work as expected, BUT it would also
# consume more memory and make the d.close() operation slower.
d.close() # close it
.. seealso::
Module :mod:`anydbm`
Generic interface to ``dbm``\ -style databases.
Module :mod:`bsddb`
BSD ``db`` database interface.
Module :mod:`dbhash`
Thin layer around the :mod:`bsddb` which provides an :func:`~dbhash.open`
function like the other database modules.
2007-08-15 11:28:01 -03:00
Module :mod:`dbm`
Standard Unix database interface.
Module :mod:`dumbdbm`
Portable implementation of the ``dbm`` interface.
Module :mod:`gdbm`
GNU database interface, based on the ``dbm`` interface.
Module :mod:`pickle`
Object serialization used by :mod:`shelve`.
Module :mod:`cPickle`
High-performance version of :mod:`pickle`.