cpython/Objects/dictnotes.txt

184 lines
7.6 KiB
Plaintext
Raw Normal View History

NOTES ON DICTIONARIES
================================
Principal Use Cases for Dictionaries
------------------------------------
Passing keyword arguments
Typically, one read and one write for 1 to 3 elements.
Occurs frequently in normal python code.
Class method lookup
Dictionaries vary in size with 8 to 16 elements being common.
Usually written once with many lookups.
When base classes are used, there are many failed lookups
followed by a lookup in a base class.
Instance attribute lookup and Global variables
Dictionaries vary in size. 4 to 10 elements are common.
Both reads and writes are common.
Builtins
Frequent reads. Almost never written.
About 150 interned strings (as of Py3.3).
A few keys are accessed much more frequently than others.
Uniquification
Dictionaries of any size. Bulk of work is in creation.
Repeated writes to a smaller set of keys.
Single read of each key.
Some use cases have two consecutive accesses to the same key.
* Removing duplicates from a sequence.
dict.fromkeys(seqn).keys()
* Counting elements in a sequence.
for e in seqn:
d[e] = d.get(e,0) + 1
* Accumulating references in a dictionary of lists:
for pagenumber, page in enumerate(pages):
for word in page:
d.setdefault(word, []).append(pagenumber)
Note, the second example is a use case characterized by a get and set
Merged revisions 53005-53303 via svnmerge from svn+ssh://pythondev@svn.python.org/python/trunk ........ r53012 | walter.doerwald | 2006-12-12 22:55:31 +0100 (Tue, 12 Dec 2006) | 2 lines Fix typo. ........ r53023 | brett.cannon | 2006-12-13 23:31:37 +0100 (Wed, 13 Dec 2006) | 2 lines Remove an unneeded import of 'warnings'. ........ r53025 | brett.cannon | 2006-12-14 00:02:38 +0100 (Thu, 14 Dec 2006) | 2 lines Remove unneeded imports of 'warnings'. ........ r53026 | brett.cannon | 2006-12-14 00:09:53 +0100 (Thu, 14 Dec 2006) | 4 lines Add test.test_support.guard_warnings_filter . This function returns a context manager that protects warnings.filter from being modified once the context is exited. ........ r53029 | george.yoshida | 2006-12-14 03:22:44 +0100 (Thu, 14 Dec 2006) | 2 lines Note that guard_warnings_filter was added in 2.6 ........ r53031 | vinay.sajip | 2006-12-14 09:53:55 +0100 (Thu, 14 Dec 2006) | 1 line Added news on recent changes to logging ........ r53032 | andrew.kuchling | 2006-12-14 19:57:53 +0100 (Thu, 14 Dec 2006) | 1 line [Patch #1599256 from David Watson] check that os.fsync is available before using it ........ r53042 | kurt.kaiser | 2006-12-15 06:13:11 +0100 (Fri, 15 Dec 2006) | 6 lines 1. Avoid hang when encountering a duplicate in a completion list. Bug 1571112. 2. Duplicate some old entries from Python's NEWS to IDLE's NEWS.txt M AutoCompleteWindow.py M NEWS.txt ........ r53048 | andrew.kuchling | 2006-12-18 18:12:31 +0100 (Mon, 18 Dec 2006) | 1 line [Bug #1618083] Add missing word; make a few grammar fixes ........ r53050 | andrew.kuchling | 2006-12-18 18:16:05 +0100 (Mon, 18 Dec 2006) | 1 line Bump version ........ r53051 | andrew.kuchling | 2006-12-18 18:22:07 +0100 (Mon, 18 Dec 2006) | 1 line [Bug #1616726] Fix description of generator.close(); if you raise some random exception, the exception is raised and doesn't trigger a RuntimeError ........ r53052 | andrew.kuchling | 2006-12-18 18:38:14 +0100 (Mon, 18 Dec 2006) | 1 line Describe new methods in Queue module ........ r53053 | andrew.kuchling | 2006-12-18 20:22:24 +0100 (Mon, 18 Dec 2006) | 1 line [Patch #1615868 by Lars Gustaebel] Use Py_off_t to fix BZ2File.seek() for offsets > 2Gb ........ r53057 | andrew.kuchling | 2006-12-18 22:29:07 +0100 (Mon, 18 Dec 2006) | 1 line Fix markup ........ r53063 | thomas.wouters | 2006-12-19 09:17:50 +0100 (Tue, 19 Dec 2006) | 5 lines Make sre's SubPattern objects accept slice objects like it already accepts simple slices. ........ r53065 | andrew.kuchling | 2006-12-19 15:13:05 +0100 (Tue, 19 Dec 2006) | 6 lines [Patch #1618455 by Ben Maurer] Improve speed of HMAC by using str.translate() instead of a more general XOR that has to construct a list. Slightly modified from Maurer's patch: the _strxor() function is no longer necessary at all. ........ r53066 | andrew.kuchling | 2006-12-19 15:28:23 +0100 (Tue, 19 Dec 2006) | 9 lines [Bug #1613651] Document socket.recv_into, socket.recvfrom_into Also, the text for recvfrom told you to read recv() for an explanation of the 'flags' argument, but recv() just pointed you at the man page. Copied the man-page text to recvfrom(), recvfrom_into, recv_into to avoid the pointless redirection. I don't have LaTeX on this machine; hope my markup is OK. ........ r53067 | andrew.kuchling | 2006-12-19 15:29:04 +0100 (Tue, 19 Dec 2006) | 1 line Comment typo ........ r53068 | andrew.kuchling | 2006-12-19 16:11:41 +0100 (Tue, 19 Dec 2006) | 1 line [Patch #1617413 from Dug Song] Fix HTTP Basic authentication via HTTPS ........ r53071 | andrew.kuchling | 2006-12-19 16:18:12 +0100 (Tue, 19 Dec 2006) | 1 line [Patch #1600491 from Jim Jewett] Describe how to build help files on Windows ........ r53073 | andrew.kuchling | 2006-12-19 16:43:10 +0100 (Tue, 19 Dec 2006) | 6 lines [Patch #1587139 by kxroberto] Protect lock acquisition/release with try...finally to ensure the lock is always released. This could use the 'with' statement, but the patch uses 'finally'. 2.5 backport candidate. ........ r53074 | vinay.sajip | 2006-12-19 19:29:11 +0100 (Tue, 19 Dec 2006) | 1 line Updated documentation for findCaller() to indicate that a 3-tuple is now returned, rather than a 2-tuple. ........ r53090 | georg.brandl | 2006-12-19 23:06:46 +0100 (Tue, 19 Dec 2006) | 3 lines Patch #1484695: The tarfile module now raises a HeaderError exception if a buffer given to frombuf() is invalid. ........ r53099 | raymond.hettinger | 2006-12-20 07:42:06 +0100 (Wed, 20 Dec 2006) | 5 lines Bug #1590891: random.randrange don't return correct value for big number Needs to be backported. ........ r53106 | georg.brandl | 2006-12-20 12:55:16 +0100 (Wed, 20 Dec 2006) | 3 lines Testcase for patch #1484695. ........ r53110 | andrew.kuchling | 2006-12-20 20:48:20 +0100 (Wed, 20 Dec 2006) | 17 lines [Apply length-checking.diff from bug #1599254] Add length checking to single-file mailbox formats: before doing a flush() on a mailbox, seek to the end and verify its length is unchanged, raising ExternalClashError if the file's length has changed. This fix avoids potential data loss if some other process appends to the mailbox file after the table of contents has been generated; instead of overwriting the modified file, you'll get the exception. I also noticed that the self._lookup() call in self.flush() wasn't necessary (everything that sets self._pending to True also calls self.lookup()), and replaced it by an assertion. 2.5 backport candidate. ........ r53112 | andrew.kuchling | 2006-12-20 20:57:10 +0100 (Wed, 20 Dec 2006) | 1 line [Bug #1619674] Make sum() use the term iterable, not sequence ........ r53113 | andrew.kuchling | 2006-12-20 20:58:11 +0100 (Wed, 20 Dec 2006) | 1 line Two grammar fixes ........ r53115 | andrew.kuchling | 2006-12-20 21:11:12 +0100 (Wed, 20 Dec 2006) | 5 lines Some other built-in functions are described with 'sequence' arguments that should really be 'iterable'; this commit changes them. Did I miss any? Did I introduce any errors? ........ r53117 | andrew.kuchling | 2006-12-20 21:20:42 +0100 (Wed, 20 Dec 2006) | 1 line [Bug #1619680] in_dll() arguments are documented in the wrong order ........ r53120 | neal.norwitz | 2006-12-21 05:38:00 +0100 (Thu, 21 Dec 2006) | 1 line Lars asked for permission on on python-dev for work on tarfile.py ........ r53125 | andrew.kuchling | 2006-12-21 14:40:29 +0100 (Thu, 21 Dec 2006) | 1 line Mention the os.SEEK_* constants ........ r53129 | walter.doerwald | 2006-12-21 19:06:30 +0100 (Thu, 21 Dec 2006) | 2 lines Fix typo. ........ r53131 | thomas.heller | 2006-12-21 19:30:56 +0100 (Thu, 21 Dec 2006) | 3 lines Fix wrong markup of an argument in a method signature. Will backport. ........ r53137 | andrew.kuchling | 2006-12-22 01:50:56 +0100 (Fri, 22 Dec 2006) | 1 line Typo fix ........ r53139 | andrew.kuchling | 2006-12-22 14:25:02 +0100 (Fri, 22 Dec 2006) | 1 line [Bug #737202; fix from Titus Brown] Make CGIHTTPServer work for scripts in sub-directories ........ r53141 | andrew.kuchling | 2006-12-22 16:04:45 +0100 (Fri, 22 Dec 2006) | 6 lines [Bug #802128] Make the mode argument of dumbdbm actually work the way it's described, and add a test for it. 2.5 bugfix candidate, maybe; arguably this patch changes the API of dumbdbm and shouldn't be added in a point-release. ........ r53142 | andrew.kuchling | 2006-12-22 16:16:58 +0100 (Fri, 22 Dec 2006) | 6 lines [Bug #802128 continued] Modify mode depending on the process umask. Is there really no other way to read the umask than to set it? Hope this works on Windows... ........ r53145 | andrew.kuchling | 2006-12-22 17:43:26 +0100 (Fri, 22 Dec 2006) | 1 line [Bug #776202] Apply Walter Doerwald's patch to use text mode for encoded files ........ r53146 | andrew.kuchling | 2006-12-22 19:41:42 +0100 (Fri, 22 Dec 2006) | 9 lines [Patch #783050 from Patrick Lynch] The emulation of forkpty() is incorrect; the master should close the slave fd. Added a test to test_pty.py that reads from the master_fd after doing a pty.fork(); without the fix it hangs forever instead of raising an exception. (<crossing fingers for the buildbots>) 2.5 backport candidate. ........ r53147 | andrew.kuchling | 2006-12-22 20:06:16 +0100 (Fri, 22 Dec 2006) | 1 line [Patch #827559 from Chris Gonnerman] Make SimpleHTTPServer redirect when a directory URL is missing the trailing slash; this lets relative links work. ........ r53149 | andrew.kuchling | 2006-12-22 20:21:27 +0100 (Fri, 22 Dec 2006) | 1 line Darn; this test works when you run test_pty.py directly, but fails when regrtest runs it (the os.read() raises os.error). I can't figure out the cause, so am commenting out the test. ........ r53150 | andrew.kuchling | 2006-12-22 22:48:19 +0100 (Fri, 22 Dec 2006) | 1 line Frak; this test also fails ........ r53153 | lars.gustaebel | 2006-12-23 17:40:13 +0100 (Sat, 23 Dec 2006) | 5 lines Patch #1230446: tarfile.py: fix ExFileObject so that read() and tell() work correctly together with readline(). Will backport to 2.5. ........ r53155 | lars.gustaebel | 2006-12-23 18:57:23 +0100 (Sat, 23 Dec 2006) | 5 lines Patch #1262036: Prevent TarFiles from being added to themselves under certain conditions. Will backport to 2.5. ........ r53159 | andrew.kuchling | 2006-12-27 04:25:31 +0100 (Wed, 27 Dec 2006) | 4 lines [Part of patch #1182394] Move the HMAC blocksize to be a class-level constant; this allows changing it in a subclass. To accommodate this, copy() now uses __class__. Also add some text to a comment. ........ r53160 | andrew.kuchling | 2006-12-27 04:31:24 +0100 (Wed, 27 Dec 2006) | 1 line [Rest of patch #1182394] Add ._current() method so that we can use the written-in-C .hexdigest() method ........ r53161 | lars.gustaebel | 2006-12-27 11:30:46 +0100 (Wed, 27 Dec 2006) | 4 lines Patch #1504073: Fix tarfile.open() for mode "r" with a fileobj argument. Will backport to 2.5. ........ r53165 | neal.norwitz | 2006-12-28 05:39:20 +0100 (Thu, 28 Dec 2006) | 1 line Remove a stray (old) macro name left around (I guess) ........ r53188 | neal.norwitz | 2006-12-29 04:01:53 +0100 (Fri, 29 Dec 2006) | 1 line SF bug #1623890, fix argument name in docstring ........ r53200 | raymond.hettinger | 2006-12-30 05:01:17 +0100 (Sat, 30 Dec 2006) | 1 line For sets with cyclical reprs, emit an ellipsis instead of infinitely recursing. ........ r53232 | brett.cannon | 2007-01-04 01:23:49 +0100 (Thu, 04 Jan 2007) | 3 lines Add EnvironmentVarGuard to test.test_support. Provides a context manager to temporarily set or unset environment variables. ........ r53235 | neal.norwitz | 2007-01-04 07:25:31 +0100 (Thu, 04 Jan 2007) | 1 line SF #1627373, fix typo in CarbonEvt. ........ r53244 | raymond.hettinger | 2007-01-04 18:53:34 +0100 (Thu, 04 Jan 2007) | 1 line Fix stability of heapq's nlargest() and nsmallest(). ........ r53249 | martin.v.loewis | 2007-01-04 22:06:12 +0100 (Thu, 04 Jan 2007) | 3 lines Bug #1566280: Explicitly invoke threading._shutdown from Py_Main, to avoid relying on atexit. Will backport to 2.5. ........ r53252 | gregory.p.smith | 2007-01-05 02:59:42 +0100 (Fri, 05 Jan 2007) | 3 lines Support linking of the bsddb module against BerkeleyDB 4.5.x (will backport to 2.5) ........ r53253 | gregory.p.smith | 2007-01-05 03:06:17 +0100 (Fri, 05 Jan 2007) | 2 lines bump module version to match supported berkeleydb version ........ r53255 | neal.norwitz | 2007-01-05 06:25:22 +0100 (Fri, 05 Jan 2007) | 6 lines Prevent crash on shutdown which can occur if we are finalizing and the module dict has been cleared already and some object raises a warning (like in a __del__). Will backport. ........ r53258 | gregory.p.smith | 2007-01-05 08:21:35 +0100 (Fri, 05 Jan 2007) | 2 lines typo fix ........ r53260 | neal.norwitz | 2007-01-05 09:06:43 +0100 (Fri, 05 Jan 2007) | 1 line Add Collin Winter for access to update PEP 3107 ........ r53262 | andrew.kuchling | 2007-01-05 15:22:17 +0100 (Fri, 05 Jan 2007) | 1 line [Bug #1622533] Make docstrings raw strings because they contain control characters (\0, \1) ........ r53264 | andrew.kuchling | 2007-01-05 16:51:24 +0100 (Fri, 05 Jan 2007) | 1 line [Patch #1520904] Fix bsddb tests to write to the temp directory instead of the Lib/bsddb/test directory ........ r53279 | brett.cannon | 2007-01-05 22:45:09 +0100 (Fri, 05 Jan 2007) | 3 lines Silence a warning from gcc 4.0.1 by specifying a function's parameter list is 'void' instead of just a set of empty parentheses. ........ r53285 | raymond.hettinger | 2007-01-06 02:14:41 +0100 (Sat, 06 Jan 2007) | 2 lines SF# 1409443: Expand comment to cover the interaction between f->f_lasti and the PREDICT macros. ........ r53286 | anthony.baxter | 2007-01-06 05:45:54 +0100 (Sat, 06 Jan 2007) | 1 line update to (c) years to include 2007 ........ r53291 | neal.norwitz | 2007-01-06 22:24:35 +0100 (Sat, 06 Jan 2007) | 1 line Add Josiah to SF for maintaining asyncore/asynchat ........ r53293 | peter.astrand | 2007-01-07 09:53:46 +0100 (Sun, 07 Jan 2007) | 1 line Re-implemented fix for #1531862 once again, in a way that works with Python 2.2. Fixes bug #1603424. ........ r53295 | peter.astrand | 2007-01-07 15:34:16 +0100 (Sun, 07 Jan 2007) | 1 line Avoid O(N**2) bottleneck in _communicate_(). Fixes #1598181. ........ r53300 | raymond.hettinger | 2007-01-08 19:09:20 +0100 (Mon, 08 Jan 2007) | 1 line Fix zero-length corner case for iterating over a mutating deque. ........ r53301 | vinay.sajip | 2007-01-08 19:50:32 +0100 (Mon, 08 Jan 2007) | 4 lines Bare except clause removed from SMTPHandler.emit(). Now, only ImportError is trapped. Bare except clause removed from SocketHandler.createSocket(). Now, only socket.error is trapped. (SF #411881) ........ r53302 | vinay.sajip | 2007-01-08 19:51:46 +0100 (Mon, 08 Jan 2007) | 2 lines Bare except clause removed from LogRecord.__init__. Now, only ValueError, TypeError and AttributeError are trapped. (SF #411881) ........ r53303 | vinay.sajip | 2007-01-08 19:52:36 +0100 (Mon, 08 Jan 2007) | 1 line Added entries about removal of some bare except clauses from logging. ........
2007-01-09 19:18:33 -04:00
to the same key. There are similar use cases with a __contains__
followed by a get, set, or del to the same key. Part of the
justification for d.setdefault is combining the two lookups into one.
Membership Testing
Dictionaries of any size. Created once and then rarely changes.
Single write to each key.
Many calls to __contains__() or has_key().
Similar access patterns occur with replacement dictionaries
such as with the % formatting operator.
Dynamic Mappings
Characterized by deletions interspersed with adds and replacements.
Performance benefits greatly from the re-use of dummy entries.
Data Layout
-----------
Dictionaries are composed of 3 components:
The dictobject struct itself
A dict-keys object (keys & hashes)
A values array
Tunable Dictionary Parameters
-----------------------------
* PyDict_STARTSIZE. Starting size of dict (unless an instance dict).
Currently set to 8. Must be a power of two.
New dicts have to zero-out every cell.
Increasing improves the sparseness of small dictionaries but costs
time to read in the additional cache lines if they are not already
in cache. That case is common when keyword arguments are passed.
Prior to version 3.3, PyDict_MINSIZE was used as the starting size
of a new dict.
* PyDict_MINSIZE. Minimum size of a dict.
Currently set to 4 (to keep instance dicts small).
Must be a power of two. Prior to version 3.3, PyDict_MINSIZE was
set to 8.
* USABLE_FRACTION. Maximum dictionary load in PyDict_SetItem.
Currently set to 2/3. Increasing this ratio makes dictionaries more
dense resulting in more collisions. Decreasing it improves sparseness
at the expense of spreading entries over more cache lines and at the
cost of total memory consumed.
* Growth rate upon hitting maximum load. Currently set to *2.
Raising this to *4 results in half the number of resizes, less
effort to resize, better sparseness for some (but not all dict sizes),
and potentially doubles memory consumption depending on the size of
the dictionary. Setting to *4 eliminates every other resize step.
Merged revisions 53623-53858 via svnmerge from svn+ssh://pythondev@svn.python.org/python/trunk ........ r53624 | peter.astrand | 2007-02-02 20:06:36 +0100 (Fri, 02 Feb 2007) | 1 line We had several if statements checking the value of a fd. This is unsafe, since valid fds might be zero. We should check for not None instead. ........ r53635 | kurt.kaiser | 2007-02-05 07:03:18 +0100 (Mon, 05 Feb 2007) | 2 lines Add 'raw' support to configHandler. Patch 1650174 Tal Einat. ........ r53641 | kurt.kaiser | 2007-02-06 00:02:16 +0100 (Tue, 06 Feb 2007) | 5 lines 1. Calltips now 'handle' tuples in the argument list (display '<tuple>' :) Suggested solution by Christos Georgiou, Bug 791968. 2. Clean up tests, were not failing when they should have been. 4. Remove some camelcase and an unneeded try/except block. ........ r53644 | kurt.kaiser | 2007-02-06 04:21:40 +0100 (Tue, 06 Feb 2007) | 2 lines Clean up ModifiedInterpreter.runcode() structure ........ r53646 | peter.astrand | 2007-02-06 16:37:50 +0100 (Tue, 06 Feb 2007) | 1 line Applied patch 1124861.3.patch to solve bug #1124861: Automatically create pipes on Windows, if GetStdHandle fails. Will backport. ........ r53648 | lars.gustaebel | 2007-02-06 19:38:13 +0100 (Tue, 06 Feb 2007) | 4 lines Patch #1652681: create nonexistent files in append mode and allow appending to empty files. ........ r53649 | kurt.kaiser | 2007-02-06 20:09:43 +0100 (Tue, 06 Feb 2007) | 4 lines Updated patch (CodeContext.061217.patch) to [ 1362975 ] CodeContext - Improved text indentation Tal Einat 16Dec06 ........ r53650 | kurt.kaiser | 2007-02-06 20:21:19 +0100 (Tue, 06 Feb 2007) | 2 lines narrow exception per [ 1540849 ] except too broad ........ r53653 | kurt.kaiser | 2007-02-07 04:39:41 +0100 (Wed, 07 Feb 2007) | 4 lines [ 1621265 ] Auto-completion list placement Move AC window below input line unless not enough space, then put it above. Patch: Tal Einat ........ r53654 | kurt.kaiser | 2007-02-07 09:07:13 +0100 (Wed, 07 Feb 2007) | 2 lines Handle AttributeError during calltip lookup ........ r53656 | raymond.hettinger | 2007-02-07 21:08:22 +0100 (Wed, 07 Feb 2007) | 3 lines SF #1615701: make d.update(m) honor __getitem__() and keys() in dict subclasses ........ r53658 | raymond.hettinger | 2007-02-07 22:04:20 +0100 (Wed, 07 Feb 2007) | 1 line SF: 1397711 Set docs conflated immutable and hashable ........ r53660 | raymond.hettinger | 2007-02-07 22:42:17 +0100 (Wed, 07 Feb 2007) | 1 line Check for a common user error with defaultdict(). ........ r53662 | raymond.hettinger | 2007-02-07 23:24:07 +0100 (Wed, 07 Feb 2007) | 1 line Bug #1575169: operator.isSequenceType() now returns False for subclasses of dict. ........ r53664 | raymond.hettinger | 2007-02-08 00:49:03 +0100 (Thu, 08 Feb 2007) | 1 line Silence compiler warning ........ r53666 | raymond.hettinger | 2007-02-08 01:07:32 +0100 (Thu, 08 Feb 2007) | 1 line Do not let overflows in enumerate() and count() pass silently. ........ r53668 | raymond.hettinger | 2007-02-08 01:50:39 +0100 (Thu, 08 Feb 2007) | 1 line Bypass set specific optimizations for set and frozenset subclasses. ........ r53670 | raymond.hettinger | 2007-02-08 02:42:35 +0100 (Thu, 08 Feb 2007) | 1 line Fix docstring bug ........ r53671 | martin.v.loewis | 2007-02-08 10:13:36 +0100 (Thu, 08 Feb 2007) | 3 lines Bug #1653736: Complain about keyword arguments to time.isoformat. Will backport to 2.5. ........ r53679 | kurt.kaiser | 2007-02-08 23:58:18 +0100 (Thu, 08 Feb 2007) | 6 lines Corrected some bugs in AutoComplete. Also, Page Up/Down in ACW implemented; mouse and cursor selection in ACWindow implemented; double Tab inserts current selection and closes ACW (similar to double-click and Return); scroll wheel now works in ACW. Added AutoComplete instructions to IDLE Help. ........ r53689 | martin.v.loewis | 2007-02-09 13:19:32 +0100 (Fri, 09 Feb 2007) | 3 lines Bug #1653736: Properly discard third argument to slot_nb_inplace_power. Will backport. ........ r53691 | martin.v.loewis | 2007-02-09 13:36:48 +0100 (Fri, 09 Feb 2007) | 4 lines Bug #1600860: Search for shared python library in LIBDIR, not lib/python/config, on "linux" and "gnu" systems. Will backport. ........ r53693 | martin.v.loewis | 2007-02-09 13:58:49 +0100 (Fri, 09 Feb 2007) | 2 lines Update broken link. Will backport to 2.5. ........ r53697 | georg.brandl | 2007-02-09 19:48:41 +0100 (Fri, 09 Feb 2007) | 2 lines Bug #1656078: typo in in profile docs. ........ r53731 | brett.cannon | 2007-02-11 06:36:00 +0100 (Sun, 11 Feb 2007) | 3 lines Change a very minor inconsistency (that is purely cosmetic) in the AST definition. ........ r53735 | skip.montanaro | 2007-02-11 19:24:37 +0100 (Sun, 11 Feb 2007) | 1 line fix trace.py --ignore-dir ........ r53741 | brett.cannon | 2007-02-11 20:44:41 +0100 (Sun, 11 Feb 2007) | 3 lines Check in changed Python-ast.c from a cosmetic change to Python.asdl (in r53731). ........ r53751 | brett.cannon | 2007-02-12 04:51:02 +0100 (Mon, 12 Feb 2007) | 5 lines Modify Parser/asdl_c.py so that the __version__ number for Python/Python-ast.c is specified at the top of the file. Also add a note that Python/Python-ast.c needs to be committed separately after a change to the AST grammar to capture the revision number of the change (which is what __version__ is set to). ........ r53752 | lars.gustaebel | 2007-02-12 10:25:53 +0100 (Mon, 12 Feb 2007) | 3 lines Bug #1656581: Point out that external file objects are supposed to be at position 0. ........ r53754 | martin.v.loewis | 2007-02-12 13:21:10 +0100 (Mon, 12 Feb 2007) | 3 lines Patch 1463026: Support default namespace in XMLGenerator. Fixes #847665. Will backport. ........ r53757 | armin.rigo | 2007-02-12 17:23:24 +0100 (Mon, 12 Feb 2007) | 4 lines Fix the line to what is my guess at the original author's meaning. (The line has no effect anyway, but is present because it's customary call the base class __init__). ........ r53763 | martin.v.loewis | 2007-02-13 09:34:45 +0100 (Tue, 13 Feb 2007) | 3 lines Patch #685268: Consider a package's __path__ in imputil. Will backport. ........ r53765 | martin.v.loewis | 2007-02-13 10:49:38 +0100 (Tue, 13 Feb 2007) | 2 lines Patch #698833: Support file decryption in zipfile. ........ r53766 | martin.v.loewis | 2007-02-13 11:10:39 +0100 (Tue, 13 Feb 2007) | 3 lines Patch #1517891: Make 'a' create the file if it doesn't exist. Fixes #1514451. ........ r53767 | martin.v.loewis | 2007-02-13 13:08:24 +0100 (Tue, 13 Feb 2007) | 3 lines Bug #1658794: Remove extraneous 'this'. Will backport to 2.5. ........ r53769 | martin.v.loewis | 2007-02-13 13:14:19 +0100 (Tue, 13 Feb 2007) | 3 lines Patch #1657276: Make NETLINK_DNRTMSG conditional. Will backport. ........ r53771 | lars.gustaebel | 2007-02-13 17:09:24 +0100 (Tue, 13 Feb 2007) | 4 lines Patch #1647484: Renamed GzipFile's filename attribute to name. The filename attribute is still accessible as a property that emits a DeprecationWarning. ........ r53772 | lars.gustaebel | 2007-02-13 17:24:00 +0100 (Tue, 13 Feb 2007) | 3 lines Strip the '.gz' extension from the filename that is written to the gzip header. ........ r53774 | martin.v.loewis | 2007-02-14 11:07:37 +0100 (Wed, 14 Feb 2007) | 2 lines Patch #1432399: Add HCI sockets. ........ r53775 | martin.v.loewis | 2007-02-14 12:30:07 +0100 (Wed, 14 Feb 2007) | 2 lines Update 1432399 to removal of _BT_SOCKADDR_MEMB. ........ r53776 | martin.v.loewis | 2007-02-14 12:30:56 +0100 (Wed, 14 Feb 2007) | 3 lines Ignore directory time stamps when considering whether to rerun libffi configure. ........ r53778 | lars.gustaebel | 2007-02-14 15:45:12 +0100 (Wed, 14 Feb 2007) | 4 lines A missing binary mode in AppendTest caused failures in Windows Buildbot. ........ r53782 | martin.v.loewis | 2007-02-15 10:51:35 +0100 (Thu, 15 Feb 2007) | 2 lines Patch #1397848: add the reasoning behind no-resize-on-shrinkage. ........ r53783 | georg.brandl | 2007-02-15 11:37:59 +0100 (Thu, 15 Feb 2007) | 2 lines Make functools.wraps() docs a bit clearer. ........ r53785 | georg.brandl | 2007-02-15 12:29:04 +0100 (Thu, 15 Feb 2007) | 2 lines Patch #1494140: Add documentation for the new struct.Struct object. ........ r53787 | georg.brandl | 2007-02-15 12:29:55 +0100 (Thu, 15 Feb 2007) | 2 lines Add missing \versionadded. ........ r53800 | brett.cannon | 2007-02-15 23:54:39 +0100 (Thu, 15 Feb 2007) | 11 lines Update the encoding package's search function to use absolute imports when calling __import__. This helps make the expected search locations for encoding modules be more explicit. One could use an explicit value for __path__ when making the call to __import__ to force the exact location searched for encodings. This would give the most strict search path possible if one is worried about malicious code being imported. The unfortunate side-effect of that is that if __path__ was modified on 'encodings' on purpose in a safe way it would not be picked up in future __import__ calls. ........ r53801 | brett.cannon | 2007-02-16 20:33:01 +0100 (Fri, 16 Feb 2007) | 2 lines Make the __import__ call in encodings.__init__ absolute with a level 0 call. ........ r53809 | vinay.sajip | 2007-02-16 23:36:24 +0100 (Fri, 16 Feb 2007) | 1 line Minor fix for currentframe (SF #1652788). ........ r53818 | raymond.hettinger | 2007-02-19 03:03:19 +0100 (Mon, 19 Feb 2007) | 3 lines Extend work on revision 52962: Eliminate redundant calls to PyObject_Hash(). ........ r53820 | raymond.hettinger | 2007-02-19 05:08:43 +0100 (Mon, 19 Feb 2007) | 1 line Add merge() function to heapq. ........ r53821 | raymond.hettinger | 2007-02-19 06:28:28 +0100 (Mon, 19 Feb 2007) | 1 line Add tie-breaker count to preserve sort stability. ........ r53822 | raymond.hettinger | 2007-02-19 07:59:32 +0100 (Mon, 19 Feb 2007) | 1 line Use C heapreplace() instead of slower _siftup() in pure python. ........ r53823 | raymond.hettinger | 2007-02-19 08:30:21 +0100 (Mon, 19 Feb 2007) | 1 line Add test for merge stability ........ r53824 | raymond.hettinger | 2007-02-19 10:14:10 +0100 (Mon, 19 Feb 2007) | 1 line Provide an example of defaultdict with non-zero constant factory function. ........ r53825 | lars.gustaebel | 2007-02-19 10:54:47 +0100 (Mon, 19 Feb 2007) | 2 lines Moved misplaced news item. ........ r53826 | martin.v.loewis | 2007-02-19 11:55:19 +0100 (Mon, 19 Feb 2007) | 3 lines Patch #1490190: posixmodule now includes os.chflags() and os.lchflags() functions on platforms where the underlying system calls are available. ........ r53827 | raymond.hettinger | 2007-02-19 19:15:04 +0100 (Mon, 19 Feb 2007) | 1 line Fixup docstrings for merge(). ........ r53829 | raymond.hettinger | 2007-02-19 21:44:04 +0100 (Mon, 19 Feb 2007) | 1 line Fixup set/dict interoperability. ........ r53837 | raymond.hettinger | 2007-02-21 06:20:38 +0100 (Wed, 21 Feb 2007) | 1 line Add itertools.izip_longest(). ........ r53838 | raymond.hettinger | 2007-02-21 18:22:05 +0100 (Wed, 21 Feb 2007) | 1 line Remove filler struct item and fix leak. ........
2007-02-23 11:07:44 -04:00
* Maximum sparseness (minimum dictionary load). What percentage
of entries can be unused before the dictionary shrinks to
free up memory and speed up iteration? (The current CPython
code does not represent this parameter directly.)
* Shrinkage rate upon exceeding maximum sparseness. The current
CPython code never even checks sparseness when deleting a
key. When a new key is added, it resizes based on the number
of active keys, so that the addition may trigger shrinkage
rather than growth.
Tune-ups should be measured across a broad range of applications and
use cases. A change to any parameter will help in some situations and
hurt in others. The key is to find settings that help the most common
cases and do the least damage to the less common cases. Results will
vary dramatically depending on the exact number of keys, whether the
keys are all strings, whether reads or writes dominate, the exact
hash values of the keys (some sets of values have fewer collisions than
others). Any one test or benchmark is likely to prove misleading.
While making a dictionary more sparse reduces collisions, it impairs
iteration and key listing. Those methods loop over every potential
entry. Doubling the size of dictionary results in twice as many
non-overlapping memory accesses for keys(), items(), values(),
__iter__(), iterkeys(), iteritems(), itervalues(), and update().
2004-03-15 11:52:22 -04:00
Also, every dictionary iterates at least twice, once for the memset()
when it is created and once by dealloc().
Dictionary operations involving only a single key can be O(1) unless
resizing is possible. By checking for a resize only when the
Merged revisions 53623-53858 via svnmerge from svn+ssh://pythondev@svn.python.org/python/trunk ........ r53624 | peter.astrand | 2007-02-02 20:06:36 +0100 (Fri, 02 Feb 2007) | 1 line We had several if statements checking the value of a fd. This is unsafe, since valid fds might be zero. We should check for not None instead. ........ r53635 | kurt.kaiser | 2007-02-05 07:03:18 +0100 (Mon, 05 Feb 2007) | 2 lines Add 'raw' support to configHandler. Patch 1650174 Tal Einat. ........ r53641 | kurt.kaiser | 2007-02-06 00:02:16 +0100 (Tue, 06 Feb 2007) | 5 lines 1. Calltips now 'handle' tuples in the argument list (display '<tuple>' :) Suggested solution by Christos Georgiou, Bug 791968. 2. Clean up tests, were not failing when they should have been. 4. Remove some camelcase and an unneeded try/except block. ........ r53644 | kurt.kaiser | 2007-02-06 04:21:40 +0100 (Tue, 06 Feb 2007) | 2 lines Clean up ModifiedInterpreter.runcode() structure ........ r53646 | peter.astrand | 2007-02-06 16:37:50 +0100 (Tue, 06 Feb 2007) | 1 line Applied patch 1124861.3.patch to solve bug #1124861: Automatically create pipes on Windows, if GetStdHandle fails. Will backport. ........ r53648 | lars.gustaebel | 2007-02-06 19:38:13 +0100 (Tue, 06 Feb 2007) | 4 lines Patch #1652681: create nonexistent files in append mode and allow appending to empty files. ........ r53649 | kurt.kaiser | 2007-02-06 20:09:43 +0100 (Tue, 06 Feb 2007) | 4 lines Updated patch (CodeContext.061217.patch) to [ 1362975 ] CodeContext - Improved text indentation Tal Einat 16Dec06 ........ r53650 | kurt.kaiser | 2007-02-06 20:21:19 +0100 (Tue, 06 Feb 2007) | 2 lines narrow exception per [ 1540849 ] except too broad ........ r53653 | kurt.kaiser | 2007-02-07 04:39:41 +0100 (Wed, 07 Feb 2007) | 4 lines [ 1621265 ] Auto-completion list placement Move AC window below input line unless not enough space, then put it above. Patch: Tal Einat ........ r53654 | kurt.kaiser | 2007-02-07 09:07:13 +0100 (Wed, 07 Feb 2007) | 2 lines Handle AttributeError during calltip lookup ........ r53656 | raymond.hettinger | 2007-02-07 21:08:22 +0100 (Wed, 07 Feb 2007) | 3 lines SF #1615701: make d.update(m) honor __getitem__() and keys() in dict subclasses ........ r53658 | raymond.hettinger | 2007-02-07 22:04:20 +0100 (Wed, 07 Feb 2007) | 1 line SF: 1397711 Set docs conflated immutable and hashable ........ r53660 | raymond.hettinger | 2007-02-07 22:42:17 +0100 (Wed, 07 Feb 2007) | 1 line Check for a common user error with defaultdict(). ........ r53662 | raymond.hettinger | 2007-02-07 23:24:07 +0100 (Wed, 07 Feb 2007) | 1 line Bug #1575169: operator.isSequenceType() now returns False for subclasses of dict. ........ r53664 | raymond.hettinger | 2007-02-08 00:49:03 +0100 (Thu, 08 Feb 2007) | 1 line Silence compiler warning ........ r53666 | raymond.hettinger | 2007-02-08 01:07:32 +0100 (Thu, 08 Feb 2007) | 1 line Do not let overflows in enumerate() and count() pass silently. ........ r53668 | raymond.hettinger | 2007-02-08 01:50:39 +0100 (Thu, 08 Feb 2007) | 1 line Bypass set specific optimizations for set and frozenset subclasses. ........ r53670 | raymond.hettinger | 2007-02-08 02:42:35 +0100 (Thu, 08 Feb 2007) | 1 line Fix docstring bug ........ r53671 | martin.v.loewis | 2007-02-08 10:13:36 +0100 (Thu, 08 Feb 2007) | 3 lines Bug #1653736: Complain about keyword arguments to time.isoformat. Will backport to 2.5. ........ r53679 | kurt.kaiser | 2007-02-08 23:58:18 +0100 (Thu, 08 Feb 2007) | 6 lines Corrected some bugs in AutoComplete. Also, Page Up/Down in ACW implemented; mouse and cursor selection in ACWindow implemented; double Tab inserts current selection and closes ACW (similar to double-click and Return); scroll wheel now works in ACW. Added AutoComplete instructions to IDLE Help. ........ r53689 | martin.v.loewis | 2007-02-09 13:19:32 +0100 (Fri, 09 Feb 2007) | 3 lines Bug #1653736: Properly discard third argument to slot_nb_inplace_power. Will backport. ........ r53691 | martin.v.loewis | 2007-02-09 13:36:48 +0100 (Fri, 09 Feb 2007) | 4 lines Bug #1600860: Search for shared python library in LIBDIR, not lib/python/config, on "linux" and "gnu" systems. Will backport. ........ r53693 | martin.v.loewis | 2007-02-09 13:58:49 +0100 (Fri, 09 Feb 2007) | 2 lines Update broken link. Will backport to 2.5. ........ r53697 | georg.brandl | 2007-02-09 19:48:41 +0100 (Fri, 09 Feb 2007) | 2 lines Bug #1656078: typo in in profile docs. ........ r53731 | brett.cannon | 2007-02-11 06:36:00 +0100 (Sun, 11 Feb 2007) | 3 lines Change a very minor inconsistency (that is purely cosmetic) in the AST definition. ........ r53735 | skip.montanaro | 2007-02-11 19:24:37 +0100 (Sun, 11 Feb 2007) | 1 line fix trace.py --ignore-dir ........ r53741 | brett.cannon | 2007-02-11 20:44:41 +0100 (Sun, 11 Feb 2007) | 3 lines Check in changed Python-ast.c from a cosmetic change to Python.asdl (in r53731). ........ r53751 | brett.cannon | 2007-02-12 04:51:02 +0100 (Mon, 12 Feb 2007) | 5 lines Modify Parser/asdl_c.py so that the __version__ number for Python/Python-ast.c is specified at the top of the file. Also add a note that Python/Python-ast.c needs to be committed separately after a change to the AST grammar to capture the revision number of the change (which is what __version__ is set to). ........ r53752 | lars.gustaebel | 2007-02-12 10:25:53 +0100 (Mon, 12 Feb 2007) | 3 lines Bug #1656581: Point out that external file objects are supposed to be at position 0. ........ r53754 | martin.v.loewis | 2007-02-12 13:21:10 +0100 (Mon, 12 Feb 2007) | 3 lines Patch 1463026: Support default namespace in XMLGenerator. Fixes #847665. Will backport. ........ r53757 | armin.rigo | 2007-02-12 17:23:24 +0100 (Mon, 12 Feb 2007) | 4 lines Fix the line to what is my guess at the original author's meaning. (The line has no effect anyway, but is present because it's customary call the base class __init__). ........ r53763 | martin.v.loewis | 2007-02-13 09:34:45 +0100 (Tue, 13 Feb 2007) | 3 lines Patch #685268: Consider a package's __path__ in imputil. Will backport. ........ r53765 | martin.v.loewis | 2007-02-13 10:49:38 +0100 (Tue, 13 Feb 2007) | 2 lines Patch #698833: Support file decryption in zipfile. ........ r53766 | martin.v.loewis | 2007-02-13 11:10:39 +0100 (Tue, 13 Feb 2007) | 3 lines Patch #1517891: Make 'a' create the file if it doesn't exist. Fixes #1514451. ........ r53767 | martin.v.loewis | 2007-02-13 13:08:24 +0100 (Tue, 13 Feb 2007) | 3 lines Bug #1658794: Remove extraneous 'this'. Will backport to 2.5. ........ r53769 | martin.v.loewis | 2007-02-13 13:14:19 +0100 (Tue, 13 Feb 2007) | 3 lines Patch #1657276: Make NETLINK_DNRTMSG conditional. Will backport. ........ r53771 | lars.gustaebel | 2007-02-13 17:09:24 +0100 (Tue, 13 Feb 2007) | 4 lines Patch #1647484: Renamed GzipFile's filename attribute to name. The filename attribute is still accessible as a property that emits a DeprecationWarning. ........ r53772 | lars.gustaebel | 2007-02-13 17:24:00 +0100 (Tue, 13 Feb 2007) | 3 lines Strip the '.gz' extension from the filename that is written to the gzip header. ........ r53774 | martin.v.loewis | 2007-02-14 11:07:37 +0100 (Wed, 14 Feb 2007) | 2 lines Patch #1432399: Add HCI sockets. ........ r53775 | martin.v.loewis | 2007-02-14 12:30:07 +0100 (Wed, 14 Feb 2007) | 2 lines Update 1432399 to removal of _BT_SOCKADDR_MEMB. ........ r53776 | martin.v.loewis | 2007-02-14 12:30:56 +0100 (Wed, 14 Feb 2007) | 3 lines Ignore directory time stamps when considering whether to rerun libffi configure. ........ r53778 | lars.gustaebel | 2007-02-14 15:45:12 +0100 (Wed, 14 Feb 2007) | 4 lines A missing binary mode in AppendTest caused failures in Windows Buildbot. ........ r53782 | martin.v.loewis | 2007-02-15 10:51:35 +0100 (Thu, 15 Feb 2007) | 2 lines Patch #1397848: add the reasoning behind no-resize-on-shrinkage. ........ r53783 | georg.brandl | 2007-02-15 11:37:59 +0100 (Thu, 15 Feb 2007) | 2 lines Make functools.wraps() docs a bit clearer. ........ r53785 | georg.brandl | 2007-02-15 12:29:04 +0100 (Thu, 15 Feb 2007) | 2 lines Patch #1494140: Add documentation for the new struct.Struct object. ........ r53787 | georg.brandl | 2007-02-15 12:29:55 +0100 (Thu, 15 Feb 2007) | 2 lines Add missing \versionadded. ........ r53800 | brett.cannon | 2007-02-15 23:54:39 +0100 (Thu, 15 Feb 2007) | 11 lines Update the encoding package's search function to use absolute imports when calling __import__. This helps make the expected search locations for encoding modules be more explicit. One could use an explicit value for __path__ when making the call to __import__ to force the exact location searched for encodings. This would give the most strict search path possible if one is worried about malicious code being imported. The unfortunate side-effect of that is that if __path__ was modified on 'encodings' on purpose in a safe way it would not be picked up in future __import__ calls. ........ r53801 | brett.cannon | 2007-02-16 20:33:01 +0100 (Fri, 16 Feb 2007) | 2 lines Make the __import__ call in encodings.__init__ absolute with a level 0 call. ........ r53809 | vinay.sajip | 2007-02-16 23:36:24 +0100 (Fri, 16 Feb 2007) | 1 line Minor fix for currentframe (SF #1652788). ........ r53818 | raymond.hettinger | 2007-02-19 03:03:19 +0100 (Mon, 19 Feb 2007) | 3 lines Extend work on revision 52962: Eliminate redundant calls to PyObject_Hash(). ........ r53820 | raymond.hettinger | 2007-02-19 05:08:43 +0100 (Mon, 19 Feb 2007) | 1 line Add merge() function to heapq. ........ r53821 | raymond.hettinger | 2007-02-19 06:28:28 +0100 (Mon, 19 Feb 2007) | 1 line Add tie-breaker count to preserve sort stability. ........ r53822 | raymond.hettinger | 2007-02-19 07:59:32 +0100 (Mon, 19 Feb 2007) | 1 line Use C heapreplace() instead of slower _siftup() in pure python. ........ r53823 | raymond.hettinger | 2007-02-19 08:30:21 +0100 (Mon, 19 Feb 2007) | 1 line Add test for merge stability ........ r53824 | raymond.hettinger | 2007-02-19 10:14:10 +0100 (Mon, 19 Feb 2007) | 1 line Provide an example of defaultdict with non-zero constant factory function. ........ r53825 | lars.gustaebel | 2007-02-19 10:54:47 +0100 (Mon, 19 Feb 2007) | 2 lines Moved misplaced news item. ........ r53826 | martin.v.loewis | 2007-02-19 11:55:19 +0100 (Mon, 19 Feb 2007) | 3 lines Patch #1490190: posixmodule now includes os.chflags() and os.lchflags() functions on platforms where the underlying system calls are available. ........ r53827 | raymond.hettinger | 2007-02-19 19:15:04 +0100 (Mon, 19 Feb 2007) | 1 line Fixup docstrings for merge(). ........ r53829 | raymond.hettinger | 2007-02-19 21:44:04 +0100 (Mon, 19 Feb 2007) | 1 line Fixup set/dict interoperability. ........ r53837 | raymond.hettinger | 2007-02-21 06:20:38 +0100 (Wed, 21 Feb 2007) | 1 line Add itertools.izip_longest(). ........ r53838 | raymond.hettinger | 2007-02-21 18:22:05 +0100 (Wed, 21 Feb 2007) | 1 line Remove filler struct item and fix leak. ........
2007-02-23 11:07:44 -04:00
dictionary can grow (and may *require* resizing), other operations
remain O(1), and the odds of resize thrashing or memory fragmentation
are reduced. In particular, an algorithm that empties a dictionary
by repeatedly invoking .pop will see no resizing, which might
not be necessary at all because the dictionary is eventually
discarded entirely.
The key differences between this implementation and earlier versions are:
1. The table can be split into two parts, the keys and the values.
2. There is an additional key-value combination: (key, NULL).
Unlike (<dummy>, NULL) which represents a deleted value, (key, NULL)
represented a yet to be inserted value. This combination can only occur
when the table is split.
3. No small table embedded in the dict,
as this would make sharing of key-tables impossible.
These changes have the following consequences.
1. General dictionaries are slightly larger.
2. All object dictionaries of a single class can share a single key-table,
saving about 60% memory for such cases.
Results of Cache Locality Experiments
--------------------------------------
Experiments on an earlier design of dictionary, in which all tables were
combined, showed the following:
When an entry is retrieved from memory, several adjacent entries are also
retrieved into a cache line. Since accessing items in cache is *much*
cheaper than a cache miss, an enticing idea is to probe the adjacent
entries as a first step in collision resolution. Unfortunately, the
introduction of any regularity into collision searches results in more
collisions than the current random chaining approach.
Exploiting cache locality at the expense of additional collisions fails
to payoff when the entries are already loaded in cache (the expense
is paid with no compensating benefit). This occurs in small dictionaries
where the whole dictionary fits into a pair of cache lines. It also
occurs frequently in large dictionaries which have a common access pattern
where some keys are accessed much more frequently than others. The
more popular entries *and* their collision chains tend to remain in cache.
To exploit cache locality, change the collision resolution section
in lookdict() and lookdict_string(). Set i^=1 at the top of the
loop and move the i = (i << 2) + i + perturb + 1 to an unrolled
version of the loop.
For split tables, the above will apply to the keys, but the value will
always be in a different cache line from the key.