Commit Graph

168 Commits

Author SHA1 Message Date
Guido van Rossum b26a97aa50 Get rid of __safe_for_unpickling__ and safe_constructors.
Also tidied up a few lines, got rid of apply(), added a comment.
2003-01-28 22:29:13 +00:00
Guido van Rossum ac5b5d2e8b Instead of bad hacks trying to worm around the inherited
object.__reduce__, do a getattr() on the class so we can explicitly
test for it.  The reduce()-calling code becomes a bit more regular as
a result.

Also add support slots: if an object has slots, the default state is
(dict, slots) where dict is the __dict__ or None, and slots is a dict
mapping slot names to slot values.  We do a best-effort approach to
find slot names, assuming the __slots__ fields of classes aren't
modified after class definition time to misrepresent the actual list
of slots defined by a class.
2003-01-28 22:01:16 +00:00
Guido van Rossum 3d8c01b31c The default __reduce__ on the base object type obscured any
possibility of calling save_reduce().  Add a special hack for this.
The tests for this are much simpler now (no __getstate__ or
__getnewargs__ needed).
2003-01-28 19:48:18 +00:00
Guido van Rossum 54fb192508 Move the NEWOBJ-generating code to a separate function, and invoke it
after checking for __reduce__.
2003-01-28 18:22:35 +00:00
Guido van Rossum 533dbcf250 Some experimental support for generating NEWOBJ with proto=2, and
fixed a bug in load_newobj().
2003-01-28 17:55:05 +00:00
Tim Peters a6ae9a2128 save_empty_tuple(): Comment on why we can't get rid of this. 2003-01-28 16:58:41 +00:00
Tim Peters 82ca59e002 save_dict(): Added a comment about the control flow NealN missed. 2003-01-28 16:47:59 +00:00
Tim Peters 13a25fb8e6 _is_string_secure(): This method is no longer used; removed it. (It
was used before string-escape codecs were added to the core.)
2003-01-28 16:42:22 +00:00
Guido van Rossum bc64e22ed6 Made save() fit on a page, while adding comments. (I moved some type
checks to save_reduce(), which can also be called from a subclass.)

Also tweaked some more comments.
2003-01-28 16:34:19 +00:00
Tim Peters ad5a771fae Got rid of the _quotes global. Used only once, and is trivial. 2003-01-28 16:23:33 +00:00
Guido van Rossum 1be3175992 Add a few comments. Change the way the protocol is checked (it must
be one of 0, 1 or 2).

I should note that the previous checkin also added NEWOBJ support to
the unpickler -- but there's nothing yet that generates this.
2003-01-28 15:19:53 +00:00
Guido van Rossum 3a41c61dd4 Rename all variables 'object' to 'obj' to avoid conflicts with the
type 'object'.  Also minor docstring tweakage, and rearranged a few
lines in save().
2003-01-28 15:10:22 +00:00
Guido van Rossum cbe2dbddda Don't memoize the empty tuple in protocol 0. 2003-01-28 14:40:16 +00:00
Tim Peters d97da80dd5 save_tuple(): So long as the charter is rewriting for clarity, the snaky
control flow had to be simplified.
2003-01-28 05:48:29 +00:00
Tim Peters ff57bff16e save_tuple(): I believe the new code for TUPLE{1,2,3} in proto 2 was
incorrect for recursive tuples.  Tried to repair; seems to work OK, but
there are no checked-in tests for this yet.
2003-01-28 05:34:53 +00:00
Guido van Rossum 7d97d31a1b OK, this is really the last one tonight!
NEWFALSE and NEWTRUE.
2003-01-28 04:25:27 +00:00
Guido van Rossum 44f0ea5f73 More protocol 2: TUPLE1, TUPLE2, TUPLE3.
Also moved the special case for empty tuples from save() to save_tuple().
2003-01-28 04:14:51 +00:00
Tim Peters 3b769835ca save_inst(): Rewrote to have only one branch on self.bin. Also got rid
of my recent XXX comment, taking a (what appears to be vanishingly small)
chance and calling self.memoize() instead.
2003-01-28 03:51:36 +00:00
Guido van Rossum d6c9e63af9 First baby steps towards implementing protocol 2: PROTO, LONG1 and LONG4. 2003-01-28 03:49:52 +00:00
Tim Peters d95c2df3a9 Fixed odd whitespace after "if", which I believe I introduced long ago. 2003-01-28 03:41:54 +00:00
Tim Peters 8fda7bc48d save_int(): Fixed two new off-by-1 glitches. 2003-01-28 03:40:52 +00:00
Guido van Rossum e0b904232f Add a comment explaining that struct.pack() beats marshal.dumps(), but
marshal.loads() beats struct.unpack()!  Possibly because the latter
creates a one-tuple. :-(
2003-01-28 03:17:21 +00:00
Guido van Rossum 5c938d00a1 Got rid of mdumps; I timed it, and struct.pack("<i", x) is more than
40% faster than marshal.dumps(x)[1:]!  (That's not counting the
module attribute lookups, which can be avoided in either case.)
2003-01-28 03:03:08 +00:00
Tim Peters f558da0f90 save_tuple(): Minor rewriting, and added a comment about the subtlety
created by recursive tuples.
2003-01-28 02:09:55 +00:00
Tim Peters 209ad95b00 load_appends(): replaced .append() loop with an .extend(). 2003-01-28 01:44:45 +00:00
Tim Peters c23d18a955 Comments. 2003-01-28 01:41:51 +00:00
Tim Peters 064567e41a save_dict(): Untangled most of the bin-vs-not-bin logic. Also used
iteritems() instead of materializing a (possibly giant) list of the
items.
2003-01-28 01:34:43 +00:00
Tim Peters 21c18f0bf5 save_list(): Rewrote, to untangle the proto 0 from the proto 1 cases.
The code is much easier to follow now, and I bet it's faster too.
2003-01-28 01:15:46 +00:00
Tim Peters 22dc6f4f7a save_list(): removed unused local "d". 2003-01-28 01:07:48 +00:00
Tim Peters 58d7846e68 save_list(): removed unused local "memo". 2003-01-28 01:07:10 +00:00
Tim Peters bd1cdb9227 save_pers(): Switched the order of cases, to get rid of a "not", and to
make the bin-vs-not-bin order consistent with what other routines try to
do (they almost all handle the bin case first).
2003-01-28 01:03:10 +00:00
Tim Peters 518df0dae4 Several routines appeared to inline the guts of memoize(), possibly for
some notion of low-level efficiency.  Undid that, but left one routine
alone:  save_inst() claims it has a reason for not using memoize().
I don't understand that comment, so added an XXX comment there.
2003-01-28 01:00:38 +00:00
Tim Peters b32a8317d7 save(): Fiddled the control flow to put the normal case where it
belongs.  This is a much smaller change than it may appear:  the bulk
of the function merely got unindented by one level.
2003-01-28 00:48:09 +00:00
Tim Peters c9d7c4a656 Added XXX about save()'s special-casing of tuples -- I don't get it. 2003-01-28 00:43:26 +00:00
Tim Peters 22987e3cbd save_bool(): simplified. 2003-01-28 00:26:14 +00:00
Tim Peters cbd0a32d09 Repaired grammar in new comment. 2003-01-28 00:24:43 +00:00
Tim Peters b377f8ac3e clear_memo(): Repaired grammar in docstring. 2003-01-28 00:23:36 +00:00
Tim Peters e105478a0c Added one-line comments to the proto 2 opcodes. 2003-01-28 00:22:12 +00:00
Tim Peters fdb8cfab08 Removed the new LONG2 opcode: it's extravagant. If LONG1 isn't enough,
then the embedded argument consumes at least 256 bytes.  The difference
between a 3-byte prefix (LONG2 + 2 bytes) and a 5-byte prefix (LONG4 +
4 bytes) is at worst less than 1%.  Note that binary strings and binary
Unicode strings also have only "size is 1 byte, or size is 4 bytes?"
flavors, and I expect for the same reason.  The only place a 2-byte
thingie was used was in BININT2, where the 2 bytes make up the *entire*
embedded argument (and now EXT2 also does this); that's a large savings
over 4 bytes, because the total opcode+argument size is so small in
the BININT2/EXT2 case.

Removed the TAKEN_FROM_ARGUMENT "number of bytes" code, and bifurcated it
into TAKEN_FROM_ARGUMENT1 and TAKEN_FROM_ARGUMENT4.  Now there's enough
info in ArgumentDescriptor objects to deduce the # of bytes consumed by
each opcode.

Rearranged the order in which proto2 opcodes are listed in pickle.py.
2003-01-28 00:13:19 +00:00
Guido van Rossum f29d3d6011 Begin the change from 'binary vs. text mode' to 'protocol 0, 1, 2'.
The protocol now defaults to 1.  Protocol 2 is still unimplemented.
2003-01-27 22:47:53 +00:00
Guido van Rossum 5a2d8f5e9a Begin documenting protocol 2. 2003-01-27 21:44:25 +00:00
Tim Peters bb38e306ae Added some comments. 2003-01-27 21:25:41 +00:00
Tim Peters e46b73f5b3 memoize(): Reworded the docs to try to disentangle the Pickler's memo
dict from the Unpickler's memo (which is a different beast!).
2003-01-27 21:22:10 +00:00
Tim Peters bbf63cd05c Using marshal functions to pack & unpack 1-byte ints is an obscure &
expensive way to spell chr() and ord().
2003-01-27 21:15:36 +00:00
Tim Peters 22a449aacc Added a brief comment to each pickle opcode declaration. 2003-01-27 20:16:36 +00:00
Jeremy Hylton 3422c99de1 Raise PicklingError when __reduce__() fails, and
add memoize() helper function to update the memo.

The first element of the tuple returned by __reduce__() must be a
callable.  If it isn't the Unpickler will raise an error.  Catch this
error in the pickler and raise the error there.

The memoize() helper also has a comment explaining how the memo
works.  So methods can't use memoize() because the write funny codes.
2003-01-24 19:29:52 +00:00
Jeremy Hylton 5e0f4e73a9 Remove inst_persistent_id() WANNI (we ain't never needed it).
Add some simple tests of the persistence hooks.
2002-11-13 22:01:27 +00:00
Jeremy Hylton f0cfdf7314 Fiddle comments and variable names in whichmodule(). 2002-09-19 23:00:12 +00:00
Jeremy Hylton 065a5ab8fb whichmodule() should skip dummy package entries in sys.modules.
This fixes the charming, but unhelpful error message for
>>> pickle.dumps(type.__new__)
Can't pickle <built-in method __new__ of type object at 0x812a440>: it's not the same object as datetime.math.__new__

Bugfix candidate.
2002-09-19 22:57:26 +00:00
Martin v. Löwis 8a8da798a5 Patch #505705: Remove eval in pickle and cPickle. 2002-08-14 07:46:28 +00:00
Jeremy Hylton 39c6116483 Given the persistent id code a shot at a class before calling save_global().
Some persistent picklers (well, probably, the *only* persistent
pickler) would like to pickle some classes in a special way.
2002-07-16 19:47:43 +00:00
Raymond Hettinger 46ac8eb3c8 Code modernization. Replace v=s[i]; del s[i] with single lookup v=s.pop(i) 2002-06-30 03:39:14 +00:00
Raymond Hettinger 54f0222547 SF 563203. Replaced 'has_key()' with 'in'. 2002-06-01 14:18:47 +00:00
Neal Norwitz efbb67b1a7 Remove comment about inheritance, look one line up 2002-05-30 12:12:04 +00:00
Raymond Hettinger aef22fb9cd Patch 560023 adding docstrings. 2.2 Candidate (after verifying modules were not updated after 2.2). 2002-05-29 16:18:42 +00:00
Tim Peters 8ac1495a6a Whitespace normalization. 2002-05-23 15:15:30 +00:00
Raymond Hettinger 97394bc795 Patch 533291. Deprecate None return form of __reduce__. 2002-05-21 17:22:02 +00:00
Fred Drake 7f781c9aab Add Pickler.clear_memo() so the pickle and cPickle modules are more similar. 2002-05-01 20:33:53 +00:00
Guido van Rossum e276339cea Implement an idea by Paul Rubin:
Change pickling format for bools to use a backwards compatible
encoding.  This means you can pickle True or False on Python 2.3
and Python 2.2 or before will read it back as 1 or 0.  The code
used for pickling bools before would create pickles that could
not be read in previous Python versions.
2002-04-05 19:30:08 +00:00
Guido van Rossum 77f6a65eb0 Add the 'bool' type and its values 'False' and 'True', as described in
PEP 285.  Everything described in the PEP is here, and there is even
some documentation.  I had to fix 12 unit tests; all but one of these
were printing Boolean outcomes that changed from 0/1 to False/True.
(The exception is test_unicode.py, which did a type(x) == type(y)
style comparison.  I could've fixed that with a single line using
issubtype(x, type(y)), but instead chose to be explicit about those
places where a bool is expected.

Still to do: perhaps more documentation; change standard library
modules to return False/True from predicates.
2002-04-03 22:41:51 +00:00
Guido van Rossum 85ee491b3a Fix for SF 502085.
Don't die when issubclass(t, TypeType) fails.

Bugfix candidate (but I think it's too late for 2.2.1).
2002-03-26 00:51:56 +00:00
Neal Norwitz d5ba4aeab0 SF #515018, delete global variable that was apparently used only
in a list comprehension.
2002-02-11 18:12:06 +00:00
Guido van Rossum f048a8f6d7 Pickler.save(): Fix for SF bug #494904: Cannot pickle a class with a
metaclass, reported by Dan Parisien.

Objects that are instances of custom metaclasses, i.e. whose class is
a subclass of 'type', should be pickled the same as new-style classes
(objects whose class is 'type').  This can't be done through a
dispatch table entry, and the __reduce__ trick doesn't work for these,
since it finds the unbound __reduce__ for instances of the class
(inherited from 'object').  So check explicitly using issubclass().
2001-12-19 16:55:02 +00:00
Barry Warsaw bf4d959d28 Two changes:
load_inst(): Implement the security hook that cPickle already had.
When unpickling callables which are not classes, we look to see if the
object has an attribute __safe_for_unpickling__.  If this exists and
has a true value, then we can call it to create the unpickled object.
Otherwise we raise an UnpicklingError.

find_class(): We no longer mask ImportError, KeyError, and
AttributeError by transforming them into SystemError.  The latter is
definitely not the right thing to do, so we let the former three
exceptions simply propagate up if they occur, i.e. we remove the
try/except!
2001-11-15 23:42:58 +00:00
Jeremy Hylton 20747fa167 A better new, unique object 2001-11-09 16:15:04 +00:00
Jeremy Hylton abe2c62bdb Use cStringIO when available.
Remove test code.  It's available in Lib/test/picklertester.py.
2001-10-15 21:29:28 +00:00
Guido van Rossum dbb718fa87 Make these modules work when Python is compiled without Unicode support. 2001-09-21 19:22:34 +00:00
Tim Peters 19ef62d5a9 pickle.py, load_int(): Match cPickle's just-repaired ability to unpickle
64-bit INTs on 32-bit boxes (where they become longs).  Also exploit that
int(str) and long(str) will ignore a trailing newline (saves creating a
new string at the Python level).

pickletester.py:  Simulate reading a pickle produced by a 64-bit box.
2001-08-28 22:21:18 +00:00
Guido van Rossum b0a98e9c94 Address SF #451547. The approach is a bit draconian: any object that
is pickled as a global must now exist by the name under which it is
pickled, otherwise the pickling fails.  Previously, such things would
fail on unpickling, or unpickle as the wrong global object.  I'm
hoping that this won't break existing code that is playing tricks with
this.

I need a volunteer to do this for cPickle too.
2001-08-17 18:49:52 +00:00
Tim Peters 6d6c1a35e0 Merge of descr-branch back into trunk. 2001-08-02 04:15:00 +00:00
Tim Peters 44714007e8 test_pickle works on sizeof(long)==8 boxes again.
pickle.py
    The code implicitly assumed that all ints fit in 4 bytes, causing all
    sorts of mischief (from nonsense results to corrupted pickles).
    Repaired that.
marshal.c
    The int marshaling code assumed that right shifts of signed longs
    sign-extend.  Repaired that.
2001-04-10 05:02:52 +00:00
Tim Peters c32d824467 Mechanical fiddling for easier reading: every "if" test was enclosed in
parens, but no "while" test.  Removed the former.
2001-04-10 02:48:53 +00:00
Skip Montanaro 23bafc6fcd add module-level constants to __all__ 2001-02-18 03:10:09 +00:00
Tim Peters 658cba6706 Whitespace normalization. 2001-02-09 20:06:00 +00:00
Skip Montanaro 352674d01c a few more __all__ lists 2001-02-07 23:14:30 +00:00
Guido van Rossum 31584cb43d Finn Bock (SF patch #103349):
Allow pickle.py to be using with Jython unicode strings
2001-01-22 14:53:29 +00:00
Tim Peters 2344fae6d0 Whitespace normalization. 2001-01-15 00:50:52 +00:00
Guido van Rossum a31b4ef7c5 Minimal fix for the complaints about pickling Unicode objects. (SF
bugs #126161 and 123634).

The solution doesn't use the unicode-escape encoding; that has other
problems (it seems not 100% reversible).  Rather, it transforms the
input Unicode object slightly before encoding it using
raw-unicode-escape, so that the decoding will reconstruct the original
string: backslash and newline characters are translated into their
\uXXXX counterparts.

This is backwards incompatible for strings containing backslashes, but
for some of those strings, the pickling was already broken.
2000-12-19 01:29:00 +00:00
Guido van Rossum ff87174db1 Get rid of string module and string exceptions. 2000-12-13 18:11:56 +00:00
Jeremy Hylton be467e5c69 Fix Bug #114293:
Strings are unpickled by calling eval on the string's repr. This
    change makes pickle work like cPickle; it checks if the pickled
    string is safe to eval and raises ValueError if it is not.

test suite modifications:
    Verify that pickle catches a variety of insecure string pickles
    Make test_pickle and test_cpickle use exactly the same test suite
    Add test for pickling recursive object
2000-09-15 15:14:51 +00:00
Guido van Rossum 26e89d0185 Change string exceptions to class exceptions. 2000-06-29 16:15:52 +00:00
Guido van Rossum b5f2f1bb6f Marc-Andre Lemburg: support pickling Unicode objects, both in text
mode ('V') and in binary mode ('X').
2000-03-10 23:20:09 +00:00
Guido van Rossum 54f22ed30b More trivial comment -> docstring transformations by Ka-Ping Yee,
who writes:

Here is batch 2, as a big collection of CVS context diffs.
Along with moving comments into docstrings, i've added a
couple of missing docstrings and attempted to make sure more
module docstrings begin with a one-line summary.

I did not add docstrings to the methods in profile.py for
fear of upsetting any careful optimizations there, though
i did move class documentation into class docstrings.

The convention i'm using is to leave credits/version/copyright
type of stuff in # comments, and move the rest of the descriptive
stuff about module usage into module docstrings.  Hope this is
okay.
2000-02-04 15:10:34 +00:00
Guido van Rossum 08a92cb568 Jeremy writes:
I found the following patch helpful in tracking down a bug in some
code.  I had appended time, the module, instead of time.time().  Not
sure if it is generally true that printing the repr of the object is
good, but I expect that most unpicklable things will have fairly
information and concise reprs (like files or sockets or modules).
1999-10-10 21:14:25 +00:00
Guido van Rossum 397032aa46 Don't use "exec" in find_class(). It's slow, unnecessary, and (as AMK
points out) it doesn't work in JPython Applets.
1999-03-25 21:58:59 +00:00
Guido van Rossum d370379186 Jim Fulton writes:
"""
I've attached a long overdue patch to pickle.py to bring it to format
1.3, which is the same as 1.2 except that the binary float format
is supported.  This is done using the new platform-indepent format
features of struct.

This patch also gets rid of the undocumented obsolete Pickler
dump_special method.
"""
1998-10-22 20:15:36 +00:00
Guido van Rossum 743d17e3aa In load_inst(), when instantiating an instance the old way (i.e. when
there's an __getinitargs__() method), if a TypeError occurs, catch and
reraise it but add info to the error about the class name being
instantiated.  This makes debugging a lot easier if __getinitargs__()
returns something bogus (e.g. a string instead of a singleton tuple).
1998-09-15 20:25:57 +00:00
Jeremy Hylton 2b9d029308 add handler for JPython's org.python.core.PyStringMap object, which
walks and quacks like a dictionary.
1998-05-27 22:38:22 +00:00
Guido van Rossum b19e2a383c Whoops! Add a missing 'instantiated = 1' to load_inst(); otherwise it
would still try to call the class...
1998-04-13 18:08:45 +00:00
Guido van Rossum b1062fc9d2 Correct dumb typo found by kjpylint (stack should be self.stack). 1998-03-31 17:00:46 +00:00
Guido van Rossum 599174f74d Correct a definite typo ('mem' should be 'memo'). 1998-03-31 16:30:28 +00:00
Guido van Rossum 45e2fbc2e7 Mass check-in after untabifying all files that need it. 1998-03-26 21:13:24 +00:00
Fred Drake fe82acc3b1 Fixed typo in docstring: "__ version__" --> "__version__" 1998-02-13 03:24:48 +00:00
Barry Warsaw abe2a457de Unpickler.load_inst(), Unpickler.load_obj(), Unpickler.load_build():
Fixed problems when unpickling in restricted execution environments.
These methods try to assign to an instance's __class__ attribute, or
access the instances __dict__, which are prohibited in REE.  For the
first two methods, I re-implemented the old behavior when assignment
to value.__class__ fails.

For the load_build() I also re-implemented the old behavior when
inst.__dict__.update() fails but this means that unpickling in REE is
semantically different than unpickling in unrestricted mode.
1998-01-26 22:47:35 +00:00
Guido van Rossum d1f4984a9b Jim Fulton writes:
The attached patch adds the following behavior to the handling
of REDUCE codes:

- A user-defined type may have a __reduce__ method that returns
  a string rather than a tuple, in which case the object is
  saved as a global object with a name given by the string returned
  by reduce.

  This was a feature added to cPickle a long time ago.


- User-defined types can now support unpickling without
  executing a constructor.

  The second value returned from '__reduce__' can now be None,
  rather than an argument tuple.  On unpickling, if the
  second value returned from '__reduce__' during pickling was
  None, then rather than calling the first value returned from
  '__reduce__', directly, the '__basicnew__' method of the
  first value returned from '__reduce__' is called without
  arguments.

I also got rid of a few of Chris' extra ()s, which he used
to make python ifs look like C ifs.
1997-12-10 23:40:18 +00:00
Guido van Rossum e680546894 Don't specify base 0 to string.atoi when unpickling integers in text
mode.  The pickler always uses base 10 so the default base should be
fine.  (The base gets us in trouble when there's no strop module, as
the atoi() in string.py only supports base 10.  This is for JPython.)
1997-12-10 19:36:41 +00:00
Guido van Rossum e467be6511 When instantiating a class with no arguments and where the class does
not define __getinitargs__, bypass the __init__ constructor
completely.  This uses the trick of instantiating an empty dummy class
and then changing inst.__class__ to the real class.  This is done in
two places: once for the INST and once for the OBJ format code.

Also replaced the much outdated long doc string with a short summary
of the module; the information of that doc string is already
incorporated in the library reference manual.
1997-12-05 19:42:42 +00:00
Guido van Rossum 4fb5b28dfc Three independent changes:
- Don't use "from copy_reg import *".

- Use cls.__module__ instead of calling whichobject(cls, cls.__name__);
  also try __module__ in whichmodule(), just in case.

- After calling save_reduce(), add the object to the memo.
1997-09-12 20:07:24 +00:00
Guido van Rossum d6ead328b3 *Semantic change*: when unpickling the instance variables of an
instance, use inst.__dict__.update(value) instead of a for loop with
setattr() over the value.keys().  This is more consistent (the
pickling doesn't use getattr() either but pickles inst.__dict__) and
avoids problems with instances that have a __setattr__ hook.

But it *is* a semantic change (because the setattr hook is no longer
used).  So beware!
1997-09-08 02:08:11 +00:00
Guido van Rossum 5ed5c4c364 Add the same _keep_alive patch (by Michael Scharff) that was added to
copy.deepcopy() a while ago.  Can't reproduce this but it doesn't
break anything and it looks like the code could have the same problem.
1997-09-03 00:23:54 +00:00