Commit Graph

169 Commits

Author SHA1 Message Date
Tim Peters bf5ca65c2d load_string(): Force use of unsigned compare in a context that was
clearly (but incorrectly) assuming it.
2001-11-12 22:26:10 +00:00
Michael W. Hudson 03f96bd8f5 Fixes to compile cPickle.c & socketmodule.c on cygwin and possibly
other platforms that have funny ideas about whether addresses of
functions in dlls are compile-time constants.
2001-11-09 10:06:23 +00:00
Jeremy Hylton 3eb46f3a5d Must terminate the Pickler_members[] and Pickler_getsets with NULL. 2001-10-16 17:10:49 +00:00
Jeremy Hylton 499ab6a653 Better fix for core dumps on recursive objects in fast mode.
Raise ValueError when an object contains an arbitrarily nested
reference to itself.  (The previous fix just produced invalid
pickles.)

Solution is very much like Py_ReprEnter() and Py_ReprLeave():
fast_save_enter() and fast_save_leave() that tracks the fast_container
limit and keeps a fast_memo of objects currently being pickled.

The cost of the solution is moderately expensive for deeply nested
structures, but it still seems to be faster than normal pickling,
based on tests with deeply nested lists.

Once FAST_LIMIT is exceeded, the new code is about twice as slow as
fast-mode code that doesn't check for recursion.  It's still twice as
fast as the normal pickling code.  In the absence of deeply nested
structures, I couldn't measure a difference.
2001-10-15 21:37:58 +00:00
Jeremy Hylton a0fb177be8 Progress on SF bug #466175 and general cleanup.
Add a fast_container member to Picklerobject.  If fast is true, then
fast_container counts the depth of nested container calls.  If the
depth exceeds FAST_LIMIT (2000), the fast flag is ignored and the
normal checks occur.  This approach is much like the approach for
prevent stack overflow for comparison and reprs of recursive objects
(e.g. [[...]]).

    - Fast container used for save_list(), save_dict(), and
      save_inst().

      XXX Not clear which other save_xxx() functions should use it.

Make Picklerobject into new-style types, using PyObject_GenericGetAttr()
and PyObject_GenericSetAttr().

    - Use PyMemberDef for binary and fast members

    - Use PyGetSetDef for persistent_id, inst_persistent_id, memo, and
      PicklingError.

      XXX Not all of these seem like they need to use getset, but it's
      not clear why the old getattr() and setattr() had such odd
      semantics.  One change is that the getvalue() attribute will
      exist on all Picklers, not just list-based picklers; I think
      this is a more rationale interface.

There is a long laundry list of other changes:

    - Remove unused #defines for PyList_SET_ITEM() etc.

    - Make some of the indentation consistent

    - Replace uses of cPickle_PyMapping_HasKey() where the first
      argument is self->memo with calls to PyDict_GetItem(), because
      self->memo must be a dictionary.

    - Don't bother to check if cPickle_PyMapping_HasKey() returns < 0,
      because it can only return 0 or 1.

    - Replace uses of PyObject_CallObject() with PyObject_Call(), when
      we can guarantee that the argument tuple is really a tuple.

Performance impacts of these changes:

    - 5% speedup for normal pickling

    - No change to fast-mode pickling.

XXX Really need tests for all the features in cPickle that aren't in
pickle.
2001-10-12 04:11:06 +00:00
Tim Peters 12778e314b load_int: The fallback to long ints was coded in such a way that it
couldn't succeed.  Fixed.
2001-08-28 22:08:34 +00:00
Guido van Rossum a92d16aaec SF patch #452239 by Gordon McMillan, to fix SF bug #451547.
This patch attempts to do to cPickle what Guido did
   for pickle.py v 1.50. That is: save_global tries
   importing the module, and fetching the name from the
   module. If that fails, or the returned object is not
   the same one we started with, it raises a
   PicklingError. (All this so pickling a lambda will
   fail at save time, rather than load time).
2001-08-18 21:22:07 +00:00
Martin v. Löwis 339d0f720e Patch #445762: Support --disable-unicode
- Do not compile unicodeobject, unicodectype, and unicodedata if Unicode is disabled
- check for Py_USING_UNICODE in all places that use Unicode functions
- disables unicode literals, and the builtin functions
- add the types.StringTypes list
- remove Unicode literals from most tests.
2001-08-17 18:39:25 +00:00
Tim Peters 6d6c1a35e0 Merge of descr-branch back into trunk. 2001-08-02 04:15:00 +00:00
Fred Drake 2c7a6851ed Remove code to initialize globals that are never used.
Add some casts to quiet warnings from an unspecified non-GCC compiler.

This closes SF patch #436258.
2001-07-17 18:34:03 +00:00
Tim Peters d8ae7c2999 Ack -- this module mixes tabs and spaces, and what appears to be a mix
of 2-space and 4-space indents.  Whatever, when I saw the checkin diff it
was clear that what my editor thinks a tab means didn't match this module's
belief.  Removed all the tabs from the lines I added and changed, left
everything else alone.
2001-04-10 04:35:28 +00:00
Tim Peters 3906eb877a On a sizeof(long)==8 machine, ints in range(2**31, 2**32) were getting
pickled into the signed(!) 4-byte BININT format, so were getting unpickled
again as negative ints.  Repaired that.
Added some minimal docs at the top about what I've learned about the pickle
format codes (little of which was obvious from staring at the code,
although that's partly because all the size-related bugs greatly obscured
the true intent of the code).
Happy side effect:  because save_int() needed to grow a *proper* range
check in order to fix this bug, it can now use the more-efficient BININT1,
BININT2 and BININT formats when the long's value is small enough to fit
in a signed 4-byte int (before this, on a sizeof(long)==8 box it always
used the general INT format for negative ints).
test_cpickle works again on sizeof(long)==8 machines.  test_pickle is
still busted big-time.
2001-04-10 04:22:00 +00:00
Tim Peters bfa18f711f Critical fix: if cPickle on a sizeof(long)==8 box is used to read a
binary pickle, and the latter contains a pickle of a negative Python
int i written on a sizeof(long)==4 box (and whether by cPickle or
pickle.py), it's read incorrectly as i + 2**32.  The patch repairs that,
and allows test_cpickle.py (to which I added a relevant test case earlier
today) to work again on sizeof(long)==8 boxes.
There's another (at least one) sizeof(long)==8 binary pickle bug, but in
pickle.py instead.  That bug is still there, and test_pickle.py doesn't
catch it yet (try pickling and unpickling, e.g., 1 << 46).
2001-04-10 01:54:42 +00:00
Fred Drake 2c77355937 Make cPickle use the recently-added PyInstance_NewRaw() API to create
instance objects without calling the constructor.  This is the same as
the new.instance() function.
2001-03-22 17:52:17 +00:00
Tim Peters 84e87f379e SF bug [ #233200 ] cPickle does not use Py_BEGIN_ALLOW_THREADS.
http://sourceforge.net/tracker/?func=detail&aid=233200&group_id=5470&atid=105470
Wrapped the fread/fwrite calls in thread BEGIN_ALLOW/END_ALLOW brackets
Afraid I hit the "delete trailing whitespace key" too!  Only two "real" sections
of code changed here.
2001-03-17 04:50:51 +00:00
Guido van Rossum fb10c3f664 Minimal fix for the complaints about pickling Unicode objects. (SF
bugs #126161 and 123634).

The solution doesn't use the unicode-escape encoding; that has other
problems (it seems not 100% reversible).  Rather, it transforms the
input Unicode object slightly before encoding it using
raw-unicode-escape, so that the decoding will reconstruct the original
string: backslash and newline characters are translated into their
\uXXXX counterparts.

This is backwards incompatible for strings containing backslashes, but
for some of those strings, the pickling was already broken.

Note that SF bug #123634 complains specifically that cPickle fails to
unpickle the pickle for u'' (the empty Unicode string) correctly.
This was an off-by-one error in load_unicode().

XXX Ugliness: in order to do the modified raw-unicode-escape, I've
cut-and-pasted a copy of PyUnicode_EncodeRawUnicodeEscape() into this
file that also encodes '\\' and '\n'.  It might be nice to migrate
this into the Unicode implementation and give this encoding a new name
('half-raw-unicode-escape'? 'pickle-unicode-escape'?); that would help
pickle.py too.  But right now I can't be bothered with the necessary
infrastructural changes.
2000-12-19 02:08:38 +00:00
Neil Schemenauer 5196c586bb - Fix a GC bug caused by PyDict_New() failing. 2000-10-04 16:22:26 +00:00
Guido van Rossum ebba420285 Oops. Jim's fix didn't. This one does -- I tested it a bit better
this time!
2000-09-07 14:35:37 +00:00
Guido van Rossum c84d8bd7be Simple fix from Jin Fulton to avoid returning a half-initialized
module when e.g. copy_reg.py doesn't exist.  This caused a core dump.

This closes SF bug 112944.
2000-09-07 00:11:40 +00:00
Trent Mick 6c116dd56b Use safer comparisons (only matters when sizeof(int) != sizeof(size_t)). fread
and fwrite return size_t, so it is safer to cast up to the largest type for the
comparison. I believe the cast is required at all to remove compiler warnings.
2000-08-12 20:58:11 +00:00
Peter Schneider-Kamp 7e01890986 merge Include/my*.h into Include/pyport.h
marked my*.h as obsolete
2000-07-31 15:28:04 +00:00
Thomas Wouters 58d0510245 ANSIfy some more forward declarations. 2000-07-24 14:43:35 +00:00
Thomas Wouters 4789b3ae05 ... and yet more ANSIfications... 2000-07-24 11:36:47 +00:00
Thomas Wouters 3b6448fbae ANSIfication: add proper prototypes to function-pointers and declarations.
Also, fix a bug found by said declarations, where a string was defined as
unsigned char*, but used as signed.
2000-07-22 23:56:07 +00:00
Jeremy Hylton 03657cfdb0 replace PyXXX_Length calls with PyXXX_Size calls 2000-07-12 13:05:33 +00:00
Jeremy Hylton c5007aa5c3 final patches from Neil Schemenauer for garbage collection 2000-06-30 05:02:53 +00:00
Guido van Rossum 534b7c5c96 Trent Mick:
This patch fixes cPickle.c for 64-bit platforms.

- The false assumption sizeof(long) == size(void*) exists where
PyInt_FromLong is used to represent a pointer. The safe Python call
for this is PyLong_FromVoidPtr. (On platforms where the above
assumption *is* true a PyInt is returned as before so there is no
effective change.)

- use size_t instead of int for some variables
2000-06-28 22:23:56 +00:00
Guido van Rossum ea2b7157ab New version from Jim Fulton to fix a problem that Eric Raymond ran
into.  Jim writes:

The core dump was due to a C decrement operation
in a macro invocation in load_pop.  (BAD)

I fixed this by moving the decrement outside
the macro call.

I added a comment to load_pop and load_mark
to document the fact that cPickle separates the
unpickling stack into two separate stacks, one for
objects and one for marks.

I also moved some increments out of some macro
calls (PyTuple_SET_ITEM and PyList_SET_ITEM).
This wasn't necessary, but made me feel better. :)

I tested these changes in *my* cPickle, which
doesn't have the new Unicode stuff.
2000-05-09 18:14:50 +00:00
Guido van Rossum b18618dab7 Vladimir Marangozov's long-awaited malloc restructuring.
For more comments, read the patches@python.org archives.
For documentation read the comments in mymalloc.h and objimpl.h.

(This is not exactly what Vladimir posted to the patches list; I've
made a few changes, and Vladimir sent me a fix in private email for a
problem that only occurs in debug mode.  I'm also holding back on his
change to main.c, which seems unnecessary to me.)
2000-05-03 23:44:39 +00:00
Guido van Rossum 83addc7a0f Charles Waldman writes:
"""
Problem description:

	Run the following script:

import test.test_cpickle
for x in xrange(1000000):
    reload(test.test_cpickle)

Watch Python's memory use go up up and away!

In the course of debugging this I also saw that cPickle is
inconsistent with pickle - if you attempt a pickle.load or pickle.dump
on a closed file, you get a ValueError, whereas the corresponding
cPickle operations give an IOError.  Since cPickle is advertised as
being compatible with pickle, I changed these exceptions to match.
"""
2000-04-21 20:49:36 +00:00
Guido van Rossum 5fccb7c58e Marc-Andre Lemburg: support pickling Unicode objects, both in text
mode ('V') and in binary mode ('X').
2000-03-10 23:11:40 +00:00
Guido van Rossum 43713e5a28 Massive patch by Skip Montanaro to add ":name" to as many
PyArg_ParseTuple() format string arguments as possible.
2000-02-29 13:59:29 +00:00
Guido van Rossum 2f80d96c04 Patch by Stephen Turner, who writes:
"""
It fixes a memory corruption error resulting from BadPickleGet
exceptions in load_get, load_binget and load_long_binget.  This was
initially reported on c.l.py as a problem with Cookie.py; see the thread
titled "python core dump (SIGBUS) on Solaris" for more details.

If PyDict_GetItem(self->memo, py_key) call failed, then py_key was being
Py_DECREF'd out of existence before call was made to
PyErr_SetObject(BadPickleGet, py_key).

The bug can be duplicated as follows:

import cPickle
cPickle.loads('garyp')

This raises a BadPickleGet exception whose value is a freed object.  A
core dump will soon follow.
"""

Jim Fulton approves of the patch.
1999-07-13 15:18:58 +00:00
Guido van Rossum c3be1a3ca1 New version from Jim:
- Don't call Py_FatalError() when initialization fails.

- Fix bogus use of return value from PyRun_String().

- Fix misc. compiler errors on some platforms.
1999-06-15 14:36:59 +00:00
Guido van Rossum c03158bfc7 Jim Fulton writes:
I've updated cPickle.c to use class exceptions:

Changed pickle error types to classes:

  PickleError
     PicklingError
        UnpickleableError
     UnpicklingError

And change the handling of unpickleable objects so that an UnpickleableError
is raised with the unpickleable object as the argument.  UnpickleableError
has a reasonable string representation and provides access to the problem
object, which is useful during debugging.

[I'm still waiting for patches to do the same to pickle.py.]
1999-06-09 15:23:31 +00:00
Guido van Rossum 1b9e0aae6e Jim Fulton writes:
I have attached a new cPickle that adds a new control attribute
to unpicklers:

  Added new Unpickler attribute, find_global.  If set to None, then
  global and instance pickles are disabled.  Otherwise, it should be set to
  a callable object that takes two arguments, a module name and an
  object name, and returns an object.  If the attribute is unset, then
  the default mechanism is used.

  This feature provides an additional mechanism for controlling which
  classes can be used for unpickling.
1999-04-19 17:58:18 +00:00
Guido van Rossum 761fcd03aa Fix accidentally reversed NULL test in load_mark(). Suggested by
Tamito Kajiyama.  (This caused a bug only on platforms where malloc(0)
returns NULL.)
1999-04-12 22:51:20 +00:00
Guido van Rossum c91fcaa43b Protection against picling to/from closed (real) file.
The problem was reported by Moshe Zadka.
1999-03-29 20:00:14 +00:00
Guido van Rossum d1f66dc198 Fix buglet in load_put -- the test for bad readline result tested the
wrong variable.
1999-02-08 22:38:25 +00:00
Guido van Rossum f9ffb03c35 Jim Fulton: this fixes seg faults with bad pickles like "c". 1999-02-04 14:54:04 +00:00
Guido van Rossum aa8d16761b Make sure not to call realloc() with a NULL pointer -- call malloc()
in that case.  Tamito Kajiyama.
1999-01-25 21:43:51 +00:00
Guido van Rossum 21ef088265 Need to initialize self->safe_constructors early on to prevent crash
in early dealloc.  Patch by Andrew Dalke.
1998-12-11 03:20:00 +00:00
Guido van Rossum e94e3fbb72 Make VC++ 5.0 compiler happy. 1998-12-08 17:37:19 +00:00
Guido van Rossum 50f385c197 Fix two small bugs; add DL_EXPORT() to initcPickle decl. 1998-12-04 18:48:44 +00:00
Guido van Rossum 053b8dfcde New version from Jim Fulton:
- New copyright. (Open source)

  - Added new protocol for binary string pickles that
    takes out unneeded puts:

      p=Pickler()
      p.dump(x)
      p.dump(y)
      thePickle=p.getvalue()

    This has little or no impact on pickling time, but
    often reduces unpickling time and pickle size, sometimes
    significantly.

  - Changed unpickler to use internal data structure instead
    of list to reduce unpickling times by about a third.

  - Many cleanups to get rid of obfuscated error handling
    involving 'goto finally' and status variables.

  - Extensive reGuidofication. (formatting :)

  - Fixed binary floating-point pickling bug. 0.0 was not
    pickled correctly.

  - Now use binary floating point format when saving
    floats in binary mode.

  - Fixed some error message spelling error.
1998-11-25 16:18:00 +00:00
Jeremy Hylton ce616e4009 Enter Jim Fulton's latest version. He writes:
I had to make a slight diddle to work with Python 1.4, which
we and some of our customers are still using. :(

I've also made a few minor enhancements:

  - You can now both get and set the memo using a 'memo'
    attribute.  This is handy for certain advanced applications
    that we have.

  - Added a 'binary' attribute to get and set the binary
    mode for a pickler.

  - Added a somewhat experimental 'fast' attribute.  When this
    is set, objects are not placed in the memo during pickling.
    This should lead to faster pickling and smaller pickles in
    cases where:

      o you *know* there are no circular references, and

      o either you've:

        - preloaded the memo with class information
          by pickling classes in non-fast mode or by
          manipilating the memo directly, or

        - aren't pickling instances.
1998-08-13 23:13:52 +00:00
Jeremy Hylton d10552379d Two fixes to find_class:
1. Only DECREF the class's module when the module is retrieved via
PyImport_Import.  If it is retrieved from the modules dictionary with
PyDict_GetItem, it is using a borrowed reference.

2. If the module doesn't define the desired class, raise the same
SystemError that pickle.py does instead of returning an AttributeError
(which is cryptic at best).

Also, fix the PyArg_ParseTuple in cpm_loads (the externally visible
loads) function:  Use "S" instead of "O" because cStringIO will croak
with a "bad arguments to internal function" if passed anything other
than a string.
1998-08-11 19:52:51 +00:00
Guido van Rossum e2d81cd4d7 Jim Fulton's patches to get rid of the class_map(). 1998-08-08 19:40:10 +00:00
Fred Drake 764b984db5 Use PyErr_ExceptionMatches(...) instead of PyErr_Occurred() == ... in two
places.
1998-05-28 04:33:37 +00:00
Guido van Rossum ed33a3f415 whichmodule(): remove redundant PyErr_Clear(); add explicit setting
of error when sys.modules isn't there.
1998-05-14 02:34:46 +00:00
Guido van Rossum 104be4a4a7 Use %.17f to format floats/doubles 1998-04-03 21:13:02 +00:00
Guido van Rossum 8a6dba3562 Clear class_map in constructor so that when it later detects an error
and the destructor is called early, it doesn't DECREF garbage.
1998-03-06 01:39:39 +00:00
Guido van Rossum 57d9f2e6ec Renamed Jim's PyErr_[JF]Format() to cPickle_ErrFormat(). It's not a
standard Python API function so it should not have a Py prefix.
1998-01-19 23:18:18 +00:00
Guido van Rossum 9716aaa14c Jim Fulton:
- Loading non-binary string pickles checks for insecure
          strings. This is needed because cPickle (still)
          uses a restricted eval to parse non-binary string pickles.
          This change is needed to prevent untrusted
          pickles like::

            "S'hello world'*2000000\012p0\012."

          from hosing an application.

        - User-defined types can now support unpickling without
          executing a constructor.

          The second value returned from __reduce__ can now be None,
          rather than an argument tuple. On unpickling, if the second
          value returned from __reduce__ during pickling was None, then
          rather than calling the first value returned from __reduce__,
          directly, the __basicnew__ method of the first value returned
          from __reduce__ is called without arguments.
1997-12-08 15:15:16 +00:00
Guido van Rossum fdde96ce98 New versions of cPickle and cStringIO, from Jim Fulton's cPickle 1.0b1
distribution.
1997-12-04 01:13:01 +00:00
Guido van Rossum f6e8316b01 Initialize __version__ to the correct version string regardless of
what RCS checkout options are used.  Problem first diagnosed by Marc
Lemburg.
1997-12-01 15:57:40 +00:00
Guido van Rossum 4518823ad0 In whichmodule(), use __module__ if set. 1997-09-28 05:38:51 +00:00
Guido van Rossum 9efe8ef7a1 #Plug small memory leaks in constructors. 1997-09-03 18:19:40 +00:00
Barry Warsaw 779133c707 Removed JF's dollar-Log-dollar RCS turd that caused compilation to
crash due to GvR's last check in message :-).  Will try to convince JF
to remove all this evilness.
1997-08-21 22:36:26 +00:00
Guido van Rossum c6ef204830 Added /**/ around #end tags 1997-08-21 02:30:45 +00:00
Guido van Rossum 725d941f0f Renamed strndup to pystrndup, to avoid conflicting prototype
in GNU libc on some platforms.
1997-08-20 23:38:57 +00:00
Guido van Rossum 142eeb8339 cPickle release 0.3 from Jim Fulton 1997-08-13 03:14:41 +00:00
Guido van Rossum 5a37d7d150 Renamed strndup to my_strndup to avoid conflict witth GNU libc. 1997-05-16 16:36:52 +00:00
Guido van Rossum de8d6d73fb Use compile-time test for 64-bit hardware instead of run-time test.
This silences some compilers.
1997-05-13 18:00:44 +00:00
Guido van Rossum b05a5c7698 Instead of importing graminit.h whenever one of the three grammar 'root'
symbols is needed, define these in Python.h with a Py_ prefix.
1997-05-07 17:46:13 +00:00
Guido van Rossum d385d59c09 Give PyErr_Format a new name and make it static. 1997-04-09 17:47:47 +00:00
Guido van Rossum 60456fdcfe Jim Fulton's version 2.2. 1997-04-09 17:36:32 +00:00
Barry Warsaw 93d29b6895 Eliminated gcc -Wall complaints:
- Quieted gcc -Wall by removing unused local variables.

    - Added some choice parentheses around assignments in conditional
      tests.

    - Removed an unused (and seemingly unreachable) err label in
      load_short_binstring().

    - in Unpickler_load(), removed \. in string format.

    - init_stuff() was declared to return an int, but had these
      problems:

	- it was returning NULL instead of 0 or 1 in some cases
	- it was falling of the end of the routine without returning
	  anything
	- the call of init_stuff() in initcPickle() was never checking
	  the return value anyway.

      I changed all this by returning 1 in the case of errors, 0 when
      no error occurred.  Then in initcPickle(), if init_stuff()
      returns non-zero, I call Py_FatalError().

Suppressing my urge to reformat according to Python coding standards!
:-)
1997-01-14 17:45:08 +00:00
Guido van Rossum 2f4caa4c48 cPickle, version 0.1. 1997-01-06 22:59:08 +00:00