Commit Graph

204 Commits

Author SHA1 Message Date
Serhiy Storchaka 1211c9a989
bpo-32503: Avoid creating too small frames in pickles. (#5127) 2018-01-20 16:42:44 +02:00
Serhiy Storchaka 5b76bdba07
bpo-31993: Do not use memoryview when pickle large strings. (#5154)
PyMemoryView_FromMemory() created a memoryview referring to
the internal data of the string.  When the string is destroyed
the memoryview become referring to a freed memory.
2018-01-13 00:28:31 +02:00
Serhiy Storchaka 0a2da50e18
bpo-31993: Do not create frames for large bytes and str objects (#5114)
when serialize into memory buffer with C pickle implementations.

This optimization already is performed when serialize into memory
with Python pickle implementations or into a file with both
implementations.
2018-01-11 13:03:20 +02:00
Olivier Grisel 3cd7c6e6eb bpo-31993: Do not allocate large temporary buffers in pickle dump. (#4353)
The picklers do no longer allocate temporary memory when dumping large
bytes and str objects into a file object. Instead the data is
directly streamed into the underlying file object.

Previously the C implementation would buffer all content and issue a
single call to file.write() at the end of the dump. With protocol 4
this behavior has changed to issue one call to file.write() per frame.

The Python pickler with protocol 4 now dumps each frame content as a
memoryview to an IOBytes instance that is never reused and the
memoryview is no longer released after the call to write. This makes it
possible for the file object to delay access to the memoryview of
previous frames without forcing any additional memory copy as was
already possible with the C pickler.
2018-01-06 17:18:54 +02:00
Serhiy Storchaka 3daaafb700
bpo-32037: Use the INT opcode for 32-bit integers in protocol 0 pickles. (#4407) 2017-11-16 09:44:43 +02:00
Serhiy Storchaka 6545256df9
bpo-32032: Test both implementations of module-level pickle API. (#4401) 2017-11-15 14:01:08 +02:00
Victor Stinner 8c663fd60e
Replace KB unit with KiB (#4293)
kB (*kilo* byte) unit means 1000 bytes, whereas KiB ("kibibyte")
means 1024 bytes. KB was misused: replace kB or KB with KiB when
appropriate.

Same change for MB and GB which become MiB and GiB.

Change the output of Tools/iobench/iobench.py.

Round also the size of the documentation from 5.5 MB to 5 MiB.
2017-11-08 14:44:44 -08:00
Serhiy Storchaka 3410c01d83 Issue #17711: Fixed unpickling by the persistent ID with protocol 0.
Original patch by Alexandre Vassalotti.
2016-07-17 11:35:35 +03:00
Serhiy Storchaka dec25afab1 Issue #17711: Fixed unpickling by the persistent ID with protocol 0.
Original patch by Alexandre Vassalotti.
2016-07-17 11:24:17 +03:00
Serhiy Storchaka 96c058b4de Fixed typo in pickle tests. 2016-03-04 09:40:03 +02:00
Serhiy Storchaka be70002866 Fixed typo in pickle tests. 2016-03-04 09:39:47 +02:00
Serhiy Storchaka 58f8833e4c Make catched exception more specific and correct a comment. 2015-12-30 21:00:29 +02:00
Serhiy Storchaka beb652ccc1 Make catched exception more specific and correct a comment. 2015-12-30 21:00:08 +02:00
Serhiy Storchaka 59fb6342a4 Issue #25761: Improved detecting errors in broken pickle data. 2015-12-06 22:01:35 +02:00
Serhiy Storchaka 92d5fbaf8f Issue #25761: Added more test cases for testing unpickling broken data.
Output raised exception at verbose level 2 (-vv).
2015-11-29 13:14:00 +02:00
Serhiy Storchaka 3c49710e9f Issue #25761: Added more test cases for testing unpickling broken data.
Output raised exception at verbose level 2 (-vv).
2015-11-29 13:13:24 +02:00
Serhiy Storchaka 7279befccb Issue #25761: Added more test cases for testing unpickling broken data.
Output raised exception at verbose level 2 (-vv).
2015-11-29 13:12:10 +02:00
Serhiy Storchaka b1ce30281d Issue #23914: Fixed SystemError raised by unpickler on broken pickle data. 2015-11-23 15:20:21 +02:00
Serhiy Storchaka 4409c6cfae Issue #23914: Fixed SystemError raised by unpickler on broken pickle data. 2015-11-23 15:19:23 +02:00
Serhiy Storchaka e9b3074cf9 Issue #23914: Fixed SystemError raised by unpickler on broken pickle data. 2015-11-23 15:17:43 +02:00
Serhiy Storchaka 41a87637c0 Issue #892902: Added new tests for pickling recursive collections. 2015-11-07 11:17:15 +02:00
Serhiy Storchaka 838b7cc0e7 Issue #892902: Added new tests for pickling recursive collections. 2015-11-07 11:17:02 +02:00
Serhiy Storchaka 608c213530 Issue #892902: Added new tests for pickling recursive collections. 2015-11-07 11:16:10 +02:00
Serhiy Storchaka 0d554d7ef1 Issue #24164: Objects that need calling ``__new__`` with keyword arguments,
can now be pickled using pickle protocols older than protocol version 4.
2015-10-10 22:42:18 +03:00
Serhiy Storchaka 525faaeffc Issue #25262. Added support for BINBYTES8 opcode in Python implementation of
unpickler.  Highest 32 bits of 64-bit size for BINUNICODE8 and BINBYTES8
opcodes no longer silently ignored on 32-bit platforms in C implementation.
2015-09-29 22:12:29 +03:00
Serhiy Storchaka e060619d4b Issue #25262. Added support for BINBYTES8 opcode in Python implementation of
unpickler.  Highest 32 bits of 64-bit size for BINUNICODE8 and BINBYTES8
opcodes no longer silently ignored on 32-bit platforms in C implementation.
2015-09-29 22:10:07 +03:00
Serhiy Storchaka cb764960c9 Added additional unpickling tests. 2015-09-29 15:50:45 +03:00
Serhiy Storchaka b8b951f6ee Added additional unpickling tests. 2015-09-29 15:49:58 +03:00
Serhiy Storchaka 104de66b76 Moved unpickling tests with prepickled data to separate class. 2015-09-29 15:34:53 +03:00
Serhiy Storchaka c6b54b45ea Moved unpickling tests with prepickled data to separate class. 2015-09-29 15:33:24 +03:00
Benjamin Peterson b32b998bf7 merge 3.4 2015-07-02 16:58:31 -05:00
Benjamin Peterson d3a2a95125 use correct __new__ method (closes #24552) 2015-07-02 16:58:22 -05:00
Benjamin Peterson 42e913aa3a merge 3.4 (#24552) 2015-07-02 16:18:58 -05:00
Benjamin Peterson 80f78a3efc fix use after free (closes #24552) 2015-07-02 16:18:38 -05:00
Serhiy Storchaka 58e4134a1c Issue #23611: Serializing more "lookupable" objects (such as unbound methods
or nested classes) now are supported with pickle protocols < 4.
2015-03-31 14:07:24 +03:00
Serhiy Storchaka d5d818d40b Issue #18473: Fixed 2to3 and 3to2 compatible pickle mappings.
Fixed ambigious reverse mappings.  Added many new mappings.  Import mapping
is no longer applied to modules already mapped with full name mapping.

Added tests for compatible pickling and unpickling and for consistency of
_compat_pickle mappings.
2015-03-31 13:17:10 +03:00
Serhiy Storchaka bfe1824d08 Issue #18473: Fixed 2to3 and 3to2 compatible pickle mappings.
Fixed ambigious reverse mappings.  Added many new mappings.  Import mapping
is no longer applied to modules already mapped with full name mapping.

Added tests for compatible pickling and unpickling and for consistency of
_compat_pickle mappings.
2015-03-31 13:12:37 +03:00
Serhiy Storchaka d362c21264 Use pickled data compatible with Python 2 for testing protocols 0-2. 2015-02-15 14:10:24 +02:00
Serhiy Storchaka fa310ee3a9 Use pickled data compatible with Python 2 for testing protocols 0-2. 2015-02-15 14:10:03 +02:00
Serhiy Storchaka 230586739c Issue #23094: Fixed readline with frames in Python implementation of pickle. 2015-01-26 10:37:44 +02:00
Serhiy Storchaka 21d7533c4c Issue #23094: Fixed readline with frames in Python implementation of pickle. 2015-01-26 10:37:01 +02:00
Serhiy Storchaka 707b5ccde5 Issue #22783: Pickling now uses the NEWOBJ opcode instead of the NEWOBJ_EX
opcode if possible.
2014-12-16 19:43:46 +02:00
Antoine Pitrou 6cd5eda0dd Fix uninitialized variable after #22676. 2014-12-02 00:20:03 +01:00
Serhiy Storchaka 8cf7c1cff0 Issue #22775: Fixed unpickling of http.cookies.SimpleCookie with protocol 2
and above.  Patch by Tim Graham.
2014-11-02 22:18:25 +02:00
Serhiy Storchaka 786ac7b27d Issue #19886: Use better estimated memory requirements for bigmem tests.
Incorrect requirements can cause memory swapping.
2014-01-10 13:39:27 +02:00
Serhiy Storchaka 4847e4e1f4 Issue #19886: Use better estimated memory requirements for bigmem tests.
Incorrect requirements can cause memory swapping.
2014-01-10 13:37:54 +02:00
Antoine Pitrou c1764dd350 Issue #19648: implement empty tests in pickletester. Patch by Gennadiy Zlobin. 2013-12-28 16:57:37 +01:00
Alexandre Vassalotti d05c9ff845 Issue #6784: Strings from Python 2 can now be unpickled as bytes objects.
Initial patch by Merlijn van Deen.

I've added a few unrelated docstring fixes in the patch while I was at
it, which makes the documentation for pickle a bit more consistent.
2013-12-07 01:09:27 -08:00
Alexandre Vassalotti 6e73ff1a31 Issue #19881: Fix bad pickling of large bytes in cpickle. 2013-12-05 19:29:32 -08:00
Antoine Pitrou 6e8bc50317 Issue #19800: make the pickle framing tests more precise. 2013-12-03 09:51:40 +01:00