Commit Graph

130 Commits

Author SHA1 Message Date
Ruben Vorderman a89e6713c4
gh-101322: Ensure test_zlib.ZlibDecompressorTest runs, fix errors in ZlibDecompressor (#101323)
* Ensure test_zlib.ZlibDecompressorTest actually runs, fix errors in ZlibDecompressor.
2023-02-04 12:07:30 -08:00
Ruben Vorderman eae7dad402
gh-95534: Improve gzip reading speed by 10% (#97664)
Change summary:
+ There is now a `gzip.READ_BUFFER_SIZE` constant that is 128KB. Other programs that read in 128KB chunks: pigz and cat. So this seems best practice among good programs. Also it is faster than 8 kb chunks.
+ a zlib._ZlibDecompressor was added. This is the _bz2.BZ2Decompressor ported to zlib. Since the zlib.Decompress object is better for in-memory decompression, the _ZlibDecompressor is hidden. It only makes sense in file decompression, and that is already implemented now in the gzip library. No need to bother the users with this.
+ The ZlibDecompressor uses the older Cpython arrange_output_buffer functions, as those are faster and more appropriate for the use case. 
+ GzipFile.read has been optimized. There is no longer a `unconsumed_tail` member to write back to padded file. This is instead handled by the ZlibDecompressor itself, which has an internal buffer. `_add_read_data` has been inlined, as it was just two calls.

EDIT: While I am adding improvements anyway, I figured I could add another one-liner optimization now to the python -m gzip application. That read chunks in io.DEFAULT_BUFFER_SIZE previously, but has been updated now to use READ_BUFFER_SIZE chunks.
2022-10-16 19:10:58 -07:00
Victor Stinner 9475dc0b8d
bpo-46623: Skip two test_zlib tests on s390x (GH-31096)
Skip test_pair() and test_speech128() of test_zlib on s390x since
they fail if zlib uses the s390x hardware accelerator.
2022-02-25 00:32:02 +01:00
Ruben Vorderman ea23e7820f
bpo-43613: Faster implementation of gzip.compress and gzip.decompress (GH-27941)
Co-authored-by: Łukasz Langa <lukasz@langa.pl>
2021-09-02 17:02:59 +02:00
Erlend Egeberg Aasland fbff5387c3
bpo-43988: Use check disallow instantiation helper (GH-26392) 2021-05-27 08:43:52 +02:00
Erlend Egeberg Aasland 9746cda705
bpo-43916: Apply Py_TPFLAGS_DISALLOW_INSTANTIATION to selected types (GH-25748)
Apply Py_TPFLAGS_DISALLOW_INSTANTIATION to the following types:

* _dbm.dbm
* _gdbm.gdbm
* _multibytecodec.MultibyteCodec
* _sre..SRE_Scanner
* _thread._localdummy
* _thread.lock
* _winapi.Overlapped
* array.arrayiterator
* functools.KeyWrapper
* functools._lru_list_elem
* pyexpat.xmlparser
* re.Match
* re.Pattern
* unicodedata.UCD
* zlib.Compress
* zlib.Decompress
2021-04-30 16:04:57 +02:00
Hai Shi a089d21df1
bpo-40275: Use new test.support helper submodules in tests (GH-21315) 2020-07-06 11:15:08 +02:00
Victor Stinner 87502ddd71
bpo-40286: Use random.randbytes() in tests (GH-19575) 2020-04-17 22:54:38 +02:00
Serhiy Storchaka 6a44f6eef3
bpo-36048: Use __index__() instead of __int__() for implicit conversion if available. (GH-11952)
Deprecate using the __int__() method in implicit conversions of Python
numbers to C integers.
2019-02-25 17:57:58 +02:00
Steve Dower 57675090b0
bpo-34582: Update syntax of Azure Pipelines builds (GH-9521) 2018-09-24 07:44:50 -04:00
Zackery Spytz d2cbfffc84 bpo-25007: Add copy protocol support to zlib compressors and decompressors (GH-7940) 2018-06-27 21:04:51 +03:00
Xiang Zhang bc3f2289b9
bpo-32969: Expose some missing constants in zlib and fix the doc (GH-5988) 2018-03-07 13:05:37 +08:00
pmp-p 4c7108a771 bpo-32682: Improve libz version parsing in test_zilb (GH-5347) 2018-02-18 21:45:11 -06:00
Victor Stinner 8c663fd60e
Replace KB unit with KiB (#4293)
kB (*kilo* byte) unit means 1000 bytes, whereas KiB ("kibibyte")
means 1024 bytes. KB was misused: replace kB or KB with KiB when
appropriate.

Same change for MB and GB which become MiB and GiB.

Change the output of Tools/iobench/iobench.py.

Round also the size of the documentation from 5.5 MB to 5 MiB.
2017-11-08 14:44:44 -08:00
Serhiy Storchaka 989df09db2 Issue #16764: Move NEWS entry to correct section and remove too strict test. 2016-08-18 09:14:47 +03:00
Serhiy Storchaka 15f3228b7c Issue #16764: Support keyword arguments to zlib.decompress(). Patch by
Xiang Zhang.
2016-08-15 10:06:16 +03:00
Martin Panter 525a949251 Issue #27130: Merge zlib 64-bit fixes from 3.5 2016-07-23 03:39:49 +00:00
Martin Panter 84544c1020 Issue #27130: Fix handling of buffers exceeding UINT_MAX in “zlib” module
Patch by Xiang Zhang.
2016-07-23 03:02:07 +00:00
Serhiy Storchaka 95657cdd40 Issue #26243: Only the level argument to zlib.compress() is keyword argument
now.  The first argument is positional-only.
2016-06-25 22:43:05 +03:00
Martin Panter 1ab2f14281 Issue #27164: Merge raw Deflate zdict support from 3.5 2016-06-05 12:07:48 +00:00
Martin Panter 3f0ee83f14 Issue #27164: Allow decompressing raw Deflate streams with predefined zdict
Based on patch by Xiang Zhang.
2016-06-05 10:48:34 +00:00
Martin Panter 5677aa75f0 Issue #5784: Merge zlib from 3.5 2016-05-27 11:30:59 +00:00
Martin Panter c618ae8ed7 Issue #5784: wbits=0 apparently added in zlib v1.2.3.5 2016-05-27 11:20:21 +00:00
Martin Panter f4affb71bc Issue #5784: Merge zlib from 3.5 2016-05-27 08:00:24 +00:00
Martin Panter 0fdf41d847 Issue #5784: Expand documentation and tests for zlib wbits parameter
Based on documentation by AM Kuchling.
2016-05-27 07:32:11 +00:00
Martin Panter cda80940ed Issue #15984: Merge PyUnicode doc from 3.5 2016-04-15 02:27:11 +00:00
Martin Panter 6245cb3c01 Correct “an” → “a” with “Unicode”, “user”, “UTF”, etc
This affects documentation, code comments, and a debugging messages.
2016-04-15 02:14:19 +00:00
Martin Panter 1fe0d13d12 Issue #26243: zlib.compress() keyword argument support by Aviv Palivoda 2016-02-10 10:06:36 +00:00
Martin Panter b82032f935 Issue #22341: Drop Python 2 workaround and document CRC initial value
Also align the parameter naming in binascii to be consistent with zlib.
2015-12-11 05:19:29 +00:00
Martin Panter e99e97762c Issue #25626: Change zlib to accept Py_ssize_t and cap to UINT_MAX
The underlying zlib library stores sizes in “unsigned int”. The corresponding
Python parameters are all sizes of buffers filled in by zlib, so it is okay
to reduce higher values to the UINT_MAX internal cap. OverflowError is still
raised for sizes that do not fit in Py_ssize_t.

Sizes are now limited to Py_ssize_t rather than unsigned long, because Python
byte strings cannot be larger than Py_ssize_t. Previously this could result
in a SystemError on 32-bit platforms.

This resolves a regression in the gzip module when reading more than UINT_MAX
or LONG_MAX bytes in one call, introduced by revision 62723172412c.
2015-11-20 08:13:35 +00:00
Serhiy Storchaka 609a2e17ad Issue #22995: Default implementation of __reduce__ and __reduce_ex__ now
rejects builtin types with not defined __new__.
Added tests for non-pickleable types.
2015-11-12 11:31:51 +02:00
Serhiy Storchaka d7a4415599 Issue #22995: Default implementation of __reduce__ and __reduce_ex__ now
rejects builtin types with not defined __new__.
Added tests for non-pickleable types.
2015-11-12 11:23:04 +02:00
Martin Panter 0ff89099c7 Issue #23738: Merge 3.4 into 3.5 2015-09-09 01:56:53 +00:00
Martin Panter bf19d16950 Issue #23738: Document and test actual keyword parameter names
Also fix signature because os.utime(..., ns=None) is not allowed.
2015-09-09 01:01:13 +00:00
Zachary Ware 38c707e7e0 Issue #21741: Update 147 test modules to use test discovery.
I have compared output between pre- and post-patch runs of these tests
to make sure there's nothing missing and nothing broken, on both
Windows and Linux.  The only differences I found were actually tests
that were previously *not* run.
2015-04-13 15:00:43 -05:00
Serhiy Storchaka 43767638a9 Issue #18702: All skipped tests now reported as skipped. 2013-11-03 21:31:38 +02:00
Nadeem Vawda ec6dfcffa0 Issue #16411: Fix a bug where zlib.decompressobj().flush() might try to access previously-freed memory.
Patch by Serhiy Storchaka.
2012-11-11 03:16:44 +01:00
Nadeem Vawda 7ee955550b Issue #16411: Fix a bug where zlib.decompressobj().flush() might try to access previously-freed memory.
Patch by Serhiy Storchaka.
2012-11-11 03:15:32 +01:00
Nadeem Vawda dd1253abdd Issue #16350, part 2: Set unused_data (and unconsumed_tail) correctly in decompressobj().flush().
Additionally, fix a bug where a MemoryError in allocating a bytes object could
leave the decompressor object in an invalid state (with its unconsumed_tail
member being NULL).

Patch by Serhiy Storchaka.
2012-11-11 02:21:22 +01:00
Nadeem Vawda ee7889dec3 Issue #16350, part 2: Set unused_data (and unconsumed_tail) correctly in decompressobj().flush().
Additionally, fix a bug where a MemoryError in allocating a bytes object could
leave the decompressor object in an invalid state (with its unconsumed_tail
member being NULL).

Patch by Serhiy Storchaka.
2012-11-11 02:14:36 +01:00
Nadeem Vawda 519f43844b Issue #16350: Fix zlib decompressor handling of unused_data with multiple calls to decompress() after EOF.
Patch by Serhiy Storchaka.
2012-11-05 00:38:48 +01:00
Nadeem Vawda 39079946a2 Issue #16350: Fix zlib decompressor handling of unused_data with multiple calls to decompress() after EOF.
Patch by Serhiy Storchaka.
2012-11-05 00:37:42 +01:00
Nadeem Vawda cf5e1d82e3 Tidy up comments from dd4f7d5c51c7 (zlib compression dictionary support). 2012-06-22 00:35:57 +02:00
Nadeem Vawda fd8a838d58 Issue #14684: Add support for predefined compression dictionaries to the zlib module.
Original patch by Sam Rushing.
2012-06-21 02:13:12 +02:00
Nadeem Vawda 05a4dd83af Issue #13873: Fix crash in test_zlib when running on a small (<4GB) tmpfs. 2012-02-27 00:42:58 +02:00
Nadeem Vawda 197e22c1d7 Simplify zlib bigmem tests. 2012-02-23 14:23:17 +02:00
Nadeem Vawda bc8c8175cc Fix previous fix (for test_zlib) to work on 32-bit systems. 2012-02-23 14:16:15 +02:00
Nadeem Vawda b063a48406 Issue #13873: Fix crash in test_zlib on bigmem buildbot. 2012-02-23 13:36:25 +02:00
Nadeem Vawda d770fe45a0 Fix typo in comment. 2012-01-28 17:32:47 +02:00
Nadeem Vawda 131c7079d3 Issue #13862: Relax zlib version test to avoid spurious failures. 2012-01-25 23:16:50 +02:00