Mirror
/
cpython
mirror of https://github.com/python/cpython
4
1
Fork 0
Code Issues Releases Wiki Activity

gh-105967: Work around a macOS bug, limit zlib C library crc32 API calls to 1gig (#112615) 

Work around a macOS bug, limit zlib crc32 calls to 1GiB.

Without this, `zlib.crc32` and `binascii.crc32` could produce incorrect
results on multi-gigabyte inputs depending on the macOS version's Apple
supplied zlib implementation.
This commit is contained in:
Gregory P. Smith 2023-12-04 12:04:05 -08:00 committed by GitHub
parent a1551b48ee
commit 4eddb4c9d9
No known key found for this signature in database
GPG Key ID: 4AEE18F83AFDEB23
3 changed files with 30 additions and 10 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

4
Misc/NEWS.d/next/Core and Builtins/2023-12-01-19-02-21.gh-issue-105967.Puq5Cn.rst Normal file
View File

@ -0,0 +1,4 @@
Workaround a bug in Apple's macOS platform zlib library where
:func:`zlib.crc32` and :func:`binascii.crc32` could produce incorrect results
on multi-gigabyte inputs. Including when using :mod:`zipfile` on zips
containing large data.

18
Modules/binascii.c
View File

@ -770,12 +770,20 @@ binascii_crc32_impl(PyObject *module, Py_buffer *data, unsigned int crc)
Py_BEGIN_ALLOW_THREADS Py_BEGIN_ALLOW_THREADS
/* Avoid truncation of length for very large buffers. crc32() takes /* Avoid truncation of length for very large buffers. crc32() takes
length as an unsigned int, which may be narrower than Py_ssize_t. */ length as an unsigned int, which may be narrower than Py_ssize_t.
while ((size_t)len > UINT_MAX) { We further limit size due to bugs in Apple's macOS zlib.
crc = crc32(crc, buf, UINT_MAX); See https://github.com/python/cpython/issues/105967
buf += (size_t) UINT_MAX; */
len -= (size_t) UINT_MAX; #define ZLIB_CRC_CHUNK_SIZE 0x40000000
#if ZLIB_CRC_CHUNK_SIZE > INT_MAX
# error "unsupported less than 32-bit platform?"
#endif
while ((size_t)len > ZLIB_CRC_CHUNK_SIZE) {
crc = crc32(crc, buf, ZLIB_CRC_CHUNK_SIZE);
buf += (size_t) ZLIB_CRC_CHUNK_SIZE;
len -= (size_t) ZLIB_CRC_CHUNK_SIZE;
} }
#undef ZLIB_CRC_CHUNK_SIZE
crc = crc32(crc, buf, (unsigned int)len); crc = crc32(crc, buf, (unsigned int)len);
Py_END_ALLOW_THREADS Py_END_ALLOW_THREADS
} else { } else {

18
Modules/zlibmodule.c
View File

@ -1896,12 +1896,20 @@ zlib_crc32_impl(PyObject *module, Py_buffer *data, unsigned int value)
Py_BEGIN_ALLOW_THREADS Py_BEGIN_ALLOW_THREADS
/* Avoid truncation of length for very large buffers. crc32() takes /* Avoid truncation of length for very large buffers. crc32() takes
length as an unsigned int, which may be narrower than Py_ssize_t. */ length as an unsigned int, which may be narrower than Py_ssize_t.
while ((size_t)len > UINT_MAX) { We further limit size due to bugs in Apple's macOS zlib.
value = crc32(value, buf, UINT_MAX); See https://github.com/python/cpython/issues/105967.
buf += (size_t) UINT_MAX; */
len -= (size_t) UINT_MAX; #define ZLIB_CRC_CHUNK_SIZE 0x40000000
#if ZLIB_CRC_CHUNK_SIZE > INT_MAX
# error "unsupported less than 32-bit platform?"
#endif
while ((size_t)len > ZLIB_CRC_CHUNK_SIZE) {
value = crc32(value, buf, ZLIB_CRC_CHUNK_SIZE);
buf += (size_t) ZLIB_CRC_CHUNK_SIZE;
len -= (size_t) ZLIB_CRC_CHUNK_SIZE;
} }
#undef ZLIB_CRC_CHUNK_SIZE
value = crc32(value, buf, (unsigned int)len); value = crc32(value, buf, (unsigned int)len);
Py_END_ALLOW_THREADS Py_END_ALLOW_THREADS
} else { } else {