Mirror
/
cpython
mirror of https://github.com/python/cpython
4
1
Fork 0
Code Issues Releases Wiki Activity

gh-117151: increase default buffer size of shutil.copyfileobj() to 256k. (GH-119783) 

* gh-117151: increase default buffer size of shutil.copyfileobj() to 256k.

it was set to 16k in the 1990s.
it was raised to 64k in 2019. the discussion at the time mentioned another 5% improvement by raising to 128k and settled for a very conservative setting.

it's 2024 now, I think it should be revisited to match modern hardware. I am measuring 0-15% performance improvement when raising to 256k on various types of disk. there is no downside as far as I can tell.

this function is only intended for sequential copy of full files (or file like objects). it's the typical use case that benefits from larger operations.

for reference, I came across this function while trying to profile pip that is using it to copy files when installing python packages.

* add news

---------

Co-authored-by: rmorotti <romain.morotti@man.com>
This commit is contained in:
morotti 2024-10-05 00:51:22 +01:00 committed by GitHub
parent 8bcf118dcb
commit 6efd95c465
No known key found for this signature in database
GPG Key ID: B5690EEEBB952194
2 changed files with 4 additions and 1 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

2
Lib/shutil.py
View File

@ -44,7 +44,7 @@ if sys.platform == 'win32':
else:
_winapi = None
COPY_BUFSIZE = 1024 * 1024 if _WINDOWS else 64 * 1024
COPY_BUFSIZE = 1024 * 1024 if _WINDOWS else 256 * 1024
# This should never be removed, see rationale in:
# https://bugs.python.org/issue43743#msg393429
_USE_CP_SENDFILE = (hasattr(os, "sendfile")

3
Misc/NEWS.d/next/Library/2024-10-03-05-00-25.gh-issue-117151.Prdw_W.rst Normal file
View File

@ -0,0 +1,3 @@
The default buffer size used by :func:`shutil.copyfileobj` has been
increased from 64k to 256k on non-Windows platforms. It was already larger
on Windows.