cpython/Modules/clinic
Giampaolo Rodola 4a172ccc73
bpo-33671: efficient zero-copy for shutil.copy* functions (Linux, OSX and Win) (#7160)
* have shutil.copyfileobj use sendfile() if possible

* refactoring: use ctx manager

* add test with non-regular file obj

* emulate case where file size can't be determined

* reference _copyfileobj_sendfile directly

* add test for offset() at certain position

* add test for empty file

* add test for non regular file dst

* small refactoring

* leave copyfileobj() alone in order to not introduce any incompatibility

* minor refactoring

* remove old test

* update docstring

* update docstring; rename exception class

* detect platforms which only support file to socket zero copy

* don't run test on platforms where file-to-file zero copy is not supported

* use tempfiles

* reset verbosity

* add test for smaller chunks

* add big file size test

* add comment

* update doc

* update whatsnew doc

* update doc

* catch Exception

* remove unused import

* add test case for error on second sendfile() call

* turn docstring into comment

* add one more test

* update comment

* add Misc/NEWS entry

* get rid of COPY_BUFSIZE; it belongs to another PR

* update doc

* expose posix._fcopyfile() for OSX

* merge from linux branch

* merge from linux branch

* expose fcopyfile

* arg clinic for the win implementation

* convert path type to path_t

* expose CopyFileW

* fix windows tests

* release GIL

* minor refactoring

* update doc

* update comment

* update docstrings

* rename functions

* rename test classes

* update doc

* update doc

* update docstrings and comments

* avoid do import nt|posix modules if unnecessary

* set nt|posix modules to None if not available

* micro speedup

* update description

* add doc note

* use better wording in doc

* rename function using 'fastcopy' prefix instead of 'zerocopy'

* use :ref: in rst doc

* change wording in doc

* add test to make sure sendfile() doesn't get called aymore in case it doesn't support file to file copies

* move CopyFileW in _winapi and actually expose CopyFileExW instead

* fix line endings

* add tests for mode bits

* add docstring

* remove test file mode class; let's keep it for later when Istart addressing OSX fcopyfile() specific copies

* update doc to reflect new changes

* update doc

* adjust tests on win

* fix argument clinic error

* update doc

* OSX: expose copyfile(3) instead of fcopyfile(3); also expose flags arg to python

* osx / copyfile: use path_t instead of char

* do not set dst name in the OSError exception in order to remain consistent with platforms which cannot do that (e.g. linux)

* add same file test

* add test for same file

* have osx copyfile() pre-emptively check if src and dst are the same, otherwise it will return immedialtey and src file content gets deleted

* turn PermissionError into appropriate SameFileError

* expose ERROR_SHARING_VIOLATION in order to raise more appropriate SameFileError

* honour follow_symlinks arg when using CopyFileEx

* update Misc/NEWS

* expose CreateDirectoryEx mock

* change C type

* CreateDirectoryExW actual implementation

* provide specific makedirs() implementation for win

* fix typo

* skeleton for SetNamedSecurityInfo

* get security info for src path

* finally set security attrs

* add unit tests

* mimick os.makedirs() behavior and raise if dst dir exists

* set 2 paths for OSError object

* set 2 paths for OSError object

* expand windows test

* in case of exception on os.sendfile() set filename and filename2 exception attributes

* set 2 filenames (src, dst) for OSError in case copyfile() fails on OSX

* update doc

* do not use CreateDirectoryEx() in copytree() if source dir is a symlink (breaks test_copytree_symlink_dir); instead just create a plain dir and remain consistent with POSIX implementation

* use bytearray() and readinto()

* use memoryview() with bytearray()

* refactoring + introduce a new _fastcopy_binfileobj() fun

* remove CopyFileEx and other C wrappers

* remove code related to CopyFileEx

* Recognize binary files in copyfileobj()
...and use fastest _fastcopy_binfileobj() when possible

* set 1MB copy bufsize on win; also add a global _COPY_BUFSIZE variable

* use ctx manager for memoryview()

* update doc

* remove outdated doc

* remove last CopyFileEx remnants

* OSX - use fcopyfile(3) instead of copyfile(3)

...as an extra safety measure: in case src/dst are "exotic" files (non
regular or living on a network fs etc.) we better fail on open() instead
of copyfile(3) as we're not quite sure what's gonna happen in that
case.

* update doc
2018-06-12 23:04:50 +02:00
..
_abc.c.h bpo-31333: Re-implement ABCMeta in C (#5273) 2018-02-18 12:41:58 +00:00
_asynciomodule.c.h bpo-32643: Drop support for a few private Task and Future APIs. (#5293) 2018-01-24 11:31:01 -05:00
_bz2module.c.h bpo-32240: Add the const qualifier to declarations of PyObject* array arguments. (#4746) 2017-12-15 13:11:11 +02:00
_codecsmodule.c.h bpo-32240: Add the const qualifier to declarations of PyObject* array arguments. (#4746) 2017-12-15 13:11:11 +02:00
_contextvarsmodule.c.h bpo-32436: Implement PEP 567 (#5027) 2018-01-22 19:11:18 -05:00
_cryptmodule.c.h bpo-32240: Add the const qualifier to declarations of PyObject* array arguments. (#4746) 2017-12-15 13:11:11 +02:00
_curses_panel.c.h bpo-20171: Convert the _curses and _curses_panel modules to Argument Clinic. (GH-4251) 2018-05-10 11:27:23 +03:00
_cursesmodule.c.h bpo-20171: Convert the _curses and _curses_panel modules to Argument Clinic. (GH-4251) 2018-05-10 11:27:23 +03:00
_datetimemodule.c.h bpo-32240: Add the const qualifier to declarations of PyObject* array arguments. (#4746) 2017-12-15 13:11:11 +02:00
_dbmmodule.c.h bpo-33383: Fix crash in get() of the dbm.ndbm database object. (#6630) 2018-04-29 12:38:06 +03:00
_elementtree.c.h bpo-32240: Add the const qualifier to declarations of PyObject* array arguments. (#4746) 2017-12-15 13:11:11 +02:00
_gdbmmodule.c.h bpo-32922: dbm.open() now encodes filename with the filesystem encoding. (GH-5832) 2018-02-26 16:02:22 +02:00
_hashopenssl.c.h bpo-32433: Optimized HMAC digest (#5023) 2018-01-27 09:53:43 +01:00
_lzmamodule.c.h bpo-32240: Add the const qualifier to declarations of PyObject* array arguments. (#4746) 2017-12-15 13:11:11 +02:00
_opcode.c.h bpo-32240: Add the const qualifier to declarations of PyObject* array arguments. (#4746) 2017-12-15 13:11:11 +02:00
_operator.c.h bpo-32240: Add the const qualifier to declarations of PyObject* array arguments. (#4746) 2017-12-15 13:11:11 +02:00
_pickle.c.h bpo-23403: Bump pickle.DEFAULT_PROTOCOL to 4 (#6355) 2018-04-03 23:06:53 -07:00
_queuemodule.c.h bpo-14976: Reentrant simple queue (#3346) 2018-01-16 00:27:16 +01:00
_sre.c.h bpo-32240: Add the const qualifier to declarations of PyObject* array arguments. (#4746) 2017-12-15 13:11:11 +02:00
_ssl.c.h bpo-30622: Fix NPN for OpenSSL 1.1.1-pre1 (#5876) 2018-02-25 12:31:33 +01:00
_struct.c.h bpo-30249: Improve struct.unpack_from() error messages (GH-6059) 2018-03-11 02:58:52 +08:00
_tkinter.c.h bpo-32240: Add the const qualifier to declarations of PyObject* array arguments. (#4746) 2017-12-15 13:11:11 +02:00
_tracemalloc.c.h bpo-32240: Add the const qualifier to declarations of PyObject* array arguments. (#4746) 2017-12-15 13:11:11 +02:00
_weakref.c.h bpo-32240: Add the const qualifier to declarations of PyObject* array arguments. (#4746) 2017-12-15 13:11:11 +02:00
_winapi.c.h bpo-19764: Implemented support for subprocess.Popen(close_fds=True) on Windows (#1218) 2017-12-18 10:28:19 +01:00
arraymodule.c.h bpo-32240: Add the const qualifier to declarations of PyObject* array arguments. (#4746) 2017-12-15 13:11:11 +02:00
audioop.c.h bpo-32240: Add the const qualifier to declarations of PyObject* array arguments. (#4746) 2017-12-15 13:11:11 +02:00
binascii.c.h bpo-32240: Add the const qualifier to declarations of PyObject* array arguments. (#4746) 2017-12-15 13:11:11 +02:00
cmathmodule.c.h bpo-32240: Add the const qualifier to declarations of PyObject* array arguments. (#4746) 2017-12-15 13:11:11 +02:00
fcntlmodule.c.h bpo-32240: Add the const qualifier to declarations of PyObject* array arguments. (#4746) 2017-12-15 13:11:11 +02:00
gcmodule.c.h bpo-9566: Fix size_t=>int downcast warnings (#5230) 2018-01-18 11:15:25 +01:00
grpmodule.c.h bpo-32240: Add the const qualifier to declarations of PyObject* array arguments. (#4746) 2017-12-15 13:11:11 +02:00
mathmodule.c.h bpo-32240: Add the const qualifier to declarations of PyObject* array arguments. (#4746) 2017-12-15 13:11:11 +02:00
md5module.c.h bpo-32240: Add the const qualifier to declarations of PyObject* array arguments. (#4746) 2017-12-15 13:11:11 +02:00
posixmodule.c.h bpo-33671: efficient zero-copy for shutil.copy* functions (Linux, OSX and Win) (#7160) 2018-06-12 23:04:50 +02:00
pwdmodule.c.h - Issue #27332: Fixed the type of the first argument of module-level functions 2016-07-07 18:20:03 +03:00
pyexpat.c.h bpo-32240: Add the const qualifier to declarations of PyObject* array arguments. (#4746) 2017-12-15 13:11:11 +02:00
resource.c.h bpo-32240: Add the const qualifier to declarations of PyObject* array arguments. (#4746) 2017-12-15 13:11:11 +02:00
sha1module.c.h bpo-32240: Add the const qualifier to declarations of PyObject* array arguments. (#4746) 2017-12-15 13:11:11 +02:00
sha256module.c.h bpo-32240: Add the const qualifier to declarations of PyObject* array arguments. (#4746) 2017-12-15 13:11:11 +02:00
sha512module.c.h bpo-32240: Add the const qualifier to declarations of PyObject* array arguments. (#4746) 2017-12-15 13:11:11 +02:00
signalmodule.c.h bpo-33441: Make the sigset_t converter available in other modules. (GH-6720) 2018-05-08 07:48:50 +03:00
spwdmodule.c.h - Issue #27332: Fixed the type of the first argument of module-level functions 2016-07-07 18:20:03 +03:00
symtablemodule.c.h bpo-32240: Add the const qualifier to declarations of PyObject* array arguments. (#4746) 2017-12-15 13:11:11 +02:00
unicodedata.c.h bpo-32240: Add the const qualifier to declarations of PyObject* array arguments. (#4746) 2017-12-15 13:11:11 +02:00
zipimport.c.h bpo-32248 - Implement `ResourceReader` and `get_resource_reader()` for zipimport (#5248) 2018-01-24 15:36:21 -05:00
zlibmodule.c.h bpo-32240: Add the const qualifier to declarations of PyObject* array arguments. (#4746) 2017-12-15 13:11:11 +02:00