Commit Graph

193 Commits

Author SHA1 Message Date
Giampaolo Rodola c7f02a9659
bpo-33671 / shutil.copyfile: use memoryview() with dynamic size on Windows (#7681)
bpo-33671
* use memoryview() with size == file size on Windows, see https://github.com/python/cpython/pull/7160#discussion_r195405230
* release intermediate (sliced) memoryview immediately
* replace "OSX" occurrences with "macOS"
* add some unittests for copyfileobj()
2018-06-19 08:27:29 -07:00
Giampaolo Rodola 4a172ccc73
bpo-33671: efficient zero-copy for shutil.copy* functions (Linux, OSX and Win) (#7160)
* have shutil.copyfileobj use sendfile() if possible

* refactoring: use ctx manager

* add test with non-regular file obj

* emulate case where file size can't be determined

* reference _copyfileobj_sendfile directly

* add test for offset() at certain position

* add test for empty file

* add test for non regular file dst

* small refactoring

* leave copyfileobj() alone in order to not introduce any incompatibility

* minor refactoring

* remove old test

* update docstring

* update docstring; rename exception class

* detect platforms which only support file to socket zero copy

* don't run test on platforms where file-to-file zero copy is not supported

* use tempfiles

* reset verbosity

* add test for smaller chunks

* add big file size test

* add comment

* update doc

* update whatsnew doc

* update doc

* catch Exception

* remove unused import

* add test case for error on second sendfile() call

* turn docstring into comment

* add one more test

* update comment

* add Misc/NEWS entry

* get rid of COPY_BUFSIZE; it belongs to another PR

* update doc

* expose posix._fcopyfile() for OSX

* merge from linux branch

* merge from linux branch

* expose fcopyfile

* arg clinic for the win implementation

* convert path type to path_t

* expose CopyFileW

* fix windows tests

* release GIL

* minor refactoring

* update doc

* update comment

* update docstrings

* rename functions

* rename test classes

* update doc

* update doc

* update docstrings and comments

* avoid do import nt|posix modules if unnecessary

* set nt|posix modules to None if not available

* micro speedup

* update description

* add doc note

* use better wording in doc

* rename function using 'fastcopy' prefix instead of 'zerocopy'

* use :ref: in rst doc

* change wording in doc

* add test to make sure sendfile() doesn't get called aymore in case it doesn't support file to file copies

* move CopyFileW in _winapi and actually expose CopyFileExW instead

* fix line endings

* add tests for mode bits

* add docstring

* remove test file mode class; let's keep it for later when Istart addressing OSX fcopyfile() specific copies

* update doc to reflect new changes

* update doc

* adjust tests on win

* fix argument clinic error

* update doc

* OSX: expose copyfile(3) instead of fcopyfile(3); also expose flags arg to python

* osx / copyfile: use path_t instead of char

* do not set dst name in the OSError exception in order to remain consistent with platforms which cannot do that (e.g. linux)

* add same file test

* add test for same file

* have osx copyfile() pre-emptively check if src and dst are the same, otherwise it will return immedialtey and src file content gets deleted

* turn PermissionError into appropriate SameFileError

* expose ERROR_SHARING_VIOLATION in order to raise more appropriate SameFileError

* honour follow_symlinks arg when using CopyFileEx

* update Misc/NEWS

* expose CreateDirectoryEx mock

* change C type

* CreateDirectoryExW actual implementation

* provide specific makedirs() implementation for win

* fix typo

* skeleton for SetNamedSecurityInfo

* get security info for src path

* finally set security attrs

* add unit tests

* mimick os.makedirs() behavior and raise if dst dir exists

* set 2 paths for OSError object

* set 2 paths for OSError object

* expand windows test

* in case of exception on os.sendfile() set filename and filename2 exception attributes

* set 2 filenames (src, dst) for OSError in case copyfile() fails on OSX

* update doc

* do not use CreateDirectoryEx() in copytree() if source dir is a symlink (breaks test_copytree_symlink_dir); instead just create a plain dir and remain consistent with POSIX implementation

* use bytearray() and readinto()

* use memoryview() with bytearray()

* refactoring + introduce a new _fastcopy_binfileobj() fun

* remove CopyFileEx and other C wrappers

* remove code related to CopyFileEx

* Recognize binary files in copyfileobj()
...and use fastest _fastcopy_binfileobj() when possible

* set 1MB copy bufsize on win; also add a global _COPY_BUFSIZE variable

* use ctx manager for memoryview()

* update doc

* remove outdated doc

* remove last CopyFileEx remnants

* OSX - use fcopyfile(3) instead of copyfile(3)

...as an extra safety measure: in case src/dst are "exotic" files (non
regular or living on a network fs etc.) we better fail on open() instead
of copyfile(3) as we're not quite sure what's gonna happen in that
case.

* update doc
2018-06-12 23:04:50 +02:00
Serhiy Storchaka b21d155f57
bpo-32964: Reuse a testing implementation of the path protocol in tests. (#5930) 2018-03-02 11:53:51 +02:00
xdegaye 92c2ca7633
bpo-28759: Skip some tests on PermissionError raised by Android (GH-4350)
Access to mkfifo(), mknod() and hard link creation is controled
by SELinux on Android.
Also remove test.support.android_not_root.
2017-11-12 17:31:07 +01:00
Serhiy Storchaka d4d79bc1ff
bpo-28564: Use os.scandir() in shutil.rmtree(). (#4085)
This speeds up it to 20-40%.
2017-11-04 14:16:35 +02:00
Gregory P. Smith 529746c905 Make test_shutil test_disk_usage not depend on the cwd fs (#2597)
Make test_shutil test_disk_usage not depend on the current working directory's filesystem.
2017-07-06 17:11:27 -07:00
Jelle Zijlstra a12df7b7d4 bpo-30218: support path-like objects in shutil.unpack_archive() (GH-1367)
Thanks to Jelle Zijlstra for the patch.
2017-05-05 14:27:12 -07:00
Victor Stinner d6debb24e0 bpo-29919: Remove unused imports found by pyflakes (#137)
Make also minor PEP8 coding style fixes on modified imports.
2017-03-27 16:05:26 +02:00
Serhiy Storchaka 70d28a184c Remove unused imports. 2016-12-16 20:00:15 +02:00
Serhiy Storchaka 9bb6fe5274 Issue #14061: Misc fixes and cleanups in archiving code in shutil.
Imporoved the documentation and tests for make_archive() and unpack_archive().
Improved error handling when corresponding compress module is not available.
Brake circular dependency between shutil and tarfile modules.
2016-12-16 19:00:55 +02:00
Serhiy Storchaka 20cdffd830 Issue #14061: Misc fixes and cleanups in archiving code in shutil.
Imporoved the documentation and tests for make_archive() and unpack_archive().
Improved error handling when corresponding compress module is not available.
Brake circular dependency between shutil and tarfile modules.
2016-12-16 18:58:33 +02:00
Xavier de Gaye 3a4e989324 Issue #28759: Fix the tests that fail with PermissionError when run as
a non-root user on Android where access rights are controled by SELinux MAC.
2016-12-13 10:00:01 +01:00
Xavier de Gaye 38c8b7d292 Issue #28662: Catch PermissionError in tests when spawning a non existent program 2016-11-14 17:14:42 +01:00
Serhiy Storchaka 7fc92bb38a Issue #28488: shutil.make_archive() no longer adds entry "./" to ZIP archive. 2016-10-23 15:57:42 +03:00
Serhiy Storchaka 666de77727 Issue #28488: shutil.make_archive() no longer adds entry "./" to ZIP archive. 2016-10-23 15:55:09 +03:00
Steve Dower e58571b7ea Fixes tests broken by issue #27781. 2016-09-08 11:11:13 -07:00
Raymond Hettinger 15f44ab043 Issue #27895: Spelling fixes (Contributed by Ville Skyttä). 2016-08-30 10:47:49 -07:00
Martin Panter eb9957065a Issue #27626: Spelling fixes in docs, comments and internal names
Based on patch by Ville Skyttä.
2016-07-28 01:11:04 +00:00
Serhiy Storchaka d30829def2 Issue #26801: shutil.get_terminal_size() now handles the case of stdout is
reopened on Windows.  Added tests for fallbacks.
2016-04-24 09:58:43 +03:00
Victor Stinner 119ebb70e9 Fix shutil.get_terminal_size() error handling
Issue #26801: Fix error handling in shutil.get_terminal_size(), catch
AttributeError instead of NameError. Patch written by Emanuel Barry.

test_shutil: skip the functional test using "stty size" command if
os.get_terminal_size() is missing.
2016-04-19 22:24:56 +02:00
Serhiy Storchaka 6e3d2ba269 Issue #25624: ZipFile now always writes a ZIP_STORED header for directory
entries.  Patch by Dingyuan Wang.
2015-11-22 14:50:25 +02:00
Serhiy Storchaka 8bc792a602 Issue #25624: ZipFile now always writes a ZIP_STORED header for directory
entries.  Patch by Dingyuan Wang.
2015-11-22 14:49:58 +02:00
Serhiy Storchaka efd83feb7d Issue #25686: test_shutil no longer uses the distutils package for searching
and running external archivers.
2015-11-21 14:09:58 +02:00
Serhiy Storchaka b42de2f309 Issue #25686: test_shutil no longer uses the distutils package for searching
and running external archivers.
2015-11-21 14:09:26 +02:00
Serhiy Storchaka 2f85555531 Issue #25607: Restore old distutils logging threshold after running tests that
parse command line arguments.
2015-11-12 19:47:14 +02:00
Serhiy Storchaka 84023247b4 Issue #25607: Restore old distutils logging threshold after running tests that
parse command line arguments.
2015-11-12 19:46:23 +02:00
Serhiy Storchaka 050a143a3f Fixed tests for shutil.make_archive() with relative base_name in the case when
the path of the directory for temporary files contains symlinks.
2015-09-08 10:00:22 +03:00
Serhiy Storchaka 5558d4f2f8 Fixed tests for shutil.make_archive() with relative base_name in the case when
the path of the directory for temporary files contains symlinks.
2015-09-08 09:59:02 +03:00
Serhiy Storchaka d941d7a586 Issue #24982: shutil.make_archive() with the "zip" format now adds entries
for directories (including empty directories) in ZIP file.
Added test for comparing shutil.make_archive() with the "zip" command.
2015-09-08 05:51:00 +03:00
Serhiy Storchaka 2504cecebd Issue #24982: shutil.make_archive() with the "zip" format now adds entries
for directories (including empty directories) in ZIP file.
Added test for comparing shutil.make_archive() with the "zip" command.
2015-09-08 05:47:23 +03:00
Serhiy Storchaka ae0d7527f3 Issue #25018: Fixed testing shutil.make_archive() with relative base_name on
Windows.  The test now makes sense on non-Windows.  Added similar test for
zip format.
2015-09-07 19:59:24 +03:00
Serhiy Storchaka eba8feedfa Issue #25018: Fixed testing shutil.make_archive() with relative base_name on
Windows.  The test now makes sense on non-Windows.  Added similar test for
zip format.
2015-09-07 19:58:23 +03:00
Serhiy Storchaka 8c85a2083f Explicitly test archive name in shutil.make_archive() tests to expose failure
details in issue25018.
2015-09-07 13:56:49 +03:00
Serhiy Storchaka a091a8221e Explicitly test archive name in shutil.make_archive() tests to expose failure
details in issue25018.
2015-09-07 13:55:25 +03:00
Serhiy Storchaka 2ba39800e5 Fix, refactor and extend tests for shutil.make_archive(). 2015-09-06 18:33:52 +03:00
Serhiy Storchaka 527ef0792f Fix, refactor and extend tests for shutil.make_archive(). 2015-09-06 18:33:19 +03:00
Serhiy Storchaka 5fbadb63ef Use support.change_cwd() in tests. 2015-09-06 14:14:49 +03:00
Serhiy Storchaka 2a23adf440 Use support.change_cwd() in tests. 2015-09-06 14:13:25 +03:00
Berker Peksag 5a294d822b Issue #21697: shutil.copytree() now correctly handles symbolic links that point to directories.
Patch by Eduardo Seabra and Thomas Kluyver.
2015-07-25 14:53:48 +03:00
Berker Peksag 940d69d444 Issue #21697: shutil.copytree() now correctly handles symbolic links that point to directories.
Patch by Eduardo Seabra and Thomas Kluyver.
2015-07-25 14:55:06 +03:00
Berker Peksag 5de4a3cfc5 Issue #21775: shutil.copytree(): fix crash when copying to VFAT
An exception handler assumed that that OSError objects always have a
'winerror' attribute. That is not the case, so the exception handler
itself raised AttributeError when run on Linux (and, presumably, any
other non-Windows OS).

Patch by Greg Ward.
2014-12-10 02:51:36 +02:00
Berker Peksag 884afd92f5 Issue #21775: shutil.copytree(): fix crash when copying to VFAT
An exception handler assumed that that OSError objects always have a
'winerror' attribute. That is not the case, so the exception handler
itself raised AttributeError when run on Linux (and, presumably, any
other non-Windows OS).

Patch by Greg Ward.
2014-12-10 02:50:32 +02:00
Serhiy Storchaka 0a99b2ab61 Issue #21280: Fixed a bug in shutil.make_archive() when create an archive of
current directory in current directory.
2014-11-28 00:49:50 +02:00
Serhiy Storchaka 9a4fc19589 Issue #21280: Fixed a bug in shutil.make_archive() when create an archive of
current directory in current directory.
2014-11-28 00:48:46 +02:00
Berker Peksag 8912270629 Issue #22665: Add missing get_terminal_size and SameFileError to shutil.__all__. 2014-11-01 11:05:36 +02:00
Berker Peksag 8083cd6c3b Issue #22665: Add missing get_terminal_size and SameFileError to shutil.__all__. 2014-11-01 11:04:06 +02:00
Serhiy Storchaka 304e542ccd Issue #20056: Fixed deprecation warning about bytes path in test_shutil on
Windows.  Path by Vajrasky Kok.
2014-08-07 19:39:30 +03:00
Serhiy Storchaka 41ad77c697 Issue #20056: Fixed deprecation warning about bytes path in test_shutil on
Windows.  Path by Vajrasky Kok.
2014-08-07 19:38:37 +03:00
Serhiy Storchaka 1121377cdb Issue #5411: Added support for the "xztar" format in the shutil module. 2014-08-06 18:50:19 +03:00
R David Murray 6ffface429 #19840: Add copy_function to shutil.move.
Patch by Claudiu Popa.
2014-06-11 14:40:13 -04:00