Commit Graph

233 Commits

Author SHA1 Message Date
Cheryl Sabella 5680f6546d bpo-18283: Add support for bytes to shutil.which (GH-11818) 2019-02-13 12:25:10 +01:00
jab 9e00d9e88f bpo-20849: add dirs_exist_ok arg to shutil.copytree (patch by Josh Bronson) 2018-12-28 19:03:40 +01:00
Giampaolo Rodola 19c46a4c96
bpo-33695 shutil.copytree() + os.scandir() cache (#7874) 2018-11-12 06:18:15 -08:00
Srinivas Thatiparthy (శ్రీనివాస్ తాటిపర్తి) 2d1bc537fe bpo-35202: Remove unused imports in Lib directory. (GH-10445) 2018-11-10 07:45:28 +02:00
Zsolt Cserna 4f399be0e7 bpo-34260, shutil: fix copy2 and copystat documentation (GH-8523)
Fix the documentation of copy2, as it does not copy file ownership (user and
group), only mode, mtime, atime and flags.

The original text was confusing to developers as it suggested that this
command is the same as 'cp -p', but according to cp(1), '-p' copies file
ownership as well.

Clarify which metadata is copied by shutil.copystat in its docstring.
2018-10-23 12:09:50 +02:00
Giampaolo Rodola c7f02a9659
bpo-33671 / shutil.copyfile: use memoryview() with dynamic size on Windows (#7681)
bpo-33671
* use memoryview() with size == file size on Windows, see https://github.com/python/cpython/pull/7160#discussion_r195405230
* release intermediate (sliced) memoryview immediately
* replace "OSX" occurrences with "macOS"
* add some unittests for copyfileobj()
2018-06-19 08:27:29 -07:00
Giampaolo Rodola 4a172ccc73
bpo-33671: efficient zero-copy for shutil.copy* functions (Linux, OSX and Win) (#7160)
* have shutil.copyfileobj use sendfile() if possible

* refactoring: use ctx manager

* add test with non-regular file obj

* emulate case where file size can't be determined

* reference _copyfileobj_sendfile directly

* add test for offset() at certain position

* add test for empty file

* add test for non regular file dst

* small refactoring

* leave copyfileobj() alone in order to not introduce any incompatibility

* minor refactoring

* remove old test

* update docstring

* update docstring; rename exception class

* detect platforms which only support file to socket zero copy

* don't run test on platforms where file-to-file zero copy is not supported

* use tempfiles

* reset verbosity

* add test for smaller chunks

* add big file size test

* add comment

* update doc

* update whatsnew doc

* update doc

* catch Exception

* remove unused import

* add test case for error on second sendfile() call

* turn docstring into comment

* add one more test

* update comment

* add Misc/NEWS entry

* get rid of COPY_BUFSIZE; it belongs to another PR

* update doc

* expose posix._fcopyfile() for OSX

* merge from linux branch

* merge from linux branch

* expose fcopyfile

* arg clinic for the win implementation

* convert path type to path_t

* expose CopyFileW

* fix windows tests

* release GIL

* minor refactoring

* update doc

* update comment

* update docstrings

* rename functions

* rename test classes

* update doc

* update doc

* update docstrings and comments

* avoid do import nt|posix modules if unnecessary

* set nt|posix modules to None if not available

* micro speedup

* update description

* add doc note

* use better wording in doc

* rename function using 'fastcopy' prefix instead of 'zerocopy'

* use :ref: in rst doc

* change wording in doc

* add test to make sure sendfile() doesn't get called aymore in case it doesn't support file to file copies

* move CopyFileW in _winapi and actually expose CopyFileExW instead

* fix line endings

* add tests for mode bits

* add docstring

* remove test file mode class; let's keep it for later when Istart addressing OSX fcopyfile() specific copies

* update doc to reflect new changes

* update doc

* adjust tests on win

* fix argument clinic error

* update doc

* OSX: expose copyfile(3) instead of fcopyfile(3); also expose flags arg to python

* osx / copyfile: use path_t instead of char

* do not set dst name in the OSError exception in order to remain consistent with platforms which cannot do that (e.g. linux)

* add same file test

* add test for same file

* have osx copyfile() pre-emptively check if src and dst are the same, otherwise it will return immedialtey and src file content gets deleted

* turn PermissionError into appropriate SameFileError

* expose ERROR_SHARING_VIOLATION in order to raise more appropriate SameFileError

* honour follow_symlinks arg when using CopyFileEx

* update Misc/NEWS

* expose CreateDirectoryEx mock

* change C type

* CreateDirectoryExW actual implementation

* provide specific makedirs() implementation for win

* fix typo

* skeleton for SetNamedSecurityInfo

* get security info for src path

* finally set security attrs

* add unit tests

* mimick os.makedirs() behavior and raise if dst dir exists

* set 2 paths for OSError object

* set 2 paths for OSError object

* expand windows test

* in case of exception on os.sendfile() set filename and filename2 exception attributes

* set 2 filenames (src, dst) for OSError in case copyfile() fails on OSX

* update doc

* do not use CreateDirectoryEx() in copytree() if source dir is a symlink (breaks test_copytree_symlink_dir); instead just create a plain dir and remain consistent with POSIX implementation

* use bytearray() and readinto()

* use memoryview() with bytearray()

* refactoring + introduce a new _fastcopy_binfileobj() fun

* remove CopyFileEx and other C wrappers

* remove code related to CopyFileEx

* Recognize binary files in copyfileobj()
...and use fastest _fastcopy_binfileobj() when possible

* set 1MB copy bufsize on win; also add a global _COPY_BUFSIZE variable

* use ctx manager for memoryview()

* update doc

* remove outdated doc

* remove last CopyFileEx remnants

* OSX - use fcopyfile(3) instead of copyfile(3)

...as an extra safety measure: in case src/dst are "exotic" files (non
regular or living on a network fs etc.) we better fail on open() instead
of copyfile(3) as we're not quite sure what's gonna happen in that
case.

* update doc
2018-06-12 23:04:50 +02:00
Serhiy Storchaka d4d79bc1ff
bpo-28564: Use os.scandir() in shutil.rmtree(). (#4085)
This speeds up it to 20-40%.
2017-11-04 14:16:35 +02:00
Jelle Zijlstra a12df7b7d4 bpo-30218: support path-like objects in shutil.unpack_archive() (GH-1367)
Thanks to Jelle Zijlstra for the patch.
2017-05-05 14:27:12 -07:00
Serhiy Storchaka 5affd23e6f bpo-29762: More use "raise from None". (#569)
This hides unwanted implementation details from tracebacks.
2017-04-05 09:37:24 +03:00
Serhiy Storchaka 9bb6fe5274 Issue #14061: Misc fixes and cleanups in archiving code in shutil.
Imporoved the documentation and tests for make_archive() and unpack_archive().
Improved error handling when corresponding compress module is not available.
Brake circular dependency between shutil and tarfile modules.
2016-12-16 19:00:55 +02:00
Serhiy Storchaka 20cdffd830 Issue #14061: Misc fixes and cleanups in archiving code in shutil.
Imporoved the documentation and tests for make_archive() and unpack_archive().
Improved error handling when corresponding compress module is not available.
Brake circular dependency between shutil and tarfile modules.
2016-12-16 18:58:33 +02:00
Serhiy Storchaka 7fc92bb38a Issue #28488: shutil.make_archive() no longer adds entry "./" to ZIP archive. 2016-10-23 15:57:42 +03:00
Serhiy Storchaka 666de77727 Issue #28488: shutil.make_archive() no longer adds entry "./" to ZIP archive. 2016-10-23 15:55:09 +03:00
Martin Panter 0be894b2f6 Issue #27895: Spelling fixes (Contributed by Ville Skyttä). 2016-09-07 12:03:06 +00:00
Raymond Hettinger 15f44ab043 Issue #27895: Spelling fixes (Contributed by Ville Skyttä). 2016-08-30 10:47:49 -07:00
Martin Panter 8bde911115 Issue #27626: Merge spelling fixes from 3.5 2016-07-28 01:30:58 +00:00
Martin Panter eb9957065a Issue #27626: Spelling fixes in docs, comments and internal names
Based on patch by Ville Skyttä.
2016-07-28 01:11:04 +00:00
Serhiy Storchaka 18a8affc8e Issue #26801: shutil.get_terminal_size() now handles the case of stdout is
reopened on Windows.  Added tests for fallbacks.
2016-04-24 09:59:17 +03:00
Serhiy Storchaka d30829def2 Issue #26801: shutil.get_terminal_size() now handles the case of stdout is
reopened on Windows.  Added tests for fallbacks.
2016-04-24 09:58:43 +03:00
Victor Stinner 5bfe0da808 Merge 3.5: issue #26801 2016-04-19 22:29:11 +02:00
Victor Stinner 119ebb70e9 Fix shutil.get_terminal_size() error handling
Issue #26801: Fix error handling in shutil.get_terminal_size(), catch
AttributeError instead of NameError. Patch written by Emanuel Barry.

test_shutil: skip the functional test using "stty size" command if
os.get_terminal_size() is missing.
2016-04-19 22:24:56 +02:00
Serhiy Storchaka 899f32fe1e Issue #24982: shutil.make_archive() with the "zip" format now adds entries
for directories (including empty directories) in ZIP file.
Added test for comparing shutil.make_archive() with the "zip" command.
2015-09-08 05:53:42 +03:00
Serhiy Storchaka d941d7a586 Issue #24982: shutil.make_archive() with the "zip" format now adds entries
for directories (including empty directories) in ZIP file.
Added test for comparing shutil.make_archive() with the "zip" command.
2015-09-08 05:51:00 +03:00
Serhiy Storchaka 2504cecebd Issue #24982: shutil.make_archive() with the "zip" format now adds entries
for directories (including empty directories) in ZIP file.
Added test for comparing shutil.make_archive() with the "zip" command.
2015-09-08 05:47:23 +03:00
Raymond Hettinger 5b798abf5b Issue #24878: Add docstrings to selected namedtuples 2015-08-17 22:04:45 -07:00
Berker Peksag 940d69d444 Issue #21697: shutil.copytree() now correctly handles symbolic links that point to directories.
Patch by Eduardo Seabra and Thomas Kluyver.
2015-07-25 14:55:06 +03:00
Berker Peksag 5a294d822b Issue #21697: shutil.copytree() now correctly handles symbolic links that point to directories.
Patch by Eduardo Seabra and Thomas Kluyver.
2015-07-25 14:53:48 +03:00
Berker Peksag 5de4a3cfc5 Issue #21775: shutil.copytree(): fix crash when copying to VFAT
An exception handler assumed that that OSError objects always have a
'winerror' attribute. That is not the case, so the exception handler
itself raised AttributeError when run on Linux (and, presumably, any
other non-Windows OS).

Patch by Greg Ward.
2014-12-10 02:51:36 +02:00
Berker Peksag 884afd92f5 Issue #21775: shutil.copytree(): fix crash when copying to VFAT
An exception handler assumed that that OSError objects always have a
'winerror' attribute. That is not the case, so the exception handler
itself raised AttributeError when run on Linux (and, presumably, any
other non-Windows OS).

Patch by Greg Ward.
2014-12-10 02:50:32 +02:00
Serhiy Storchaka 0a99b2ab61 Issue #21280: Fixed a bug in shutil.make_archive() when create an archive of
current directory in current directory.
2014-11-28 00:49:50 +02:00
Serhiy Storchaka 9a4fc19589 Issue #21280: Fixed a bug in shutil.make_archive() when create an archive of
current directory in current directory.
2014-11-28 00:48:46 +02:00
Berker Peksag 8912270629 Issue #22665: Add missing get_terminal_size and SameFileError to shutil.__all__. 2014-11-01 11:05:36 +02:00
Berker Peksag 8083cd6c3b Issue #22665: Add missing get_terminal_size and SameFileError to shutil.__all__. 2014-11-01 11:04:06 +02:00
Berker Peksag 3715da5c97 Issue #21391: Use os.path.abspath in the shutil module. 2014-09-18 05:11:15 +03:00
Serhiy Storchaka 1121377cdb Issue #5411: Added support for the "xztar" format in the shutil module. 2014-08-06 18:50:19 +03:00
R David Murray 6ffface429 #19840: Add copy_function to shutil.move.
Patch by Claudiu Popa.
2014-06-11 14:40:13 -04:00
Andrew Kuchling a0934b2c1b #20744: don't try running an external 'zip' in shutil.make_archive()
Instead we'll just use the stdlib zipfile module.  Patch by Derek Chiang
2014-03-20 16:11:16 -04:00
Serhiy Storchaka 6a45021084 Issue #19856: shutil.move() failed to move a directory to other directory
on Windows if source name ends with os.altsep.
2014-02-11 10:32:41 +02:00
Serhiy Storchaka 3a308b9f37 Issue #19856: shutil.move() failed to move a directory to other directory
on Windows if source name ends with os.altsep.
2014-02-11 10:30:59 +02:00
Benjamin Peterson 85f1074438 merge 3.3 (#20102) 2014-02-02 15:31:07 -05:00
Benjamin Peterson 091d386f78 use with statement to ensure zipfile is always closed (closes #20102) 2014-02-02 15:30:22 -05:00
Victor Stinner 1d006a2c0e Backout a1a05e2724dd: shutil.which(bytes) is a new feature and my patch does not work on Windows 2013-12-16 23:39:40 +01:00
Victor Stinner fffb96ba66 Issue #18283: shutil.which() now supports bytes argument, not only text argument. 2013-12-16 22:48:48 +01:00
Ezio Melotti 9a3777e525 #18705: merge with 3.3. 2013-08-17 15:53:55 +03:00
Ezio Melotti 30b9d5d3af #18705: fix a number of typos. Patch by Févry Thibault. 2013-08-17 15:50:46 +03:00
Brett Cannon cd171c8e92 Issue #18200: Back out usage of ModuleNotFoundError (8d28d44f3a9a) 2013-07-04 17:43:24 -04:00
Brett Cannon 0a140668fa Issue #18200: Update the stdlib (except tests) to use
ModuleNotFoundError.
2013-06-13 20:57:26 -04:00
Barry Warsaw ecaefcf44e - Issue #17012: shutil.which() no longer fallbacks to the PATH environment
variable if empty path argument is specified.  Patch by Serhiy Storchaka.
2013-04-16 11:18:18 -04:00
Barry Warsaw 618738b921 - Issue #17012: shutil.which() no longer fallbacks to the PATH environment
variable if empty path argument is specified.  Patch by Serhiy Storchaka.
2013-04-16 11:05:03 -04:00
Hynek Schlawack 4cd7b9c3e9 #17076: Make copying of xattrs more permissive of missing FS support
Patch by Thomas Wouters.
2013-02-05 08:25:24 +01:00
Hynek Schlawack 0beab058dd #17076: Make copying of xattrs more permissive of missing FS support
Patch by Thomas Wouters.
2013-02-05 08:22:44 +01:00
Serhiy Storchaka e9a63600b3 Issue #16957: shutil.which() no longer searches a bare file name in the
current directory on Unix and no longer searches a relative file path with
a directory part in PATH directories.  Patch by Thomas Kluyver.
2013-01-23 10:45:33 +02:00
Serhiy Storchaka 8bea200b98 Issue #16957: shutil.which() no longer searches a bare file name in the
current directory on Unix and no longer searches a relative file path with
a directory part in PATH directories.  Patch by Thomas Kluyver.
2013-01-23 10:44:21 +02:00
Serhiy Storchaka 5b987c2e68 Issue #16993: shutil.which() now preserves the case of the path and extension
on Windows.
2013-01-21 15:01:34 +02:00
Serhiy Storchaka 014791f848 Issue #16993: shutil.which() now preserves the case of the path and extension
on Windows.
2013-01-21 15:00:27 +02:00
Andrew Svetlov 2606a6f197 Issue #16719: Get rid of WindowsError. Use OSError instead
Patch by Serhiy Storchaka.
2012-12-19 14:33:35 +02:00
Andrew Svetlov ad28c7f9da Issue #16706: get rid of os.error 2012-12-18 22:02:39 +02:00
Andrew Svetlov 3438fa496d Get rig of EnvironmentError (#16705) 2012-12-17 23:35:18 +02:00
Hynek Schlawack ae9a9e9a2d #15872: Fix 3.3 regression introduced by the new fd-based shutil.rmtree
It caused rmtree to not ignore certain errors when ignore_errors was set.

Patch by Alessandro Moura and Serhiy Storchaka.
2012-12-10 09:15:23 +01:00
Hynek Schlawack b550110f64 #15872: Fix 3.3 regression introduced by the new fd-based shutil.rmtree
It caused rmtree to not ignore certain errors when ignore_errors was set.

Patch by Alessandro Moura and Serhiy Storchaka.
2012-12-10 09:11:25 +01:00
Hynek Schlawack 4865376c44 Closes #1492704: Make shutil.copyfile() raise a distinct SameFileError
Patch by Atsuo Ishimoto.
2012-10-07 12:49:58 +02:00
Georg Brandl cc0274b90f merge with 3.2 2012-08-25 10:12:47 +02:00
Georg Brandl c8076dfa9d Fix wrong way of adding Error information in shutil.copytree. 2012-08-25 10:11:57 +02:00
Hynek Schlawack 26fe37dd3f #1492704: Backout and wait for 3.4 2012-07-19 21:41:02 +02:00
Hynek Schlawack 77d3283370 #1492704: Make shutil.copyfile() raise a distinct SameFileError
Patch by Atsuo Ishimoto.
2012-07-19 20:23:49 +02:00
Larry Hastings b40380667c Issue #15202: Consistently use the name "follow_symlinks" for
new parameters in os and shutil functions.  Patch by Serhiy Storchaka.
2012-07-15 10:57:38 -07:00
Larry Hastings ad5ae0456e - Issue #15238: shutil.copystat now copies Linux "extended attributes". 2012-07-14 17:55:11 -07:00
Hynek Schlawack d0f6e0a6e6 Check for all used fd-based functions in shutil.rmdir, closes #15218 2012-06-29 08:28:20 +02:00
Hynek Schlawack 9f558ccefd #4489: Use dir_fd in rmdir in _rmtree_safe_fd()
Now that rmdir supports dir_fd, we also use it. Attackers can't even delete
empty directories anymore.
2012-06-28 15:30:47 +02:00
Hynek Schlawack a75cd1ce73 #4489: Don't follow ever symlinks in rmtree
Also added several regression tests.
2012-06-28 12:07:29 +02:00
Hynek Schlawack 3b52778c74 #4489 Make fd based rmtree work on bytes 2012-06-25 13:27:31 +02:00
Ned Deily 5fddf866d8 Issue #14662: Prevent shutil failures on OS X when destination does not
support chflag operations.  (Patch by Hynek Schlawack)
2012-05-10 17:21:23 -07:00
Éric Araujo 89dfd5cf80 Really make bztar support in shutil conditional.
This dict entry is added a few lines after if the bzip2 module is
available, but removing this line was forgotten.
2012-02-05 13:40:08 +01:00
Florent Xicluna 5d1155c08e Closes #13258: Use callable() built-in in the standard library. 2011-10-28 14:45:05 +02:00
Éric Araujo 5fa8e7a559 Add missing name in shutil 2011-08-21 14:29:18 +02:00
Éric Araujo 43a7ee1f3b Make logging call lazy 2011-08-19 02:55:11 +02:00
Ronald Oussoren 78349b06af merge from 3.1 2011-05-06 10:57:22 +02:00
Ronald Oussoren f51738b10e Fix for issue 10684: Folders get deleted when trying to change case with shutil.move (case insensitive file systems only) 2011-05-06 10:23:04 +02:00
Nick Coghlan abf202d798 Close #11548: Correctly handle format argument in shutil.unpack_archive 2011-03-16 13:52:20 -04:00
Éric Araujo ac4e58eb61 Protect logging call against None argument (fixes #11045).
Initial patch by Kelsey Hightower.  Approved by Raymond.  A test was
non-trivial to write without calling the private function directly, so
we moved that for later.
2011-01-29 20:32:11 +00:00
Éric Araujo 4433a5fdc0 Fix wrong name in docstring and doc (#10693). Original patch by Eli Bendersky. 2010-12-15 20:26:30 +00:00
Nick Coghlan 5b0eca116a Issue #4489: Rename the feature marker for the symlink resistant rmtree and store it as a function attribute 2012-06-24 16:43:06 +10:00
Philip Jenvey 88bc0d2640 small refactor and PEP8 line width/docstring 2012-06-23 15:54:38 -07:00
Hynek Schlawack 2100b42317 #4489: Fix usage of fd-based functions to new api introduced earlier today
Also add an explicit test for safe implementation usage on supported platforms.

As a side effect, this commit adds a module-level attribute 'rmtree_is_safe'
which offers introspection whether the current rmtree implementation is safe
against symlink attacks.
2012-06-23 20:28:32 +02:00
Hynek Schlawack 67be92bed4 #4489: Add a shutil.rmtree that isn't suspectible to symlink attacks
It is used automatically on platforms supporting the necessary os.openat() and
os.unlinkat() functions. Main code by Martin von Löwis.
2012-06-23 17:58:42 +02:00
Brian Curtin dc00f1eba0 Fix #15148. Capitalize PATH, hopefully leading to less confusion 2012-06-22 22:49:12 -05:00
Brian Curtin 2193536e79 Fix #15148. Make the shutil.which docstring more thorough 2012-06-22 22:48:06 -05:00
Brian Curtin e3f3940fc3 file was the old arg name from an earlier patch. command matches the implementation 2012-06-22 21:14:34 -05:00
Alexander Belopolsky de74de5d5d Issue #15148: Fixed typos in shutil.which() docstring 2012-06-22 21:56:42 -04:00
Larry Hastings 9cf065cfdc Issue #14626: Large refactoring of functions / parameters in the os module.
Many functions now support "dir_fd" and "follow_symlinks" parameters;
some also support accepting an open file descriptor in place of of a path
string.  Added os.support_* collections as LBYL helpers.  Removed many
functions only previously seen in 3.3 alpha releases (often starting with
"f" or "l", or ending with "at").  Originally suggested by Serhiy Storchaka;
implemented by Larry Hastings.
2012-06-22 16:30:09 -07:00
Antoine Pitrou 07c24d13ed Issue #444582: shutil.which() respects relative paths. 2012-06-22 23:33:05 +02:00
Brian Curtin 849349de05 whitespace normalize 2012-06-22 16:03:06 -05:00
Brian Curtin c57a34577c Fix #444582. Add shutil.which function for finding programs on the system path. 2012-06-22 16:00:30 -05:00
Brian Curtin 0d0a1dedbc Fix #14772: Return the destination from some shutil functions. 2012-06-18 18:41:07 -05:00
Antoine Pitrou 424246fbf3 Issue #14082: shutil.copy2() now copies extended attributes, if possible.
Patch by Hynek Schlawack.
2012-05-12 19:02:01 +02:00
Ned Deily baf75713c7 Issue #14662: Prevent shutil failures on OS X when destination does not
support chflag operations.  (Patch by Hynek Schlawack)
2012-05-10 17:05:19 -07:00
Larry Hastings b14544807e Issue #14127: Fix no-op stub for platforms that lack some "os" functions. 2012-05-03 12:56:44 -07:00
Larry Hastings 76ad59b7e8 Issue #14127: Add ns= parameter to utime, futimes, and lutimes.
Removed futimens as it is now redundant.
Changed shutil.copystat to use st_atime_ns and st_mtime_ns from os.stat
and ns= parameter to utime--it once again preserves exact metadata on Linux!
2012-05-03 00:30:07 -07:00
Sandro Tosi f8ae4faf93 Issue #14650: fix typo in shutil.disk_usage() docstring; patch by Hobson Lane 2012-04-23 20:07:15 +02:00