cpython/Doc/library/shutil.rst

807 lines
31 KiB
ReStructuredText
Raw Normal View History

2007-08-15 11:28:22 -03:00
:mod:`shutil` --- High-level file operations
============================================
.. module:: shutil
:synopsis: High-level file operations, including copying.
2007-08-15 11:28:22 -03:00
.. sectionauthor:: Fred L. Drake, Jr. <fdrake@acm.org>
.. partly based on the docstrings
2007-08-15 11:28:22 -03:00
**Source code:** :source:`Lib/shutil.py`
2007-08-15 11:28:22 -03:00
.. index::
single: file; copying
single: copying files
2011-01-10 15:54:11 -04:00
--------------
2007-08-15 11:28:22 -03:00
The :mod:`shutil` module offers a number of high-level operations on files and
collections of files. In particular, functions are provided which support file
Merged revisions 58742-58816 via svnmerge from svn+ssh://pythondev@svn.python.org/python/trunk ........ r58745 | georg.brandl | 2007-11-01 10:19:33 -0700 (Thu, 01 Nov 2007) | 2 lines #1364: os.lstat is available on Windows too, as an alias to os.stat. ........ r58750 | christian.heimes | 2007-11-01 12:48:10 -0700 (Thu, 01 Nov 2007) | 1 line Backport of import tests for bug http://bugs.python.org/issue1293 and bug http://bugs.python.org/issue1342 ........ r58751 | christian.heimes | 2007-11-01 13:11:06 -0700 (Thu, 01 Nov 2007) | 1 line Removed non ASCII text from test as requested by Guido. Sorry :/ ........ r58753 | georg.brandl | 2007-11-01 13:37:02 -0700 (Thu, 01 Nov 2007) | 2 lines Fix markup glitch. ........ r58757 | gregory.p.smith | 2007-11-01 14:08:14 -0700 (Thu, 01 Nov 2007) | 4 lines Fix bug introduced in revision 58385. Database keys could no longer have NULL bytes in them. Replace the errant strdup with a malloc+memcpy. Adds a unit test for the correct behavior. ........ r58758 | gregory.p.smith | 2007-11-01 14:15:36 -0700 (Thu, 01 Nov 2007) | 3 lines Undo revision 58533 58534 fixes. Those were a workaround for a problem introduced by 58385. ........ r58759 | gregory.p.smith | 2007-11-01 14:17:47 -0700 (Thu, 01 Nov 2007) | 2 lines false "fix" undone as correct problem was found and fixed. ........ r58765 | mark.summerfield | 2007-11-02 01:24:59 -0700 (Fri, 02 Nov 2007) | 3 lines Added more file-handling related cross-references. ........ r58766 | nick.coghlan | 2007-11-02 03:09:12 -0700 (Fri, 02 Nov 2007) | 1 line Fix for bug 1705170 - contextmanager swallowing StopIteration (2.5 backport candidate) ........ r58784 | thomas.heller | 2007-11-02 12:10:24 -0700 (Fri, 02 Nov 2007) | 4 lines Issue #1292: On alpha, arm, ppc, and s390 linux systems the --with-system-ffi configure option defaults to "yes" because the bundled libffi sources are too old. ........ r58785 | thomas.heller | 2007-11-02 12:11:23 -0700 (Fri, 02 Nov 2007) | 1 line Enable the full ctypes c_longdouble tests again. ........ r58796 | georg.brandl | 2007-11-02 13:06:17 -0700 (Fri, 02 Nov 2007) | 4 lines Make "hashable" a glossary entry and clarify docs on __cmp__, __eq__ and __hash__. I hope the concept of hashability is better understandable now. Thanks to Tim Hatch for pointing out the flaws here. ........
2007-11-02 20:46:40 -03:00
copying and removal. For operations on individual files, see also the
:mod:`os` module.
2007-08-15 11:28:22 -03:00
.. warning::
2012-02-13 11:30:47 -04:00
Even the higher-level file copying functions (:func:`shutil.copy`,
:func:`shutil.copy2`) cannot copy all file metadata.
2009-01-03 17:18:54 -04:00
On POSIX platforms, this means that file owner and group are lost as well
as ACLs. On Mac OS, the resource fork and other metadata are not used.
This means that resources will be lost and file type and creator codes will
not be correct. On Windows, file owners, ACLs and alternate data streams
are not copied.
2007-08-15 11:28:22 -03:00
.. _file-operations:
Directory and files operations
------------------------------
2007-08-15 11:28:22 -03:00
.. function:: copyfileobj(fsrc, fdst[, length])
Copy the contents of the file-like object *fsrc* to the file-like object *fdst*.
The integer *length*, if given, is the buffer size. In particular, a negative
*length* value means to copy the data without looping over the source data in
chunks; by default the data is read in chunks to avoid uncontrolled memory
consumption. Note that if the current file position of the *fsrc* object is not
0, only the contents from the current file position to the end of the file will
be copied.
.. function:: copyfile(src, dst, *, follow_symlinks=True)
Merged revisions 62386-62387,62389-62393,62396,62400-62402,62407,62409-62410,62412-62414,62418-62419 via svnmerge from svn+ssh://pythondev@svn.python.org/python/trunk ........ r62386 | christian.heimes | 2008-04-19 04:23:57 +0200 (Sat, 19 Apr 2008) | 2 lines Added kill, terminate and send_signal to subprocess.Popen The bits and pieces for the Windows side were already in place. The POSIX side is trivial (as usual) and uses os.kill(). ........ r62387 | georg.brandl | 2008-04-19 10:23:59 +0200 (Sat, 19 Apr 2008) | 2 lines Fix-up docs for revision 62386. ........ r62389 | georg.brandl | 2008-04-19 18:57:43 +0200 (Sat, 19 Apr 2008) | 2 lines #2369: clarify that copyfile() doesn't take a target directory. ........ r62390 | georg.brandl | 2008-04-19 18:58:28 +0200 (Sat, 19 Apr 2008) | 2 lines #2634: clarify meaning of env parameter to spawn/exec*e. ........ r62391 | georg.brandl | 2008-04-19 18:58:49 +0200 (Sat, 19 Apr 2008) | 2 lines #2633: clarify meaning of env parameter. ........ r62392 | georg.brandl | 2008-04-19 18:59:16 +0200 (Sat, 19 Apr 2008) | 2 lines #2631: clarify IMPORT_NAME semantics. ........ r62393 | georg.brandl | 2008-04-19 19:00:14 +0200 (Sat, 19 Apr 2008) | 2 lines :func: et al. should *not* include the parens. ........ r62396 | mark.dickinson | 2008-04-19 20:51:48 +0200 (Sat, 19 Apr 2008) | 5 lines Additional tests for math.pow, and extra special-case handling code in math.pow, in the hope of making all tests pass on the alpha Tru64 buildbot. ........ r62400 | mark.dickinson | 2008-04-19 21:41:52 +0200 (Sat, 19 Apr 2008) | 3 lines Additional special-case handling for math.pow. Windows/VS2008 doesn't like (-1)**(+-inf). ........ r62401 | benjamin.peterson | 2008-04-19 21:47:34 +0200 (Sat, 19 Apr 2008) | 2 lines Complete documentation for errors argument of io's open and TextIOWrapper ........ r62402 | mark.dickinson | 2008-04-19 22:31:16 +0200 (Sat, 19 Apr 2008) | 2 lines Document updates to math and cmath modules. ........ r62407 | georg.brandl | 2008-04-19 23:28:38 +0200 (Sat, 19 Apr 2008) | 2 lines Update template for newest Sphinx. ........ r62409 | mark.dickinson | 2008-04-19 23:35:35 +0200 (Sat, 19 Apr 2008) | 5 lines Correct documentation for math.pow; 0**nan is nan, not 0. (But nan**0 and 1**nan are 1.) Also fix minor typo: 'quite NaN' -> 'quiet NaN' ........ r62410 | mark.dickinson | 2008-04-19 23:49:22 +0200 (Sat, 19 Apr 2008) | 4 lines Move asinh documentation to the proper place. Remove meaningless 'in radians' from inverse hyperbolic functions. ........ r62412 | mark.dickinson | 2008-04-20 03:22:30 +0200 (Sun, 20 Apr 2008) | 5 lines Report additional diagnostic information in test_math, to help track down debian-alpha buildbot failure. ........ r62413 | mark.dickinson | 2008-04-20 03:39:24 +0200 (Sun, 20 Apr 2008) | 3 lines FreeBSD doesn't follow C99 for modf(inf); so add explicit special-value handling to math.modf code. ........ r62414 | mark.dickinson | 2008-04-20 06:13:13 +0200 (Sun, 20 Apr 2008) | 5 lines Yet more explicit special case handling to make math.pow behave on alpha Tru64. All IEEE 754 special values are now handled directly; only the finite**finite case is handled by libm. ........ r62418 | mark.dickinson | 2008-04-20 18:13:17 +0200 (Sun, 20 Apr 2008) | 7 lines Issue 2662: Initialize special value tables dynamically (i.e. when cmath module is loaded) instead of statically. This fixes compile-time problems on platforms where HUGE_VAL is an extern variable rather than a constant. Thanks Hirokazu Yamamoto for the patch. ........ r62419 | andrew.kuchling | 2008-04-20 18:54:02 +0200 (Sun, 20 Apr 2008) | 1 line Move description of math module changes; various edits to description of cmath changes ........
2008-04-20 18:01:16 -03:00
2012-02-13 11:30:47 -04:00
Copy the contents (no metadata) of the file named *src* to a file named
bpo-33671: efficient zero-copy for shutil.copy* functions (Linux, OSX and Win) (#7160) * have shutil.copyfileobj use sendfile() if possible * refactoring: use ctx manager * add test with non-regular file obj * emulate case where file size can't be determined * reference _copyfileobj_sendfile directly * add test for offset() at certain position * add test for empty file * add test for non regular file dst * small refactoring * leave copyfileobj() alone in order to not introduce any incompatibility * minor refactoring * remove old test * update docstring * update docstring; rename exception class * detect platforms which only support file to socket zero copy * don't run test on platforms where file-to-file zero copy is not supported * use tempfiles * reset verbosity * add test for smaller chunks * add big file size test * add comment * update doc * update whatsnew doc * update doc * catch Exception * remove unused import * add test case for error on second sendfile() call * turn docstring into comment * add one more test * update comment * add Misc/NEWS entry * get rid of COPY_BUFSIZE; it belongs to another PR * update doc * expose posix._fcopyfile() for OSX * merge from linux branch * merge from linux branch * expose fcopyfile * arg clinic for the win implementation * convert path type to path_t * expose CopyFileW * fix windows tests * release GIL * minor refactoring * update doc * update comment * update docstrings * rename functions * rename test classes * update doc * update doc * update docstrings and comments * avoid do import nt|posix modules if unnecessary * set nt|posix modules to None if not available * micro speedup * update description * add doc note * use better wording in doc * rename function using 'fastcopy' prefix instead of 'zerocopy' * use :ref: in rst doc * change wording in doc * add test to make sure sendfile() doesn't get called aymore in case it doesn't support file to file copies * move CopyFileW in _winapi and actually expose CopyFileExW instead * fix line endings * add tests for mode bits * add docstring * remove test file mode class; let's keep it for later when Istart addressing OSX fcopyfile() specific copies * update doc to reflect new changes * update doc * adjust tests on win * fix argument clinic error * update doc * OSX: expose copyfile(3) instead of fcopyfile(3); also expose flags arg to python * osx / copyfile: use path_t instead of char * do not set dst name in the OSError exception in order to remain consistent with platforms which cannot do that (e.g. linux) * add same file test * add test for same file * have osx copyfile() pre-emptively check if src and dst are the same, otherwise it will return immedialtey and src file content gets deleted * turn PermissionError into appropriate SameFileError * expose ERROR_SHARING_VIOLATION in order to raise more appropriate SameFileError * honour follow_symlinks arg when using CopyFileEx * update Misc/NEWS * expose CreateDirectoryEx mock * change C type * CreateDirectoryExW actual implementation * provide specific makedirs() implementation for win * fix typo * skeleton for SetNamedSecurityInfo * get security info for src path * finally set security attrs * add unit tests * mimick os.makedirs() behavior and raise if dst dir exists * set 2 paths for OSError object * set 2 paths for OSError object * expand windows test * in case of exception on os.sendfile() set filename and filename2 exception attributes * set 2 filenames (src, dst) for OSError in case copyfile() fails on OSX * update doc * do not use CreateDirectoryEx() in copytree() if source dir is a symlink (breaks test_copytree_symlink_dir); instead just create a plain dir and remain consistent with POSIX implementation * use bytearray() and readinto() * use memoryview() with bytearray() * refactoring + introduce a new _fastcopy_binfileobj() fun * remove CopyFileEx and other C wrappers * remove code related to CopyFileEx * Recognize binary files in copyfileobj() ...and use fastest _fastcopy_binfileobj() when possible * set 1MB copy bufsize on win; also add a global _COPY_BUFSIZE variable * use ctx manager for memoryview() * update doc * remove outdated doc * remove last CopyFileEx remnants * OSX - use fcopyfile(3) instead of copyfile(3) ...as an extra safety measure: in case src/dst are "exotic" files (non regular or living on a network fs etc.) we better fail on open() instead of copyfile(3) as we're not quite sure what's gonna happen in that case. * update doc
2018-06-12 18:04:50 -03:00
*dst* and return *dst* in the most efficient way possible.
*src* and *dst* are path-like objects or path names given as strings.
bpo-33671: efficient zero-copy for shutil.copy* functions (Linux, OSX and Win) (#7160) * have shutil.copyfileobj use sendfile() if possible * refactoring: use ctx manager * add test with non-regular file obj * emulate case where file size can't be determined * reference _copyfileobj_sendfile directly * add test for offset() at certain position * add test for empty file * add test for non regular file dst * small refactoring * leave copyfileobj() alone in order to not introduce any incompatibility * minor refactoring * remove old test * update docstring * update docstring; rename exception class * detect platforms which only support file to socket zero copy * don't run test on platforms where file-to-file zero copy is not supported * use tempfiles * reset verbosity * add test for smaller chunks * add big file size test * add comment * update doc * update whatsnew doc * update doc * catch Exception * remove unused import * add test case for error on second sendfile() call * turn docstring into comment * add one more test * update comment * add Misc/NEWS entry * get rid of COPY_BUFSIZE; it belongs to another PR * update doc * expose posix._fcopyfile() for OSX * merge from linux branch * merge from linux branch * expose fcopyfile * arg clinic for the win implementation * convert path type to path_t * expose CopyFileW * fix windows tests * release GIL * minor refactoring * update doc * update comment * update docstrings * rename functions * rename test classes * update doc * update doc * update docstrings and comments * avoid do import nt|posix modules if unnecessary * set nt|posix modules to None if not available * micro speedup * update description * add doc note * use better wording in doc * rename function using 'fastcopy' prefix instead of 'zerocopy' * use :ref: in rst doc * change wording in doc * add test to make sure sendfile() doesn't get called aymore in case it doesn't support file to file copies * move CopyFileW in _winapi and actually expose CopyFileExW instead * fix line endings * add tests for mode bits * add docstring * remove test file mode class; let's keep it for later when Istart addressing OSX fcopyfile() specific copies * update doc to reflect new changes * update doc * adjust tests on win * fix argument clinic error * update doc * OSX: expose copyfile(3) instead of fcopyfile(3); also expose flags arg to python * osx / copyfile: use path_t instead of char * do not set dst name in the OSError exception in order to remain consistent with platforms which cannot do that (e.g. linux) * add same file test * add test for same file * have osx copyfile() pre-emptively check if src and dst are the same, otherwise it will return immedialtey and src file content gets deleted * turn PermissionError into appropriate SameFileError * expose ERROR_SHARING_VIOLATION in order to raise more appropriate SameFileError * honour follow_symlinks arg when using CopyFileEx * update Misc/NEWS * expose CreateDirectoryEx mock * change C type * CreateDirectoryExW actual implementation * provide specific makedirs() implementation for win * fix typo * skeleton for SetNamedSecurityInfo * get security info for src path * finally set security attrs * add unit tests * mimick os.makedirs() behavior and raise if dst dir exists * set 2 paths for OSError object * set 2 paths for OSError object * expand windows test * in case of exception on os.sendfile() set filename and filename2 exception attributes * set 2 filenames (src, dst) for OSError in case copyfile() fails on OSX * update doc * do not use CreateDirectoryEx() in copytree() if source dir is a symlink (breaks test_copytree_symlink_dir); instead just create a plain dir and remain consistent with POSIX implementation * use bytearray() and readinto() * use memoryview() with bytearray() * refactoring + introduce a new _fastcopy_binfileobj() fun * remove CopyFileEx and other C wrappers * remove code related to CopyFileEx * Recognize binary files in copyfileobj() ...and use fastest _fastcopy_binfileobj() when possible * set 1MB copy bufsize on win; also add a global _COPY_BUFSIZE variable * use ctx manager for memoryview() * update doc * remove outdated doc * remove last CopyFileEx remnants * OSX - use fcopyfile(3) instead of copyfile(3) ...as an extra safety measure: in case src/dst are "exotic" files (non regular or living on a network fs etc.) we better fail on open() instead of copyfile(3) as we're not quite sure what's gonna happen in that case. * update doc
2018-06-12 18:04:50 -03:00
*dst* must be the complete target file name; look at :func:`~shutil.copy`
for a copy that accepts a target directory path. If *src* and *dst*
specify the same file, :exc:`SameFileError` is raised.
2012-02-13 11:35:44 -04:00
The destination location must be writable; otherwise, an :exc:`OSError`
exception will be raised. If *dst* already exists, it will be replaced.
Special files such as character or block devices and pipes cannot be
copied with this function.
Merged revisions 62386-62387,62389-62393,62396,62400-62402,62407,62409-62410,62412-62414,62418-62419 via svnmerge from svn+ssh://pythondev@svn.python.org/python/trunk ........ r62386 | christian.heimes | 2008-04-19 04:23:57 +0200 (Sat, 19 Apr 2008) | 2 lines Added kill, terminate and send_signal to subprocess.Popen The bits and pieces for the Windows side were already in place. The POSIX side is trivial (as usual) and uses os.kill(). ........ r62387 | georg.brandl | 2008-04-19 10:23:59 +0200 (Sat, 19 Apr 2008) | 2 lines Fix-up docs for revision 62386. ........ r62389 | georg.brandl | 2008-04-19 18:57:43 +0200 (Sat, 19 Apr 2008) | 2 lines #2369: clarify that copyfile() doesn't take a target directory. ........ r62390 | georg.brandl | 2008-04-19 18:58:28 +0200 (Sat, 19 Apr 2008) | 2 lines #2634: clarify meaning of env parameter to spawn/exec*e. ........ r62391 | georg.brandl | 2008-04-19 18:58:49 +0200 (Sat, 19 Apr 2008) | 2 lines #2633: clarify meaning of env parameter. ........ r62392 | georg.brandl | 2008-04-19 18:59:16 +0200 (Sat, 19 Apr 2008) | 2 lines #2631: clarify IMPORT_NAME semantics. ........ r62393 | georg.brandl | 2008-04-19 19:00:14 +0200 (Sat, 19 Apr 2008) | 2 lines :func: et al. should *not* include the parens. ........ r62396 | mark.dickinson | 2008-04-19 20:51:48 +0200 (Sat, 19 Apr 2008) | 5 lines Additional tests for math.pow, and extra special-case handling code in math.pow, in the hope of making all tests pass on the alpha Tru64 buildbot. ........ r62400 | mark.dickinson | 2008-04-19 21:41:52 +0200 (Sat, 19 Apr 2008) | 3 lines Additional special-case handling for math.pow. Windows/VS2008 doesn't like (-1)**(+-inf). ........ r62401 | benjamin.peterson | 2008-04-19 21:47:34 +0200 (Sat, 19 Apr 2008) | 2 lines Complete documentation for errors argument of io's open and TextIOWrapper ........ r62402 | mark.dickinson | 2008-04-19 22:31:16 +0200 (Sat, 19 Apr 2008) | 2 lines Document updates to math and cmath modules. ........ r62407 | georg.brandl | 2008-04-19 23:28:38 +0200 (Sat, 19 Apr 2008) | 2 lines Update template for newest Sphinx. ........ r62409 | mark.dickinson | 2008-04-19 23:35:35 +0200 (Sat, 19 Apr 2008) | 5 lines Correct documentation for math.pow; 0**nan is nan, not 0. (But nan**0 and 1**nan are 1.) Also fix minor typo: 'quite NaN' -> 'quiet NaN' ........ r62410 | mark.dickinson | 2008-04-19 23:49:22 +0200 (Sat, 19 Apr 2008) | 4 lines Move asinh documentation to the proper place. Remove meaningless 'in radians' from inverse hyperbolic functions. ........ r62412 | mark.dickinson | 2008-04-20 03:22:30 +0200 (Sun, 20 Apr 2008) | 5 lines Report additional diagnostic information in test_math, to help track down debian-alpha buildbot failure. ........ r62413 | mark.dickinson | 2008-04-20 03:39:24 +0200 (Sun, 20 Apr 2008) | 3 lines FreeBSD doesn't follow C99 for modf(inf); so add explicit special-value handling to math.modf code. ........ r62414 | mark.dickinson | 2008-04-20 06:13:13 +0200 (Sun, 20 Apr 2008) | 5 lines Yet more explicit special case handling to make math.pow behave on alpha Tru64. All IEEE 754 special values are now handled directly; only the finite**finite case is handled by libm. ........ r62418 | mark.dickinson | 2008-04-20 18:13:17 +0200 (Sun, 20 Apr 2008) | 7 lines Issue 2662: Initialize special value tables dynamically (i.e. when cmath module is loaded) instead of statically. This fixes compile-time problems on platforms where HUGE_VAL is an extern variable rather than a constant. Thanks Hirokazu Yamamoto for the patch. ........ r62419 | andrew.kuchling | 2008-04-20 18:54:02 +0200 (Sun, 20 Apr 2008) | 1 line Move description of math module changes; various edits to description of cmath changes ........
2008-04-20 18:01:16 -03:00
If *follow_symlinks* is false and *src* is a symbolic link,
a new symbolic link will be created instead of copying the
file *src* points to.
.. audit-event:: shutil.copyfile src,dst shutil.copyfile
2011-10-12 15:10:51 -03:00
.. versionchanged:: 3.3
:exc:`IOError` used to be raised instead of :exc:`OSError`.
Added *follow_symlinks* argument.
Now returns *dst*.
2011-10-12 15:10:51 -03:00
.. versionchanged:: 3.4
Raise :exc:`SameFileError` instead of :exc:`Error`. Since the former is
a subclass of the latter, this change is backward compatible.
bpo-33671: efficient zero-copy for shutil.copy* functions (Linux, OSX and Win) (#7160) * have shutil.copyfileobj use sendfile() if possible * refactoring: use ctx manager * add test with non-regular file obj * emulate case where file size can't be determined * reference _copyfileobj_sendfile directly * add test for offset() at certain position * add test for empty file * add test for non regular file dst * small refactoring * leave copyfileobj() alone in order to not introduce any incompatibility * minor refactoring * remove old test * update docstring * update docstring; rename exception class * detect platforms which only support file to socket zero copy * don't run test on platforms where file-to-file zero copy is not supported * use tempfiles * reset verbosity * add test for smaller chunks * add big file size test * add comment * update doc * update whatsnew doc * update doc * catch Exception * remove unused import * add test case for error on second sendfile() call * turn docstring into comment * add one more test * update comment * add Misc/NEWS entry * get rid of COPY_BUFSIZE; it belongs to another PR * update doc * expose posix._fcopyfile() for OSX * merge from linux branch * merge from linux branch * expose fcopyfile * arg clinic for the win implementation * convert path type to path_t * expose CopyFileW * fix windows tests * release GIL * minor refactoring * update doc * update comment * update docstrings * rename functions * rename test classes * update doc * update doc * update docstrings and comments * avoid do import nt|posix modules if unnecessary * set nt|posix modules to None if not available * micro speedup * update description * add doc note * use better wording in doc * rename function using 'fastcopy' prefix instead of 'zerocopy' * use :ref: in rst doc * change wording in doc * add test to make sure sendfile() doesn't get called aymore in case it doesn't support file to file copies * move CopyFileW in _winapi and actually expose CopyFileExW instead * fix line endings * add tests for mode bits * add docstring * remove test file mode class; let's keep it for later when Istart addressing OSX fcopyfile() specific copies * update doc to reflect new changes * update doc * adjust tests on win * fix argument clinic error * update doc * OSX: expose copyfile(3) instead of fcopyfile(3); also expose flags arg to python * osx / copyfile: use path_t instead of char * do not set dst name in the OSError exception in order to remain consistent with platforms which cannot do that (e.g. linux) * add same file test * add test for same file * have osx copyfile() pre-emptively check if src and dst are the same, otherwise it will return immedialtey and src file content gets deleted * turn PermissionError into appropriate SameFileError * expose ERROR_SHARING_VIOLATION in order to raise more appropriate SameFileError * honour follow_symlinks arg when using CopyFileEx * update Misc/NEWS * expose CreateDirectoryEx mock * change C type * CreateDirectoryExW actual implementation * provide specific makedirs() implementation for win * fix typo * skeleton for SetNamedSecurityInfo * get security info for src path * finally set security attrs * add unit tests * mimick os.makedirs() behavior and raise if dst dir exists * set 2 paths for OSError object * set 2 paths for OSError object * expand windows test * in case of exception on os.sendfile() set filename and filename2 exception attributes * set 2 filenames (src, dst) for OSError in case copyfile() fails on OSX * update doc * do not use CreateDirectoryEx() in copytree() if source dir is a symlink (breaks test_copytree_symlink_dir); instead just create a plain dir and remain consistent with POSIX implementation * use bytearray() and readinto() * use memoryview() with bytearray() * refactoring + introduce a new _fastcopy_binfileobj() fun * remove CopyFileEx and other C wrappers * remove code related to CopyFileEx * Recognize binary files in copyfileobj() ...and use fastest _fastcopy_binfileobj() when possible * set 1MB copy bufsize on win; also add a global _COPY_BUFSIZE variable * use ctx manager for memoryview() * update doc * remove outdated doc * remove last CopyFileEx remnants * OSX - use fcopyfile(3) instead of copyfile(3) ...as an extra safety measure: in case src/dst are "exotic" files (non regular or living on a network fs etc.) we better fail on open() instead of copyfile(3) as we're not quite sure what's gonna happen in that case. * update doc
2018-06-12 18:04:50 -03:00
.. versionchanged:: 3.8
Platform-specific fast-copy syscalls may be used internally in order to
copy the file more efficiently. See
:ref:`shutil-platform-dependent-efficient-copy-operations` section.
.. exception:: SameFileError
This exception is raised if source and destination in :func:`copyfile`
are the same file.
.. versionadded:: 3.4
.. function:: copymode(src, dst, *, follow_symlinks=True)
2007-08-15 11:28:22 -03:00
Copy the permission bits from *src* to *dst*. The file contents, owner, and
group are unaffected. *src* and *dst* are path-like objects or path names
given as strings.
If *follow_symlinks* is false, and both *src* and *dst* are symbolic links,
:func:`copymode` will attempt to modify the mode of *dst* itself (rather
than the file it points to). This functionality is not available on every
platform; please see :func:`copystat` for more information. If
:func:`copymode` cannot modify symbolic links on the local platform, and it
is asked to do so, it will do nothing and return.
2007-08-15 11:28:22 -03:00
.. audit-event:: shutil.copymode src,dst shutil.copymode
.. versionchanged:: 3.3
Added *follow_symlinks* argument.
2007-08-15 11:28:22 -03:00
.. function:: copystat(src, dst, *, follow_symlinks=True)
2007-08-15 11:28:22 -03:00
Copy the permission bits, last access time, last modification time, and
flags from *src* to *dst*. On Linux, :func:`copystat` also copies the
"extended attributes" where possible. The file contents, owner, and
group are unaffected. *src* and *dst* are path-like objects or path
names given as strings.
If *follow_symlinks* is false, and *src* and *dst* both
refer to symbolic links, :func:`copystat` will operate on
the symbolic links themselves rather than the files the
symbolic links refer to—reading the information from the
*src* symbolic link, and writing the information to the
*dst* symbolic link.
.. note::
Not all platforms provide the ability to examine and
modify symbolic links. Python itself can tell you what
functionality is locally available.
* If ``os.chmod in os.supports_follow_symlinks`` is
``True``, :func:`copystat` can modify the permission
bits of a symbolic link.
* If ``os.utime in os.supports_follow_symlinks`` is
``True``, :func:`copystat` can modify the last access
and modification times of a symbolic link.
* If ``os.chflags in os.supports_follow_symlinks`` is
``True``, :func:`copystat` can modify the flags of
a symbolic link. (``os.chflags`` is not available on
all platforms.)
On platforms where some or all of this functionality
is unavailable, when asked to modify a symbolic link,
:func:`copystat` will copy everything it can.
:func:`copystat` never returns failure.
Please see :data:`os.supports_follow_symlinks`
for more information.
2007-08-15 11:28:22 -03:00
.. audit-event:: shutil.copystat src,dst shutil.copystat
.. versionchanged:: 3.3
Added *follow_symlinks* argument and support for Linux extended attributes.
2007-08-15 11:28:22 -03:00
.. function:: copy(src, dst, *, follow_symlinks=True)
2007-08-15 11:28:22 -03:00
Copies the file *src* to the file or directory *dst*. *src* and *dst*
should be strings. If *dst* specifies a directory, the file will be
copied into *dst* using the base filename from *src*. Returns the
path to the newly created file.
If *follow_symlinks* is false, and *src* is a symbolic link,
*dst* will be created as a symbolic link. If *follow_symlinks*
is true and *src* is a symbolic link, *dst* will be a copy of
the file *src* refers to.
:func:`~shutil.copy` copies the file data and the file's permission
mode (see :func:`os.chmod`). Other metadata, like the
file's creation and modification times, is not preserved.
To preserve all file metadata from the original, use
:func:`~shutil.copy2` instead.
2007-08-15 11:28:22 -03:00
.. audit-event:: shutil.copyfile src,dst shutil.copy
.. audit-event:: shutil.copymode src,dst shutil.copy
.. versionchanged:: 3.3
Added *follow_symlinks* argument.
Now returns path to the newly created file.
2007-08-15 11:28:22 -03:00
bpo-33671: efficient zero-copy for shutil.copy* functions (Linux, OSX and Win) (#7160) * have shutil.copyfileobj use sendfile() if possible * refactoring: use ctx manager * add test with non-regular file obj * emulate case where file size can't be determined * reference _copyfileobj_sendfile directly * add test for offset() at certain position * add test for empty file * add test for non regular file dst * small refactoring * leave copyfileobj() alone in order to not introduce any incompatibility * minor refactoring * remove old test * update docstring * update docstring; rename exception class * detect platforms which only support file to socket zero copy * don't run test on platforms where file-to-file zero copy is not supported * use tempfiles * reset verbosity * add test for smaller chunks * add big file size test * add comment * update doc * update whatsnew doc * update doc * catch Exception * remove unused import * add test case for error on second sendfile() call * turn docstring into comment * add one more test * update comment * add Misc/NEWS entry * get rid of COPY_BUFSIZE; it belongs to another PR * update doc * expose posix._fcopyfile() for OSX * merge from linux branch * merge from linux branch * expose fcopyfile * arg clinic for the win implementation * convert path type to path_t * expose CopyFileW * fix windows tests * release GIL * minor refactoring * update doc * update comment * update docstrings * rename functions * rename test classes * update doc * update doc * update docstrings and comments * avoid do import nt|posix modules if unnecessary * set nt|posix modules to None if not available * micro speedup * update description * add doc note * use better wording in doc * rename function using 'fastcopy' prefix instead of 'zerocopy' * use :ref: in rst doc * change wording in doc * add test to make sure sendfile() doesn't get called aymore in case it doesn't support file to file copies * move CopyFileW in _winapi and actually expose CopyFileExW instead * fix line endings * add tests for mode bits * add docstring * remove test file mode class; let's keep it for later when Istart addressing OSX fcopyfile() specific copies * update doc to reflect new changes * update doc * adjust tests on win * fix argument clinic error * update doc * OSX: expose copyfile(3) instead of fcopyfile(3); also expose flags arg to python * osx / copyfile: use path_t instead of char * do not set dst name in the OSError exception in order to remain consistent with platforms which cannot do that (e.g. linux) * add same file test * add test for same file * have osx copyfile() pre-emptively check if src and dst are the same, otherwise it will return immedialtey and src file content gets deleted * turn PermissionError into appropriate SameFileError * expose ERROR_SHARING_VIOLATION in order to raise more appropriate SameFileError * honour follow_symlinks arg when using CopyFileEx * update Misc/NEWS * expose CreateDirectoryEx mock * change C type * CreateDirectoryExW actual implementation * provide specific makedirs() implementation for win * fix typo * skeleton for SetNamedSecurityInfo * get security info for src path * finally set security attrs * add unit tests * mimick os.makedirs() behavior and raise if dst dir exists * set 2 paths for OSError object * set 2 paths for OSError object * expand windows test * in case of exception on os.sendfile() set filename and filename2 exception attributes * set 2 filenames (src, dst) for OSError in case copyfile() fails on OSX * update doc * do not use CreateDirectoryEx() in copytree() if source dir is a symlink (breaks test_copytree_symlink_dir); instead just create a plain dir and remain consistent with POSIX implementation * use bytearray() and readinto() * use memoryview() with bytearray() * refactoring + introduce a new _fastcopy_binfileobj() fun * remove CopyFileEx and other C wrappers * remove code related to CopyFileEx * Recognize binary files in copyfileobj() ...and use fastest _fastcopy_binfileobj() when possible * set 1MB copy bufsize on win; also add a global _COPY_BUFSIZE variable * use ctx manager for memoryview() * update doc * remove outdated doc * remove last CopyFileEx remnants * OSX - use fcopyfile(3) instead of copyfile(3) ...as an extra safety measure: in case src/dst are "exotic" files (non regular or living on a network fs etc.) we better fail on open() instead of copyfile(3) as we're not quite sure what's gonna happen in that case. * update doc
2018-06-12 18:04:50 -03:00
.. versionchanged:: 3.8
Platform-specific fast-copy syscalls may be used internally in order to
copy the file more efficiently. See
:ref:`shutil-platform-dependent-efficient-copy-operations` section.
.. function:: copy2(src, dst, *, follow_symlinks=True)
2007-08-15 11:28:22 -03:00
Identical to :func:`~shutil.copy` except that :func:`copy2`
also attempts to preserve file metadata.
When *follow_symlinks* is false, and *src* is a symbolic
link, :func:`copy2` attempts to copy all metadata from the
*src* symbolic link to the newly-created *dst* symbolic link.
However, this functionality is not available on all platforms.
On platforms where some or all of this functionality is
unavailable, :func:`copy2` will preserve all the metadata
it can; :func:`copy2` never raises an exception because it
cannot preserve file metadata.
:func:`copy2` uses :func:`copystat` to copy the file metadata.
Please see :func:`copystat` for more information
about platform support for modifying symbolic link metadata.
2007-08-15 11:28:22 -03:00
.. audit-event:: shutil.copyfile src,dst shutil.copy2
.. audit-event:: shutil.copystat src,dst shutil.copy2
.. versionchanged:: 3.3
Added *follow_symlinks* argument, try to copy extended
file system attributes too (currently Linux only).
Now returns path to the newly created file.
bpo-33671: efficient zero-copy for shutil.copy* functions (Linux, OSX and Win) (#7160) * have shutil.copyfileobj use sendfile() if possible * refactoring: use ctx manager * add test with non-regular file obj * emulate case where file size can't be determined * reference _copyfileobj_sendfile directly * add test for offset() at certain position * add test for empty file * add test for non regular file dst * small refactoring * leave copyfileobj() alone in order to not introduce any incompatibility * minor refactoring * remove old test * update docstring * update docstring; rename exception class * detect platforms which only support file to socket zero copy * don't run test on platforms where file-to-file zero copy is not supported * use tempfiles * reset verbosity * add test for smaller chunks * add big file size test * add comment * update doc * update whatsnew doc * update doc * catch Exception * remove unused import * add test case for error on second sendfile() call * turn docstring into comment * add one more test * update comment * add Misc/NEWS entry * get rid of COPY_BUFSIZE; it belongs to another PR * update doc * expose posix._fcopyfile() for OSX * merge from linux branch * merge from linux branch * expose fcopyfile * arg clinic for the win implementation * convert path type to path_t * expose CopyFileW * fix windows tests * release GIL * minor refactoring * update doc * update comment * update docstrings * rename functions * rename test classes * update doc * update doc * update docstrings and comments * avoid do import nt|posix modules if unnecessary * set nt|posix modules to None if not available * micro speedup * update description * add doc note * use better wording in doc * rename function using 'fastcopy' prefix instead of 'zerocopy' * use :ref: in rst doc * change wording in doc * add test to make sure sendfile() doesn't get called aymore in case it doesn't support file to file copies * move CopyFileW in _winapi and actually expose CopyFileExW instead * fix line endings * add tests for mode bits * add docstring * remove test file mode class; let's keep it for later when Istart addressing OSX fcopyfile() specific copies * update doc to reflect new changes * update doc * adjust tests on win * fix argument clinic error * update doc * OSX: expose copyfile(3) instead of fcopyfile(3); also expose flags arg to python * osx / copyfile: use path_t instead of char * do not set dst name in the OSError exception in order to remain consistent with platforms which cannot do that (e.g. linux) * add same file test * add test for same file * have osx copyfile() pre-emptively check if src and dst are the same, otherwise it will return immedialtey and src file content gets deleted * turn PermissionError into appropriate SameFileError * expose ERROR_SHARING_VIOLATION in order to raise more appropriate SameFileError * honour follow_symlinks arg when using CopyFileEx * update Misc/NEWS * expose CreateDirectoryEx mock * change C type * CreateDirectoryExW actual implementation * provide specific makedirs() implementation for win * fix typo * skeleton for SetNamedSecurityInfo * get security info for src path * finally set security attrs * add unit tests * mimick os.makedirs() behavior and raise if dst dir exists * set 2 paths for OSError object * set 2 paths for OSError object * expand windows test * in case of exception on os.sendfile() set filename and filename2 exception attributes * set 2 filenames (src, dst) for OSError in case copyfile() fails on OSX * update doc * do not use CreateDirectoryEx() in copytree() if source dir is a symlink (breaks test_copytree_symlink_dir); instead just create a plain dir and remain consistent with POSIX implementation * use bytearray() and readinto() * use memoryview() with bytearray() * refactoring + introduce a new _fastcopy_binfileobj() fun * remove CopyFileEx and other C wrappers * remove code related to CopyFileEx * Recognize binary files in copyfileobj() ...and use fastest _fastcopy_binfileobj() when possible * set 1MB copy bufsize on win; also add a global _COPY_BUFSIZE variable * use ctx manager for memoryview() * update doc * remove outdated doc * remove last CopyFileEx remnants * OSX - use fcopyfile(3) instead of copyfile(3) ...as an extra safety measure: in case src/dst are "exotic" files (non regular or living on a network fs etc.) we better fail on open() instead of copyfile(3) as we're not quite sure what's gonna happen in that case. * update doc
2018-06-12 18:04:50 -03:00
.. versionchanged:: 3.8
Platform-specific fast-copy syscalls may be used internally in order to
copy the file more efficiently. See
:ref:`shutil-platform-dependent-efficient-copy-operations` section.
Merged revisions 64623,64640,64665,64687,64689-64690,64719,64721,64735,64742,64744-64746,64756-64761,64767-64769,64771-64772,64774-64775,64788,64793,64835-64836 via svnmerge from svn+ssh://pythondev@svn.python.org/python/trunk ........ r64623 | benjamin.peterson | 2008-07-01 21:51:54 +0200 (Tue, 01 Jul 2008) | 1 line write a short little section for multiprocessing; it still needs help ........ r64640 | georg.brandl | 2008-07-01 22:56:03 +0200 (Tue, 01 Jul 2008) | 2 lines Add a comment about incref'ing w. ........ r64665 | jesse.noller | 2008-07-02 18:56:51 +0200 (Wed, 02 Jul 2008) | 1 line Add #!/usr/bin/env python for ben ........ r64687 | andrew.kuchling | 2008-07-03 14:50:03 +0200 (Thu, 03 Jul 2008) | 1 line Tweak wording ........ r64689 | benjamin.peterson | 2008-07-03 14:57:35 +0200 (Thu, 03 Jul 2008) | 1 line lowercase glossary term ........ r64690 | benjamin.peterson | 2008-07-03 15:01:17 +0200 (Thu, 03 Jul 2008) | 1 line let the term be linked ........ r64719 | raymond.hettinger | 2008-07-05 04:11:55 +0200 (Sat, 05 Jul 2008) | 1 line Update comment on prediction macros. ........ r64721 | georg.brandl | 2008-07-05 12:07:18 +0200 (Sat, 05 Jul 2008) | 2 lines Fix tabs. ........ r64735 | mark.dickinson | 2008-07-05 17:25:48 +0200 (Sat, 05 Jul 2008) | 3 lines Minor rewrite of cmath_log to work around a Sun compiler bug. See issue #3168. ........ r64742 | benjamin.peterson | 2008-07-05 18:29:38 +0200 (Sat, 05 Jul 2008) | 1 line make regrtest aware of the lib2to3 resource ........ r64744 | georg.brandl | 2008-07-05 18:43:45 +0200 (Sat, 05 Jul 2008) | 2 lines Keep below 80 chars. ........ r64745 | facundo.batista | 2008-07-05 21:19:50 +0200 (Sat, 05 Jul 2008) | 3 lines Issue 3289. Removed two lines that ended doing nothing. ........ r64746 | facundo.batista | 2008-07-05 22:39:59 +0200 (Sat, 05 Jul 2008) | 4 lines Issue #3239. Differentiate the ascii call from the curses one and the builtin one. ........ r64756 | gregory.p.smith | 2008-07-06 09:16:40 +0200 (Sun, 06 Jul 2008) | 3 lines - Issue #2113: Fix error in subprocess.Popen if the select system call is interrupted by a signal. ........ r64757 | benjamin.peterson | 2008-07-06 14:39:09 +0200 (Sun, 06 Jul 2008) | 1 line remove test_compact_freelists from test_sys ........ r64758 | gregory.p.smith | 2008-07-06 19:06:29 +0200 (Sun, 06 Jul 2008) | 2 lines fix issue3304 - remove an incorrect PyMem_Free in fileio_init ........ r64759 | georg.brandl | 2008-07-06 19:36:20 +0200 (Sun, 06 Jul 2008) | 2 lines Fix opensearch template. ........ r64760 | andrew.kuchling | 2008-07-06 19:43:16 +0200 (Sun, 06 Jul 2008) | 1 line Wording fix ........ r64761 | andrew.kuchling | 2008-07-06 19:44:17 +0200 (Sun, 06 Jul 2008) | 1 line Add two items; rewrap paragraph ........ r64767 | gregory.p.smith | 2008-07-07 06:31:58 +0200 (Mon, 07 Jul 2008) | 4 lines - Issue #3309: Fix bz2.BZFile itererator to release its internal lock properly when raising an exception due to the bz2file being closed. Prevents a deadlock. ........ r64768 | josiah.carlson | 2008-07-07 06:51:46 +0200 (Mon, 07 Jul 2008) | 2 lines Fixed bugs 760475, 953599, and 1519. ........ r64769 | gregory.p.smith | 2008-07-07 06:54:31 +0200 (Mon, 07 Jul 2008) | 2 lines Add commented out #_sha256 and #_sha512 lines per issue 3183. ........ r64771 | gregory.p.smith | 2008-07-07 07:09:12 +0200 (Mon, 07 Jul 2008) | 4 lines - Issue #3094: httplib.HTTPSConnection Host: headers no longer include the redundant ":443" port number designation when the connection is using the default https port (443). ........ r64772 | skip.montanaro | 2008-07-07 13:16:14 +0200 (Mon, 07 Jul 2008) | 2 lines Correct grammar. ........ r64774 | andrew.kuchling | 2008-07-07 18:51:09 +0200 (Mon, 07 Jul 2008) | 1 line Fix example to match text ........ r64775 | facundo.batista | 2008-07-07 19:02:59 +0200 (Mon, 07 Jul 2008) | 3 lines Issue 3306. Better control for a lenght in findmax() function. ........ r64788 | georg.brandl | 2008-07-08 09:05:23 +0200 (Tue, 08 Jul 2008) | 2 lines Add missing ABCs to list. ........ r64793 | nick.coghlan | 2008-07-08 16:21:42 +0200 (Tue, 08 Jul 2008) | 1 line Add missing NEWS and ACK entries for r64791 ........ r64835 | raymond.hettinger | 2008-07-10 11:31:08 +0200 (Thu, 10 Jul 2008) | 1 line Issue 3287: Raise correct exception for float inputs. ........ r64836 | raymond.hettinger | 2008-07-10 12:28:41 +0200 (Thu, 10 Jul 2008) | 1 line Use operator.index() instead of n.__index__(). ........
2008-07-16 00:43:04 -03:00
.. function:: ignore_patterns(\*patterns)
This factory function creates a function that can be used as a callable for
:func:`copytree`\'s *ignore* argument, ignoring files and directories that
match one of the glob-style *patterns* provided. See the example below.
.. function:: copytree(src, dst, symlinks=False, ignore=None, \
copy_function=copy2, ignore_dangling_symlinks=False, \
dirs_exist_ok=False)
2007-08-15 11:28:22 -03:00
Recursively copy an entire directory tree rooted at *src* to a directory
named *dst* and return the destination directory. *dirs_exist_ok* dictates
whether to raise an exception in case *dst* or any missing parent directory
already exists.
Permissions and times of directories are copied with :func:`copystat`,
individual files are copied using :func:`~shutil.copy2`.
Merged revisions 64623,64640,64665,64687,64689-64690,64719,64721,64735,64742,64744-64746,64756-64761,64767-64769,64771-64772,64774-64775,64788,64793,64835-64836 via svnmerge from svn+ssh://pythondev@svn.python.org/python/trunk ........ r64623 | benjamin.peterson | 2008-07-01 21:51:54 +0200 (Tue, 01 Jul 2008) | 1 line write a short little section for multiprocessing; it still needs help ........ r64640 | georg.brandl | 2008-07-01 22:56:03 +0200 (Tue, 01 Jul 2008) | 2 lines Add a comment about incref'ing w. ........ r64665 | jesse.noller | 2008-07-02 18:56:51 +0200 (Wed, 02 Jul 2008) | 1 line Add #!/usr/bin/env python for ben ........ r64687 | andrew.kuchling | 2008-07-03 14:50:03 +0200 (Thu, 03 Jul 2008) | 1 line Tweak wording ........ r64689 | benjamin.peterson | 2008-07-03 14:57:35 +0200 (Thu, 03 Jul 2008) | 1 line lowercase glossary term ........ r64690 | benjamin.peterson | 2008-07-03 15:01:17 +0200 (Thu, 03 Jul 2008) | 1 line let the term be linked ........ r64719 | raymond.hettinger | 2008-07-05 04:11:55 +0200 (Sat, 05 Jul 2008) | 1 line Update comment on prediction macros. ........ r64721 | georg.brandl | 2008-07-05 12:07:18 +0200 (Sat, 05 Jul 2008) | 2 lines Fix tabs. ........ r64735 | mark.dickinson | 2008-07-05 17:25:48 +0200 (Sat, 05 Jul 2008) | 3 lines Minor rewrite of cmath_log to work around a Sun compiler bug. See issue #3168. ........ r64742 | benjamin.peterson | 2008-07-05 18:29:38 +0200 (Sat, 05 Jul 2008) | 1 line make regrtest aware of the lib2to3 resource ........ r64744 | georg.brandl | 2008-07-05 18:43:45 +0200 (Sat, 05 Jul 2008) | 2 lines Keep below 80 chars. ........ r64745 | facundo.batista | 2008-07-05 21:19:50 +0200 (Sat, 05 Jul 2008) | 3 lines Issue 3289. Removed two lines that ended doing nothing. ........ r64746 | facundo.batista | 2008-07-05 22:39:59 +0200 (Sat, 05 Jul 2008) | 4 lines Issue #3239. Differentiate the ascii call from the curses one and the builtin one. ........ r64756 | gregory.p.smith | 2008-07-06 09:16:40 +0200 (Sun, 06 Jul 2008) | 3 lines - Issue #2113: Fix error in subprocess.Popen if the select system call is interrupted by a signal. ........ r64757 | benjamin.peterson | 2008-07-06 14:39:09 +0200 (Sun, 06 Jul 2008) | 1 line remove test_compact_freelists from test_sys ........ r64758 | gregory.p.smith | 2008-07-06 19:06:29 +0200 (Sun, 06 Jul 2008) | 2 lines fix issue3304 - remove an incorrect PyMem_Free in fileio_init ........ r64759 | georg.brandl | 2008-07-06 19:36:20 +0200 (Sun, 06 Jul 2008) | 2 lines Fix opensearch template. ........ r64760 | andrew.kuchling | 2008-07-06 19:43:16 +0200 (Sun, 06 Jul 2008) | 1 line Wording fix ........ r64761 | andrew.kuchling | 2008-07-06 19:44:17 +0200 (Sun, 06 Jul 2008) | 1 line Add two items; rewrap paragraph ........ r64767 | gregory.p.smith | 2008-07-07 06:31:58 +0200 (Mon, 07 Jul 2008) | 4 lines - Issue #3309: Fix bz2.BZFile itererator to release its internal lock properly when raising an exception due to the bz2file being closed. Prevents a deadlock. ........ r64768 | josiah.carlson | 2008-07-07 06:51:46 +0200 (Mon, 07 Jul 2008) | 2 lines Fixed bugs 760475, 953599, and 1519. ........ r64769 | gregory.p.smith | 2008-07-07 06:54:31 +0200 (Mon, 07 Jul 2008) | 2 lines Add commented out #_sha256 and #_sha512 lines per issue 3183. ........ r64771 | gregory.p.smith | 2008-07-07 07:09:12 +0200 (Mon, 07 Jul 2008) | 4 lines - Issue #3094: httplib.HTTPSConnection Host: headers no longer include the redundant ":443" port number designation when the connection is using the default https port (443). ........ r64772 | skip.montanaro | 2008-07-07 13:16:14 +0200 (Mon, 07 Jul 2008) | 2 lines Correct grammar. ........ r64774 | andrew.kuchling | 2008-07-07 18:51:09 +0200 (Mon, 07 Jul 2008) | 1 line Fix example to match text ........ r64775 | facundo.batista | 2008-07-07 19:02:59 +0200 (Mon, 07 Jul 2008) | 3 lines Issue 3306. Better control for a lenght in findmax() function. ........ r64788 | georg.brandl | 2008-07-08 09:05:23 +0200 (Tue, 08 Jul 2008) | 2 lines Add missing ABCs to list. ........ r64793 | nick.coghlan | 2008-07-08 16:21:42 +0200 (Tue, 08 Jul 2008) | 1 line Add missing NEWS and ACK entries for r64791 ........ r64835 | raymond.hettinger | 2008-07-10 11:31:08 +0200 (Thu, 10 Jul 2008) | 1 line Issue 3287: Raise correct exception for float inputs. ........ r64836 | raymond.hettinger | 2008-07-10 12:28:41 +0200 (Thu, 10 Jul 2008) | 1 line Use operator.index() instead of n.__index__(). ........
2008-07-16 00:43:04 -03:00
If *symlinks* is true, symbolic links in the source tree are represented as
symbolic links in the new tree and the metadata of the original links will
be copied as far as the platform allows; if false or omitted, the contents
and metadata of the linked files are copied to the new tree.
Merged revisions 64623,64640,64665,64687,64689-64690,64719,64721,64735,64742,64744-64746,64756-64761,64767-64769,64771-64772,64774-64775,64788,64793,64835-64836 via svnmerge from svn+ssh://pythondev@svn.python.org/python/trunk ........ r64623 | benjamin.peterson | 2008-07-01 21:51:54 +0200 (Tue, 01 Jul 2008) | 1 line write a short little section for multiprocessing; it still needs help ........ r64640 | georg.brandl | 2008-07-01 22:56:03 +0200 (Tue, 01 Jul 2008) | 2 lines Add a comment about incref'ing w. ........ r64665 | jesse.noller | 2008-07-02 18:56:51 +0200 (Wed, 02 Jul 2008) | 1 line Add #!/usr/bin/env python for ben ........ r64687 | andrew.kuchling | 2008-07-03 14:50:03 +0200 (Thu, 03 Jul 2008) | 1 line Tweak wording ........ r64689 | benjamin.peterson | 2008-07-03 14:57:35 +0200 (Thu, 03 Jul 2008) | 1 line lowercase glossary term ........ r64690 | benjamin.peterson | 2008-07-03 15:01:17 +0200 (Thu, 03 Jul 2008) | 1 line let the term be linked ........ r64719 | raymond.hettinger | 2008-07-05 04:11:55 +0200 (Sat, 05 Jul 2008) | 1 line Update comment on prediction macros. ........ r64721 | georg.brandl | 2008-07-05 12:07:18 +0200 (Sat, 05 Jul 2008) | 2 lines Fix tabs. ........ r64735 | mark.dickinson | 2008-07-05 17:25:48 +0200 (Sat, 05 Jul 2008) | 3 lines Minor rewrite of cmath_log to work around a Sun compiler bug. See issue #3168. ........ r64742 | benjamin.peterson | 2008-07-05 18:29:38 +0200 (Sat, 05 Jul 2008) | 1 line make regrtest aware of the lib2to3 resource ........ r64744 | georg.brandl | 2008-07-05 18:43:45 +0200 (Sat, 05 Jul 2008) | 2 lines Keep below 80 chars. ........ r64745 | facundo.batista | 2008-07-05 21:19:50 +0200 (Sat, 05 Jul 2008) | 3 lines Issue 3289. Removed two lines that ended doing nothing. ........ r64746 | facundo.batista | 2008-07-05 22:39:59 +0200 (Sat, 05 Jul 2008) | 4 lines Issue #3239. Differentiate the ascii call from the curses one and the builtin one. ........ r64756 | gregory.p.smith | 2008-07-06 09:16:40 +0200 (Sun, 06 Jul 2008) | 3 lines - Issue #2113: Fix error in subprocess.Popen if the select system call is interrupted by a signal. ........ r64757 | benjamin.peterson | 2008-07-06 14:39:09 +0200 (Sun, 06 Jul 2008) | 1 line remove test_compact_freelists from test_sys ........ r64758 | gregory.p.smith | 2008-07-06 19:06:29 +0200 (Sun, 06 Jul 2008) | 2 lines fix issue3304 - remove an incorrect PyMem_Free in fileio_init ........ r64759 | georg.brandl | 2008-07-06 19:36:20 +0200 (Sun, 06 Jul 2008) | 2 lines Fix opensearch template. ........ r64760 | andrew.kuchling | 2008-07-06 19:43:16 +0200 (Sun, 06 Jul 2008) | 1 line Wording fix ........ r64761 | andrew.kuchling | 2008-07-06 19:44:17 +0200 (Sun, 06 Jul 2008) | 1 line Add two items; rewrap paragraph ........ r64767 | gregory.p.smith | 2008-07-07 06:31:58 +0200 (Mon, 07 Jul 2008) | 4 lines - Issue #3309: Fix bz2.BZFile itererator to release its internal lock properly when raising an exception due to the bz2file being closed. Prevents a deadlock. ........ r64768 | josiah.carlson | 2008-07-07 06:51:46 +0200 (Mon, 07 Jul 2008) | 2 lines Fixed bugs 760475, 953599, and 1519. ........ r64769 | gregory.p.smith | 2008-07-07 06:54:31 +0200 (Mon, 07 Jul 2008) | 2 lines Add commented out #_sha256 and #_sha512 lines per issue 3183. ........ r64771 | gregory.p.smith | 2008-07-07 07:09:12 +0200 (Mon, 07 Jul 2008) | 4 lines - Issue #3094: httplib.HTTPSConnection Host: headers no longer include the redundant ":443" port number designation when the connection is using the default https port (443). ........ r64772 | skip.montanaro | 2008-07-07 13:16:14 +0200 (Mon, 07 Jul 2008) | 2 lines Correct grammar. ........ r64774 | andrew.kuchling | 2008-07-07 18:51:09 +0200 (Mon, 07 Jul 2008) | 1 line Fix example to match text ........ r64775 | facundo.batista | 2008-07-07 19:02:59 +0200 (Mon, 07 Jul 2008) | 3 lines Issue 3306. Better control for a lenght in findmax() function. ........ r64788 | georg.brandl | 2008-07-08 09:05:23 +0200 (Tue, 08 Jul 2008) | 2 lines Add missing ABCs to list. ........ r64793 | nick.coghlan | 2008-07-08 16:21:42 +0200 (Tue, 08 Jul 2008) | 1 line Add missing NEWS and ACK entries for r64791 ........ r64835 | raymond.hettinger | 2008-07-10 11:31:08 +0200 (Thu, 10 Jul 2008) | 1 line Issue 3287: Raise correct exception for float inputs. ........ r64836 | raymond.hettinger | 2008-07-10 12:28:41 +0200 (Thu, 10 Jul 2008) | 1 line Use operator.index() instead of n.__index__(). ........
2008-07-16 00:43:04 -03:00
When *symlinks* is false, if the file pointed by the symlink doesn't
exist, an exception will be added in the list of errors raised in
an :exc:`Error` exception at the end of the copy process.
You can set the optional *ignore_dangling_symlinks* flag to true if you
want to silence this exception. Notice that this option has no effect
on platforms that don't support :func:`os.symlink`.
Merged revisions 64623,64640,64665,64687,64689-64690,64719,64721,64735,64742,64744-64746,64756-64761,64767-64769,64771-64772,64774-64775,64788,64793,64835-64836 via svnmerge from svn+ssh://pythondev@svn.python.org/python/trunk ........ r64623 | benjamin.peterson | 2008-07-01 21:51:54 +0200 (Tue, 01 Jul 2008) | 1 line write a short little section for multiprocessing; it still needs help ........ r64640 | georg.brandl | 2008-07-01 22:56:03 +0200 (Tue, 01 Jul 2008) | 2 lines Add a comment about incref'ing w. ........ r64665 | jesse.noller | 2008-07-02 18:56:51 +0200 (Wed, 02 Jul 2008) | 1 line Add #!/usr/bin/env python for ben ........ r64687 | andrew.kuchling | 2008-07-03 14:50:03 +0200 (Thu, 03 Jul 2008) | 1 line Tweak wording ........ r64689 | benjamin.peterson | 2008-07-03 14:57:35 +0200 (Thu, 03 Jul 2008) | 1 line lowercase glossary term ........ r64690 | benjamin.peterson | 2008-07-03 15:01:17 +0200 (Thu, 03 Jul 2008) | 1 line let the term be linked ........ r64719 | raymond.hettinger | 2008-07-05 04:11:55 +0200 (Sat, 05 Jul 2008) | 1 line Update comment on prediction macros. ........ r64721 | georg.brandl | 2008-07-05 12:07:18 +0200 (Sat, 05 Jul 2008) | 2 lines Fix tabs. ........ r64735 | mark.dickinson | 2008-07-05 17:25:48 +0200 (Sat, 05 Jul 2008) | 3 lines Minor rewrite of cmath_log to work around a Sun compiler bug. See issue #3168. ........ r64742 | benjamin.peterson | 2008-07-05 18:29:38 +0200 (Sat, 05 Jul 2008) | 1 line make regrtest aware of the lib2to3 resource ........ r64744 | georg.brandl | 2008-07-05 18:43:45 +0200 (Sat, 05 Jul 2008) | 2 lines Keep below 80 chars. ........ r64745 | facundo.batista | 2008-07-05 21:19:50 +0200 (Sat, 05 Jul 2008) | 3 lines Issue 3289. Removed two lines that ended doing nothing. ........ r64746 | facundo.batista | 2008-07-05 22:39:59 +0200 (Sat, 05 Jul 2008) | 4 lines Issue #3239. Differentiate the ascii call from the curses one and the builtin one. ........ r64756 | gregory.p.smith | 2008-07-06 09:16:40 +0200 (Sun, 06 Jul 2008) | 3 lines - Issue #2113: Fix error in subprocess.Popen if the select system call is interrupted by a signal. ........ r64757 | benjamin.peterson | 2008-07-06 14:39:09 +0200 (Sun, 06 Jul 2008) | 1 line remove test_compact_freelists from test_sys ........ r64758 | gregory.p.smith | 2008-07-06 19:06:29 +0200 (Sun, 06 Jul 2008) | 2 lines fix issue3304 - remove an incorrect PyMem_Free in fileio_init ........ r64759 | georg.brandl | 2008-07-06 19:36:20 +0200 (Sun, 06 Jul 2008) | 2 lines Fix opensearch template. ........ r64760 | andrew.kuchling | 2008-07-06 19:43:16 +0200 (Sun, 06 Jul 2008) | 1 line Wording fix ........ r64761 | andrew.kuchling | 2008-07-06 19:44:17 +0200 (Sun, 06 Jul 2008) | 1 line Add two items; rewrap paragraph ........ r64767 | gregory.p.smith | 2008-07-07 06:31:58 +0200 (Mon, 07 Jul 2008) | 4 lines - Issue #3309: Fix bz2.BZFile itererator to release its internal lock properly when raising an exception due to the bz2file being closed. Prevents a deadlock. ........ r64768 | josiah.carlson | 2008-07-07 06:51:46 +0200 (Mon, 07 Jul 2008) | 2 lines Fixed bugs 760475, 953599, and 1519. ........ r64769 | gregory.p.smith | 2008-07-07 06:54:31 +0200 (Mon, 07 Jul 2008) | 2 lines Add commented out #_sha256 and #_sha512 lines per issue 3183. ........ r64771 | gregory.p.smith | 2008-07-07 07:09:12 +0200 (Mon, 07 Jul 2008) | 4 lines - Issue #3094: httplib.HTTPSConnection Host: headers no longer include the redundant ":443" port number designation when the connection is using the default https port (443). ........ r64772 | skip.montanaro | 2008-07-07 13:16:14 +0200 (Mon, 07 Jul 2008) | 2 lines Correct grammar. ........ r64774 | andrew.kuchling | 2008-07-07 18:51:09 +0200 (Mon, 07 Jul 2008) | 1 line Fix example to match text ........ r64775 | facundo.batista | 2008-07-07 19:02:59 +0200 (Mon, 07 Jul 2008) | 3 lines Issue 3306. Better control for a lenght in findmax() function. ........ r64788 | georg.brandl | 2008-07-08 09:05:23 +0200 (Tue, 08 Jul 2008) | 2 lines Add missing ABCs to list. ........ r64793 | nick.coghlan | 2008-07-08 16:21:42 +0200 (Tue, 08 Jul 2008) | 1 line Add missing NEWS and ACK entries for r64791 ........ r64835 | raymond.hettinger | 2008-07-10 11:31:08 +0200 (Thu, 10 Jul 2008) | 1 line Issue 3287: Raise correct exception for float inputs. ........ r64836 | raymond.hettinger | 2008-07-10 12:28:41 +0200 (Thu, 10 Jul 2008) | 1 line Use operator.index() instead of n.__index__(). ........
2008-07-16 00:43:04 -03:00
If *ignore* is given, it must be a callable that will receive as its
arguments the directory being visited by :func:`copytree`, and a list of its
contents, as returned by :func:`os.listdir`. Since :func:`copytree` is
called recursively, the *ignore* callable will be called once for each
directory that is copied. The callable must return a sequence of directory
and file names relative to the current directory (i.e. a subset of the items
in its second argument); these names will then be ignored in the copy
process. :func:`ignore_patterns` can be used to create such a callable that
ignores names based on glob-style patterns.
If exception(s) occur, an :exc:`Error` is raised with a list of reasons.
2012-02-13 11:30:47 -04:00
If *copy_function* is given, it must be a callable that will be used to copy
each file. It will be called with the source path and the destination path
  as arguments. By default, :func:`~shutil.copy2` is used, but any function
  that supports the same signature (like :func:`~shutil.copy`) can be used.
2007-08-15 11:28:22 -03:00
.. audit-event:: shutil.copytree src,dst shutil.copytree
.. versionchanged:: 3.3
Copy metadata when *symlinks* is false.
Now returns *dst*.
.. versionchanged:: 3.2
Added the *copy_function* argument to be able to provide a custom copy
function.
Added the *ignore_dangling_symlinks* argument to silent dangling symlinks
errors when *symlinks* is false.
bpo-33671: efficient zero-copy for shutil.copy* functions (Linux, OSX and Win) (#7160) * have shutil.copyfileobj use sendfile() if possible * refactoring: use ctx manager * add test with non-regular file obj * emulate case where file size can't be determined * reference _copyfileobj_sendfile directly * add test for offset() at certain position * add test for empty file * add test for non regular file dst * small refactoring * leave copyfileobj() alone in order to not introduce any incompatibility * minor refactoring * remove old test * update docstring * update docstring; rename exception class * detect platforms which only support file to socket zero copy * don't run test on platforms where file-to-file zero copy is not supported * use tempfiles * reset verbosity * add test for smaller chunks * add big file size test * add comment * update doc * update whatsnew doc * update doc * catch Exception * remove unused import * add test case for error on second sendfile() call * turn docstring into comment * add one more test * update comment * add Misc/NEWS entry * get rid of COPY_BUFSIZE; it belongs to another PR * update doc * expose posix._fcopyfile() for OSX * merge from linux branch * merge from linux branch * expose fcopyfile * arg clinic for the win implementation * convert path type to path_t * expose CopyFileW * fix windows tests * release GIL * minor refactoring * update doc * update comment * update docstrings * rename functions * rename test classes * update doc * update doc * update docstrings and comments * avoid do import nt|posix modules if unnecessary * set nt|posix modules to None if not available * micro speedup * update description * add doc note * use better wording in doc * rename function using 'fastcopy' prefix instead of 'zerocopy' * use :ref: in rst doc * change wording in doc * add test to make sure sendfile() doesn't get called aymore in case it doesn't support file to file copies * move CopyFileW in _winapi and actually expose CopyFileExW instead * fix line endings * add tests for mode bits * add docstring * remove test file mode class; let's keep it for later when Istart addressing OSX fcopyfile() specific copies * update doc to reflect new changes * update doc * adjust tests on win * fix argument clinic error * update doc * OSX: expose copyfile(3) instead of fcopyfile(3); also expose flags arg to python * osx / copyfile: use path_t instead of char * do not set dst name in the OSError exception in order to remain consistent with platforms which cannot do that (e.g. linux) * add same file test * add test for same file * have osx copyfile() pre-emptively check if src and dst are the same, otherwise it will return immedialtey and src file content gets deleted * turn PermissionError into appropriate SameFileError * expose ERROR_SHARING_VIOLATION in order to raise more appropriate SameFileError * honour follow_symlinks arg when using CopyFileEx * update Misc/NEWS * expose CreateDirectoryEx mock * change C type * CreateDirectoryExW actual implementation * provide specific makedirs() implementation for win * fix typo * skeleton for SetNamedSecurityInfo * get security info for src path * finally set security attrs * add unit tests * mimick os.makedirs() behavior and raise if dst dir exists * set 2 paths for OSError object * set 2 paths for OSError object * expand windows test * in case of exception on os.sendfile() set filename and filename2 exception attributes * set 2 filenames (src, dst) for OSError in case copyfile() fails on OSX * update doc * do not use CreateDirectoryEx() in copytree() if source dir is a symlink (breaks test_copytree_symlink_dir); instead just create a plain dir and remain consistent with POSIX implementation * use bytearray() and readinto() * use memoryview() with bytearray() * refactoring + introduce a new _fastcopy_binfileobj() fun * remove CopyFileEx and other C wrappers * remove code related to CopyFileEx * Recognize binary files in copyfileobj() ...and use fastest _fastcopy_binfileobj() when possible * set 1MB copy bufsize on win; also add a global _COPY_BUFSIZE variable * use ctx manager for memoryview() * update doc * remove outdated doc * remove last CopyFileEx remnants * OSX - use fcopyfile(3) instead of copyfile(3) ...as an extra safety measure: in case src/dst are "exotic" files (non regular or living on a network fs etc.) we better fail on open() instead of copyfile(3) as we're not quite sure what's gonna happen in that case. * update doc
2018-06-12 18:04:50 -03:00
.. versionchanged:: 3.8
Platform-specific fast-copy syscalls may be used internally in order to
copy the file more efficiently. See
:ref:`shutil-platform-dependent-efficient-copy-operations` section.
.. versionadded:: 3.8
The *dirs_exist_ok* parameter.
.. function:: rmtree(path, ignore_errors=False, onerror=None)
2007-08-15 11:28:22 -03:00
.. index:: single: directory; deleting
Merged revisions 60124-60142 via svnmerge from svn+ssh://pythondev@svn.python.org/python/trunk ........ r60131 | georg.brandl | 2008-01-20 12:13:29 +0100 (Sun, 20 Jan 2008) | 3 lines #1351692: in pprint, always call format() for dict and list items to enable custom formatting of contents via subclassing PrettyPrinter. ........ r60133 | georg.brandl | 2008-01-20 12:43:03 +0100 (Sun, 20 Jan 2008) | 2 lines #1178141: add addinfourl.code to get http status code from urllib. ........ r60134 | georg.brandl | 2008-01-20 13:05:43 +0100 (Sun, 20 Jan 2008) | 4 lines #856047: respect the ``no_proxy`` env var when checking for proxies in urllib and using the other ``_proxy`` env vars. Original patch by Donovan Baarda. ........ r60135 | georg.brandl | 2008-01-20 13:18:17 +0100 (Sun, 20 Jan 2008) | 4 lines #1664522: in urllib, don't read non-existing directories in ftp mode, returning a 0-byte file -- raise an IOError instead. Original patch from Phil Knirsch. ........ r60136 | georg.brandl | 2008-01-20 13:57:47 +0100 (Sun, 20 Jan 2008) | 2 lines #799369: document possible sys.platform values. ........ r60137 | georg.brandl | 2008-01-20 14:08:37 +0100 (Sun, 20 Jan 2008) | 2 lines #652749: document the constants added to the builtins by site.py. ........ r60138 | georg.brandl | 2008-01-20 14:59:46 +0100 (Sun, 20 Jan 2008) | 2 lines #1648: add sys.gettrace() and sys.getprofile(). ........ r60139 | georg.brandl | 2008-01-20 15:17:42 +0100 (Sun, 20 Jan 2008) | 2 lines #1669: don't allow shutil.rmtree() to be called on a symlink. ........ r60140 | georg.brandl | 2008-01-20 15:20:02 +0100 (Sun, 20 Jan 2008) | 2 lines Fix test_pyclbr after urllib change. ........ r60141 | christian.heimes | 2008-01-20 15:28:28 +0100 (Sun, 20 Jan 2008) | 1 line Fixed a wrong assumption in configure.in and Include/pyport.h. The is finite function is not called isfinite() but finite(). Sorry, my fault. :) ........ r60142 | georg.brandl | 2008-01-20 15:31:27 +0100 (Sun, 20 Jan 2008) | 2 lines #1876: fix typos in test_operator. ........
2008-01-20 11:14:11 -04:00
Delete an entire directory tree; *path* must point to a directory (but not a
symbolic link to a directory). If *ignore_errors* is true, errors resulting
from failed removals will be ignored; if false or omitted, such errors are
handled by calling a handler specified by *onerror* or, if that is omitted,
they raise an exception.
.. note::
On platforms that support the necessary fd-based functions a symlink
attack resistant version of :func:`rmtree` is used by default. On other
platforms, the :func:`rmtree` implementation is susceptible to a symlink
attack: given proper timing and circumstances, attackers can manipulate
symlinks on the filesystem to delete files they wouldn't be able to access
otherwise. Applications can use the :data:`rmtree.avoids_symlink_attacks`
function attribute to determine which case applies.
Merged revisions 60124-60142 via svnmerge from svn+ssh://pythondev@svn.python.org/python/trunk ........ r60131 | georg.brandl | 2008-01-20 12:13:29 +0100 (Sun, 20 Jan 2008) | 3 lines #1351692: in pprint, always call format() for dict and list items to enable custom formatting of contents via subclassing PrettyPrinter. ........ r60133 | georg.brandl | 2008-01-20 12:43:03 +0100 (Sun, 20 Jan 2008) | 2 lines #1178141: add addinfourl.code to get http status code from urllib. ........ r60134 | georg.brandl | 2008-01-20 13:05:43 +0100 (Sun, 20 Jan 2008) | 4 lines #856047: respect the ``no_proxy`` env var when checking for proxies in urllib and using the other ``_proxy`` env vars. Original patch by Donovan Baarda. ........ r60135 | georg.brandl | 2008-01-20 13:18:17 +0100 (Sun, 20 Jan 2008) | 4 lines #1664522: in urllib, don't read non-existing directories in ftp mode, returning a 0-byte file -- raise an IOError instead. Original patch from Phil Knirsch. ........ r60136 | georg.brandl | 2008-01-20 13:57:47 +0100 (Sun, 20 Jan 2008) | 2 lines #799369: document possible sys.platform values. ........ r60137 | georg.brandl | 2008-01-20 14:08:37 +0100 (Sun, 20 Jan 2008) | 2 lines #652749: document the constants added to the builtins by site.py. ........ r60138 | georg.brandl | 2008-01-20 14:59:46 +0100 (Sun, 20 Jan 2008) | 2 lines #1648: add sys.gettrace() and sys.getprofile(). ........ r60139 | georg.brandl | 2008-01-20 15:17:42 +0100 (Sun, 20 Jan 2008) | 2 lines #1669: don't allow shutil.rmtree() to be called on a symlink. ........ r60140 | georg.brandl | 2008-01-20 15:20:02 +0100 (Sun, 20 Jan 2008) | 2 lines Fix test_pyclbr after urllib change. ........ r60141 | christian.heimes | 2008-01-20 15:28:28 +0100 (Sun, 20 Jan 2008) | 1 line Fixed a wrong assumption in configure.in and Include/pyport.h. The is finite function is not called isfinite() but finite(). Sorry, my fault. :) ........ r60142 | georg.brandl | 2008-01-20 15:31:27 +0100 (Sun, 20 Jan 2008) | 2 lines #1876: fix typos in test_operator. ........
2008-01-20 11:14:11 -04:00
If *onerror* is provided, it must be a callable that accepts three
parameters: *function*, *path*, and *excinfo*.
The first parameter, *function*, is the function which raised the exception;
it depends on the platform and implementation. The second parameter,
*path*, will be the path name passed to *function*. The third parameter,
*excinfo*, will be the exception information returned by
:func:`sys.exc_info`. Exceptions raised by *onerror* will not be caught.
.. audit-event:: shutil.rmtree path shutil.rmtree
.. versionchanged:: 3.3
Added a symlink attack resistant version that is used automatically
if platform supports fd-based functions.
Merged revisions 60124-60142 via svnmerge from svn+ssh://pythondev@svn.python.org/python/trunk ........ r60131 | georg.brandl | 2008-01-20 12:13:29 +0100 (Sun, 20 Jan 2008) | 3 lines #1351692: in pprint, always call format() for dict and list items to enable custom formatting of contents via subclassing PrettyPrinter. ........ r60133 | georg.brandl | 2008-01-20 12:43:03 +0100 (Sun, 20 Jan 2008) | 2 lines #1178141: add addinfourl.code to get http status code from urllib. ........ r60134 | georg.brandl | 2008-01-20 13:05:43 +0100 (Sun, 20 Jan 2008) | 4 lines #856047: respect the ``no_proxy`` env var when checking for proxies in urllib and using the other ``_proxy`` env vars. Original patch by Donovan Baarda. ........ r60135 | georg.brandl | 2008-01-20 13:18:17 +0100 (Sun, 20 Jan 2008) | 4 lines #1664522: in urllib, don't read non-existing directories in ftp mode, returning a 0-byte file -- raise an IOError instead. Original patch from Phil Knirsch. ........ r60136 | georg.brandl | 2008-01-20 13:57:47 +0100 (Sun, 20 Jan 2008) | 2 lines #799369: document possible sys.platform values. ........ r60137 | georg.brandl | 2008-01-20 14:08:37 +0100 (Sun, 20 Jan 2008) | 2 lines #652749: document the constants added to the builtins by site.py. ........ r60138 | georg.brandl | 2008-01-20 14:59:46 +0100 (Sun, 20 Jan 2008) | 2 lines #1648: add sys.gettrace() and sys.getprofile(). ........ r60139 | georg.brandl | 2008-01-20 15:17:42 +0100 (Sun, 20 Jan 2008) | 2 lines #1669: don't allow shutil.rmtree() to be called on a symlink. ........ r60140 | georg.brandl | 2008-01-20 15:20:02 +0100 (Sun, 20 Jan 2008) | 2 lines Fix test_pyclbr after urllib change. ........ r60141 | christian.heimes | 2008-01-20 15:28:28 +0100 (Sun, 20 Jan 2008) | 1 line Fixed a wrong assumption in configure.in and Include/pyport.h. The is finite function is not called isfinite() but finite(). Sorry, my fault. :) ........ r60142 | georg.brandl | 2008-01-20 15:31:27 +0100 (Sun, 20 Jan 2008) | 2 lines #1876: fix typos in test_operator. ........
2008-01-20 11:14:11 -04:00
.. versionchanged:: 3.8
On Windows, will no longer delete the contents of a directory junction
before removing the junction.
2012-06-24 14:53:48 -03:00
.. attribute:: rmtree.avoids_symlink_attacks
Indicates whether the current platform and implementation provides a
symlink attack resistant version of :func:`rmtree`. Currently this is
only true for platforms supporting fd-based directory access functions.
.. versionadded:: 3.3
2007-08-15 11:28:22 -03:00
.. function:: move(src, dst, copy_function=copy2)
2007-08-15 11:28:22 -03:00
Recursively move a file or directory (*src*) to another location (*dst*)
and return the destination.
2007-08-15 11:28:22 -03:00
If the destination is an existing directory, then *src* is moved inside that
directory. If the destination already exists but is not a directory, it may
be overwritten depending on :func:`os.rename` semantics.
If the destination is on the current filesystem, then :func:`os.rename` is
used. Otherwise, *src* is copied to *dst* using *copy_function* and then
removed. In case of symlinks, a new symlink pointing to the target of *src*
will be created in or as *dst* and *src* will be removed.
If *copy_function* is given, it must be a callable that takes two arguments
*src* and *dst*, and will be used to copy *src* to *dest* if
:func:`os.rename` cannot be used. If the source is a directory,
:func:`copytree` is called, passing it the :func:`copy_function`. The
default *copy_function* is :func:`copy2`. Using :func:`~shutil.copy` as the
*copy_function* allows the move to succeed when it is not possible to also
copy the metadata, at the expense of not copying any of the metadata.
.. audit-event:: shutil.move src,dst shutil.move
.. versionchanged:: 3.3
Added explicit symlink handling for foreign filesystems, thus adapting
it to the behavior of GNU's :program:`mv`.
Now returns *dst*.
.. versionchanged:: 3.5
Added the *copy_function* keyword argument.
bpo-33671: efficient zero-copy for shutil.copy* functions (Linux, OSX and Win) (#7160) * have shutil.copyfileobj use sendfile() if possible * refactoring: use ctx manager * add test with non-regular file obj * emulate case where file size can't be determined * reference _copyfileobj_sendfile directly * add test for offset() at certain position * add test for empty file * add test for non regular file dst * small refactoring * leave copyfileobj() alone in order to not introduce any incompatibility * minor refactoring * remove old test * update docstring * update docstring; rename exception class * detect platforms which only support file to socket zero copy * don't run test on platforms where file-to-file zero copy is not supported * use tempfiles * reset verbosity * add test for smaller chunks * add big file size test * add comment * update doc * update whatsnew doc * update doc * catch Exception * remove unused import * add test case for error on second sendfile() call * turn docstring into comment * add one more test * update comment * add Misc/NEWS entry * get rid of COPY_BUFSIZE; it belongs to another PR * update doc * expose posix._fcopyfile() for OSX * merge from linux branch * merge from linux branch * expose fcopyfile * arg clinic for the win implementation * convert path type to path_t * expose CopyFileW * fix windows tests * release GIL * minor refactoring * update doc * update comment * update docstrings * rename functions * rename test classes * update doc * update doc * update docstrings and comments * avoid do import nt|posix modules if unnecessary * set nt|posix modules to None if not available * micro speedup * update description * add doc note * use better wording in doc * rename function using 'fastcopy' prefix instead of 'zerocopy' * use :ref: in rst doc * change wording in doc * add test to make sure sendfile() doesn't get called aymore in case it doesn't support file to file copies * move CopyFileW in _winapi and actually expose CopyFileExW instead * fix line endings * add tests for mode bits * add docstring * remove test file mode class; let's keep it for later when Istart addressing OSX fcopyfile() specific copies * update doc to reflect new changes * update doc * adjust tests on win * fix argument clinic error * update doc * OSX: expose copyfile(3) instead of fcopyfile(3); also expose flags arg to python * osx / copyfile: use path_t instead of char * do not set dst name in the OSError exception in order to remain consistent with platforms which cannot do that (e.g. linux) * add same file test * add test for same file * have osx copyfile() pre-emptively check if src and dst are the same, otherwise it will return immedialtey and src file content gets deleted * turn PermissionError into appropriate SameFileError * expose ERROR_SHARING_VIOLATION in order to raise more appropriate SameFileError * honour follow_symlinks arg when using CopyFileEx * update Misc/NEWS * expose CreateDirectoryEx mock * change C type * CreateDirectoryExW actual implementation * provide specific makedirs() implementation for win * fix typo * skeleton for SetNamedSecurityInfo * get security info for src path * finally set security attrs * add unit tests * mimick os.makedirs() behavior and raise if dst dir exists * set 2 paths for OSError object * set 2 paths for OSError object * expand windows test * in case of exception on os.sendfile() set filename and filename2 exception attributes * set 2 filenames (src, dst) for OSError in case copyfile() fails on OSX * update doc * do not use CreateDirectoryEx() in copytree() if source dir is a symlink (breaks test_copytree_symlink_dir); instead just create a plain dir and remain consistent with POSIX implementation * use bytearray() and readinto() * use memoryview() with bytearray() * refactoring + introduce a new _fastcopy_binfileobj() fun * remove CopyFileEx and other C wrappers * remove code related to CopyFileEx * Recognize binary files in copyfileobj() ...and use fastest _fastcopy_binfileobj() when possible * set 1MB copy bufsize on win; also add a global _COPY_BUFSIZE variable * use ctx manager for memoryview() * update doc * remove outdated doc * remove last CopyFileEx remnants * OSX - use fcopyfile(3) instead of copyfile(3) ...as an extra safety measure: in case src/dst are "exotic" files (non regular or living on a network fs etc.) we better fail on open() instead of copyfile(3) as we're not quite sure what's gonna happen in that case. * update doc
2018-06-12 18:04:50 -03:00
.. versionchanged:: 3.8
Platform-specific fast-copy syscalls may be used internally in order to
copy the file more efficiently. See
:ref:`shutil-platform-dependent-efficient-copy-operations` section.
bpo-32689: Updates shutil.move to allow for Path objects to be used as source arg (GH-15326) Important work originally done by @emilyemorehouse two years ago and nearly ready to go in. This bug has affected many people and in some cases has been a dealbreaker to the adoption of the otherwise wonderful pathlib and PEP519. https://stackoverflow.com/questions/33625931/copy-file-with-pathlib-in-python. This adds the outstanding test request from that PR @vstinner (https://github.com/python/cpython/pull/5393). Test fails without the change, passes with it, along with every other test in test_shutil. Some variants were experimented with to make the one line change and the most performant one was picked. # Added Test for PathLike directory destination, the current fail case ``` Lib/test/test_shutil.py::TestMove::test_move_file_pathlike FAILED [100%] ============================================================== FAILURES =============================================================== __________________________________________________ TestMove.test_move_file_pathlike ___________________________________________________ self = <test.test_shutil.TestMove testMethod=test_move_file_pathlike> def test_move_file_pathlike(self): # Move a file to another location on the same filesystem. src = pathlib.Path(self.src_file) > self._check_move_file(src, self.dst_dir, self.dst_file) Lib/test/test_shutil.py:1563: _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ Lib/test/test_shutil.py:1545: in _check_move_file shutil.move(src, dst) /Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/shutil.py:562: in move real_dst = os.path.join(dst, _basename(src)) _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ path = PosixPath('/var/folders/r2/psq74t5x3nbfzlph8bh2pvdw0000gn/T/tmp9ie0wh9_/foo') def _basename(path): # A basename() variant which first strips the trailing slash, if present. # Thus we always get the last component of the path, even for directories. sep = os.path.sep + (os.path.altsep or '') > return os.path.basename(path.rstrip(sep)) E AttributeError: 'PosixPath' object has no attribute 'rstrip' /Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/shutil.py:526: AttributeError ============================================== 1 failed, 102 deselected in 0.30 seconds =============================================== ``` After change: ``` ========================================================= test session starts ========================================================= platform darwin -- Python 3.7.4, pytest-5.0.1, py-1.8.0, pluggy-0.12.0 -- /Users/maxwellmckinnon/.venvs/TA3.7/bin/python3.7 cachedir: .pytest_cache rootdir: /Users/maxwellmckinnon/dev/cpython plugins: cov-2.7.1, mock-1.10.4 collected 103 items / 102 deselected / 1 selected Lib/test/test_shutil.py::TestMove::test_move_file_pathlike PASSED [100%] ============================================== 1 passed, 102 deselected in 0.06 seconds =============================================== ``` Running all the tests in test_shutil.py ``` ╰─ pytest Lib/test/test_shutil.py -v ========================================================= test session starts ========================================================= platform darwin -- Python 3.7.4, pytest-5.0.1, py-1.8.0, pluggy-0.12.0 -- /Users/maxwellmckinnon/.venvs/TA3.7/bin/python3.7 cachedir: .pytest_cache rootdir: /Users/maxwellmckinnon/dev/cpython plugins: cov-2.7.1, mock-1.10.4 collected 103 items Lib/test/test_shutil.py::TestShutil::test_chown PASSED [ 0%] Lib/test/test_shutil.py::TestShutil::test_copy PASSED [ 1%] ... Lib/test/test_shutil.py::TermsizeTests::test_stty_match SKIPPED [ 99%] Lib/test/test_shutil.py::PublicAPITests::test_module_all_attribute PASSED [100%] ================================================ 96 passed, 7 skipped in 1.25 seconds ================================================= ``` # Performance Considerations Is it considered poor form to get rid of _basename altogether and make use of pathlib in the move function? I'm not sure if the idea is for all these modules to strictly avoid circular dependencies. They are already using os.path which is just as much a citizen in 3.8 as pathlib right? e.g. `real_dst = os.path.join(dst, _basename(src))` becomes `real_dst = Path(dst) / Path(src).name` I've looked around and familiarized myself, and I now think importing pathlib here is fine. My only remaining concern is that of performance. Here's the performance difference for this step. ``` In [46]: %timeit real_dst = os.path.join("a/b/c", _basename('b/')) 2.71 µs ± 62.6 ns per loop (mean ± std. dev. of 7 runs, 100000 loops each) In [47]: %timeit real_dst = Path("a/b/c") / Path('b/').name 12.4 µs ± 65.3 ns per loop (mean ± std. dev. of 7 runs, 100000 loops each) ``` Is 10us significant or insignificant compared to the least expensive operation this function will do? I don't know. Let's find out. ``` In [55]: %timeit os.rename('/tmp/a/a.txt', '/tmp/a/b.txt'); os.rename('/tmp/a/b.txt', '/tmp/a/a.txt') 124 µs ± 2.18 µs per loop (mean ± std. dev. of 7 runs, 10000 loops each) ``` 62us to rename. 10us seems significant enough that we wouldn't want to favor the Path sugar suggestion. 16% speed decrease from adding the 10us. What do people think? I was hoping to get to use pathlib.Path here, but I suspect for this low level move, it should be as fast as possible, and 16% is not worth one line of sugary code to me. https://bugs.python.org/issue32689 Automerge-Triggered-By: @gvanrossum
2019-09-30 23:41:16 -03:00
.. versionchanged:: 3.9
Accepts a :term:`path-like object` for both *src* and *dst*.
2011-07-01 08:55:36 -03:00
.. function:: disk_usage(path)
Return disk usage statistics about the given path as a :term:`named tuple`
with the attributes *total*, *used* and *free*, which are the amount of
total, used and free space, in bytes. *path* may be a file or a
directory.
2011-07-01 08:55:36 -03:00
.. versionadded:: 3.3
.. versionchanged:: 3.8
On Windows, *path* can now be a file or directory.
.. availability:: Unix, Windows.
2007-08-15 11:28:22 -03:00
.. function:: chown(path, user=None, group=None)
Change owner *user* and/or *group* of the given *path*.
*user* can be a system user name or a uid; the same applies to *group*. At
least one argument is required.
See also :func:`os.chown`, the underlying function.
.. audit-event:: shutil.chown path,user,group shutil.chown
.. availability:: Unix.
.. versionadded:: 3.3
.. function:: which(cmd, mode=os.F_OK | os.X_OK, path=None)
Return the path to an executable which would be run if the given *cmd* was
called. If no *cmd* would be called, return ``None``.
*mode* is a permission mask passed to :func:`os.access`, by default
determining if the file exists and executable.
When no *path* is specified, the results of :func:`os.environ` are used,
returning either the "PATH" value or a fallback of :attr:`os.defpath`.
On Windows, the current directory is always prepended to the *path* whether
or not you use the default or provide your own, which is the behavior the
command shell uses when finding executables. Additionally, when finding the
*cmd* in the *path*, the ``PATHEXT`` environment variable is checked. For
example, if you call ``shutil.which("python")``, :func:`which` will search
``PATHEXT`` to know that it should look for ``python.exe`` within the *path*
directories. For example, on Windows::
>>> shutil.which("python")
'C:\\Python33\\python.EXE'
.. versionadded:: 3.3
.. versionchanged:: 3.8
The :class:`bytes` type is now accepted. If *cmd* type is
:class:`bytes`, the result type is also :class:`bytes`.
2007-08-15 11:28:22 -03:00
.. exception:: Error
This exception collects exceptions that are raised during a multi-file
operation. For :func:`copytree`, the exception argument is a list of 3-tuples
(*srcname*, *dstname*, *exception*).
2007-08-15 11:28:22 -03:00
bpo-33671: efficient zero-copy for shutil.copy* functions (Linux, OSX and Win) (#7160) * have shutil.copyfileobj use sendfile() if possible * refactoring: use ctx manager * add test with non-regular file obj * emulate case where file size can't be determined * reference _copyfileobj_sendfile directly * add test for offset() at certain position * add test for empty file * add test for non regular file dst * small refactoring * leave copyfileobj() alone in order to not introduce any incompatibility * minor refactoring * remove old test * update docstring * update docstring; rename exception class * detect platforms which only support file to socket zero copy * don't run test on platforms where file-to-file zero copy is not supported * use tempfiles * reset verbosity * add test for smaller chunks * add big file size test * add comment * update doc * update whatsnew doc * update doc * catch Exception * remove unused import * add test case for error on second sendfile() call * turn docstring into comment * add one more test * update comment * add Misc/NEWS entry * get rid of COPY_BUFSIZE; it belongs to another PR * update doc * expose posix._fcopyfile() for OSX * merge from linux branch * merge from linux branch * expose fcopyfile * arg clinic for the win implementation * convert path type to path_t * expose CopyFileW * fix windows tests * release GIL * minor refactoring * update doc * update comment * update docstrings * rename functions * rename test classes * update doc * update doc * update docstrings and comments * avoid do import nt|posix modules if unnecessary * set nt|posix modules to None if not available * micro speedup * update description * add doc note * use better wording in doc * rename function using 'fastcopy' prefix instead of 'zerocopy' * use :ref: in rst doc * change wording in doc * add test to make sure sendfile() doesn't get called aymore in case it doesn't support file to file copies * move CopyFileW in _winapi and actually expose CopyFileExW instead * fix line endings * add tests for mode bits * add docstring * remove test file mode class; let's keep it for later when Istart addressing OSX fcopyfile() specific copies * update doc to reflect new changes * update doc * adjust tests on win * fix argument clinic error * update doc * OSX: expose copyfile(3) instead of fcopyfile(3); also expose flags arg to python * osx / copyfile: use path_t instead of char * do not set dst name in the OSError exception in order to remain consistent with platforms which cannot do that (e.g. linux) * add same file test * add test for same file * have osx copyfile() pre-emptively check if src and dst are the same, otherwise it will return immedialtey and src file content gets deleted * turn PermissionError into appropriate SameFileError * expose ERROR_SHARING_VIOLATION in order to raise more appropriate SameFileError * honour follow_symlinks arg when using CopyFileEx * update Misc/NEWS * expose CreateDirectoryEx mock * change C type * CreateDirectoryExW actual implementation * provide specific makedirs() implementation for win * fix typo * skeleton for SetNamedSecurityInfo * get security info for src path * finally set security attrs * add unit tests * mimick os.makedirs() behavior and raise if dst dir exists * set 2 paths for OSError object * set 2 paths for OSError object * expand windows test * in case of exception on os.sendfile() set filename and filename2 exception attributes * set 2 filenames (src, dst) for OSError in case copyfile() fails on OSX * update doc * do not use CreateDirectoryEx() in copytree() if source dir is a symlink (breaks test_copytree_symlink_dir); instead just create a plain dir and remain consistent with POSIX implementation * use bytearray() and readinto() * use memoryview() with bytearray() * refactoring + introduce a new _fastcopy_binfileobj() fun * remove CopyFileEx and other C wrappers * remove code related to CopyFileEx * Recognize binary files in copyfileobj() ...and use fastest _fastcopy_binfileobj() when possible * set 1MB copy bufsize on win; also add a global _COPY_BUFSIZE variable * use ctx manager for memoryview() * update doc * remove outdated doc * remove last CopyFileEx remnants * OSX - use fcopyfile(3) instead of copyfile(3) ...as an extra safety measure: in case src/dst are "exotic" files (non regular or living on a network fs etc.) we better fail on open() instead of copyfile(3) as we're not quite sure what's gonna happen in that case. * update doc
2018-06-12 18:04:50 -03:00
.. _shutil-platform-dependent-efficient-copy-operations:
Platform-dependent efficient copy operations
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
Starting from Python 3.8 all functions involving a file copy (:func:`copyfile`,
:func:`copy`, :func:`copy2`, :func:`copytree`, and :func:`move`) may use
platform-specific "fast-copy" syscalls in order to copy the file more
efficiently (see :issue:`33671`).
"fast-copy" means that the copying operation occurs within the kernel, avoiding
the use of userspace buffers in Python as in "``outfd.write(infd.read())``".
On macOS `fcopyfile`_ is used to copy the file content (not metadata).
bpo-33671: efficient zero-copy for shutil.copy* functions (Linux, OSX and Win) (#7160) * have shutil.copyfileobj use sendfile() if possible * refactoring: use ctx manager * add test with non-regular file obj * emulate case where file size can't be determined * reference _copyfileobj_sendfile directly * add test for offset() at certain position * add test for empty file * add test for non regular file dst * small refactoring * leave copyfileobj() alone in order to not introduce any incompatibility * minor refactoring * remove old test * update docstring * update docstring; rename exception class * detect platforms which only support file to socket zero copy * don't run test on platforms where file-to-file zero copy is not supported * use tempfiles * reset verbosity * add test for smaller chunks * add big file size test * add comment * update doc * update whatsnew doc * update doc * catch Exception * remove unused import * add test case for error on second sendfile() call * turn docstring into comment * add one more test * update comment * add Misc/NEWS entry * get rid of COPY_BUFSIZE; it belongs to another PR * update doc * expose posix._fcopyfile() for OSX * merge from linux branch * merge from linux branch * expose fcopyfile * arg clinic for the win implementation * convert path type to path_t * expose CopyFileW * fix windows tests * release GIL * minor refactoring * update doc * update comment * update docstrings * rename functions * rename test classes * update doc * update doc * update docstrings and comments * avoid do import nt|posix modules if unnecessary * set nt|posix modules to None if not available * micro speedup * update description * add doc note * use better wording in doc * rename function using 'fastcopy' prefix instead of 'zerocopy' * use :ref: in rst doc * change wording in doc * add test to make sure sendfile() doesn't get called aymore in case it doesn't support file to file copies * move CopyFileW in _winapi and actually expose CopyFileExW instead * fix line endings * add tests for mode bits * add docstring * remove test file mode class; let's keep it for later when Istart addressing OSX fcopyfile() specific copies * update doc to reflect new changes * update doc * adjust tests on win * fix argument clinic error * update doc * OSX: expose copyfile(3) instead of fcopyfile(3); also expose flags arg to python * osx / copyfile: use path_t instead of char * do not set dst name in the OSError exception in order to remain consistent with platforms which cannot do that (e.g. linux) * add same file test * add test for same file * have osx copyfile() pre-emptively check if src and dst are the same, otherwise it will return immedialtey and src file content gets deleted * turn PermissionError into appropriate SameFileError * expose ERROR_SHARING_VIOLATION in order to raise more appropriate SameFileError * honour follow_symlinks arg when using CopyFileEx * update Misc/NEWS * expose CreateDirectoryEx mock * change C type * CreateDirectoryExW actual implementation * provide specific makedirs() implementation for win * fix typo * skeleton for SetNamedSecurityInfo * get security info for src path * finally set security attrs * add unit tests * mimick os.makedirs() behavior and raise if dst dir exists * set 2 paths for OSError object * set 2 paths for OSError object * expand windows test * in case of exception on os.sendfile() set filename and filename2 exception attributes * set 2 filenames (src, dst) for OSError in case copyfile() fails on OSX * update doc * do not use CreateDirectoryEx() in copytree() if source dir is a symlink (breaks test_copytree_symlink_dir); instead just create a plain dir and remain consistent with POSIX implementation * use bytearray() and readinto() * use memoryview() with bytearray() * refactoring + introduce a new _fastcopy_binfileobj() fun * remove CopyFileEx and other C wrappers * remove code related to CopyFileEx * Recognize binary files in copyfileobj() ...and use fastest _fastcopy_binfileobj() when possible * set 1MB copy bufsize on win; also add a global _COPY_BUFSIZE variable * use ctx manager for memoryview() * update doc * remove outdated doc * remove last CopyFileEx remnants * OSX - use fcopyfile(3) instead of copyfile(3) ...as an extra safety measure: in case src/dst are "exotic" files (non regular or living on a network fs etc.) we better fail on open() instead of copyfile(3) as we're not quite sure what's gonna happen in that case. * update doc
2018-06-12 18:04:50 -03:00
On Linux :func:`os.sendfile` is used.
bpo-33671: efficient zero-copy for shutil.copy* functions (Linux, OSX and Win) (#7160) * have shutil.copyfileobj use sendfile() if possible * refactoring: use ctx manager * add test with non-regular file obj * emulate case where file size can't be determined * reference _copyfileobj_sendfile directly * add test for offset() at certain position * add test for empty file * add test for non regular file dst * small refactoring * leave copyfileobj() alone in order to not introduce any incompatibility * minor refactoring * remove old test * update docstring * update docstring; rename exception class * detect platforms which only support file to socket zero copy * don't run test on platforms where file-to-file zero copy is not supported * use tempfiles * reset verbosity * add test for smaller chunks * add big file size test * add comment * update doc * update whatsnew doc * update doc * catch Exception * remove unused import * add test case for error on second sendfile() call * turn docstring into comment * add one more test * update comment * add Misc/NEWS entry * get rid of COPY_BUFSIZE; it belongs to another PR * update doc * expose posix._fcopyfile() for OSX * merge from linux branch * merge from linux branch * expose fcopyfile * arg clinic for the win implementation * convert path type to path_t * expose CopyFileW * fix windows tests * release GIL * minor refactoring * update doc * update comment * update docstrings * rename functions * rename test classes * update doc * update doc * update docstrings and comments * avoid do import nt|posix modules if unnecessary * set nt|posix modules to None if not available * micro speedup * update description * add doc note * use better wording in doc * rename function using 'fastcopy' prefix instead of 'zerocopy' * use :ref: in rst doc * change wording in doc * add test to make sure sendfile() doesn't get called aymore in case it doesn't support file to file copies * move CopyFileW in _winapi and actually expose CopyFileExW instead * fix line endings * add tests for mode bits * add docstring * remove test file mode class; let's keep it for later when Istart addressing OSX fcopyfile() specific copies * update doc to reflect new changes * update doc * adjust tests on win * fix argument clinic error * update doc * OSX: expose copyfile(3) instead of fcopyfile(3); also expose flags arg to python * osx / copyfile: use path_t instead of char * do not set dst name in the OSError exception in order to remain consistent with platforms which cannot do that (e.g. linux) * add same file test * add test for same file * have osx copyfile() pre-emptively check if src and dst are the same, otherwise it will return immedialtey and src file content gets deleted * turn PermissionError into appropriate SameFileError * expose ERROR_SHARING_VIOLATION in order to raise more appropriate SameFileError * honour follow_symlinks arg when using CopyFileEx * update Misc/NEWS * expose CreateDirectoryEx mock * change C type * CreateDirectoryExW actual implementation * provide specific makedirs() implementation for win * fix typo * skeleton for SetNamedSecurityInfo * get security info for src path * finally set security attrs * add unit tests * mimick os.makedirs() behavior and raise if dst dir exists * set 2 paths for OSError object * set 2 paths for OSError object * expand windows test * in case of exception on os.sendfile() set filename and filename2 exception attributes * set 2 filenames (src, dst) for OSError in case copyfile() fails on OSX * update doc * do not use CreateDirectoryEx() in copytree() if source dir is a symlink (breaks test_copytree_symlink_dir); instead just create a plain dir and remain consistent with POSIX implementation * use bytearray() and readinto() * use memoryview() with bytearray() * refactoring + introduce a new _fastcopy_binfileobj() fun * remove CopyFileEx and other C wrappers * remove code related to CopyFileEx * Recognize binary files in copyfileobj() ...and use fastest _fastcopy_binfileobj() when possible * set 1MB copy bufsize on win; also add a global _COPY_BUFSIZE variable * use ctx manager for memoryview() * update doc * remove outdated doc * remove last CopyFileEx remnants * OSX - use fcopyfile(3) instead of copyfile(3) ...as an extra safety measure: in case src/dst are "exotic" files (non regular or living on a network fs etc.) we better fail on open() instead of copyfile(3) as we're not quite sure what's gonna happen in that case. * update doc
2018-06-12 18:04:50 -03:00
On Windows :func:`shutil.copyfile` uses a bigger default buffer size (1 MiB
instead of 64 KiB) and a :func:`memoryview`-based variant of
:func:`shutil.copyfileobj` is used.
bpo-33671: efficient zero-copy for shutil.copy* functions (Linux, OSX and Win) (#7160) * have shutil.copyfileobj use sendfile() if possible * refactoring: use ctx manager * add test with non-regular file obj * emulate case where file size can't be determined * reference _copyfileobj_sendfile directly * add test for offset() at certain position * add test for empty file * add test for non regular file dst * small refactoring * leave copyfileobj() alone in order to not introduce any incompatibility * minor refactoring * remove old test * update docstring * update docstring; rename exception class * detect platforms which only support file to socket zero copy * don't run test on platforms where file-to-file zero copy is not supported * use tempfiles * reset verbosity * add test for smaller chunks * add big file size test * add comment * update doc * update whatsnew doc * update doc * catch Exception * remove unused import * add test case for error on second sendfile() call * turn docstring into comment * add one more test * update comment * add Misc/NEWS entry * get rid of COPY_BUFSIZE; it belongs to another PR * update doc * expose posix._fcopyfile() for OSX * merge from linux branch * merge from linux branch * expose fcopyfile * arg clinic for the win implementation * convert path type to path_t * expose CopyFileW * fix windows tests * release GIL * minor refactoring * update doc * update comment * update docstrings * rename functions * rename test classes * update doc * update doc * update docstrings and comments * avoid do import nt|posix modules if unnecessary * set nt|posix modules to None if not available * micro speedup * update description * add doc note * use better wording in doc * rename function using 'fastcopy' prefix instead of 'zerocopy' * use :ref: in rst doc * change wording in doc * add test to make sure sendfile() doesn't get called aymore in case it doesn't support file to file copies * move CopyFileW in _winapi and actually expose CopyFileExW instead * fix line endings * add tests for mode bits * add docstring * remove test file mode class; let's keep it for later when Istart addressing OSX fcopyfile() specific copies * update doc to reflect new changes * update doc * adjust tests on win * fix argument clinic error * update doc * OSX: expose copyfile(3) instead of fcopyfile(3); also expose flags arg to python * osx / copyfile: use path_t instead of char * do not set dst name in the OSError exception in order to remain consistent with platforms which cannot do that (e.g. linux) * add same file test * add test for same file * have osx copyfile() pre-emptively check if src and dst are the same, otherwise it will return immedialtey and src file content gets deleted * turn PermissionError into appropriate SameFileError * expose ERROR_SHARING_VIOLATION in order to raise more appropriate SameFileError * honour follow_symlinks arg when using CopyFileEx * update Misc/NEWS * expose CreateDirectoryEx mock * change C type * CreateDirectoryExW actual implementation * provide specific makedirs() implementation for win * fix typo * skeleton for SetNamedSecurityInfo * get security info for src path * finally set security attrs * add unit tests * mimick os.makedirs() behavior and raise if dst dir exists * set 2 paths for OSError object * set 2 paths for OSError object * expand windows test * in case of exception on os.sendfile() set filename and filename2 exception attributes * set 2 filenames (src, dst) for OSError in case copyfile() fails on OSX * update doc * do not use CreateDirectoryEx() in copytree() if source dir is a symlink (breaks test_copytree_symlink_dir); instead just create a plain dir and remain consistent with POSIX implementation * use bytearray() and readinto() * use memoryview() with bytearray() * refactoring + introduce a new _fastcopy_binfileobj() fun * remove CopyFileEx and other C wrappers * remove code related to CopyFileEx * Recognize binary files in copyfileobj() ...and use fastest _fastcopy_binfileobj() when possible * set 1MB copy bufsize on win; also add a global _COPY_BUFSIZE variable * use ctx manager for memoryview() * update doc * remove outdated doc * remove last CopyFileEx remnants * OSX - use fcopyfile(3) instead of copyfile(3) ...as an extra safety measure: in case src/dst are "exotic" files (non regular or living on a network fs etc.) we better fail on open() instead of copyfile(3) as we're not quite sure what's gonna happen in that case. * update doc
2018-06-12 18:04:50 -03:00
If the fast-copy operation fails and no data was written in the destination
file then shutil will silently fallback on using less efficient
:func:`copyfileobj` function internally.
.. versionchanged:: 3.8
2007-08-15 11:28:22 -03:00
.. _shutil-copytree-example:
2007-08-15 11:28:22 -03:00
copytree example
~~~~~~~~~~~~~~~~
2007-08-15 11:28:22 -03:00
This example is the implementation of the :func:`copytree` function, described
above, with the docstring omitted. It demonstrates many of the other functions
provided by this module. ::
def copytree(src, dst, symlinks=False):
names = os.listdir(src)
os.makedirs(dst)
errors = []
for name in names:
srcname = os.path.join(src, name)
dstname = os.path.join(dst, name)
try:
if symlinks and os.path.islink(srcname):
linkto = os.readlink(srcname)
os.symlink(linkto, dstname)
elif os.path.isdir(srcname):
copytree(srcname, dstname, symlinks)
else:
copy2(srcname, dstname)
# XXX What about devices, sockets etc.?
except OSError as why:
2007-08-15 11:28:22 -03:00
errors.append((srcname, dstname, str(why)))
# catch the Error from the recursive copytree so that we can
# continue with other files
except Error as err:
errors.extend(err.args[0])
try:
copystat(src, dst)
except OSError as why:
# can't copy file access times on Windows
if why.winerror is None:
errors.extend((src, dst, str(why)))
2007-08-15 11:28:22 -03:00
if errors:
raise Error(errors)
2007-08-15 11:28:22 -03:00
Another example that uses the :func:`ignore_patterns` helper::
from shutil import copytree, ignore_patterns
copytree(source, destination, ignore=ignore_patterns('*.pyc', 'tmp*'))
This will copy everything except ``.pyc`` files and files or directories whose
name starts with ``tmp``.
Another example that uses the *ignore* argument to add a logging call::
from shutil import copytree
import logging
def _logpath(path, names):
logging.info('Working in %s', path)
return [] # nothing will be ignored
copytree(source, destination, ignore=_logpath)
.. _shutil-rmtree-example:
rmtree example
~~~~~~~~~~~~~~
This example shows how to remove a directory tree on Windows where some
of the files have their read-only bit set. It uses the onerror callback
to clear the readonly bit and reattempt the remove. Any subsequent failure
will propagate. ::
import os, stat
import shutil
2014-05-07 14:08:08 -03:00
def remove_readonly(func, path, _):
"Clear the readonly bit and reattempt the removal"
os.chmod(path, stat.S_IWRITE)
2014-05-07 14:08:08 -03:00
func(path)
shutil.rmtree(directory, onerror=remove_readonly)
.. _archiving-operations:
Archiving operations
--------------------
.. versionadded:: 3.2
.. versionchanged:: 3.5
Added support for the *xztar* format.
High-level utilities to create and read compressed and archived files are also
provided. They rely on the :mod:`zipfile` and :mod:`tarfile` modules.
.. function:: make_archive(base_name, format, [root_dir, [base_dir, [verbose, [dry_run, [owner, [group, [logger]]]]]]])
Create an archive file (such as zip or tar) and return its name.
*base_name* is the name of the file to create, including the path, minus
any format-specific extension. *format* is the archive format: one of
"zip" (if the :mod:`zlib` module is available), "tar", "gztar" (if the
:mod:`zlib` module is available), "bztar" (if the :mod:`bz2` module is
available), or "xztar" (if the :mod:`lzma` module is available).
*root_dir* is a directory that will be the root directory of the
archive, all paths in the archive will be relative to it; for example,
we typically chdir into *root_dir* before creating the archive.
*base_dir* is the directory where we start archiving from;
i.e. *base_dir* will be the common prefix of all files and
directories in the archive. *base_dir* must be given relative
to *root_dir*. See :ref:`shutil-archiving-example-with-basedir` for how to
use *base_dir* and *root_dir* together.
*root_dir* and *base_dir* both default to the current directory.
If *dry_run* is true, no archive is created, but the operations that would be
executed are logged to *logger*.
*owner* and *group* are used when creating a tar archive. By default,
uses the current owner and group.
*logger* must be an object compatible with :pep:`282`, usually an instance of
:class:`logging.Logger`.
The *verbose* argument is unused and deprecated.
.. audit-event:: shutil.make_archive base_name,format,root_dir,base_dir shutil.make_archive
.. versionchanged:: 3.8
The modern pax (POSIX.1-2001) format is now used instead of
the legacy GNU format for archives created with ``format="tar"``.
.. function:: get_archive_formats()
Return a list of supported formats for archiving.
Each element of the returned sequence is a tuple ``(name, description)``.
By default :mod:`shutil` provides these formats:
- *zip*: ZIP file (if the :mod:`zlib` module is available).
- *tar*: Uncompressed tar file. Uses POSIX.1-2001 pax format for new archives.
- *gztar*: gzip'ed tar-file (if the :mod:`zlib` module is available).
- *bztar*: bzip2'ed tar-file (if the :mod:`bz2` module is available).
- *xztar*: xz'ed tar-file (if the :mod:`lzma` module is available).
You can register new formats or provide your own archiver for any existing
formats, by using :func:`register_archive_format`.
.. function:: register_archive_format(name, function, [extra_args, [description]])
Register an archiver for the format *name*.
*function* is the callable that will be used to unpack archives. The callable
will receive the *base_name* of the file to create, followed by the
*base_dir* (which defaults to :data:`os.curdir`) to start archiving from.
Further arguments are passed as keyword arguments: *owner*, *group*,
*dry_run* and *logger* (as passed in :func:`make_archive`).
If given, *extra_args* is a sequence of ``(name, value)`` pairs that will be
used as extra keywords arguments when the archiver callable is used.
*description* is used by :func:`get_archive_formats` which returns the
list of archivers. Defaults to an empty string.
.. function:: unregister_archive_format(name)
Remove the archive format *name* from the list of supported formats.
.. function:: unpack_archive(filename[, extract_dir[, format]])
Unpack an archive. *filename* is the full path of the archive.
*extract_dir* is the name of the target directory where the archive is
unpacked. If not provided, the current working directory is used.
*format* is the archive format: one of "zip", "tar", "gztar", "bztar", or
"xztar". Or any other format registered with
:func:`register_unpack_format`. If not provided, :func:`unpack_archive`
will use the archive file name extension and see if an unpacker was
registered for that extension. In case none is found,
a :exc:`ValueError` is raised.
.. audit-event:: shutil.unpack_archive filename,extract_dir,format shutil.unpack_archive
.. versionchanged:: 3.7
Accepts a :term:`path-like object` for *filename* and *extract_dir*.
.. function:: register_unpack_format(name, extensions, function[, extra_args[, description]])
Registers an unpack format. *name* is the name of the format and
*extensions* is a list of extensions corresponding to the format, like
``.zip`` for Zip files.
*function* is the callable that will be used to unpack archives. The
callable will receive the path of the archive, followed by the directory
the archive must be extracted to.
When provided, *extra_args* is a sequence of ``(name, value)`` tuples that
will be passed as keywords arguments to the callable.
*description* can be provided to describe the format, and will be returned
by the :func:`get_unpack_formats` function.
.. function:: unregister_unpack_format(name)
Unregister an unpack format. *name* is the name of the format.
.. function:: get_unpack_formats()
Return a list of all registered formats for unpacking.
Each element of the returned sequence is a tuple
``(name, extensions, description)``.
By default :mod:`shutil` provides these formats:
- *zip*: ZIP file (unpacking compressed files works only if the corresponding
module is available).
- *tar*: uncompressed tar file.
- *gztar*: gzip'ed tar-file (if the :mod:`zlib` module is available).
- *bztar*: bzip2'ed tar-file (if the :mod:`bz2` module is available).
- *xztar*: xz'ed tar-file (if the :mod:`lzma` module is available).
You can register new formats or provide your own unpacker for any existing
formats, by using :func:`register_unpack_format`.
.. _shutil-archiving-example:
Archiving example
~~~~~~~~~~~~~~~~~
In this example, we create a gzip'ed tar-file archive containing all files
found in the :file:`.ssh` directory of the user::
>>> from shutil import make_archive
>>> import os
>>> archive_name = os.path.expanduser(os.path.join('~', 'myarchive'))
>>> root_dir = os.path.expanduser(os.path.join('~', '.ssh'))
>>> make_archive(archive_name, 'gztar', root_dir)
'/Users/tarek/myarchive.tar.gz'
The resulting archive contains:
.. code-block:: shell-session
$ tar -tzvf /Users/tarek/myarchive.tar.gz
drwx------ tarek/staff 0 2010-02-01 16:23:40 ./
-rw-r--r-- tarek/staff 609 2008-06-09 13:26:54 ./authorized_keys
-rwxr-xr-x tarek/staff 65 2008-06-09 13:26:54 ./config
-rwx------ tarek/staff 668 2008-06-09 13:26:54 ./id_dsa
-rwxr-xr-x tarek/staff 609 2008-06-09 13:26:54 ./id_dsa.pub
-rw------- tarek/staff 1675 2008-06-09 13:26:54 ./id_rsa
-rw-r--r-- tarek/staff 397 2008-06-09 13:26:54 ./id_rsa.pub
-rw-r--r-- tarek/staff 37192 2010-02-06 18:23:10 ./known_hosts
.. _shutil-archiving-example-with-basedir:
Archiving example with *base_dir*
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
In this example, similar to the `one above <shutil-archiving-example_>`_,
we show how to use :func:`make_archive`, but this time with the usage of
*base_dir*. We now have the following directory structure:
.. code-block:: shell-session
$ tree tmp
tmp
└── root
└── structure
├── content
└── please_add.txt
└── do_not_add.txt
In the final archive, :file:`please_add.txt` should be included, but
:file:`do_not_add.txt` should not. Therefore we use the following::
>>> from shutil import make_archive
>>> import os
>>> archive_name = os.path.expanduser(os.path.join('~', 'myarchive'))
>>> make_archive(
... archive_name,
... 'tar',
... root_dir='tmp/root',
... base_dir='structure/content',
... )
'/Users/tarek/my_archive.tar'
Listing the files in the resulting archive gives us:
.. code-block:: shell-session
$ python -m tarfile -l /Users/tarek/myarchive.tar
structure/content/
structure/content/please_add.txt
Querying the size of the output terminal
----------------------------------------
.. function:: get_terminal_size(fallback=(columns, lines))
Get the size of the terminal window.
For each of the two dimensions, the environment variable, ``COLUMNS``
and ``LINES`` respectively, is checked. If the variable is defined and
the value is a positive integer, it is used.
When ``COLUMNS`` or ``LINES`` is not defined, which is the common case,
the terminal connected to :data:`sys.__stdout__` is queried
by invoking :func:`os.get_terminal_size`.
If the terminal size cannot be successfully queried, either because
the system doesn't support querying, or because we are not
connected to a terminal, the value given in ``fallback`` parameter
is used. ``fallback`` defaults to ``(80, 24)`` which is the default
size used by many terminal emulators.
The value returned is a named tuple of type :class:`os.terminal_size`.
See also: The Single UNIX Specification, Version 2,
`Other Environment Variables`_.
.. versionadded:: 3.3
bpo-33671: efficient zero-copy for shutil.copy* functions (Linux, OSX and Win) (#7160) * have shutil.copyfileobj use sendfile() if possible * refactoring: use ctx manager * add test with non-regular file obj * emulate case where file size can't be determined * reference _copyfileobj_sendfile directly * add test for offset() at certain position * add test for empty file * add test for non regular file dst * small refactoring * leave copyfileobj() alone in order to not introduce any incompatibility * minor refactoring * remove old test * update docstring * update docstring; rename exception class * detect platforms which only support file to socket zero copy * don't run test on platforms where file-to-file zero copy is not supported * use tempfiles * reset verbosity * add test for smaller chunks * add big file size test * add comment * update doc * update whatsnew doc * update doc * catch Exception * remove unused import * add test case for error on second sendfile() call * turn docstring into comment * add one more test * update comment * add Misc/NEWS entry * get rid of COPY_BUFSIZE; it belongs to another PR * update doc * expose posix._fcopyfile() for OSX * merge from linux branch * merge from linux branch * expose fcopyfile * arg clinic for the win implementation * convert path type to path_t * expose CopyFileW * fix windows tests * release GIL * minor refactoring * update doc * update comment * update docstrings * rename functions * rename test classes * update doc * update doc * update docstrings and comments * avoid do import nt|posix modules if unnecessary * set nt|posix modules to None if not available * micro speedup * update description * add doc note * use better wording in doc * rename function using 'fastcopy' prefix instead of 'zerocopy' * use :ref: in rst doc * change wording in doc * add test to make sure sendfile() doesn't get called aymore in case it doesn't support file to file copies * move CopyFileW in _winapi and actually expose CopyFileExW instead * fix line endings * add tests for mode bits * add docstring * remove test file mode class; let's keep it for later when Istart addressing OSX fcopyfile() specific copies * update doc to reflect new changes * update doc * adjust tests on win * fix argument clinic error * update doc * OSX: expose copyfile(3) instead of fcopyfile(3); also expose flags arg to python * osx / copyfile: use path_t instead of char * do not set dst name in the OSError exception in order to remain consistent with platforms which cannot do that (e.g. linux) * add same file test * add test for same file * have osx copyfile() pre-emptively check if src and dst are the same, otherwise it will return immedialtey and src file content gets deleted * turn PermissionError into appropriate SameFileError * expose ERROR_SHARING_VIOLATION in order to raise more appropriate SameFileError * honour follow_symlinks arg when using CopyFileEx * update Misc/NEWS * expose CreateDirectoryEx mock * change C type * CreateDirectoryExW actual implementation * provide specific makedirs() implementation for win * fix typo * skeleton for SetNamedSecurityInfo * get security info for src path * finally set security attrs * add unit tests * mimick os.makedirs() behavior and raise if dst dir exists * set 2 paths for OSError object * set 2 paths for OSError object * expand windows test * in case of exception on os.sendfile() set filename and filename2 exception attributes * set 2 filenames (src, dst) for OSError in case copyfile() fails on OSX * update doc * do not use CreateDirectoryEx() in copytree() if source dir is a symlink (breaks test_copytree_symlink_dir); instead just create a plain dir and remain consistent with POSIX implementation * use bytearray() and readinto() * use memoryview() with bytearray() * refactoring + introduce a new _fastcopy_binfileobj() fun * remove CopyFileEx and other C wrappers * remove code related to CopyFileEx * Recognize binary files in copyfileobj() ...and use fastest _fastcopy_binfileobj() when possible * set 1MB copy bufsize on win; also add a global _COPY_BUFSIZE variable * use ctx manager for memoryview() * update doc * remove outdated doc * remove last CopyFileEx remnants * OSX - use fcopyfile(3) instead of copyfile(3) ...as an extra safety measure: in case src/dst are "exotic" files (non regular or living on a network fs etc.) we better fail on open() instead of copyfile(3) as we're not quite sure what's gonna happen in that case. * update doc
2018-06-12 18:04:50 -03:00
.. _`fcopyfile`:
http://www.manpagez.com/man/3/copyfile/
.. _`Other Environment Variables`:
http://pubs.opengroup.org/onlinepubs/7908799/xbd/envvar.html#tag_002_003