Issue 24230: The tempfile module now accepts bytes for prefix, suffix and dir

parameters and returns bytes in such situations (matching the os module APIs).
This commit is contained in:
Gregory P. Smith 2015-05-22 16:18:14 -07:00
parent 4a7fe7e397
commit ad577b938b
5 changed files with 281 additions and 50 deletions

View File

@ -119,7 +119,7 @@ The module defines the following user-callable items:
.. versionadded:: 3.2
.. function:: mkstemp(suffix='', prefix='tmp', dir=None, text=False)
.. function:: mkstemp(suffix=None, prefix=None, dir=None, text=False)
Creates a temporary file in the most secure manner possible. There are
no race conditions in the file's creation, assuming that the platform
@ -148,6 +148,16 @@ The module defines the following user-callable items:
filename will have any nice properties, such as not requiring quoting
when passed to external commands via ``os.popen()``.
*suffix*, *prefix*, and *dir* must all contain the same type, if specified.
If they are bytes, the returned name will be bytes instead of str.
If you want to force a bytes return value with otherwise default behavior,
pass ``suffix=b''``.
A *prefix* value of ``None`` means use the return value of
:func:`gettempprefix` or :func:`gettempprefixb` as appropriate.
A *suffix* value of ``None`` means use an appropriate empty value.
If *text* is specified, it indicates whether to open the file in binary
mode (the default) or text mode. On some platforms, this makes no
difference.
@ -156,8 +166,14 @@ The module defines the following user-callable items:
file (as would be returned by :func:`os.open`) and the absolute pathname
of that file, in that order.
.. versionchanged:: 3.5
*suffix*, *prefix*, and *dir* may now be supplied in bytes in order to
obtain a bytes return value. Prior to this, only str was allowed.
*suffix* and *prefix* now accept and default to ``None`` to cause
an appropriate default value to be used.
.. function:: mkdtemp(suffix='', prefix='tmp', dir=None)
.. function:: mkdtemp(suffix=None, prefix=None, dir=None)
Creates a temporary directory in the most secure manner possible. There
are no race conditions in the directory's creation. The directory is
@ -171,6 +187,12 @@ The module defines the following user-callable items:
:func:`mkdtemp` returns the absolute pathname of the new directory.
.. versionchanged:: 3.5
*suffix*, *prefix*, and *dir* may now be supplied in bytes in order to
obtain a bytes return value. Prior to this, only str was allowed.
*suffix* and *prefix* now accept and default to ``None`` to cause
an appropriate default value to be used.
.. function:: mktemp(suffix='', prefix='tmp', dir=None)
@ -239,12 +261,23 @@ the appropriate function arguments, instead.
:data:`tempdir` is not ``None``, this simply returns its contents; otherwise,
the search described above is performed, and the result returned.
.. function:: gettempdirb()
Same as :func:`gettempdir` but the return value is in bytes.
.. versionadded:: 3.5
.. function:: gettempprefix()
Return the filename prefix used to create temporary files. This does not
contain the directory component.
.. function:: gettempprefixb()
Same as :func:`gettempprefixb` but the return value is in bytes.
.. versionadded:: 3.5
Examples
--------

View File

@ -96,7 +96,12 @@ Implementation improvements:
Significantly Improved Library Modules:
* None yet.
* You may now pass bytes to the :mod:`tempfile` module's APIs and it will
return the temporary pathname as bytes instead of str. It also accepts
a value of ``None`` on parameters where only str was accepted in the past to
do the right thing based on the types of the other inputs. Two functions,
:func:`gettempdirb` and :func:`gettempprefixb`, have been added to go along
with this. This behavior matches that of the :mod:`os` APIs.
Security improvements:

View File

@ -6,6 +6,14 @@ provided by this module can be used without fear of race conditions
except for 'mktemp'. 'mktemp' is subject to race conditions and
should not be used; it is provided for backward compatibility only.
The default path names are returned as str. If you supply bytes as
input, all return values will be in bytes. Ex:
>>> tempfile.mkstemp()
(4, '/tmp/tmptpu9nin8')
>>> tempfile.mkdtemp(suffix=b'')
b'/tmp/tmppbi8f0hy'
This module also provides some data items to the user:
TMP_MAX - maximum number of names that will be tried before
@ -21,7 +29,8 @@ __all__ = [
"mkstemp", "mkdtemp", # low level safe interfaces
"mktemp", # deprecated unsafe interface
"TMP_MAX", "gettempprefix", # constants
"tempdir", "gettempdir"
"tempdir", "gettempdir",
"gettempprefixb", "gettempdirb",
]
@ -55,8 +64,10 @@ if hasattr(_os, 'TMP_MAX'):
else:
TMP_MAX = 10000
# Although it does not have an underscore for historical reasons, this
# variable is an internal implementation detail (see issue 10354).
# This variable _was_ unused for legacy reasons, see issue 10354.
# But as of 3.5 we actually use it at runtime so changing it would
# have a possibly desirable side effect... But we do not want to support
# that as an API. It is undocumented on purpose. Do not depend on this.
template = "tmp"
# Internal routines.
@ -82,6 +93,46 @@ def _exists(fn):
else:
return True
def _infer_return_type(*args):
"""Look at the type of all args and divine their implied return type."""
return_type = None
for arg in args:
if arg is None:
continue
if isinstance(arg, bytes):
if return_type is str:
raise TypeError("Can't mix bytes and non-bytes in "
"path components.")
return_type = bytes
else:
if return_type is bytes:
raise TypeError("Can't mix bytes and non-bytes in "
"path components.")
return_type = str
if return_type is None:
return str # tempfile APIs return a str by default.
return return_type
def _sanitize_params(prefix, suffix, dir):
"""Common parameter processing for most APIs in this module."""
output_type = _infer_return_type(prefix, suffix, dir)
if suffix is None:
suffix = output_type()
if prefix is None:
if output_type is str:
prefix = template
else:
prefix = _os.fsencode(template)
if dir is None:
if output_type is str:
dir = gettempdir()
else:
dir = gettempdirb()
return prefix, suffix, dir, output_type
class _RandomNameSequence:
"""An instance of _RandomNameSequence generates an endless
sequence of unpredictable strings which can safely be incorporated
@ -195,17 +246,18 @@ def _get_candidate_names():
return _name_sequence
def _mkstemp_inner(dir, pre, suf, flags):
def _mkstemp_inner(dir, pre, suf, flags, output_type):
"""Code common to mkstemp, TemporaryFile, and NamedTemporaryFile."""
names = _get_candidate_names()
if output_type is bytes:
names = map(_os.fsencode, names)
for seq in range(TMP_MAX):
name = next(names)
file = _os.path.join(dir, pre + name + suf)
try:
fd = _os.open(file, flags, 0o600)
return (fd, _os.path.abspath(file))
except FileExistsError:
continue # try again
except PermissionError:
@ -216,6 +268,7 @@ def _mkstemp_inner(dir, pre, suf, flags):
continue
else:
raise
return (fd, _os.path.abspath(file))
raise FileExistsError(_errno.EEXIST,
"No usable temporary file name found")
@ -224,9 +277,13 @@ def _mkstemp_inner(dir, pre, suf, flags):
# User visible interfaces.
def gettempprefix():
"""Accessor for tempdir.template."""
"""The default prefix for temporary directories."""
return template
def gettempprefixb():
"""The default prefix for temporary directories as bytes."""
return _os.fsencode(gettempprefix())
tempdir = None
def gettempdir():
@ -241,7 +298,11 @@ def gettempdir():
_once_lock.release()
return tempdir
def mkstemp(suffix="", prefix=template, dir=None, text=False):
def gettempdirb():
"""A bytes version of tempfile.gettempdir()."""
return _os.fsencode(gettempdir())
def mkstemp(suffix=None, prefix=None, dir=None, text=False):
"""User-callable function to create and return a unique temporary
file. The return value is a pair (fd, name) where fd is the
file descriptor returned by os.open, and name is the filename.
@ -259,6 +320,10 @@ def mkstemp(suffix="", prefix=template, dir=None, text=False):
mode. Else (the default) the file is opened in binary mode. On
some operating systems, this makes no difference.
suffix, prefix and dir must all contain the same type if specified.
If they are bytes, the returned name will be bytes; str otherwise.
A value of None will cause an appropriate default to be used.
The file is readable and writable only by the creating user ID.
If the operating system uses permission bits to indicate whether a
file is executable, the file is executable by no one. The file
@ -267,18 +332,17 @@ def mkstemp(suffix="", prefix=template, dir=None, text=False):
Caller is responsible for deleting the file when done with it.
"""
if dir is None:
dir = gettempdir()
prefix, suffix, dir, output_type = _sanitize_params(prefix, suffix, dir)
if text:
flags = _text_openflags
else:
flags = _bin_openflags
return _mkstemp_inner(dir, prefix, suffix, flags)
return _mkstemp_inner(dir, prefix, suffix, flags, output_type)
def mkdtemp(suffix="", prefix=template, dir=None):
def mkdtemp(suffix=None, prefix=None, dir=None):
"""User-callable function to create and return a unique temporary
directory. The return value is the pathname of the directory.
@ -291,17 +355,17 @@ def mkdtemp(suffix="", prefix=template, dir=None):
Caller is responsible for deleting the directory when done with it.
"""
if dir is None:
dir = gettempdir()
prefix, suffix, dir, output_type = _sanitize_params(prefix, suffix, dir)
names = _get_candidate_names()
if output_type is bytes:
names = map(_os.fsencode, names)
for seq in range(TMP_MAX):
name = next(names)
file = _os.path.join(dir, prefix + name + suffix)
try:
_os.mkdir(file, 0o700)
return file
except FileExistsError:
continue # try again
except PermissionError:
@ -312,6 +376,7 @@ def mkdtemp(suffix="", prefix=template, dir=None):
continue
else:
raise
return file
raise FileExistsError(_errno.EEXIST,
"No usable temporary directory name found")
@ -323,8 +388,8 @@ def mktemp(suffix="", prefix=template, dir=None):
Arguments are as for mkstemp, except that the 'text' argument is
not accepted.
This function is unsafe and should not be used. The file name
refers to a file that did not exist at some point, but by the time
THIS FUNCTION IS UNSAFE AND SHOULD NOT BE USED. The file name may
refer to a file that did not exist at some point, but by the time
you get around to creating it, someone else may have beaten you to
the punch.
"""
@ -454,7 +519,7 @@ class _TemporaryFileWrapper:
def NamedTemporaryFile(mode='w+b', buffering=-1, encoding=None,
newline=None, suffix="", prefix=template,
newline=None, suffix=None, prefix=None,
dir=None, delete=True):
"""Create and return a temporary file.
Arguments:
@ -471,8 +536,7 @@ def NamedTemporaryFile(mode='w+b', buffering=-1, encoding=None,
when it is closed unless the 'delete' argument is set to False.
"""
if dir is None:
dir = gettempdir()
prefix, suffix, dir, output_type = _sanitize_params(prefix, suffix, dir)
flags = _bin_openflags
@ -481,7 +545,7 @@ def NamedTemporaryFile(mode='w+b', buffering=-1, encoding=None,
if _os.name == 'nt' and delete:
flags |= _os.O_TEMPORARY
(fd, name) = _mkstemp_inner(dir, prefix, suffix, flags)
(fd, name) = _mkstemp_inner(dir, prefix, suffix, flags, output_type)
try:
file = _io.open(fd, mode, buffering=buffering,
newline=newline, encoding=encoding)
@ -503,7 +567,7 @@ else:
_O_TMPFILE_WORKS = hasattr(_os, 'O_TMPFILE')
def TemporaryFile(mode='w+b', buffering=-1, encoding=None,
newline=None, suffix="", prefix=template,
newline=None, suffix=None, prefix=None,
dir=None):
"""Create and return a temporary file.
Arguments:
@ -519,8 +583,7 @@ else:
"""
global _O_TMPFILE_WORKS
if dir is None:
dir = gettempdir()
prefix, suffix, dir, output_type = _sanitize_params(prefix, suffix, dir)
flags = _bin_openflags
if _O_TMPFILE_WORKS:
@ -544,7 +607,7 @@ else:
raise
# Fallback to _mkstemp_inner().
(fd, name) = _mkstemp_inner(dir, prefix, suffix, flags)
(fd, name) = _mkstemp_inner(dir, prefix, suffix, flags, output_type)
try:
_os.unlink(name)
return _io.open(fd, mode, buffering=buffering,
@ -562,7 +625,7 @@ class SpooledTemporaryFile:
def __init__(self, max_size=0, mode='w+b', buffering=-1,
encoding=None, newline=None,
suffix="", prefix=template, dir=None):
suffix=None, prefix=None, dir=None):
if 'b' in mode:
self._file = _io.BytesIO()
else:
@ -713,7 +776,7 @@ class TemporaryDirectory(object):
in it are removed.
"""
def __init__(self, suffix="", prefix=template, dir=None):
def __init__(self, suffix=None, prefix=None, dir=None):
self.name = mkdtemp(suffix, prefix, dir)
self._finalizer = _weakref.finalize(
self, self._cleanup, self.name,

View File

@ -36,10 +36,38 @@ else:
# in order of their appearance in the file. Testing which requires
# threads is not done here.
class TestLowLevelInternals(unittest.TestCase):
def test_infer_return_type_singles(self):
self.assertIs(str, tempfile._infer_return_type(''))
self.assertIs(bytes, tempfile._infer_return_type(b''))
self.assertIs(str, tempfile._infer_return_type(None))
def test_infer_return_type_multiples(self):
self.assertIs(str, tempfile._infer_return_type('', ''))
self.assertIs(bytes, tempfile._infer_return_type(b'', b''))
with self.assertRaises(TypeError):
tempfile._infer_return_type('', b'')
with self.assertRaises(TypeError):
tempfile._infer_return_type(b'', '')
def test_infer_return_type_multiples_and_none(self):
self.assertIs(str, tempfile._infer_return_type(None, ''))
self.assertIs(str, tempfile._infer_return_type('', None))
self.assertIs(str, tempfile._infer_return_type(None, None))
self.assertIs(bytes, tempfile._infer_return_type(b'', None))
self.assertIs(bytes, tempfile._infer_return_type(None, b''))
with self.assertRaises(TypeError):
tempfile._infer_return_type('', None, b'')
with self.assertRaises(TypeError):
tempfile._infer_return_type(b'', None, '')
# Common functionality.
class BaseTestCase(unittest.TestCase):
str_check = re.compile(r"^[a-z0-9_-]{8}$")
b_check = re.compile(br"^[a-z0-9_-]{8}$")
def setUp(self):
self._warnings_manager = support.check_warnings()
@ -56,18 +84,31 @@ class BaseTestCase(unittest.TestCase):
npre = nbase[:len(pre)]
nsuf = nbase[len(nbase)-len(suf):]
if dir is not None:
self.assertIs(type(name), str if type(dir) is str else bytes,
"unexpected return type")
if pre is not None:
self.assertIs(type(name), str if type(pre) is str else bytes,
"unexpected return type")
if suf is not None:
self.assertIs(type(name), str if type(suf) is str else bytes,
"unexpected return type")
if (dir, pre, suf) == (None, None, None):
self.assertIs(type(name), str, "default return type must be str")
# check for equality of the absolute paths!
self.assertEqual(os.path.abspath(ndir), os.path.abspath(dir),
"file '%s' not in directory '%s'" % (name, dir))
"file %r not in directory %r" % (name, dir))
self.assertEqual(npre, pre,
"file '%s' does not begin with '%s'" % (nbase, pre))
"file %r does not begin with %r" % (nbase, pre))
self.assertEqual(nsuf, suf,
"file '%s' does not end with '%s'" % (nbase, suf))
"file %r does not end with %r" % (nbase, suf))
nbase = nbase[len(pre):len(nbase)-len(suf)]
self.assertTrue(self.str_check.match(nbase),
"random string '%s' does not match ^[a-z0-9_-]{8}$"
% nbase)
check = self.str_check if isinstance(nbase, str) else self.b_check
self.assertTrue(check.match(nbase),
"random characters %r do not match %r"
% (nbase, check.pattern))
class TestExports(BaseTestCase):
@ -83,7 +124,9 @@ class TestExports(BaseTestCase):
"mktemp" : 1,
"TMP_MAX" : 1,
"gettempprefix" : 1,
"gettempprefixb" : 1,
"gettempdir" : 1,
"gettempdirb" : 1,
"tempdir" : 1,
"template" : 1,
"SpooledTemporaryFile" : 1,
@ -320,7 +363,8 @@ class TestMkstempInner(TestBadTempdir, BaseTestCase):
if bin: flags = self._bflags
else: flags = self._tflags
(self.fd, self.name) = tempfile._mkstemp_inner(dir, pre, suf, flags)
output_type = tempfile._infer_return_type(dir, pre, suf)
(self.fd, self.name) = tempfile._mkstemp_inner(dir, pre, suf, flags, output_type)
def write(self, str):
os.write(self.fd, str)
@ -329,9 +373,17 @@ class TestMkstempInner(TestBadTempdir, BaseTestCase):
self._close(self.fd)
self._unlink(self.name)
def do_create(self, dir=None, pre="", suf="", bin=1):
def do_create(self, dir=None, pre=None, suf=None, bin=1):
output_type = tempfile._infer_return_type(dir, pre, suf)
if dir is None:
dir = tempfile.gettempdir()
if output_type is str:
dir = tempfile.gettempdir()
else:
dir = tempfile.gettempdirb()
if pre is None:
pre = output_type()
if suf is None:
suf = output_type()
file = self.mkstemped(dir, pre, suf, bin)
self.nameCheck(file.name, dir, pre, suf)
@ -345,6 +397,23 @@ class TestMkstempInner(TestBadTempdir, BaseTestCase):
self.do_create(pre="a", suf="b").write(b"blat")
self.do_create(pre="aa", suf=".txt").write(b"blat")
def test_basic_with_bytes_names(self):
# _mkstemp_inner can create files when given name parts all
# specified as bytes.
dir_b = tempfile.gettempdirb()
self.do_create(dir=dir_b, suf=b"").write(b"blat")
self.do_create(dir=dir_b, pre=b"a").write(b"blat")
self.do_create(dir=dir_b, suf=b"b").write(b"blat")
self.do_create(dir=dir_b, pre=b"a", suf=b"b").write(b"blat")
self.do_create(dir=dir_b, pre=b"aa", suf=b".txt").write(b"blat")
# Can't mix str & binary types in the args.
with self.assertRaises(TypeError):
self.do_create(dir="", suf=b"").write(b"blat")
with self.assertRaises(TypeError):
self.do_create(dir=dir_b, pre="").write(b"blat")
with self.assertRaises(TypeError):
self.do_create(dir=dir_b, pre=b"", suf="").write(b"blat")
def test_basic_many(self):
# _mkstemp_inner can create many files (stochastic)
extant = list(range(TEST_FILES))
@ -424,9 +493,10 @@ class TestMkstempInner(TestBadTempdir, BaseTestCase):
def make_temp(self):
return tempfile._mkstemp_inner(tempfile.gettempdir(),
tempfile.template,
tempfile.gettempprefix(),
'',
tempfile._bin_openflags)
tempfile._bin_openflags,
str)
def test_collision_with_existing_file(self):
# _mkstemp_inner tries another name when a file with
@ -462,7 +532,12 @@ class TestGetTempPrefix(BaseTestCase):
p = tempfile.gettempprefix()
self.assertIsInstance(p, str)
self.assertTrue(len(p) > 0)
self.assertGreater(len(p), 0)
pb = tempfile.gettempprefixb()
self.assertIsInstance(pb, bytes)
self.assertGreater(len(pb), 0)
def test_usable_template(self):
# gettempprefix returns a usable prefix string
@ -487,11 +562,11 @@ class TestGetTempDir(BaseTestCase):
def test_directory_exists(self):
# gettempdir returns a directory which exists
dir = tempfile.gettempdir()
self.assertTrue(os.path.isabs(dir) or dir == os.curdir,
"%s is not an absolute path" % dir)
self.assertTrue(os.path.isdir(dir),
"%s is not a directory" % dir)
for d in (tempfile.gettempdir(), tempfile.gettempdirb()):
self.assertTrue(os.path.isabs(d) or d == os.curdir,
"%r is not an absolute path" % d)
self.assertTrue(os.path.isdir(d),
"%r is not a directory" % d)
def test_directory_writable(self):
# gettempdir returns a directory writable by the user
@ -507,8 +582,11 @@ class TestGetTempDir(BaseTestCase):
# gettempdir always returns the same object
a = tempfile.gettempdir()
b = tempfile.gettempdir()
c = tempfile.gettempdirb()
self.assertTrue(a is b)
self.assertNotEqual(type(a), type(c))
self.assertEqual(a, os.fsdecode(c))
def test_case_sensitive(self):
# gettempdir should not flatten its case
@ -528,9 +606,17 @@ class TestGetTempDir(BaseTestCase):
class TestMkstemp(BaseTestCase):
"""Test mkstemp()."""
def do_create(self, dir=None, pre="", suf=""):
def do_create(self, dir=None, pre=None, suf=None):
output_type = tempfile._infer_return_type(dir, pre, suf)
if dir is None:
dir = tempfile.gettempdir()
if output_type is str:
dir = tempfile.gettempdir()
else:
dir = tempfile.gettempdirb()
if pre is None:
pre = output_type()
if suf is None:
suf = output_type()
(fd, name) = tempfile.mkstemp(dir=dir, prefix=pre, suffix=suf)
(ndir, nbase) = os.path.split(name)
adir = os.path.abspath(dir)
@ -552,6 +638,24 @@ class TestMkstemp(BaseTestCase):
self.do_create(pre="aa", suf=".txt")
self.do_create(dir=".")
def test_basic_with_bytes_names(self):
# mkstemp can create files when given name parts all
# specified as bytes.
d = tempfile.gettempdirb()
self.do_create(dir=d, suf=b"")
self.do_create(dir=d, pre=b"a")
self.do_create(dir=d, suf=b"b")
self.do_create(dir=d, pre=b"a", suf=b"b")
self.do_create(dir=d, pre=b"aa", suf=b".txt")
self.do_create(dir=b".")
with self.assertRaises(TypeError):
self.do_create(dir=".", pre=b"aa", suf=b".txt")
with self.assertRaises(TypeError):
self.do_create(dir=b".", pre="aa", suf=b".txt")
with self.assertRaises(TypeError):
self.do_create(dir=b".", pre=b"aa", suf=".txt")
def test_choose_directory(self):
# mkstemp can create directories in a user-selected directory
dir = tempfile.mkdtemp()
@ -567,9 +671,17 @@ class TestMkdtemp(TestBadTempdir, BaseTestCase):
def make_temp(self):
return tempfile.mkdtemp()
def do_create(self, dir=None, pre="", suf=""):
def do_create(self, dir=None, pre=None, suf=None):
output_type = tempfile._infer_return_type(dir, pre, suf)
if dir is None:
dir = tempfile.gettempdir()
if output_type is str:
dir = tempfile.gettempdir()
else:
dir = tempfile.gettempdirb()
if pre is None:
pre = output_type()
if suf is None:
suf = output_type()
name = tempfile.mkdtemp(dir=dir, prefix=pre, suffix=suf)
try:
@ -587,6 +699,21 @@ class TestMkdtemp(TestBadTempdir, BaseTestCase):
os.rmdir(self.do_create(pre="a", suf="b"))
os.rmdir(self.do_create(pre="aa", suf=".txt"))
def test_basic_with_bytes_names(self):
# mkdtemp can create directories when given all binary parts
d = tempfile.gettempdirb()
os.rmdir(self.do_create(dir=d))
os.rmdir(self.do_create(dir=d, pre=b"a"))
os.rmdir(self.do_create(dir=d, suf=b"b"))
os.rmdir(self.do_create(dir=d, pre=b"a", suf=b"b"))
os.rmdir(self.do_create(dir=d, pre=b"aa", suf=b".txt"))
with self.assertRaises(TypeError):
os.rmdir(self.do_create(dir=d, pre="aa", suf=b".txt"))
with self.assertRaises(TypeError):
os.rmdir(self.do_create(dir=d, pre=b"aa", suf=".txt"))
with self.assertRaises(TypeError):
os.rmdir(self.do_create(dir="", pre=b"aa", suf=b".txt"))
def test_basic_many(self):
# mkdtemp can create many directories (stochastic)
extant = list(range(TEST_FILES))

View File

@ -61,6 +61,9 @@ Core and Builtins
Library
-------
- Issue 24230: The tempfile module now accepts bytes for prefix, suffix and dir
parameters and returns bytes in such situations (matching the os module APIs).
- Issue 24244: Prevents termination when an invalid format string is
encountered on Windows in strftime.