new CSV file processing module - see PEP 305
This commit is contained in:
Skip Montanaro
parent
4cee220ff3
commit
b4a0417e91
|
|
@ -0,0 +1,281 @@
|
|||
\section{\module{csv} --- CSV File Reading and Writing}
|
||||
|
||||
\declaremodule{standard}{csv}
|
||||
\modulesynopsis{Write and read tabular data to and from delimited files.}
|
||||
|
||||
\versionadded{2.3}
|
||||
\index{csv}
|
||||
\indexii{data}{tabular}
|
||||
|
||||
The so-called CSV (Comma Separated Values) format is the most common import
|
||||
and export format for spreadsheets and databases. There is no ``CSV
|
||||
standard'', so the format is operationally defined by the many applications
|
||||
which read and write it. The lack of a standard means that subtle
|
||||
differences often exist in the data produced and consumed by different
|
||||
applications. These differences can make it annoying to process CSV files
|
||||
from multiple sources. Still, while the delimiters and quoting characters
|
||||
vary, the overall format is similar enough that it is possible to write a
|
||||
single module which can efficiently manipulate such data, hiding the details
|
||||
of reading and writing the data from the programmer.
|
||||
|
||||
The \module{csv} module implements classes to read and write tabular data in
|
||||
CSV format. It allows programmers to say, ``write this data in the format
|
||||
preferred by Excel,'' or ``read data from this file which was generated by
|
||||
Excel,'' without knowing the precise details of the CSV format used by
|
||||
Excel. Programmers can also describe the CSV formats understood by other
|
||||
applications or define their own special-purpose CSV formats.
|
||||
|
||||
The \module{csv} module's \class{reader} and \class{writer} objects read and
|
||||
write sequences. Programmers can also read and write data in dictionary
|
||||
form using the \class{DictReader} and \class{DictWriter} classes.
|
||||
|
||||
\note{The first version of the \module{csv} module doesn't support Unicode
|
||||
input. Also, there are currently some issues regarding \ASCII{} NUL
|
||||
characters. Accordingly, all input should generally be plain \ASCII{} to be
|
||||
safe. These restrictions will be removed in the future.}
|
||||
|
||||
\begin{seealso}
|
||||
% \seemodule{array}{Arrays of uniformly types numeric values.}
|
||||
\seepep{305}{CSV File API}
|
||||
{The Python Enhancement Proposal which proposed this addition
|
||||
to Python.}
|
||||
\end{seealso}
|
||||
|
||||
|
||||
\subsection{Module Contents}
|
||||
|
||||
|
||||
The \module{csv} module defines the following functions:
|
||||
|
||||
\begin{funcdesc}{reader}{csvfile\optional{,
|
||||
dialect=\code{'excel'}\optional{, fmtparam}}}
|
||||
Return a reader object which will iterate over lines in the given
|
||||
{}\var{csvfile}. \var{csvfile} can be any object which supports the
|
||||
iterator protocol and returns a string each time its \method{next}
|
||||
method is called. An optional \var{dialect} parameter can be given
|
||||
which is used to define a set of parameters specific to a particular CSV
|
||||
dialect. It may be an instance of a subclass of the \class{Dialect}
|
||||
class or one of the strings returned by the \function{list_dialects}
|
||||
function. The other optional {}\var{fmtparam} keyword arguments can be
|
||||
given to override individual formatting parameters in the current
|
||||
dialect. For more information about the dialect and formatting
|
||||
parameters, see section~\ref{fmt-params}, ``Dialects and Formatting
|
||||
Parameters'' for details of these parameters.
|
||||
|
||||
All data read are returned as strings. No automatic data type
|
||||
conversion is performed.
|
||||
\end{funcdesc}
|
||||
|
||||
\begin{funcdesc}{writer}{csvfile\optional{,
|
||||
dialect=\code{'excel'}\optional{, fmtparam}}}
|
||||
Return a writer object responsible for converting the user's data into
|
||||
delimited strings on the given file-like object. An optional
|
||||
{}\var{dialect} parameter can be given which is used to define a set of
|
||||
parameters specific to a particular CSV dialect. It may be an instance
|
||||
of a subclass of the \class{Dialect} class or one of the strings
|
||||
returned by the \function{list_dialects} function. The other optional
|
||||
{}\var{fmtparam} keyword arguments can be given to override individual
|
||||
formatting parameters in the current dialect. For more information
|
||||
about the dialect and formatting parameters, see
|
||||
section~\ref{fmt-params}, ``Dialects and Formatting Parameters'' for
|
||||
details of these parameters. To make it as easy as possible to
|
||||
interface with modules which implement the DB API, the value
|
||||
\constant{None} is written as the empty string. While this isn't a
|
||||
reversible transformation, it makes it easier to dump SQL NULL data values
|
||||
to CSV files without preprocessing the data returned from a
|
||||
\code{cursor.fetch*()} call. All other non-string data are stringified
|
||||
with \function{str()} before being written.
|
||||
\end{funcdesc}
|
||||
|
||||
\begin{funcdesc}{register_dialect}{name, dialect}
|
||||
Associate \var{dialect} with \var{name}. \var{dialect} must be a subclass
|
||||
of \class{csv.Dialect}. \var{name} must be a string or Unicode object.
|
||||
\end{funcdesc}
|
||||
|
||||
\begin{funcdesc}{unregister_dialect}{name}
|
||||
Delete the dialect associated with \var{name} from the dialect registry. An
|
||||
\exception{Error} is raised if \var{name} is not a registered dialect
|
||||
name.
|
||||
\end{funcdesc}
|
||||
|
||||
\begin{funcdesc}{get_dialect}{name}
|
||||
Return the dialect associated with \var{name}. An \exception{Error} is
|
||||
raised if \var{name} is not a registered dialect name.
|
||||
\end{funcdesc}
|
||||
|
||||
\begin{funcdesc}{list_dialects}{}
|
||||
Return the names of all registered dialects.
|
||||
\end{funcdesc}
|
||||
|
||||
|
||||
The \module{csv} module defines the following classes:
|
||||
|
||||
\begin{classdesc}{DictReader}{csvfile, fieldnames\optional{,
|
||||
restkey=\code{None}\optional{,
|
||||
restval=\code{None}\optional{,
|
||||
dialect=\code{'excel'}\optional{,
|
||||
fmtparam}}}}}
|
||||
Create an object which operates like a regular reader but maps the
|
||||
information read into a dict whose keys are given by the \var{fieldnames}
|
||||
parameter. If the row read has fewer fields than the fieldnames sequence,
|
||||
the value of \var{restval} will be used as the default value. If the row
|
||||
read has more fields than the fieldnames sequence, the remaining data is
|
||||
added as a sequence keyed by the value of \var{restkey}. If the row read
|
||||
has fewer fields than the fieldnames sequence, the remaining keys take the
|
||||
value of the optiona \var{restval} parameter. All other parameters are
|
||||
interpreted as for regular readers.
|
||||
\end{classdesc}
|
||||
|
||||
|
||||
\begin{classdesc}{DictWriter}{csvfile, fieldnames\optional{,
|
||||
restval=""\optional{,
|
||||
extrasaction=\code{'raise'}\optional{,
|
||||
dialect=\code{'excel'}\optional{, fmtparam}}}}}
|
||||
Create an object which operates like a regular writer but maps dictionaries
|
||||
onto output rows. The \var{fieldnames} parameter identifies the order in
|
||||
which values in the dictionary passed to the \method{writerow()} method are
|
||||
written to the \var{csvfile}. The optional \var{restval} parameter
|
||||
specifies the value to be written if the dictionary is missing a key in
|
||||
\var{fieldnames}. If the dictionary passed to the \method{writerow()}
|
||||
method contains a key not found in \var{fieldnames}, the optional
|
||||
\var{extrasaction} parameter indicates what action to take. If it is set
|
||||
to \code{'raise'} a \exception{ValueError} is raised. If it is set to
|
||||
\code{'ignore'}, extra values in the dictionary are ignored. All other
|
||||
parameters are interpreted as for regular writers.
|
||||
\end{classdesc}
|
||||
|
||||
|
||||
\begin{classdesc*}{Dialect}{}
|
||||
The \class{Dialect} class is a container class relied on primarily for its
|
||||
attributes, which are used to define the parameters for a specific
|
||||
\class{reader} or \class{writer} instance. Dialect objects support the
|
||||
following data attributes:
|
||||
|
||||
\begin{memberdesc}[string]{delimiter}
|
||||
A one-character string used to separate fields. It defaults to \code{","}.
|
||||
\end{memberdesc}
|
||||
|
||||
\begin{memberdesc}[boolean]{doublequote}
|
||||
Controls how instances of \var{quotechar} appearing inside a field should be
|
||||
themselves be quoted. When \constant{True}, the character is doubledd.
|
||||
When \constant{False}, the \var{escapechar} must be a one-character string
|
||||
which is used as a prefix to the \var{quotechar}. It defaults to
|
||||
\constant{True}.
|
||||
\end{memberdesc}
|
||||
|
||||
\begin{memberdesc}{escapechar}
|
||||
A one-character string used to escape the \var{delimiter} if \var{quoting}
|
||||
is set to \constant{QUOTE_NONE}. It defaults to \constant{None}.
|
||||
\end{memberdesc}
|
||||
|
||||
\begin{memberdesc}[string]{lineterminator}
|
||||
The string used to terminate lines in the CSV file. It defaults to
|
||||
\code{"\e r\e n"}.
|
||||
\end{memberdesc}
|
||||
|
||||
\begin{memberdesc}[string]{quotechar}
|
||||
A one-character string used to quote elements containing the \var{delimiter}
|
||||
or which start with the \var{quotechar}. It defaults to \code{'"'}.
|
||||
\end{memberdesc}
|
||||
|
||||
\begin{memberdesc}[integer]{quoting}
|
||||
Controls when quotes should be generated by the writer. It can take on any
|
||||
of the \code{QUOTE_*} constants defined below and defaults to
|
||||
\constant{QUOTE_MINIMAL}.
|
||||
\end{memberdesc}
|
||||
|
||||
\begin{memberdesc}[boolean]{skipinitialspace}
|
||||
When \constant{True}, whitespace immediately following the \var{delimiter}
|
||||
is ignored. The default is \constant{False}.
|
||||
\end{memberdesc}
|
||||
|
||||
\end{classdesc*}
|
||||
|
||||
The \module{csv} module defines the following constants:
|
||||
|
||||
\begin{datadesc}{QUOTE_ALWAYS}
|
||||
Instructs \class{writer} objects to quote all fields.
|
||||
\end{datadesc}
|
||||
|
||||
\begin{datadesc}{QUOTE_MINIMAL}
|
||||
Instructs \class{writer} objects to only quote those fields which contain
|
||||
the current \var{delimiter} or begin with the current \var{quotechar}.
|
||||
\end{datadesc}
|
||||
|
||||
\begin{datadesc}{QUOTE_NONNUMERIC}
|
||||
Instructs \class{writer} objects to quote all non-numeric fields.
|
||||
\end{datadesc}
|
||||
|
||||
\begin{datadesc}{QUOTE_NONE}
|
||||
Instructs \class{writer} objects to never quote fields. When the current
|
||||
\var{delimiter} occurs in output data it is preceded by the current
|
||||
\var{escapechar} character. When \constant{QUOTE_NONE} is in effect, it
|
||||
is an error not to have a single-character \var{escapechar} defined, even if
|
||||
no data to be written contains the \var{delimiter} character.
|
||||
\end{datadesc}
|
||||
|
||||
|
||||
The \module{csv} module defines the following exception:
|
||||
|
||||
\begin{excdesc}{Error}
|
||||
Raised by any of the functions when an error is detected.
|
||||
\end{excdesc}
|
||||
|
||||
|
||||
\subsection{Dialects and Formatting Parameters\label{fmt-params}}
|
||||
|
||||
To make it easier to specify the format of input and output records,
|
||||
specific formatting parameters are grouped together into dialects. A
|
||||
dialect is a subclass of the \class{Dialect} class having a set of specific
|
||||
methods and a single \method{validate()} method. When creating \class{reader}
|
||||
or \class{writer} objects, the programmer can specify a string or a subclass
|
||||
of the \class{Dialect} class as the dialect parameter. In addition to, or
|
||||
instead of, the \var{dialect} parameter, the programmer can also specify
|
||||
individual formatting parameters, which have the same names as the
|
||||
attributes defined above for the \class{Dialect} class.
|
||||
|
||||
|
||||
\subsection{Reader Objects}
|
||||
|
||||
\class{DictReader} and \var{reader} objects have the following public
|
||||
methods:
|
||||
|
||||
\begin{methoddesc}{next}{}
|
||||
Return the next row of the reader's iterable object as a list, parsed
|
||||
according to the current dialect.
|
||||
\end{methoddesc}
|
||||
|
||||
|
||||
\subsection{Writer Objects}
|
||||
|
||||
\class{DictWriter} and \var{writer} objects have the following public
|
||||
methods:
|
||||
|
||||
\begin{methoddesc}{writerow}{row}
|
||||
Write the \var{row} parameter to the writer's file object, formatted
|
||||
according to the current dialect.
|
||||
\end{methoddesc}
|
||||
|
||||
\begin{methoddesc}{writerows}{rows}
|
||||
Write all the \var{rows} parameters to the writer's file object, formatted
|
||||
according to the current dialect.
|
||||
\end{methoddesc}
|
||||
|
||||
|
||||
\subsection{Examples}
|
||||
|
||||
The ``Hello, world'' of csv reading is
|
||||
|
||||
\begin{verbatim}
|
||||
reader = csv.reader(file("some.csv"))
|
||||
for row in reader:
|
||||
print row
|
||||
\end{verbatim}
|
||||
|
||||
The corresponding simplest possible writing example is
|
||||
|
||||
\begin{verbatim}
|
||||
writer = csv.writer(file("some.csv", "w"))
|
||||
for row in someiterable:
|
||||
writer.writerow(row)
|
||||
\end{verbatim}
|
||||
|
|
@ -0,0 +1 @@
|
|||
|
||||
|
|
@ -0,0 +1,138 @@
|
|||
from _csv import Error, __version__, writer, reader, register_dialect, \
|
||||
unregister_dialect, get_dialect, list_dialects, \
|
||||
QUOTE_MINIMAL, QUOTE_ALL, QUOTE_NONNUMERIC, QUOTE_NONE, \
|
||||
__doc__
|
||||
|
||||
__all__ = [ "QUOTE_MINIMAL", "QUOTE_ALL", "QUOTE_NONNUMERIC", "QUOTE_NONE",
|
||||
"Error", "Dialect", "excel", "excel_tab", "reader", "writer",
|
||||
"register_dialect", "get_dialect", "list_dialects",
|
||||
"unregister_dialect", "__version__", "DictReader", "DictWriter" ]
|
||||
|
||||
class Dialect:
|
||||
_name = ""
|
||||
_valid = False
|
||||
# placeholders
|
||||
delimiter = None
|
||||
quotechar = None
|
||||
escapechar = None
|
||||
doublequote = None
|
||||
skipinitialspace = None
|
||||
lineterminator = None
|
||||
quoting = None
|
||||
|
||||
def __init__(self):
|
||||
if self.__class__ != Dialect:
|
||||
self._valid = True
|
||||
errors = self._validate()
|
||||
if errors != []:
|
||||
raise Error, "Dialect did not validate: %s" % ", ".join(errors)
|
||||
|
||||
def _validate(self):
|
||||
errors = []
|
||||
if not self._valid:
|
||||
errors.append("can't directly instantiate Dialect class")
|
||||
|
||||
if self.delimiter is None:
|
||||
errors.append("delimiter character not set")
|
||||
elif (not isinstance(self.delimiter, str) or
|
||||
len(self.delimiter) > 1):
|
||||
errors.append("delimiter must be one-character string")
|
||||
|
||||
if self.quotechar is None:
|
||||
if self.quoting != QUOTE_NONE:
|
||||
errors.append("quotechar not set")
|
||||
elif (not isinstance(self.quotechar, str) or
|
||||
len(self.quotechar) > 1):
|
||||
errors.append("quotechar must be one-character string")
|
||||
|
||||
if self.lineterminator is None:
|
||||
errors.append("lineterminator not set")
|
||||
elif not isinstance(self.lineterminator, str):
|
||||
errors.append("lineterminator must be a string")
|
||||
|
||||
if self.doublequote not in (True, False):
|
||||
errors.append("doublequote parameter must be True or False")
|
||||
|
||||
if self.skipinitialspace not in (True, False):
|
||||
errors.append("skipinitialspace parameter must be True or False")
|
||||
|
||||
if self.quoting is None:
|
||||
errors.append("quoting parameter not set")
|
||||
|
||||
if self.quoting is QUOTE_NONE:
|
||||
if (not isinstance(self.escapechar, (unicode, str)) or
|
||||
len(self.escapechar) > 1):
|
||||
errors.append("escapechar must be a one-character string or unicode object")
|
||||
|
||||
return errors
|
||||
|
||||
class excel(Dialect):
|
||||
delimiter = ','
|
||||
quotechar = '"'
|
||||
doublequote = True
|
||||
skipinitialspace = False
|
||||
lineterminator = '\r\n'
|
||||
quoting = QUOTE_MINIMAL
|
||||
register_dialect("excel", excel)
|
||||
|
||||
class excel_tab(excel):
|
||||
delimiter = '\t'
|
||||
register_dialect("excel-tab", excel_tab)
|
||||
|
||||
|
||||
class DictReader:
|
||||
def __init__(self, f, fieldnames, restkey=None, restval=None,
|
||||
dialect="excel", *args):
|
||||
self.fieldnames = fieldnames # list of keys for the dict
|
||||
self.restkey = restkey # key to catch long rows
|
||||
self.restval = restval # default value for short rows
|
||||
self.reader = reader(f, dialect, *args)
|
||||
|
||||
def __iter__(self):
|
||||
return self
|
||||
|
||||
def next(self):
|
||||
row = self.reader.next()
|
||||
# unlike the basic reader, we prefer not to return blanks,
|
||||
# because we will typically wind up with a dict full of None
|
||||
# values
|
||||
while row == []:
|
||||
row = self.reader.next()
|
||||
d = dict(zip(self.fieldnames, row))
|
||||
lf = len(self.fieldnames)
|
||||
lr = len(row)
|
||||
if lf < lr:
|
||||
d[self.restkey] = row[lf:]
|
||||
elif lf > lr:
|
||||
for key in self.fieldnames[lr:]:
|
||||
d[key] = self.restval
|
||||
return d
|
||||
|
||||
|
||||
class DictWriter:
|
||||
def __init__(self, f, fieldnames, restval="", extrasaction="raise",
|
||||
dialect="excel", *args):
|
||||
self.fieldnames = fieldnames # list of keys for the dict
|
||||
self.restval = restval # for writing short dicts
|
||||
if extrasaction.lower() not in ("raise", "ignore"):
|
||||
raise ValueError, \
|
||||
("extrasaction (%s) must be 'raise' or 'ignore'" %
|
||||
extrasaction)
|
||||
self.extrasaction = extrasaction
|
||||
self.writer = writer(f, dialect, *args)
|
||||
|
||||
def _dict_to_list(self, rowdict):
|
||||
if self.extrasaction == "raise":
|
||||
for k in rowdict.keys():
|
||||
if k not in self.fieldnames:
|
||||
raise ValueError, "dict contains fields not in fieldnames"
|
||||
return [rowdict.get(key, self.restval) for key in self.fieldnames]
|
||||
|
||||
def writerow(self, rowdict):
|
||||
return self.writer.writerow(self._dict_to_list(rowdict))
|
||||
|
||||
def writerows(self, rowdicts):
|
||||
rows = []
|
||||
for rowdict in rowdicts:
|
||||
rows.append(self._dict_to_list(rowdict))
|
||||
return self.writer.writerows(rows)
|
||||
|
|
@ -0,0 +1,619 @@
|
|||
# Copyright (C) 2001,2002 Python Software Foundation
|
||||
# csv package unit tests
|
||||
|
||||
import sys
|
||||
import unittest
|
||||
from StringIO import StringIO
|
||||
from csv import csv
|
||||
import gc
|
||||
|
||||
class Test_Csv(unittest.TestCase):
|
||||
"""
|
||||
Test the underlying C csv parser in ways that are not appropriate
|
||||
from the high level interface. Further tests of this nature are done
|
||||
in TestDialectRegistry.
|
||||
"""
|
||||
def test_reader_arg_valid(self):
|
||||
self.assertRaises(TypeError, csv.reader)
|
||||
self.assertRaises(TypeError, csv.reader, None)
|
||||
self.assertRaises(AttributeError, csv.reader, [], bad_attr = 0)
|
||||
self.assertRaises(csv.Error, csv.reader, [], 'foo')
|
||||
class BadClass:
|
||||
def __init__(self):
|
||||
raise IOError
|
||||
self.assertRaises(IOError, csv.reader, [], BadClass)
|
||||
self.assertRaises(TypeError, csv.reader, [], None)
|
||||
class BadDialect:
|
||||
bad_attr = 0
|
||||
self.assertRaises(AttributeError, csv.reader, [], BadDialect)
|
||||
|
||||
def test_writer_arg_valid(self):
|
||||
self.assertRaises(TypeError, csv.writer)
|
||||
self.assertRaises(TypeError, csv.writer, None)
|
||||
self.assertRaises(AttributeError, csv.writer, StringIO(), bad_attr = 0)
|
||||
|
||||
def _test_attrs(self, obj):
|
||||
self.assertEqual(obj.dialect.delimiter, ',')
|
||||
obj.dialect.delimiter = '\t'
|
||||
self.assertEqual(obj.dialect.delimiter, '\t')
|
||||
self.assertRaises(TypeError, delattr, obj.dialect, 'delimiter')
|
||||
self.assertRaises(TypeError, setattr, obj.dialect,
|
||||
'lineterminator', None)
|
||||
obj.dialect.escapechar = None
|
||||
self.assertEqual(obj.dialect.escapechar, None)
|
||||
self.assertRaises(TypeError, delattr, obj.dialect, 'quoting')
|
||||
self.assertRaises(TypeError, setattr, obj.dialect, 'quoting', None)
|
||||
obj.dialect.quoting = csv.QUOTE_MINIMAL
|
||||
self.assertEqual(obj.dialect.quoting, csv.QUOTE_MINIMAL)
|
||||
|
||||
def test_reader_attrs(self):
|
||||
self._test_attrs(csv.reader([]))
|
||||
|
||||
def test_writer_attrs(self):
|
||||
self._test_attrs(csv.writer(StringIO()))
|
||||
|
||||
def _write_test(self, fields, expect, **kwargs):
|
||||
fileobj = StringIO()
|
||||
writer = csv.writer(fileobj, **kwargs)
|
||||
writer.writerow(fields)
|
||||
self.assertEqual(fileobj.getvalue(),
|
||||
expect + writer.dialect.lineterminator)
|
||||
|
||||
def test_write_arg_valid(self):
|
||||
self.assertRaises(csv.Error, self._write_test, None, '')
|
||||
self._write_test((), '')
|
||||
self._write_test([None], '""')
|
||||
self.assertRaises(csv.Error, self._write_test,
|
||||
[None], None, quoting = csv.QUOTE_NONE)
|
||||
# Check that exceptions are passed up the chain
|
||||
class BadList:
|
||||
def __len__(self):
|
||||
return 10;
|
||||
def __getitem__(self, i):
|
||||
if i > 2:
|
||||
raise IOError
|
||||
self.assertRaises(IOError, self._write_test, BadList(), '')
|
||||
class BadItem:
|
||||
def __str__(self):
|
||||
raise IOError
|
||||
self.assertRaises(IOError, self._write_test, [BadItem()], '')
|
||||
|
||||
def test_write_bigfield(self):
|
||||
# This exercises the buffer realloc functionality
|
||||
bigstring = 'X' * 50000
|
||||
self._write_test([bigstring,bigstring], '%s,%s' % \
|
||||
(bigstring, bigstring))
|
||||
|
||||
def test_write_quoting(self):
|
||||
self._write_test(['a','1','p,q'], 'a,1,"p,q"')
|
||||
self.assertRaises(csv.Error,
|
||||
self._write_test,
|
||||
['a','1','p,q'], 'a,1,"p,q"',
|
||||
quoting = csv.QUOTE_NONE)
|
||||
self._write_test(['a','1','p,q'], 'a,1,"p,q"',
|
||||
quoting = csv.QUOTE_MINIMAL)
|
||||
self._write_test(['a','1','p,q'], '"a",1,"p,q"',
|
||||
quoting = csv.QUOTE_NONNUMERIC)
|
||||
self._write_test(['a','1','p,q'], '"a","1","p,q"',
|
||||
quoting = csv.QUOTE_ALL)
|
||||
|
||||
def test_write_escape(self):
|
||||
self._write_test(['a','1','p,q'], 'a,1,"p,q"',
|
||||
escapechar='\\')
|
||||
# FAILED - needs to be fixed [am]:
|
||||
# self._write_test(['a','1','p,"q"'], 'a,1,"p,\\"q\\"',
|
||||
# escapechar='\\', doublequote = 0)
|
||||
self._write_test(['a','1','p,q'], 'a,1,p\\,q',
|
||||
escapechar='\\', quoting = csv.QUOTE_NONE)
|
||||
|
||||
def test_writerows(self):
|
||||
class BrokenFile:
|
||||
def write(self, buf):
|
||||
raise IOError
|
||||
writer = csv.writer(BrokenFile())
|
||||
self.assertRaises(IOError, writer.writerows, [['a']])
|
||||
fileobj = StringIO()
|
||||
writer = csv.writer(fileobj)
|
||||
self.assertRaises(TypeError, writer.writerows, None)
|
||||
writer.writerows([['a','b'],['c','d']])
|
||||
self.assertEqual(fileobj.getvalue(), "a,b\r\nc,d\r\n")
|
||||
|
||||
def _read_test(self, input, expect, **kwargs):
|
||||
reader = csv.reader(input, **kwargs)
|
||||
result = list(reader)
|
||||
self.assertEqual(result, expect)
|
||||
|
||||
def test_read_oddinputs(self):
|
||||
self._read_test([], [])
|
||||
self._read_test([''], [[]])
|
||||
self.assertRaises(csv.Error, self._read_test,
|
||||
['"ab"c'], None, strict = 1)
|
||||
# cannot handle null bytes for the moment
|
||||
self.assertRaises(csv.Error, self._read_test,
|
||||
['ab\0c'], None, strict = 1)
|
||||
self._read_test(['"ab"c'], [['abc']], doublequote = 0)
|
||||
|
||||
def test_read_eol(self):
|
||||
self._read_test(['a,b'], [['a','b']])
|
||||
self._read_test(['a,b\n'], [['a','b']])
|
||||
self._read_test(['a,b\r\n'], [['a','b']])
|
||||
self._read_test(['a,b\r'], [['a','b']])
|
||||
self.assertRaises(csv.Error, self._read_test, ['a,b\rc,d'], [])
|
||||
self.assertRaises(csv.Error, self._read_test, ['a,b\nc,d'], [])
|
||||
self.assertRaises(csv.Error, self._read_test, ['a,b\r\nc,d'], [])
|
||||
|
||||
def test_read_escape(self):
|
||||
self._read_test(['a,\\b,c'], [['a', '\\b', 'c']], escapechar='\\')
|
||||
self._read_test(['a,b\\,c'], [['a', 'b,c']], escapechar='\\')
|
||||
self._read_test(['a,"b\\,c"'], [['a', 'b,c']], escapechar='\\')
|
||||
self._read_test(['a,"b,\\c"'], [['a', 'b,\\c']], escapechar='\\')
|
||||
self._read_test(['a,"b,c\\""'], [['a', 'b,c"']], escapechar='\\')
|
||||
self._read_test(['a,"b,c"\\'], [['a', 'b,c\\']], escapechar='\\')
|
||||
|
||||
def test_read_bigfield(self):
|
||||
# This exercises the buffer realloc functionality
|
||||
bigstring = 'X' * 50000
|
||||
bigline = '%s,%s' % (bigstring, bigstring)
|
||||
self._read_test([bigline], [[bigstring, bigstring]])
|
||||
|
||||
class TestDialectRegistry(unittest.TestCase):
|
||||
def test_registry_badargs(self):
|
||||
self.assertRaises(TypeError, csv.list_dialects, None)
|
||||
self.assertRaises(TypeError, csv.get_dialect)
|
||||
self.assertRaises(csv.Error, csv.get_dialect, None)
|
||||
self.assertRaises(csv.Error, csv.get_dialect, "nonesuch")
|
||||
self.assertRaises(TypeError, csv.unregister_dialect)
|
||||
self.assertRaises(csv.Error, csv.unregister_dialect, None)
|
||||
self.assertRaises(csv.Error, csv.unregister_dialect, "nonesuch")
|
||||
self.assertRaises(TypeError, csv.register_dialect, None)
|
||||
self.assertRaises(TypeError, csv.register_dialect, None, None)
|
||||
self.assertRaises(TypeError, csv.register_dialect, "nonesuch", None)
|
||||
class bogus:
|
||||
def __init__(self):
|
||||
raise KeyError
|
||||
self.assertRaises(KeyError, csv.register_dialect, "nonesuch", bogus)
|
||||
|
||||
def test_registry(self):
|
||||
class myexceltsv(csv.excel):
|
||||
delimiter = "\t"
|
||||
name = "myexceltsv"
|
||||
expected_dialects = csv.list_dialects() + [name]
|
||||
expected_dialects.sort()
|
||||
csv.register_dialect(name, myexceltsv)
|
||||
try:
|
||||
self.failUnless(isinstance(csv.get_dialect(name), myexceltsv))
|
||||
got_dialects = csv.list_dialects()
|
||||
got_dialects.sort()
|
||||
self.assertEqual(expected_dialects, got_dialects)
|
||||
finally:
|
||||
csv.unregister_dialect(name)
|
||||
|
||||
def test_incomplete_dialect(self):
|
||||
class myexceltsv(csv.Dialect):
|
||||
delimiter = "\t"
|
||||
self.assertRaises(csv.Error, myexceltsv)
|
||||
|
||||
def test_space_dialect(self):
|
||||
class space(csv.excel):
|
||||
delimiter = " "
|
||||
quoting = csv.QUOTE_NONE
|
||||
escapechar = "\\"
|
||||
|
||||
s = StringIO("abc def\nc1ccccc1 benzene\n")
|
||||
rdr = csv.reader(s, dialect=space())
|
||||
self.assertEqual(rdr.next(), ["abc", "def"])
|
||||
self.assertEqual(rdr.next(), ["c1ccccc1", "benzene"])
|
||||
|
||||
def test_dialect_apply(self):
|
||||
class testA(csv.excel):
|
||||
delimiter = "\t"
|
||||
class testB(csv.excel):
|
||||
delimiter = ":"
|
||||
class testC(csv.excel):
|
||||
delimiter = "|"
|
||||
|
||||
csv.register_dialect('testC', testC)
|
||||
try:
|
||||
fileobj = StringIO()
|
||||
writer = csv.writer(fileobj)
|
||||
writer.writerow([1,2,3])
|
||||
self.assertEqual(fileobj.getvalue(), "1,2,3\r\n")
|
||||
|
||||
fileobj = StringIO()
|
||||
writer = csv.writer(fileobj, testA)
|
||||
writer.writerow([1,2,3])
|
||||
self.assertEqual(fileobj.getvalue(), "1\t2\t3\r\n")
|
||||
|
||||
fileobj = StringIO()
|
||||
writer = csv.writer(fileobj, dialect=testB())
|
||||
writer.writerow([1,2,3])
|
||||
self.assertEqual(fileobj.getvalue(), "1:2:3\r\n")
|
||||
|
||||
fileobj = StringIO()
|
||||
writer = csv.writer(fileobj, dialect='testC')
|
||||
writer.writerow([1,2,3])
|
||||
self.assertEqual(fileobj.getvalue(), "1|2|3\r\n")
|
||||
|
||||
fileobj = StringIO()
|
||||
writer = csv.writer(fileobj, dialect=testA, delimiter=';')
|
||||
writer.writerow([1,2,3])
|
||||
self.assertEqual(fileobj.getvalue(), "1;2;3\r\n")
|
||||
finally:
|
||||
csv.unregister_dialect('testC')
|
||||
|
||||
def test_bad_dialect(self):
|
||||
# Unknown parameter
|
||||
self.assertRaises(AttributeError, csv.reader, [], bad_attr = 0)
|
||||
# Bad values
|
||||
self.assertRaises(TypeError, csv.reader, [], delimiter = None)
|
||||
self.assertRaises(TypeError, csv.reader, [], quoting = -1)
|
||||
self.assertRaises(TypeError, csv.reader, [], quoting = 100)
|
||||
|
||||
class TestCsvBase(unittest.TestCase):
|
||||
def readerAssertEqual(self, input, expected_result):
|
||||
reader = csv.reader(StringIO(input), dialect = self.dialect)
|
||||
fields = list(reader)
|
||||
self.assertEqual(fields, expected_result)
|
||||
|
||||
def writerAssertEqual(self, input, expected_result):
|
||||
fileobj = StringIO()
|
||||
writer = csv.writer(fileobj, dialect = self.dialect)
|
||||
writer.writerows(input)
|
||||
self.assertEqual(fileobj.getvalue(), expected_result)
|
||||
|
||||
class TestDialectExcel(TestCsvBase):
|
||||
dialect = 'excel'
|
||||
|
||||
def test_single(self):
|
||||
self.readerAssertEqual('abc', [['abc']])
|
||||
|
||||
def test_simple(self):
|
||||
self.readerAssertEqual('1,2,3,4,5', [['1','2','3','4','5']])
|
||||
|
||||
def test_blankline(self):
|
||||
self.readerAssertEqual('', [])
|
||||
|
||||
def test_empty_fields(self):
|
||||
self.readerAssertEqual(',', [['', '']])
|
||||
|
||||
def test_singlequoted(self):
|
||||
self.readerAssertEqual('""', [['']])
|
||||
|
||||
def test_singlequoted_left_empty(self):
|
||||
self.readerAssertEqual('"",', [['','']])
|
||||
|
||||
def test_singlequoted_right_empty(self):
|
||||
self.readerAssertEqual(',""', [['','']])
|
||||
|
||||
def test_single_quoted_quote(self):
|
||||
self.readerAssertEqual('""""', [['"']])
|
||||
|
||||
def test_quoted_quotes(self):
|
||||
self.readerAssertEqual('""""""', [['""']])
|
||||
|
||||
def test_inline_quote(self):
|
||||
self.readerAssertEqual('a""b', [['a""b']])
|
||||
|
||||
def test_inline_quotes(self):
|
||||
self.readerAssertEqual('a"b"c', [['a"b"c']])
|
||||
|
||||
def test_quotes_and_more(self):
|
||||
self.readerAssertEqual('"a"b', [['ab']])
|
||||
|
||||
def test_lone_quote(self):
|
||||
self.readerAssertEqual('a"b', [['a"b']])
|
||||
|
||||
def test_quote_and_quote(self):
|
||||
self.readerAssertEqual('"a" "b"', [['a "b"']])
|
||||
|
||||
def test_space_and_quote(self):
|
||||
self.readerAssertEqual(' "a"', [[' "a"']])
|
||||
|
||||
def test_quoted(self):
|
||||
self.readerAssertEqual('1,2,3,"I think, therefore I am",5,6',
|
||||
[['1', '2', '3',
|
||||
'I think, therefore I am',
|
||||
'5', '6']])
|
||||
|
||||
def test_quoted_quote(self):
|
||||
self.readerAssertEqual('1,2,3,"""I see,"" said the blind man","as he picked up his hammer and saw"',
|
||||
[['1', '2', '3',
|
||||
'"I see," said the blind man',
|
||||
'as he picked up his hammer and saw']])
|
||||
|
||||
def test_quoted_nl(self):
|
||||
input = '''\
|
||||
1,2,3,"""I see,""
|
||||
said the blind man","as he picked up his
|
||||
hammer and saw"
|
||||
9,8,7,6'''
|
||||
self.readerAssertEqual(input,
|
||||
[['1', '2', '3',
|
||||
'"I see,"\nsaid the blind man',
|
||||
'as he picked up his\nhammer and saw'],
|
||||
['9','8','7','6']])
|
||||
|
||||
def test_dubious_quote(self):
|
||||
self.readerAssertEqual('12,12,1",', [['12', '12', '1"', '']])
|
||||
|
||||
def test_null(self):
|
||||
self.writerAssertEqual([], '')
|
||||
|
||||
def test_single(self):
|
||||
self.writerAssertEqual([['abc']], 'abc\r\n')
|
||||
|
||||
def test_simple(self):
|
||||
self.writerAssertEqual([[1, 2, 'abc', 3, 4]], '1,2,abc,3,4\r\n')
|
||||
|
||||
def test_quotes(self):
|
||||
self.writerAssertEqual([[1, 2, 'a"bc"', 3, 4]], '1,2,"a""bc""",3,4\r\n')
|
||||
|
||||
def test_quote_fieldsep(self):
|
||||
self.writerAssertEqual([['abc,def']], '"abc,def"\r\n')
|
||||
|
||||
def test_newlines(self):
|
||||
self.writerAssertEqual([[1, 2, 'a\nbc', 3, 4]], '1,2,"a\nbc",3,4\r\n')
|
||||
|
||||
class EscapedExcel(csv.excel):
|
||||
quoting = csv.QUOTE_NONE
|
||||
escapechar = '\\'
|
||||
|
||||
class TestEscapedExcel(TestCsvBase):
|
||||
dialect = EscapedExcel()
|
||||
|
||||
def test_escape_fieldsep(self):
|
||||
self.writerAssertEqual([['abc,def']], 'abc\\,def\r\n')
|
||||
|
||||
def test_read_escape_fieldsep(self):
|
||||
self.readerAssertEqual('abc\\,def\r\n', [['abc,def']])
|
||||
|
||||
class QuotedEscapedExcel(csv.excel):
|
||||
quoting = csv.QUOTE_NONNUMERIC
|
||||
escapechar = '\\'
|
||||
|
||||
class TestQuotedEscapedExcel(TestCsvBase):
|
||||
dialect = QuotedEscapedExcel()
|
||||
|
||||
def test_write_escape_fieldsep(self):
|
||||
self.writerAssertEqual([['abc,def']], '"abc,def"\r\n')
|
||||
|
||||
def test_read_escape_fieldsep(self):
|
||||
self.readerAssertEqual('"abc\\,def"\r\n', [['abc,def']])
|
||||
|
||||
# Disabled, pending support in csv.utils module
|
||||
class TestDictFields(unittest.TestCase):
|
||||
### "long" means the row is longer than the number of fieldnames
|
||||
### "short" means there are fewer elements in the row than fieldnames
|
||||
def test_write_simple_dict(self):
|
||||
fileobj = StringIO()
|
||||
writer = csv.DictWriter(fileobj, fieldnames = ["f1", "f2", "f3"])
|
||||
writer.writerow({"f1": 10, "f3": "abc"})
|
||||
self.assertEqual(fileobj.getvalue(), "10,,abc\r\n")
|
||||
|
||||
def test_write_no_fields(self):
|
||||
fileobj = StringIO()
|
||||
self.assertRaises(TypeError, csv.DictWriter, fileobj)
|
||||
|
||||
def test_read_dict_fields(self):
|
||||
reader = csv.DictReader(StringIO("1,2,abc\r\n"),
|
||||
fieldnames=["f1", "f2", "f3"])
|
||||
self.assertEqual(reader.next(), {"f1": '1', "f2": '2', "f3": 'abc'})
|
||||
|
||||
def test_read_long(self):
|
||||
reader = csv.DictReader(StringIO("1,2,abc,4,5,6\r\n"),
|
||||
fieldnames=["f1", "f2"])
|
||||
self.assertEqual(reader.next(), {"f1": '1', "f2": '2',
|
||||
None: ["abc", "4", "5", "6"]})
|
||||
|
||||
def test_read_long_with_rest(self):
|
||||
reader = csv.DictReader(StringIO("1,2,abc,4,5,6\r\n"),
|
||||
fieldnames=["f1", "f2"], restkey="_rest")
|
||||
self.assertEqual(reader.next(), {"f1": '1', "f2": '2',
|
||||
"_rest": ["abc", "4", "5", "6"]})
|
||||
|
||||
def test_read_short(self):
|
||||
reader = csv.DictReader(["1,2,abc,4,5,6\r\n","1,2,abc\r\n"],
|
||||
fieldnames="1 2 3 4 5 6".split(),
|
||||
restval="DEFAULT")
|
||||
self.assertEqual(reader.next(), {"1": '1', "2": '2', "3": 'abc',
|
||||
"4": '4', "5": '5', "6": '6'})
|
||||
self.assertEqual(reader.next(), {"1": '1', "2": '2', "3": 'abc',
|
||||
"4": 'DEFAULT', "5": 'DEFAULT',
|
||||
"6": 'DEFAULT'})
|
||||
|
||||
def test_read_with_blanks(self):
|
||||
reader = csv.DictReader(["1,2,abc,4,5,6\r\n","\r\n",
|
||||
"1,2,abc,4,5,6\r\n"],
|
||||
fieldnames="1 2 3 4 5 6".split())
|
||||
self.assertEqual(reader.next(), {"1": '1', "2": '2', "3": 'abc',
|
||||
"4": '4', "5": '5', "6": '6'})
|
||||
self.assertEqual(reader.next(), {"1": '1', "2": '2', "3": 'abc',
|
||||
"4": '4', "5": '5', "6": '6'})
|
||||
|
||||
class TestArrayWrites(unittest.TestCase):
|
||||
def test_int_write(self):
|
||||
import array
|
||||
contents = [(20-i) for i in range(20)]
|
||||
a = array.array('i', contents)
|
||||
fileobj = StringIO()
|
||||
writer = csv.writer(fileobj, dialect="excel")
|
||||
writer.writerow(a)
|
||||
expected = ",".join([str(i) for i in a])+"\r\n"
|
||||
self.assertEqual(fileobj.getvalue(), expected)
|
||||
|
||||
def test_double_write(self):
|
||||
import array
|
||||
contents = [(20-i)*0.1 for i in range(20)]
|
||||
a = array.array('d', contents)
|
||||
fileobj = StringIO()
|
||||
writer = csv.writer(fileobj, dialect="excel")
|
||||
writer.writerow(a)
|
||||
expected = ",".join([str(i) for i in a])+"\r\n"
|
||||
self.assertEqual(fileobj.getvalue(), expected)
|
||||
|
||||
def test_float_write(self):
|
||||
import array
|
||||
contents = [(20-i)*0.1 for i in range(20)]
|
||||
a = array.array('f', contents)
|
||||
fileobj = StringIO()
|
||||
writer = csv.writer(fileobj, dialect="excel")
|
||||
writer.writerow(a)
|
||||
expected = ",".join([str(i) for i in a])+"\r\n"
|
||||
self.assertEqual(fileobj.getvalue(), expected)
|
||||
|
||||
def test_char_write(self):
|
||||
import array, string
|
||||
a = array.array('c', string.letters)
|
||||
fileobj = StringIO()
|
||||
writer = csv.writer(fileobj, dialect="excel")
|
||||
writer.writerow(a)
|
||||
expected = ",".join(a)+"\r\n"
|
||||
self.assertEqual(fileobj.getvalue(), expected)
|
||||
|
||||
class TestDialectValidity(unittest.TestCase):
|
||||
def test_quoting(self):
|
||||
class mydialect(csv.Dialect):
|
||||
delimiter = ";"
|
||||
escapechar = '\\'
|
||||
doublequote = False
|
||||
skipinitialspace = True
|
||||
lineterminator = '\r\n'
|
||||
quoting = csv.QUOTE_NONE
|
||||
d = mydialect()
|
||||
|
||||
mydialect.quoting = None
|
||||
self.assertRaises(csv.Error, mydialect)
|
||||
|
||||
mydialect.quoting = csv.QUOTE_NONE
|
||||
mydialect.escapechar = None
|
||||
self.assertRaises(csv.Error, mydialect)
|
||||
|
||||
mydialect.doublequote = True
|
||||
mydialect.quoting = csv.QUOTE_ALL
|
||||
mydialect.quotechar = '"'
|
||||
d = mydialect()
|
||||
|
||||
mydialect.quotechar = "''"
|
||||
self.assertRaises(csv.Error, mydialect)
|
||||
|
||||
mydialect.quotechar = 4
|
||||
self.assertRaises(csv.Error, mydialect)
|
||||
|
||||
def test_delimiter(self):
|
||||
class mydialect(csv.Dialect):
|
||||
delimiter = ";"
|
||||
escapechar = '\\'
|
||||
doublequote = False
|
||||
skipinitialspace = True
|
||||
lineterminator = '\r\n'
|
||||
quoting = csv.QUOTE_NONE
|
||||
d = mydialect()
|
||||
|
||||
mydialect.delimiter = ":::"
|
||||
self.assertRaises(csv.Error, mydialect)
|
||||
|
||||
mydialect.delimiter = 4
|
||||
self.assertRaises(csv.Error, mydialect)
|
||||
|
||||
def test_lineterminator(self):
|
||||
class mydialect(csv.Dialect):
|
||||
delimiter = ";"
|
||||
escapechar = '\\'
|
||||
doublequote = False
|
||||
skipinitialspace = True
|
||||
lineterminator = '\r\n'
|
||||
quoting = csv.QUOTE_NONE
|
||||
d = mydialect()
|
||||
|
||||
mydialect.lineterminator = ":::"
|
||||
d = mydialect()
|
||||
|
||||
mydialect.lineterminator = 4
|
||||
self.assertRaises(csv.Error, mydialect)
|
||||
|
||||
|
||||
if not hasattr(sys, "gettotalrefcount"):
|
||||
print "*** skipping leakage tests ***"
|
||||
else:
|
||||
class NUL:
|
||||
def write(s, *args):
|
||||
pass
|
||||
writelines = write
|
||||
|
||||
class TestLeaks(unittest.TestCase):
|
||||
def test_create_read(self):
|
||||
delta = 0
|
||||
lastrc = sys.gettotalrefcount()
|
||||
for i in xrange(20):
|
||||
gc.collect()
|
||||
self.assertEqual(gc.garbage, [])
|
||||
rc = sys.gettotalrefcount()
|
||||
csv.reader(["a,b,c\r\n"])
|
||||
csv.reader(["a,b,c\r\n"])
|
||||
csv.reader(["a,b,c\r\n"])
|
||||
delta = rc-lastrc
|
||||
lastrc = rc
|
||||
# if csv.reader() leaks, last delta should be 3 or more
|
||||
self.assertEqual(delta < 3, True)
|
||||
|
||||
def test_create_write(self):
|
||||
delta = 0
|
||||
lastrc = sys.gettotalrefcount()
|
||||
s = NUL()
|
||||
for i in xrange(20):
|
||||
gc.collect()
|
||||
self.assertEqual(gc.garbage, [])
|
||||
rc = sys.gettotalrefcount()
|
||||
csv.writer(s)
|
||||
csv.writer(s)
|
||||
csv.writer(s)
|
||||
delta = rc-lastrc
|
||||
lastrc = rc
|
||||
# if csv.writer() leaks, last delta should be 3 or more
|
||||
self.assertEqual(delta < 3, True)
|
||||
|
||||
def test_read(self):
|
||||
delta = 0
|
||||
rows = ["a,b,c\r\n"]*5
|
||||
lastrc = sys.gettotalrefcount()
|
||||
for i in xrange(20):
|
||||
gc.collect()
|
||||
self.assertEqual(gc.garbage, [])
|
||||
rc = sys.gettotalrefcount()
|
||||
rdr = csv.reader(rows)
|
||||
for row in rdr:
|
||||
pass
|
||||
delta = rc-lastrc
|
||||
lastrc = rc
|
||||
# if reader leaks during read, delta should be 5 or more
|
||||
self.assertEqual(delta < 5, True)
|
||||
|
||||
def test_write(self):
|
||||
delta = 0
|
||||
rows = [[1,2,3]]*5
|
||||
s = NUL()
|
||||
lastrc = sys.gettotalrefcount()
|
||||
for i in xrange(20):
|
||||
gc.collect()
|
||||
self.assertEqual(gc.garbage, [])
|
||||
rc = sys.gettotalrefcount()
|
||||
writer = csv.writer(s)
|
||||
for row in rows:
|
||||
writer.writerow(row)
|
||||
delta = rc-lastrc
|
||||
lastrc = rc
|
||||
# if writer leaks during write, last delta should be 5 or more
|
||||
self.assertEqual(delta < 5, True)
|
||||
|
||||
def _testclasses():
|
||||
mod = sys.modules[__name__]
|
||||
return [getattr(mod, name) for name in dir(mod) if name.startswith('Test')]
|
||||
|
||||
def suite():
|
||||
suite = unittest.TestSuite()
|
||||
for testclass in _testclasses():
|
||||
suite.addTest(unittest.makeSuite(testclass))
|
||||
return suite
|
||||
|
||||
if __name__ == '__main__':
|
||||
unittest.main(defaultTest='suite')
|
||||
File diff suppressed because it is too large
Load Diff
Loading…
Reference in New Issue