\section{\module{codecs} --- Codec registry and base classes} \declaremodule{standard}{codecs} \modulesynopsis{Encode and decode data and streams.} \moduleauthor{Marc-Andre Lemburg}{mal@lemburg.com} \sectionauthor{Marc-Andre Lemburg}{mal@lemburg.com} \index{Unicode} \index{Codecs} \indexii{Codecs}{encode} \indexii{Codecs}{decode} \index{streams} \indexii{stackable}{streams} This module defines base classes for standard Python codecs (encoders and decoders) and provides access to the internal Python codec registry which manages the codec and error handling lookup process. It defines the following functions: \begin{funcdesc}{register}{search_function} Register a codec search function. Search functions are expected to take one argument, the encoding name in all lower case letters, and return a tuple of functions \code{(\var{encoder}, \var{decoder}, \var{stream_reader}, \var{stream_writer})} taking the following arguments: \var{encoder} and \var{decoder}: These must be functions or methods which have the same interface as the \method{encode()}/\method{decode()} methods of Codec instances (see Codec Interface). The functions/methods are expected to work in a stateless mode. \var{stream_reader} and \var{stream_writer}: These have to be factory functions providing the following interface: \code{factory(\var{stream}, \var{errors}='strict')} The factory functions must return objects providing the interfaces defined by the base classes \class{StreamWriter} and \class{StreamReader}, respectively. Stream codecs can maintain state. Possible values for errors are \code{'strict'} (raise an exception in case of an encoding error), \code{'replace'} (replace malformed data with a suitable replacement marker, such as \character{?}), \code{'ignore'} (ignore malformed data and continue without further notice), \code{'xmlcharrefreplace'} (replace with the appropriate XML character reference (for encoding only)) and \code{'backslashreplace'} (replace with backslashed escape sequences (for encoding only)) as well as any other error handling name defined via \function{register_error()}. In case a search function cannot find a given encoding, it should return \code{None}. \end{funcdesc} \begin{funcdesc}{lookup}{encoding} Looks up a codec tuple in the Python codec registry and returns the function tuple as defined above. Encodings are first looked up in the registry's cache. If not found, the list of registered search functions is scanned. If no codecs tuple is found, a \exception{LookupError} is raised. Otherwise, the codecs tuple is stored in the cache and returned to the caller. \end{funcdesc} To simplify access to the various codecs, the module provides these additional functions which use \function{lookup()} for the codec lookup: \begin{funcdesc}{getencoder}{encoding} Lookup up the codec for the given encoding and return its encoder function. Raises a \exception{LookupError} in case the encoding cannot be found. \end{funcdesc} \begin{funcdesc}{getdecoder}{encoding} Lookup up the codec for the given encoding and return its decoder function. Raises a \exception{LookupError} in case the encoding cannot be found. \end{funcdesc} \begin{funcdesc}{getreader}{encoding} Lookup up the codec for the given encoding and return its StreamReader class or factory function. Raises a \exception{LookupError} in case the encoding cannot be found. \end{funcdesc} \begin{funcdesc}{getwriter}{encoding} Lookup up the codec for the given encoding and return its StreamWriter class or factory function. Raises a \exception{LookupError} in case the encoding cannot be found. \end{funcdesc} \begin{funcdesc}{register_error}{name, error_handler} Register the error handling function \var{error_handler} under the name \var{name}. \var{error_handler} will be called during encoding and decoding in case of an error, when \var{name} is specified as the errors parameter. For encoding \var{error_handler} will be called with a \exception{UnicodeEncodeError} instance, which contains information about the location of the error. The error handler must either raise this or a different exception or return a tuple with a replacement for the unencodable part of the input and a position where encoding should continue. The encoder will encode the replacement and continue encoding the original input at the specified position. Negative position values will be treated as being relative to the end of the input string. If the resulting position is out of bound an IndexError will be raised. Decoding and translating works similar, except \exception{UnicodeDecodeError} or \exception{UnicodeTranslateError} will be passed to the handler and that the replacement from the error handler will be put into the output directly. \end{funcdesc} \begin{funcdesc}{lookup_error}{name} Return the error handler previously register under the name \var{name}. Raises a \exception{LookupError} in case the handler cannot be found. \end{funcdesc} \begin{funcdesc}{strict_errors}{exception} Implements the \code{strict} error handling. \end{funcdesc} \begin{funcdesc}{replace_errors}{exception} Implements the \code{replace} error handling. \end{funcdesc} \begin{funcdesc}{ignore_errors}{exception} Implements the \code{ignore} error handling. \end{funcdesc} \begin{funcdesc}{xmlcharrefreplace_errors_errors}{exception} Implements the \code{xmlcharrefreplace} error handling. \end{funcdesc} \begin{funcdesc}{backslashreplace_errors_errors}{exception} Implements the \code{backslashreplace} error handling. \end{funcdesc} To simplify working with encoded files or stream, the module also defines these utility functions: \begin{funcdesc}{open}{filename, mode\optional{, encoding\optional{, errors\optional{, buffering}}}} Open an encoded file using the given \var{mode} and return a wrapped version providing transparent encoding/decoding. \note{The wrapped version will only accept the object format defined by the codecs, i.e.\ Unicode objects for most built-in codecs. Output is also codec-dependent and will usually be Unicode as well.} \var{encoding} specifies the encoding which is to be used for the the file. \var{errors} may be given to define the error handling. It defaults to \code{'strict'} which causes a \exception{ValueError} to be raised in case an encoding error occurs. \var{buffering} has the same meaning as for the built-in \function{open()} function. It defaults to line buffered. \end{funcdesc} \begin{funcdesc}{EncodedFile}{file, input\optional{, output\optional{, errors}}} Return a wrapped version of file which provides transparent encoding translation. Strings written to the wrapped file are interpreted according to the given \var{input} encoding and then written to the original file as strings using the \var{output} encoding. The intermediate encoding will usually be Unicode but depends on the specified codecs. If \var{output} is not given, it defaults to \var{input}. \var{errors} may be given to define the error handling. It defaults to \code{'strict'}, which causes \exception{ValueError} to be raised in case an encoding error occurs. \end{funcdesc} The module also provides the following constants which are useful for reading and writing to platform dependent files: \begin{datadesc}{BOM} \dataline{BOM_BE} \dataline{BOM_LE} \dataline{BOM_UTF8} \dataline{BOM_UTF16} \dataline{BOM_UTF16_BE} \dataline{BOM_UTF16_LE} \dataline{BOM_UTF32} \dataline{BOM_UTF32_BE} \dataline{BOM_UTF32_LE} These constants define various encodings of the Unicode byte order mark (BOM) used in UTF-16 and UTF-32 data streams to indicate the byte order used in the stream or file and in UTF-8 as a Unicode signature. \constant{BOM_UTF16} is either \constant{BOM_UTF16_BE} or \constant{BOM_UTF16_LE} depending on the platform's native byte order, \constant{BOM} is an alias for \constant{BOM_UTF16}, \constant{BOM_LE} for \constant{BOM_UTF16_LE} and \constant{BOM_BE} for \constant{BOM_UTF16_BE}. The others represent the BOM in UTF-8 and UTF-32 encodings. \end{datadesc} \begin{seealso} \seeurl{http://sourceforge.net/projects/python-codecs/}{A SourceForge project working on additional support for Asian codecs for use with Python. They are in the early stages of development at the time of this writing --- look in their FTP area for downloadable files.} \end{seealso} \subsection{Codec Base Classes} The \module{codecs} defines a set of base classes which define the interface and can also be used to easily write you own codecs for use in Python. Each codec has to define four interfaces to make it usable as codec in Python: stateless encoder, stateless decoder, stream reader and stream writer. The stream reader and writers typically reuse the stateless encoder/decoder to implement the file protocols. The \class{Codec} class defines the interface for stateless encoders/decoders. To simplify and standardize error handling, the \method{encode()} and \method{decode()} methods may implement different error handling schemes by providing the \var{errors} string argument. The following string values are defined and implemented by all standard Python codecs: \begin{tableii}{l|l}{code}{Value}{Meaning} \lineii{'strict'}{Raise \exception{UnicodeError} (or a subclass); this is the default.} \lineii{'ignore'}{Ignore the character and continue with the next.} \lineii{'replace'}{Replace with a suitable replacement character; Python will use the official U+FFFD REPLACEMENT CHARACTER for the built-in Unicode codecs on decoding and '?' on encoding.} \lineii{'xmlcharrefreplace'}{Replace with the appropriate XML character reference (only for encoding).} \lineii{'backslashreplace'}{Replace with backslashed escape sequences (only for encoding).} \end{tableii} The set of allowed values can be extended via \method{register_error}. \subsubsection{Codec Objects \label{codec-objects}} The \class{Codec} class defines these methods which also define the function interfaces of the stateless encoder and decoder: \begin{methoddesc}{encode}{input\optional{, errors}} Encodes the object \var{input} and returns a tuple (output object, length consumed). While codecs are not restricted to use with Unicode, in a Unicode context, encoding converts a Unicode object to a plain string using a particular character set encoding (e.g., \code{cp1252} or \code{iso-8859-1}). \var{errors} defines the error handling to apply. It defaults to \code{'strict'} handling. The method may not store state in the \class{Codec} instance. Use \class{StreamCodec} for codecs which have to keep state in order to make encoding/decoding efficient. The encoder must be able to handle zero length input and return an empty object of the output object type in this situation. \end{methoddesc} \begin{methoddesc}{decode}{input\optional{, errors}} Decodes the object \var{input} and returns a tuple (output object, length consumed). In a Unicode context, decoding converts a plain string encoded using a particular character set encoding to a Unicode object. \var{input} must be an object which provides the \code{bf_getreadbuf} buffer slot. Python strings, buffer objects and memory mapped files are examples of objects providing this slot. \var{errors} defines the error handling to apply. It defaults to \code{'strict'} handling. The method may not store state in the \class{Codec} instance. Use \class{StreamCodec} for codecs which have to keep state in order to make encoding/decoding efficient. The decoder must be able to handle zero length input and return an empty object of the output object type in this situation. \end{methoddesc} The \class{StreamWriter} and \class{StreamReader} classes provide generic working interfaces which can be used to implement new encodings submodules very easily. See \module{encodings.utf_8} for an example on how this is done. \subsubsection{StreamWriter Objects \label{stream-writer-objects}} The \class{StreamWriter} class is a subclass of \class{Codec} and defines the following methods which every stream writer must define in order to be compatible to the Python codec registry. \begin{classdesc}{StreamWriter}{stream\optional{, errors}} Constructor for a \class{StreamWriter} instance. All stream writers must provide this constructor interface. They are free to add additional keyword arguments, but only the ones defined here are used by the Python codec registry. \var{stream} must be a file-like object open for writing (binary) data. The \class{StreamWriter} may implement different error handling schemes by providing the \var{errors} keyword argument. These parameters are predefined: \begin{itemize} \item \code{'strict'} Raise \exception{ValueError} (or a subclass); this is the default. \item \code{'ignore'} Ignore the character and continue with the next. \item \code{'replace'} Replace with a suitable replacement character \item \code{'xmlcharrefreplace'} Replace with the appropriate XML character reference \item \code{'backslashreplace'} Replace with backslashed escape sequences. \end{itemize} The \var{errors} argument will be assigned to an attribute of the same name. Assigning to this attribute makes it possible to switch between different error handling strategies during the lifetime of the \class{StreamWriter} object. The set of allowed values for the \var{errors} argument can be extended with \function{register_error()}. \end{classdesc} \begin{methoddesc}{write}{object} Writes the object's contents encoded to the stream. \end{methoddesc} \begin{methoddesc}{writelines}{list} Writes the concatenated list of strings to the stream (possibly by reusing the \method{write()} method). \end{methoddesc} \begin{methoddesc}{reset}{} Flushes and resets the codec buffers used for keeping state. Calling this method should ensure that the data on the output is put into a clean state, that allows appending of new fresh data without having to rescan the whole stream to recover state. \end{methoddesc} In addition to the above methods, the \class{StreamWriter} must also inherit all other methods and attribute from the underlying stream. \subsubsection{StreamReader Objects \label{stream-reader-objects}} The \class{StreamReader} class is a subclass of \class{Codec} and defines the following methods which every stream reader must define in order to be compatible to the Python codec registry. \begin{classdesc}{StreamReader}{stream\optional{, errors}} Constructor for a \class{StreamReader} instance. All stream readers must provide this constructor interface. They are free to add additional keyword arguments, but only the ones defined here are used by the Python codec registry. \var{stream} must be a file-like object open for reading (binary) data. The \class{StreamReader} may implement different error handling schemes by providing the \var{errors} keyword argument. These parameters are defined: \begin{itemize} \item \code{'strict'} Raise \exception{ValueError} (or a subclass); this is the default. \item \code{'ignore'} Ignore the character and continue with the next. \item \code{'replace'} Replace with a suitable replacement character. \end{itemize} The \var{errors} argument will be assigned to an attribute of the same name. Assigning to this attribute makes it possible to switch between different error handling strategies during the lifetime of the \class{StreamReader} object. The set of allowed values for the \var{errors} argument can be extended with \function{register_error()}. \end{classdesc} \begin{methoddesc}{read}{\optional{size}} Decodes data from the stream and returns the resulting object. \var{size} indicates the approximate maximum number of bytes to read from the stream for decoding purposes. The decoder can modify this setting as appropriate. The default value -1 indicates to read and decode as much as possible. \var{size} is intended to prevent having to decode huge files in one step. The method should use a greedy read strategy meaning that it should read as much data as is allowed within the definition of the encoding and the given size, e.g. if optional encoding endings or state markers are available on the stream, these should be read too. \end{methoddesc} \begin{methoddesc}{readline}{[size]} Read one line from the input stream and return the decoded data. Unlike the \method{readlines()} method, this method inherits the line breaking knowledge from the underlying stream's \method{readline()} method -- there is currently no support for line breaking using the codec decoder due to lack of line buffering. Sublcasses should however, if possible, try to implement this method using their own knowledge of line breaking. \var{size}, if given, is passed as size argument to the stream's \method{readline()} method. \end{methoddesc} \begin{methoddesc}{readlines}{[sizehint]} Read all lines available on the input stream and return them as list of lines. Line breaks are implemented using the codec's decoder method and are included in the list entries. \var{sizehint}, if given, is passed as \var{size} argument to the stream's \method{read()} method. \end{methoddesc} \begin{methoddesc}{reset}{} Resets the codec buffers used for keeping state. Note that no stream repositioning should take place. This method is primarily intended to be able to recover from decoding errors. \end{methoddesc} In addition to the above methods, the \class{StreamReader} must also inherit all other methods and attribute from the underlying stream. The next two base classes are included for convenience. They are not needed by the codec registry, but may provide useful in practice. \subsubsection{StreamReaderWriter Objects \label{stream-reader-writer}} The \class{StreamReaderWriter} allows wrapping streams which work in both read and write modes. The design is such that one can use the factory functions returned by the \function{lookup()} function to construct the instance. \begin{classdesc}{StreamReaderWriter}{stream, Reader, Writer, errors} Creates a \class{StreamReaderWriter} instance. \var{stream} must be a file-like object. \var{Reader} and \var{Writer} must be factory functions or classes providing the \class{StreamReader} and \class{StreamWriter} interface resp. Error handling is done in the same way as defined for the stream readers and writers. \end{classdesc} \class{StreamReaderWriter} instances define the combined interfaces of \class{StreamReader} and \class{StreamWriter} classes. They inherit all other methods and attribute from the underlying stream. \subsubsection{StreamRecoder Objects \label{stream-recoder-objects}} The \class{StreamRecoder} provide a frontend - backend view of encoding data which is sometimes useful when dealing with different encoding environments. The design is such that one can use the factory functions returned by the \function{lookup()} function to construct the instance. \begin{classdesc}{StreamRecoder}{stream, encode, decode, Reader, Writer, errors} Creates a \class{StreamRecoder} instance which implements a two-way conversion: \var{encode} and \var{decode} work on the frontend (the input to \method{read()} and output of \method{write()}) while \var{Reader} and \var{Writer} work on the backend (reading and writing to the stream). You can use these objects to do transparent direct recodings from e.g.\ Latin-1 to UTF-8 and back. \var{stream} must be a file-like object. \var{encode}, \var{decode} must adhere to the \class{Codec} interface, \var{Reader}, \var{Writer} must be factory functions or classes providing objects of the the \class{StreamReader} and \class{StreamWriter} interface respectively. \var{encode} and \var{decode} are needed for the frontend translation, \var{Reader} and \var{Writer} for the backend translation. The intermediate format used is determined by the two sets of codecs, e.g. the Unicode codecs will use Unicode as intermediate encoding. Error handling is done in the same way as defined for the stream readers and writers. \end{classdesc} \class{StreamRecoder} instances define the combined interfaces of \class{StreamReader} and \class{StreamWriter} classes. They inherit all other methods and attribute from the underlying stream. \subsection{Standard Encodings} Python comes with a number of codecs builtin, either implemented as C functions, or with dictionaries as mapping tables. The following table lists the codecs by name, together with a few common aliases, and the languages for which the encoding is likely used. Neither the list of aliases nor the list of languages is meant to be exhaustive. Notice that spelling alternatives that only differ in case or use a hyphen instead of an underscore are also valid aliases. Many of the character sets support the same languages. They vary in individual characters (e.g. whether the EURO SIGN is supported or not), and in the assignment of characters to code positions. For the European languages in particular, the following variants typically exist: \begin{itemize} \item an ISO 8859 codeset \item a Microsoft Windows code page, which is typically derived from a 8859 codeset, but replaces control characters with additional graphic characters \item an IBM EBCDIC code page \item an IBM PC code page, which is ASCII compatible \end{itemize} \begin{longtableiii}{l|l|l}{textrm}{Codec}{Aliases}{Languages} \lineiii{ascii} {646, us-ascii} {English} \lineiii{cp037} {IBM037, IBM039} {English} \lineiii{cp424} {EBCDIC-CP-HE, IBM424} {Hebrew} \lineiii{cp437} {437, IBM437} {English} \lineiii{cp500} {EBCDIC-CP-BE, EBCDIC-CP-CH, IBM500} {Western Europe} \lineiii{cp737} {} {Greek} \lineiii{cp775} {IBM775} {Baltic languages} \lineiii{cp850} {850, IBM850} {Western Europe} \lineiii{cp852} {852, IBM852} {Central and Eastern Europe} \lineiii{cp855} {855, IBM855} {Bulgarian, Byelorussian, Macedonian, Russian, Serbian} \lineiii{cp856} {} {Hebrew} \lineiii{cp857} {857, IBM857} {Turkish} \lineiii{cp860} {860, IBM860} {Portuguese} \lineiii{cp861} {861, CP-IS, IBM861} {Icelandic} \lineiii{cp862} {862, IBM862} {Hebrew} \lineiii{cp863} {863, IBM863} {Canadian} \lineiii{cp864} {IBM864} {Arabic} \lineiii{cp865} {865, IBM865} {Danish, Norwegian} \lineiii{cp869} {869, CP-GR, IBM869} {Greek} \lineiii{cp874} {} {Thai} \lineiii{cp875} {} {Greek} \lineiii{cp1006} {} {Urdu} \lineiii{cp1026} {ibm1026} {Turkish} \lineiii{cp1140} {ibm1140} {Western Europe} \lineiii{cp1250} {windows-1250} {Central and Eastern Europe} \lineiii{cp1251} {windows-1251} {Bulgarian, Byelorussian, Macedonian, Russian, Serbian} \lineiii{cp1252} {windows-1252} {Western Europe} \lineiii{cp1253} {windows-1253} {Greek} \lineiii{cp1254} {windows-1254} {Turkish} \lineiii{cp1255} {windows-1255} {Hebrew} \lineiii{cp1256} {windows1256} {Arabic} \lineiii{cp1257} {windows-1257} {Baltic languages} \lineiii{cp1258} {windows-1258} {Vietnamese} \lineiii{latin_1} {iso-8859-1, iso8859-1, 8859, cp819, latin, latin1, L1} {West Europe} \lineiii{iso8859_2} {iso-8859-2, latin2, L2} {Central and Eastern Europe} \lineiii{iso8859_3} {iso-8859-3, latin3, L3} {Esperanto, Maltese} \lineiii{iso8859_4} {iso-8859-4, latin4, L4} {Baltic languagues} \lineiii{iso8859_5} {iso-8859-5, cyrillic} {Bulgarian, Byelorussian, Macedonian, Russian, Serbian} \lineiii{iso8859_6} {iso-8859-6, arabic} {Arabic} \lineiii{iso8859_7} {iso-8859-7, greek, greek8} {Greek} \lineiii{iso8859_8} {iso-8859-8, hebrew} {Hebrew} \lineiii{iso8859_9} {iso-8859-9, latin5, L5} {Turkish} \lineiii{iso8859_10} {iso-8859-10, latin6, L6} {Nordic languages} \lineiii{iso8859_13} {iso-8859-13} {Baltic languages} \lineiii{iso8859_14} {iso-8859-14, latin8, L8} {Celtic languages} \lineiii{iso8859_15} {iso-8859-15} {Western Europe} \lineiii{koi8_r} {} {Russian} \lineiii{koi8_u} {} {Ukrainian} \lineiii{mac_cyrillic} {maccyrillic} {Bulgarian, Byelorussian, Macedonian, Russian, Serbian} \lineiii{mac_greek} {macgreek} {Greek} \lineiii{mac_iceland} {maciceland} {Icelandic} \lineiii{mac_latin2} {maclatin2, maccentraleurope} {Central and Eastern Europe} \lineiii{mac_roman} {macroman} {Western Europe} \lineiii{mac_turkish} {macturkish} {Turkish} \lineiii{utf_16} {U16, utf16} {all languages} \lineiii{utf_16_be} {UTF-16BE} {all languages (BMP only)} \lineiii{utf_16_le} {UTF-16LE} {all languages (BMP only)} \lineiii{utf_7} {U7} {all languages} \lineiii{utf_8} {U8, UTF, utf8} {all languages} \end{longtableiii} A number of codecs are specific to Python, so their codec names have no meaning outside Python. Some of them don't convert from Unicode strings to byte strings, but instead use the property of the Python codecs machinery that any bijective function with one argument can be considered as an encoding. For the codecs listed below, the result in the ``encoding'' direction is always a byte string. The result of the ``decoding'' direction is listed as operand type in the table. \begin{tableiv}{l|l|l|l}{textrm}{Codec}{Aliases}{Operand type}{Purpose} \lineiv{base64_codec} {base64, base-64} {byte string} {Convert operand to MIME base64} \lineiv{hex_codec} {hex} {byte string} {Convert operand to hexadecimal representation, with two digits per byte} \lineiv{mbcs} {dbcs} {Unicode string} {Windows only: Encode operand according to the ANSI codepage (CP_ACP)} \lineiv{palmos} {} {Unicode string} {Encoding of PalmOS 3.5} \lineiv{quopri_codec} {quopri, quoted-printable, quotedprintable} {byte string} {Convert operand to MIME quoted printable} \lineiv{raw_unicode_escape} {} {Unicode string} {Produce a string that is suitable as raw Unicode literal in Python source code} \lineiv{rot_13} {rot13} {byte string} {Returns the Caesar-cypher encryption of the operand} \lineiv{string_escape} {} {byte string} {Produce a string that is suitable as string literal in Python source code} \lineiv{undefined} {} {any} {Raise an exception for all conversion. Can be used as the system encoding if no automatic coercion between byte and Unicode strings is desired.} \lineiv{unicode_escape} {} {Unicode string} {Produce a string that is suitable as Unicode literal in Python source code} \lineiv{unicode_internal} {} {Unicode string} {Return the internal represenation of the operand} \lineiv{uu_codec} {uu} {byte string} {Convert the operand using uuencode} \lineiv{zlib_codec} {zip, zlib} {byte string} {Compress the operand using gzip} \end{tableiv}