767 lines
32 KiB
ReStructuredText
767 lines
32 KiB
ReStructuredText
Unicode HOWTO
|
|
================
|
|
|
|
**Version 1.02**
|
|
|
|
This HOWTO discusses Python's support for Unicode, and explains various
|
|
problems that people commonly encounter when trying to work with Unicode.
|
|
|
|
Introduction to Unicode
|
|
------------------------------
|
|
|
|
History of Character Codes
|
|
''''''''''''''''''''''''''''''
|
|
|
|
In 1968, the American Standard Code for Information Interchange,
|
|
better known by its acronym ASCII, was standardized. ASCII defined
|
|
numeric codes for various characters, with the numeric values running from 0 to
|
|
127. For example, the lowercase letter 'a' is assigned 97 as its code
|
|
value.
|
|
|
|
ASCII was an American-developed standard, so it only defined
|
|
unaccented characters. There was an 'e', but no 'é' or 'Í'. This
|
|
meant that languages which required accented characters couldn't be
|
|
faithfully represented in ASCII. (Actually the missing accents matter
|
|
for English, too, which contains words such as 'naïve' and 'café', and some
|
|
publications have house styles which require spellings such as
|
|
'coöperate'.)
|
|
|
|
For a while people just wrote programs that didn't display accents. I
|
|
remember looking at Apple ][ BASIC programs, published in French-language
|
|
publications in the mid-1980s, that had lines like these::
|
|
|
|
PRINT "FICHER EST COMPLETE."
|
|
PRINT "CARACTERE NON ACCEPTE."
|
|
|
|
Those messages should contain accents, and they just look wrong to
|
|
someone who can read French.
|
|
|
|
In the 1980s, almost all personal computers were 8-bit, meaning that
|
|
bytes could hold values ranging from 0 to 255. ASCII codes only went
|
|
up to 127, so some machines assigned values between 128 and 255 to
|
|
accented characters. Different machines had different codes, however,
|
|
which led to problems exchanging files. Eventually various commonly
|
|
used sets of values for the 128-255 range emerged. Some were true
|
|
standards, defined by the International Standards Organization, and
|
|
some were **de facto** conventions that were invented by one company
|
|
or another and managed to catch on.
|
|
|
|
255 characters aren't very many. For example, you can't fit
|
|
both the accented characters used in Western Europe and the Cyrillic
|
|
alphabet used for Russian into the 128-255 range because there are more than
|
|
127 such characters.
|
|
|
|
You could write files using different codes (all your Russian
|
|
files in a coding system called KOI8, all your French files in
|
|
a different coding system called Latin1), but what if you wanted
|
|
to write a French document that quotes some Russian text? In the
|
|
1980s people began to want to solve this problem, and the Unicode
|
|
standardization effort began.
|
|
|
|
Unicode started out using 16-bit characters instead of 8-bit characters. 16
|
|
bits means you have 2^16 = 65,536 distinct values available, making it
|
|
possible to represent many different characters from many different
|
|
alphabets; an initial goal was to have Unicode contain the alphabets for
|
|
every single human language. It turns out that even 16 bits isn't enough to
|
|
meet that goal, and the modern Unicode specification uses a wider range of
|
|
codes, 0-1,114,111 (0x10ffff in base-16).
|
|
|
|
There's a related ISO standard, ISO 10646. Unicode and ISO 10646 were
|
|
originally separate efforts, but the specifications were merged with
|
|
the 1.1 revision of Unicode.
|
|
|
|
(This discussion of Unicode's history is highly simplified. I don't
|
|
think the average Python programmer needs to worry about the
|
|
historical details; consult the Unicode consortium site listed in the
|
|
References for more information.)
|
|
|
|
|
|
Definitions
|
|
''''''''''''''''''''''''
|
|
|
|
A **character** is the smallest possible component of a text. 'A',
|
|
'B', 'C', etc., are all different characters. So are 'È' and
|
|
'Í'. Characters are abstractions, and vary depending on the
|
|
language or context you're talking about. For example, the symbol for
|
|
ohms (Ω) is usually drawn much like the capital letter
|
|
omega (Ω) in the Greek alphabet (they may even be the same in
|
|
some fonts), but these are two different characters that have
|
|
different meanings.
|
|
|
|
The Unicode standard describes how characters are represented by
|
|
**code points**. A code point is an integer value, usually denoted in
|
|
base 16. In the standard, a code point is written using the notation
|
|
U+12ca to mean the character with value 0x12ca (4810 decimal). The
|
|
Unicode standard contains a lot of tables listing characters and their
|
|
corresponding code points::
|
|
|
|
0061 'a'; LATIN SMALL LETTER A
|
|
0062 'b'; LATIN SMALL LETTER B
|
|
0063 'c'; LATIN SMALL LETTER C
|
|
...
|
|
007B '{'; LEFT CURLY BRACKET
|
|
|
|
Strictly, these definitions imply that it's meaningless to say 'this is
|
|
character U+12ca'. U+12ca is a code point, which represents some particular
|
|
character; in this case, it represents the character 'ETHIOPIC SYLLABLE WI'.
|
|
In informal contexts, this distinction between code points and characters will
|
|
sometimes be forgotten.
|
|
|
|
A character is represented on a screen or on paper by a set of graphical
|
|
elements that's called a **glyph**. The glyph for an uppercase A, for
|
|
example, is two diagonal strokes and a horizontal stroke, though the exact
|
|
details will depend on the font being used. Most Python code doesn't need
|
|
to worry about glyphs; figuring out the correct glyph to display is
|
|
generally the job of a GUI toolkit or a terminal's font renderer.
|
|
|
|
|
|
Encodings
|
|
'''''''''
|
|
|
|
To summarize the previous section:
|
|
a Unicode string is a sequence of code points, which are
|
|
numbers from 0 to 0x10ffff. This sequence needs to be represented as
|
|
a set of bytes (meaning, values from 0-255) in memory. The rules for
|
|
translating a Unicode string into a sequence of bytes are called an
|
|
**encoding**.
|
|
|
|
The first encoding you might think of is an array of 32-bit integers.
|
|
In this representation, the string "Python" would look like this::
|
|
|
|
P y t h o n
|
|
0x50 00 00 00 79 00 00 00 74 00 00 00 68 00 00 00 6f 00 00 00 6e 00 00 00
|
|
0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23
|
|
|
|
This representation is straightforward but using
|
|
it presents a number of problems.
|
|
|
|
1. It's not portable; different processors order the bytes
|
|
differently.
|
|
|
|
2. It's very wasteful of space. In most texts, the majority of the code
|
|
points are less than 127, or less than 255, so a lot of space is occupied
|
|
by zero bytes. The above string takes 24 bytes compared to the 6
|
|
bytes needed for an ASCII representation. Increased RAM usage doesn't
|
|
matter too much (desktop computers have megabytes of RAM, and strings
|
|
aren't usually that large), but expanding our usage of disk and
|
|
network bandwidth by a factor of 4 is intolerable.
|
|
|
|
3. It's not compatible with existing C functions such as ``strlen()``,
|
|
so a new family of wide string functions would need to be used.
|
|
|
|
4. Many Internet standards are defined in terms of textual data, and
|
|
can't handle content with embedded zero bytes.
|
|
|
|
Generally people don't use this encoding, choosing other encodings
|
|
that are more efficient and convenient.
|
|
|
|
Encodings don't have to handle every possible Unicode character, and
|
|
most encodings don't. For example, Python's default encoding is the
|
|
'ascii' encoding. The rules for converting a Unicode string into the
|
|
ASCII encoding are simple; for each code point:
|
|
|
|
1. If the code point is <128, each byte is the same as the value of the
|
|
code point.
|
|
|
|
2. If the code point is 128 or greater, the Unicode string can't
|
|
be represented in this encoding. (Python raises a
|
|
``UnicodeEncodeError`` exception in this case.)
|
|
|
|
Latin-1, also known as ISO-8859-1, is a similar encoding. Unicode
|
|
code points 0-255 are identical to the Latin-1 values, so converting
|
|
to this encoding simply requires converting code points to byte
|
|
values; if a code point larger than 255 is encountered, the string
|
|
can't be encoded into Latin-1.
|
|
|
|
Encodings don't have to be simple one-to-one mappings like Latin-1.
|
|
Consider IBM's EBCDIC, which was used on IBM mainframes. Letter
|
|
values weren't in one block: 'a' through 'i' had values from 129 to
|
|
137, but 'j' through 'r' were 145 through 153. If you wanted to use
|
|
EBCDIC as an encoding, you'd probably use some sort of lookup table to
|
|
perform the conversion, but this is largely an internal detail.
|
|
|
|
UTF-8 is one of the most commonly used encodings. UTF stands for
|
|
"Unicode Transformation Format", and the '8' means that 8-bit numbers
|
|
are used in the encoding. (There's also a UTF-16 encoding, but it's
|
|
less frequently used than UTF-8.) UTF-8 uses the following rules:
|
|
|
|
1. If the code point is <128, it's represented by the corresponding byte value.
|
|
2. If the code point is between 128 and 0x7ff, it's turned into two byte values
|
|
between 128 and 255.
|
|
3. Code points >0x7ff are turned into three- or four-byte sequences, where
|
|
each byte of the sequence is between 128 and 255.
|
|
|
|
UTF-8 has several convenient properties:
|
|
|
|
1. It can handle any Unicode code point.
|
|
2. A Unicode string is turned into a string of bytes containing no embedded zero bytes. This avoids byte-ordering issues, and means UTF-8 strings can be processed by C functions such as ``strcpy()`` and sent through protocols that can't handle zero bytes.
|
|
3. A string of ASCII text is also valid UTF-8 text.
|
|
4. UTF-8 is fairly compact; the majority of code points are turned into two bytes, and values less than 128 occupy only a single byte.
|
|
5. If bytes are corrupted or lost, it's possible to determine the start of the next UTF-8-encoded code point and resynchronize. It's also unlikely that random 8-bit data will look like valid UTF-8.
|
|
|
|
|
|
|
|
References
|
|
''''''''''''''
|
|
|
|
The Unicode Consortium site at <http://www.unicode.org> has character
|
|
charts, a glossary, and PDF versions of the Unicode specification. Be
|
|
prepared for some difficult reading.
|
|
<http://www.unicode.org/history/> is a chronology of the origin and
|
|
development of Unicode.
|
|
|
|
To help understand the standard, Jukka Korpela has written an
|
|
introductory guide to reading the Unicode character tables,
|
|
available at <http://www.cs.tut.fi/~jkorpela/unicode/guide.html>.
|
|
|
|
Roman Czyborra wrote another explanation of Unicode's basic principles;
|
|
it's at <http://czyborra.com/unicode/characters.html>.
|
|
Czyborra has written a number of other Unicode-related documentation,
|
|
available from <http://www.cyzborra.com>.
|
|
|
|
Two other good introductory articles were written by Joel Spolsky
|
|
<http://www.joelonsoftware.com/articles/Unicode.html> and Jason
|
|
Orendorff <http://www.jorendorff.com/articles/unicode/>. If this
|
|
introduction didn't make things clear to you, you should try reading
|
|
one of these alternate articles before continuing.
|
|
|
|
Wikipedia entries are often helpful; see the entries for "character
|
|
encoding" <http://en.wikipedia.org/wiki/Character_encoding> and UTF-8
|
|
<http://en.wikipedia.org/wiki/UTF-8>, for example.
|
|
|
|
|
|
Python's Unicode Support
|
|
------------------------
|
|
|
|
Now that you've learned the rudiments of Unicode, we can look at
|
|
Python's Unicode features.
|
|
|
|
|
|
The Unicode Type
|
|
'''''''''''''''''''
|
|
|
|
Unicode strings are expressed as instances of the ``unicode`` type,
|
|
one of Python's repertoire of built-in types. It derives from an
|
|
abstract type called ``basestring``, which is also an ancestor of the
|
|
``str`` type; you can therefore check if a value is a string type with
|
|
``isinstance(value, basestring)``. Under the hood, Python represents
|
|
Unicode strings as either 16- or 32-bit integers, depending on how the
|
|
Python interpreter was compiled.
|
|
|
|
The ``unicode()`` constructor has the signature ``unicode(string[, encoding, errors])``.
|
|
All of its arguments should be 8-bit strings. The first argument is converted
|
|
to Unicode using the specified encoding; if you leave off the ``encoding`` argument,
|
|
the ASCII encoding is used for the conversion, so characters greater than 127 will
|
|
be treated as errors::
|
|
|
|
>>> unicode('abcdef')
|
|
u'abcdef'
|
|
>>> s = unicode('abcdef')
|
|
>>> type(s)
|
|
<type 'unicode'>
|
|
>>> unicode('abcdef' + chr(255))
|
|
Traceback (most recent call last):
|
|
File "<stdin>", line 1, in ?
|
|
UnicodeDecodeError: 'ascii' codec can't decode byte 0xff in position 6:
|
|
ordinal not in range(128)
|
|
|
|
The ``errors`` argument specifies the response when the input string can't be converted according to the encoding's rules. Legal values for this argument
|
|
are 'strict' (raise a ``UnicodeDecodeError`` exception),
|
|
'replace' (add U+FFFD, 'REPLACEMENT CHARACTER'),
|
|
or 'ignore' (just leave the character out of the Unicode result).
|
|
The following examples show the differences::
|
|
|
|
>>> unicode('\x80abc', errors='strict')
|
|
Traceback (most recent call last):
|
|
File "<stdin>", line 1, in ?
|
|
UnicodeDecodeError: 'ascii' codec can't decode byte 0x80 in position 0:
|
|
ordinal not in range(128)
|
|
>>> unicode('\x80abc', errors='replace')
|
|
u'\ufffdabc'
|
|
>>> unicode('\x80abc', errors='ignore')
|
|
u'abc'
|
|
|
|
Encodings are specified as strings containing the encoding's name.
|
|
Python 2.4 comes with roughly 100 different encodings; see the Python
|
|
Library Reference at
|
|
<http://docs.python.org/lib/standard-encodings.html> for a list. Some
|
|
encodings have multiple names; for example, 'latin-1', 'iso_8859_1'
|
|
and '8859' are all synonyms for the same encoding.
|
|
|
|
One-character Unicode strings can also be created with the
|
|
``unichr()`` built-in function, which takes integers and returns a
|
|
Unicode string of length 1 that contains the corresponding code point.
|
|
The reverse operation is the built-in `ord()` function that takes a
|
|
one-character Unicode string and returns the code point value::
|
|
|
|
>>> unichr(40960)
|
|
u'\ua000'
|
|
>>> ord(u'\ua000')
|
|
40960
|
|
|
|
Instances of the ``unicode`` type have many of the same methods as
|
|
the 8-bit string type for operations such as searching and formatting::
|
|
|
|
>>> s = u'Was ever feather so lightly blown to and fro as this multitude?'
|
|
>>> s.count('e')
|
|
5
|
|
>>> s.find('feather')
|
|
9
|
|
>>> s.find('bird')
|
|
-1
|
|
>>> s.replace('feather', 'sand')
|
|
u'Was ever sand so lightly blown to and fro as this multitude?'
|
|
>>> s.upper()
|
|
u'WAS EVER FEATHER SO LIGHTLY BLOWN TO AND FRO AS THIS MULTITUDE?'
|
|
|
|
Note that the arguments to these methods can be Unicode strings or 8-bit strings.
|
|
8-bit strings will be converted to Unicode before carrying out the operation;
|
|
Python's default ASCII encoding will be used, so characters greater than 127 will cause an exception::
|
|
|
|
>>> s.find('Was\x9f')
|
|
Traceback (most recent call last):
|
|
File "<stdin>", line 1, in ?
|
|
UnicodeDecodeError: 'ascii' codec can't decode byte 0x9f in position 3: ordinal not in range(128)
|
|
>>> s.find(u'Was\x9f')
|
|
-1
|
|
|
|
Much Python code that operates on strings will therefore work with
|
|
Unicode strings without requiring any changes to the code. (Input and
|
|
output code needs more updating for Unicode; more on this later.)
|
|
|
|
Another important method is ``.encode([encoding], [errors='strict'])``,
|
|
which returns an 8-bit string version of the
|
|
Unicode string, encoded in the requested encoding. The ``errors``
|
|
parameter is the same as the parameter of the ``unicode()``
|
|
constructor, with one additional possibility; as well as 'strict',
|
|
'ignore', and 'replace', you can also pass 'xmlcharrefreplace' which
|
|
uses XML's character references. The following example shows the
|
|
different results::
|
|
|
|
>>> u = unichr(40960) + u'abcd' + unichr(1972)
|
|
>>> u.encode('utf-8')
|
|
'\xea\x80\x80abcd\xde\xb4'
|
|
>>> u.encode('ascii')
|
|
Traceback (most recent call last):
|
|
File "<stdin>", line 1, in ?
|
|
UnicodeEncodeError: 'ascii' codec can't encode character '\ua000' in position 0: ordinal not in range(128)
|
|
>>> u.encode('ascii', 'ignore')
|
|
'abcd'
|
|
>>> u.encode('ascii', 'replace')
|
|
'?abcd?'
|
|
>>> u.encode('ascii', 'xmlcharrefreplace')
|
|
'ꀀabcd޴'
|
|
|
|
Python's 8-bit strings have a ``.decode([encoding], [errors])`` method
|
|
that interprets the string using the given encoding::
|
|
|
|
>>> u = unichr(40960) + u'abcd' + unichr(1972) # Assemble a string
|
|
>>> utf8_version = u.encode('utf-8') # Encode as UTF-8
|
|
>>> type(utf8_version), utf8_version
|
|
(<type 'str'>, '\xea\x80\x80abcd\xde\xb4')
|
|
>>> u2 = utf8_version.decode('utf-8') # Decode using UTF-8
|
|
>>> u == u2 # The two strings match
|
|
True
|
|
|
|
The low-level routines for registering and accessing the available
|
|
encodings are found in the ``codecs`` module. However, the encoding
|
|
and decoding functions returned by this module are usually more
|
|
low-level than is comfortable, so I'm not going to describe the
|
|
``codecs`` module here. If you need to implement a completely new
|
|
encoding, you'll need to learn about the ``codecs`` module interfaces,
|
|
but implementing encodings is a specialized task that also won't be
|
|
covered here. Consult the Python documentation to learn more about
|
|
this module.
|
|
|
|
The most commonly used part of the ``codecs`` module is the
|
|
``codecs.open()`` function which will be discussed in the section
|
|
on input and output.
|
|
|
|
|
|
Unicode Literals in Python Source Code
|
|
''''''''''''''''''''''''''''''''''''''''''
|
|
|
|
In Python source code, Unicode literals are written as strings
|
|
prefixed with the 'u' or 'U' character: ``u'abcdefghijk'``. Specific
|
|
code points can be written using the ``\u`` escape sequence, which is
|
|
followed by four hex digits giving the code point. The ``\U`` escape
|
|
sequence is similar, but expects 8 hex digits, not 4.
|
|
|
|
Unicode literals can also use the same escape sequences as 8-bit
|
|
strings, including ``\x``, but ``\x`` only takes two hex digits so it
|
|
can't express an arbitrary code point. Octal escapes can go up to
|
|
U+01ff, which is octal 777.
|
|
|
|
::
|
|
|
|
>>> s = u"a\xac\u1234\u20ac\U00008000"
|
|
^^^^ two-digit hex escape
|
|
^^^^^^ four-digit Unicode escape
|
|
^^^^^^^^^^ eight-digit Unicode escape
|
|
>>> for c in s: print ord(c),
|
|
...
|
|
97 172 4660 8364 32768
|
|
|
|
Using escape sequences for code points greater than 127 is fine in
|
|
small doses, but becomes an annoyance if you're using many accented
|
|
characters, as you would in a program with messages in French or some
|
|
other accent-using language. You can also assemble strings using the
|
|
``unichr()`` built-in function, but this is even more tedious.
|
|
|
|
Ideally, you'd want to be able to write literals in your language's
|
|
natural encoding. You could then edit Python source code with your
|
|
favorite editor which would display the accented characters naturally,
|
|
and have the right characters used at runtime.
|
|
|
|
Python supports writing Unicode literals in any encoding, but you have
|
|
to declare the encoding being used. This is done by including a
|
|
special comment as either the first or second line of the source
|
|
file::
|
|
|
|
#!/usr/bin/env python
|
|
# -*- coding: latin-1 -*-
|
|
|
|
u = u'abcdé'
|
|
print ord(u[-1])
|
|
|
|
The syntax is inspired by Emacs's notation for specifying variables local to a file.
|
|
Emacs supports many different variables, but Python only supports 'coding'.
|
|
The ``-*-`` symbols indicate that the comment is special; within them,
|
|
you must supply the name ``coding`` and the name of your chosen encoding,
|
|
separated by ``':'``.
|
|
|
|
If you don't include such a comment, the default encoding used will be
|
|
ASCII. Versions of Python before 2.4 were Euro-centric and assumed
|
|
Latin-1 as a default encoding for string literals; in Python 2.4,
|
|
characters greater than 127 still work but result in a warning. For
|
|
example, the following program has no encoding declaration::
|
|
|
|
#!/usr/bin/env python
|
|
u = u'abcdé'
|
|
print ord(u[-1])
|
|
|
|
When you run it with Python 2.4, it will output the following warning::
|
|
|
|
amk:~$ python p263.py
|
|
sys:1: DeprecationWarning: Non-ASCII character '\xe9'
|
|
in file p263.py on line 2, but no encoding declared;
|
|
see http://www.python.org/peps/pep-0263.html for details
|
|
|
|
|
|
Unicode Properties
|
|
'''''''''''''''''''
|
|
|
|
The Unicode specification includes a database of information about
|
|
code points. For each code point that's defined, the information
|
|
includes the character's name, its category, the numeric value if
|
|
applicable (Unicode has characters representing the Roman numerals and
|
|
fractions such as one-third and four-fifths). There are also
|
|
properties related to the code point's use in bidirectional text and
|
|
other display-related properties.
|
|
|
|
The following program displays some information about several
|
|
characters, and prints the numeric value of one particular character::
|
|
|
|
import unicodedata
|
|
|
|
u = unichr(233) + unichr(0x0bf2) + unichr(3972) + unichr(6000) + unichr(13231)
|
|
|
|
for i, c in enumerate(u):
|
|
print i, '%04x' % ord(c), unicodedata.category(c),
|
|
print unicodedata.name(c)
|
|
|
|
# Get numeric value of second character
|
|
print unicodedata.numeric(u[1])
|
|
|
|
When run, this prints::
|
|
|
|
0 00e9 Ll LATIN SMALL LETTER E WITH ACUTE
|
|
1 0bf2 No TAMIL NUMBER ONE THOUSAND
|
|
2 0f84 Mn TIBETAN MARK HALANTA
|
|
3 1770 Lo TAGBANWA LETTER SA
|
|
4 33af So SQUARE RAD OVER S SQUARED
|
|
1000.0
|
|
|
|
The category codes are abbreviations describing the nature of the
|
|
character. These are grouped into categories such as "Letter",
|
|
"Number", "Punctuation", or "Symbol", which in turn are broken up into
|
|
subcategories. To take the codes from the above output, ``'Ll'``
|
|
means 'Letter, lowercase', ``'No'`` means "Number, other", ``'Mn'`` is
|
|
"Mark, nonspacing", and ``'So'`` is "Symbol, other". See
|
|
<http://www.unicode.org/Public/UNIDATA/UCD.html#General_Category_Values>
|
|
for a list of category codes.
|
|
|
|
References
|
|
''''''''''''''
|
|
|
|
The Unicode and 8-bit string types are described in the Python library
|
|
reference at <http://docs.python.org/lib/typesseq.html>.
|
|
|
|
The documentation for the ``unicodedata`` module is at
|
|
<http://docs.python.org/lib/module-unicodedata.html>.
|
|
|
|
The documentation for the ``codecs`` module is at
|
|
<http://docs.python.org/lib/module-codecs.html>.
|
|
|
|
Marc-André Lemburg gave a presentation at EuroPython 2002
|
|
titled "Python and Unicode". A PDF version of his slides
|
|
is available at <http://www.egenix.com/files/python/Unicode-EPC2002-Talk.pdf>,
|
|
and is an excellent overview of the design of Python's Unicode features.
|
|
|
|
|
|
Reading and Writing Unicode Data
|
|
----------------------------------------
|
|
|
|
Once you've written some code that works with Unicode data, the next
|
|
problem is input/output. How do you get Unicode strings into your
|
|
program, and how do you convert Unicode into a form suitable for
|
|
storage or transmission?
|
|
|
|
It's possible that you may not need to do anything depending on your
|
|
input sources and output destinations; you should check whether the
|
|
libraries used in your application support Unicode natively. XML
|
|
parsers often return Unicode data, for example. Many relational
|
|
databases also support Unicode-valued columns and can return Unicode
|
|
values from an SQL query.
|
|
|
|
Unicode data is usually converted to a particular encoding before it
|
|
gets written to disk or sent over a socket. It's possible to do all
|
|
the work yourself: open a file, read an 8-bit string from it, and
|
|
convert the string with ``unicode(str, encoding)``. However, the
|
|
manual approach is not recommended.
|
|
|
|
One problem is the multi-byte nature of encodings; one Unicode
|
|
character can be represented by several bytes. If you want to read
|
|
the file in arbitrary-sized chunks (say, 1K or 4K), you need to write
|
|
error-handling code to catch the case where only part of the bytes
|
|
encoding a single Unicode character are read at the end of a chunk.
|
|
One solution would be to read the entire file into memory and then
|
|
perform the decoding, but that prevents you from working with files
|
|
that are extremely large; if you need to read a 2Gb file, you need 2Gb
|
|
of RAM. (More, really, since for at least a moment you'd need to have
|
|
both the encoded string and its Unicode version in memory.)
|
|
|
|
The solution would be to use the low-level decoding interface to catch
|
|
the case of partial coding sequences. The work of implementing this
|
|
has already been done for you: the ``codecs`` module includes a
|
|
version of the ``open()`` function that returns a file-like object
|
|
that assumes the file's contents are in a specified encoding and
|
|
accepts Unicode parameters for methods such as ``.read()`` and
|
|
``.write()``.
|
|
|
|
The function's parameters are
|
|
``open(filename, mode='rb', encoding=None, errors='strict', buffering=1)``. ``mode`` can be
|
|
``'r'``, ``'w'``, or ``'a'``, just like the corresponding parameter to the
|
|
regular built-in ``open()`` function; add a ``'+'`` to
|
|
update the file. ``buffering`` is similarly
|
|
parallel to the standard function's parameter.
|
|
``encoding`` is a string giving
|
|
the encoding to use; if it's left as ``None``, a regular Python file
|
|
object that accepts 8-bit strings is returned. Otherwise, a wrapper
|
|
object is returned, and data written to or read from the wrapper
|
|
object will be converted as needed. ``errors`` specifies the action
|
|
for encoding errors and can be one of the usual values of 'strict',
|
|
'ignore', and 'replace'.
|
|
|
|
Reading Unicode from a file is therefore simple::
|
|
|
|
import codecs
|
|
f = codecs.open('unicode.rst', encoding='utf-8')
|
|
for line in f:
|
|
print repr(line)
|
|
|
|
It's also possible to open files in update mode,
|
|
allowing both reading and writing::
|
|
|
|
f = codecs.open('test', encoding='utf-8', mode='w+')
|
|
f.write(u'\u4500 blah blah blah\n')
|
|
f.seek(0)
|
|
print repr(f.readline()[:1])
|
|
f.close()
|
|
|
|
Unicode character U+FEFF is used as a byte-order mark (BOM),
|
|
and is often written as the first character of a file in order
|
|
to assist with autodetection of the file's byte ordering.
|
|
Some encodings, such as UTF-16, expect a BOM to be present at
|
|
the start of a file; when such an encoding is used,
|
|
the BOM will be automatically written as the first character
|
|
and will be silently dropped when the file is read. There are
|
|
variants of these encodings, such as 'utf-16-le' and 'utf-16-be'
|
|
for little-endian and big-endian encodings, that specify
|
|
one particular byte ordering and don't
|
|
skip the BOM.
|
|
|
|
|
|
Unicode filenames
|
|
'''''''''''''''''''''''''
|
|
|
|
Most of the operating systems in common use today support filenames
|
|
that contain arbitrary Unicode characters. Usually this is
|
|
implemented by converting the Unicode string into some encoding that
|
|
varies depending on the system. For example, MacOS X uses UTF-8 while
|
|
Windows uses a configurable encoding; on Windows, Python uses the name
|
|
"mbcs" to refer to whatever the currently configured encoding is. On
|
|
Unix systems, there will only be a filesystem encoding if you've set
|
|
the ``LANG`` or ``LC_CTYPE`` environment variables; if you haven't,
|
|
the default encoding is ASCII.
|
|
|
|
The ``sys.getfilesystemencoding()`` function returns the encoding to
|
|
use on your current system, in case you want to do the encoding
|
|
manually, but there's not much reason to bother. When opening a file
|
|
for reading or writing, you can usually just provide the Unicode
|
|
string as the filename, and it will be automatically converted to the
|
|
right encoding for you::
|
|
|
|
filename = u'filename\u4500abc'
|
|
f = open(filename, 'w')
|
|
f.write('blah\n')
|
|
f.close()
|
|
|
|
Functions in the ``os`` module such as ``os.stat()`` will also accept
|
|
Unicode filenames.
|
|
|
|
``os.listdir()``, which returns filenames, raises an issue: should it
|
|
return the Unicode version of filenames, or should it return 8-bit
|
|
strings containing the encoded versions? ``os.listdir()`` will do
|
|
both, depending on whether you provided the directory path as an 8-bit
|
|
string or a Unicode string. If you pass a Unicode string as the path,
|
|
filenames will be decoded using the filesystem's encoding and a list
|
|
of Unicode strings will be returned, while passing an 8-bit path will
|
|
return the 8-bit versions of the filenames. For example, assuming the
|
|
default filesystem encoding is UTF-8, running the following program::
|
|
|
|
fn = u'filename\u4500abc'
|
|
f = open(fn, 'w')
|
|
f.close()
|
|
|
|
import os
|
|
print os.listdir('.')
|
|
print os.listdir(u'.')
|
|
|
|
will produce the following output::
|
|
|
|
amk:~$ python t.py
|
|
['.svn', 'filename\xe4\x94\x80abc', ...]
|
|
[u'.svn', u'filename\u4500abc', ...]
|
|
|
|
The first list contains UTF-8-encoded filenames, and the second list
|
|
contains the Unicode versions.
|
|
|
|
|
|
|
|
Tips for Writing Unicode-aware Programs
|
|
''''''''''''''''''''''''''''''''''''''''''''
|
|
|
|
This section provides some suggestions on writing software that
|
|
deals with Unicode.
|
|
|
|
The most important tip is:
|
|
|
|
Software should only work with Unicode strings internally,
|
|
converting to a particular encoding on output.
|
|
|
|
If you attempt to write processing functions that accept both
|
|
Unicode and 8-bit strings, you will find your program vulnerable to
|
|
bugs wherever you combine the two different kinds of strings. Python's
|
|
default encoding is ASCII, so whenever a character with an ASCII value >127
|
|
is in the input data, you'll get a ``UnicodeDecodeError``
|
|
because that character can't be handled by the ASCII encoding.
|
|
|
|
It's easy to miss such problems if you only test your software
|
|
with data that doesn't contain any
|
|
accents; everything will seem to work, but there's actually a bug in your
|
|
program waiting for the first user who attempts to use characters >127.
|
|
A second tip, therefore, is:
|
|
|
|
Include characters >127 and, even better, characters >255 in your
|
|
test data.
|
|
|
|
When using data coming from a web browser or some other untrusted source,
|
|
a common technique is to check for illegal characters in a string
|
|
before using the string in a generated command line or storing it in a
|
|
database. If you're doing this, be careful to check
|
|
the string once it's in the form that will be used or stored; it's
|
|
possible for encodings to be used to disguise characters. This is especially
|
|
true if the input data also specifies the encoding;
|
|
many encodings leave the commonly checked-for characters alone,
|
|
but Python includes some encodings such as ``'base64'``
|
|
that modify every single character.
|
|
|
|
For example, let's say you have a content management system that takes a
|
|
Unicode filename, and you want to disallow paths with a '/' character.
|
|
You might write this code::
|
|
|
|
def read_file (filename, encoding):
|
|
if '/' in filename:
|
|
raise ValueError("'/' not allowed in filenames")
|
|
unicode_name = filename.decode(encoding)
|
|
f = open(unicode_name, 'r')
|
|
# ... return contents of file ...
|
|
|
|
However, if an attacker could specify the ``'base64'`` encoding,
|
|
they could pass ``'L2V0Yy9wYXNzd2Q='``, which is the base-64
|
|
encoded form of the string ``'/etc/passwd'``, to read a
|
|
system file. The above code looks for ``'/'`` characters
|
|
in the encoded form and misses the dangerous character
|
|
in the resulting decoded form.
|
|
|
|
References
|
|
''''''''''''''
|
|
|
|
The PDF slides for Marc-André Lemburg's presentation "Writing
|
|
Unicode-aware Applications in Python" are available at
|
|
<http://www.egenix.com/files/python/LSM2005-Developing-Unicode-aware-applications-in-Python.pdf>
|
|
and discuss questions of character encodings as well as how to
|
|
internationalize and localize an application.
|
|
|
|
|
|
Revision History and Acknowledgements
|
|
------------------------------------------
|
|
|
|
Thanks to the following people who have noted errors or offered
|
|
suggestions on this article: Nicholas Bastin,
|
|
Marius Gedminas, Kent Johnson, Ken Krugler,
|
|
Marc-André Lemburg, Martin von Löwis, Chad Whitacre.
|
|
|
|
Version 1.0: posted August 5 2005.
|
|
|
|
Version 1.01: posted August 7 2005. Corrects factual and markup
|
|
errors; adds several links.
|
|
|
|
Version 1.02: posted August 16 2005. Corrects factual errors.
|
|
|
|
|
|
.. comment Additional topic: building Python w/ UCS2 or UCS4 support
|
|
.. comment Describe obscure -U switch somewhere?
|
|
.. comment Describe use of codecs.StreamRecoder and StreamReaderWriter
|
|
|
|
.. comment
|
|
Original outline:
|
|
|
|
- [ ] Unicode introduction
|
|
- [ ] ASCII
|
|
- [ ] Terms
|
|
- [ ] Character
|
|
- [ ] Code point
|
|
- [ ] Encodings
|
|
- [ ] Common encodings: ASCII, Latin-1, UTF-8
|
|
- [ ] Unicode Python type
|
|
- [ ] Writing unicode literals
|
|
- [ ] Obscurity: -U switch
|
|
- [ ] Built-ins
|
|
- [ ] unichr()
|
|
- [ ] ord()
|
|
- [ ] unicode() constructor
|
|
- [ ] Unicode type
|
|
- [ ] encode(), decode() methods
|
|
- [ ] Unicodedata module for character properties
|
|
- [ ] I/O
|
|
- [ ] Reading/writing Unicode data into files
|
|
- [ ] Byte-order marks
|
|
- [ ] Unicode filenames
|
|
- [ ] Writing Unicode programs
|
|
- [ ] Do everything in Unicode
|
|
- [ ] Declaring source code encodings (PEP 263)
|
|
- [ ] Other issues
|
|
- [ ] Building Python (UCS2, UCS4)
|