mirror of https://github.com/python/cpython
978 lines
36 KiB
ReStructuredText
978 lines
36 KiB
ReStructuredText
.. highlightlang:: c
|
|
|
|
.. _unicodeobjects:
|
|
|
|
Unicode Objects and Codecs
|
|
--------------------------
|
|
|
|
.. sectionauthor:: Marc-Andre Lemburg <mal@lemburg.com>
|
|
|
|
Unicode Objects
|
|
^^^^^^^^^^^^^^^
|
|
|
|
|
|
Unicode Type
|
|
""""""""""""
|
|
|
|
These are the basic Unicode object types used for the Unicode implementation in
|
|
Python:
|
|
|
|
|
|
.. ctype:: Py_UNICODE
|
|
|
|
This type represents the storage type which is used by Python internally as
|
|
basis for holding Unicode ordinals. Python's default builds use a 16-bit type
|
|
for :ctype:`Py_UNICODE` and store Unicode values internally as UCS2. It is also
|
|
possible to build a UCS4 version of Python (most recent Linux distributions come
|
|
with UCS4 builds of Python). These builds then use a 32-bit type for
|
|
:ctype:`Py_UNICODE` and store Unicode data internally as UCS4. On platforms
|
|
where :ctype:`wchar_t` is available and compatible with the chosen Python
|
|
Unicode build variant, :ctype:`Py_UNICODE` is a typedef alias for
|
|
:ctype:`wchar_t` to enhance native platform compatibility. On all other
|
|
platforms, :ctype:`Py_UNICODE` is a typedef alias for either :ctype:`unsigned
|
|
short` (UCS2) or :ctype:`unsigned long` (UCS4).
|
|
|
|
Note that UCS2 and UCS4 Python builds are not binary compatible. Please keep
|
|
this in mind when writing extensions or interfaces.
|
|
|
|
|
|
.. ctype:: PyUnicodeObject
|
|
|
|
This subtype of :ctype:`PyObject` represents a Python Unicode object.
|
|
|
|
|
|
.. cvar:: PyTypeObject PyUnicode_Type
|
|
|
|
This instance of :ctype:`PyTypeObject` represents the Python Unicode type. It
|
|
is exposed to Python code as ``unicode`` and ``types.UnicodeType``.
|
|
|
|
The following APIs are really C macros and can be used to do fast checks and to
|
|
access internal read-only data of Unicode objects:
|
|
|
|
|
|
.. cfunction:: int PyUnicode_Check(PyObject *o)
|
|
|
|
Return true if the object *o* is a Unicode object or an instance of a Unicode
|
|
subtype.
|
|
|
|
.. versionchanged:: 2.2
|
|
Allowed subtypes to be accepted.
|
|
|
|
|
|
.. cfunction:: int PyUnicode_CheckExact(PyObject *o)
|
|
|
|
Return true if the object *o* is a Unicode object, but not an instance of a
|
|
subtype.
|
|
|
|
.. versionadded:: 2.2
|
|
|
|
|
|
.. cfunction:: Py_ssize_t PyUnicode_GET_SIZE(PyObject *o)
|
|
|
|
Return the size of the object. *o* has to be a :ctype:`PyUnicodeObject` (not
|
|
checked).
|
|
|
|
.. versionchanged:: 2.5
|
|
This function returned an :ctype:`int` type. This might require changes
|
|
in your code for properly supporting 64-bit systems.
|
|
|
|
|
|
.. cfunction:: Py_ssize_t PyUnicode_GET_DATA_SIZE(PyObject *o)
|
|
|
|
Return the size of the object's internal buffer in bytes. *o* has to be a
|
|
:ctype:`PyUnicodeObject` (not checked).
|
|
|
|
.. versionchanged:: 2.5
|
|
This function returned an :ctype:`int` type. This might require changes
|
|
in your code for properly supporting 64-bit systems.
|
|
|
|
|
|
.. cfunction:: Py_UNICODE* PyUnicode_AS_UNICODE(PyObject *o)
|
|
|
|
Return a pointer to the internal :ctype:`Py_UNICODE` buffer of the object. *o*
|
|
has to be a :ctype:`PyUnicodeObject` (not checked).
|
|
|
|
|
|
.. cfunction:: const char* PyUnicode_AS_DATA(PyObject *o)
|
|
|
|
Return a pointer to the internal buffer of the object. *o* has to be a
|
|
:ctype:`PyUnicodeObject` (not checked).
|
|
|
|
|
|
.. cfunction:: int PyUnicode_ClearFreeList()
|
|
|
|
Clear the free list. Return the total number of freed items.
|
|
|
|
.. versionadded:: 2.6
|
|
|
|
|
|
Unicode Character Properties
|
|
""""""""""""""""""""""""""""
|
|
|
|
Unicode provides many different character properties. The most often needed ones
|
|
are available through these macros which are mapped to C functions depending on
|
|
the Python configuration.
|
|
|
|
|
|
.. cfunction:: int Py_UNICODE_ISSPACE(Py_UNICODE ch)
|
|
|
|
Return 1 or 0 depending on whether *ch* is a whitespace character.
|
|
|
|
|
|
.. cfunction:: int Py_UNICODE_ISLOWER(Py_UNICODE ch)
|
|
|
|
Return 1 or 0 depending on whether *ch* is a lowercase character.
|
|
|
|
|
|
.. cfunction:: int Py_UNICODE_ISUPPER(Py_UNICODE ch)
|
|
|
|
Return 1 or 0 depending on whether *ch* is an uppercase character.
|
|
|
|
|
|
.. cfunction:: int Py_UNICODE_ISTITLE(Py_UNICODE ch)
|
|
|
|
Return 1 or 0 depending on whether *ch* is a titlecase character.
|
|
|
|
|
|
.. cfunction:: int Py_UNICODE_ISLINEBREAK(Py_UNICODE ch)
|
|
|
|
Return 1 or 0 depending on whether *ch* is a linebreak character.
|
|
|
|
|
|
.. cfunction:: int Py_UNICODE_ISDECIMAL(Py_UNICODE ch)
|
|
|
|
Return 1 or 0 depending on whether *ch* is a decimal character.
|
|
|
|
|
|
.. cfunction:: int Py_UNICODE_ISDIGIT(Py_UNICODE ch)
|
|
|
|
Return 1 or 0 depending on whether *ch* is a digit character.
|
|
|
|
|
|
.. cfunction:: int Py_UNICODE_ISNUMERIC(Py_UNICODE ch)
|
|
|
|
Return 1 or 0 depending on whether *ch* is a numeric character.
|
|
|
|
|
|
.. cfunction:: int Py_UNICODE_ISALPHA(Py_UNICODE ch)
|
|
|
|
Return 1 or 0 depending on whether *ch* is an alphabetic character.
|
|
|
|
|
|
.. cfunction:: int Py_UNICODE_ISALNUM(Py_UNICODE ch)
|
|
|
|
Return 1 or 0 depending on whether *ch* is an alphanumeric character.
|
|
|
|
These APIs can be used for fast direct character conversions:
|
|
|
|
|
|
.. cfunction:: Py_UNICODE Py_UNICODE_TOLOWER(Py_UNICODE ch)
|
|
|
|
Return the character *ch* converted to lower case.
|
|
|
|
|
|
.. cfunction:: Py_UNICODE Py_UNICODE_TOUPPER(Py_UNICODE ch)
|
|
|
|
Return the character *ch* converted to upper case.
|
|
|
|
|
|
.. cfunction:: Py_UNICODE Py_UNICODE_TOTITLE(Py_UNICODE ch)
|
|
|
|
Return the character *ch* converted to title case.
|
|
|
|
|
|
.. cfunction:: int Py_UNICODE_TODECIMAL(Py_UNICODE ch)
|
|
|
|
Return the character *ch* converted to a decimal positive integer. Return
|
|
``-1`` if this is not possible. This macro does not raise exceptions.
|
|
|
|
|
|
.. cfunction:: int Py_UNICODE_TODIGIT(Py_UNICODE ch)
|
|
|
|
Return the character *ch* converted to a single digit integer. Return ``-1`` if
|
|
this is not possible. This macro does not raise exceptions.
|
|
|
|
|
|
.. cfunction:: double Py_UNICODE_TONUMERIC(Py_UNICODE ch)
|
|
|
|
Return the character *ch* converted to a double. Return ``-1.0`` if this is not
|
|
possible. This macro does not raise exceptions.
|
|
|
|
|
|
Plain Py_UNICODE
|
|
""""""""""""""""
|
|
|
|
To create Unicode objects and access their basic sequence properties, use these
|
|
APIs:
|
|
|
|
|
|
.. cfunction:: PyObject* PyUnicode_FromUnicode(const Py_UNICODE *u, Py_ssize_t size)
|
|
|
|
Create a Unicode Object from the Py_UNICODE buffer *u* of the given size. *u*
|
|
may be *NULL* which causes the contents to be undefined. It is the user's
|
|
responsibility to fill in the needed data. The buffer is copied into the new
|
|
object. If the buffer is not *NULL*, the return value might be a shared object.
|
|
Therefore, modification of the resulting Unicode object is only allowed when *u*
|
|
is *NULL*.
|
|
|
|
.. versionchanged:: 2.5
|
|
This function used an :ctype:`int` type for *size*. This might require
|
|
changes in your code for properly supporting 64-bit systems.
|
|
|
|
|
|
.. cfunction:: Py_UNICODE* PyUnicode_AsUnicode(PyObject *unicode)
|
|
|
|
Return a read-only pointer to the Unicode object's internal :ctype:`Py_UNICODE`
|
|
buffer, *NULL* if *unicode* is not a Unicode object.
|
|
|
|
|
|
.. cfunction:: Py_ssize_t PyUnicode_GetSize(PyObject *unicode)
|
|
|
|
Return the length of the Unicode object.
|
|
|
|
.. versionchanged:: 2.5
|
|
This function returned an :ctype:`int` type. This might require changes
|
|
in your code for properly supporting 64-bit systems.
|
|
|
|
|
|
.. cfunction:: PyObject* PyUnicode_FromEncodedObject(PyObject *obj, const char *encoding, const char *errors)
|
|
|
|
Coerce an encoded object *obj* to an Unicode object and return a reference with
|
|
incremented refcount.
|
|
|
|
String and other char buffer compatible objects are decoded according to the
|
|
given encoding and using the error handling defined by errors. Both can be
|
|
*NULL* to have the interface use the default values (see the next section for
|
|
details).
|
|
|
|
All other objects, including Unicode objects, cause a :exc:`TypeError` to be
|
|
set.
|
|
|
|
The API returns *NULL* if there was an error. The caller is responsible for
|
|
decref'ing the returned objects.
|
|
|
|
|
|
.. cfunction:: PyObject* PyUnicode_FromObject(PyObject *obj)
|
|
|
|
Shortcut for ``PyUnicode_FromEncodedObject(obj, NULL, "strict")`` which is used
|
|
throughout the interpreter whenever coercion to Unicode is needed.
|
|
|
|
If the platform supports :ctype:`wchar_t` and provides a header file wchar.h,
|
|
Python can interface directly to this type using the following functions.
|
|
Support is optimized if Python's own :ctype:`Py_UNICODE` type is identical to
|
|
the system's :ctype:`wchar_t`.
|
|
|
|
|
|
wchar_t Support
|
|
"""""""""""""""
|
|
|
|
wchar_t support for platforms which support it:
|
|
|
|
.. cfunction:: PyObject* PyUnicode_FromWideChar(const wchar_t *w, Py_ssize_t size)
|
|
|
|
Create a Unicode object from the :ctype:`wchar_t` buffer *w* of the given size.
|
|
Return *NULL* on failure.
|
|
|
|
.. versionchanged:: 2.5
|
|
This function used an :ctype:`int` type for *size*. This might require
|
|
changes in your code for properly supporting 64-bit systems.
|
|
|
|
|
|
.. cfunction:: Py_ssize_t PyUnicode_AsWideChar(PyUnicodeObject *unicode, wchar_t *w, Py_ssize_t size)
|
|
|
|
Copy the Unicode object contents into the :ctype:`wchar_t` buffer *w*. At most
|
|
*size* :ctype:`wchar_t` characters are copied (excluding a possibly trailing
|
|
0-termination character). Return the number of :ctype:`wchar_t` characters
|
|
copied or -1 in case of an error. Note that the resulting :ctype:`wchar_t`
|
|
string may or may not be 0-terminated. It is the responsibility of the caller
|
|
to make sure that the :ctype:`wchar_t` string is 0-terminated in case this is
|
|
required by the application.
|
|
|
|
.. versionchanged:: 2.5
|
|
This function returned an :ctype:`int` type and used an :ctype:`int`
|
|
type for *size*. This might require changes in your code for properly
|
|
supporting 64-bit systems.
|
|
|
|
|
|
.. _builtincodecs:
|
|
|
|
Built-in Codecs
|
|
^^^^^^^^^^^^^^^
|
|
|
|
Python provides a set of built-in codecs which are written in C for speed. All of
|
|
these codecs are directly usable via the following functions.
|
|
|
|
Many of the following APIs take two arguments encoding and errors. These
|
|
parameters encoding and errors have the same semantics as the ones of the
|
|
built-in :func:`unicode` Unicode object constructor.
|
|
|
|
Setting encoding to *NULL* causes the default encoding to be used which is
|
|
ASCII. The file system calls should use :cdata:`Py_FileSystemDefaultEncoding`
|
|
as the encoding for file names. This variable should be treated as read-only: On
|
|
some systems, it will be a pointer to a static string, on others, it will change
|
|
at run-time (such as when the application invokes setlocale).
|
|
|
|
Error handling is set by errors which may also be set to *NULL* meaning to use
|
|
the default handling defined for the codec. Default error handling for all
|
|
built-in codecs is "strict" (:exc:`ValueError` is raised).
|
|
|
|
The codecs all use a similar interface. Only deviation from the following
|
|
generic ones are documented for simplicity.
|
|
|
|
|
|
Generic Codecs
|
|
""""""""""""""
|
|
|
|
These are the generic codec APIs:
|
|
|
|
|
|
.. cfunction:: PyObject* PyUnicode_Decode(const char *s, Py_ssize_t size, const char *encoding, const char *errors)
|
|
|
|
Create a Unicode object by decoding *size* bytes of the encoded string *s*.
|
|
*encoding* and *errors* have the same meaning as the parameters of the same name
|
|
in the :func:`unicode` built-in function. The codec to be used is looked up
|
|
using the Python codec registry. Return *NULL* if an exception was raised by
|
|
the codec.
|
|
|
|
.. versionchanged:: 2.5
|
|
This function used an :ctype:`int` type for *size*. This might require
|
|
changes in your code for properly supporting 64-bit systems.
|
|
|
|
|
|
.. cfunction:: PyObject* PyUnicode_Encode(const Py_UNICODE *s, Py_ssize_t size, const char *encoding, const char *errors)
|
|
|
|
Encode the :ctype:`Py_UNICODE` buffer of the given size and return a Python
|
|
string object. *encoding* and *errors* have the same meaning as the parameters
|
|
of the same name in the Unicode :meth:`encode` method. The codec to be used is
|
|
looked up using the Python codec registry. Return *NULL* if an exception was
|
|
raised by the codec.
|
|
|
|
.. versionchanged:: 2.5
|
|
This function used an :ctype:`int` type for *size*. This might require
|
|
changes in your code for properly supporting 64-bit systems.
|
|
|
|
|
|
.. cfunction:: PyObject* PyUnicode_AsEncodedString(PyObject *unicode, const char *encoding, const char *errors)
|
|
|
|
Encode a Unicode object and return the result as Python string object.
|
|
*encoding* and *errors* have the same meaning as the parameters of the same name
|
|
in the Unicode :meth:`encode` method. The codec to be used is looked up using
|
|
the Python codec registry. Return *NULL* if an exception was raised by the
|
|
codec.
|
|
|
|
|
|
UTF-8 Codecs
|
|
""""""""""""
|
|
|
|
These are the UTF-8 codec APIs:
|
|
|
|
|
|
.. cfunction:: PyObject* PyUnicode_DecodeUTF8(const char *s, Py_ssize_t size, const char *errors)
|
|
|
|
Create a Unicode object by decoding *size* bytes of the UTF-8 encoded string
|
|
*s*. Return *NULL* if an exception was raised by the codec.
|
|
|
|
.. versionchanged:: 2.5
|
|
This function used an :ctype:`int` type for *size*. This might require
|
|
changes in your code for properly supporting 64-bit systems.
|
|
|
|
|
|
.. cfunction:: PyObject* PyUnicode_DecodeUTF8Stateful(const char *s, Py_ssize_t size, const char *errors, Py_ssize_t *consumed)
|
|
|
|
If *consumed* is *NULL*, behave like :cfunc:`PyUnicode_DecodeUTF8`. If
|
|
*consumed* is not *NULL*, trailing incomplete UTF-8 byte sequences will not be
|
|
treated as an error. Those bytes will not be decoded and the number of bytes
|
|
that have been decoded will be stored in *consumed*.
|
|
|
|
.. versionadded:: 2.4
|
|
|
|
.. versionchanged:: 2.5
|
|
This function used an :ctype:`int` type for *size*. This might require
|
|
changes in your code for properly supporting 64-bit systems.
|
|
|
|
|
|
.. cfunction:: PyObject* PyUnicode_EncodeUTF8(const Py_UNICODE *s, Py_ssize_t size, const char *errors)
|
|
|
|
Encode the :ctype:`Py_UNICODE` buffer of the given size using UTF-8 and return a
|
|
Python string object. Return *NULL* if an exception was raised by the codec.
|
|
|
|
.. versionchanged:: 2.5
|
|
This function used an :ctype:`int` type for *size*. This might require
|
|
changes in your code for properly supporting 64-bit systems.
|
|
|
|
|
|
.. cfunction:: PyObject* PyUnicode_AsUTF8String(PyObject *unicode)
|
|
|
|
Encode a Unicode object using UTF-8 and return the result as Python string
|
|
object. Error handling is "strict". Return *NULL* if an exception was raised
|
|
by the codec.
|
|
|
|
|
|
UTF-32 Codecs
|
|
"""""""""""""
|
|
|
|
These are the UTF-32 codec APIs:
|
|
|
|
|
|
.. cfunction:: PyObject* PyUnicode_DecodeUTF32(const char *s, Py_ssize_t size, const char *errors, int *byteorder)
|
|
|
|
Decode *length* bytes from a UTF-32 encoded buffer string and return the
|
|
corresponding Unicode object. *errors* (if non-*NULL*) defines the error
|
|
handling. It defaults to "strict".
|
|
|
|
If *byteorder* is non-*NULL*, the decoder starts decoding using the given byte
|
|
order::
|
|
|
|
*byteorder == -1: little endian
|
|
*byteorder == 0: native order
|
|
*byteorder == 1: big endian
|
|
|
|
If ``*byteorder`` is zero, and the first four bytes of the input data are a
|
|
byte order mark (BOM), the decoder switches to this byte order and the BOM is
|
|
not copied into the resulting Unicode string. If ``*byteorder`` is ``-1`` or
|
|
``1``, any byte order mark is copied to the output.
|
|
|
|
After completion, *\*byteorder* is set to the current byte order at the end
|
|
of input data.
|
|
|
|
In a narrow build codepoints outside the BMP will be decoded as surrogate pairs.
|
|
|
|
If *byteorder* is *NULL*, the codec starts in native order mode.
|
|
|
|
Return *NULL* if an exception was raised by the codec.
|
|
|
|
.. versionadded:: 2.6
|
|
|
|
|
|
.. cfunction:: PyObject* PyUnicode_DecodeUTF32Stateful(const char *s, Py_ssize_t size, const char *errors, int *byteorder, Py_ssize_t *consumed)
|
|
|
|
If *consumed* is *NULL*, behave like :cfunc:`PyUnicode_DecodeUTF32`. If
|
|
*consumed* is not *NULL*, :cfunc:`PyUnicode_DecodeUTF32Stateful` will not treat
|
|
trailing incomplete UTF-32 byte sequences (such as a number of bytes not divisible
|
|
by four) as an error. Those bytes will not be decoded and the number of bytes
|
|
that have been decoded will be stored in *consumed*.
|
|
|
|
.. versionadded:: 2.6
|
|
|
|
|
|
.. cfunction:: PyObject* PyUnicode_EncodeUTF32(const Py_UNICODE *s, Py_ssize_t size, const char *errors, int byteorder)
|
|
|
|
Return a Python bytes object holding the UTF-32 encoded value of the Unicode
|
|
data in *s*. Output is written according to the following byte order::
|
|
|
|
byteorder == -1: little endian
|
|
byteorder == 0: native byte order (writes a BOM mark)
|
|
byteorder == 1: big endian
|
|
|
|
If byteorder is ``0``, the output string will always start with the Unicode BOM
|
|
mark (U+FEFF). In the other two modes, no BOM mark is prepended.
|
|
|
|
If *Py_UNICODE_WIDE* is not defined, surrogate pairs will be output
|
|
as a single codepoint.
|
|
|
|
Return *NULL* if an exception was raised by the codec.
|
|
|
|
.. versionadded:: 2.6
|
|
|
|
|
|
.. cfunction:: PyObject* PyUnicode_AsUTF32String(PyObject *unicode)
|
|
|
|
Return a Python string using the UTF-32 encoding in native byte order. The
|
|
string always starts with a BOM mark. Error handling is "strict". Return
|
|
*NULL* if an exception was raised by the codec.
|
|
|
|
.. versionadded:: 2.6
|
|
|
|
|
|
UTF-16 Codecs
|
|
"""""""""""""
|
|
|
|
These are the UTF-16 codec APIs:
|
|
|
|
|
|
.. cfunction:: PyObject* PyUnicode_DecodeUTF16(const char *s, Py_ssize_t size, const char *errors, int *byteorder)
|
|
|
|
Decode *length* bytes from a UTF-16 encoded buffer string and return the
|
|
corresponding Unicode object. *errors* (if non-*NULL*) defines the error
|
|
handling. It defaults to "strict".
|
|
|
|
If *byteorder* is non-*NULL*, the decoder starts decoding using the given byte
|
|
order::
|
|
|
|
*byteorder == -1: little endian
|
|
*byteorder == 0: native order
|
|
*byteorder == 1: big endian
|
|
|
|
If ``*byteorder`` is zero, and the first two bytes of the input data are a
|
|
byte order mark (BOM), the decoder switches to this byte order and the BOM is
|
|
not copied into the resulting Unicode string. If ``*byteorder`` is ``-1`` or
|
|
``1``, any byte order mark is copied to the output (where it will result in
|
|
either a ``\ufeff`` or a ``\ufffe`` character).
|
|
|
|
After completion, *\*byteorder* is set to the current byte order at the end
|
|
of input data.
|
|
|
|
If *byteorder* is *NULL*, the codec starts in native order mode.
|
|
|
|
Return *NULL* if an exception was raised by the codec.
|
|
|
|
.. versionchanged:: 2.5
|
|
This function used an :ctype:`int` type for *size*. This might require
|
|
changes in your code for properly supporting 64-bit systems.
|
|
|
|
|
|
.. cfunction:: PyObject* PyUnicode_DecodeUTF16Stateful(const char *s, Py_ssize_t size, const char *errors, int *byteorder, Py_ssize_t *consumed)
|
|
|
|
If *consumed* is *NULL*, behave like :cfunc:`PyUnicode_DecodeUTF16`. If
|
|
*consumed* is not *NULL*, :cfunc:`PyUnicode_DecodeUTF16Stateful` will not treat
|
|
trailing incomplete UTF-16 byte sequences (such as an odd number of bytes or a
|
|
split surrogate pair) as an error. Those bytes will not be decoded and the
|
|
number of bytes that have been decoded will be stored in *consumed*.
|
|
|
|
.. versionadded:: 2.4
|
|
|
|
.. versionchanged:: 2.5
|
|
This function used an :ctype:`int` type for *size* and an :ctype:`int *`
|
|
type for *consumed*. This might require changes in your code for
|
|
properly supporting 64-bit systems.
|
|
|
|
|
|
.. cfunction:: PyObject* PyUnicode_EncodeUTF16(const Py_UNICODE *s, Py_ssize_t size, const char *errors, int byteorder)
|
|
|
|
Return a Python string object holding the UTF-16 encoded value of the Unicode
|
|
data in *s*. Output is written according to the following byte order::
|
|
|
|
byteorder == -1: little endian
|
|
byteorder == 0: native byte order (writes a BOM mark)
|
|
byteorder == 1: big endian
|
|
|
|
If byteorder is ``0``, the output string will always start with the Unicode BOM
|
|
mark (U+FEFF). In the other two modes, no BOM mark is prepended.
|
|
|
|
If *Py_UNICODE_WIDE* is defined, a single :ctype:`Py_UNICODE` value may get
|
|
represented as a surrogate pair. If it is not defined, each :ctype:`Py_UNICODE`
|
|
values is interpreted as an UCS-2 character.
|
|
|
|
Return *NULL* if an exception was raised by the codec.
|
|
|
|
.. versionchanged:: 2.5
|
|
This function used an :ctype:`int` type for *size*. This might require
|
|
changes in your code for properly supporting 64-bit systems.
|
|
|
|
|
|
.. cfunction:: PyObject* PyUnicode_AsUTF16String(PyObject *unicode)
|
|
|
|
Return a Python string using the UTF-16 encoding in native byte order. The
|
|
string always starts with a BOM mark. Error handling is "strict". Return
|
|
*NULL* if an exception was raised by the codec.
|
|
|
|
|
|
Unicode-Escape Codecs
|
|
"""""""""""""""""""""
|
|
|
|
These are the "Unicode Escape" codec APIs:
|
|
|
|
|
|
.. cfunction:: PyObject* PyUnicode_DecodeUnicodeEscape(const char *s, Py_ssize_t size, const char *errors)
|
|
|
|
Create a Unicode object by decoding *size* bytes of the Unicode-Escape encoded
|
|
string *s*. Return *NULL* if an exception was raised by the codec.
|
|
|
|
.. versionchanged:: 2.5
|
|
This function used an :ctype:`int` type for *size*. This might require
|
|
changes in your code for properly supporting 64-bit systems.
|
|
|
|
|
|
.. cfunction:: PyObject* PyUnicode_EncodeUnicodeEscape(const Py_UNICODE *s, Py_ssize_t size)
|
|
|
|
Encode the :ctype:`Py_UNICODE` buffer of the given size using Unicode-Escape and
|
|
return a Python string object. Return *NULL* if an exception was raised by the
|
|
codec.
|
|
|
|
.. versionchanged:: 2.5
|
|
This function used an :ctype:`int` type for *size*. This might require
|
|
changes in your code for properly supporting 64-bit systems.
|
|
|
|
|
|
.. cfunction:: PyObject* PyUnicode_AsUnicodeEscapeString(PyObject *unicode)
|
|
|
|
Encode a Unicode object using Unicode-Escape and return the result as Python
|
|
string object. Error handling is "strict". Return *NULL* if an exception was
|
|
raised by the codec.
|
|
|
|
|
|
Raw-Unicode-Escape Codecs
|
|
"""""""""""""""""""""""""
|
|
|
|
These are the "Raw Unicode Escape" codec APIs:
|
|
|
|
|
|
.. cfunction:: PyObject* PyUnicode_DecodeRawUnicodeEscape(const char *s, Py_ssize_t size, const char *errors)
|
|
|
|
Create a Unicode object by decoding *size* bytes of the Raw-Unicode-Escape
|
|
encoded string *s*. Return *NULL* if an exception was raised by the codec.
|
|
|
|
.. versionchanged:: 2.5
|
|
This function used an :ctype:`int` type for *size*. This might require
|
|
changes in your code for properly supporting 64-bit systems.
|
|
|
|
|
|
.. cfunction:: PyObject* PyUnicode_EncodeRawUnicodeEscape(const Py_UNICODE *s, Py_ssize_t size, const char *errors)
|
|
|
|
Encode the :ctype:`Py_UNICODE` buffer of the given size using Raw-Unicode-Escape
|
|
and return a Python string object. Return *NULL* if an exception was raised by
|
|
the codec.
|
|
|
|
.. versionchanged:: 2.5
|
|
This function used an :ctype:`int` type for *size*. This might require
|
|
changes in your code for properly supporting 64-bit systems.
|
|
|
|
|
|
.. cfunction:: PyObject* PyUnicode_AsRawUnicodeEscapeString(PyObject *unicode)
|
|
|
|
Encode a Unicode object using Raw-Unicode-Escape and return the result as
|
|
Python string object. Error handling is "strict". Return *NULL* if an exception
|
|
was raised by the codec.
|
|
|
|
|
|
Latin-1 Codecs
|
|
""""""""""""""
|
|
|
|
These are the Latin-1 codec APIs: Latin-1 corresponds to the first 256 Unicode
|
|
ordinals and only these are accepted by the codecs during encoding.
|
|
|
|
|
|
.. cfunction:: PyObject* PyUnicode_DecodeLatin1(const char *s, Py_ssize_t size, const char *errors)
|
|
|
|
Create a Unicode object by decoding *size* bytes of the Latin-1 encoded string
|
|
*s*. Return *NULL* if an exception was raised by the codec.
|
|
|
|
.. versionchanged:: 2.5
|
|
This function used an :ctype:`int` type for *size*. This might require
|
|
changes in your code for properly supporting 64-bit systems.
|
|
|
|
|
|
.. cfunction:: PyObject* PyUnicode_EncodeLatin1(const Py_UNICODE *s, Py_ssize_t size, const char *errors)
|
|
|
|
Encode the :ctype:`Py_UNICODE` buffer of the given size using Latin-1 and return
|
|
a Python string object. Return *NULL* if an exception was raised by the codec.
|
|
|
|
.. versionchanged:: 2.5
|
|
This function used an :ctype:`int` type for *size*. This might require
|
|
changes in your code for properly supporting 64-bit systems.
|
|
|
|
|
|
.. cfunction:: PyObject* PyUnicode_AsLatin1String(PyObject *unicode)
|
|
|
|
Encode a Unicode object using Latin-1 and return the result as Python string
|
|
object. Error handling is "strict". Return *NULL* if an exception was raised
|
|
by the codec.
|
|
|
|
|
|
ASCII Codecs
|
|
""""""""""""
|
|
|
|
These are the ASCII codec APIs. Only 7-bit ASCII data is accepted. All other
|
|
codes generate errors.
|
|
|
|
|
|
.. cfunction:: PyObject* PyUnicode_DecodeASCII(const char *s, Py_ssize_t size, const char *errors)
|
|
|
|
Create a Unicode object by decoding *size* bytes of the ASCII encoded string
|
|
*s*. Return *NULL* if an exception was raised by the codec.
|
|
|
|
.. versionchanged:: 2.5
|
|
This function used an :ctype:`int` type for *size*. This might require
|
|
changes in your code for properly supporting 64-bit systems.
|
|
|
|
|
|
.. cfunction:: PyObject* PyUnicode_EncodeASCII(const Py_UNICODE *s, Py_ssize_t size, const char *errors)
|
|
|
|
Encode the :ctype:`Py_UNICODE` buffer of the given size using ASCII and return a
|
|
Python string object. Return *NULL* if an exception was raised by the codec.
|
|
|
|
.. versionchanged:: 2.5
|
|
This function used an :ctype:`int` type for *size*. This might require
|
|
changes in your code for properly supporting 64-bit systems.
|
|
|
|
|
|
.. cfunction:: PyObject* PyUnicode_AsASCIIString(PyObject *unicode)
|
|
|
|
Encode a Unicode object using ASCII and return the result as Python string
|
|
object. Error handling is "strict". Return *NULL* if an exception was raised
|
|
by the codec.
|
|
|
|
|
|
Character Map Codecs
|
|
""""""""""""""""""""
|
|
|
|
These are the mapping codec APIs:
|
|
|
|
This codec is special in that it can be used to implement many different codecs
|
|
(and this is in fact what was done to obtain most of the standard codecs
|
|
included in the :mod:`encodings` package). The codec uses mapping to encode and
|
|
decode characters.
|
|
|
|
Decoding mappings must map single string characters to single Unicode
|
|
characters, integers (which are then interpreted as Unicode ordinals) or None
|
|
(meaning "undefined mapping" and causing an error).
|
|
|
|
Encoding mappings must map single Unicode characters to single string
|
|
characters, integers (which are then interpreted as Latin-1 ordinals) or None
|
|
(meaning "undefined mapping" and causing an error).
|
|
|
|
The mapping objects provided must only support the __getitem__ mapping
|
|
interface.
|
|
|
|
If a character lookup fails with a LookupError, the character is copied as-is
|
|
meaning that its ordinal value will be interpreted as Unicode or Latin-1 ordinal
|
|
resp. Because of this, mappings only need to contain those mappings which map
|
|
characters to different code points.
|
|
|
|
|
|
.. cfunction:: PyObject* PyUnicode_DecodeCharmap(const char *s, Py_ssize_t size, PyObject *mapping, const char *errors)
|
|
|
|
Create a Unicode object by decoding *size* bytes of the encoded string *s* using
|
|
the given *mapping* object. Return *NULL* if an exception was raised by the
|
|
codec. If *mapping* is *NULL* latin-1 decoding will be done. Else it can be a
|
|
dictionary mapping byte or a unicode string, which is treated as a lookup table.
|
|
Byte values greater that the length of the string and U+FFFE "characters" are
|
|
treated as "undefined mapping".
|
|
|
|
.. versionchanged:: 2.4
|
|
Allowed unicode string as mapping argument.
|
|
|
|
.. versionchanged:: 2.5
|
|
This function used an :ctype:`int` type for *size*. This might require
|
|
changes in your code for properly supporting 64-bit systems.
|
|
|
|
|
|
.. cfunction:: PyObject* PyUnicode_EncodeCharmap(const Py_UNICODE *s, Py_ssize_t size, PyObject *mapping, const char *errors)
|
|
|
|
Encode the :ctype:`Py_UNICODE` buffer of the given size using the given
|
|
*mapping* object and return a Python string object. Return *NULL* if an
|
|
exception was raised by the codec.
|
|
|
|
.. versionchanged:: 2.5
|
|
This function used an :ctype:`int` type for *size*. This might require
|
|
changes in your code for properly supporting 64-bit systems.
|
|
|
|
|
|
.. cfunction:: PyObject* PyUnicode_AsCharmapString(PyObject *unicode, PyObject *mapping)
|
|
|
|
Encode a Unicode object using the given *mapping* object and return the result
|
|
as Python string object. Error handling is "strict". Return *NULL* if an
|
|
exception was raised by the codec.
|
|
|
|
The following codec API is special in that maps Unicode to Unicode.
|
|
|
|
|
|
.. cfunction:: PyObject* PyUnicode_TranslateCharmap(const Py_UNICODE *s, Py_ssize_t size, PyObject *table, const char *errors)
|
|
|
|
Translate a :ctype:`Py_UNICODE` buffer of the given length by applying a
|
|
character mapping *table* to it and return the resulting Unicode object. Return
|
|
*NULL* when an exception was raised by the codec.
|
|
|
|
The *mapping* table must map Unicode ordinal integers to Unicode ordinal
|
|
integers or None (causing deletion of the character).
|
|
|
|
Mapping tables need only provide the :meth:`__getitem__` interface; dictionaries
|
|
and sequences work well. Unmapped character ordinals (ones which cause a
|
|
:exc:`LookupError`) are left untouched and are copied as-is.
|
|
|
|
.. versionchanged:: 2.5
|
|
This function used an :ctype:`int` type for *size*. This might require
|
|
changes in your code for properly supporting 64-bit systems.
|
|
|
|
These are the MBCS codec APIs. They are currently only available on Windows and
|
|
use the Win32 MBCS converters to implement the conversions. Note that MBCS (or
|
|
DBCS) is a class of encodings, not just one. The target encoding is defined by
|
|
the user settings on the machine running the codec.
|
|
|
|
|
|
MBCS codecs for Windows
|
|
"""""""""""""""""""""""
|
|
|
|
|
|
.. cfunction:: PyObject* PyUnicode_DecodeMBCS(const char *s, Py_ssize_t size, const char *errors)
|
|
|
|
Create a Unicode object by decoding *size* bytes of the MBCS encoded string *s*.
|
|
Return *NULL* if an exception was raised by the codec.
|
|
|
|
.. versionchanged:: 2.5
|
|
This function used an :ctype:`int` type for *size*. This might require
|
|
changes in your code for properly supporting 64-bit systems.
|
|
|
|
|
|
.. cfunction:: PyObject* PyUnicode_DecodeMBCSStateful(const char *s, int size, const char *errors, int *consumed)
|
|
|
|
If *consumed* is *NULL*, behave like :cfunc:`PyUnicode_DecodeMBCS`. If
|
|
*consumed* is not *NULL*, :cfunc:`PyUnicode_DecodeMBCSStateful` will not decode
|
|
trailing lead byte and the number of bytes that have been decoded will be stored
|
|
in *consumed*.
|
|
|
|
.. versionadded:: 2.5
|
|
|
|
|
|
.. cfunction:: PyObject* PyUnicode_EncodeMBCS(const Py_UNICODE *s, Py_ssize_t size, const char *errors)
|
|
|
|
Encode the :ctype:`Py_UNICODE` buffer of the given size using MBCS and return a
|
|
Python string object. Return *NULL* if an exception was raised by the codec.
|
|
|
|
.. versionchanged:: 2.5
|
|
This function used an :ctype:`int` type for *size*. This might require
|
|
changes in your code for properly supporting 64-bit systems.
|
|
|
|
|
|
.. cfunction:: PyObject* PyUnicode_AsMBCSString(PyObject *unicode)
|
|
|
|
Encode a Unicode object using MBCS and return the result as Python string
|
|
object. Error handling is "strict". Return *NULL* if an exception was raised
|
|
by the codec.
|
|
|
|
|
|
Methods & Slots
|
|
"""""""""""""""
|
|
|
|
.. _unicodemethodsandslots:
|
|
|
|
Methods and Slot Functions
|
|
^^^^^^^^^^^^^^^^^^^^^^^^^^
|
|
|
|
The following APIs are capable of handling Unicode objects and strings on input
|
|
(we refer to them as strings in the descriptions) and return Unicode objects or
|
|
integers as appropriate.
|
|
|
|
They all return *NULL* or ``-1`` if an exception occurs.
|
|
|
|
|
|
.. cfunction:: PyObject* PyUnicode_Concat(PyObject *left, PyObject *right)
|
|
|
|
Concat two strings giving a new Unicode string.
|
|
|
|
|
|
.. cfunction:: PyObject* PyUnicode_Split(PyObject *s, PyObject *sep, Py_ssize_t maxsplit)
|
|
|
|
Split a string giving a list of Unicode strings. If sep is *NULL*, splitting
|
|
will be done at all whitespace substrings. Otherwise, splits occur at the given
|
|
separator. At most *maxsplit* splits will be done. If negative, no limit is
|
|
set. Separators are not included in the resulting list.
|
|
|
|
.. versionchanged:: 2.5
|
|
This function used an :ctype:`int` type for *maxsplit*. This might require
|
|
changes in your code for properly supporting 64-bit systems.
|
|
|
|
|
|
.. cfunction:: PyObject* PyUnicode_Splitlines(PyObject *s, int keepend)
|
|
|
|
Split a Unicode string at line breaks, returning a list of Unicode strings.
|
|
CRLF is considered to be one line break. If *keepend* is 0, the Line break
|
|
characters are not included in the resulting strings.
|
|
|
|
|
|
.. cfunction:: PyObject* PyUnicode_Translate(PyObject *str, PyObject *table, const char *errors)
|
|
|
|
Translate a string by applying a character mapping table to it and return the
|
|
resulting Unicode object.
|
|
|
|
The mapping table must map Unicode ordinal integers to Unicode ordinal integers
|
|
or None (causing deletion of the character).
|
|
|
|
Mapping tables need only provide the :meth:`__getitem__` interface; dictionaries
|
|
and sequences work well. Unmapped character ordinals (ones which cause a
|
|
:exc:`LookupError`) are left untouched and are copied as-is.
|
|
|
|
*errors* has the usual meaning for codecs. It may be *NULL* which indicates to
|
|
use the default error handling.
|
|
|
|
|
|
.. cfunction:: PyObject* PyUnicode_Join(PyObject *separator, PyObject *seq)
|
|
|
|
Join a sequence of strings using the given separator and return the resulting
|
|
Unicode string.
|
|
|
|
|
|
.. cfunction:: int PyUnicode_Tailmatch(PyObject *str, PyObject *substr, Py_ssize_t start, Py_ssize_t end, int direction)
|
|
|
|
Return 1 if *substr* matches *str*[*start*:*end*] at the given tail end
|
|
(*direction* == -1 means to do a prefix match, *direction* == 1 a suffix match),
|
|
0 otherwise. Return ``-1`` if an error occurred.
|
|
|
|
.. versionchanged:: 2.5
|
|
This function used an :ctype:`int` type for *start* and *end*. This
|
|
might require changes in your code for properly supporting 64-bit
|
|
systems.
|
|
|
|
|
|
.. cfunction:: Py_ssize_t PyUnicode_Find(PyObject *str, PyObject *substr, Py_ssize_t start, Py_ssize_t end, int direction)
|
|
|
|
Return the first position of *substr* in *str*[*start*:*end*] using the given
|
|
*direction* (*direction* == 1 means to do a forward search, *direction* == -1 a
|
|
backward search). The return value is the index of the first match; a value of
|
|
``-1`` indicates that no match was found, and ``-2`` indicates that an error
|
|
occurred and an exception has been set.
|
|
|
|
.. versionchanged:: 2.5
|
|
This function used an :ctype:`int` type for *start* and *end*. This
|
|
might require changes in your code for properly supporting 64-bit
|
|
systems.
|
|
|
|
|
|
.. cfunction:: Py_ssize_t PyUnicode_Count(PyObject *str, PyObject *substr, Py_ssize_t start, Py_ssize_t end)
|
|
|
|
Return the number of non-overlapping occurrences of *substr* in
|
|
``str[start:end]``. Return ``-1`` if an error occurred.
|
|
|
|
.. versionchanged:: 2.5
|
|
This function returned an :ctype:`int` type and used an :ctype:`int`
|
|
type for *start* and *end*. This might require changes in your code for
|
|
properly supporting 64-bit systems.
|
|
|
|
|
|
.. cfunction:: PyObject* PyUnicode_Replace(PyObject *str, PyObject *substr, PyObject *replstr, Py_ssize_t maxcount)
|
|
|
|
Replace at most *maxcount* occurrences of *substr* in *str* with *replstr* and
|
|
return the resulting Unicode object. *maxcount* == -1 means replace all
|
|
occurrences.
|
|
|
|
.. versionchanged:: 2.5
|
|
This function used an :ctype:`int` type for *maxcount*. This might
|
|
require changes in your code for properly supporting 64-bit systems.
|
|
|
|
|
|
.. cfunction:: int PyUnicode_Compare(PyObject *left, PyObject *right)
|
|
|
|
Compare two strings and return -1, 0, 1 for less than, equal, and greater than,
|
|
respectively.
|
|
|
|
|
|
.. cfunction:: int PyUnicode_RichCompare(PyObject *left, PyObject *right, int op)
|
|
|
|
Rich compare two unicode strings and return one of the following:
|
|
|
|
* ``NULL`` in case an exception was raised
|
|
* :const:`Py_True` or :const:`Py_False` for successful comparisons
|
|
* :const:`Py_NotImplemented` in case the type combination is unknown
|
|
|
|
Note that :const:`Py_EQ` and :const:`Py_NE` comparisons can cause a
|
|
:exc:`UnicodeWarning` in case the conversion of the arguments to Unicode fails
|
|
with a :exc:`UnicodeDecodeError`.
|
|
|
|
Possible values for *op* are :const:`Py_GT`, :const:`Py_GE`, :const:`Py_EQ`,
|
|
:const:`Py_NE`, :const:`Py_LT`, and :const:`Py_LE`.
|
|
|
|
|
|
.. cfunction:: PyObject* PyUnicode_Format(PyObject *format, PyObject *args)
|
|
|
|
Return a new string object from *format* and *args*; this is analogous to
|
|
``format % args``. The *args* argument must be a tuple.
|
|
|
|
|
|
.. cfunction:: int PyUnicode_Contains(PyObject *container, PyObject *element)
|
|
|
|
Check whether *element* is contained in *container* and return true or false
|
|
accordingly.
|
|
|
|
*element* has to coerce to a one element Unicode string. ``-1`` is returned if
|
|
there was an error.
|