eaab604829
Fix fixup() for unchanged unicode subtype
...
If maxchar_new == 0 and self is a unicode subtype, return u instead of duplicating u.
2011-12-11 22:22:39 +01:00
e6b2d4407a
unicode_fromascii() doesn't check string content twice in debug mode
...
_PyUnicode_CheckConsistency() also checks string content.
2011-12-11 21:54:30 +01:00
a1d12bb119
Call directly PyUnicode_DecodeUTF8Stateful() instead of PyUnicode_DecodeUTF8()
...
* Remove micro-optimization from PyUnicode_FromStringAndSize():
PyUnicode_DecodeUTF8Stateful() has already these optimizations (for size=0
and one ascii char).
* Rename utf8_max_char_size_and_char_count() to utf8_scanner(), and remove an
useless variable
2011-12-11 21:53:09 +01:00
382955ff4e
Use directly unicode_empty instead of PyUnicode_New(0, 0)
2011-12-11 21:44:00 +01:00
785938eebd
Move the slowest UTF-8 decoder to its own subfunction
...
* Create decode_utf8_errors()
* Reuse unicode_fromascii()
* decode_utf8_errors() doesn't refit at the beginning
* Remove refit_partial_string(), use unicode_adjust_maxchar() instead
2011-12-11 20:09:03 +01:00
84def3774d
Fix error handling in resize_compact()
2011-12-11 20:04:56 +01:00
8faf8216e4
PyUnicode_FromWideChar() and PyUnicode_FromUnicode() raise a ValueError if a
...
character in not in range [U+0000; U+10ffff].
2011-12-08 22:14:11 +01:00
551ac95733
Py_UNICODE_HIGH_SURROGATE() and Py_UNICODE_LOW_SURROGATE() macros
...
And use surrogates macros everywhere in unicodeobject.c
2011-11-29 22:58:13 +01:00
6345be9a14
Close #13093 : PyUnicode_EncodeDecimal() doesn't support error handlers
...
different than "strict" anymore. The caller was unable to compute the
size of the output buffer: it depends on the error handler.
2011-11-25 20:09:01 +01:00
1518e8713d
and back to the "magic" formula (with a comment) it is
2011-11-23 10:44:52 -06:00
5944c36931
cave to those who like readable code
2011-11-22 19:05:49 -06:00
0268675193
fix compiler warning by implementing this more cleverly
2011-11-22 15:29:32 -05:00
ca4f20782e
find_maxchar_surrogates() reuses surrogate macros
2011-11-22 03:38:40 +01:00
0d3721d986
Issue #13441 : Disable temporary the check on the maximum character until
...
the Solaris issue is solved.
But add assertion on the maximum character in various encoders: UTF-7, UTF-8,
wide character (wchar_t*, Py_UNICODE*), unicode-escape, raw-unicode-escape.
Fix also unicode_encode_ucs1() for backslashreplace error handler: Python is
now always "wide".
2011-11-22 03:27:53 +01:00
f8facacf30
Fix compiler warnings
2011-11-22 02:30:47 +01:00
b84d723509
(Merge 3.2) Issue #13093 : Fix error handling on PyUnicode_EncodeDecimal()
2011-11-22 01:50:07 +01:00
cfed46e00a
PyUnicode_FromKindAndData() fails with a ValueError if size < 0
2011-11-22 01:29:14 +01:00
42885206ec
UTF-8 decoder: set consumed value in the latin1 fast-path
2011-11-22 01:23:02 +01:00
d3df8ab377
Replace _PyUnicode_READY_REPLACE() and _PyUnicode_ReadyReplace() with unicode_ready()
...
* unicode_ready() has a simpler API
* try to reuse unicode_empty and latin1_char singleton everywhere
* Fix a reference leak in _PyUnicode_TranslateCharmap()
* PyUnicode_InternInPlace() doesn't try to get a singleton anymore, to avoid
having to handle a failure
2011-11-22 01:22:34 +01:00
f01245067a
Rewrite PyUnicode_TransformDecimalToASCII() to use the new Unicode API
2011-11-21 23:12:56 +01:00
2d718f39a5
Remove an unused variable from PyUnicode_Copy()
2011-11-21 23:11:52 +01:00
87af4f2f3a
Simplify PyUnicode_Copy()
...
USe PyUnicode_Copy() in fixup()
2011-11-21 23:03:47 +01:00
5bbe5e7c85
Fix a compiler warning in _PyUnicode_CheckConsistency()
2011-11-21 22:54:05 +01:00
42bf77537e
Rewrite PyUnicode_EncodeDecimal() to use the new Unicode API
...
Add tests for PyUnicode_EncodeDecimal() and
PyUnicode_TransformDecimalToASCII().
2011-11-21 22:52:58 +01:00
0a3229de6b
Issue #13417 : speed up utf-8 decoding by around 2x for the non-fully-ASCII case.
...
This almost catches up with pre-PEP 393 performance, when decoding needed
only one pass.
2011-11-21 20:39:13 +01:00
da29cc36aa
Issue #13441 : _PyUnicode_CheckConsistency() dumps the string if the maximum
...
character is bigger than U+10FFFF and locale.localeconv() dumps the string
before decoding it.
Temporary hack to debug the issue #13441 .
2011-11-21 14:31:41 +01:00
9e30aa52fd
Fix misuse of PyUnicode_GET_SIZE() => PyUnicode_GET_LENGTH()
...
And PyUnicode_GetSize() => PyUnicode_GetLength()
2011-11-21 02:49:52 +01:00
4ead7c7be8
PyObject_Str() ensures that the result string is ready
...
and check the string consistency.
_PyUnicode_CheckConsistency() doesn't check the hash anymore. It should be
possible to call this function even if hash(str) was already called.
2011-11-20 19:48:36 +01:00
b960b34577
PyUnicode_AsUTF32String() calls directly _PyUnicode_EncodeUTF32(),
...
instead of calling the deprecated PyUnicode_EncodeUTF32() function
2011-11-20 19:12:52 +01:00
77faf69ca1
_PyUnicode_CheckConsistency() also checks maxchar maximum value,
...
not only its minimum value
2011-11-20 18:56:05 +01:00
d5c4022d2a
Remove the two ugly and unused WRITE_ASCII_OR_WSTR and WRITE_WSTR macros
2011-11-20 18:41:31 +01:00
2e9cfadd7c
Reuse surrogate macros in UTF-16 decoder
2011-11-20 18:40:27 +01:00
ae4f7c8e59
charmap_encoding_error() uses the new Unicode API
2011-11-20 18:28:55 +01:00
ac931b1e5b
Use PyUnicode_EncodeCodePage() instead of PyUnicode_EncodeMBCS() with
...
PyUnicode_AsUnicodeAndSize()
2011-11-20 18:27:03 +01:00
22168998f5
charmap encoders uses Py_UCS4, not Py_UNICODE
2011-11-20 17:09:18 +01:00
1f7951711c
Catch PyUnicode_AS_UNICODE() errors
2011-11-17 00:45:54 +01:00
11060a4a48
#13406 : silence deprecation warnings in test_codecs.
2011-11-16 09:39:10 +02:00
78edf7576e
Issue #13333 : The UTF-7 decoder now accepts lone surrogates
...
(the encoder already accepts them).
2011-11-15 01:44:16 +01:00
5418ee0b9a
Issue #13333 : The UTF-7 decoder now accepts lone surrogates
...
(the encoder already accepts them).
2011-11-15 01:42:21 +01:00
31b92a534f
Sanitize reference management in the utf-8 encoder
2011-11-12 18:35:19 +01:00
0290c7a811
Fix regression on 2-byte wchar_t systems (Windows)
2011-11-11 13:29:12 +01:00
44c6affc79
Avoid crashing because of an unaligned word access
2011-11-11 02:59:42 +01:00
de20b0b50e
Issue #13149 : Speed up append-only StringIO objects.
...
This is very similar to the "lazy strings" idea.
2011-11-10 21:47:38 +01:00
9f4b1e9c50
Fix and deprecated the unicode_internal codec
...
unicode_internal codec uses Py_UNICODE instead of the real internal
representation (PEP 393: Py_UCS1, Py_UCS2 or Py_UCS4) for backward
compatibility.
2011-11-10 20:56:30 +01:00
24729f36bf
Prefer Py_UCS4 or wchar_t over Py_UNICODE
2011-11-10 20:31:37 +01:00
ebf3ba808e
PyUnicode_DecodeCharmap() uses the new Unicode API
2011-11-10 20:30:22 +01:00
a98b28c1bf
Avoid PyUnicode_AS_UNICODE in the UTF-8 encoder
2011-11-10 20:21:49 +01:00
3326cb6a36
Fix "unicode_escape" encoder
2011-11-10 20:15:25 +01:00
0e36826a04
Fix UTF-7 encoder on Windows
2011-11-10 20:12:49 +01:00
1db7c13be1
Port encoders from Py_UNICODE API to unicode object API.
2011-11-10 18:24:32 +01:00
Victor Stinner
Benjamin Peterson
Antoine Pitrou
Ezio Melotti
Martin v. Löwis