998c9cdd42
Issue #28561 : Clean up UTF-8 encoder: remove dead code, update comments, etc.
...
Patch by Xiang Zhang.
2016-10-30 18:25:27 +02:00
f051e43b22
Issue #28126 : Replace Py_MEMCPY with memcpy(). Visual Studio can properly optimize memcpy().
2016-09-13 20:22:02 +02:00
621b430a14
remove all usage of Py_LOCAL
2016-09-09 13:54:34 -07:00
1a05d6c04d
PEP 7 style for if/else in C
...
Add also a newline for readability in normalize_encoding().
2016-09-02 12:12:23 +02:00
15f44ab043
Issue #27895 : Spelling fixes (Contributed by Ville Skyttä).
2016-08-30 10:47:49 -07:00
e09132f2c7
Backed out changeset b0087e17cd5e (issue #26765 )
...
For unknown reasons it perhaps caused a crash on 32-bit Windows (issue #).
2016-07-03 13:57:48 +03:00
355048970b
Issue #26765 : Moved wrappers for bytes and bytearray methods to common header
...
file.
2016-07-01 17:57:30 +03:00
bcde10aa7e
Issue #26765 : Ensure that bytes- and unicode-specific stringlib files are used
...
with correct type.
2016-05-16 09:42:29 +03:00
fb81d3cbe7
Issue #26765 : Moved common code for the replace() method of bytes and bytearray
...
to a template file.
2016-05-05 09:26:07 +03:00
dd40fc3e57
Issue #26765 : Moved common code and docstrings for bytes and bytearray methods
...
to bytes_methods.c.
2016-05-04 22:23:26 +03:00
b6a9c9761c
Issue #26778 : Fixed "a/an/and" typos in code comment, documentation and error
...
messages.
2016-04-17 09:39:28 +03:00
6a7b3a77b4
Issue #26778 : Fixed "a/an/and" typos in code comment and documentation.
2016-04-17 08:32:47 +03:00
21a663ea28
Issue #26057 : Got rid of nonneeded use of PyUnicode_FromObject().
2016-04-13 15:37:23 +03:00
413fdcea21
Issue #24821 : Refactor STRINGLIB(fastsearch_memchr_1char) and split it on
...
STRINGLIB(find_char) and STRINGLIB(rfind_char) that can be used independedly
without special preconditions.
2015-11-14 15:42:17 +02:00
6bd525b656
Optimize error handlers of ASCII and Latin1 encoders when the replacement
...
string is pure ASCII: use _PyBytesWriter_WriteBytes(), don't check individual
character.
Cleanup unicode_encode_ucs1():
* Rename repunicode to rep
* Clear rep object on error
* Factorize code between bytes and unicode path
2015-10-09 13:10:05 +02:00
ce179bf6ba
Add _PyBytesWriter_WriteBytes() to factorize the code
2015-10-09 12:57:22 +02:00
ad7715891e
_PyBytesWriter: simplify code to avoid "prealloc" parameters
...
Substract preallocate bytes from min_size before calling
_PyBytesWriter_Prepare().
2015-10-09 12:38:53 +02:00
e7bf86cd7d
Optimize backslashreplace error handler
...
Issue #25318 : Optimize backslashreplace and xmlcharrefreplace error handlers in
UTF-8 encoder. Optimize also backslashreplace error handler for ASCII and
Latin1 encoders.
Use the new _PyBytesWriter API to optimize these error handlers for the
encoders. It avoids to create an exception and call the slow implementation of
the error handler.
2015-10-09 01:39:28 +02:00
fdfbf78114
Issue #25318 : Add _PyBytesWriter API
...
Add a new private API to optimize Unicode encoders. It uses a small buffer
allocated on the stack and supports overallocation.
Use _PyBytesWriter API for UCS1 (ASCII and Latin1) and UTF-8 encoders. Enable
overallocation for the UTF-8 encoder with error handlers.
unicode_encode_ucs1(): initialize collend to collstart+1 to not check the
current character twice, we already know that it is not ASCII.
2015-10-09 00:33:49 +02:00
01ada3996b
Issue #25267 : The UTF-8 encoder is now up to 75 times as fast for error
...
handlers: ``ignore``, ``replace``, ``surrogateescape``, ``surrogatepass``.
Patch co-written with Serhiy Storchaka.
2015-10-01 21:54:51 +02:00
ab2aa6dc91
Fixed an incorrect comment.
2015-08-26 14:10:32 -04:00
9ce71a6475
Fixed typos in comments.
2015-05-18 22:20:18 +03:00
7e29eea926
Fixed typos in comments.
2015-05-18 22:19:42 +03:00
0d4df752ac
Issue #15027 : The UTF-32 encoder is now 3x to 7x faster.
2015-05-12 23:12:45 +03:00
d9d769fcdd
Issue #23573 : Increased performance of string search operations (str.find,
...
str.index, str.count, the in operator, str.split, str.partition) with
arguments of different kinds (UCS1, UCS2, UCS4).
2015-03-24 21:55:47 +02:00
009b811d67
Removed unintentional trailing spaces in non-external and non-generated C files.
2015-03-18 21:53:15 +02:00
4fdb68491e
Issue #22896 : Avoid to use PyObject_AsCharBuffer(), PyObject_AsReadBuffer()
...
and PyObject_AsWriteBuffer().
2015-02-03 01:21:08 +02:00
b757c83ec6
Issue #22581 : Use more "bytes-like object" throughout the docs and comments.
2014-12-05 22:25:22 +02:00
1cc9520327
s/stringobject/bytesobject/ ( closes #22036 )
...
Patch by Martin Matusiak.
2014-07-23 21:39:37 -07:00
d455ce4fd4
merge 3.3
2014-03-30 19:52:39 -04:00
0ad6098b67
merge 3.2
2014-03-30 19:52:22 -04:00
23cf403ca1
fix expandtabs overflow detection to be consistent and not rely on signed overflow
2014-03-30 19:47:57 -04:00
3079328d29
Reverted changeset b72c5573c5e7 (issue #15027 ).
2014-01-04 22:44:01 +02:00
583a93943c
Issue #15027 : Rewrite the UTF-32 encoder. It is now 1.6x to 3.5x faster.
2014-01-04 19:25:37 +02:00
0ee22bf774
fix format spec recursive expansion ( closes #19729 )
2013-11-26 19:22:36 -06:00
dc2fd5101a
Remove dead code committed in issue #12892 .
2013-11-19 15:56:05 +02:00
58cf607d13
Issue #12892 : The utf-16* and utf-32* codecs now reject (lone) surrogates.
...
The utf-16* and utf-32* encoders no longer allow surrogate code points
(U+D800-U+DFFF) to be encoded.
The utf-32* decoders no longer decode byte sequences that correspond to
surrogate code points.
The surrogatepass error handler now works with the utf-16* and utf-32* codecs.
Based on patches by Victor Stinner and Kang-Hao (Kenny) Lu.
2013-11-19 11:32:41 +02:00
745d54d2fa
#17806 : Added keyword-argument support for "tabsize" to str/bytes.expandtabs().
2013-11-16 19:10:57 +02:00
cc64eb5b9f
Issue #18408 : Fix bytearrayiter.partition()/rpartition(), handle
...
PyByteArray_FromStringAndSize() failure (ex: on memory allocation failure)
2013-10-29 03:15:37 +01:00
8fa8ee3970
Issue #18701 : Remove support of old CPython versions (<3.0) from C code.
2013-08-17 00:48:02 +03:00
d06eeb4a24
merge
2013-08-13 18:20:55 -07:00
b1b915c796
Issue 18719: Remove a false optimization
...
Remove an unused early-out test from the critical path for
dict and set lookups.
When the strings already have matching lengths, kinds, and hashes,
there is no additional information gained by checking the first
characters (the probability of a mismatch is already known to
be less than 1 in 2**64).
2013-08-13 18:16:34 -07:00
9ed5f27266
Issue #18722 : Remove uses of the "register" keyword in C code.
2013-08-13 20:18:52 +02:00
d2b58a9880
only recursively expand in the format spec ( closes #17644 )
2013-05-17 17:34:30 -05:00
4d94474ba3
rewrite the parsing of field names to be more consistent wrt recursive expansion
2013-05-17 18:22:31 -05:00
48953632df
merge 3.3
2013-05-17 17:35:28 -05:00
5263c13801
Merge removal of trailing whitespace from 3.3.
2013-04-21 04:08:18 +03:00
6b02772c13
Remove trailing whitespace.
2013-04-21 04:07:51 +03:00
8f674ccd64
Close #17694 : Add minimum length to _PyUnicodeWriter
...
* Add also min_char attribute to _PyUnicodeWriter structure (currently unused)
* _PyUnicodeWriter_Init() has no more argument (except the writer itself):
min_length and overallocate must be set explicitly
* In error handlers, only enable overallocation if the replacement string
is longer than 1 character
* CJK decoders don't use overallocation anymore
* Set min_length, instead of preallocating memory using
_PyUnicodeWriter_Prepare(), in many decoders
* _PyUnicode_DecodeUnicodeInternal() checks for integer overflow
2013-04-17 23:02:17 +02:00
76b3b2726c
stringlib: remove unused STRINGLIB_RESIZE macro
2013-04-14 16:29:09 +02:00
e2cef885a2
Issue #16061 : Speed up str.replace() for replacing 1-character strings.
2013-04-13 22:45:04 +03:00
7efa3b8242
Close #13126 : "Simplify" FASTSEARCH() code to help the compiler to emit more
...
efficient machine code. Patch written by Antoine Pitrou.
Without this change, str.find() was 10% slower than str.rfind() in the worst
case.
2013-04-08 00:26:43 +02:00
cfc4c13b04
Add _PyUnicodeWriter_WriteSubstring() function
...
Write a function to enable more optimizations:
* If the substring is the whole string and overallocation is disabled, just
keep a reference to the string, don't copy characters
* Avoid a call to the expensive _PyUnicode_FindMaxChar() function when
possible
2013-04-03 01:48:39 +02:00
06b16f879f
Remove unused defines.
2013-02-23 14:49:09 +02:00
18809fa94e
Remove unused defines.
2013-02-23 14:48:16 +02:00
4de7457009
Issue #17173 : Remove uses of locale-dependent C functions (isalpha() etc.) in the interpreter.
...
I've left a couple of them in: zlib (third-party lib), getaddrinfo.c
(doesn't include Python.h, and probably obsolete), _sre.c (legitimate
use for the re.LOCALE flag).
2013-02-09 23:11:27 +01:00
b946af5897
Check for NULL before the pointer aligning in fastsearch_memchr_1char.
...
There is no guarantee that NULL is aligned.
2013-01-15 13:32:41 +02:00
18ba40b945
Check for NULL before the pointer aligning in fastsearch_memchr_1char.
...
There is no guarantee that NULL is aligned.
2013-01-15 13:27:28 +02:00
5f7e8dab11
Issue #16592 : stringlib_bytes_join doesn't raise MemoryError on allocation failure
2012-12-02 07:56:42 +01:00
6caa6fb535
(Merge 3.3) Issue #8271 : Fix compilation on Windows
2012-11-05 00:00:50 +01:00
ab60de478d
Issue #8271 : Fix compilation on Windows
2012-11-04 23:59:15 +01:00
cfa9636404
#8271 : merge with 3.3.
2012-11-04 23:23:09 +02:00
f7ed5d111b
#8271 : the utf-8 decoder now outputs the correct number of U+FFFD characters when used with the "replace" error handler on invalid utf-8 sequences. Patch by Serhiy Storchaka, tests by Ezio Melotti.
2012-11-04 23:21:38 +02:00
6f7b0da6bc
Issue #12805 : Make bytes.join and bytearray.join faster when the separator is empty.
...
Patch by Serhiy Storchaka.
2012-10-20 23:08:34 +02:00
743e0cd6b5
Issue #16166 : Add PY_LITTLE_ENDIAN and PY_BIG_ENDIAN macros and unified
...
endianess detection and handling.
2012-10-17 23:52:17 +02:00
cfc22b4a9b
Issue #15958 : bytes.join and bytearray.join now accept arbitrary buffer objects.
2012-10-16 21:07:23 +02:00
ca8aa4acf6
Issue #15144 : Fix possible integer overflow when handling pointers as integer values, by using Py_uintptr_t instead of size_t.
...
Patch by Serhiy Storchaka.
2012-09-20 20:56:47 +02:00
b3f5501250
Close #15534 : Fix a typo in the fast search function of the string library (_s => s)
...
Replace _s with ptr to avoid future confusion. Add also non regression tests.
2012-08-02 23:05:01 +02:00
fb90c0934c
Issue #14700 : Fix buggy overflow checks for large precision and width in new-style and old-style formatting.
2012-10-28 10:18:03 +00:00
01ac8b6ab1
Use correct types for ASCII_CHAR_MASK integer constants.
2012-07-07 14:08:48 +02:00
106c4145ff
Issue #14923 : Optimize continuation-byte check in UTF-8 decoding. Patch by Serhiy Storchaka.
2012-06-23 21:45:14 +01:00
a759d4e9f4
Make private function static (from `make smelly`)
2012-06-21 17:26:28 +02:00
27f6a3b0bf
Issue #15026 : utf-16 encoding is now significantly faster (up to 10x).
...
Patch by Serhiy Storchaka.
2012-06-15 22:15:23 +02:00
d7b7c7472b
Issue #14993 : Use standard "unsigned char" instead of a unsigned char bitfield
2012-06-04 22:52:12 +02:00
d3f0882dfb
Issue #14744 : Use the new _PyUnicodeWriter internal API to speed up str%args and str.format(args)
...
* Formatting string, int, float and complex use the _PyUnicodeWriter API. It
avoids a temporary buffer in most cases.
* Add _PyUnicodeWriter_WriteStr() to restore the PyAccu optimization: just
keep a reference to the string if the output is only composed of one string
* Disable overallocation when formatting the last argument of str%args and
str.format(args)
* Overallocation allocates at least 100 characters: add min_length attribute
to the _PyUnicodeWriter structure
* Add new private functions: _PyUnicode_FastCopyCharacters(),
_PyUnicode_FastFill() and _PyUnicode_FromASCII()
The speed up is around 20% in average.
2012-05-29 12:57:52 +02:00
63065d761e
Issue #14624 : UTF-16 decoding is now 3x to 4x faster on various inputs.
...
Patch by Serhiy Storchaka.
2012-05-15 23:48:04 +02:00
ca5f91b888
Issue #14738 : Speed-up UTF-8 decoding on non-ASCII data. Patch by Serhiy Storchaka.
2012-05-10 16:36:02 +02:00
3b1a74a9c3
Rename unicode_write_t structure and its methods to "_PyUnicodeWriter"
2012-05-09 22:25:00 +02:00
ee4544c920
Issue #14744 : Inline unicode_writer_write_char() and unicode_write_str()
...
Optimize also PyUnicode_Format(): call unicode_writer_prepare() only once
per argument.
2012-05-09 22:24:08 +02:00
202fdca133
Close #14716 : str.format() now uses the new "unicode writer" API instead of the
...
PyAccu API. For example, it makes str.format() from 25% to 30% faster on Linux.
2012-05-07 12:47:02 +02:00
d0acb411ef
Issue #14387 : Do not include accu.h from Python.h.
2012-03-22 14:42:18 +01:00
41a863cb81
Issue #13706 : Fix format(int, "n") for locale with non-ASCII thousands separator
...
* Decode thousands separator and decimal point using PyUnicode_DecodeLocale()
(from the locale encoding), instead of decoding them implicitly from latin1
* Remove _PyUnicode_InsertThousandsGroupingLocale(), it was not used
* Change _PyUnicode_InsertThousandsGrouping() API to return the maximum
character if unicode is NULL
* Replace MIN/MAX macros by Py_MIN/Py_MAX
* stringlib/undef.h undefines STRINGLIB_IS_UNICODE
* stringlib/localeutil.h only supports Unicode
2012-02-24 00:37:51 +01:00
21e0da228d
remove some usage of Py_UNICODE_TOUPPER/LOWER
2012-01-11 21:00:42 -05:00
6099a03202
Issue #13624 : Write a specialized UTF-8 encoder to allow more optimization
...
The main bottleneck was the PyUnicode_READ() macro.
2011-12-18 14:22:26 +01:00
f8eac00779
Issue #13623 : Fix a performance regression introduced by issue #12170 in
...
bytes.find() and handle correctly OverflowError (raise the same ValueError than
the error for -1).
2011-12-18 01:17:41 +01:00
b37b17423b
Replace PyUnicode_FromUnicode(NULL, 0) by PyUnicode_New(0, 0)
...
Create an empty string with the new Unicode API.
2011-12-01 03:18:59 +01:00
0a3229de6b
Issue #13417 : speed up utf-8 decoding by around 2x for the non-fully-ASCII case.
...
This almost catches up with pre-PEP 393 performance, when decoding needed
only one pass.
2011-11-21 20:39:13 +01:00
0fc35196bb
stringlib: remove unused STRINGLIB_FILL
2011-11-20 19:30:15 +01:00
7931d9a951
Replace PyUnicodeObject type by PyObject
...
* _PyUnicode_CheckConsistency() now takes a PyObject* instead of void*
* Remove now useless casts to PyObject*
2011-11-04 00:22:48 +01:00
9db1a8b69f
Replace PyUnicodeObject* by PyObject* where it was irrevelant
...
A Unicode string can now be a PyASCIIObject, PyCompactUnicodeObject or
PyUnicodeObject. Aliasing a PyASCIIObject* or PyCompactUnicodeObject* to
PyUnicodeObject* is wrong
2011-10-23 20:04:37 +02:00
ac65d96777
Issue #12170 : The count(), find(), rfind(), index() and rindex() methods
...
of bytes and bytearray objects now accept an integer between 0 and 255
as their first argument. Patch by Petri Lehtinen.
2011-10-20 23:54:17 +02:00
5b9f4c1539
Fix typo
2011-10-17 19:21:04 +02:00
c198d0599b
Add a comment explaining this heuristic.
2011-10-13 18:07:37 +02:00
dda339e6d2
Simplify heuristic for when to use memchr
2011-10-13 17:58:11 +02:00
dd4e2f0153
Issue #13155 : Optimize finding the optimal character width of an unicode string
2011-10-13 00:02:27 +02:00
d218bf14cc
stringlib: Fix STRINGLIB_STR for UCS2/UCS4
2011-10-12 00:14:32 +02:00
8cc70dcf70
Fix fastsearch for UCS2 and UCS4
...
* If needle is 0, try (p[0] >> 16) & 0xff for UCS4
* Disable fastsearch_memchr_1char() if needle is zero for UCS2 and UCS4
2011-10-11 23:22:22 +02:00
2c3b2302ad
Issue #13134 : optimize finding single-character strings using memchr
2011-10-11 20:29:21 +02:00
c47adb04b3
Change PyUnicode_KIND to 1,2,4. Drop _KIND_SIZE and _CHARACTER_SIZE.
2011-10-07 20:55:35 +02:00
4574e62c6e
Fix massive slowdown in string formatting with str.format.
...
Example:
./python -m timeit -s "f='{}' + '-' * 1024 + '{}'; s='abcd' * 16384" "f.format(s, s)"
-> before: 547 usec per loop
-> after: 13 usec per loop
-> 3.2: 22.5 usec per loop
-> 2.7: 12.6 usec per loop
2011-10-07 02:26:47 +02:00
Serhiy Storchaka
Christian Heimes
Benjamin Peterson
Victor Stinner
Raymond Hettinger
Eric V. Smith
Ezio Melotti
Antoine Pitrou
Christian Heimes
Mark Dickinson
Victor Stinner
Martin v. Löwis