c4f281eba3
Fix misuse of PyUnicode_GET_SIZE, use PyUnicode_GET_LENGTH instead
2011-10-11 22:11:42 +02:00
e459a0877e
Issue #13136 : speed up conversion between different character widths.
2011-10-11 20:58:41 +02:00
2871698546
/* Remove unused code. It has been committed out since 2000 (!). */
2011-10-11 03:17:47 +02:00
53bb548f22
Avoid exporting private helpers
...
(thanks "make smelly")
2011-10-10 23:49:24 +02:00
794d567b17
any_find_slice() doesn't use callbacks anymore
...
* Call directly the right find/rfind method: allow inlining functions
* Remove Py_LOCAL_CALLBACK (added for any_find_slice)
2011-10-10 03:21:36 +02:00
afe55bba33
Add API for static strings, primarily good for identifiers.
...
Thanks to Konrad Schöbel and Jasper Schulz for helping with the mass-editing.
2011-10-09 10:38:36 +02:00
eaf139b3fc
Fix typo in the PyUnicode_Find() implementation
2011-10-09 00:33:09 +02:00
c47adb04b3
Change PyUnicode_KIND to 1,2,4. Drop _KIND_SIZE and _CHARACTER_SIZE.
2011-10-07 20:55:35 +02:00
dd07732af5
PyUnicode_Join() calls directly memcpy() if all strings are of the same kind
2011-10-07 17:02:31 +02:00
978b9d2a27
Fix formatting memory consumption with very large padding specifications
2011-10-07 12:35:48 +02:00
59de0ee9e0
str.replace(a, a) is now returning str unchanged if a is a
2011-10-07 10:01:28 +02:00
5c0ba36d5f
Fix massive slowdown in string formatting with the % operator
2011-10-07 01:54:09 +02:00
7c46da7993
Ensure that 1-char singletons get used
2011-10-06 22:07:51 +02:00
c6f0df7b20
Fix PyUnicode_Join() for len==1 and non-exact string
2011-10-06 15:58:54 +02:00
15a66cf134
Fix compilation under Windows
2011-10-06 15:25:32 +02:00
200f21340d
Fix assertion in unicode_adjust_maxchar()
2011-10-06 13:27:56 +02:00
acf47b807f
Fix my last change on PyUnicode_Join(): don't process separator if len==1
2011-10-06 12:32:37 +02:00
25a4b29c95
str.replace() avoids memory when it's possible
2011-10-06 12:31:55 +02:00
56c161ab00
_copy_characters() fails more quickly in debug mode on inconsistent state
2011-10-06 02:47:11 +02:00
c729b8e92f
Fix a compiler warning: don't define unicode_is_singleton() in release mode
2011-10-06 02:36:59 +02:00
fb9ea8c57e
Don't check for the maximum character when copying from unicodeobject.c
...
* Create copy_characters() function which doesn't check for the maximum
character in release mode
* _PyUnicode_CheckConsistency() is no more static to be able to use it
in _PyUnicode_FormatAdvanced() (in formatter_unicode.c)
* _PyUnicode_CheckConsistency() checks the string hash
2011-10-06 01:45:57 +02:00
05d1189566
Fix post-condition in unicode_repr(): check the result, not the input
2011-10-06 01:13:58 +02:00
f48323e3b3
replace() uses unicode_fromascii() if the input and replace string is ASCII
2011-10-05 23:27:08 +02:00
0617b6e18b
unicode_fromascii() checks that the input is ASCII in debug mode
2011-10-05 23:26:01 +02:00
c3cec7868b
Add asciilib: similar to ucs1, ucs2 and ucs4 library, but specialized to ASCII
...
ucs1, ucs2 and ucs4 libraries have to scan created substring to find the
maximum character, whereas it is not need to ASCII strings. Because ASCII
strings are common, it is useful to optimize ASCII.
2011-10-05 21:24:08 +02:00
14f8f02826
Fix PyUnicode_Partition(): str_in->str_obj
2011-10-05 20:58:25 +02:00
bb10a1f759
Ensure that newly created strings use the most efficient store in debug mode
2011-10-05 01:34:17 +02:00
9310abbf40
Replace PyUnicodeObject* with PyObject* where it was inappropriate
2011-10-05 00:59:23 +02:00
ce5faf673e
unicodeobject.c doesn't make output strings ready in debug mode
...
Try to only create non ready strings in debug mode to ensure that all functions
(not only in unicodeobject.c, everywhere) make input strings ready.
2011-10-05 00:42:43 +02:00
7597addbd4
More typoes.
2011-10-05 16:36:47 +02:00
c80d6d20d5
Speedup str[a 🅱️ step] for step != 1
...
Try to stop the scanner of the maximum character before the end using a limit
depending on the kind (e.g. 256 for PyUnicode_2BYTE_KIND).
2011-10-05 14:13:28 +02:00
ae86485517
Speedup find_maxchar_surrogates() for 32-bit wchar_t
...
If we have at least one character in U+10000-U+10FFFF, we know that we must use
PyUnicode_4BYTE_KIND kind.
2011-10-05 14:02:44 +02:00
b9275c104e
Speedup str[a:b] and PyUnicode_FromKindAndData
...
* str[a:b] doesn't scan the string for the maximum character if the string
is ascii only
* PyUnicode_FromKindAndData() stops if we are sure that we cannot use a
shorter character type. For example, _PyUnicode_FromUCS1() stops if we
have at least one character in range U+0080-U+00FF
2011-10-05 14:01:42 +02:00
702c734395
Speedup the ASCII decoder
...
It is faster for long string and a little bit faster for short strings,
benchmark on Linux 32 bits, Intel Core i5 @ 3.33GHz:
./python -m timeit 'x=b"a"' 'x.decode("ascii")'
./python -m timeit 'x=b"x"*80' 'x.decode("ascii")'
./python -m timeit 'x=b"abc"*4096' 'x.decode("ascii")'
length | before | after
-------+------------+-----------
1 | 0.234 usec | 0.229 usec
80 | 0.381 usec | 0.357 usec
12,288 | 11.2 usec | 3.01 usec
2011-10-05 13:50:52 +02:00
e1335c711c
Fix usage og PyUnicode_READY()
2011-10-04 20:53:03 +02:00
e06e145943
_PyUnicode_READY_REPLACE() cannot be used in unicode_subtype_new()
2011-10-04 20:52:31 +02:00
17efeed284
Add DONT_MAKE_RESULT_READY to unicodeobject.c to help detecting bugs
...
Use also _PyUnicode_READY_REPLACE() when it's applicable.
2011-10-04 20:05:46 +02:00
6b56a7fd3d
Add assertion to _Py_ReleaseInternedUnicodeStrings() if READY fails
2011-10-04 20:04:52 +02:00
875f29bb95
Fix naïve heuristic in unicode slicing (followup to 1b4f886dc9e2)
2011-10-04 20:00:49 +02:00
2242522fde
Add a necessary call to PyUnicode_READY() (followup to ab5086539ab9)
2011-10-04 19:10:51 +02:00
7aec401966
Optimize string slicing to use the new API
2011-10-04 19:08:01 +02:00
e19aa388e8
When expandtabs() would be a no-op, don't create a duplicate string
2011-10-04 16:04:01 +02:00
e71d574a39
Migrate str.expandtabs to the new API
2011-10-04 15:55:09 +02:00
7f3140ef80
fix parens
2011-10-03 19:37:29 -04:00
4bfce8f81f
fix formatting
2011-10-03 19:35:07 -04:00
ccc51c1fc6
fix compiler warnings
2011-10-03 19:34:12 -04:00
b092365cc6
Move in-place Unicode append to its own subfunction
2011-10-04 01:17:31 +02:00
a5f9163501
Reindent internal Unicode macros
2011-10-04 01:07:11 +02:00
a41463c203
Document utf8_length and wstr_length states
...
Ensure these states with assertions in _PyUnicode_CheckConsistency().
2011-10-04 01:05:08 +02:00
9566311014
resize_inplace() sets utf8_length to zero if the utf8 is not shared8
...
Cleanup also the code.
2011-10-04 01:03:50 +02:00
9e9d689d85
PyUnicode_New() sets utf8_length to zero for latin1
2011-10-04 01:02:02 +02:00
016980454e
Unicode: raise SystemError instead of ValueError or RuntimeError on invalid
...
state
2011-10-04 00:04:26 +02:00
7f11ad4594
Unicode: document when the wstr pointer is shared with data
...
Add also related assertions to _PyUnicode_CheckConsistency().
2011-10-04 00:00:20 +02:00
03490918b7
Add _PyUnicode_HAS_WSTR_MEMORY() macro
2011-10-03 23:45:12 +02:00
9ce5a835bb
PyUnicode_Join() checks output length in debug mode
...
PyUnicode_CopyCharacters() may copies less character than requested size, if
the input string is smaller than the argument. (This is very unlikely, but who
knows!?)
Avoid also calling PyUnicode_CopyCharacters() if the string is empty.
2011-10-03 23:36:02 +02:00
b803895355
Fix a compiler warning in PyUnicode_Append()
...
Don't check PyUnicode_CopyCharacters() in release mode. Rename also some
variables.
2011-10-03 23:27:56 +02:00
8cfcbed4e3
Improve string forms and PyUnicode_Resize() documentation
...
Remove also the FIXME for resize_copy(): as discussed with Martin, copy the
string on resize if the string is not resizable is just fine.
2011-10-03 23:19:21 +02:00
77bb47b312
Simplify unicode_resizable(): singletons reference count is at least 2
2011-10-03 20:06:05 +02:00
85041a54bd
_PyUnicode_CheckConsistency() checks utf8 field consistency
2011-10-03 14:42:39 +02:00
3cf4637e4e
unicode_subtype_new() copies also the ascii flag
2011-10-03 14:42:15 +02:00
42dfd71333
unicode_kind_name() doesn't check consistency anymore
...
It is is called from _PyUnicode_Dump() and so must not fail.
2011-10-03 14:41:45 +02:00
a3b334da6d
PyUnicode_Ready() now sets ascii=1 if maxchar < 128
...
ascii=1 is no more reserved to PyASCIIObject. Use
PyUnicode_IS_COMPACT_ASCII(obj) to check if obj is a PyASCIIObject (as before).
2011-10-03 13:53:37 +02:00
1b4f9ceca7
Create _PyUnicode_READY_REPLACE() to reuse singleton
...
Only use _PyUnicode_READY_REPLACE() on just created strings.
2011-10-03 13:28:14 +02:00
c379ead9af
Fix resize_compact() and resize_inplace(); reenable full resize optimizations
...
* resize_compact() updates also wstr_len for non-ascii strings sharing wstr
* resize_inplace() updates also utf8_len/wstr_len for strings sharing
utf8/wstr
2011-10-03 12:52:27 +02:00
34411e17b0
resize_inplace() has been fixed: reenable this optimization
2011-10-03 12:21:33 +02:00
a849a4b6b4
_PyUnicode_Dump() indicates if wstr and/or utf8 are shared
2011-10-03 12:12:11 +02:00
1c8d0c76a1
Fix resize_inplace(): update shared utf8 pointer
2011-10-03 12:11:00 +02:00
ca4f7a4298
Disable unicode_resize() optimization on Windows (16-bit wchar_t)
2011-10-03 04:18:04 +02:00
126c559d05
_PyUnicode_Ready() for 16-bit wchar_t
2011-10-03 04:17:10 +02:00
2fd82278cb
Fix compilation error on Windows
...
Fix also a compiler warning.
2011-10-03 04:06:05 +02:00
a3be613a56
Use PyUnicode_WCHAR_KIND to check if a string is a wstr string
...
Simplify the test in wstr pointer in unicode_sizeof().
2011-10-03 02:16:37 +02:00
910337b42e
Add _PyUnicode_CheckConsistency() macro to help debugging
...
* Document Unicode string states
* Use _PyUnicode_CheckConsistency() to ensure that objects are always
consistent.
2011-10-03 03:20:16 +02:00
4fae54cb0e
In release mode, PyUnicode_InternInPlace() does nothing if the input is NULL or
...
not a unicode, instead of failing with a fatal error.
Use assertions in debug mode (provide better error messages).
2011-10-03 02:01:52 +02:00
23e5668214
PyUnicode_Append() now works in-place when it's possible
2011-10-03 03:54:37 +02:00
fe226c0d37
Rewrite PyUnicode_Resize()
...
* Rename _PyUnicode_Resize() to unicode_resize()
* unicode_resize() creates a copy if the string cannot be resized instead
of failing
* Optimize resize_copy() for wstr strings
* Disable temporary resize_inplace()
2011-10-03 03:52:20 +02:00
829c0adca9
Add _PyUnicode_HAS_UTF8_MEMORY() macro
2011-10-03 01:08:02 +02:00
fe0c155c4f
Write _PyUnicode_Dump() to help debugging
2011-10-03 02:59:31 +02:00
f42dc448e0
PyUnicode_CopyCharacters() fails when copying latin1 into ascii
2011-10-02 23:33:16 +02:00
c53be96c54
unicode_convert_wchar_to_ucs4() cannot fail
2011-10-02 21:33:54 +02:00
c3c7415639
Add _PyUnicode_DATA_ANY(op) private macro
2011-10-02 20:39:55 +02:00
a464fc141d
unicode_empty and unicode_latin1 are PyObject* objects, not PyUnicodeObject*
2011-10-02 20:39:30 +02:00
267aa24365
PyUnicode_FindChar() raises a IndexError on invalid index
2011-10-02 01:08:37 +02:00
bc603d12b7
Optimize _PyUnicode_AsKind() for UCS1->UCS4 and UCS2->UCS4
...
* Ensure that the input string is ready
* Raise a ValueError instead of of a fatal error
2011-10-02 01:00:40 +02:00
5a706cf8c0
Fix usage of PyUnicode_READY() in PyUnicode_GetLength()
2011-10-02 00:36:53 +02:00
cd9950fd09
PyUnicode_WriteChar() raises IndexError on invalid index
...
PyUnicode_WriteChar() raises also a ValueError if the string has more than 1
reference.
2011-10-02 00:34:53 +02:00
2fe5ced752
PyUnicode_ReadChar() raises a IndexError if the index in invalid
...
unicode_getitem() reuses PyUnicode_ReadChar()
2011-10-02 00:25:40 +02:00
202b62bd90
PyUnicode_FromKindAndData() raises a ValueError if the kind is unknown
2011-10-01 23:48:37 +02:00
07ac3ebd7b
Optimize unicode_subtype_new(): don't encode to wchar_t and decode from wchar_t
...
Rewrite unicode_subtype_new(): allocate directly the right type.
2011-10-01 16:16:43 +02:00
e90fe6a8f4
Add _PyUnicode_UTF8() and _PyUnicode_UTF8_LENGTH() macros
...
* Rename existing _PyUnicode_UTF8() macro to PyUnicode_UTF8()
* Rename existing _PyUnicode_UTF8_LENGTH() macro to PyUnicode_UTF8_LENGTH()
* PyUnicode_UTF8() and PyUnicode_UTF8_LENGTH() are more strict
2011-10-01 16:48:13 +02:00
0b1d348990
Issue 13085: Fix some memory leaks. Patch by Stefan Krah.
2011-10-01 16:35:40 +02:00
5c0fb00ad8
merge heads
2011-10-01 00:12:20 -04:00
31616ea2ff
remove reference to non-existent file
2011-10-01 00:11:09 -04:00
de636f3c34
PyUnicode_Substring() now accepts end bigger than string length
...
Fix also a bug: call PyUnicode_READY() before reading string length.
2011-10-01 03:55:54 +02:00
c759f3e7ec
Ooops, avoid a division by zero in unicode_repeat()
2011-10-01 03:09:58 +02:00
d3a83d5eb3
PyUnicode_FromObject() ensures that its output is a ready string
2011-10-01 03:09:33 +02:00
67ca64ce54
I want a super fast 'a' * n!
...
* Optimize unicode_repeat() for a special case with memset()
* Simplify integer overflow checking; remove the second check because
PyUnicode_New() already does it and uses a smaller limit (Py_ssize_t vs
size_t)
2011-10-01 02:47:29 +02:00
e9a2935c1f
Fix usage of PyUnicode_READY in unicodeobject.c
2011-10-01 02:14:59 +02:00
12bab6dace
Remove private substring() function, reuse public PyUnicode_Substring()
...
* PyUnicode_Substring() now fails if start or end is invalid
* PyUnicode_Substring() reuses PyUnicode_Copy() for non-exact strings
2011-10-01 01:53:49 +02:00
c841e7db1f
Optimize PyUnicode_Copy(): don't recompute maximum character
2011-10-01 01:34:32 +02:00
2219e0a37e
PyUnicode_FromObject() reuses PyUnicode_Copy()
...
* PyUnicode_Copy() is faster than substring()
* Fix also a compiler warning
2011-10-01 01:16:59 +02:00
034f6cf10c
Add PyUnicode_Copy() function, include it to the public API
2011-09-30 02:26:44 +02:00
b153615008
PyUnicode_CopyCharacters() uses exceptions instead of assertions
...
Call PyErr_BadInternalCall() if inputs are not unicode strings.
2011-09-30 02:26:10 +02:00
d8f6510acc
_PyUnicode_Ready() cannot be used on ready strings anymore
...
* Change its prototype: PyObject* instead of PyUnicodeoObject*.
* Remove an old assertion, the result of PyUnicode_READY (_PyUnicode_Ready)
must be checked instead
2011-09-29 19:43:17 +02:00
bc8b81bc4e
Move _PyUnicode_UTF8() and _PyUnicode_UTF8_LENGTH() outside unicodeobject.h
...
Move these macros to unicodeobject.c
2011-09-29 19:31:34 +02:00
a0702ab1fe
Add a note in PyUnicode_CopyCharacters() doc: it doesn't write null character
...
Cleanup also the code (avoid the goto).
2011-09-29 14:14:38 +02:00
639418812f
Use the new Py_ARRAY_LENGTH macro
2011-09-29 00:42:28 +02:00
b9dcffb51e
Fix 'c' format of PyUnicode_Format()
...
formatbuf is now an array of Py_UCS4, not of Py_UNICODE
2011-09-29 00:39:24 +02:00
c17f540b7a
Oops, fix my previous commit: unicode => to
2011-09-29 00:16:58 +02:00
b15d4d899c
PyUnicode_CopyCharacters() marks the string as dirty (reset the hash)
2011-09-28 23:59:20 +02:00
f5ca1a21a5
PyUnicode_CopyCharacters() fails if 'to' has more than 1 reference
2011-09-28 23:54:59 +02:00
2aa2b3b4d5
Clean up a few tabs that went in with PEP393.
2011-09-29 00:58:57 +03:00
48a2f8fd97
#13054 : sys.maxunicode is now always 0x10FFFF.
2011-09-29 00:18:19 +03:00
506f592769
Check size of wchar_t using the preprocessor
2011-09-28 22:34:18 +02:00
73f01c65c8
PyUnicode_CopyCharacters() initializes overflow
2011-09-28 22:28:04 +02:00
e57b1c0da1
Mark PyUnicode_FromUCS[124] as private
2011-09-28 22:20:48 +02:00
ff9e50fd04
Oops, fix Py_MIN/Py_MAX case
2011-09-28 22:17:19 +02:00
17222160e7
Mark _PyUnicode_FindMaxCharAndNumSurrogatePairs() as private
2011-09-28 22:15:37 +02:00
157f83fcfc
Strip trailing spaces in unicodeobject.[ch]
2011-09-28 21:41:31 +02:00
6c7a52a46f
Check for PyUnicode_CopyCharacters() failure
2011-09-28 21:39:17 +02:00
be78eaf2de
PyUnicode_CopyCharacters() checks for buffer and character overflow
...
It now returns the number of written characters on success.
2011-09-28 21:37:03 +02:00
fb5f5f2420
Mark PyUnicode_CONVERT_BYTES as private
2011-09-28 21:39:49 +02:00
4cb0de246c
Rename new macros to conform to naming rules (function macros have "Py" prefix, not "PY").
2011-09-28 21:49:49 +02:00
9c6e6a0c7f
don't check that the first character is XID_Continue
...
Current, XID_Continue is a superset of XID_Start, but that may sometime change.
2011-09-28 08:09:05 -04:00
d63a3b8beb
Implement PEP 393.
2011-09-28 07:41:54 +02:00
57e683e53e
Issue #1621 : Fix undefined behaviour in bytes.__hash__, str.__hash__, tuple.__hash__, frozenset.__hash__ and set indexing operations.
2011-09-24 18:18:40 +01:00
0d5f6adbb3
Issue #13012 : Allow 'keepends' to be passed as a keyword argument in str.splitlines, bytes.splitlines and bytearray.splitlines.
2011-09-24 09:14:39 +01:00
f955eb210f
Merge 3.2: Fix PyUnicode_AsWideCharString() doc
...
- Fix PyUnicode_AsWideCharString() doc: size doesn't contain the null
character
- Fix spelling of the null character
2011-09-06 02:01:29 +02:00
d88d9836c5
Fix PyUnicode_AsWideCharString() doc: size doesn't contain the null character
...
Fix also spelling of the null character.
2011-09-06 02:00:05 +02:00
6f2a683a0c
#9200 : merge with 3.2.
2011-08-22 20:31:11 +03:00
93e7afc5d9
#9200 : The str.is* methods now work with strings that contain non-BMP characters even in narrow Unicode builds.
2011-08-22 14:08:38 +03:00
e518d4c18a
merge 3.2
2011-08-18 13:52:19 -05:00
7a6b44ab62
the named of the character is actually NUL
2011-08-18 13:51:47 -05:00
020340f284
merge 3.2
2011-08-18 10:49:16 -05:00
5ad517a7d9
NUL -> NULL
2011-08-18 10:48:50 -05:00
269e3ee3db
#12266 : merge with 3.2.
2011-08-15 09:26:28 +03:00
ee8d998ecf
#12266 : Fix str.capitalize() to correctly uppercase/lowercase titlecased and cased non-letter characters.
2011-08-15 09:09:57 +03:00
f8e7543df9
merge 3.2 ( #12732 )
2011-08-12 22:18:19 -05:00
f413b80806
in narrow builds, make sure to test codepoints as identifier characters ( closes #12732 )
...
This fixes the use of Unicode identifiers outside the BMP in narrow builds.
2011-08-12 22:17:18 -05:00
dfc80e3d97
Replace Py_NotImplemented returns with the macro form Py_RETURN_NOTIMPLEMENTED.
...
The macro was introduced in #12724 .
2011-08-10 20:28:54 -05:00
fcdaaa9011
merge from 3.2 - Fix closes Issue12621 - Fix docstrings of find and rfind methods of bytes/bytearry/unicodeobject.
2011-07-27 23:34:29 +08:00
53516a82df
Fix closes Issue12621 - Fix docstrings of find and rfind methods of bytes/bytearry/unicodeobject.
2011-07-27 23:33:54 +08:00
99b9538636
Issue #9642 : Uniformize the tests on the availability of the mbcs codec
...
Add a new HAVE_MBCS define.
2011-07-04 14:23:54 +02:00
bc9d8f838b
merge from 3.2
2011-07-03 21:05:25 -07:00
9ebe08d2f6
Fix closes issue12471 - wrong TypeError message when '%i' format spec was used.
2011-07-03 21:03:16 -07:00
3cbf14bfb1
Issue #10914 : Initialize correctly the filesystem codec when creating a new
...
subinterpreter to fix a bootstrap issue with codecs implemented in Python, as
the ISO-8859-15 codec.
Add fscodec_initialized attribute to the PyInterpreterState structure.
2011-04-27 00:24:21 +02:00
793b531756
Issue #10914 : Initialize correctly the filesystem codec when creating a new
...
subinterpreter to fix a bootstrap issue with codecs implemented in Python, as
the ISO-8859-15 codec.
Add fscodec_initialized attribute to the PyInterpreterState structure.
2011-04-27 00:24:21 +02:00
bf1253b25a
#6780 : merge with 3.2.
2011-04-26 06:45:24 +03:00
f2b3f780a1
#6780 : merge with 3.1.
2011-04-26 06:40:59 +03:00
ba42fd5801
#6780 : fix starts/endswith error message to mention that tuples are accepted too.
2011-04-26 06:09:45 +03:00
c1ceb64e41
MERGE: startswith and endswith don't accept None as slice index. Patch by Torsten Becker. ( closes #11828 )
2011-04-20 17:59:29 +02:00
Victor Stinner
Antoine Pitrou
Martin v. Löwis
Victor Stinner
Georg Brandl
Benjamin Peterson
Ezio Melotti
Mark Dickinson
Brian Curtin
Senthil Kumaran
Jesus Cea