Without this change, test_unicode.UnicodeTest.test_codecs_utf7 crashes in
debug mode. What happens is the unicode string u'\U000abcde' with a length
of 1 encodes to the string '+2m/c3g-' of length 8. Since only 5 bytes is
reserved in the buffer, a buffer overrun occurs.
svn+ssh://pythondev@svn.python.org/python/trunk
........
r72260 | walter.doerwald | 2009-05-04 00:36:33 +0200 (Mo, 04 Mai 2009) | 5 lines
Issue #5108: Handle %s like %S and %R in PyUnicode_FromFormatV(): Call
PyUnicode_DecodeUTF8() once, remember the result and output it in a second
step. This avoids problems with counting UTF-8 bytes that ignores the effect
of using the replace error handler in PyUnicode_DecodeUTF8().
........
PyUnicode_DecodeUTF8() once, remember the result and output it in a second
step. This avoids problems with counting UTF-8 bytes that ignores the effect
of using the replace error handler in PyUnicode_DecodeUTF8().
If anyone wants to clean up the documentation, feel free. It's my first documentation foray, and it's not that great.
Will port to py3k with a different strategy.
svn+ssh://pythondev@svn.python.org/python/trunk
........
r70499 | hirokazu.yamamoto | 2009-03-21 19:32:52 +0900 | 1 line
There is no macro named SIZEOF_SSIZE_T. Should use SIZEOF_SIZE_T instead.
........
sizeof(Py_UNICODE) == 2, PyUnicode_FromWideChar now converts
each character outside the BMP to the appropriate surrogate pair.
Thanks Victor Stinner for the patch.
(backport of r70452 from py3k to trunk)
svn+ssh://pythondev@svn.python.org/python/trunk
........
r67898 | benjamin.peterson | 2008-12-21 15:00:53 -0600 (Sun, 21 Dec 2008) | 1 line
compute DISTVERSION with patchlevel.py
........
r67904 | benjamin.peterson | 2008-12-22 14:44:58 -0600 (Mon, 22 Dec 2008) | 1 line
less attitude
........
r67905 | benjamin.peterson | 2008-12-22 14:51:15 -0600 (Mon, 22 Dec 2008) | 1 line
fix#4720: the format to PyArg_ParseTupleAndKeywords can now start with '|'
........
r67906 | benjamin.peterson | 2008-12-22 14:52:53 -0600 (Mon, 22 Dec 2008) | 1 line
add NEWS note
........
r67907 | benjamin.peterson | 2008-12-22 16:12:19 -0600 (Mon, 22 Dec 2008) | 1 line
silence compiler warning
........
r67912 | georg.brandl | 2008-12-23 06:37:21 -0600 (Tue, 23 Dec 2008) | 2 lines
Fix missing "svn" command.
........
r67918 | georg.brandl | 2008-12-23 09:44:25 -0600 (Tue, 23 Dec 2008) | 2 lines
Markup fix.
........
r67920 | benjamin.peterson | 2008-12-23 14:09:28 -0600 (Tue, 23 Dec 2008) | 1 line
use a global variable, so the compiler doesn't optimize the assignment out
........
r67921 | benjamin.peterson | 2008-12-23 14:12:33 -0600 (Tue, 23 Dec 2008) | 1 line
make global static
........
r67923 | benjamin.peterson | 2008-12-24 09:10:27 -0600 (Wed, 24 Dec 2008) | 1 line
#4736 BufferRWPair.closed shouldn't try to call another property as a function
........
r67924 | benjamin.peterson | 2008-12-24 10:10:05 -0600 (Wed, 24 Dec 2008) | 1 line
pretend exceptions don't exist a while longer
........
r67927 | benjamin.peterson | 2008-12-26 17:26:30 -0600 (Fri, 26 Dec 2008) | 1 line
python version is included in file name now
........
r67930 | hirokazu.yamamoto | 2008-12-26 22:19:48 -0600 (Fri, 26 Dec 2008) | 2 lines
Issue #4740: Use HIGHEST_PROTOCOL in pickle test.
(There is no behavior difference in 2.x because HIGHEST_PROTOCOL == 2)
........
r67932 | alexandre.vassalotti | 2008-12-27 00:36:10 -0600 (Sat, 27 Dec 2008) | 5 lines
Remove unnecessary casts related to unicode_decode_call_errorhandler.
Make the _PyUnicode_Resize macro a static function.
These changes are needed to avoid breaking strict aliasing rules.
........
r67943 | alexandre.vassalotti | 2008-12-27 04:02:59 -0600 (Sat, 27 Dec 2008) | 2 lines
Fix bogus unicode tests in pickletester.
........
svn+ssh://pythondev@svn.python.org/python/trunk
........
r67398 | benjamin.peterson | 2008-11-26 18:39:17 +0100 (Wed, 26 Nov 2008) | 1 line
fix typo in sqlite3 docs
........
r67423 | jesse.noller | 2008-11-28 19:59:35 +0100 (Fri, 28 Nov 2008) | 2 lines
issue4238: bsd support for cpu_count
........
r67424 | christian.heimes | 2008-11-28 20:33:33 +0100 (Fri, 28 Nov 2008) | 1 line
Retain copyright of processing examples. This was requested by a Debian maintainer during packaging of the multiprocessing package for 2.4/2.5
........
r67432 | benjamin.peterson | 2008-11-29 00:18:46 +0100 (Sat, 29 Nov 2008) | 1 line
SVN format 9 is the same it seems
........
r67440 | jeremy.hylton | 2008-11-29 00:42:59 +0100 (Sat, 29 Nov 2008) | 4 lines
Move definition int sval into branch of ifdef where it is used.
Otherwise, you get a warning about an undefined variable.
........
r67441 | jeremy.hylton | 2008-11-29 01:09:16 +0100 (Sat, 29 Nov 2008) | 2 lines
Reflow long lines.
........
r67444 | amaury.forgeotdarc | 2008-11-29 03:03:32 +0100 (Sat, 29 Nov 2008) | 2 lines
Fix a small typo in docstring
........
r67445 | benjamin.peterson | 2008-11-30 04:07:33 +0100 (Sun, 30 Nov 2008) | 1 line
StringIO.close() stops you from using the buffer, too
........
r67454 | benjamin.peterson | 2008-11-30 15:43:23 +0100 (Sun, 30 Nov 2008) | 1 line
note the version that works
........
r67457 | christian.heimes | 2008-11-30 22:16:28 +0100 (Sun, 30 Nov 2008) | 1 line
w# requires Py_ssize_t
........
r67463 | skip.montanaro | 2008-12-01 02:55:22 +0100 (Mon, 01 Dec 2008) | 1 line
typo in comment
........
svn+ssh://pythondev@svn.python.org/python/trunk
........
r66748 | christian.heimes | 2008-10-02 21:47:50 +0200 (Thu, 02 Oct 2008) | 1 line
Fixed a couple more C99 comments and one occurence of inline.
........
the freelist contained half-initialized objects with freed pointers.
The comment
/* XXX UNREF/NEWREF interface should be more symmetrical */
was copied from tupleobject.c, and appears in some other places.
I sign the petition.
renamed Include/bytesobject.h to Include/bytearrayobject.h
renamed Include/stringobject.h to Include/bytesobject.h
added Include/stringobject.h with aliases
Rather than sprinkle casts throughout the code, change Py_CHARMASK to
always cast it's result to an unsigned char. This should ensure we
do the right thing when accessing an array with the result.
svn+ssh://pythondev@svn.python.org/python/branches/trunk-bytearray
........
r61750 | christian.heimes | 2008-03-22 20:47:44 +0100 (Sat, 22 Mar 2008) | 1 line
Copied files from py3k w/o modifications
........
r61752 | christian.heimes | 2008-03-22 20:53:20 +0100 (Sat, 22 Mar 2008) | 7 lines
Take One
* Added initialization code, warnings, flags etc. to the appropriate places
* Added new buffer interface to string type
* Modified tests
* Modified Makefile.pre.in to compile the new files
* Added bytesobject.c to Python.h
........
r61754 | christian.heimes | 2008-03-22 21:22:19 +0100 (Sat, 22 Mar 2008) | 2 lines
Disabled bytearray.extend for now since it causes an infinite recursion
Fixed serveral unit tests
........
r61756 | christian.heimes | 2008-03-22 21:43:38 +0100 (Sat, 22 Mar 2008) | 5 lines
Added PyBytes support to several places:
str + bytearray
ord(bytearray)
bytearray(str, encoding)
........
r61760 | christian.heimes | 2008-03-22 21:56:32 +0100 (Sat, 22 Mar 2008) | 1 line
Fixed more unit tests related to type('') is not unicode
........
r61763 | christian.heimes | 2008-03-22 22:20:28 +0100 (Sat, 22 Mar 2008) | 2 lines
Fixed more unit tests
Fixed bytearray.extend
........
r61768 | christian.heimes | 2008-03-22 22:40:50 +0100 (Sat, 22 Mar 2008) | 1 line
Implemented old buffer interface for bytearray
........
r61772 | christian.heimes | 2008-03-22 23:24:52 +0100 (Sat, 22 Mar 2008) | 1 line
Added backport of the io module
........
r61775 | christian.heimes | 2008-03-23 03:50:49 +0100 (Sun, 23 Mar 2008) | 1 line
Fix str assignement to bytearray. Assignment of a str of size 1 is interpreted as a single byte
........
r61805 | christian.heimes | 2008-03-23 19:33:48 +0100 (Sun, 23 Mar 2008) | 3 lines
Fixed more tests
Fixed bytearray() comparsion with unicode()
Fixed iterator assignment of bytearray
........
r61809 | christian.heimes | 2008-03-23 21:02:21 +0100 (Sun, 23 Mar 2008) | 2 lines
str(bytesarray()) now returns the bytes and not the representation of the bytearray object
Enabled and fixed more unit tests
........
r61812 | christian.heimes | 2008-03-23 21:53:08 +0100 (Sun, 23 Mar 2008) | 3 lines
Clear error PyNumber_AsSsize_t() fails
Use CHARMASK for ob_svall access
disabled a test with memoryview again
........
r61819 | christian.heimes | 2008-03-23 23:05:57 +0100 (Sun, 23 Mar 2008) | 1 line
Untested updates to the PCBuild directory
........
r61917 | christian.heimes | 2008-03-26 00:57:06 +0100 (Wed, 26 Mar 2008) | 1 line
The type system of Python 2.6 has subtle differences to 3.0's. I've removed the Py_TPFLAGS_BASETYPE flags from bytearray for now. bytearray can't be subclasses until the issues with bytearray subclasses are fixed.
........
r61920 | christian.heimes | 2008-03-26 01:44:08 +0100 (Wed, 26 Mar 2008) | 2 lines
Disabled last failing test
I don't understand what the test is testing and how it suppose to work. Ka-Ping, please check it out.
........
r61930 | christian.heimes | 2008-03-26 12:46:18 +0100 (Wed, 26 Mar 2008) | 1 line
Re-enabled bytes warning code
........
r61933 | christian.heimes | 2008-03-26 13:20:46 +0100 (Wed, 26 Mar 2008) | 1 line
Fixed a bug in the new buffer protocol. The buffer slots weren't copied into a subclass.
........
r61934 | christian.heimes | 2008-03-26 13:25:09 +0100 (Wed, 26 Mar 2008) | 1 line
Re-enabled bytearray subclassing - all tests are passing.
........
Highlights:
- Adding PyObject_Format.
- Adding string.Format class.
- Adding __format__ for str, unicode, int, long, float, datetime.
- Adding builtin format.
- Adding ''.format and u''.format.
- str/unicode fixups for formatters.
The files in Objects/stringlib that implement PEP 3101 (stringdefs.h,
unicodedefs.h, formatter.h, string_format.h) are identical in trunk
and py3k. Any changes from here on should be made to trunk, and
changes will propogate to py3k).
in Object/ are named ``free_list``, the counter ``numfree`` and the upper
limit is a macro ``PyName_MAXFREELIST`` inside an #ifndef block.
The chances should make it easier to adjust Python for platforms with
less memory, e.g. mobile phones.
First chapter of the Python 3.0 io framework back port: _fileio
The next step depends on a working bytearray type which itself depends on a backport of the nwe buffer API.
as usual with slicing (both with str and unicode strings). This
fixes issue 1259.
For str only the stringobject.c file was modified. But for unicode,
I needed to repeat in the four functions a lot of code, so created
a new function that does part of the job for them (and placed it in
find.h, following a suggestion of Barry).
Also added tests for this behaviour.
also hex escapes) -- this was reaching beyond the end of the input string
buffer, even though it is not supposed to be \0-terminated.
This has no visible effect but is clearly the correct thing to do.
(In 3.0 it had a visible effect after removing ob_sstate from PyString.)
- Specialcase extended slices that amount to a shallow copy the same way as
is done for simple slices, in the tuple, string and unicode case.
- Specialcase step-1 extended slices to optimize the common case for all
involved types.
- For lists, allow extended slice assignment of differing lengths as long
as the step is 1. (Previously, 'l[:2:1] = []' failed even though
'l[:2] = []' and 'l[:2:None] = []' do not.)
- Implement extended slicing for buffer, array, structseq, mmap and
UserString.UserString.
- Implement slice-object support (but not non-step-1 slice assignment) for
UserString.MutableString.
- Add tests for all new functionality.
a large width is passed on 32-bit platforms. Found by Google.
It would be good for people to review this especially carefully and verify
I don't have an off by one error and there is no other way to cause overflow.
of some of the common builtin types.
Use a bit in tp_flags for each common builtin type. Check the bit
to determine if any instance is a subclass of these common types.
The check avoids a function call and O(n) search of the base classes.
The check is done in the various Py*_Check macros rather than calling
PyType_IsSubtype().
All the bits are set in tp_flags when the type is declared
in the Objects/*object.c files because PyType_Ready() is not called
for all the types. Should PyType_Ready() be called for all types?
If so and the change is made, the changes to the Objects/*object.c files
can be reverted (remove setting the tp_flags). Objects/typeobject.c
would also have to be modified to add conditions
for Py*_CheckExact() in addition to each the PyType_IsSubtype check.
* unified the way intobject, longobject and mystrtoul handle
values around -sys.maxint-1.
* in general, trying to entierely avoid overflows in any computation
involving signed ints or longs is extremely involved. Fixed a few
simple cases where a compiler might be too clever (but that's all
guesswork).
* more overflow checks against bad data in marshal.c.
* 2.5 specific: fixed a number of places that were still confusing int
and Py_ssize_t. Some of them could potentially have caused
"real-world" breakage.
* list.pop(x): fixing overflow issues on x was messy. I just reverted
to PyArg_ParseTuple("n"), which does the right thing. (An obscure
test was trying to give a Decimal to list.pop()... doesn't make
sense any more IMHO)
* trying to write a few tests...
Replace UnicodeDecodeErrors raised during == and !=
compares of Unicode and other objects with a new
UnicodeWarning.
All other comparisons continue to raise exceptions.
Exceptions other than UnicodeDecodeErrors are also left
untouched.
I modified this patch some by fixing style, some error checking, and adding
XXX comments. This patch requires review and some changes are to be expected.
I'm checking in now to get the greatest possible review and establish a
baseline for moving forward. I don't want this to hold up release if possible.
results in a 2.5x speedup on the stringbench count tests, and a 20x (!)
speedup on the stringbench search/find/contains test, compared to 2.5a2.
for more on the algorithm, see:
http://effbot.org/zone/stringlib.htm
if you get weird results, you can disable the new algoritm by undefining
USE_FAST in Objects/unicodeobject.c.
enjoy /F
speed up splitlines and strip with charsets; etc. rsplit is now as
fast as split in all our tests (reverse takes no time at all), and
splitlines() is nearly as fast as a plain split("\n") in our tests.
and we're not done yet... ;-)
compiler warnings on Windows (signed vs unsigned mismatch
in comparisons). Cleaned that up by switching more locals
to Py_ssize_t. Simplified overflow checking (it can _be_
simpler because while these things are declared as
Py_ssize_t, then should in fact never be negative).
there)
- Add missing DECREFs of inner-scope 'temp' variable
- Add various missing DECREFs by changing 'return NULL' into 'goto onError'
- Avoid double DECREF when last _PyUnicode_Resize() fails
Coverity found one of the missing DECREFs, but oddly enough not the others.
Mark Dickinson
Alexandre Vassalotti
Jesus Cea
Georg Brandl
Antoine Pitrou
Walter Dörwald
Eric Smith
Hirokazu Yamamoto
Benjamin Peterson
Amaury Forgeot d'Arc
Andrew M. Kuchling
Christian Heimes
Neal Norwitz
Robert Schuppenies
Gregory P. Smith
Martin v. Löwis
Guido van Rossum
Facundo Batista
Thomas Wouters
Armin Rigo
Raymond Hettinger
Marc-André Lemburg
Fredrik Lundh
Andrew Dalke
Tim Peters
Anthony Baxter
Hye-Shik Chang