Commit Graph

326 Commits

Author SHA1 Message Date
Amaury Forgeot d'Arc afa0d58a2d Issue #3751: str.rpartition would perform a left-partition when called with
a unicode argument.

Backport of r66119
2008-09-01 20:05:08 +00:00
Neal Norwitz 4f3be8a0a9 Security patches from Apple: prevent int overflow when allocating memory 2008-07-31 17:08:14 +00:00
Gregory P. Smith a6add69a63 backport r62271 from trunk.
remove the now obsoleted assertion.
2008-04-10 19:54:13 +00:00
Gregory P. Smith 14acde30f6 Backport r62261 from trunk:
Prevent PyString_FromStringAndSize() from passing negative sizes on to lower
level memory allocation functions.  Raise a SystemError and return NULL
instead.
2008-04-09 23:41:13 +00:00
Guido van Rossum 44a93e54f4 Fix the overflows in expandtabs(). "This time for sure!"
(Exploit at request.)
2008-03-11 21:14:54 +00:00
Guido van Rossum e6a6f39cc7 Backport r58892.
Add missing "return NULL" in overflow check in PyString_Repr().
2007-11-07 01:19:49 +00:00
Georg Brandl 1dcb9c93fd Backport r58709 from trunk:
Backport fixes for the code that decodes octal escapes (and for PyString
also hex escapes) -- this was reaching beyond the end of the input string
buffer, even though it is not supposed to be \0-terminated.
This has no visible effect but is clearly the correct thing to do.
(In 3.0 it had a visible effect after removing ob_sstate from PyString.)
Also fixes #1098.
2007-11-02 22:46:38 +00:00
Georg Brandl b4d100cf2d Bug #1763149: use proper slice syntax in docstring.
(backport)
2007-07-29 17:37:22 +00:00
Georg Brandl c5db923994 Patch #1673759: add a missing overflow check when formatting floats
with %G.
 (backport from rev. 56298)
2007-07-12 08:38:04 +00:00
Neal Norwitz 8355dd5806 Backport 55874:
Fix a bug when there was a newline in the string expandtabs was called on.
This also catches another condition that can overflow.
2007-06-11 04:32:41 +00:00
Neal Norwitz 66e64e2b6a Prevent expandtabs() on string and unicode objects from causing a segfault when
a large width is passed on 32-bit platforms.  Found by Google.

It would be good for people to review this especially carefully and verify
I don't have an off by one error and there is no other way to cause overflow.
2007-06-09 04:06:30 +00:00
Georg Brandl 5f795865da Backport from Py3k branch: fix refleak in PyString_Format.
(backport from rev. 53935)
2007-02-26 13:51:34 +00:00
Armin Rigo 4b63c21d6f Forward-port of r52136: a review of overflow-detecting code.
* unified the way intobject, longobject and mystrtoul handle
  values around -sys.maxint-1.

* in general, trying to entierely avoid overflows in any computation
  involving signed ints or longs is extremely involved.  Fixed a few
  simple cases where a compiler might be too clever (but that's all
  guesswork).

* more overflow checks against bad data in marshal.c.

* 2.5 specific: fixed a number of places that were still confusing int
  and Py_ssize_t.  Some of them could potentially have caused
  "real-world" breakage.

* list.pop(x): fixing overflow issues on x was messy.  I just reverted
  to PyArg_ParseTuple("n"), which does the right thing.  (An obscure
  test was trying to give a Decimal to list.pop()... doesn't make
  sense any more IMHO)

* trying to write a few tests...
2006-10-04 11:44:06 +00:00
Neal Norwitz 29a5fdb7ca Fix str.rpartition(sep) when sep is not found in str.
Partially from SF patch #1551339, but also taken from head.
2006-09-05 02:21:38 +00:00
Georg Brandl 26a07b5198 Fix refleak introduced in rev. 51248. 2006-08-14 20:25:39 +00:00
Neal Norwitz 56423e5762 Fix segfault when doing string formatting on subclasses of long if
__oct__, __hex__ don't return a string.

Klocwork 308
2006-08-13 18:11:08 +00:00
Neal Norwitz 8a87f5d37e Patch #1538606, Patch to fix __index__() clipping.
I modified this patch some by fixing style, some error checking, and adding
XXX comments.  This patch requires review and some changes are to be expected.
I'm checking in now to get the greatest possible review and establish a
baseline for moving forward.  I don't want this to hold up release if possible.
2006-08-12 17:03:09 +00:00
Neal Norwitz a7edb11122 Whitespace normalization 2006-07-30 06:59:13 +00:00
Neal Norwitz f71ec5a0ac Bug #1515471: string.replace() accepts character buffers again.
Pass the char* and size around rather than PyObject's.
2006-07-30 06:57:04 +00:00
Neal Norwitz 8e6675a7dc Update doc to make it agree with code.
Bottom factor out some common code.
2006-06-11 05:47:14 +00:00
Georg Brandl 90e27d38f5 Apply perky's fix for #1503157: "/".join([u"", u""]) raising OverflowError.
Also improve error message on overflow.
2006-06-10 06:40:50 +00:00
Georg Brandl 242508160e RFE #1491485: str/unicode.endswith()/startswith() now accept a tuple as first argument. 2006-06-09 18:45:48 +00:00
Neal Norwitz b16e4e7860 Remove ; at end of macro. There was a compiler recently that warned
about extra semi-colons.  It may have been the HP C compiler.
This file will trigger a bunch of those warnings now.
2006-06-01 05:32:49 +00:00
Fredrik Lundh 80f8e80c15 needforspeed: added Py_MEMCPY macro (currently tuned for Visual C only),
and use it for string copy operations.  this gives a 20% speedup on some
string benchmarks.
2006-05-28 12:06:46 +00:00
Fredrik Lundh 0b7ef46950 needforspeed: stringlib refactoring: use find_slice for stringobject 2006-05-27 15:26:19 +00:00
Fredrik Lundh c2d29c5a6d needforspeed: replace improvements, changed to Py_LOCAL_INLINE
where appropriate
2006-05-27 14:58:20 +00:00
Andrew Dalke d49d5c49ba cleanup - removed trailing whitespace 2006-05-27 14:16:40 +00:00
Fredrik Lundh 2d23d5bf2e needforspeed: more stringlib refactoring 2006-05-27 10:05:10 +00:00
Andrew Dalke 7e0a62ea90 Added description of why splitlines doesn't use the prealloc strategy 2006-05-26 22:49:03 +00:00
Andrew Dalke 5132407868 Added limits to the replace code so it does not count all of the matching
patterns in a string, only the number needed by the max limit.
2006-05-26 20:25:22 +00:00
Fredrik Lundh e6e43c867d needforspeed: stringlib refactoring: use stringlib/find for string find 2006-05-26 19:48:07 +00:00
Fredrik Lundh 58b5e84d52 needforspeed: stringlib refactoring, continued. added count and
find helpers; updated unicodeobject to use stringlib_count
2006-05-26 19:24:53 +00:00
Andrew Dalke c5da53ba78 substring split now uses /F's fast string matching algorithm.
(If compiled without FAST search support, changed the pre-memcmp test
   to check the last character as well as the first.  This gave a 25%
   speedup for my test case.)

Rewrote the split algorithms so they stop when maxsplit gets to 0.
Previously they did a string match first then checked if the maxsplit
was reached.  The new way prevents a needless string search.
2006-05-26 19:02:09 +00:00
Fredrik Lundh b3167cbcd7 needforspeed: added rpartition implementation 2006-05-26 18:15:38 +00:00
Fredrik Lundh 3a65d87e8c needforspeed: remove remaining USE_FAST macros; if fastsearch was
broken, someone would have noticed by now ;-)
2006-05-26 17:31:41 +00:00
Fredrik Lundh c2032fb86a needforspeed: cleanup 2006-05-26 17:26:39 +00:00
Fredrik Lundh b947948c61 needforspeed: stringlib refactoring (in progress) 2006-05-26 17:22:38 +00:00
Fredrik Lundh a50d201bd9 needforspeed: stringlib refactoring (in progress) 2006-05-26 17:04:58 +00:00
Fredrik Lundh 7c940d1d68 needforspeed: use Py_LOCAL on a few more locals in stringobject.c 2006-05-26 16:32:42 +00:00
Andrew Dalke 02758d66ce Eeked out another 3% or so performance in split whitespace by cleaning up the algorithm. 2006-05-26 15:21:01 +00:00
Andrew Dalke 525eab3712 Changes to string.split/rsplit on whitespace to preallocate space in the
results list.

Originally it allocated 0 items and used the list growth during append.  Now
it preallocates 12 items so the first few appends don't need list reallocs.

("Here are some words ."*2).split(None, 1) is 7% faster
("Here are some words ."*2).split() is is 15% faster

  (Your milage may vary, see dealership for details.)

File parsing like this

    for line in f:
        count += len(line.split())

is also about 15% faster.  There is a slowdown of about 3% for large
strings because of the additional overhead of checking if the append is
to a preallocated region of the list or not.  This will be the rare case.
It could be improved with special case code but we decided it was not
useful enough.

There is a cost of 12*sizeof(PyObject *) bytes per list.  For the normal
case of file parsing this is not a problem because of the lists have
a short lifetime.  We have not come up with cases where this is a problem
in real life.

I chose 12 because human text averages about 11 words per line in books,
one of my data sets averages 6.2 words with a final peak at 11 words per
line, and I work with a tab delimited data set with 8 tabs per line (or
9 words per line).  12 encompasses all of these.

Also changed the last rstrip code to append then reverse, rather than
doing insert(0).  The strip() and rstrip() times are now comparable.
2006-05-26 14:00:45 +00:00
Fredrik Lundh 95e2a91615 use Py_LOCAL also for string and unicode objects 2006-05-26 11:38:15 +00:00
Fredrik Lundh f2c0dfdb13 needforspeed: use Py_ssize_t for the fastsearch counter and skip
length (thanks, neal!).  and yes, I've verified that this doesn't
slow things down ;-)
2006-05-26 10:27:17 +00:00
Fredrik Lundh 450277fef5 needforspeed: use METH_O for argument handling, which made partition some
~15% faster for the current tests (which is noticable faster than a corre-
sponding find call).  thanks to neal-who-never-sleeps for the tip.
2006-05-26 09:46:59 +00:00
Fredrik Lundh 06a69dd8ff needforspeed: partition implementation, part two.
feel free to improve the documentation and the docstrings.
2006-05-26 08:54:28 +00:00
Fredrik Lundh fe5bb7e6d9 needforspeed: partition for 8-bit strings. for some simple tests,
this is on par with a corresponding find, and nearly twice as fast
as split(sep, 1)

full tests, a unicode version, and documentation will follow to-
morrow.
2006-05-25 23:27:53 +00:00
Bob Ippolito 955b64c031 squelch gcc4 darwin/x86 compiler warnings 2006-05-25 20:52:38 +00:00
Fredrik Lundh 554da412a8 needforspeed: use insert+reverse instead of append 2006-05-25 19:19:05 +00:00
Jack Diederich 60cbb3fe49 * eliminate warning by reverting tmp_s type to 'const char*' 2006-05-25 18:47:15 +00:00
Fredrik Lundh c3434b3834 needforspeed: use fastsearch also for find/index and contains. the
related tests are now about 10x faster.
2006-05-25 18:44:29 +00:00