Commit Graph

376 Commits

Author SHA1 Message Date
Amaury Forgeot d'Arc 5087980c1e The incremental decoder for utf-7 must preserve its state between calls.
Solves issue1460.

Might not be a backport candidate: a new API function was added,
and some code may rely on details in utf-7.py.
2007-11-20 23:31:27 +00:00
Facundo Batista 6f7e6fb7a2 Made _ParseTupleFinds only defined to unicodeobject.c 2007-11-16 19:16:15 +00:00
Facundo Batista 57d5669f4b Now in find, rfind, index, and rindex, you can use None as defaults,
as usual with slicing (both with str and unicode strings).  This
fixes issue 1259.

For str only the stringobject.c file was modified.  But for unicode,
I needed to repeat in the four functions a lot of code, so created
a new function that does part of the job for them (and placed it in
find.h, following a suggestion of Barry).

Also added tests for this behaviour.
2007-11-16 18:04:14 +00:00
Guido van Rossum 1c1ac38157 Backport fixes for the code that decodes octal escapes (and for PyString
also hex escapes) -- this was reaching beyond the end of the input string
buffer, even though it is not supposed to be \0-terminated.
This has no visible effect but is clearly the correct thing to do.
(In 3.0 it had a visible effect after removing ob_sstate from PyString.)
2007-10-29 22:15:05 +00:00
Walter Dörwald 9d04542cc9 Set startinpos before calling the error handler. 2007-08-30 15:34:55 +00:00
Walter Dörwald 8757878b12 Rewrap line. 2007-08-30 15:30:09 +00:00
Thomas Wouters 3ccec68a05 Improve extended slicing support in builtin types and classes. Specifically:
- Specialcase extended slices that amount to a shallow copy the same way as
   is done for simple slices, in the tuple, string and unicode case.

 - Specialcase step-1 extended slices to optimize the common case for all
   involved types.

 - For lists, allow extended slice assignment of differing lengths as long
   as the step is 1. (Previously, 'l[:2:1] = []' failed even though
   'l[:2] = []' and 'l[:2:None] = []' do not.)

 - Implement extended slicing for buffer, array, structseq, mmap and
   UserString.UserString.

 - Implement slice-object support (but not non-step-1 slice assignment) for
   UserString.MutableString.

 - Add tests for all new functionality.
2007-08-28 15:28:19 +00:00
Walter Dörwald 9ab80a9fb4 Move another variable declaration up. 2007-08-17 16:58:43 +00:00
Walter Dörwald 20b40d3bce Move variable declaration up. 2007-08-17 16:52:50 +00:00
Walter Dörwald 6e39080649 Backport r57105 and r57145 from the py3k branch: UTF-32 codecs. 2007-08-17 16:41:28 +00:00
Georg Brandl 9efd9b6fa4 Bug #1763149: use proper slice syntax in docstring.
(backport)
2007-07-29 17:38:35 +00:00
Martin v. Löwis 6819210b9e PEP 3123: Provide forward compatibility with Python 3.0, while keeping
backwards compatibility. Add Py_Refcnt, Py_Type, Py_Size, and
PyVarObject_HEAD_INIT.
2007-07-21 06:55:02 +00:00
Georg Brandl 7c3b50db66 Patch #1673759: add a missing overflow check when formatting floats
with %G.
2007-07-12 08:38:00 +00:00
Neal Norwitz 5c9a81a3d8 Fix a bug when there was a newline in the string expandtabs was called on.
This also catches another condition that can overflow.

Will backport.
2007-06-11 02:16:10 +00:00
Neal Norwitz 7dbd2a3720 Prevent expandtabs() on string and unicode objects from causing a segfault when
a large width is passed on 32-bit platforms.  Found by Google.

It would be good for people to review this especially carefully and verify
I don't have an off by one error and there is no other way to cause overflow.
2007-06-09 03:36:34 +00:00
Neal Norwitz ee3a1b5244 Variation of patch # 1624059 to speed up checking if an object is a subclass
of some of the common builtin types.

Use a bit in tp_flags for each common builtin type.  Check the bit
to determine if any instance is a subclass of these common types.
The check avoids a function call and O(n) search of the base classes.
The check is done in the various Py*_Check macros rather than calling
PyType_IsSubtype().

All the bits are set in tp_flags when the type is declared
in the Objects/*object.c files because PyType_Ready() is not called
for all the types.  Should PyType_Ready() be called for all types?
If so and the change is made, the changes to the Objects/*object.c files
can be reverted (remove setting the tp_flags).  Objects/typeobject.c
would also have to be modified to add conditions
for Py*_CheckExact() in addition to each the PyType_IsSubtype check.
2007-02-25 19:44:48 +00:00
Armin Rigo 7ccbca93a2 Forward-port of r52136,52138: a review of overflow-detecting code.
* unified the way intobject, longobject and mystrtoul handle
  values around -sys.maxint-1.

* in general, trying to entierely avoid overflows in any computation
  involving signed ints or longs is extremely involved.  Fixed a few
  simple cases where a compiler might be too clever (but that's all
  guesswork).

* more overflow checks against bad data in marshal.c.

* 2.5 specific: fixed a number of places that were still confusing int
  and Py_ssize_t.  Some of them could potentially have caused
  "real-world" breakage.

* list.pop(x): fixing overflow issues on x was messy.  I just reverted
  to PyArg_ParseTuple("n"), which does the right thing.  (An obscure
  test was trying to give a Decimal to list.pop()... doesn't make
  sense any more IMHO)

* trying to write a few tests...
2006-10-04 12:17:45 +00:00
Raymond Hettinger a0c95fa4d8 Fix endcase for str.rpartition() 2006-09-04 15:32:48 +00:00
Neal Norwitz 17753ecbfa Patch #1541585: fix buffer overrun when performing repr() on
a unicode string in a build with wide unicode (UCS-4) support.

This code could be improved, so add an XXX comment.
2006-08-21 22:21:19 +00:00
Marc-André Lemburg 3a457790c7 Correct an accidentally removed previous patch. 2006-08-14 12:57:27 +00:00
Marc-André Lemburg 040f76b79c Slightly revised version of patch #1538956:
Replace UnicodeDecodeErrors raised during == and !=
compares of Unicode and other objects with a new
UnicodeWarning.

All other comparisons continue to raise exceptions.
Exceptions other than UnicodeDecodeErrors are also left
untouched.
2006-08-14 10:55:19 +00:00
Neal Norwitz 8a87f5d37e Patch #1538606, Patch to fix __index__() clipping.
I modified this patch some by fixing style, some error checking, and adding
XXX comments.  This patch requires review and some changes are to be expected.
I'm checking in now to get the greatest possible review and establish a
baseline for moving forward.  I don't want this to hold up release if possible.
2006-08-12 17:03:09 +00:00
Neal Norwitz e1fdb32ff2 Handle allocation failures gracefully. Found with failmalloc.
Many (all?) of these could be backported.
2006-07-21 05:32:28 +00:00
Martin v. Löwis d825143be1 Patch #1455898: Incremental mode for "mbcs" codec. 2006-06-14 05:21:04 +00:00
Neal Norwitz de4c78a1d7 Initialize the type object so pychecker can't crash the interpreter. 2006-06-13 08:28:19 +00:00
Georg Brandl 90e27d38f5 Apply perky's fix for #1503157: "/".join([u"", u""]) raising OverflowError.
Also improve error message on overflow.
2006-06-10 06:40:50 +00:00
Georg Brandl 242508160e RFE #1491485: str/unicode.endswith()/startswith() now accept a tuple as first argument. 2006-06-09 18:45:48 +00:00
Georg Brandl 9f16760666 Repair refleaks in unicodeobject. 2006-06-04 21:46:16 +00:00
Martin v. Löwis 3f767795f6 Patch #1359618: Speed-up charmap encoder. 2006-06-04 19:36:28 +00:00
Fredrik Lundh 60d8b18831 needforspeed: stringlib refactoring: changed find_obj to find_slice,
to enable use from stringobject
2006-05-27 15:20:22 +00:00
Fredrik Lundh c2d29c5a6d needforspeed: replace improvements, changed to Py_LOCAL_INLINE
where appropriate
2006-05-27 14:58:20 +00:00
Martin v. Löwis 2e3f6b77d5 Revert bogus change committed in 46432 to this file. 2006-05-27 11:07:49 +00:00
Andrew Dalke e0df762719 fixed typo 2006-05-27 11:04:36 +00:00
Fredrik Lundh 2d23d5bf2e needforspeed: more stringlib refactoring 2006-05-27 10:05:10 +00:00
Martin v. Löwis d004fc810a Patch 1494554: Update numeric properties to Unicode 4.1. 2006-05-27 08:36:52 +00:00
Neal Norwitz d1b6cd7bfb Fix Coverity warnings.
- Check the correct variable (str_obj, not str) for NULL
 - sep_len was already verified it wasn't 0
2006-05-27 05:21:30 +00:00
Andrew M. Kuchling 07bbfc6a51 Comment typo 2006-05-26 19:51:10 +00:00
Fredrik Lundh e6e43c867d needforspeed: stringlib refactoring: use stringlib/find for string find 2006-05-26 19:48:07 +00:00
Fredrik Lundh c816281304 needforspeed: use a macro to fix slice indexes 2006-05-26 19:33:03 +00:00
Fredrik Lundh ce4eccb0c4 needforspeed: stringlib refactoring: use stringlib/find for unicode
find
2006-05-26 19:29:05 +00:00
Fredrik Lundh 58b5e84d52 needforspeed: stringlib refactoring, continued. added count and
find helpers; updated unicodeobject to use stringlib_count
2006-05-26 19:24:53 +00:00
Fredrik Lundh 9c0e9c089c needspeed: rpartition documentation, tests, and a bug fixes.
feel free to add more tests and improve the documentation.
2006-05-26 18:24:15 +00:00
Fredrik Lundh b3167cbcd7 needforspeed: added rpartition implementation 2006-05-26 18:15:38 +00:00
Fredrik Lundh b947948c61 needforspeed: stringlib refactoring (in progress) 2006-05-26 17:22:38 +00:00
Fredrik Lundh a50d201bd9 needforspeed: stringlib refactoring (in progress) 2006-05-26 17:04:58 +00:00
Fredrik Lundh 95e2a91615 use Py_LOCAL also for string and unicode objects 2006-05-26 11:38:15 +00:00
Fredrik Lundh f2c0dfdb13 needforspeed: use Py_ssize_t for the fastsearch counter and skip
length (thanks, neal!).  and yes, I've verified that this doesn't
slow things down ;-)
2006-05-26 10:27:17 +00:00
Fredrik Lundh 450277fef5 needforspeed: use METH_O for argument handling, which made partition some
~15% faster for the current tests (which is noticable faster than a corre-
sponding find call).  thanks to neal-who-never-sleeps for the tip.
2006-05-26 09:46:59 +00:00
Fredrik Lundh 06a69dd8ff needforspeed: partition implementation, part two.
feel free to improve the documentation and the docstrings.
2006-05-26 08:54:28 +00:00
Andrew Dalke b552c4d848 Code had returned an ssize_t, upcast to long, then converted with PyInt_FromLong.
Now using PyInt_FromSsize_t.
2006-05-25 18:03:25 +00:00
Fredrik Lundh 0c71f88fc9 needforspeed: check for overflow in replace (from Andrew Dalke) 2006-05-25 16:46:54 +00:00
Fredrik Lundh 347ee277aa needforspeed: refactored the replace code slightly; special-case
constant-length changes; use fastsearch to locate the first match.
2006-05-24 16:35:18 +00:00
Fredrik Lundh d5e0dc51cf needforspeedindeed: use fastsearch also for __contains__ 2006-05-24 15:11:01 +00:00
Fredrik Lundh 6471ee4f18 needforspeed: use "fastsearch" for count and findstring helpers. this
results in a 2.5x speedup on the stringbench count tests, and a 20x (!)
speedup on the stringbench search/find/contains test, compared to 2.5a2.

for more on the algorithm, see:

    http://effbot.org/zone/stringlib.htm

if you get weird results, you can disable the new algoritm by undefining
USE_FAST in Objects/unicodeobject.c.

enjoy /F
2006-05-24 14:28:11 +00:00
Fredrik Lundh 240bf2a8e4 use Py_ssize_t for string indexes (thanks, neal!) 2006-05-24 10:20:36 +00:00
Fredrik Lundh 7763351808 return 0 on misses, not -1. 2006-05-23 19:47:35 +00:00
Fredrik Lundh b63588c188 needforspeed: use append+reverse for rsplit, use "bloom filters" to
speed up splitlines and strip with charsets; etc.  rsplit is now as
fast as split in all our tests (reverse takes no time at all), and
splitlines() is nearly as fast as a plain split("\n") in our tests.
and we're not done yet... ;-)
2006-05-23 18:44:25 +00:00
Fredrik Lundh 833bf9422e needforspeed: fixed unicode "in" operator to use same implementation
approach as find/index
2006-05-23 10:12:21 +00:00
Tim Peters 1bacc641a0 unicode_repeat(): Change type of local to Py_ssize_t,
since that's what it should be.
2006-05-23 05:47:16 +00:00
Tim Peters 286085c781 PyUnicode_Join(): Recent code changes introduced new
compiler warnings on Windows (signed vs unsigned mismatch
in comparisons).  Cleaned that up by switching more locals
to Py_ssize_t.  Simplified overflow checking (it can _be_
simpler because while these things are declared as
Py_ssize_t, then should in fact never be negative).
2006-05-22 19:17:04 +00:00
Fredrik Lundh 8a8e05a2b9 needforspeed: use memcpy for "long" strings; use a better algorithm
for long repeats.
2006-05-22 17:12:58 +00:00
Fredrik Lundh f1d60a5384 needforspeed: speed up unicode repeat, unicode string copy 2006-05-22 16:29:30 +00:00
Fredrik Lundh 763b50f9d9 docstring tweaks: count counts non-overlapping substrings, not
total number of occurences
2006-05-22 15:35:12 +00:00
Neal Norwitz 1004a5339a Patch #1488312, Fix memory alignment problem on SPARC in unicode. Will backport 2006-05-15 07:17:23 +00:00
Thomas Wouters 715a4cdea2 Use %zd instead of %i as format character (in call to PyErr_Format) for
Py_ssize_t argument.
2006-04-16 22:04:49 +00:00
Martin v. Löwis 5cb6936672 Make Py_BuildValue, PyObject_CallFunction and
PyObject_CallMethod aware of PY_SSIZE_T_CLEAN.
2006-04-14 09:08:42 +00:00
Martin v. Löwis f15da6995b Remove another INT_MAX limitation 2006-04-13 07:24:50 +00:00
Martin v. Löwis 412fb67368 Change more ints to Py_ssize_t. 2006-04-13 06:34:32 +00:00
Martin v. Löwis 80d2e591d5 Revert 34153: Py_UNICODE should not be signed. 2006-04-13 06:06:08 +00:00
Anthony Baxter ac6bd46d5c spread the extern "C" { } magic pixie dust around. Python itself builds now
using a C++ compiler. Still lots and lots of errors in the modules built by
setup.py, and a bunch of warnings from g++ in the core.
2006-04-13 02:06:09 +00:00
Anthony Baxter a62862120d More low-hanging fruit. Still need to re-arrange some code (or find a better
solution) in the same way as listobject.c got changed. Hoping for a better
solution.
2006-04-11 07:42:36 +00:00
Georg Brandl ecdc0a9f46 That one was a mistake. 2006-03-30 12:19:07 +00:00
Georg Brandl 347b30042b Remove unnecessary casts in type object initializers. 2006-03-30 11:57:00 +00:00
Thomas Wouters a96affe1fc - Reindent a confusingly indented piece of code (no intended code changes
there)
 - Add missing DECREFs of inner-scope 'temp' variable
 - Add various missing DECREFs by changing 'return NULL' into 'goto onError'
 - Avoid double DECREF when last _PyUnicode_Resize() fails

Coverity found one of the missing DECREFs, but oddly enough not the others.
2006-03-12 00:29:36 +00:00
Martin v. Löwis 480f1bb67b Update Unicode database to Unicode 4.1. 2006-03-09 23:38:20 +00:00
Guido van Rossum 38fff8c4e4 Checking in the code for PEP 357.
This was mostly written by Travis Oliphant.
I've inspected it all; Neal Norwitz and MvL have also looked at it
(in an earlier incarnation).
2006-03-07 18:50:55 +00:00
Hye-Shik Chang 4af5c8cee4 SF #1444030: Fix several potential defects found by Coverity.
(reviewed by Neal Norwitz)
2006-03-07 15:39:21 +00:00
Martin v. Löwis 15e62742fa Revert backwards-incompatible const changes. 2006-02-27 16:46:16 +00:00
Thomas Wouters de01774dae Use correct PyArg_Parse format char for Py_ssize_t in unicode.center().
Fixes:

>>> u"".center(10)
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
MemoryError

on 64-bit systems.
2006-02-16 19:34:37 +00:00
Martin v. Löwis eb079f1c25 Use Py_ssize_t for counts and sizes.
Convert Py_ssize_t using PyInt_FromSsize_t
2006-02-16 14:32:27 +00:00
Martin v. Löwis 2c95cc6d72 Support %zd in PyErr_Format and PyString_FromFormat. 2006-02-16 06:54:25 +00:00
Tim Peters 15231548d2 doubletounicode(), longtounicode():
Py_SAFE_DOWNCAST can evaluate its first argument multiple
times in a debug build.  This caused two distinct assert-
failures in test_unicode run under a debug build.  Rewrote
the code in trivial ways so that multiple evaluation of the
first argument doesn't hurt.
2006-02-16 01:08:01 +00:00
Thomas Wouters 4701af5bf5 Remove two unused Py_ssize_t variables (merge glitches, looks like.) 2006-02-15 23:10:32 +00:00
Martin v. Löwis 18e165558b Merge ssize_t branch. 2006-02-15 17:27:45 +00:00
Neal Norwitz fc76d633e8 - Patch #1400181, fix unicode string formatting to not use the locale.
This is how string objects work.  u'%f' could use , instead of .
  for the decimal point.  Now both strings and unicode always use periods.

This is the code that would break:

import locale
locale.setlocale(locale.LC_NUMERIC, 'de_DE')
u'%.1f' % 1.0
assert '1.0' == u'%.1f' % 1.0

I couldn't create a test case which fails, but this fixes the problem.

Will backport.
2006-01-10 06:03:13 +00:00
Neal Norwitz d43069ce95 Fix icc warnings: remove (sometimes) unused variable conditionally 2006-01-08 01:12:10 +00:00
Martin v. Löwis dea59e5755 Stop maintaining the buildno file.
Also, stop determining Unicode sizes with PyString_GET_SIZE.
2006-01-05 10:00:36 +00:00
Hye-Shik Chang 835b243c71 Bug #1379994: Fix *unicode_escape codecs to encode r'\' as r'\\'
just like string codecs.
2005-12-17 04:38:31 +00:00
Jeremy Hylton af68c874a6 Add const to several API functions that take char *.
In C++, it's an error to pass a string literal to a char* function
without a const_cast().  Rather than require every C++ extension
module to put a cast around string literals, fix the API to state the
const-ness.

I focused on parts of the API where people usually pass literals:
PyArg_ParseTuple() and friends, Py_BuildValue(), PyMethodDef, the type
slots, etc.  Predictably, there were a large set of functions that
needed to be fixed as a result of these changes.  The most pervasive
change was to make the keyword args list passed to
PyArg_ParseTupleAndKewords() to be a const char *kwlist[].

One cast was required as a result of the changes:  A type object
mallocs the memory for its tp_doc slot and later frees it.
PyTypeObject says that tp_doc is const char *; but if the type was
created by type_new(), we know it is safe to cast to char *.
2005-12-10 18:50:16 +00:00
Walter Dörwald d4fff1731c Fix leaked reference to None. 2005-11-28 22:15:56 +00:00
Andrew M. Kuchling 8294de5673 Another comment typo fix 2005-11-02 16:36:12 +00:00
Walter Dörwald 2e2c02fedb Fix typo in comment. 2005-11-02 08:57:11 +00:00
Fred Drake db390c1ad8 fix typos, mostly in comments 2005-10-28 14:39:47 +00:00
Michael W. Hudson b2308bb9be Fix bug:
[ 1327110 ] wrong TypeError traceback in generator expressions

by removing the code that can stomp on the users' TypeError raised by the
iterable argument to ''.join() -- PySequence_Fast (now?) gives a perfectly
reasonable message itself.  Also, a couple of tests.
2005-10-21 11:45:01 +00:00
Marc-André Lemburg 5c4a9d6591 Whitespace corrections. 2005-10-19 22:39:02 +00:00
Marc-André Lemburg e115ec832c Bug fix for [ 1331062 ] utf 7 codec broken.
Backport candidate.
2005-10-19 22:33:31 +00:00
Walter Dörwald d1c1e10f70 Part of SF patch #1313939: Speedup charmap decoding by extending
PyUnicode_DecodeCharmap() the accept a unicode string as the mapping
argument which is used as a mapping table.

This code isn't used by any of the codecs yet.
2005-10-06 20:29:57 +00:00
Walter Dörwald a47d1c08d0 SF bug #1251300: On UCS-4 builds the "unicode-internal" codec will now complain
about illegal code points. The codec now supports PEP 293 style error handlers.
(This is a variant of the Nik Haldimann's patch that detects truncated data)
2005-08-30 10:23:14 +00:00
Marc-André Lemburg a9cadcd41b Correct the handling of 0-termination of PyUnicode_AsWideChar()
and its usage in PyLocale_strcoll().

Clarify the documentation on this.

Thanks to Andreas Degert for pointing this out.
2004-11-22 13:02:31 +00:00
Marc-André Lemburg 204bd6d9d2 Applied patch for [ 1047269 ] Buffer overwrite in PyUnicode_AsWideChar.
Python 2.3.x candidate.
2004-10-15 07:45:05 +00:00