Commit Graph

100 Commits

Author SHA1 Message Date
Andrew M. Kuchling 9b3a824097 Comment grammar 2006-10-06 18:51:55 +00:00
Neal Norwitz 71e05f1e0c Don't truncate if size_t is bigger than uint 2006-06-12 02:07:57 +00:00
Neal Norwitz d21a7fffb1 Patch #1357836:
Prevent an invalid memory read from test_coding in case the done flag is set.
In that case, the loop isn't entered.  I wonder if rather than setting
the done flag in the cases before the loop, if they should just exit early.

This code looks like it should be refactored.

Backport candidate (also the early break above if decoding_fgets fails)
2006-06-02 06:23:00 +00:00
Skip Montanaro a0b6338823 C++ compiler cleanup: cast signed to unsigned 2006-04-18 00:53:06 +00:00
Neal Norwitz 08062d6665 As discussed on python-dev, really fix the PyMem_*/PyObject_* memory API
mismatches.  At least I hope this fixes them all.

This reverts part of my change from yesterday that converted everything
in Parser/*.c to use PyObject_* API.  The encoding doesn't really need
to use PyMem_*, however, it uses new_string() which must return PyMem_*
for handling the result of PyOS_Readline() which returns PyMem_* memory.

If there were 2 versions of new_string() one that returned PyMem_*
for tokens and one that return PyObject_* for encodings that could
also fix this problem.  I'm not sure which version would be clearer.
This seems to fix both Guido's and Phillip's problems, so it's good enough
for now.  After this change, it would be good to review Parser/*.c
for consistent use of the 2 memory APIs.
2006-04-11 08:19:15 +00:00
Anthony Baxter 114900298e Fix the code in Parser/ to also compile with C++. This was mostly casts for
malloc/realloc type functions, as well as renaming one variable called 'new'
in tokensizer.c. Still lots more to be done, going to be checking in one
chunk at a time or the patch will be massively huge. Still compiles ok with
gcc.
2006-04-11 05:39:14 +00:00
Neal Norwitz 2c4e4f9839 SF patch #1467512, fix double free with triple quoted string in standard build.
This was the result of inconsistent use of PyMem_* and PyObject_* allocators.
By changing to use PyObject_* allocator almost everywhere, this removes
the inconsistency.
2006-04-10 06:42:25 +00:00
Tim Peters c9d78aa470 Years in the making.
objimpl.h, pymem.h:  Stop mapping PyMem_{Del, DEL} and PyMem_{Free, FREE}
to PyObject_{Free, FREE} in a release build.  They're aliases for the
system free() now.

_subprocess.c/sp_handle_dealloc():  Since the memory was originally
obtained via PyObject_NEW, it must be released via PyObject_FREE (or
_DEL).

pythonrun.c, tokenizer.c, parsermodule.c:  I lost count of the number of
PyObject vs PyMem mismatches in these -- it's like the specific
function called at each site was picked at random, sometimes even with
memory obtained via PyMem getting released via PyObject.  Changed most
to use PyObject uniformly, since the blobs allocated are predictably
small in most cases, and obmalloc is generally faster than system
mallocs then.

If extension modules in real life prove as sloppy as Python's front
end, we'll have to revert the objimpl.h + pymem.h part of this patch.
Note that no problems will show up in a debug build (all calls still go
thru obmalloc then). Problems will show up only in a release build, most
likely segfaults.
2006-03-26 23:27:58 +00:00
Neal Norwitz 2aa9a5dfdd Use macro versions instead of function versions when we already know the type.
This will hopefully get rid of some Coverity warnings, be a hint to
developers, and be marginally faster.

Some asserts were added when the type is currently known, but depends
on values from another function.
2006-03-20 01:53:23 +00:00
Thomas Wouters 7eaf2aaf48 Fix crashing bug in tokenizer, when tokenizing files with non-ASCII bytes
but without a specified encoding: decoding_fgets() (and decoding_feof()) can
return NULL and fiddle with the 'tok' struct, making tok->buf NULL. This is
okay in the other cases of calls to decoding_*(), it seems, but not in this
one.

This should get a test added, somewhere, but the testsuite doesn't seem to
test encoding anywhere (although plenty of tests use it.)

It seems to me that decoding errors in other places in the code (like at the
start of a token, instead of in the middle of one) make the code end up
adding small integers to NULL pointers, but happen to check for error states
before using the calculated new pointers. I haven't been able to trigger any
other crashes, in any case.

I would nominate this file for a comlete rewrite for Py3k. The whole
decoding trick is too bolted-on for my tastes.
2006-03-02 20:41:27 +00:00
Martin v. Löwis 49c5da1d88 Patch #1440601: Add col_offset attribute to AST nodes. 2006-03-01 22:49:05 +00:00
Martin v. Löwis 6cba25666c Change non-ASCII warning into a SyntaxError. 2006-02-28 22:41:29 +00:00
Martin v. Löwis f5adf1eb72 Use Py_ssize_t to count the length. 2006-02-16 14:35:38 +00:00
Martin v. Löwis 18e165558b Merge ssize_t branch. 2006-02-15 17:27:45 +00:00
Neal Norwitz 30b5c5d011 Fix SF bug #1072182, problems with signed characters.
Most of these can be backported.
2005-12-19 06:05:18 +00:00
Neal Norwitz db83eb3170 Fix Bug #1378022, UTF-8 files with a leading BOM crashed the interpreter.
Needs backport.
2005-12-18 05:29:30 +00:00
Neal Norwitz dee2fd5448 Fix some more memory leaks.
Call error_ret() in decode_str().  It was called in some other places,
but seemed inconsistent.  It is safe to call PyTokenizer_Free() after
calling error_ret().
2005-11-16 05:12:59 +00:00
Neal Norwitz c0d5faa9b4 Free coding spec (cs) if there was an error to prevent mem leak. Maybe backport candidate 2005-10-21 06:05:33 +00:00
Neal Norwitz 40d3781416 - Fix segfault with invalid coding.
- SF Bug #772896, unknown encoding results in MemoryError, which is not helpful

I will only backport the segfault fix.  I'll let Anthony decide if he wants
the other changes backported.  I will do the backport if asked.
2005-10-02 01:48:49 +00:00
Walter Dörwald c1f5fff2b7 Apply SF patch #1101726: Fix buffer overrun in tokenizer.c when a source file
with a PEP 263 encoding declaration results in long decoded line.
2005-07-12 21:53:43 +00:00
Martin v. Löwis 4bf108d74f Patch #802188: better parser error message for non-EOL following line cont. 2005-03-03 11:45:45 +00:00
Hye-Shik Chang 7df44b384a SF #941229: Decode source code with sys.stdin.encoding in interactive
modes like non-interactive modes.  This allows for non-latin-1 users
to write unicode strings directly and sets Japanese users free from
weird manual escaping <wink> in shift_jis environments.
(Reviewed by Martin v. Loewis)
2004-08-04 17:36:41 +00:00
Anthony Baxter c2a5a63654 PEP-0318, @decorator-style. In Guido's words:
"@ seems the syntax that everybody can hate equally"
Implementation by Mark Russell, from SF #979728.
2004-08-02 06:10:11 +00:00
Jack Jansen eddc1449ba Getting rid of all the code inside #ifdef macintosh too. 2003-11-20 01:44:59 +00:00
Marc-André Lemburg 1fb1400d08 Add URL for PEP to the source code encoding warning.
Remove the usage of PyErr_WarnExplicit() since this could cause
sensitive information from the source files to appear in e.g.
log files.
2003-02-17 18:31:57 +00:00
Just van Rossum f032f86e9e patch 680474 that fixes bug 679880: compile/eval/exec refused utf-8 bom
mark. Added unit test.
2003-02-09 20:38:48 +00:00
Mark Hammond a2e303c32d Fix [ 665014 ] files with long lines and an encoding crash.
Ensure that the 'size' arg is correctly passed to the encoding reader to
prevent buffer overflows.
2003-01-14 23:15:22 +00:00
Martin v. Löwis 95292d6caa Constify filenames and scripts. Fixes #651362. 2002-12-11 14:04:59 +00:00
Neal Norwitz e08e1bc80a Fix compiler warning on HP-UX.
Cast param to isalnum() to int.
2002-11-02 20:43:25 +00:00
Martin v. Löwis 566f6afe9a Patch #512981: Update readline input stream on sys.stdin/out change. 2002-10-26 14:39:10 +00:00
Tim Peters 17db21ffd0 Removed reliance on gcc/C99 extension. 2002-09-03 15:39:58 +00:00
Martin v. Löwis f62a89b1e0 Ignore encoding declarations inside strings. Fixes #603509. 2002-09-03 11:52:44 +00:00
Guido van Rossum 84b2bed435 Squash a few calls to the hideously expensive PyObject_CallObject(o,a)
-- replace then with slightly faster PyObject_Call(o,a,NULL).  (The
difference is that the latter requires a to be a tuple; the former
allows other values and wraps them in a tuple if necessary; it
involves two more levels of C function calls to accomplish all that.)
2002-08-16 17:01:09 +00:00
Skip Montanaro 118ec70ea2 provide less mysterious error messages when seeing end-of-line in
single-quoted strings or end-of-file in triple-quoted strings.
closes patch 586561.
2002-08-15 01:20:16 +00:00
Martin v. Löwis 2863c10a86 Use Py_FatalError instead of abort. 2002-08-07 15:18:57 +00:00
Martin v. Löwis 019934b3cc Fix PEP 263 code --without-unicode. Fixes #591943. 2002-08-07 12:33:18 +00:00
Jack Jansen cf0a2cfdb2 Added a cast to shut up a compiler warning. 2002-08-05 14:14:05 +00:00
Martin v. Löwis 725bb233b9 Add 1 to lineno in deprecation warning. Fixes #590888. 2002-08-05 01:49:16 +00:00
Martin v. Löwis 1ee99d31d9 Make pgen compile with pydebug. Duplicate normalized names, as it may
be longer than the old string.
2002-08-04 20:10:29 +00:00
Martin v. Löwis cd280fb59c Group statements properly. 2002-08-04 18:28:44 +00:00
Tim Peters 2c3f9c6f04 Repaired a fatal compiler error in the debug build: it's not clear what
this was trying to assert, but the name it referenced didn't exist.
2002-08-04 17:58:34 +00:00
Tim Peters 919603b27a Squash compiler wng about signed-vs-unsigned mismatch. 2002-08-04 17:56:42 +00:00
Martin v. Löwis 00f1e3f5a5 Patch #534304: Implement phase 1 of PEP 263. 2002-08-04 17:29:52 +00:00
Jack Jansen 7b8c7546eb Mass checkin of universal newline support.
Highlights: import and friends will understand any of \r, \n and \r\n
as end of line. Python file input will do the same if you use mode 'U'.
Everything can be disabled by configuring with --without-universal-newlines.

See PEP278 for details.
2002-04-14 20:12:41 +00:00
Tim Peters d507dab91f SF patch #455966: Allow leading 0 in float/imag literals.
Consequences for Jython still unknown (but raised on Jython-Dev).
2001-08-30 20:51:59 +00:00
Tim Peters 9aa70d93aa SF bug [#455775] float parsing discrepancy.
PyTokenizer_Get:  error if exponent contains no digits (3e, 2.0e+, ...).
2001-08-27 19:19:28 +00:00
Guido van Rossum 4668b000a1 Implement PEP 238 in its (almost) full glory.
This introduces:

- A new operator // that means floor division (the kind of division
  where 1/2 is 0).

- The "future division" statement ("from __future__ import division)
  which changes the meaning of the / operator to implement "true
  division" (where 1/2 is 0.5).

- New overloadable operators __truediv__ and __floordiv__.

- New slots in the PyNumberMethods struct for true and floor division,
  new abstract APIs for them, new opcodes, and so on.

I emphasize that without the future division statement, the semantics
of / will remain unchanged until Python 3.0.

Not yet implemented are warnings (default off) when / is used with int
or long arguments.

This has been on display since 7/31 as SF patch #443474.

Flames to /dev/null.
2001-08-08 05:00:18 +00:00
Tim Peters cf96de052f SF but #417587: compiler warnings compiling 2.1.
Repaired *some* of the SGI compiler warnings Sjoerd Mullender reported.
2001-04-21 02:46:11 +00:00
Guido van Rossum 8586991099 REMOVED all CWI, CNRI and BeOpen copyright markings.
This should match the situation in the 1.6b1 tree.
2000-09-01 23:29:29 +00:00
Thomas Wouters 434d0828d8 Support for three-token characters (**=, >>=, <<=) which was written by
Michael Hudson, and support in general for the augmented assignment syntax.
The graminit.c patch is large!
2000-08-24 20:11:32 +00:00