Andy Robinson noted a core dump in the codecs.c file. This
was introduced by my latest patch which fixed a memory leak
in codecs.c. The bug causes all successful codec lookups to fail.
Attached you find an update of the Unicode implementation.
The patch is against the current CVS version. I would appreciate
if someone with CVS checkin permissions could check the changes
in.
The patch contains all bugs and patches sent this week and also
fixes a leak in the codecs code and a bug in the free list code
for Unicode objects (which only shows up when compiling Python
with Py_DEBUG; thanks to MarkH for spotting this one).
Added wrapping macros to dictobject.c, listobject.c, tupleobject.c,
frameobject.c, traceback.c that safely prevends core dumps
on stack overflow. Macros and functions in object.c, object.h.
The method is an "elevator destructor" that turns cascading
deletes into tail recursive behavior when some limit is hit.
* Changes to a recent patch by Chris Tismer to errors.c. Chris' patch
always used FormatMessage() to get the error message passing the error code
from errno - but errno and FormatMessage use a different numbering scheme.
The main reason the patch looked OK was that ENOFILE==ERROR_FILE_NOT_FOUND -
but that is about the only shared error code :-). The MS CRT docs tell you
to use _sys_errlist()/_sys_nerr. My patch does also this, and adds a very
similar function specifically for win32 error codes.
PR#175 -- when exec is passed a code object, it didn't sync the locals
from the dictionary back into their fast representation.
Also took the time to remove some repetitive code there and to do the
syncing even when an exception is raised (since a partial effect
should still be synced).
* in import.c, #ifdef out references to dynamic loading based on
HAVE_DYNAMIC_LOADING
* clean out the platform-specific crud from importdl.c.
[ maybe fold this function into import.c and drop the importdl.c file? Greg.]
* change GetDynLoadFunc's "funcname" parameter to "shortname". change
"name" to "fqname" for clarification.
* each GetDynLoadFunc now creates its own funcname value.
WARNING: as I mentioned previously, we may run into an issue with a
missing "_" on some platforms. Testing will show this pretty quickly,
however.
* move pathname munging into dynload_shlib.c
Here's a patch that avoids a warning caused by the "const char* pathname"
declaration for _PyImport_GetDynLoadFunc (in dynload_aix). The "aix_load"
function's 1st arg is prototyped as "char *pathname".
file per platform (really: per style of Dl API; e.g. all platforms
using dlopen() are grouped together in dynload_shlib.c.).
This is part of a set of patches by Greg Stein.
Duzan, for AIX, to support C++ objects with static initializers, when
using the genuine IBM C++ compiler (namely xlC/xlC_r).
See accompanying patches to configure.in and acconfig.h.
not as descriptive as what Barry suggests, but this also catches the
(in my opinion important) case where some other C code besides apply()
constructs a kwdict that doesn't have the right format. All the other
possibilities of getting it wrong (non-dict, wrong keywords etc) are
already caught so this makes sense to check here.
For a long time I've seen absurd tracebacks under -O (e.g., negative
line numbers), but very rarely. Since I was looking at tracebacks
anyway, thought I'd track it down. Turns out to be Guido's only
predictable blind spot <wink -- "char" is signed on some non-GvR
systems>. Patch follows.
tracefunc (or profilefunc -- we're not sure which), zap the global
trace and profile funcs so that we can't get into recursive loop when
instantiating the resulting class based exception.
"""
Following up Robin Dunn's troubles with freeze, here's a patch that
fixes an oddity regarding the import logic of shared modules on AIX.
Symbol resolution of shared modules is now handled properly for the cases
when the python library is linked to a binary with an arbitrary name.
This includes the standard python[version] executable, but also applications
that are embedding the python core (i.e. linked with libpython[version].a,
the latter being static or shared).
"""
Introduce a new builtin exception, UnboundLocalError, raised when ceval.c
tries to retrieve or delete a local name that isn't bound to a value.
Currently raises NameError, which makes this behavior a FAQ since the same
error is raised for "missing" global names too: when the user has a global
of the same name as the unbound local, NameError makes no sense to them.
Even in the absence of shadowing, knowing whether a bogus name is local or
global is a real aid to quick understanding.
Example:
D:\src\PCbuild>type local.py
x = 42
def f():
print x
x = 13
return x
f()
D:\src\PCbuild>python local.py
Traceback (innermost last):
File "local.py", line 8, in ?
f()
File "local.py", line 4, in f
print x
UnboundLocalError: x
D:\src\PCbuild>
Note that UnboundLocalError is a subclass of NameError, for compatibility
with existing class-exception code that may be trying to catch this as a
NameError. Unfortunately, I see no way to make this wholly compatible
with -X (see comments in bltinmodule.c): under -X, [UnboundLocalError
is an alias for NameError --GvR].
[The ceval.c patch differs slightly from the second version that Tim
submitted; I decided not to raise UnboundLocalError for DELETE_NAME,
only for DELETE_LOCAL. DELETE_NAME is only generated at the module
level, and since at that level a NameError is raised for referencing
an undefined name, it should also be raised for deleting one.]
We occasionally received reports from people getting "invalid tstate"
crashes (this is a fatal error in PyThreadState_Delete()). Finally
several people were able to reproduce it reliably and Tim Peters
discovered that there is a race condition when multiple threads are
calling this function without holding the global interpreter lock (the
function may be called without holding that).
Solved the race condition by adding a lock around the mutating uses of
interp->tstate_head. Tim and Jonathan Giddy have run tests that make
it likely that this fixes the crashes -- although Tim hasn't heard
from the person who reported the original problem.
ExtensionClasses in isinstance() and issubclass().
- abstract instance and class protocols are used *only* in those
cases that would generate errors before the patch. That is, there's
no penalty for the normal case.
- instance protocol: an object smells like an instance if it
has a __class__ attribute that smells like a class.
- class protocol: an object smells like a class if it has a
__bases__ attribute that is a tuple with elements that
smell like classes (although not all elements may actually get
sniffed ;).
man pages suggest that the proper thing to do is to add THR_NEW_LWP to
the flags on thr_create(), and that there really isn't a downside, so
I'll do that.
"""
Spec says that on success pthread_create returns 0. It does not say
that an error code will be < 0. Linux glibc2 pthread_create() returns
ENOMEM (12) when one exceed process limits. (It looks like it should
return EAGAIN, but that's another story.)
For reference, see:
http://www.opengroup.org/onlinepubs/7908799/xsh/pthread_create.html
"""
[I have a feeling that similar bugs were fixed before; perhaps someone
could check that all error checks no check for != 0?]
xrange(), especially for platforms where int and long are different
sizes (so sys.maxint isn't actually the theoretical limit for the
length of a list, but the largest C int is -- sys.maxint is the
largest Python int, which is actually a C long).
test for classes with a __complex__() method. The attribute is pulled
out of the instance with PyObject_GetAttr() but this transfers
ownership and the function object was never DECREF'd.
v temporary variable was never decref'd. Test this by starting up the
interpreter, hitting C-c, then immediately exiting.
Same potential leak can occur if error is E_NOMEM, since the return is
done in the case block. Added Py_XDECREF(v); to both blocks, just
before the return.
think we have our own DOS box (i.e. we're not started from a command
line shell), we print a message and wait for the user to hit a key
before the DOS box is closed.
The hacky heuristic for determining whether we have our *own* DOS box
(due to Mark Hammond) is to test whether we're on line zero...
The following patches (relative to 1.5.2b1) enable Python dynamic
loading to work on NetBSD platforms that use ELF (presnetly mips and
alpha systems). They automaticly determine wether the system is ELF or
a.out rather than using astatic list of platforms so that when other
NetBSD platforms move to ELF, python will continue to work without
change.
In other words, hex(sys.hexversion) == 0x010502b2 for Python 1.5.2b2.
This is derived from the new variable PY_VERSION_HEX defined in patchlevel.h.
(Cute, eh?)
Also (non-BSDI specific):
- Change the CHECK_STATUS() macro so it tests for nonzero error codes
instead of negative error codes only (this was needed for BSDI, but
appears to be correct according to the PTHREADS spec).
- use memset() to zero out the allocated lock structure. Again, this
was needed for BSDI, but can't hurt elsewhere either.
initialization of class exceptions. Specifically:
init_class_exc(): This function now returns an integer status of the
class exception initialization. No fatal errors in this method now.
Also, use PySys_WriteStderr() when writing error messages. When an
error occurs in this function, 0 is returned, but the partial creation
of the exception classes is not undone (this happens elsewhere).
Things that could trigger the fallback:
- exceptions.py fails to be imported (due to syntax error, etc.)
- one of the exception classes is missing (e.g. due to library
version mismatch)
- exception class can't be inserted into __builtin__'s dictionary
- MemoryError instance can't be pre-allocated
- some other PyErr_Occurred
newstdexception(): Changed the error message. This is still a fatal
error because if the string based exceptions can't be created, we
really can't continue.
initerrors(): Be sure to xdecref the .exc field, which might be
non-NULL if class exceptions init was aborted.
_PyBuiltin_Init_2(): If class exception init fails, print a warning
message and reinstate the string based exceptions.
that file in fact did not exist or at least was not used. Change this
so that __file__ is *only* set to the .pyc/.pyo file when it actually
read the code object from it; otherwise __file__ is set to the .py
file.
happen when you use a non-keyword argument after a keyword argument,
and in this case you also get a syntax error. I fully suspect that
the underflow is caused by the code that stops generating code when it
detects the syntax error, but I can't find the culprit right now. I
know, I know.)
The MS compiler doesn't call it 'long long', it uses __int64,
so a new #define, LONG_LONG, has been added and all occurrences
of 'long long' are replaced with it.
This is a patch that Bill Bummgarner did for 1.4 that hasn't made its
way into the distribution yet. This is important if you want to use
the ObjC module.
frozen packages. (I *think* this means that we can now have a
built-in module bar that's a submodule of a frozen package foo, by
registering the built-in module with a name "foo.bar" in the table of
builtin modules.)
an exception from errno, with a supplied filename (primarily used by
IOError and OSError). If class exceptions are used then the exception
is instantiated with a 3-tuple: (errno, strerror, filename). For
backwards compatibility reasons, if string exceptions are used,
filename is ignored.
PyErr_SetFromErrno(): Implement in terms of
PyErr_SetFromErrnoWithFilename().
OSError. The EnvironmentError serves primarily as the (common
implementation) base class for IOError and OSError. OSError is used
by posixmodule.c
Also added tuple definition of EnvironmentError when using string
based exceptions.
(1) If a sequence S is shorter than len(S) indicated, don't fail --
just use the shorter size. (I.e, len(S) is just a hint.)
(2) Implement the special case map(None, S) as list(S) -- it's faster.
must be enabled here, otherwise the errno we set on overflows is not
the errno that's being read by compile.c. Wonder how many other files
that do their own "#include config.h" need this too :-(
(Because of the structure of autoconf, it's not so simple to get this
into config.h...)
the filename contains at least a rudimentary pathname.
(The bad part is that we need to call getcwd() because only a prefix
of ".\\" is not enough -- we prefix the drive letter.)
and lists; if the size is negative, raise an exception. Also raise an
exception when an undefined type is found -- all this to increase the
chance that garbage input causes an exception instead of a core dump.
swapped arguments].
Also make sure that no use of a function pointer gotten from a
tp_as_sequence or tp_as_mapping structure is made without checking it
for NULL first.
the code here becomes much simpler. In particular: abs(), divmod(),
pow(), int(), long(), float(), len(), tuple(), list().
Also make sure that no use of a function pointer gotten from a
tp_as_sequence or tp_as_mapping structure is made without checking it
for NULL first.
A few other cosmetic things, such as properly reindenting slice().
old value in a temporary and XDECREF it only after then new value has
been set. This prevents the (unlikely) case where the destructor of
the member uses the containing object -- it would find it in an
undefined state.
because the path through the code would notice that sys.__path__ did
not exist and it would fall back to the default path (builtins +
sys.path) instead of failing). No longer.
Date: Thu, 14 Sep 1995 12:18:20 -0400
From: Alan Morse <alan@dvcorp.com>
To: python-list@cwi.nl
Subject: getargs bug in 1.2 and 1.3 BETA
We have found a bug in the part of the getargs code that we added
and submitted, and which was incorporated into 1.1.
The parsing of "O?" format specifiers is not handled correctly;
there is no "else" for the "if" and therefore it can never fail.
What's worse, the advancing of the varargs pointer is not
handled properly, so from then on it is out of sync, wreaking
all sorts of havoc. (If it had failed properly, then the out-of-sync
varargs would not have been an issue.)
Below is the context diff for the change.
Note that I have made a few stylistic changes beyond adding the
else case, namely:
1) Making the "O" case follow the convention established by the other
format specifiers of getting all their vararg arguments before
performing the test, rather than getting some before and some after
the test passes.
2) Making the logic of the tests parallel, so the "if" part indicates
that the format is accepted and the "else" part indicates that the
format has failed. They were inconsistent with each other and with the
the other format specifiers.
-Alan Morse (amorse@dvcorp.com)
to the table of built-in modules. This should normally be called
*before* Py_Initialize(). When the malloc() or realloc() call fails,
-1 is returned and the existing table is unchanged.
After a similar function by Just van Rossum.
int PyImport_ExtendInittab(struct _inittab *newtab);
int PyImport_AppendInittab(char *name, void (*initfunc)());
Adapted from code submitted by Just van Rossum.
PySys_WriteStdout(format, ...)
PySys_WriteStderr(format, ...)
The first function writes to sys.stdout; the second to sys.stderr. When
there is a problem, they write to the real (C level) stdout or stderr;
no exceptions are raised (but a pending exception may be cleared when a
new exception is caught).
Both take a printf-style format string as their first argument followed
by a variable length argument list determined by the format string.
*** WARNING ***
The format should limit the total size of the formatted output string to
1000 bytes. In particular, this means that no unrestricted "%s" formats
should occur; these should be limited using "%.<N>s where <N> is a
decimal number calculated so that <N> plus the maximum size of other
formatted text does not exceed 1000 bytes. Also watch out for "%f",
which can print hundreds of digits for very large numbers.
PyThreadState_GetDict() returns a dictionary that can be used to hold such
state; the caller should pick a unique key and store its state there. If
PyThreadState_GetDict() returns NULL, an exception has been raised (most
likely MemoryError) and the caller should pass on the exception. */
PyObject *
PyThreadState_GetDict()
Frozen packages are indicated by a negative size (the code string
is the __import__.py file). A frozen package module has its __path__
set to a string, the package name.
time can be in PyImport_ImportModuleEx(). Recursive calls from the
same thread are okay.
Potential problems:
- The lock should really be part of the interpreter state rather than
global, but that would require modifying more files, and I first want
to figure out whether this works at all.
- One could argue that the lock should be per module -- however that
would be complicated to implement. We would have to have a linked
list of locks per module name, *or* invent a new object type to
represent a lock, so we can store the locks in the module or in a
separate dictionary. Both seem unwarranted. The one situation where
this can cause problems is when loading a module takes a long time,
e.g. when the module's initialization code interacts with the user --
during that time, no other threads can run. I say, "too bad."
(modified) and use that.
Some differences in the cleanup algorithm:
- Clear __main__ before the other modules.
- Delete more sys variables: including ps1, ps2, exitfunc, argv, and
even path -- this will prevent new imports!
- Restore stdin, stdout, stderr from __stdin__, __stdout__,
__stderr__, effectively deleting hooks that the user might have
installed -- so their (the hooks') destructors will run.
This is an option for OS-es with case-insensitive but case-preserving
filesystems. It is currently supported for Win32 and MacOS. To
enable it, #define CHECK_IMPORT_CASE in your platform specific
config.h. It is enabled by default on those systems where it is
supported. On Win32, it can be disabled at runtime by setting the
environment variable PYTHONCASEOK (to any value).
When enabled, the feature checks that the case of the requested module
name matches that of the filename found in the filesystem, and raises
a NameError exception when they don't match.
pass it the true file. This is used to set __file__ properly, instead
of believing what the code object carries with it. (If the pointer
is NULL, the code object's co_filename is still used.)
- Add Py_FrozenFlag, intended to suppress error messages fron
getpath.c in frozen binaries.
- Add Py_GetPythonHome() and Py_SetPythonHome(), intended to allow
embedders to force a different PYTHONHOME.
- Add new interface PyErr_PrintEx(flag); same as PyErr_Print() but
flag determines whether sys.last_* are set or not. PyErr_Print()
now simply calls PyErr_PrintEx(1).
(1) Explicitly clear __builtin__._ and sys.{last,exc}_* before
clearing anything else. These are common places where user values
hide and people complain when their destructors fail. Since the
modules containing them are deleted *last* of all, they would come too
late in the normal destruction order. Sigh.
(2) Add some debugging aid to cleanup (after a suggestion by Marc
Lemburg) -- print the names of the modules being cleaned, and (when
-vv is used) print the names of the variables being cleared.
now implement the following finalization strategy.
1. Whenever this code deletes a module, its directory is cleared
carefully, as follows:
- set all names to None that begin with exactly one underscore
- set all names to None that don't begin with two underscores
- clear the directory
2. Modules are deleted in the following order:
- modules with a reference count of 1, except __builtin__ or __sys__
- repeat until no more are found with a reference count of 1
- __main__ if it's still there
- all remaining modules except __builtin__ or sys
- sys
_ __builtin__
This is a bit of a hack: when the shared library is loaded, the module
name is "package.module", but the module calls Py_InitModule*() with just
"module" for the name. The shared library loader squirrels away the true
name of the module in _Py_PackageContext, and Py_InitModule*() will
substitute this (if the name actually matches).
1) The __builtins__ variable in the __main__ module is set to the
__builtin__ module instead of its __dict__.
2) Get rid of the SIGHUP and SIGTERM handlers. They can't be made to
work reliably when threads may be in use, they are Unix specific, and
Python programmers can now program this functionality is a safer way
using the signal module.
Setting interp->builtins to the __builtin__ module instead of to its
dictionary had the unfortunate side effect of always running in
restricted execution mode :-(
I will check in a different way of setting __main__.__builtins__ to
the __builtin__ module later.
Also, there was a typo -- a comment was unfinished, and as a result
some finalizations were not being executed.
In Bart Simpson style,
I Will Not Check In Untested Changes.
I Will Not Check In Untested Changes.
I Will Not Check In Untested Changes.
I Will Not Check In Untested Changes.
I Will Not Check In Untested Changes.
I Will Not Check In Untested Changes.
I Will Not Check In Untested Changes.
I Will Not Check In Untested Changes.
I Will Not Check In Untested Changes.
I Will Not Check In Untested Changes.
- The interp->builtins variable (and hence, __main__.__builtins__) is
once again initialized to the built-in *module* instead of its
dictionary.
- The finalization order is once again changed. Signals are finalized
relatively early, because (1) it DECREF's the signal handlers, and if
a signal handler happens to be a bound method, deleting it could cause
problems when there's no current thread around, and (2) we don't want
to risk executing signal handlers during finalization.
__init__.py (or __init__.pyc/.pyo, whichever applies) is considered a
package. All other subdirectories are left alone. Should make Konrad
Hinsen happy!
tstate swapping. Only the acquiring and releasing of the lock is
conditional (twice, under ``#ifdef WITH_THREAD'' and inside ``if
(interpreter_lock)'').
but annoying memory leak. This was introduced when PyExc_Exception
was added; the loop above populating the PyExc_StandardError exception
tuple started at index 1 in bltin_exc, but PyExc_Exception was added
at index 0, so PyExc_StandardError was getting inserted in itself!
How else can a tuple include itself?!
Change the loop to start at index 2.
This was a *fun* one! :-)
Guido van Rossum
Barry Warsaw
Andrew M. Kuchling
Fred Drake