currently return inconsistent results for ints and longs; in
particular: hex/oct/%u/%o/%x/%X of negative short ints, and x<<n that
either loses bits or changes sign. (No warnings for repr() of a long,
though that will also change to lose the trailing 'L' eventually.)
This introduces some warnings in the test suite; I'll take care of
those later.
Change the parser and compiler to use PyMalloc.
Only the files implementing processes that will request memory
allocations small enough for PyMalloc to be a win have been
changed, which are:-
- Python/compile.c
- Parser/acceler.c
- Parser/node.c
- Parser/parsetok.c
This augments the aggressive overallocation strategy implemented by
Tim Peters in PyNode_AddChild() [Parser/node.c], in reducing the
impact of platform malloc()/realloc()/free() corner case behaviour.
Such corner cases are known to be triggered by test_longexp and
test_import.
Jeremy Hylton, in accepting this patch, recommended this as a
bugfix candidate for 2.2. While the changes to Python/compile.c
and Parser/node.c backport easily (and could go in), the changes
to Parser/acceler.c and Parser/parsetok.c require other not
insignificant changes as a result of the differences in the memory
APIs between 2.3 and 2.2, which I'm not in a position to work
through at the moment. This is a pity, as the Parser/parsetok.c
changes are the most important after the Parser/node.c changes, due
to the size of the memory requests involved and their frequency.
actual script to run in case we are running from an applet. If we are indeed
running an applet we skip the normal option processing leaving it all to the
applet code.
This allows us to get use the normal python binary in the Python.app bundle,
giving us all the normal command line options through PythonLauncher while
still allowing Python.app to be used as the template for building applets.
Consequently, pythonforbundle is gone, and Mac/Python/macmain.c isn't used
on OSX anymore.
See there for a description.
Added test case.
Bugfix candidate for 2.2.x, not sure about previous versions:
probably low priority, because virtually no one runs debug builds.
PyErr_SetExcFromWindowsErr(), PyErr_SetExcFromWindowsErrWithFilename().
Similar to PyErr_SetFromWindowsErrWithFilename() and
PyErr_SetFromWindowsErr(), but they allow to specify
the exception type to raise. Available on Windows.
See SF patch #576458.
is slow things down unnecessarily and make tracing much more verbose.
Something like
def f(n):
return [i for i in range(n) if i%2]
should have at most two SET_LINENO instructions, not four. When tracing,
the current line number should be printed once, not 2*n+1 times.
The staticforward define was needed to support certain broken C
compilers (notably SCO ODT 3.0, perhaps early AIX as well) botched the
static keyword when it was used with a forward declaration of a static
initialized structure. Standard C allows the forward declaration with
static, and we've decided to stop catering to broken C compilers. (In
fact, we expect that the compilers are all fixed eight years later.)
I'm leaving staticforward and statichere defined in object.h as
static. This is only for backwards compatibility with C extensions
that might still use it.
XXX I haven't updated the documentation.
more trivial lexical helper macros so that uses of these guys expand
to nothing at all when they're not enabled. This should help sub-
standard compilers that can't do a good job of optimizing away the
previous "(void)0" expressions.
Py_DECREF: There's only one definition of this now. Yay! That
was that last one in the family defined multiple times in an #ifdef
maze.
Py_FatalError(): Changed the char* signature to const char*.
_Py_NegativeRefcount(): New helper function for the Py_REF_DEBUG
expansion of Py_DECREF. Calling an external function cuts down on
the volume of generated code. The previous inline expansion of abort()
didn't work as intended on Windows (the program often kept going, and
the error msg scrolled off the screen unseen). _Py_NegativeRefcount
calls Py_FatalError instead, which captures our best knowledge of
how to abort effectively across platforms.
that have taken me "too long" to reverse-engineer over the years.
Vastly reduced the nesting level and redundancy of #ifdef-ery.
Took a light stab at repairing comments that are no longer true.
sys_gettotalrefcount(): Changed to enable under Py_REF_DEBUG.
It was enabled under Py_TRACE_REFS, which was much heavier than
necessary. sys.gettotalrefcount() is now available in a
Py_REF_DEBUG-only build.
This patch enhances Python/import.c/find_module() so
that unicode objects found in sys.path will be treated
as legal directory names (The current code ignores
anything that is not a str). The unicode name is
converted to str using Py_FileSystemDefaultEncoding.
These built-in functions are replaced by their (now callable) type:
slice()
buffer()
and these types can also be called (but have no built-in named
function named after them)
classobj (type name used to be "class")
code
function
instance
instancemethod (type name used to be "instance method")
The module "new" has been replaced with a small backward compatibility
placeholder in Python.
A large portion of the patch simply removes the new module from
various platform-specific build recipes. The following binary Mac
project files still have references to it:
Mac/Build/PythonCore.mcp
Mac/Build/PythonStandSmall.mcp
Mac/Build/PythonStandalone.mcp
[I've tweaked the code layout and the doc strings here and there, and
added a comment to types.py about StringTypes vs. basestring. --Guido]
Write 4 bytes for co_stacksize, etc. to prevent writing out
bad .pyc files which can cause a crash when read back in.
(I forgot that frozen needs to be updated too for the test.)
This was a simple typo. Strange that the compiler didn't catch it!
Instead of WHY_CONTINUE, two tests used CONTINUE_LOOP, which isn't a
why_code at all, but an opcode; but even though 'why' is declared as
an enum, comparing it to an int is apparently not even worth a
warning -- not in gcc, and not in VC++. :-(
Will fix in 2.2 too.
get_file() must convert 'U' to "r" PY_STDIOTEXTMODE before calling
fopen().
imp_load_module() must accept 'r' or 'U' or something with '+'.
Also reflow some long lines.
for 'str' and 'unicode', and can be used instead of
types.StringTypes, e.g. to test whether something is "a string":
isinstance(x, string) is True for Unicode and 8-bit strings. This
is an abstract base class and cannot be instantiated directly.
The old syntax suggested that a trailing comma was OK inside backticks,
but in fact (due to ideosyncrasies of pgen) it was not. Fix the grammar
to avoid the ambiguity. Fred: you may want to update the refman.
[ 558249 ] softspace vs --disable-unicode
And #endif was in the wrong place.
Bugfix candidate, almost surely.
I think I will embark on squashing test failures in --disable-unicode builds --
a Real Bug was hiding under them.
don't understand how this function works, also beefed up the docs. The
most common usage error is of this form (often spread out across gotos):
if (_PyString_Resize(&s, n) < 0) {
Py_DECREF(s);
s = NULL;
goto outtahere;
}
The error is that if _PyString_Resize runs out of memory, it automatically
decrefs the input string object s (which also deallocates it, since its
refcount must be 1 upon entry), and sets s to NULL. So if the "if"
branch ever triggers, it's an error to call Py_DECREF(s): s is already
NULL! A correct way to write the above is the simpler (and intended)
if (_PyString_Resize(&s, n) < 0)
goto outtahere;
Bugfix candidate.
The SIGXFSZ signal is sent when the maximum file size limit is
exceeded (RLIMIT_FSIZE). Apparently, it is also sent when the 2GB
file limit is reached on platforms without large file support.
The default action for SIGXFSZ is to terminate the process and dump
core. When it is ignored, the system call that caused the limit to be
exceeded returns an error and sets errno to EFBIG. Python
always checks errno on I/O syscalls, so there is nothing to do with
the signal.
+ Redirect PyMem_{Del, DEL} to the object allocator's free() when
pymalloc is enabled. Needed so old extensions can continue to
mix PyObject_New with PyMem_DEL.
+ This implies that pgen needs to be able to see the PyObject_XYZ
declarations too. pgenheaders.h now includes Python.h. An
implication is that I expect obmalloc.o needs to get linked into
pgen on non-Windows boxes.
+ When PYMALLOC_DEBUG is defined, *all* Py memory API functions
now funnel through the debug allocator wrapper around pymalloc.
This is the default in a debug build.
+ That caused compile.c to fail: it indirectly mixed PyMem_Malloc
with raw platform free() in one place. This is verbotten.
Highlights: import and friends will understand any of \r, \n and \r\n
as end of line. Python file input will do the same if you use mode 'U'.
Everything can be disabled by configuring with --without-universal-newlines.
See PEP278 for details.
Added code to call this when PYMALLOC_DEBUG is enabled, and envar
PYTHONMALLOCSTATS is set, whenever a new arena is obtained and once
late in the Python shutdown process.
PEP 285. Everything described in the PEP is here, and there is even
some documentation. I had to fix 12 unit tests; all but one of these
were printing Boolean outcomes that changed from 0/1 to False/True.
(The exception is test_unicode.py, which did a type(x) == type(y)
style comparison. I could've fixed that with a single line using
issubtype(x, type(y)), but instead chose to be explicit about those
places where a bool is expected.
Still to do: perhaps more documentation; change standard library
modules to return False/True from predicates.
SF bug 535905 (Evil Trashcan and GC interaction).
The SETLOCAL() macro should not DECREF the local variable in-place and
then store the new value; it should copy the old value to a temporary
value, then store the new value, and then DECREF the temporary value.
This is because it is possible that during the DECREF the frame is
accessed by other code (e.g. a __del__ method or gc.collect()) and the
variable would be pointing to already-freed memory.
BUGFIX CANDIDATE!
descriptor, as used for the tp_methods slot of a type. These new flag
bits are both optional, and mutually exclusive. Most methods will not
use either. These flags are used to create special method types which
exist in the same namespace as normal methods without having to use
tedious construction code to insert the new special method objects in
the type's tp_dict after PyType_Ready() has been called.
If METH_CLASS is specified, the method will represent a class method
like that returned by the classmethod() built-in.
If METH_STATIC is specified, the method will represent a static method
like that returned by the staticmethod() built-in.
These flags may not be used in the PyMethodDef table for modules since
these special method types are not meaningful in that case; a
ValueError will be raised if these flags are found in that context.
This fixes the symptom, but PRINT_ITEM has no way to know what (if
anything) PyFile_WriteObject() writes unless the object being printed
is a string. When the object isn't a string, this fix retains the
guess that softspace should be set after PyFile_WriteObject().
We might want to say that it's the job of filelike-object write methods
to leave the file's softspace in the correct state. That would probably
be better -- but everyone relies on PRINT_ITEM to guess for them now.
A file-static "threads" dict mapped thread IDs to Windows handles, but
was never referenced, and entries never got removed. This gets rid of
the YAGNI-dict entirely.
Bugfix candidate.
Python/
dynload_shlib.c // EMX port emulates dlopen() etc. for DL extensions
import.c // changes to support 8.3 DLL name limit (VACPP+EMX)
// and case sensitive import semantics
importdl.h
thread_os2.h
Fix for the UTF-8 decoder: it will now accept isolated surrogates
(previously it raised an exception which causes round-trips to
fail).
Added new tests for UTF-8 round-trip safety (we rely on UTF-8 for
marshalling Unicode objects, so we better make sure it works for
all Unicode code points, including isolated surrogates).
Bumped the PYC magic in a non-standard way -- please review. This
was needed because the old PYC format used illegal UTF-8 sequences
for isolated high surrogates which now raise an exception.
By default every module is imported in its own namespace, but this can
be changed by defining USE_DYLD_GLOBAL_NAMESPACE. In a future version this
define will be replaced by a runtime setting, but that needs a bit more
thought.
This code is largely based on code and feedback from Steven Majewski,
Marcel Prastawa, Manoj Plakal and other on pythonmac-sig.
type.__module__ problems (again?)
This simply initializes the __module__ local in a class statement from
the __name__ global. I'm not 100% sure that this is the correct fix,
although it usually does the right thing. The problem is that if the
class statement executes in a custom namespace, the __name__ global
may be taken from __builtins__, in which case it would have the value
__builtin__, or it may not exist at all (if the custom namespace also
has a custom __builtins__), in which case the class statement will
fail.
Nevertheless, unless someone finds a better solution, this is a 2.2.1
bugfix too.
eval_frame(): Under -Qnew, INPLACE_DIVIDE wasn't getting handed off to
INPLACE_TRUE_DIVIDE (like BINARY_DIVIDE was getting handed off to
BINARY_TRUE_DIVIDE).
Bugfix candidate.
(ditto for PyMem_Free() -> PyMem_FREE()) to fix and close SF bug
#495875 on systems that HAVE_SNPRINTF=0.
Check in on both release-22 branch and trunk.
Based on the patch from Danny Yoo. The fix is in exec_statement() in
ceval.c.
There are also changes to introduce use of PyCode_GetNumFree() in
several places.
was obviously leaking an int object when whatever the heck it's looking for
was found. Repaired that. This accounts for why entering function and
class definitions at an interactive prompt leaked a reference to the
integer 1 each time.
Bugfix candidate.
still fail on importing modules that link with libraries that fail
their initialization code (such as windowing libraries when we don't have
access to the window server) and that is what I really wanted to fix.
Had nothing to do with rich comparisons -- some stack cleanup code was
lost as a result of merging in Neil Schemenauer's generators patch.
Reinserted the stack cleanup code, skipping it when yielding.
leak when a class defined a __metaclass__. This fixes the problem
reported on python-dev by Ping; I dunno if it's the same as SF bug
#489669 (since that mentions Unicode).
Big Hammer to implement -Qnew as PEP 238 says it should work (a global
option affecting all instances of "/").
pydebug.h, main.c, pythonrun.c: define a private _Py_QnewFlag flag, true
iff -Qnew is passed on the command line. This should go away (as the
comments say) when true division becomes The Rule. This is
deliberately not exposed to runtime inspection or modification: it's
a one-way one-shot switch to pretend you're using Python 3.
ceval.c: when _Py_QnewFlag is set, treat BINARY_DIVIDE as
BINARY_TRUE_DIVIDE.
test_{descr, generators, zipfile}.py: fiddle so these pass under
-Qnew too. This was just a matter of s!/!//! in test_generators and
test_zipfile. test_descr was trickier, as testbinop() is passed
assumptions that "/" is the same as calling a "__div__" method; put
a temporary hack there to call "__truediv__" instead when the method
name is "__div__" and 1/2 evaluates to 0.5.
Three standard tests still fail under -Qnew (on Windows; somebody
please try the Linux tests with -Qnew too! Linux runs a whole bunch
of tests Windows doesn't):
test_augassign
test_class
test_coercion
I can't stay awake longer to stare at this (be my guest). Offhand
cures weren't obvious, nor was it even obvious that cures are possible
without major hackery.
Question: when -Qnew is in effect, should calls to __div__ magically
change into calls to __truediv__? See "major hackery" at tail end of
last paragraph <wink>.
There's now a new structmember code, T_OBJECT_EX, which is used for
all __slot__ variables (except __weakref__, which has special behavior
anyway). This new code raises AttributeError when the variable is
NULL rather than converting NULL to None.
The error for assignment to __debug__ used ste->ste_opt_lineno instead
of n->n_lineno. The latter was at best incorrect; often the slot was
uninitialized. Two fixes here: Use the correct lineno for the error.
Initialize ste_opt_lineno in PySymtable_New(); while there are no
current cases where it is referenced unless it has already been
assigned to, there is no harm in initializing it.
use wrappers on all platforms, to make this as consistent as possible x-
platform (in particular, make sure there's at least one \0 byte in
the output buffer). Also document more of the truth about what these do.
getargs.c, seterror(): Three computations of remaining buffer size were
backwards, thus telling PyOS_snprintf the buffer is larger than it
actually is. This matters a lot now that PyOS_snprintf ensures there's a
trailing \0 byte (because it didn't get the truth about the buffer size,
it was storing \0 beyond the true end of the buffer).
sysmodule.c, mywrite(): Simplify, now that PyOS_vsnprintf guarantees to
produce a \0 byte.
than the argument string size, copy as many bytes as will fit
(including a terminating '\0'), rather than not copying anything.
This to make it satisfy the C99 spec.
vgetargskeywords(): Now that this routine is checking for bad input
(rather than dump core in some cases), some bad calls are raising errors
that previously "worked". This patch makes the error strings more
revealing, and changes the exceptions from SystemError to RuntimeError
(under the theory that SystemError is more of a "can't happen!" assert-
like thing, and so inappropriate for bad arguments to a public C API
function).
seterror() uses a char array and a pointer to the current position in
that array. Use snprintf() and compute the amount of space left in
the buffer based on the current pointer position.
If it returns -1 (which indicates overflow on old Linux platforms and
perhaps on Windows) or size greater than buffer, write a message
indicating that the previous message was truncated.
com_arglist(), symtable_check_unoptimized(), symtable_params(),
symtable_global(), symtable_list_comprehension():
Conversion of sprintf() to PyOS_snprintf() for buffer overrun
avoidance.
PyEval_EvalCodeEx(): increment tstate->recursion_depth around the
decref of the frame, because the C stack for this call is still in
use and the decref can lead to __del__ methods getting called.
While this gives tstate->recursion_depth a value proportional to the
depth of the C stack (instead of a small constant no matter how
deeply __del__s recurse), it's not enough to stop the reported crash
when using the default recursion limit on Windows.
Bugfix candidate.
Bugfix candidate.
tb_displayline(): the sprintf format was choking off the file name, but
used plain %s for the function name (which can be arbitrarily long).
Limit both to 500 chars max.
uninitialized memory reads reported in bug #478001.
Note that this doesn't address the following larger issues:
- Error conditions are not documented for PyOS_*sig() in the C API.
- Nothing that actually calls PyOS_*sig() in the core interpreter and
extension modules actually /checks/ the return value of the call.
Fixing those is left as an exercise for a later day.
This patch boosts performance for comparing identical string object
by some 20% on my machine while not causing any noticable slow-down
for other operations (according to tests done with pybench).
routines. As of 10.1 using Carbon will crash Python if no window server is
available (ssh connection, console mode, MacOSX Server). This fixes bug
#466907.
A result of this mod is that the default 8bit encoding on OSX is now ASCII,
for the time being. Also, the extension modules that need the Carbon
framework now explicitly include it in setup.py.
+ Squash another potential buffer overrun.
+ Simplify the keyword-arg loop by decrementing the count of keywords
remaining instead of incrementing Yet Another Variable; also break
out early if the number of keyword args remaining hits 0.
Since I hit the function's closing curly brace with this patch, that's
enough of this for now <wink>.
The "need" for this was probably removed by an earlier patch that stopped
the loop right before it from passing NULL to a dict lookup routine.
I still haven't convinced myself that the next loop is correct, so am
leaving the next mysterious PyErr_Clear() call in for now.
+ Generally test nkeywords against 0 instead of keywords against NULL
(saves a little work if an empty keywords dict is passed, and is
conceptually more on-target regardless).
+ When a call erroneously specifies a keyword argument both by position
and by keyword name:
- It was easy to provoke this routine into an internal buffer overrun
by using a long argument name. Now uses PyErr_format instead (which
computes a safe buffer size).
- Improved the error msg.
+ Got rid of now-redundant dict typecheck.
+ Renamed nkwds to nkwlist. Now all the "counting" vrbls have names
related to the things they're counting in an obvious way.
+ Renamed argslen to nargs.
+ Renamed kwlen to nkeywords. This one was especially confusing because
kwlen wasn't the length of the kwlist argument, but of the keywords
argument.
+ Removed now-redundant tuple typecheck.
+ Renamed "tplen" local to "argslen" (it's the length of the "args"
argument; I suppose "tp" was for "Tim Peters should rename me
someday <wink>).
introduced this bug just a little while ago, when *adding* internal error
checks).
vgetargskeywords: Rewrote the section that crawls over the format string.
+ Added block comment so it won't take the next person 15 minutes to
reverse-engineer what it's doing.
+ Lined up the "else" clauses.
+ Rearranged the ifs in decreasing order of likelihood (for speed).
and raise an error if they're insane.
vgetargskeywords: the same, except that since this is an internal routine,
just assert that the arguments are sane.
the kwlist vector whenever there was a mix of positional and keyword
arguments, and the number of positional arguments exceeded the length
of the kwlist vector. If there was just one more positional arg than
keyword, the kwlist-terminating NULL got passed to PyMapping_HasKeyString,
which set an internal error that vgetargskeywords() then squashed (but
it's impossible to say whether it knew it was masking an error). If
more than one more positional argument, it went on to pass random trash
to PyMapping_HasKeyString, which is why the example at the start
happened to kill the process.
Pure bugfix candidate.
to call the corresponding methods. This is not a performance improvement
since the times are still swamped by disk I/O, but cleans up the code just
a little.
In Include/, marshal.h declares both
PyMarshal_ReadLongFromFile()
and PyMarshal_ReadShortFromFile(),
but the second is missing from marshal.c.
[Shouldn't the return type be declared as 'short' instead of 'int'?
But 'int' is what was in marshal.h all those years... --Guido]
This fixes the behavior reported by SF bug #404545, where a file
x.y.py could be imported by the statement "import x.y" when there's a
frozen package x (I believe even if x.y also exists as a frozen
module).
:-).
Add a test that prevents the __hello__ bytecode from going stale
unnoticed again.
The test also tests the loophole noted in SF bug #404545. This test
will fail right now; I'll check in the fix in a minute.
The symbol table pass didn't have an explicit case for the list_iter
node which is used only for a nested list comprehension. As a result,
the target of the list comprehension was treated as a use instead of
an assignment. Fix is to add a case to symtable_node() to handle
list_iter.
Also, rework and document a couple of the subtler implementation
issues in the symbol table pass. The symtable_node() switch statement
depends on falling through the last several cases, in order to handle
some of the more complicated nodes like atom. Add a comment
explaining the behavior before the first fall through case. Add a
comment /* fall through */ at the end of case so that it is explicitly
marked as such.
Move the for_stmt case out of the fall through logic, which simplifies
both for_stmt and default. (The default used the local variable start
to skip the first three nodes of a for_stmt when it fell through.)
Rename the flag argument to symtable_assign() to def_flag and add a
comment explaining its use:
The third argument to symatble_assign() is a flag to be passed to
symtable_add_def() if it is eventually called. The flag is useful
to specify the particular type of assignment that should be
recorded, e.g. an assignment caused by import.
Also minor tweaks to internal routines.
Use PyCF_MASK instead of explicit list of flags.
For the MAKE_CLOSURE opcode, the number of items popped off the stack
depends on both the oparg and the number of free variables for the
code object. Fix the code so it accounts for the free variables.
In com_classdef(), record an extra pop to account for the STORE call
after the BUILD_CLASS.
Get rid of some commented out debugging code in com_push() and
com_pop().
Factor string resize logic into helper routine com_check_size().
In com_addbyte(), remove redudant if statement after assert. (They
test the same condition.)
In several routines, use string macros instead of string functions.
Guido van Rossum
Fred Drake
Martin v. Löwis
Skip Montanaro
Andrew MacIntyre
Jack Jansen
Thomas Heller
Mark Hammond
Tim Peters
Jeremy Hylton
Neal Norwitz
Raymond Hettinger
Walter Dörwald
Michael W. Hudson
Neil Schemenauer
Marc-André Lemburg
Barry Warsaw