For more comments, read the patches@python.org archives.
For documentation read the comments in mymalloc.h and objimpl.h.
(This is not exactly what Vladimir posted to the patches list; I've
made a few changes, and Vladimir sent me a fix in private email for a
problem that only occurs in debug mode. I'm also holding back on his
change to main.c, which seems unnecessary to me.)
"""
Running "test_extcall" repeatedly results in memory leaks.
One of these can't be fixed (at least not easily!), it happens since
this code:
def saboteur(**kw):
kw['x'] = locals()
d = {}
saboteur(a=1, **d)
creates a circular reference - d['x']['d']==d
The others are due to some missing decrefs in ceval.c, fixed by the
patch attached below.
Note: I originally wrote this without the "goto", just adding the
missing decref's where needed. But I think the goto is justified in
keeping the executable code size of ceval as small as possible.
"""
[I think the circular reference is more like kw['x']['kw'] == kw. --GvR]
If a non-tuple sequence is passed as the *arg, convert it to a tuple
before checking its length.
If named keyword arguments are used in combination with **kwargs, make
a copy of kwargs before inserting the new keys.
the return value of PySequence_Length(). If an exception occurred,
the returned length will be -1. Make sure this doesn't get obscurred,
and that the bogus length isn't used.
executive summary:
Instead of typing 'apply(f, args, kwargs)' you can type 'f(*arg, **kwargs)'.
Some file-by-file details follow.
Grammar/Grammar:
simplify varargslist, replacing '*' '*' with '**'
add * & ** options to arglist
Include/opcode.h & Lib/dis.py:
define three new opcodes
CALL_FUNCTION_VAR
CALL_FUNCTION_KW
CALL_FUNCTION_VAR_KW
Python/ceval.c:
extend TypeError "keyword parameter redefined" message to include
the name of the offending keyword
reindent CALL_FUNCTION using four spaces
add handling of sequences and dictionaries using extend calls
fix function import_from to use PyErr_Format
PR#175 -- when exec is passed a code object, it didn't sync the locals
from the dictionary back into their fast representation.
Also took the time to remove some repetitive code there and to do the
syncing even when an exception is raised (since a partial effect
should still be synced).
not as descriptive as what Barry suggests, but this also catches the
(in my opinion important) case where some other C code besides apply()
constructs a kwdict that doesn't have the right format. All the other
possibilities of getting it wrong (non-dict, wrong keywords etc) are
already caught so this makes sense to check here.
tracefunc (or profilefunc -- we're not sure which), zap the global
trace and profile funcs so that we can't get into recursive loop when
instantiating the resulting class based exception.
Introduce a new builtin exception, UnboundLocalError, raised when ceval.c
tries to retrieve or delete a local name that isn't bound to a value.
Currently raises NameError, which makes this behavior a FAQ since the same
error is raised for "missing" global names too: when the user has a global
of the same name as the unbound local, NameError makes no sense to them.
Even in the absence of shadowing, knowing whether a bogus name is local or
global is a real aid to quick understanding.
Example:
D:\src\PCbuild>type local.py
x = 42
def f():
print x
x = 13
return x
f()
D:\src\PCbuild>python local.py
Traceback (innermost last):
File "local.py", line 8, in ?
f()
File "local.py", line 4, in f
print x
UnboundLocalError: x
D:\src\PCbuild>
Note that UnboundLocalError is a subclass of NameError, for compatibility
with existing class-exception code that may be trying to catch this as a
NameError. Unfortunately, I see no way to make this wholly compatible
with -X (see comments in bltinmodule.c): under -X, [UnboundLocalError
is an alias for NameError --GvR].
[The ceval.c patch differs slightly from the second version that Tim
submitted; I decided not to raise UnboundLocalError for DELETE_NAME,
only for DELETE_LOCAL. DELETE_NAME is only generated at the module
level, and since at that level a NameError is raised for referencing
an undefined name, it should also be raised for deleting one.]
swapped arguments].
Also make sure that no use of a function pointer gotten from a
tp_as_sequence or tp_as_mapping structure is made without checking it
for NULL first.
tstate swapping. Only the acquiring and releasing of the lock is
conditional (twice, under ``#ifdef WITH_THREAD'' and inside ``if
(interpreter_lock)'').
This doesn't yet support "import a.b.c" or "from a.b.c import x", but
it does recognize directories. When importing a directory, it
initializes __path__ to a list containing the directory name, and
loads the __init__ module if found.
The (internal) find_module() and load_module() functions are
restructured so that they both also handle built-in and frozen modules
and Mac resources (and directories of course). The imp module's
find_module() and (new) load_module() also have this functionality.
Moreover, imp unconditionally defines constants for all module types,
and has two more new functions: find_module_in_package() and
find_module_in_directory().
There's also a new API function, PyImport_ImportModuleEx(), which
takes all four __import__ arguments (name, globals, locals, fromlist).
The last three may be NULL. This is currently the same as
PyImport_ImportModule() but in the future it will be able to do
relative dotted-path imports.
Other changes:
- bltinmodule.c: in __import__, call PyImport_ImportModuleEx().
- ceval.c: always pass the fromlist to __import__, even if it is a C
function, so PyImport_ImportModuleEx() is useful.
- getmtime.c: the function has a second argument, the FILE*, on which
it applies fstat(). According to Sjoerd this is much faster. The
first (pathname) argument is ignored, but remains for backward
compatibility (so the Mac version still works without changes).
By cleverly combining the new imp functionality, the full support for
dotted names in Python (mini.py, not checked in) is now about 7K,
lavishly commented (vs. 14K for ni plus 11K for ihooks, also lavishly
commented).
Good night!
former rather than the latter, since PyErr_NormalizeException takes
PyObject** and I didn't want to change the interface for set_exc_info
(but I did want the changes propagated to eval_code2!).
UNPACK_LIST byte codes and added a third code path that allows
generalized sequence unpacking. Now both syntaxes:
a, b, c = seq
[a, b, c] = seq
can be used to unpack any sequence with the exact right number of
items.
unpack_sequence(): out-lined implementation of generalized sequence
unpacking. tuple and list unpacking are still inlined.
PyErr_GivenExceptionMatches().
set_exc_info(): make sure to normalize exceptions.
do_raise(): Use PyErr_NormalizeException() if type is a class.
loop_subscript(): Use PyErr_ExceptionMatches() instead of raw pointer
compare for PyExc_IndexError.
PyThreadState pointer instead of a (frame) PyObject pointer. This
makes much more sense. It is backward incompatible, but that's no
problem, because (a) the heaviest users are the Py_{BEGIN,END}_
ALLOW_THREADS macros here, which have been fixed too; (b) there are
very few direct users; (c) those who use it are there will probably
appreciate the change.
Also, added new functions PyEval_AcquireThread() and
PyEval_ReleaseThread() which allows the threads created by the thread
module as well threads created by others (!) to set/reset the current
thread, and at the same time acquire/release the interpreter lock.
Much saner.
int+int, int-int, int <compareop> int, and list[int].
(Unfortunately, int*int is way too much code to inline.)
Also corrected a NULL that should have been a zero.
get/set/del item). This removes a pile of duplication. There's no
abstract operator for 'not' but I removed the function call for it
anyway -- it's a little faster in-line.
by the frameobject dealloc when it is time for the locals to go. When
there's still a traceback object referencing this stack frame, we
don't want the local variables to disappear yet.
(Hmm... Shouldn't they be copied to the f_locals dictionary?)
- Use co->... instead of f->f_code->...; save an extra lookup of what
we already have in a local variable).
- Remove test for nlocals > 0 before setting fastlocals to
f->f_localsplus; 0 is a rare case and the assignment is safe even
then.
called with keyword arguments -- the keyword and value were leaked.
This affected for instance with a __call__() method.
Bug reported and fix supplied by Jim Fulton.
i.e., counting opcode frequencies, or (with DXPAIRS defined) opcode
pair frequencies. Define DYNAMIC_EXECUTION_PROFILE on the command
line (for this file and for sysmodule.c) to enable.
This is safe now that both intrcheck() and signalmodule.c schedule a
sigcheck() call via Py_AddPendingCall().
This gives another 7% speedup (never run such a test twice ;-).
The raise logic has one additional feature: if you raise <class>,
<value> where <value> is not an instance, it will construct an
instance using <value> as argument. If <value> is None, <class> is
instantiated without arguments. If <value> is a tuple, it is used as
the argument list.
This feature is intended to make it easier to upgrade code from using
string exceptions to using class exceptions; without this feature,
you'd have to change every raise statement from ``raise X'' to ``raise
X()'' and from ``raise X, y'' to ``raise X(y)''. The latter is still
the recommended form (because it has no ambiguities about the number
of arguments), but this change makes the transition less painful.
bltinmodule.c: fixed coerce() nightmare in ternary pow().
modsupport.c (initmodule2): pass METH_FREENAME flag to newmethodobject().
pythonrun.c: move flushline() into and around print_error().
global: Py_MakePendingCalls. Also guard against recursive calls
* Include/classobject.h, Objects/classobject.c,
Python/{ceval.c,bltinmodule.c}: entirely redone operator
overloading. The rules for class instances are now much more
relaxed than for other built-in types
(whose coerce must still return two objects of the same type)
* traceback.c (tb_print): use sys.tracebacklimit as a maximum number of
traceback entries to print (default 1000).
* ceval.c (printtraceback): Don't print stack trace header -- this is now
done by tb_print().
* object.[ch], bltinmodule.c, fileobject.c: changed str() to call
strobject() which calls an object's __str__ method if it has one.
strobject() is also called by writeobject() when PRINT_RAW is passed.
* ceval.c: rationalize code for PRINT_ITEM (no change in function!)
* funcobject.c, codeobject.c: added compare and hash functionality.
Functions with identical code objects and the same global dictionary are
equal. Code objects are equal when their code, constants list and names
list are identical (i.e. the filename and code name don't count).
(hash doesn't work yet since the constants are in a list and lists can't
be hashed -- suppose this should really be done with a tuple now we have
resizetuple!)
* Grammar: add exec statement; allow testlist in expr statement.
* ceval.c, compile.c, opcode.h: support exec statement;
avoid optimizing locals when it is used
* fileobject.{c,h}: add getfilename() internal function.
* many files: made some functions static; removed "extern int errno;".
* frozenmain.c: fixed bugs introduced on 24 June...
* flmodule.c: remove 1.5 bw compat hacks, add new functions in 2.2a
(and some old functions that were omitted).
* timemodule.c: added MSDOS floatsleep version .
* pgenmain.c: changed exit() to goaway() and added defn of goaway().
* intrcheck.c: add hack (to UNIX only) so interrupting 3 times
will exit from a hanging program. The second interrupt prints
a message explaining this to the user.
(the latter also in frozenmain.c)
* ceval.c: global 'killprint' flag raises exception when printing an
expression statement's value (useful for finding stray output)
* timemodule.c: add asctime() and ctime(). Change julian date to
1-based origin (as intended and documented).
* Removed unused DO_TIMES stuff from timemodule.c. Added 'epoch' and
'day0' globals (year where time.time() == 0 and day of the week the
epoch started).
Added $(SYSDEF) to its build rule in Makefile.
* cgensupport.[ch], modsupport.[ch]: removed some old stuff. Also
changed files that still used it... And made several things static
that weren't but should have been... And other minor cleanups...
* listobject.[ch]: add external interfaces {set,get}listslice
* socketmodule.c: fix bugs in new send() argument parsing.
* sunaudiodevmodule.c: added flush() and close().
function found as instance data.
* socketmodule.c: added 'flags' argument sendto/recvfrom, rewrite
argument parsing in send/recv.
* More changes related to access (terminology change: owner instead of
class; allow any object as owner; local/global variables are owned
by their dictionary, only class/instance data is owned by the class;
"from...import *" now only imports objects with public access; etc.)
* Added "access *: ...", made access work for class methods.
* Introduced subclass check: make sure that when calling
ClassName.methodname(instance, ...), the instance is an instance of
ClassName or of a subclass thereof (this might break some old code!)
yet). The class is now passed to eval_code and stored in the current
frame. It is also stored in instance method objects. An "unbound"
instance method is now returned when a function is retrieved through
"classname.funcname", which when called passes the class to eval_code.
(1) dictionaries/mappings now have attributes values() and items() as
well as keys(); at the C level, use the new function mappinggetnext()
to iterate over a dictionary.
(2) "class C(): ..." is now illegal; you must write "class C: ...".
(3) Class objects now know their own name (finally!); and minor
improvements to the way how classes, functions and methods are
represented as strings.
(4) Added an "access" statement and semantics. (This is still
experimental -- as long as you don't use the keyword 'access' nothing
should be changed.)
before it.
* ceval.c, object.c: moved testbool() to object.c (now extern visible)
* stringobject.c: fix bugs in and rationalize string resize in formatstring()
* tokenizer.[ch]: fix non-working code for lines longer than BUFSIZ
f_fastlocals in a traceback object (this is a core dump hazard
if there are <nil> entries), but instead eval_code() merges the fast
locals back into the locals dictionary if it looks like the local
variables will be retained. Also, the merge routines save
exceptions since this is sometimes needed (alas!).
* Added id() to bltinmodule.c, which returns an object's address
(identity). Useful to walk arbitrary data structures containing
cycles.
* Added compile() to bltinmodule.c and compile_string() to
pythonrun.[ch]: support to exec/eval arbitrary code objects. The
code that defaults globals and locals is moved from run_node in
pythonrun.c (which is now identical to eval_node) to eval_code in
ceval.c. [XXX For elegance a clean-up session is necessary.]
lookup (opcode.h, ceval.[ch], compile.c, frameobject.[ch],
pythonrun.c, import.c). The .pyc MAGIC number is changed again.
Added get_menu_text to flmodule.
* Stubs for faster implementation of local variables (not yet finished)
* Added function name to code object. Print it for code and function
objects. THIS MAKES THE .PYC FILE FORMAT INCOMPATIBLE (the version
number has changed accordingly)
* Print address of self for built-in methods
* New internal functions getattro and setattro (getattr/setattr with
string object arg)
* Replaced "dictobject" with more powerful "mappingobject"
* New per-type functio tp_hash to implement arbitrary object hashing,
and hashobject() to interface to it
* Added built-in functions hash(v) and hasattr(v, 'name')
* classobject: made some functions static that accidentally weren't;
added __hash__ special instance method to implement hash()
* Added proper comparison for built-in methods and functions
* Fixcprt.py: added [-y file] option, do only files younger than file.
* modsupport.[ch]: added vmkvalue().
* intobject.c: use mkvalue().
* stringobject.c: added "formatstring"; renamed string* to string_*;
ceval.c: call formatstring for string % value.
* longobject.c: close memory leak in divmod.
* parsetok.c: set result node to NULL when returning an error.
* various modules: added 1993 to copyright.
* thread.c: added copyright notice.
* ceval.c: minor change to error message for "+"
* stdwinmodule.c: check for error from wfetchcolor
* config.c: MS-DOS fixes (define PYTHONPATH, use DELIM, use osdefs.h)
* Add declaration of inittab to import.h
* sysmodule.c: added sys.builtin_module_names
* xxmodule.c, xxobject.c: fix minor errors
* ceval.c: use #ifdef COMPAT_HACKS instead of #if 0
* Makefile: fix to make clmodule.c compile;
make config.o dependent on libpython.a (so date is always correct)
* timemodule.c: now sleep() also takes a float argument.
* posixmodule.c: added nice().
* Makefile: svmodule.c.proto and svgen.py are gone, svmodule.c came in
their stead. Also, pass -DUSE_DL flag to thread.c and give
the user a possibility to add the -DDEBUG to just thread.c.
* ceval.c: init_save_thread() can be called more than once now.
* svgen.py, svmodule.c.proto, svmodule.c: Removed prototype file and
replaced it by the generated file.
* thread.c: Added some more checks; added call to DL library when it
is also used to tell it where the shared arena is so that DL
can use some other area.
* threadmodule.c: Call init_save_thread from another place. Also,
added new function getlocklock() which does to lock objects
what getfilefile does to file objects.
return NULL for malloc(0) or realloc(p, 0). (This should be done
differently than wasting one byte, but alas...)
* Moved "add'l libraries" option in Makefile to an earlier place.
* Remove argument compatibility hacks (b) and (c).
* Add grey2mono, dither2mono and mono2grey to imageop.
* Dup the fd in socket.fromfd().
* Added new modules mpz, md5 (by JH, requiring GNU MP 1.2). Affects
Makefile and config.c.
* socketmodule.c: added socket.fromfd(fd, family, type, [proto]),
converted socket() to use of getargs().
* flmodule.c: added {do,check}_only_forms to fl's list of functions;
and don't print a message when an unknown object is returned.
* pythonrun.c: catch SIGHUP and SIGTERM to do essential cleanup.
* Made jpegmodule.c smaller by using getargs() and mkvalue() consistently.
* Increased parser stack size to 500 in parser.h.
* Implemented custom allocation of stack frames to frameobject.c and
added dynamic stack overflow checks (value stack only) to ceval.c.
(There seems to be a bug left: sometimes stack traces don't make sense.)
sys.stderr or sys.stdin, and to work with any object as long as it has
a write() (respectively readline()) methods. Some functions that took
a FILE* argument now take an object* argument.
coercion is now completely generic.
* ceval.c: for instances, don't coerce for + and *; * reverses
arguments if left one is non-instance numeric and right one sequence.
* socketmodule.c: get rid of makepair(); fix makesocketaddr to fix
broken recvfrom()
* socketmodule: get rid of getStrarg()
* ceval.h: move eval_code() to new file eval.h, so compile.h is no
longer needed.
* ceval.c: move thread comments to ceval.h; always make save/restore
thread functions available (for dynloaded modules)
* cdmodule.c, listobject.c: don't include compile.h
* flmodule.c: include ceval.h
* import.c: include eval.h instead of ceval.h
* cgen.py: add forground(); noport(); winopen(""); to initgl().
* bltinmodule.c, socketmodule.c, fileobject.c, posixmodule.c,
selectmodule.c:
adapt to threads (add BGN/END SAVE macros)
* stdwinmodule.c: adapt to threads and use a special stdwin lock.
* pythonmain.c: don't include getpythonpath().
* pythonrun.c: use BGN/END SAVE instead of direct calls; also more
BGN/END SAVE calls etc.
* thread.c: bigger stack size for sun; change exit() to _exit()
* threadmodule.c: use BGN/END SAVE macros where possible
* timemodule.c: adapt better to threads; use BGN/END SAVE; add
longsleep internal function if BSD_TIME; cosmetics
* split pythonmain.c in two: most stuff goes to pythonrun.c, in the library.
* new optional built-in threadmodule.c, build upon Sjoerd's thread.{c,h}.
* new module from Sjoerd: mmmodule.c (dynamically loaded).
* new module from Sjoerd: sv (svgen.py, svmodule.c.proto).
* new files thread.{c,h} (from Sjoerd).
* new xxmodule.c (example only).
* myselect.h: bzero -> memset
* select.c: bzero -> memset; removed global variable
Added support for class definitions.
Reorganized main interpreter loop to fetch op and arg once at the head.
Use two bytes for arguments (see ceval.c).
Guido van Rossum
Jeremy Hylton
Barry Warsaw
Andrew M. Kuchling
Fred Drake
Sjoerd Mullender