Add configure option --enable-unicode.
Add config.h macros Py_USING_UNICODE, PY_UNICODE_TYPE, Py_UNICODE_SIZE,
SIZEOF_WCHAR_T.
Define Py_UCS2.
Encode and decode large UTF-8 characters into single Py_UNICODE values
for wide Unicode types; likewise for UTF-16.
Remove test whether sizeof Py_UNICODE is two.
NEEDS DOC CHANGES.
More AttributeErrors transmuted into TypeErrors, in test_b2.py, and,
again, this strikes me as a good thing.
This checkin completes the iterator generalization work that obviously
needed to be done. Can anyone think of others that should be changed?
NEEDS DOC CHANGES.
Possibly contentious: The first time s.next() yields StopIteration (for
a given map argument s) is the last time map() *tries* s.next(). That
is, if other sequence args are longer, s will never again contribute
anything but None values to the result, even if trying s.next() again
could yield another result. This is the same behavior map() used to have
wrt IndexError, so it's the only way to be wholly backward-compatible.
I'm not a fan of letting StopIteration mean "try again later" anyway.
Also a 2.1 bugfix candidate (am I supposed to do something with those?).
Took away map()'s insistence that sequences support __len__, and cleaned
up the convoluted code that made it *look* like it really cared about
__len__ (in fact the old ->len field was only *used* as a flag bit, as
the main loop only looked at its sign bit, setting the field to -1 when
IndexError got raised; renamed the field to ->saw_IndexError instead).
new slot tp_iter in type object, plus new flag Py_TPFLAGS_HAVE_ITER
new C API PyObject_GetIter(), calls tp_iter
new builtin iter(), with two forms: iter(obj), and iter(function, sentinel)
new internal object types iterobject and calliterobject
new exception StopIteration
new opcodes for "for" loops, GET_ITER and FOR_ITER (also supported by dis.py)
new magic number for .pyc files
new special method for instances: __iter__() returns an iterator
iteration over dictionaries: "for x in dict" iterates over the keys
iteration over files: "for x in file" iterates over lines
TODO:
documentation
test suite
decide whether to use a different way to spell iter(function, sentinal)
decide whether "for key in dict" is a good idea
use iterators in map/filter/reduce, min/max, and elsewhere (in/not in?)
speed tuning (make next() a slot tp_next???)
Jeffery Collins pointed out that filterstring decrefs a character object
before it's done using it. This works by accident today because another
module always happens to have an active reference too at the time. The
accident doesn't work after his Pippy modifications, and since it *is*
an accident even in the mainline Python, it should work by design there too.
The patch accomplishes that.
If a module has a future statement enabling nested scopes, they are
also enable for the exec statement and the functions compile() and
execfile() if they occur in the module.
If Python is run with the -i option, which enters interactive mode
after executing a script, and the script it runs enables nested
scopes, they are also enabled in interactive mode.
XXX The use of -i with -c "from __future__ import nested_scopes" is
not supported. What's the point?
To support these changes, many function variants have been added to
pythonrun.c. All the variants names end with Flags and they take an
extra PyCompilerFlags * argument. It is possible that this complexity
will be eliminated in a future version of the interpreter in which
nested scopes are not optional.
except that it always returns Unicode objects.
A new C API PyObject_Unicode() is also provided.
This closes patch #101664.
Written by Marc-Andre Lemburg. Copyright assigned to Guido van Rossum.
- Make error messages from issubclass() and isinstance() a bit more
descriptive (Ping, modified by Guido)
- Couple of tiny fixes to other docstrings (Ping)
- Get rid of trailing whitespace (Guido)
Add definitions of INT_MAX and LONG_MAX to pyport.h.
Remove includes of limits.h and conditional definitions of INT_MAX
and LONG_MAX elsewhere.
This closes SourceForge patch #101659 and bug #115323.
which implements the automatic conversion from Unicode to a string
object using the default encoding.
The new API is then put to use to have eval() and exec accept
Unicode objects as code parameter. This closes bugs #110924
and #113890.
As side-effect, the traditional C APIs PyString_Size() and
PyString_AsString() will also accept Unicode objects as
parameters.
PyRun_FileEx(). These are the same as their non-Ex counterparts but
have an extra argument, a flag telling them to close the file when
done.
Then this is used by Py_Main() and execfile() to close the file after
it is parsed but before it is executed.
Adding APIs seems strange given the feature freeze but it's the only
way I see to close the bug report without incompatible changes.
[ Bug #110616 ] source file stays open after parsing is done (PR#209)
scope. Previously, s_buffer[] was defined inside the
PyUnicode_Check() scope, but referred to in the outer scope via
assignment to s. This quiets an Insure portability warning.
accepted by the BDFL.
builtin_zip(): New function to implement the zip() function described
in the above proposal.
zip_doc[]: Docstring for zip().
builtin_methods[]: added entry for zip()
This adds support for instance to the constructor (instances
have to define __str__ and can return Unicode objects via that
hook; string return values are decoded into Unicode using the
current default encoding).
Various small fixes to the builtin module to ensure no buffer
overflows.
- chunk #1:
Proper casting to ensure no truncation, and hence no surprises, in the
comparison.
- chunk #2:
The id() function guarantees a unique return value for different
objects. It does this by returning the pointer to the object. By
returning a PyInt, on Win64 (sizeof(long) < sizeof(void*)) the pointer
is truncated and the guarantee may be proven false. The appropriate
return function is PyLong_FromVoidPtr, this returns a PyLong if that
is necessary to return the pointer without truncation.
[GvR: note that this means that id() can now return a long on Win32
platforms. This *might* break some code...]
- chunk #3:
Ensure no overflow in raw_input(). Granted the user would have to pass
in >2GB of data but it *is* a possible buffer overflow condition.
module and into _exceptions.c. This includes all the PyExc_* globals,
the bltin_exc table, init_class_exc(), fini_instances(),
finierrors().
Renamed _PyBuiltin_Init_1() to _PyBuiltin_Init() since the two phase
initializations are necessary any more.
Removed as obsolete _PyBuiltin_Init_2(), _PyBuiltin_Fini_1() and
_PyBuiltin_Fini_2().
For more comments, read the patches@python.org archives.
For documentation read the comments in mymalloc.h and objimpl.h.
(This is not exactly what Vladimir posted to the patches list; I've
made a few changes, and Vladimir sent me a fix in private email for a
problem that only occurs in debug mode. I'm also holding back on his
change to main.c, which seems unnecessary to me.)
- When 'import exceptions' fails, don't suggest to use -v to print the traceback;
this doesn't actually work.
- Remove comment about fallback to string exceptions.
- Remove a PyErr_Occurred() check after all is said and done that can
never trigger.
- Remove static function newstdexception() which is no longer called.
are no longer supported (i.e. -X option is removed).
_PyBuiltin_Init_1(): Don't call initerrors(). This does mean that it
is possible to raise an ImportError before that exception has been
initialized, say because exceptions.py can't be found, or contains
bogosity. See changes to errors.c for how this is handled.
_PyBuiltin_Init_2(): Don't test Py_UseClassExceptionsFlag, just go
ahead and initialize the class-based standard exceptions. If this
fails, we throw a Py_FatalError.
Added special case to unicode(): when being passed a
Unicode object as first argument, return the object as-is.
Raises an exception when given a Unicode object *and* an
encoding name.
his copy of test_contains.py seems to be broken -- the lines he
deleted were already absent). Checkin messages:
New Unicode support for int(), float(), complex() and long().
- new APIs PyInt_FromUnicode() and PyLong_FromUnicode()
- added support for Unicode to PyFloat_FromString()
- new encoding API PyUnicode_EncodeDecimal() which converts
Unicode to a decimal char* string (used in the above new
APIs)
- shortcuts for calls like int(<int object>) and float(<float obj>)
- tests for all of the above
Unicode compares and contains checks:
- comparing Unicode and non-string types now works; TypeErrors
are masked, all other errors such as ValueError during
Unicode coercion are passed through (note that PyUnicode_Compare
does not implement the masking -- PyObject_Compare does this)
- contains now works for non-string types too; TypeErrors are
masked and 0 returned; all other errors are passed through
Better testing support for the standard codecs.
Misc minor enhancements, such as an alias dbcs for the mbcs codec.
Changes:
- PyLong_FromString() now applies the same error checks as
does PyInt_FromString(): trailing garbage is reported
as error and not longer silently ignored. The only characters
which may be trailing the digits are 'L' and 'l' -- these
are still silently ignored.
- string.ato?() now directly interface to int(), long() and
float(). The error strings are now a little different, but
the type still remains the same. These functions are now
ready to get declared obsolete ;-)
- PyNumber_Int() now also does a check for embedded NULL chars
in the input string; PyNumber_Long() already did this (and
still does)
Followed by:
Looks like I've gone a step too far there... (and test_contains.py
seem to have a bug too).
I've changed back to reporting all errors in PyUnicode_Contains()
and added a few more test cases to test_contains.py (plus corrected
the join() NameError).
Introduce a new builtin exception, UnboundLocalError, raised when ceval.c
tries to retrieve or delete a local name that isn't bound to a value.
Currently raises NameError, which makes this behavior a FAQ since the same
error is raised for "missing" global names too: when the user has a global
of the same name as the unbound local, NameError makes no sense to them.
Even in the absence of shadowing, knowing whether a bogus name is local or
global is a real aid to quick understanding.
Example:
D:\src\PCbuild>type local.py
x = 42
def f():
print x
x = 13
return x
f()
D:\src\PCbuild>python local.py
Traceback (innermost last):
File "local.py", line 8, in ?
f()
File "local.py", line 4, in f
print x
UnboundLocalError: x
D:\src\PCbuild>
Note that UnboundLocalError is a subclass of NameError, for compatibility
with existing class-exception code that may be trying to catch this as a
NameError. Unfortunately, I see no way to make this wholly compatible
with -X (see comments in bltinmodule.c): under -X, [UnboundLocalError
is an alias for NameError --GvR].
[The ceval.c patch differs slightly from the second version that Tim
submitted; I decided not to raise UnboundLocalError for DELETE_NAME,
only for DELETE_LOCAL. DELETE_NAME is only generated at the module
level, and since at that level a NameError is raised for referencing
an undefined name, it should also be raised for deleting one.]
ExtensionClasses in isinstance() and issubclass().
- abstract instance and class protocols are used *only* in those
cases that would generate errors before the patch. That is, there's
no penalty for the normal case.
- instance protocol: an object smells like an instance if it
has a __class__ attribute that smells like a class.
- class protocol: an object smells like a class if it has a
__bases__ attribute that is a tuple with elements that
smell like classes (although not all elements may actually get
sniffed ;).
xrange(), especially for platforms where int and long are different
sizes (so sys.maxint isn't actually the theoretical limit for the
length of a list, but the largest C int is -- sys.maxint is the
largest Python int, which is actually a C long).
test for classes with a __complex__() method. The attribute is pulled
out of the instance with PyObject_GetAttr() but this transfers
ownership and the function object was never DECREF'd.
initialization of class exceptions. Specifically:
init_class_exc(): This function now returns an integer status of the
class exception initialization. No fatal errors in this method now.
Also, use PySys_WriteStderr() when writing error messages. When an
error occurs in this function, 0 is returned, but the partial creation
of the exception classes is not undone (this happens elsewhere).
Things that could trigger the fallback:
- exceptions.py fails to be imported (due to syntax error, etc.)
- one of the exception classes is missing (e.g. due to library
version mismatch)
- exception class can't be inserted into __builtin__'s dictionary
- MemoryError instance can't be pre-allocated
- some other PyErr_Occurred
newstdexception(): Changed the error message. This is still a fatal
error because if the string based exceptions can't be created, we
really can't continue.
initerrors(): Be sure to xdecref the .exc field, which might be
non-NULL if class exceptions init was aborted.
_PyBuiltin_Init_2(): If class exception init fails, print a warning
message and reinstate the string based exceptions.
OSError. The EnvironmentError serves primarily as the (common
implementation) base class for IOError and OSError. OSError is used
by posixmodule.c
Also added tuple definition of EnvironmentError when using string
based exceptions.
(1) If a sequence S is shorter than len(S) indicated, don't fail --
just use the shorter size. (I.e, len(S) is just a hint.)
(2) Implement the special case map(None, S) as list(S) -- it's faster.
the code here becomes much simpler. In particular: abs(), divmod(),
pow(), int(), long(), float(), len(), tuple(), list().
Also make sure that no use of a function pointer gotten from a
tp_as_sequence or tp_as_mapping structure is made without checking it
for NULL first.
A few other cosmetic things, such as properly reindenting slice().
but annoying memory leak. This was introduced when PyExc_Exception
was added; the loop above populating the PyExc_StandardError exception
tuple started at index 1 in bltin_exc, but PyExc_Exception was added
at index 0, so PyExc_StandardError was getting inserted in itself!
How else can a tuple include itself?!
Change the loop to start at index 2.
This was a *fun* one! :-)
This doesn't yet support "import a.b.c" or "from a.b.c import x", but
it does recognize directories. When importing a directory, it
initializes __path__ to a list containing the directory name, and
loads the __init__ module if found.
The (internal) find_module() and load_module() functions are
restructured so that they both also handle built-in and frozen modules
and Mac resources (and directories of course). The imp module's
find_module() and (new) load_module() also have this functionality.
Moreover, imp unconditionally defines constants for all module types,
and has two more new functions: find_module_in_package() and
find_module_in_directory().
There's also a new API function, PyImport_ImportModuleEx(), which
takes all four __import__ arguments (name, globals, locals, fromlist).
The last three may be NULL. This is currently the same as
PyImport_ImportModule() but in the future it will be able to do
relative dotted-path imports.
Other changes:
- bltinmodule.c: in __import__, call PyImport_ImportModuleEx().
- ceval.c: always pass the fromlist to __import__, even if it is a C
function, so PyImport_ImportModuleEx() is useful.
- getmtime.c: the function has a second argument, the FILE*, on which
it applies fstat(). According to Sjoerd this is much faster. The
first (pathname) argument is ignored, but remains for backward
compatibility (so the Mac version still works without changes).
By cleverly combining the new imp functionality, the full support for
dotted names in Python (mini.py, not checked in) is now about 7K,
lavishly commented (vs. 14K for ni plus 11K for ihooks, also lavishly
commented).
Good night!
Added PyErr_MemoryErrorInst to hold the pre-instantiated instance when
using class based exceptions.
Simplified the creation of all built-in exceptions, both class based
and string based. Actually, for class based exceptions, the string
ones are still created just in case there's a problem creating the
class based ones (so you still get *some* exception handling!). Now
the init and fini functions run through a list of structure elements,
creating the strings (and optionally classes) for every entry.
initerrors(): the new base class exceptions StandardError,
LookupError, and NumberError are initialized when using string
exceptions, to tuples containing the list of derived string
exceptions. This GvR trick enables forward compatibility! One bit of
nastiness is that the C code has to know the inheritance tree embodied
in exceptions.py.
Added the two phase init and fini functions.
classes as their second arguments. The former takes a class as the
first argument and returns true iff first is second, or is a subclass
of second.
The latter takes any object as the first argument and returns true iff
first is an instance of the second, or any subclass of second.
Also, change all occurances of pointer compares against
PyExc_IndexError with PyErr_ExceptionMatches() calls.
2. Fix two bugs in complex():
- Memory leak when using complex(classinstance) -- r was never
DECREF'ed.
- Conversion of the second argument, if not complex, was done using
the type vector of the 1st.
be Ellipsis!).
Bumped the API version because a linker-visible symbol is affected.
Old C code will still compile -- there's a b/w compat macro.
Similarly, old Python code will still run, builtin exports both
Ellipses and Ellipsis.
bltinmodule.c: fixed coerce() nightmare in ternary pow().
modsupport.c (initmodule2): pass METH_FREENAME flag to newmethodobject().
pythonrun.c: move flushline() into and around print_error().
global: Py_MakePendingCalls. Also guard against recursive calls
* Include/classobject.h, Objects/classobject.c,
Python/{ceval.c,bltinmodule.c}: entirely redone operator
overloading. The rules for class instances are now much more
relaxed than for other built-in types
(whose coerce must still return two objects of the same type)
* funcobject.c (func_repr): don't call getstringvalue(None) for anonymous
functions.
* bltinmodule.c: removed lambda (which is now a built-in function);
removed implied lambda for string arg to filter/map/reduce.
* Grammar, graminit.[ch], compile.[ch]: replaced lambda as built-in
function by lambda as grammar entity: instead of "lambda('x: x+1')" you
write "lambda x: x+1".
* Xtmodule.c (checkargdict): return 0, not NULL, for error.
* object.[ch], bltinmodule.c, fileobject.c: changed str() to call
strobject() which calls an object's __str__ method if it has one.
strobject() is also called by writeobject() when PRINT_RAW is passed.
* ceval.c: rationalize code for PRINT_ITEM (no change in function!)
* funcobject.c, codeobject.c: added compare and hash functionality.
Functions with identical code objects and the same global dictionary are
equal. Code objects are equal when their code, constants list and names
list are identical (i.e. the filename and code name don't count).
(hash doesn't work yet since the constants are in a list and lists can't
be hashed -- suppose this should really be done with a tuple now we have
resizetuple!)
careful about these.
* arraymodule.c: added 8 byte swap; added 'i' format character; added
reverse() method; rename read/write to fromfile/tofile.
* config.c: Set version to 0.9.9++.
* rotormodule.c (r_rand): declare k1..k5 as unsigned longs so the shifts
will have a well-defined effect independent of word size.
* bltinmodule.c: renamed bagof() to filter().
setlistslice() can be used to cut the unused part out of a freshly made
slice (as done by bagof()). [needed by the next mod!]
* structural changes to bagof(), map() etc.
* PROTO.h, mymalloc.h: added #ifdefs for TURBOC and GNUC.
* allobjects.h: added #include "rangeobject.h"
* Grammar: added lambda_input; relaxed syntax for exec.
* bltinmodule.c: added bagof, map, reduce, lambda, xrange.
* tupleobject.[ch]: added resizetuple().
* rangeobject.[ch]: new object type to speed up range operations (not
convinced this is needed!!!)
function vs. exec statement
* bltinmodule.c: renamed the module to __builtin__.
* posixmodule.c (posix_execv): renamed exec --> execv since it is now a
reserved word.
* Grammar: add exec statement; allow testlist in expr statement.
* ceval.c, compile.c, opcode.h: support exec statement;
avoid optimizing locals when it is used
* fileobject.{c,h}: add getfilename() internal function.
Added $(SYSDEF) to its build rule in Makefile.
* cgensupport.[ch], modsupport.[ch]: removed some old stuff. Also
changed files that still used it... And made several things static
that weren't but should have been... And other minor cleanups...
* listobject.[ch]: add external interfaces {set,get}listslice
* socketmodule.c: fix bugs in new send() argument parsing.
* sunaudiodevmodule.c: added flush() and close().
yet). The class is now passed to eval_code and stored in the current
frame. It is also stored in instance method objects. An "unbound"
instance method is now returned when a function is retrieved through
"classname.funcname", which when called passes the class to eval_code.
(1) dictionaries/mappings now have attributes values() and items() as
well as keys(); at the C level, use the new function mappinggetnext()
to iterate over a dictionary.
(2) "class C(): ..." is now illegal; you must write "class C: ...".
(3) Class objects now know their own name (finally!); and minor
improvements to the way how classes, functions and methods are
represented as strings.
(4) Added an "access" statement and semantics. (This is still
experimental -- as long as you don't use the keyword 'access' nothing
should be changed.)
f_fastlocals in a traceback object (this is a core dump hazard
if there are <nil> entries), but instead eval_code() merges the fast
locals back into the locals dictionary if it looks like the local
variables will be retained. Also, the merge routines save
exceptions since this is sometimes needed (alas!).
* Added id() to bltinmodule.c, which returns an object's address
(identity). Useful to walk arbitrary data structures containing
cycles.
* Added compile() to bltinmodule.c and compile_string() to
pythonrun.[ch]: support to exec/eval arbitrary code objects. The
code that defaults globals and locals is moved from run_node in
pythonrun.c (which is now identical to eval_node) to eval_code in
ceval.c. [XXX For elegance a clean-up session is necessary.]
* Stubs for faster implementation of local variables (not yet finished)
* Added function name to code object. Print it for code and function
objects. THIS MAKES THE .PYC FILE FORMAT INCOMPATIBLE (the version
number has changed accordingly)
* Print address of self for built-in methods
* New internal functions getattro and setattro (getattr/setattr with
string object arg)
* Replaced "dictobject" with more powerful "mappingobject"
* New per-type functio tp_hash to implement arbitrary object hashing,
and hashobject() to interface to it
* Added built-in functions hash(v) and hasattr(v, 'name')
* classobject: made some functions static that accidentally weren't;
added __hash__ special instance method to implement hash()
* Added proper comparison for built-in methods and functions
* Fixcprt.py: added [-y file] option, do only files younger than file.
* modsupport.[ch]: added vmkvalue().
* intobject.c: use mkvalue().
* stringobject.c: added "formatstring"; renamed string* to string_*;
ceval.c: call formatstring for string % value.
* longobject.c: close memory leak in divmod.
* parsetok.c: set result node to NULL when returning an error.
* flmodule.c: added {do,check}_only_forms to fl's list of functions;
and don't print a message when an unknown object is returned.
* pythonrun.c: catch SIGHUP and SIGTERM to do essential cleanup.
* Made jpegmodule.c smaller by using getargs() and mkvalue() consistently.
* Increased parser stack size to 500 in parser.h.
* Implemented custom allocation of stack frames to frameobject.c and
added dynamic stack overflow checks (value stack only) to ceval.c.
(There seems to be a bug left: sometimes stack traces don't make sense.)
sys.stderr or sys.stdin, and to work with any object as long as it has
a write() (respectively readline()) methods. Some functions that took
a FILE* argument now take an object* argument.
coercion is now completely generic.
* ceval.c: for instances, don't coerce for + and *; * reverses
arguments if left one is non-instance numeric and right one sequence.
* socketmodule.c: get rid of makepair(); fix makesocketaddr to fix
broken recvfrom()
* socketmodule: get rid of getStrarg()
* ceval.h: move eval_code() to new file eval.h, so compile.h is no
longer needed.
* ceval.c: move thread comments to ceval.h; always make save/restore
thread functions available (for dynloaded modules)
* cdmodule.c, listobject.c: don't include compile.h
* flmodule.c: include ceval.h
* import.c: include eval.h instead of ceval.h
* cgen.py: add forground(); noport(); winopen(""); to initgl().
* bltinmodule.c, socketmodule.c, fileobject.c, posixmodule.c,
selectmodule.c:
adapt to threads (add BGN/END SAVE macros)
* stdwinmodule.c: adapt to threads and use a special stdwin lock.
* pythonmain.c: don't include getpythonpath().
* pythonrun.c: use BGN/END SAVE instead of direct calls; also more
BGN/END SAVE calls etc.
* thread.c: bigger stack size for sun; change exit() to _exit()
* threadmodule.c: use BGN/END SAVE macros where possible
* timemodule.c: adapt better to threads; use BGN/END SAVE; add
longsleep internal function if BSD_TIME; cosmetics
Martin v. Löwis
Fredrik Lundh
Tim Peters
Mark Hammond
Guido van Rossum
Jeremy Hylton
Marc-André Lemburg
Fred Drake
Neil Schemenauer
Andrew M. Kuchling
Barry Warsaw
Trent Mick
Peter Schneider-Kamp
Thomas Wouters
Jack Jansen
Sjoerd Mullender