Highlights:
- Adding PyObject_Format.
- Adding string.Format class.
- Adding __format__ for str, unicode, int, long, float, datetime.
- Adding builtin format.
- Adding ''.format and u''.format.
- str/unicode fixups for formatters.
The files in Objects/stringlib that implement PEP 3101 (stringdefs.h,
unicodedefs.h, formatter.h, string_format.h) are identical in trunk
and py3k. Any changes from here on should be made to trunk, and
changes will propogate to py3k).
The mapping between bytecode offsets and source lines (lnotab) did not contain
an entry for the beginning of the loop.
Now it does, and the lnotab can be a bit larger:
in particular, several statements on the same line generate several entries.
However, this does not bother the settrace function, which will trigger only
one 'line' event.
The lnotab seems to be exactly the same as with python2.4.
I implemented the function sys._compact_freelists() and C API functions PyInt_/PyFloat_CompactFreeList() to compact the pre-allocated blocks of ints and floats. They allow the user to reduce the memory usage of a Python process that deals with lots of numbers.
The patch also renames sys._cleartypecache to sys._clear_type_cache
whole construct away, even when an 'else' clause is present::
while 0:
print("no")
else:
print("yes")
did not generate any code at all.
Now the compiler emits the 'else' block, like it already does for 'if' statements.
Will backport.
The "can't load dll" message box on Windows is suppressed while an extension is loaded by calling SetErrorMode in dynload_win.c. The error is still reported properly.
PyThreadState_Delete() to avoid an infinite loop when the tstate list
is messed up and has somehow becomes circular and does not contain the
current thread.
I don't know how this happens but it does, *very* rarely. On more than
one hardware platform. I have not been able to reproduce it manually.
Attaching to a process where its happening: it has always been in an
infinite loop over a single element tstate list that is not the tstate
we're looking to delete. It has been in t_bootstrap()'s call to
PyThreadState_DeleteCurrent() as a pthread is exiting.
local_ptr_assign_local: Assigning address of stack variable "namebuf" to pointer "filename"
out_of_scope: Variable "namebuf" goes out of scope
use_invalid: Used "filename" pointing to out-of-scope variable "namebuf"
round included:
* Revert round to its 2.6 behavior (half away from 0).
* Because round, floor, and ceil always return float again, it's no
longer necessary to have them delegate to __xxx___, so I've ripped
that out of their implementations and the Real ABC. This also helps
in implementing types that work in both 2.6 and 3.0: you return int
from the __xxx__ methods, and let it get enabled by the version
upgrade.
* Make pow(-1, .5) raise a ValueError again.
On Windows, when import fails to load a dll module, the message says
"error code 193" instead of a more informative text.
It turns out that FormatMessage needs additional parameters for some error codes.
For example: 193 means "%1 is not a valid Win32 application".
Since it is impossible to know which parameter to pass, we use
FORMAT_MESSAGE_IGNORE_INSERTS to get the raw message, which is still better
than the number.
the complex_pow part), r56649, r56652, r56715, r57296, r57302, r57359, r57361,
r57372, r57738, r57739, r58017, r58039, r58040, and r59390, and new
documentation. The only significant difference is that round(x) returns a float
to preserve backward-compatibility. See http://bugs.python.org/issue1689.
Allows dictionaries to be pre-sized (upto 255 elements) saving time lost
to re-sizes with their attendant mallocs and re-insertions.
Has zero effect on small dictionaries (5 elements or fewer), a slight
benefit for dicts upto 22 elements (because they had to resize once
anyway), and more benefit for dicts upto 255 elements (saving multiple
resizes during the build-up and reducing the number of collisions on
the first insertions). Beyond 255 elements, there is no addional benefit.
Added PyFloat_GetMax(), PyFloat_GetMin() and PyFloat_GetInfo() to the float API.
Added a dictionary sys.float_info with information about the internal floating point type to the sys module.
(in deallocation of running threads, for example), so the PyGILState_Release()
function must still be functional.
On the other hand, _PyGILState_Fini() only frees memory, and can be called later.
Backport candidate, but only after some experts comment on it.
I've finished the last task for the PCbuild9 directory today. I don't think there is much left to do. Now you can all play around with the shiny new VS 2008 and try the PGO builds. I was able to get a speed improvement of about 10% on py3k.
Have fun! :)
Correction for issue1265 (pdb bug with "with" statement).
When an unfinished generator-iterator is garbage collected, PyEval_EvalFrameEx
is called with a GeneratorExit exception set. This leads to funny results
if the sys.settrace function itself makes use of generators.
A visible effect is that the settrace function is reset to None.
Another is that the eventual "finally" block of the generator is not called.
It is necessary to save/restore the exception around the call to the trace
function.
This happens a lot with py3k: isinstance() of an ABCMeta instance runs
def __instancecheck__(cls, instance):
"""Override for isinstance(instance, cls)."""
return any(cls.__subclasscheck__(c)
for c in {instance.__class__, type(instance)})
which lets an opened generator expression each time it returns True.
Backport candidate, even if the case is less frequent in 2.5.
ever going back out to Python code in PyObject_Call(). Required introducing a
static RuntimeError instance so that normalizing an exception there is no
reliance on a recursive call that would put the exception system over the
recursion check itself.
object's co_consts tuple; add a test to show that the previous behavior (where
these two constants were "collapsed" into one) causes serious malfunctioning.
The value is always written to the returned pointer if getting it was
successful, even if a warning causes an error. (This probably doesn't matter
as the caller will probably discard the value.)
Will backport.
Py_ssize_t members.
Simplify the implementation of UnicodeError objects:
start and end attributes are now stored directly as
Py_ssize_t members, which simplifies various get and
set functions.
- Reenable modules on x64 that had been disabled aeons ago for Itanium.
- Cleared up confusion about compilers for 64 bit windows. There is only Itanium and x64. Added macros MS_WINI64 and MS_WINX64 for those rare cases where it matters, such as the disabling of modules above.
- Set target platform (_WIN32_WINNT and WINVER) to 0x0501 (XP) for x64, and 0x0400 (NT 4.0) otherwise, which are the targeted minimum platforms.
- Fixed thread_nt.h. The emulated InterlockedCompareExchange function didn´t work on x64, probaby due to the lack of a "volatile" specifier. Anyway, win95 is no longer a target platform.
- Itertools module used wrong constant to check for overflow in count()
- PyInt_AsSsize_t couldn't deal with attribute error when accessing the __long__ member.
- PyLong_FromSsize_t() incorrectly specified that the operand were unsigned.
With these changes, the x64 passes the testsuite, for those modules present.
The file should now follow PEP 7, except that it uses 4 space indents
(in the style of Py3k). This particular code would be really hard to
read with the regular tab idents.
Other changes:
- reflow long lines
- change multi-line conditionals to have test at end of line
* lines too long
* wrong indentation
* space after a function name
* wrong function name in error string
* simplifying some logic
Also add an error check to PyDict_SetItemString.
Patch #1591665: implement the __dir__() special function lookup in PyObject_Dir.
Had to change a few bits of the patch because classobjs and __methods__ are still
in Py2.6.
exception if the -i command line option or PYTHONINSPECT environment
variable is given, but break into the interactive interpreter just like
on other exceptions or normal program exit.
(backport)
* use %r instead of backticks since backticks are going away in Py3k
* PyArena_Malloc() already sets PyErr_NoMemory so we don't need to do it again
* the signature for ast2obj_int incorrectly used a bool, rather than a long
I can't think of an easy way to test this behavior. It only occurs
when the file system default encoding and the interpreter default
encoding are different, such that you can open the file but not decode
its name.
is specified at the top of the file. Also add a note that Python/Python-ast.c
needs to be committed separately after a change to the AST grammar to capture
the revision number of the change (which is what __version__ is set to).
The next step of PEP 352 (for 2.6) causes raising a string exception to trigger
a TypeError. Trying to catch a string exception raises a DeprecationWarning.
References to string exceptions has been removed from the docs since they are
now just an error.
When running the interpreter in an environment that would cause it to set
stdout/stderr/stdin's encoding, having a sitecustomize that would replace
them with something other than PyFile objects would crash the interpreter.
Fix it by simply ignoring the encoding-setting for non-files.
This could do with a test, but I can think of no maintainable and portable
way to test this bug, short of adding a sitecustomize.py to the buildsystem
and have it always run with it (hmmm....)
It seems like this should be a different error than SystemError, but
I don't have any great ideas and SystemError was raised in 2.4 and earlier.
Will backport.
some warnings from Klokwork. They verify the assumptions of the format
of svn version output.
The assert in the thread module helped debug a problem on HP-UX.
* unified the way intobject, longobject and mystrtoul handle
values around -sys.maxint-1.
* in general, trying to entierely avoid overflows in any computation
involving signed ints or longs is extremely involved. Fixed a few
simple cases where a compiler might be too clever (but that's all
guesswork).
* more overflow checks against bad data in marshal.c.
* 2.5 specific: fixed a number of places that were still confusing int
and Py_ssize_t. Some of them could potentially have caused
"real-world" breakage.
* list.pop(x): fixing overflow issues on x was messy. I just reverted
to PyArg_ParseTuple("n"), which does the right thing. (An obscure
test was trying to give a Decimal to list.pop()... doesn't make
sense any more IMHO)
* trying to write a few tests...
The compiler was checking that there was something on the fblock
stack, but not that there was a loop on the stack. Fixed that and
added a test for the specific syntax error.
Bug fix candidate.
Building with HP's cc on HP-UX turned up a couple of problems.
_PyGILState_NoteThreadState was declared as static inconsistently.
Make it static as it's not necessary outside of this module.
Some tests failed because errno was reset to 0. (I think the tests
that failed were at least: test_fcntl and test_mailbox).
Ensure that errno doesn't change after a call to Py_END_ALLOW_THREADS.
This only affected debug builds.
generator expressions (x for x, in ... ) works again.
Sigh, I only fixed for loops the first time, not list comps and genexprs too.
I couldn't find any more unpacking cases where there is a similar bug lurking.
This code should be refactored to eliminate the duplication. I'm sure
the listcomp/genexpr code can be refactored. I'm not sure if the for loop
can re-use any of the same code though.
Will backport to 2.5 (the only place it matters).
I modified this patch some by fixing style, some error checking, and adding
XXX comments. This patch requires review and some changes are to be expected.
I'm checking in now to get the greatest possible review and establish a
baseline for moving forward. I don't want this to hold up release if possible.
However, there was no error checking that PyFloat_FromDouble returned
a valid pointer. I believe this change is correct as it seemed
to follow other code in the area.
Klocwork # 292.
there was no verification that privateobj was a PyString. If it wasn't
a string, this could have allowed a NULL pointer to creep in below and crash.
I wonder if this should be PyString_CheckExact? Must identifiers be strings
or can they be subclasses?
Klocwork #275
This is the first batch of fixes that should be easy to verify based on context.
This fixes problem numbers: 220 (ast), 323-324 (symtable),
321-322 (structseq), 215 (array), 210 (hotshot), 182 (codecs), 209 (etree).
PyThreadState_SetAsyncExc(): internal correctness changes wrt
refcount safety and deadlock avoidance. Also added a basic test
case (relying on ctypes) and repaired the docs.
on each iteration. I'm not positive this is the best way to handle
this. I'm also not sure that there aren't other cases where
the lnotab is generated incorrectly. It would be great if people
that use pdb or tracing could test heavily.
Also:
* Remove dead/duplicated code that wasn't used/necessary
because we already handled the docstring prior to entering the loop.
* add some debugging code into the compiler (#if 0'd out).
This provides the proper warning for struct.pack().
PyErr_Warn() is now deprecated in favor of PyErr_WarnEx().
As mentioned by Tim Peters on python-dev.
with PEP 302. This was fixed by adding an ``imp.NullImporter`` type that is
used in ``sys.path_importer_cache`` to cache non-directory paths and avoid
excessive filesystem operations during imports.
In general, C doesn't define anything about what happens when
an operation on a signed integral type overflows, and PyOS_strtol()
did several formally undefined things of that nature on signed
longs. Some version of gcc apparently tries to exploit that now,
and PyOS_strtol() could fail to detect overflow then.
Tried to repair all that, although it seems at least as likely to me
that we'll get screwed by bad platform definitions for LONG_MIN
and/or LONG_MAX now. For that reason, I don't recommend backporting
this.
Note that I have no box on which this makes a lick of difference --
can't really test it, except to note that it didn't break anything
on my boxes.
Silent change: PyOS_strtol() used to return the hard-coded 0x7fffffff
in case of overflow. Now it returns LONG_MAX. They're the same only on
32-bit boxes (although C doesn't guarantee that either ...).
Moved the code for _PyThread_CurrentFrames() up, so it's no longer
in a huge "#ifdef WITH_THREAD" block (I didn't realize it /was/ in
one).
Changed test_sys's test_current_frames() so it passes with or without
thread supported compiled in.
Note that test_sys fails when Python is compiled without threads,
but for an unrelated reason (the old test_exit() fails with an
indirect ImportError on the `thread` module). There are also
other unrelated compilation failures without threads, in extension
modules (like ctypes); at least the core compiles again.
Do we really support --without-threads? If so, there are several
problems remaining.
v2 can be NULL if exception2 is NULL. I don't think that condition can happen,
but I'm not sure it can't either. Now the code will protect against either
being NULL.
omit a default "error" argument for NULL pointer. This allows
the parser to take a codec from cjkcodecs again.
(Reported by Taewook Kang and reviewed by Walter Doerwald)
Heavily revised, comprising revisions:
46640 - original trunk revision (backed out in r46655)
46647 - markup fix (backed out in r46655)
46692:46918 merged from branch aimacintyre-sf1454481
branch tested on buildbots (Windows buildbots had problems
not related to these changes).
exact maximum size someone guesses is needed. In this case, if
we're really worried about extreme integers, then "cp%d" can
actually need 14 bytes (2 for "cp" + 1 for \0 at the end +
11 for -(2**31-1)). So reserve 128 bytes instead -- nothing is
actually saved by making a stack-local buffer tiny.
46640 Patch #1454481: Make thread stack size runtime tunable.
46647 Markup fix
The first is causing many buildbots to fail test runs, and there
are multiple causes with seemingly no immediate prospects for
repairing them. See python-dev discussion.
Note that a branch can (and should) be created for resolving these
problems, like
svn copy svn+ssh://svn.python.org/python/trunk -r46640 svn+ssh://svn.python.org/python/branches/NEW_BRANCH
followed by merging rev 46647 to the new branch.
set_exc_info(), reset_exc_info(): By exploiting the
likely (who knows?) invariant that when an exception's
`type` is NULL, its `value` and `traceback` are also NULL,
save some cycles in heavily-executed code.
This is a "a kronar saved is a kronar earned" patch: the
speedup isn't reliably measurable, but it obviously does
reduce the operation count in the normal (no exception
raised) path through PyEval_EvalFrameEx().
The tim-exc_sanity branch tries to push this harder, but
is still blowing up (at least in part due to pre-existing
subtle bugs that appear to have no other visible
consequences!).
Not a bugfix candidate.
invalid file paths for the built-in import machinery which leads to
fewer open calls on startup.
Also fix issue with PEP 302 style import hooks which lead to more open()
calls than necessary.
both mystrtoul.c and longobject.c. Share the table instead. Also
cut its size by 64 entries (they had been used for an inscrutable
trick originally, but the code no longer tries to use that trick).
In rare cases of strings specifying true values near sys.maxint,
and oddball bases (not decimal or a power of 2), int(string, base)
could deliver insane answers. This repairs all such problems, and
also speeds string->int significantly. On my box, here are %
speedups for decimal strings of various lengths:
length speedup
------ -------
1 12.4%
2 15.7%
3 20.6%
4 28.1%
5 33.2%
6 37.5%
7 41.9%
8 46.3%
9 51.2%
10 19.5%
11 19.9%
12 23.9%
13 23.7%
14 23.3%
15 24.9%
16 25.3%
17 28.3%
18 27.9%
19 35.7%
Note that the difference between 9 and 10 is the difference between
short and long Python ints on a 32-bit box. The patch doesn't
actually do anything to speed conversion to long: the speedup is
due to detecting "unsigned long" overflow more quickly.
This is a bugfix candidate, but it's a non-trivial patch and it
would be painful to separate the "bug fix" from the "speed up" parts.
discussion.
There are two places of documentation that still mention __context__:
Doc/lib/libstdtypes.tex -- I wasn't quite sure how to rewrite that without
spending a whole lot of time thinking about it; and whatsnew, which Andrew
usually likes to change himself.
- Warn-raise ImportWarning when importing would have picked up a directory
as package, if only it'd had an __init__.py. This swaps two tests (for
case-ness and __init__-ness), but case-test is not really more expensive,
and it's not in a speed-critical section.
- Test for the new warning by importing a common non-package directory on
sys.path: site-packages
- In regrtest.py, silence warnings generated by the build-environment
because Modules/ (which is added to sys.path for Setup-created modules)
has 'zlib' and '_ctypes' directories without __init__.py's.
MAXPATHLEN-sized buffers for various output-buffers (like to realpath()),
and that's correct on BSD platforms, but not Linux (which uses PATH_MAX, and
does not define MAXPATHLEN.) Cursory googling suggests Linux is following a
newer standard than BSD, but in cases like this, who knows. Using the
greater of PATH_MAX and 1024 as a fallback for MAXPATHLEN seems to be the
most portable solution.
exceptions that can't be raised any further, because (for instance) they
occur in __del__ methods. The coroutine tests in test_generators was
triggering this leak. Remove the leakers' testcase, and add a simpler
testcase that explicitly tests this leak to test_generators.
test_generators now no longer leaks at all, on my machine. This fix may also
solve other leaks, but my full refleakhunting run is still busy, so who
knows?
using a custom, nearly-identical macro. This probably changes how some of
these functions are compiled, which may result in fractionally slower (or
faster) execution. Considering the nature of traversal, visiting much of the
address space in unpredictable patterns, I'd argue the code readability and
maintainability is well worth it ;P
passing a string. Martin already fixed the actual crash by ensuring
Py_UNICODE is unsigned. As discussed on python-dev, this fix
removes the possibility of creating a unicode string from a raw buffer.
There is an outstanding question of how to fix the crash in 2.4.
that are suspended outside of any try/except/finally blocks to be
garbage collected even if they are part of a cycle. Generators that
suspend inside of an active try/except or try/finally block (including
those created by a ``with`` statement) are still not GC-able if they
are part of a cycle, however.
Re-enable all the tests in test_trace.py except one. Still not sure that these tests test what they used to test, but they pass. One failing test seems to be caused by undocumented line number table behavior in Python 2.4.
tracing/line number table in except blocks.
Reflow long lines introduced by col_offset changes. Update test_ast
to handle new fields in excepthandler.
As note in Python.asdl says, we might want to rethink how attributes
are handled. Perhaps they should be the same as other fields, with
the primary difference being how they are defined for all types within
a sum.
Also fix asdl_c so that constructors with int fields don't fail when
passed a zero value.
Explicitly clear all elements from arena->a_objects and remove
assert() that refcount is 1. It's possible for a program to get a
reference to the list via sys.getobjects() or via gc functions.
def foo((x)): was getting recognized as requiring tuple unpacking
which is not correct.
Add tests for this case and the proper way to unpack a tuple of one:
def foo((x,)):
test_inpsect was incorrect before. I'm not sure why it was passing,
but that has been corrected with a test for both functions above.
This means the test (and therefore inspect.getargspec()) are broken in 2.4.
objimpl.h, pymem.h: Stop mapping PyMem_{Del, DEL} and PyMem_{Free, FREE}
to PyObject_{Free, FREE} in a release build. They're aliases for the
system free() now.
_subprocess.c/sp_handle_dealloc(): Since the memory was originally
obtained via PyObject_NEW, it must be released via PyObject_FREE (or
_DEL).
pythonrun.c, tokenizer.c, parsermodule.c: I lost count of the number of
PyObject vs PyMem mismatches in these -- it's like the specific
function called at each site was picked at random, sometimes even with
memory obtained via PyMem getting released via PyObject. Changed most
to use PyObject uniformly, since the blobs allocated are predictably
small in most cases, and obmalloc is generally faster than system
mallocs then.
If extension modules in real life prove as sloppy as Python's front
end, we'll have to revert the objimpl.h + pymem.h part of this patch.
Note that no problems will show up in a debug build (all calls still go
thru obmalloc then). Problems will show up only in a release build, most
likely segfaults.
This will hopefully get rid of some Coverity warnings, be a hint to
developers, and be marginally faster.
Some asserts were added when the type is currently known, but depends
on values from another function.
PyCodec_IncrementalDecoder().
Factor out common code from PyCodec_Encoder()/PyCodec_Decoder(),
PyCodec_IncrementalEncoder()/PyCodec_IncrementalDecoder() and
PyCodec_StreamReader()/PyCodec_StreamWriter().
of tuple) that provides incremental decoders and encoders (a way to use
stateful codecs without the stream API). Functions
codecs.getincrementaldecoder() and codecs.getincrementalencoder() have
been added.
Anyway, this is the changes to the with-statement
so that __exit__ must return a true value in order
for a pending exception to be ignored.
The PEP (343) is already updated.
added message attribute compared to the previous version of Exception. It is
also a new-style class, making all exceptions now new-style. KeyboardInterrupt
and SystemExit inherit from BaseException directly. String exceptions now
raise DeprecationWarning.
Applies patch 1104669, and closes bugs 1012952 and 518846.
- New semantics for __exit__() -- it must re-raise the exception
if type is not None; the with-statement itself doesn't do this.
(See the updated PEP for motivation.)
- Added context managers to:
- file
- thread.LockType
- threading.{Lock,RLock,Condition,Semaphore,BoundedSemaphore}
- decimal.Context
- Added contextlib.py, which defines @contextmanager, nested(), closing().
- Unit tests all around; bot no docs yet.
Christian Heimes
Eric Smith
Kristján Valur Jónsson
Brett Cannon
Amaury Forgeot d'Arc
Neal Norwitz
Jeffrey Yasskin
Mark Dickinson
Raymond Hettinger
Guido van Rossum
Georg Brandl
Gregory P. Smith
Andrew M. Kuchling
Skip Montanaro
Nick Coghlan
Martin v. Löwis
Thomas Wouters
Alex Martelli
Walter Dörwald
Collin Winter
Jeremy Hylton
Anthony Baxter
Armin Rigo
Tim Peters
Thomas Heller
Phillip J. Eby
Andrew MacIntyre
Neil Schemenauer
Hye-Shik Chang
Richard Jones
Fredrik Lundh
Ronald Oussoren