This work is substantially Anthony Baxter's, from issue
1633807. I just freshened it, made a few minor tweaks,
and added the test cases. I also created issue 2412,
which is to check for 2to3's behavior with the print
function. I also added myself to ACKS.
Added 0b and 0o literals to tokenizer.
Modified PyOS_strtoul to support 0b and 0o inputs.
Modified PyLong_FromString to support guessing 0b and 0o inputs.
Renamed test_hexoct.py to test_int_literal.py and added binary tests.
Added upper and lower case 0b, 0O, and 0X tests to test_int_literal.py
Added "Z" format_char to PyOS_ascii_formatd to support empty float presentation type.
Renamed buf_size in PyOS_ascii_formatd to more accurately reflect it's meaning.
Modified format.__float__ to use the new "Z" format as the default.
Added test cases.
assert (0, 'message')
An empty tuple does not create a warning. While questionable usage:
assert (), 'message'
should not display a warning. Tested manually.
The warning message could be improved. Feel free to update it.
would give bogus error messages, because of untested exceptions::
>>> f(**g(1=2))
XXX undetected error
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
TypeError: 'int' object is not iterable
instead of the expected SyntaxError: keyword can't be an expression
Will backport.
My tests don't show the promised speed up of 10%. The code is as fast as the old code for simple cases and slightly faster for complex cases with several of args and kwargs. But the patch simplifies the code, too.
Highlights:
- Adding PyObject_Format.
- Adding string.Format class.
- Adding __format__ for str, unicode, int, long, float, datetime.
- Adding builtin format.
- Adding ''.format and u''.format.
- str/unicode fixups for formatters.
The files in Objects/stringlib that implement PEP 3101 (stringdefs.h,
unicodedefs.h, formatter.h, string_format.h) are identical in trunk
and py3k. Any changes from here on should be made to trunk, and
changes will propogate to py3k).
The mapping between bytecode offsets and source lines (lnotab) did not contain
an entry for the beginning of the loop.
Now it does, and the lnotab can be a bit larger:
in particular, several statements on the same line generate several entries.
However, this does not bother the settrace function, which will trigger only
one 'line' event.
The lnotab seems to be exactly the same as with python2.4.
I implemented the function sys._compact_freelists() and C API functions PyInt_/PyFloat_CompactFreeList() to compact the pre-allocated blocks of ints and floats. They allow the user to reduce the memory usage of a Python process that deals with lots of numbers.
The patch also renames sys._cleartypecache to sys._clear_type_cache
whole construct away, even when an 'else' clause is present::
while 0:
print("no")
else:
print("yes")
did not generate any code at all.
Now the compiler emits the 'else' block, like it already does for 'if' statements.
Will backport.
The "can't load dll" message box on Windows is suppressed while an extension is loaded by calling SetErrorMode in dynload_win.c. The error is still reported properly.
PyThreadState_Delete() to avoid an infinite loop when the tstate list
is messed up and has somehow becomes circular and does not contain the
current thread.
I don't know how this happens but it does, *very* rarely. On more than
one hardware platform. I have not been able to reproduce it manually.
Attaching to a process where its happening: it has always been in an
infinite loop over a single element tstate list that is not the tstate
we're looking to delete. It has been in t_bootstrap()'s call to
PyThreadState_DeleteCurrent() as a pthread is exiting.
local_ptr_assign_local: Assigning address of stack variable "namebuf" to pointer "filename"
out_of_scope: Variable "namebuf" goes out of scope
use_invalid: Used "filename" pointing to out-of-scope variable "namebuf"
round included:
* Revert round to its 2.6 behavior (half away from 0).
* Because round, floor, and ceil always return float again, it's no
longer necessary to have them delegate to __xxx___, so I've ripped
that out of their implementations and the Real ABC. This also helps
in implementing types that work in both 2.6 and 3.0: you return int
from the __xxx__ methods, and let it get enabled by the version
upgrade.
* Make pow(-1, .5) raise a ValueError again.
On Windows, when import fails to load a dll module, the message says
"error code 193" instead of a more informative text.
It turns out that FormatMessage needs additional parameters for some error codes.
For example: 193 means "%1 is not a valid Win32 application".
Since it is impossible to know which parameter to pass, we use
FORMAT_MESSAGE_IGNORE_INSERTS to get the raw message, which is still better
than the number.
the complex_pow part), r56649, r56652, r56715, r57296, r57302, r57359, r57361,
r57372, r57738, r57739, r58017, r58039, r58040, and r59390, and new
documentation. The only significant difference is that round(x) returns a float
to preserve backward-compatibility. See http://bugs.python.org/issue1689.
Allows dictionaries to be pre-sized (upto 255 elements) saving time lost
to re-sizes with their attendant mallocs and re-insertions.
Has zero effect on small dictionaries (5 elements or fewer), a slight
benefit for dicts upto 22 elements (because they had to resize once
anyway), and more benefit for dicts upto 255 elements (saving multiple
resizes during the build-up and reducing the number of collisions on
the first insertions). Beyond 255 elements, there is no addional benefit.
Added PyFloat_GetMax(), PyFloat_GetMin() and PyFloat_GetInfo() to the float API.
Added a dictionary sys.float_info with information about the internal floating point type to the sys module.
(in deallocation of running threads, for example), so the PyGILState_Release()
function must still be functional.
On the other hand, _PyGILState_Fini() only frees memory, and can be called later.
Backport candidate, but only after some experts comment on it.
I've finished the last task for the PCbuild9 directory today. I don't think there is much left to do. Now you can all play around with the shiny new VS 2008 and try the PGO builds. I was able to get a speed improvement of about 10% on py3k.
Have fun! :)
Correction for issue1265 (pdb bug with "with" statement).
When an unfinished generator-iterator is garbage collected, PyEval_EvalFrameEx
is called with a GeneratorExit exception set. This leads to funny results
if the sys.settrace function itself makes use of generators.
A visible effect is that the settrace function is reset to None.
Another is that the eventual "finally" block of the generator is not called.
It is necessary to save/restore the exception around the call to the trace
function.
This happens a lot with py3k: isinstance() of an ABCMeta instance runs
def __instancecheck__(cls, instance):
"""Override for isinstance(instance, cls)."""
return any(cls.__subclasscheck__(c)
for c in {instance.__class__, type(instance)})
which lets an opened generator expression each time it returns True.
Backport candidate, even if the case is less frequent in 2.5.
ever going back out to Python code in PyObject_Call(). Required introducing a
static RuntimeError instance so that normalizing an exception there is no
reliance on a recursive call that would put the exception system over the
recursion check itself.
object's co_consts tuple; add a test to show that the previous behavior (where
these two constants were "collapsed" into one) causes serious malfunctioning.
The value is always written to the returned pointer if getting it was
successful, even if a warning causes an error. (This probably doesn't matter
as the caller will probably discard the value.)
Will backport.
Py_ssize_t members.
Simplify the implementation of UnicodeError objects:
start and end attributes are now stored directly as
Py_ssize_t members, which simplifies various get and
set functions.
- Reenable modules on x64 that had been disabled aeons ago for Itanium.
- Cleared up confusion about compilers for 64 bit windows. There is only Itanium and x64. Added macros MS_WINI64 and MS_WINX64 for those rare cases where it matters, such as the disabling of modules above.
- Set target platform (_WIN32_WINNT and WINVER) to 0x0501 (XP) for x64, and 0x0400 (NT 4.0) otherwise, which are the targeted minimum platforms.
- Fixed thread_nt.h. The emulated InterlockedCompareExchange function didn´t work on x64, probaby due to the lack of a "volatile" specifier. Anyway, win95 is no longer a target platform.
- Itertools module used wrong constant to check for overflow in count()
- PyInt_AsSsize_t couldn't deal with attribute error when accessing the __long__ member.
- PyLong_FromSsize_t() incorrectly specified that the operand were unsigned.
With these changes, the x64 passes the testsuite, for those modules present.
The file should now follow PEP 7, except that it uses 4 space indents
(in the style of Py3k). This particular code would be really hard to
read with the regular tab idents.
Other changes:
- reflow long lines
- change multi-line conditionals to have test at end of line
* lines too long
* wrong indentation
* space after a function name
* wrong function name in error string
* simplifying some logic
Also add an error check to PyDict_SetItemString.
Patch #1591665: implement the __dir__() special function lookup in PyObject_Dir.
Had to change a few bits of the patch because classobjs and __methods__ are still
in Py2.6.
exception if the -i command line option or PYTHONINSPECT environment
variable is given, but break into the interactive interpreter just like
on other exceptions or normal program exit.
(backport)
* use %r instead of backticks since backticks are going away in Py3k
* PyArena_Malloc() already sets PyErr_NoMemory so we don't need to do it again
* the signature for ast2obj_int incorrectly used a bool, rather than a long
I can't think of an easy way to test this behavior. It only occurs
when the file system default encoding and the interpreter default
encoding are different, such that you can open the file but not decode
its name.
is specified at the top of the file. Also add a note that Python/Python-ast.c
needs to be committed separately after a change to the AST grammar to capture
the revision number of the change (which is what __version__ is set to).
The next step of PEP 352 (for 2.6) causes raising a string exception to trigger
a TypeError. Trying to catch a string exception raises a DeprecationWarning.
References to string exceptions has been removed from the docs since they are
now just an error.
When running the interpreter in an environment that would cause it to set
stdout/stderr/stdin's encoding, having a sitecustomize that would replace
them with something other than PyFile objects would crash the interpreter.
Fix it by simply ignoring the encoding-setting for non-files.
This could do with a test, but I can think of no maintainable and portable
way to test this bug, short of adding a sitecustomize.py to the buildsystem
and have it always run with it (hmmm....)
It seems like this should be a different error than SystemError, but
I don't have any great ideas and SystemError was raised in 2.4 and earlier.
Will backport.
some warnings from Klokwork. They verify the assumptions of the format
of svn version output.
The assert in the thread module helped debug a problem on HP-UX.
* unified the way intobject, longobject and mystrtoul handle
values around -sys.maxint-1.
* in general, trying to entierely avoid overflows in any computation
involving signed ints or longs is extremely involved. Fixed a few
simple cases where a compiler might be too clever (but that's all
guesswork).
* more overflow checks against bad data in marshal.c.
* 2.5 specific: fixed a number of places that were still confusing int
and Py_ssize_t. Some of them could potentially have caused
"real-world" breakage.
* list.pop(x): fixing overflow issues on x was messy. I just reverted
to PyArg_ParseTuple("n"), which does the right thing. (An obscure
test was trying to give a Decimal to list.pop()... doesn't make
sense any more IMHO)
* trying to write a few tests...
The compiler was checking that there was something on the fblock
stack, but not that there was a loop on the stack. Fixed that and
added a test for the specific syntax error.
Bug fix candidate.
Building with HP's cc on HP-UX turned up a couple of problems.
_PyGILState_NoteThreadState was declared as static inconsistently.
Make it static as it's not necessary outside of this module.
Some tests failed because errno was reset to 0. (I think the tests
that failed were at least: test_fcntl and test_mailbox).
Ensure that errno doesn't change after a call to Py_END_ALLOW_THREADS.
This only affected debug builds.
generator expressions (x for x, in ... ) works again.
Sigh, I only fixed for loops the first time, not list comps and genexprs too.
I couldn't find any more unpacking cases where there is a similar bug lurking.
This code should be refactored to eliminate the duplication. I'm sure
the listcomp/genexpr code can be refactored. I'm not sure if the for loop
can re-use any of the same code though.
Will backport to 2.5 (the only place it matters).
I modified this patch some by fixing style, some error checking, and adding
XXX comments. This patch requires review and some changes are to be expected.
I'm checking in now to get the greatest possible review and establish a
baseline for moving forward. I don't want this to hold up release if possible.
However, there was no error checking that PyFloat_FromDouble returned
a valid pointer. I believe this change is correct as it seemed
to follow other code in the area.
Klocwork # 292.
there was no verification that privateobj was a PyString. If it wasn't
a string, this could have allowed a NULL pointer to creep in below and crash.
I wonder if this should be PyString_CheckExact? Must identifiers be strings
or can they be subclasses?
Klocwork #275
This is the first batch of fixes that should be easy to verify based on context.
This fixes problem numbers: 220 (ast), 323-324 (symtable),
321-322 (structseq), 215 (array), 210 (hotshot), 182 (codecs), 209 (etree).
PyThreadState_SetAsyncExc(): internal correctness changes wrt
refcount safety and deadlock avoidance. Also added a basic test
case (relying on ctypes) and repaired the docs.
on each iteration. I'm not positive this is the best way to handle
this. I'm also not sure that there aren't other cases where
the lnotab is generated incorrectly. It would be great if people
that use pdb or tracing could test heavily.
Also:
* Remove dead/duplicated code that wasn't used/necessary
because we already handled the docstring prior to entering the loop.
* add some debugging code into the compiler (#if 0'd out).
This provides the proper warning for struct.pack().
PyErr_Warn() is now deprecated in favor of PyErr_WarnEx().
As mentioned by Tim Peters on python-dev.
with PEP 302. This was fixed by adding an ``imp.NullImporter`` type that is
used in ``sys.path_importer_cache`` to cache non-directory paths and avoid
excessive filesystem operations during imports.
In general, C doesn't define anything about what happens when
an operation on a signed integral type overflows, and PyOS_strtol()
did several formally undefined things of that nature on signed
longs. Some version of gcc apparently tries to exploit that now,
and PyOS_strtol() could fail to detect overflow then.
Tried to repair all that, although it seems at least as likely to me
that we'll get screwed by bad platform definitions for LONG_MIN
and/or LONG_MAX now. For that reason, I don't recommend backporting
this.
Note that I have no box on which this makes a lick of difference --
can't really test it, except to note that it didn't break anything
on my boxes.
Silent change: PyOS_strtol() used to return the hard-coded 0x7fffffff
in case of overflow. Now it returns LONG_MAX. They're the same only on
32-bit boxes (although C doesn't guarantee that either ...).
Moved the code for _PyThread_CurrentFrames() up, so it's no longer
in a huge "#ifdef WITH_THREAD" block (I didn't realize it /was/ in
one).
Changed test_sys's test_current_frames() so it passes with or without
thread supported compiled in.
Note that test_sys fails when Python is compiled without threads,
but for an unrelated reason (the old test_exit() fails with an
indirect ImportError on the `thread` module). There are also
other unrelated compilation failures without threads, in extension
modules (like ctypes); at least the core compiles again.
Do we really support --without-threads? If so, there are several
problems remaining.
v2 can be NULL if exception2 is NULL. I don't think that condition can happen,
but I'm not sure it can't either. Now the code will protect against either
being NULL.
omit a default "error" argument for NULL pointer. This allows
the parser to take a codec from cjkcodecs again.
(Reported by Taewook Kang and reviewed by Walter Doerwald)
Heavily revised, comprising revisions:
46640 - original trunk revision (backed out in r46655)
46647 - markup fix (backed out in r46655)
46692:46918 merged from branch aimacintyre-sf1454481
branch tested on buildbots (Windows buildbots had problems
not related to these changes).
exact maximum size someone guesses is needed. In this case, if
we're really worried about extreme integers, then "cp%d" can
actually need 14 bytes (2 for "cp" + 1 for \0 at the end +
11 for -(2**31-1)). So reserve 128 bytes instead -- nothing is
actually saved by making a stack-local buffer tiny.
46640 Patch #1454481: Make thread stack size runtime tunable.
46647 Markup fix
The first is causing many buildbots to fail test runs, and there
are multiple causes with seemingly no immediate prospects for
repairing them. See python-dev discussion.
Note that a branch can (and should) be created for resolving these
problems, like
svn copy svn+ssh://svn.python.org/python/trunk -r46640 svn+ssh://svn.python.org/python/branches/NEW_BRANCH
followed by merging rev 46647 to the new branch.
set_exc_info(), reset_exc_info(): By exploiting the
likely (who knows?) invariant that when an exception's
`type` is NULL, its `value` and `traceback` are also NULL,
save some cycles in heavily-executed code.
This is a "a kronar saved is a kronar earned" patch: the
speedup isn't reliably measurable, but it obviously does
reduce the operation count in the normal (no exception
raised) path through PyEval_EvalFrameEx().
The tim-exc_sanity branch tries to push this harder, but
is still blowing up (at least in part due to pre-existing
subtle bugs that appear to have no other visible
consequences!).
Not a bugfix candidate.
invalid file paths for the built-in import machinery which leads to
fewer open calls on startup.
Also fix issue with PEP 302 style import hooks which lead to more open()
calls than necessary.
both mystrtoul.c and longobject.c. Share the table instead. Also
cut its size by 64 entries (they had been used for an inscrutable
trick originally, but the code no longer tries to use that trick).
Georg Brandl
Neal Norwitz
Christian Heimes
Raymond Hettinger
Martin v. Löwis
Eric Smith
David Wolever
Travis E. Oliphant
Guido van Rossum
Brett Cannon
Nick Coghlan
Amaury Forgeot d'Arc
Jeffrey Yasskin
Barry Warsaw
Kristján Valur Jónsson
Mark Dickinson
Gregory P. Smith
Andrew M. Kuchling
Skip Montanaro
Thomas Wouters
Alex Martelli
Walter Dörwald
Collin Winter
Jeremy Hylton
Anthony Baxter
Armin Rigo
Tim Peters
Thomas Heller
Phillip J. Eby
Andrew MacIntyre
Neil Schemenauer
Hye-Shik Chang
Richard Jones
Fredrik Lundh