The problem is that _DB_get_type() can't be called without the GIL
because it calls a bunch of PyErr_* APIs when an error occurs.
There were no other cases in this file that it was called without the GIL.
Removing the BEGIN/END THREAD around _DB_get_type() made everything work.
Will backport.
some warnings from Klokwork. They verify the assumptions of the format
of svn version output.
The assert in the thread module helped debug a problem on HP-UX.
Check this and raise an error when something else is used - before
this change ctypes would hang or crash when such a callback was
called. This is a partial fix for #1574584.
Will backport to release25-maint.
* unified the way intobject, longobject and mystrtoul handle
values around -sys.maxint-1.
* in general, trying to entierely avoid overflows in any computation
involving signed ints or longs is extremely involved. Fixed a few
simple cases where a compiler might be too clever (but that's all
guesswork).
* more overflow checks against bad data in marshal.c.
* 2.5 specific: fixed a number of places that were still confusing int
and Py_ssize_t. Some of them could potentially have caused
"real-world" breakage.
* list.pop(x): fixing overflow issues on x was messy. I just reverted
to PyArg_ParseTuple("n"), which does the right thing. (An obscure
test was trying to give a Decimal to list.pop()... doesn't make
sense any more IMHO)
* trying to write a few tests...
if an input hook has been defined. Patch by Richard Boulton.
This select() code is only executed with readline 2.1, or if
READLINE_CALLBACKS is defined.
Backport candidate for 2.5, 2.4, probably earlier versions too.
- gbk and gb18030 codec now handle U+30FB KATAKANA MIDDLE DOT correctly.
- iso2022_jp_2 codec now encodes into G0 for KS X 1001, GB2312
codepoints to conform the standard.
- iso2022_jp_3 and iso2022_jp_2004 codec can encode JIS X 2013:2
codepoints now.
In one place where I'n not 1000% sure about the non-NULL, raise
a RuntimeError for safety.
This should fix the klocwork issues that Neal sent me. If so,
it should be applied to the release25-maint branch also.
of the Python part of a callback function to C. If it cannot be
converted, call PyErr_WriteUnraisable with the exception we got.
Before, arbitrary data has been passed to the calling C code in this
case.
(I'm not really sure the NEWS entry is understandable, but I cannot
find better words)
PyObject_stgdict() and PyType_stgdict() calls are non-NULL before
dereferencing the result. Hopefully this fixes what klocwork is
complaining about.
Fix a few other nits as well.
ctypes instances no longer have the internal and undocumented
'_as_parameter_' attribute which was used to adapt them to foreign
function calls; this mechanism is replaced by a function pointer in
the type's stgdict.
In the 'from_param' class methods, try the _as_parameter_ attribute if
other conversions are not possible.
This makes the documented _as_parameter_ mechanism work as intended.
Change the ctypes version number to 1.0.1.
I modified this patch some by fixing style, some error checking, and adding
XXX comments. This patch requires review and some changes are to be expected.
I'm checking in now to get the greatest possible review and establish a
baseline for moving forward. I don't want this to hold up release if possible.
This is the first batch of fixes that should be easy to verify based on context.
This fixes problem numbers: 220 (ast), 323-324 (symtable),
321-322 (structseq), 215 (array), 210 (hotshot), 182 (codecs), 209 (etree).
storage before converting the result to C data. See the comment in
the code for details.
Provide a better context for errors when the conversion of a callback
function's result cannot be converted.
This provides the proper warning for struct.pack().
PyErr_Warn() is now deprecated in favor of PyErr_WarnEx().
As mentioned by Tim Peters on python-dev.
We didn't notice these before because test_defaultdict didn't
actually do anything before Georg fixed that earlier today.
Neal's next refleak run then showed test_defaultdict leaking
9 references on each run. That's repaired by this checkin.
inspect.py, and pydoc.py. Specifically, this allows for querying the type of
an object against these built-in C types and more importantly, for getting
their docstrings printed in the interactive interpreter's help() function.
This patch includes a new built-in module called _types which provides
definitions of getset and member descriptors for use by the types.py module.
These types are exposed as types.GetSetDescriptorType and
types.MemberDescriptorType. Query functions are provided as
inspect.isgetsetdescriptor() and inspect.ismemberdescriptor(). The
implementations of these are robust enough to work with Python implementations
other than CPython, which may not have these fundamental types.
The patch also includes documentation and test suite updates.
I commit these changes now under these guiding principles:
1. Silence is assent. The release manager has not said "no", and of the few
people that cared enough to respond to the thread, the worst vote was "0".
2. It's easier to ask for forgiveness than permission.
3. It's so dang easy to revert stuff in svn, that you could view this as a
forcing function. :)
Windows build patches will follow.
it definitely could use some review to ensure I'm not off by one
and there's no possible overflow/wrap-around of bytes_left.
Reported by Klocwork #1.
Fix a problem if there is a failure allocating self->db.
Found with failmalloc.
Also fix a few memory leaks in other failure scenarios.
It seems that if b_objects == Py_None, we will have an extra ref to
b_objects. Add XXX comment so hopefully someone documents why the
else isn't necessary or adds it in.
Reported by Klocwork #20
of values in the time tuple passed in. Unfortunately people came to rely on
undocumented behaviour of setting unneeded values to 0, regardless of if it was
within the valid range. Now those values force the value internally to the
minimum value when 0 is passed in.
Add a new function uses_seh() to the _ctypes extension module. This
will return True if Windows Structured Exception handling (SEH) is
used when calling functions, False otherwise.
of bugs in that port. The most annoying ones were due to some subtle differences
between the document ABI and the actual implementation :-(
(there are no python unittests that fail without this patch, but without it
some of libffi's unittests fail).
will return True if Windows Structured Exception handling (SEH) is
used when calling functions, False otherwise.
Currently, only MSVC supports SEH.
Fix the test so that it doesn't crash when run with MingW compiled
_ctypes. Note that two tests are still failing when mingw is used, I
suspect structure layout differences and function calling conventions
between MSVC and MingW.
It seems that the definition of '__attribute__(x)' was responsible for
the compiler ignoring the '__fastcall' attribute on the
ffi_closure_SYSV function in libffi_msvc/ffi.c, took me quite some
time to figure this out.
handler would cause a segfault. This merges in Expat's lib/xmlparse.c
revisions 1.154 and 1.155, which fix this and a closely related problem
(the later does not affect Python).
Moved the crasher test to the tests for xml.parsers.expat.
This backs out the test changes in 46962 which prevented crashes
by not running the tests via a version check. All the version checks
added in that rev were removed from the tests.
Code was added to the error handler in connection.c that seems
to work with older versions of sqlite including 3.1.3.
SQLite versions.
- Added version checks in test suite so that we don't execute tests that we
know will fail with older (buggy) SQLite versions.
Now, all tests should run against all SQLite versions from 3.0.8 until 3.3.6
(latest one now). The sqlite3 module can be built against all these SQLite
versions and the sqlite3 module does its best to not trigger bugs in SQLite,
but using SQLite 3.3.3 or later is recommended.
Heavily revised, comprising revisions:
46640 - original trunk revision (backed out in r46655)
46647 - markup fix (backed out in r46655)
46692:46918 merged from branch aimacintyre-sf1454481
branch tested on buildbots (Windows buildbots had problems
not related to these changes).
Summary of changes:
- support for 'variable sized' data
- support for anonymous structure/union fields
- fix severe bug with certain arrays or structures containing more than 256 fields
the char buffer was requested. Now it actually returns the char buffer if
available or raises a TypeError if it isn't (as is raised for the other buffer
types if they are not present but requested).
Not a backport candidate since it does change semantics of the buffer object
(although it could be argued this is enough of a bug to bother backporting).
endian machines. Should fix the remaininf failure in the PPC64
Debian buildbot.
Thanks to Matthias Klose for providing access to a machine to debug
and test this.
parameter strings") changed this function's signature
seemingly by mistake, which is causing buildbots to fail
test_bsddb3. Restored the pre-46688 signature.
structmember typecode for Py_ssize_t fields. This should fix some of
the errors on the PPC64 debian machine (64-bit, big endian).
Assigning to readonly fields now raises AttributeError instead of
TypeError, so the testcase has to be changed as well.
in BerkeleyDB >= 4.2 it tried to construct a list out of an uninitialized
char **log_list.
feature: export the DB_ARCH_REMOVE flag by name in the module on BerkeleyDB >= 4.2.
46640 Patch #1454481: Make thread stack size runtime tunable.
46647 Markup fix
The first is causing many buildbots to fail test runs, and there
are multiple causes with seemingly no immediate prospects for
repairing them. See python-dev discussion.
Note that a branch can (and should) be created for resolving these
problems, like
svn copy svn+ssh://svn.python.org/python/trunk -r46640 svn+ssh://svn.python.org/python/branches/NEW_BRANCH
followed by merging rev 46647 to the new branch.
- Following Guido's comments, renamed
* pack_to -> pack_into
* recv_buf -> recv_into
* recvfrom_buf -> recvfrom_into
- Made fixes to _struct.c according to Neal Norwitz comments on the checkins
list.
- Converted some ints into the appropriate -- I hope -- ssize_t and size_t.
Fix by renaming the variable.
In a different module, Neal fixed it by renaming _self to self. There's
already a variable named 'self' here, so I used selfptr.
(I'm committing this on a Mac without Tk, but it's a simple search-and-replace.
<crosses fingers>, so I'll watch the buildbots and see what happens.)
Renames functional extension module to _functools and adds a Python
functools module so that utility functions like update_wrapper can be
added easily.
* Added socket.recv_buf() and socket.recvfrom_buf() methods, that use the buffer
protocol (send and sendto already did).
* Added struct.pack_to(), that is the corresponding buffer compatible method to
unpack_from().
* Fixed minor typos in arraymodule.
QueryPerformanceCounter(), but we believe Win64 does
support it now. So use in time.clock().
It would be peachy if someone with a Win64 box tried
this ;-)
about "%u", "%lu" and "%zu" formats.
Since PyString_FromFormat and PyErr_Format have exactly the same rules
(both inherited from PyString_FromFormatV), it would be good if someone
with more LaTeX Fu changed one of them to just point to the other.
Their docs were way out of synch before this patch, and I just did a
mass copy+paste to repair that.
Not a backport candidate (this is a new feature).
longobject.c: also fix an ssize_t problem
<a> could have been NULL, so hoist the size calc to not use <a>.
_ssl.c: under fail: self is DECREF'd, but it would have been NULL.
_elementtree.c: delete self if there was an error.
_csv.c: I'm not sure if lineterminator could have been anything other than
a string. However, other string method calls are checked, so check this
one too.
discussion.
There are two places of documentation that still mention __context__:
Doc/lib/libstdtypes.tex -- I wasn't quite sure how to rewrite that without
spending a whole lot of time thinking about it; and whatsnew, which Andrew
usually likes to change himself.
The problem was that pyconfig.h was being included before some system headers
which caused redefinitions and other breakage. This moves system headers
after expat_config.h which includes pyconfig.h.
After the patch (45590) to add extra debug stats to the gc module, Python
was crashing on OpenBSD due to:
Fatal Python error: Interpreter not initialized (version mismatch?)
This seems to occur due to calling collect() when initialized (in pythonrun.c)
is set to 0. Now, the import will occur in the init function which
shouldn't suffer this problem.
MAXPATHLEN-sized buffers for various output-buffers (like to realpath()),
and that's correct on BSD platforms, but not Linux (which uses PATH_MAX, and
does not define MAXPATHLEN.) Cursory googling suggests Linux is following a
newer standard than BSD, but in cases like this, who knows. Using the
greater of PATH_MAX and 1024 as a fallback for MAXPATHLEN seems to be the
most portable solution.
AF_PACKET cases in getsockaddrarg were missing their own checks for
tuple-ness of the address argument, which means a confusing SystemError was
raised by PyArg_ParseTuple instead.
The new char-array used in ioctl calls wasn't explicitly NUL-terminated;
quite probably the cause for the test_pty failures on Solaris that we
circumvented earlier. (I wasn't able to reproduce it with this patch, but it
has been somewhat elusive to start with.)
This patch checks if poll is broken when the select module is loaded instead
of doing so at configure-time. This functionality is only active on Mac OS X.
This patch causes several symbols in the socket and posix module to be weakly
linked on OSX and disables usage of ftime on OSX. These changes make it possible
to use a binary build on OSX 10.4 on a 10.3 system.
I tested this with valgrind on amd64.
The man pages I found for diff architectures are inconsistent on this.
I'm not entirely sure this change is correct for all architectures either.
Perhaps we should just over-allocate and not worry about it?
the StgDictObject's ffi_type member had the same name as its type. I
changed that to ffi_type_pointer. Feel free to change it to something else
more meaningful, just not ffi_type.
not be tracked by GC. This fixes 254 of test_generators' refleaks on my
machine, but I'm sure something else will make them come back :>
Not adding a separate test for this kind of cycle, since the existing
fib/m235 already test them in more extensive ways than any 'minimal' test
has been able to manage.
using a custom, nearly-identical macro. This probably changes how some of
these functions are compiled, which may result in fractionally slower (or
faster) execution. Considering the nature of traversal, visiting much of the
address space in unpredictable patterns, I'd argue the code readability and
maintainability is well worth it ;P
PySequence_GetItem of the time.strptime() result. Not a high probability
bug, but not inconceivable either, considering people can provide their own
'time' module.
that are suspended outside of any try/except/finally blocks to be
garbage collected even if they are part of a cycle. Generators that
suspend inside of an active try/except or try/finally block (including
those created by a ``with`` statement) are still not GC-able if they
are part of a cycle, however.
If RTLD_LOCAL is not #defined in any header file (Windows), set it to 0.
If RTLD_GLOBAL is not #defined, set it equal to RTLD_LOCAL.
This should fix ctypes on cygwin.
of SQLite3 from 3.2.2 to 3.0.8, by providing an alternative to
sqlite3_transfer_bindings. setup.py also handles the common (in debian
and ubuntu, at least) case of a buggy sqlite3.h SQLITE_VERSION_NUMBER.
This is based on pysqlite2.1.3, and provides a DB-API interface in
the standard library. You'll need sqlite 3.2.2 or later to build
this - if you have an earlier version, the C extension module will
not be built.
glibc, for example, does this already on its own, but it seems that
the solaris libc doesn't. This leads to Python code being able to over-
write file contents even though having specified "a" mode.
objimpl.h, pymem.h: Stop mapping PyMem_{Del, DEL} and PyMem_{Free, FREE}
to PyObject_{Free, FREE} in a release build. They're aliases for the
system free() now.
_subprocess.c/sp_handle_dealloc(): Since the memory was originally
obtained via PyObject_NEW, it must be released via PyObject_FREE (or
_DEL).
pythonrun.c, tokenizer.c, parsermodule.c: I lost count of the number of
PyObject vs PyMem mismatches in these -- it's like the specific
function called at each site was picked at random, sometimes even with
memory obtained via PyMem getting released via PyObject. Changed most
to use PyObject uniformly, since the blobs allocated are predictably
small in most cases, and obmalloc is generally faster than system
mallocs then.
If extension modules in real life prove as sloppy as Python's front
end, we'll have to revert the objimpl.h + pymem.h part of this patch.
Note that no problems will show up in a debug build (all calls still go
thru obmalloc then). Problems will show up only in a release build, most
likely segfaults.
Don't use SEH when compiling wth mingw.
Use IS_INTRESOURCE to determine function name from function ordinal.
Rewrite the code that allocates and frees callback functions, hopefully
this avoids the coverty warnings: Remove the THUNK typedef, and move the
definition of struct ffi_info into the header file.
as diagnosed by Nick Coghlan.
test_capi.py: A test module should never spawn a thread as
a side effect of being imported. Because this one did, the
segfault one of its thread tests caused didn't occur until
a few tests after test_regrtest.py thought test_capi was
finished. Repair that. Also join() the thread spawned
at the end, so that test_capi is truly finished when
regrtest reports that it's done.
_testcapimodule.c test_thread_state(): this spawns a
couple of non-threading.py threads, passing them a PyObject*
argument, but did nothing to ensure that those threads
finished before returning. As a result, the PyObject*
_could_ (although this was unlikely) get decref'ed out of
existence before the threads got around to using it.
Added explicit synchronization (via a Python mutex) so
that test_thread_state can reliably wait for its spawned
threads to finish.
This was a fair amount of rework of the patch. Refactored test_fork1 so it
could be reused by the new tests for wait3/4. Also made them into new style
unittests (derive from unittest.TestCase).
This patch adds a-LAW encoding to audioop and replaces the old
u-LAW encoding/decoding code with the current code from sox.
Possible issues: the code from sox uses int16_t.
Code by Lars Immisch
This will hopefully get rid of some Coverity warnings, be a hint to
developers, and be marginally faster.
Some asserts were added when the type is currently known, but depends
on values from another function.
"""
The attached patch fixes all the ctypes tests so they pass on amd64.
It also fixes several warnings. I'm not sure what else to do with the
patch. Let me know how you want to handle these in the future.
I'm not sure the patch is 100% correct. You will need to decide what
can be 64 bits and what can't. I believe
sq_{item,slice,ass_item,ass_slice} all need to use Py_ssize_t. The
types in ctypes.h may not require all the changes I made. I don't
know how you want to support older version, so I unconditionally
changed the types to Py_ssize_t.
"""
The patch is also in the ctypes SVN repository now, after small
changes to add compatibility with older Python versions.
PyObject_Unicode(). This problem was originally reported from Coverity
and addresses mail on python-dev "checkin r43015".
This inlines the conversion of the string to unicode and cleans
up/simplifies some code at the end of the PyObject_Unicode().
We really need a complete C API test module for all public APIs
and passing good and bad parameter values.
Will backport.
Anyway, this is the changes to the with-statement
so that __exit__ must return a true value in order
for a pending exception to be ignored.
The PEP (343) is already updated.
missing PyObject_Del()'s, simplify some code by using Py_BuildValue()
instead of creating a tuple with items manually, stop clobbering builtin
exceptions in a few places, and guard against NULL-returning functions some
more.
This fixes 117 of the 780 (!?!#%@#$!!) reference leaks in test_bsddb3. I
ain't not done yet, although this review of 5kloc was just the easy part.
an error code, this let `self` leak. This is a disaster
on Windows, since `self` already points to a newly-opened
file object, and it was impossible for Python code to
close the thing since the only reference to it was in a
blob of leaked C memory.
test_hotshot test_bad_sys_path(): This new test provoked
the C bug above. This test passed, but left an open
"@test" file behind, which caused a massive cascade of
bogus test failures in later, unrelated tests on Windows.
Changed the test code to remove the @test file it leaves
behind, which relies on the change above to close that
file first.
to an unsigned int (and back again) on 64-bit machines, even though the
actual value of the Py_ssize_t variable is way below 31 bits. I suspect
compiler-error.
the sentinel value in the main function, rather than the helper. This
function could possibly do with an early-out if any of the helper calls ends
up with a len of 0, but I doubt it really matters (how common are malformed
hangul syllables, really?)
posix__getfullpathname().
In partial answer to the now-deleted XXX comment:
/* XXX(twouters) Why use 'et#' here at all? insize isn't used */
`insize` is an input parameter too, and it was left uninitialized,
leading to seemingly random failures.
non-32bit platforms. Will still only allow 32 bits in a timestamp on Win64,
but at least it won't crash, and it'll work right on platforms where longs
are big enough to contain time_t's.
(A better-working, although conceptually less-right fix would have been to
use Py_ssize_t here, but Martin and Tim won't let me.)
- New semantics for __exit__() -- it must re-raise the exception
if type is not None; the with-statement itself doesn't do this.
(See the updated PEP for motivation.)
- Added context managers to:
- file
- thread.LockType
- threading.{Lock,RLock,Condition,Semaphore,BoundedSemaphore}
- decimal.Context
- Added contextlib.py, which defines @contextmanager, nested(), closing().
- Unit tests all around; bot no docs yet.
- IMPORT_NAME takes an extra argument from the stack: the relativeness of
the import. Only passed to __import__ when it's not -1.
- __import__() takes an optional 5th argument for the same thing; it
__defaults to -1 (old semantics: try relative, then absolute)
- 'from . import name' imports name (be it module or regular attribute)
from the current module's *package*. Likewise, 'from .module import name'
will import name from a sibling to the current module.
- Importing from outside a package is not allowed; 'from . import sys' in a
toplevel module will not work, nor will 'from .. import sys' in a
(single-level) package.
- 'from __future__ import absolute_import' will turn on the new semantics
for import and from-import: imports will be absolute, except for
from-import with dots.
Includes tests for regular imports and importhooks, parser changes and a
NEWS item, but no compiler-package changes or documentation changes.
- The copy module now "copies" function objects (as atomic objects).
- dict.__getitem__ now looks for a __missing__ hook before raising
KeyError.
- Added a new type, defaultdict, to the collections module.
This uses the new __missing__ hook behavior added to dict (see above).
On a box where sizeof(size_t) == 4, C doesn't define
what happens when a size_t value is shifted right by
32 bits, and this caused test_mmap to fail on Windows
in a debug build. So use different code to break
the size apart depending on how large size_t actually
is.
This looks like an illusion, since lots of code in this
module still appears to assume sizes can't be more
than 32 bits (e.g., the internal _GetMapSize() still
returns an int), but at least test_mmap passes again.
has been applied fairly arbitrarily in this module (nsmallest uses
Py_ssize_t, nlargest does not) and it probably deserves a more complete
review. Fixes heapq.nsmallest() always returning the empty list (on
platforms with 64-bit ssize_t/long)
Based on lsprof (patch #1212837) by Brett Rosen and Ted Czotter.
With further editing by Michael Hudson and myself.
History in svn repo: http://codespeak.net/svn/user/arigo/hack/misc/lsprof
* Module/_lsprof.c is the internal C module, Lib/cProfile.py a wrapper.
* pstats.py updated to display cProfile's caller/callee timings if available.
* setup.py and NEWS updated.
* documentation updates in the profiler section:
- explain the differences between the three profilers that we have now
- profile and cProfile can use a unified documentation, like (c)Pickle
- mention that hotshot is "for specialized usage" now
- removed references to the "old profiler" that no longer exists
* test updates:
- extended test_profile to cover delicate cases like recursion
- added tests for the caller/callee displays
- added test_cProfile, performing the same tests for cProfile
* TO-DO:
- cProfile gives a nicer name to built-in, particularly built-in methods,
which could be backported to profile.
- not tested on Windows recently!
is larger than FD_SETSIZE.
This can only be acheived with ulimit -n SOME_NUMBER_BIGGER_THAN_FD_SETSIZE
which is typically only available to root. Since this wouldn't normally
be run in a test (ie, run as root), it doesn't seem too worthwhile to
add a normal test. The bug report has one version of a test. I've
written another. Not sure what the best thing to do is.
Do the check before calling internal_select() because we can't set
an error in between Py_BEGIN_ALLOW_THREADS and Py_END_ALLOW_THREADS.
This seemed the clearest solution, ie handle before calling internal_select()
rather than inside. Plus there is at least one place outside
of internal_select() that needed to be handled.
Will backport.
on both Unix (SVR4 and BSD) and Windows. Restores behaviour of passing -1
for anonymous memory on Unix. Use MAP_ANONYMOUS instead of _ANON since
the latter is deprecated according to Linux (gentoo) man pages.
Should we continue to allow mmap.mmap(0, length) to work on Windows?
0 is a valid fd.
Will backport bugfix portions.
Thomas Heller
Raymond Hettinger
Brett Cannon
Gregory P. Smith
Martin v. Löwis
Walter Dörwald
Andrew M. Kuchling
Neal Norwitz
Georg Brandl
Ronald Oussoren
Tim Peters
Barry Warsaw
Hye-Shik Chang
Armin Rigo
Jack Diederich
Thomas Wouters
Jeremy Hylton
Fredrik Lundh
Bob Ippolito
Matt Fleming
Anthony Baxter
Fred Drake
Kristján Valur Jónsson
Gerhard Häring
Trent Mick
Andrew MacIntyre
Nick Coghlan
Martin Blais
Richard Jones
Guido van Rossum
Skip Montanaro
Phillip J. Eby
Matthias Klose