This patch publishes the work done until now
for Python 3.0 compatibility. Still a lot
to be done.
When possible, we use 3.0 features in Python 2.6,
easing development and testing, and exposing internal
changes to a wider audience, for better test coverage.
Some mode details:
http://www.jcea.es/programacion/pybsddb.htm#bsddb3-4.7.2
was not always being done properly in some python types and extension
modules. PyMem_MALLOC, PyMem_REALLOC, PyMem_NEW and PyMem_RESIZE have
all been updated to perform better checks and places in the code that
would previously leak memory on the error path when such an allocation
failed have been fixed.
process rather than both parent and child.
Does anyone actually use fork1()? It appears to be a Solaris thing
but if Python is built with pthreads on Solaris, fork1() and fork()
should be the same.
When a thread touches such an object for the first time, a new thread-local __dict__ is created,
and the __init__ method is run.
But a thread switch can occur here; if the other thread touches the same object, it installs another
__dict__; when the first thread resumes, it updates the dictionary of the second...
This is the deep cause of the failures in test_multiprocessing involving "managers" objects.
Also a 2.5 backport candidate.
behaviours. I left the original test commented out (note
that that test came from #2702, which seems to have a
problem in FreeBSD and Windows, but not in Linux).
I included a new test, to watch over the now-broken
behaviour, I took it from #3179.
Some functions in the msvcrt module are skipped,
and socket.ioctl is enabled only when using a more recent Platform SDK.
(and yes, there are still companies that use a 10-years old compiler)
Added checks for integer overflows, contributed by Google. Some are
only available if asserts are left in the code, in cases where they
can't be triggered from Python code.
occurances of PyBytes_ in the code to their original PyString_ names. The
bytesobject.c file will be renamed back to stringobject.c in a future checkin.
ctypes maintains thread-local storage that has space for two error
numbers: private copies of the system 'errno' value and, on Windows,
the system error code accessed by the GetLastError() and
SetLastError() api functions.
Foreign functions created with CDLL(..., use_errno=True), when called,
swap the system 'errno' value with the private copy just before the
actual function call, and swapped again immediately afterwards. The
'use_errno' parameter defaults to False, in this case 'ctypes_errno'
is not touched.
On Windows, foreign functions created with CDLL(...,
use_last_error=True) or WinDLL(..., use_last_error=True) swap the
system LastError value with the ctypes private copy.
The values are also swapped immeditately before and after ctypes
callback functions are called, if the callbacks are constructed using
the new optional use_errno parameter set to True: CFUNCTYPE(...,
use_errno=TRUE) or WINFUNCTYPE(..., use_errno=True).
New ctypes functions are provided to access the ctypes private copies
from Python:
- ctypes.set_errno(value) and ctypes.set_last_error(value) store
'value' in the private copy and returns the previous value.
- ctypes.get_errno() and ctypes.get_last_error() returns the current
ctypes private copies value.
This patch adds a new configure argument on OSX:
--with-universal-archs=[32-bit|64-bit|all]
When used with the --enable-universalsdk option this controls which
CPU architectures are includes in the framework. The default is 32-bit,
meaning i386 and ppc. The most useful alternative is 'all', which includes
all 4 CPU architectures supported by MacOS X (i386, ppc, x86_64 and ppc64).
This includes limited support for the Carbon bindings in 64-bit mode as well,
limited because (a) I haven't done extensive testing and (b) a large portion
of the Carbon API's aren't available in 64-bit mode anyway.
I've also duplicated a feature of Apple's build of python: setting the
environment variable 'ARCHFLAGS' controls the '-arch' flags used for building
extensions using distutils.
convention that allows safe access to errno)
This code does not yet work on OS X (__thread storage specifier not
available), so i needs a configure check plus a more portable
solution.
errno (and LastError, on Windows).
ctypes maintains a module-global, but thread-local, variable that
contains an error number; called 'ctypes_errno' for this discussion.
This variable is a private copy of the systems 'errno' value; the copy
is swapped with the 'errno' variable on several occasions.
Foreign functions created with CDLL(..., use_errno=True), when called,
swap the values just before the actual function call, and swapped
again immediately afterwards. The 'use_errno' parameter defaults to
False, in this case 'ctypes_errno' is not touched.
The values are also swapped immeditately before and after ctypes
callback functions are called, if the callbacks are constructed using
the new optional use_errno parameter set to True: CFUNCTYPE(..., use_errno=TRUE)
or WINFUNCTYPE(..., use_errno=True).
Two new ctypes functions are provided to access the 'ctypes_errno'
value from Python:
- ctypes.set_errno(value) sets ctypes_errno to 'value', the previous
ctypes_errno value is returned.
- ctypes.get_errno() returns the current ctypes_errno value.
---
On Windows, the same scheme is implemented for the error value which
is managed by the GetLastError() and SetLastError() windows api calls.
The ctypes functions are 'ctypes.set_last_error(value)' and
'ctypes.get_last_error()', the CDLL and WinDLL optional parameter is
named 'use_last_error', defaults to False.
---
On Windows, TlsSetValue and TlsGetValue calls are used to provide
thread local storage for the variables; ctypes compiled with __GNUC__
uses __thread variables.
* Expand comments.
* Swap variable names in the sum_exact code so that x and y
are consistently chosen as the larger and smaller magnitude
values respectively.
Renamed copy_reg to copyreg in the standard library, to avoid
spurious warnings and ease later merging to py3k branch. Public
documentation remains intact.
characters. This avoids the common case of something like 'NUMBER(10)' not
being parsed as 'NUMBER', like expected. Also corrected the docs about
converter names being case-sensitive. They aren't any longer.
cmath module is loaded) instead of statically. This fixes compile-time
problems on platforms where HUGE_VAL is an extern variable rather than
a constant.
Thanks Hirokazu Yamamoto for the patch.
The patch also adds acosh, asinh, atanh, log1p and copysign to all platforms. Finally it fixes differences between platforms like different results or exceptions for edge cases. Have fun :)
'warnings' code in places where it was previously not possible (e.g., the
parser). It could also potentially lead to a speed-up in interpreter start-up
if the C version of the code (_warnings) is imported over the use of the
Python version in key places.
Closes issue #1631171.
-J and -X are now reserved for Jython and non-standard arguments (e.g. IronPython). I've added some extra comments to make sure the reservation don't get missed in the future.
It was only used as a helper in types.py to access types (GetSetDescriptorType and MemberDescriptorType),
when they can easily be obtained with python code.
These expressions even work with Jython.
I don't know what the future of the types module is; (cf. discussion in http://bugs.python.org/issue1605 )
at least this change makes it simpler.
I applied the same changes manually to VS7.1 and VC6 files; completely untested.
(Christian, don't try too hard merging this change into py3k.
It will be easier to do the same work again on the branch)
svn+ssh://pythondev@svn.python.org/python/branches/trunk-bytearray
........
r61750 | christian.heimes | 2008-03-22 20:47:44 +0100 (Sat, 22 Mar 2008) | 1 line
Copied files from py3k w/o modifications
........
r61752 | christian.heimes | 2008-03-22 20:53:20 +0100 (Sat, 22 Mar 2008) | 7 lines
Take One
* Added initialization code, warnings, flags etc. to the appropriate places
* Added new buffer interface to string type
* Modified tests
* Modified Makefile.pre.in to compile the new files
* Added bytesobject.c to Python.h
........
r61754 | christian.heimes | 2008-03-22 21:22:19 +0100 (Sat, 22 Mar 2008) | 2 lines
Disabled bytearray.extend for now since it causes an infinite recursion
Fixed serveral unit tests
........
r61756 | christian.heimes | 2008-03-22 21:43:38 +0100 (Sat, 22 Mar 2008) | 5 lines
Added PyBytes support to several places:
str + bytearray
ord(bytearray)
bytearray(str, encoding)
........
r61760 | christian.heimes | 2008-03-22 21:56:32 +0100 (Sat, 22 Mar 2008) | 1 line
Fixed more unit tests related to type('') is not unicode
........
r61763 | christian.heimes | 2008-03-22 22:20:28 +0100 (Sat, 22 Mar 2008) | 2 lines
Fixed more unit tests
Fixed bytearray.extend
........
r61768 | christian.heimes | 2008-03-22 22:40:50 +0100 (Sat, 22 Mar 2008) | 1 line
Implemented old buffer interface for bytearray
........
r61772 | christian.heimes | 2008-03-22 23:24:52 +0100 (Sat, 22 Mar 2008) | 1 line
Added backport of the io module
........
r61775 | christian.heimes | 2008-03-23 03:50:49 +0100 (Sun, 23 Mar 2008) | 1 line
Fix str assignement to bytearray. Assignment of a str of size 1 is interpreted as a single byte
........
r61805 | christian.heimes | 2008-03-23 19:33:48 +0100 (Sun, 23 Mar 2008) | 3 lines
Fixed more tests
Fixed bytearray() comparsion with unicode()
Fixed iterator assignment of bytearray
........
r61809 | christian.heimes | 2008-03-23 21:02:21 +0100 (Sun, 23 Mar 2008) | 2 lines
str(bytesarray()) now returns the bytes and not the representation of the bytearray object
Enabled and fixed more unit tests
........
r61812 | christian.heimes | 2008-03-23 21:53:08 +0100 (Sun, 23 Mar 2008) | 3 lines
Clear error PyNumber_AsSsize_t() fails
Use CHARMASK for ob_svall access
disabled a test with memoryview again
........
r61819 | christian.heimes | 2008-03-23 23:05:57 +0100 (Sun, 23 Mar 2008) | 1 line
Untested updates to the PCBuild directory
........
r61917 | christian.heimes | 2008-03-26 00:57:06 +0100 (Wed, 26 Mar 2008) | 1 line
The type system of Python 2.6 has subtle differences to 3.0's. I've removed the Py_TPFLAGS_BASETYPE flags from bytearray for now. bytearray can't be subclasses until the issues with bytearray subclasses are fixed.
........
r61920 | christian.heimes | 2008-03-26 01:44:08 +0100 (Wed, 26 Mar 2008) | 2 lines
Disabled last failing test
I don't understand what the test is testing and how it suppose to work. Ka-Ping, please check it out.
........
r61930 | christian.heimes | 2008-03-26 12:46:18 +0100 (Wed, 26 Mar 2008) | 1 line
Re-enabled bytes warning code
........
r61933 | christian.heimes | 2008-03-26 13:20:46 +0100 (Wed, 26 Mar 2008) | 1 line
Fixed a bug in the new buffer protocol. The buffer slots weren't copied into a subclass.
........
r61934 | christian.heimes | 2008-03-26 13:25:09 +0100 (Wed, 26 Mar 2008) | 1 line
Re-enabled bytearray subclassing - all tests are passing.
........
2.x signed value. Also, don't waste space on a table full of unsigned longs
when all it needs are unsigned ints (incase anyone builds this without zlib on
a 64-bit unix for some strange reason).
tested by forcing it to compile this version on both 32-bit and 64-bit linux.
platforms with a 64-bit long.
The Alpha/Tru64 test problem is a problem in either tarfile or test_tarfile,
not zlib.
crc32 and adler32 return 32-bit values. by using a long thats larger than
32-bits in these functions they were prevented from wrapping around to their
signed 32-bit value that we want them to return in python 2.x.
which made the return value signed. On the Alpha that also lost data
since sizeof(int) != sizeof(long) and apparently adler32/crc32 return
64 bits of data. This change keeps the signedness and continues to store the
data in a long rather than an int as was the case before r61449.
The patch adds wrappers for the Linux epoll syscalls and the BSD kqueue syscalls. Thanks to Thomas Herve and the Twisted people for their support and help.
TODO: Finish documentation documentation
when used on platforms that actually define ioctl as taking an unsigned long.
(the BSDs and OS X / Darwin)
Adds a unittest for fcntl.ioctl that tests what happens with both positive and
negative numbers.
This was done because of issue1471 but I'm not able to reproduce -that- problem
in the first place on Linux 32bit or 64bit or OS X 10.4 & 10.5 32bit or 64 bit.
uid and gid input to accept values >=2**31 as valid while still accepting
negative numbers to pass -1 to chown for "no change".
Fixes issue1747858.
This should be backported to release25-maint.
regardless of the native sizeof(long) used in the integer object.
This somewhat odd behavior of returning a signed is maintained in 2.x for
compatibility reasons of always returning an integer rather than a long object.
Fixes Issue1202 for Python 2.6
The bundled libffi copy is now in sync with the recently released
libffi3.0.4 version, apart from some small changes to
Modules/_ctypes/libffi/configure.ac.
I gave up on using libffi3 files on os x.
Instead, static configuration with files from pyobjc is used.
My tests don't show the promised speed up of 10%. The code is as fast as the old code for simple cases and slightly faster for complex cases with several of args and kwargs. But the patch simplifies the code, too.
were two module_methods and the one used depended on the order the
modules were loaded. By making the test module_methods static,
it is not exported and the correct version is picked up.
* Fix-up issues pointed-out by Neal Norwitz.
* Add extensive comments.
* The lz->result variable is now a tuple instead of a list.
* Use fast macro getitem/setitem calls so most code is in-line.
* Re-use the result tuple if available (modify in-place instead of copy).
Highlights:
- Adding PyObject_Format.
- Adding string.Format class.
- Adding __format__ for str, unicode, int, long, float, datetime.
- Adding builtin format.
- Adding ''.format and u''.format.
- str/unicode fixups for formatters.
The files in Objects/stringlib that implement PEP 3101 (stringdefs.h,
unicodedefs.h, formatter.h, string_format.h) are identical in trunk
and py3k. Any changes from here on should be made to trunk, and
changes will propogate to py3k).
function can be called recursively.
This was discussed in issue1020188.
In python codebase, all occurrences of Py_[X]DECREF(xxx->yyy) are suspect,
except when they appear in tp_new or tp_dealloc functions, or when
the member cannot be of a user-defined class.
Note that tp_init is not safe.
I do have a (crashing) example for every changed line.
Is it worth adding them to the test suite?
Example:
class SpecialStr(str):
def __del__(self):
s.close()
import cStringIO
s = cStringIO.StringIO(SpecialStr("text"))
s.close() # Segfault
First chapter of the Python 3.0 io framework back port: _fileio
The next step depends on a working bytearray type which itself depends on a backport of the nwe buffer API.
from the WeakValueDictionary was slower by nearly a factor of 3.
To avoid leaks, weakref proxies for the array types are put into the
cache dict, with weakref callbacks that removes the entries when the
type goes away.
a default timeout has been set, by using getsockopt() to get the error
condition (instead of trying another connect() call, which seems to be
a Linuxism).
2.5 bugfix candidate, assuming no one reports any problems with this change.
var_decl: Declared variable "stm" without initializer
ninit_use_in_call: Using uninitialized value "stm" (field "stm".tm_zone uninitialized) in call to function "mktime"
ctypes takes some liberties when creating python types: it modifies the types'
__dict__ directly, bypassing all the machinery of type objects which deal with
special methods. And this broke recent optimisations of method lookup.
Now we try to modify the type with more "official" functions.
SRE_MATCH so that signal handlers can be invoked during
long regular expression matches. It also adds a new
error return value indicating that an exception
occurred in a signal handler during the match, allowing
exceptions in the signal handler to propagate up to the
main loop. Thanks Josh Hoyt and Ralf Schmitt.
This prevents traceback prints pointing to files in this way:
File "\loewis\25\python\Modules\_ctypes\callbacks.c", line 206, in 'calling callback function'
round included:
* Revert round to its 2.6 behavior (half away from 0).
* Because round, floor, and ceil always return float again, it's no
longer necessary to have them delegate to __xxx___, so I've ripped
that out of their implementations and the Real ABC. This also helps
in implementing types that work in both 2.6 and 3.0: you return int
from the __xxx__ methods, and let it get enabled by the version
upgrade.
* Make pow(-1, .5) raise a ValueError again.
the complex_pow part), r56649, r56652, r56715, r57296, r57302, r57359, r57361,
r57372, r57738, r57739, r58017, r58039, r58040, and r59390, and new
documentation. The only significant difference is that round(x) returns a float
to preserve backward-compatibility. See http://bugs.python.org/issue1689.
This adds signal.set_wakeup_fd(fd) which sets a file descriptor to
which a zero byte will be written whenever a C exception handler runs.
I added a simple C API as well, PySignal_SetWakeupFd(fd).
- Backported a workaround for a bug in SQLite 3.2.x/3.3.x versions where a
statement recompilation with no bound parameters lead to a segfault
- Backported a fix necessary because of an SQLite API change in version
3.5.
This prevents segfaults when executing empty queries, like our test suite
does
on 32-bit systems on 64-bit systems, and vice versa. As a consequence
of the change, Random pickles created by Python 2.6 cannot be loaded
in Python 2.5.
The new msvc9compiler module supports VS 2005 and VS 2008. I've also fixed build_ext to support PCbuild8 and PCbuild9 and backported my fix for xxmodule.c from py3k. The old code msvccompiler is still in place in case somebody likes to build an extension with VS 2003 or earlier.
I've also updated the cygwin compiler module for VS 2005 and VS 2008. It works with VS 2005 but I'm unable to test it with VS 2008. We have to wait for a new version of cygwin.
Georg Brandl has added fchmod() and fchown(). I've contributed lchown but I'm not able to test it on Linux. However it should be available on Mac and some other flavors of Unix.
I've made a quick test of fchmod() and fchown() on my system. They are working as expected.
I've finished the last task for the PCbuild9 directory today. I don't think there is much left to do. Now you can all play around with the shiny new VS 2008 and try the PGO builds. I was able to get a speed improvement of about 10% on py3k.
Have fun! :)
Currently on Windows set_error() make use of a large array which maps
socket error numbers to error messages.
This patch removes that array and just lets PyErr_SetExcFromWindowsErr()
generate the message by using the Win32 function FormatMessage().
fixed indention to tabs
use Py_RETURN_NONE macro
added more error checks to on_completion_display_matches_hook
open question: Does PyList_SetItem(l, i, o) steal a reference to o in the case of an error?
string key (and probably a few other situations with string keys).
This was reported with a patch as pybsddb sourceforge bug 1708868 by
jjjhhhlll at gmail.
This code was broken if save() returned a negative number since i contained
a boolean value and then we compared i < 0 which should never be true.
Will backport (assuming it's necessary)
Add maxlen support to deque() and fixup docs.
Partially fix __reduce__(). The None as a third arg was no longer supported.
Still needs work on __reduce__() to handle recursive inputs.
adding the 'makefile' method to ssl.SSLSocket, and importing the
requisite fakefile class from socket.py, and making the appropriate
changes to it to make it use the SSL connection.
Added sample HTTPS server to test_ssl.py, and test that uses it.
Change SSL tests to use https://svn.python.org/, instead of
www.sf.net and pop.gmail.com.
Added utility function to ssl module, get_server_certificate,
to wrap up the several things to be done to pull a certificate
from a remote server.
Make sure the type of the return value of re.sub(x, y, z) is the type
of y+x (i.e. unicode if either is unicode, str if they are both str)
even if there are no substitutions or if x==z (which triggered various
special cases in join_list()).
Could be backported to 2.5; no need to port to 3.0.
* Much expanded test suite:
All protocols tested against all other protocols.
All protocols tested with all certificate options.
Tests for bad key and bad cert.
Test of STARTTLS functionality.
Test of RAND_* functions.
* Fixes for threading/malloc bug.
* Issue 1065 fixed:
sslsocket class renamed to SSLSocket.
sslerror class renamed to SSLError.
Function "wrap_socket" now used to wrap an existing socket.
* Issue 1583946 finally fixed:
Support for subjectAltName added.
Subject name now returned as proper DN list of RDNs.
* SSLError exported from socket as "sslerror".
* RAND_* functions properly exported from ssl.py.
* Documentation improved:
Example of how to create a self-signed certificate.
Better indexing.
alone class. This addresses the primary concern in
http://bugs.python.org/issue1706815
python-dev discussion here:
http://mail.python.org/pipermail/python-dev/2007-July/073749.html
I chose IOError rather than EnvironmentError as the base class since
socket objects are often used as transparent duck typed file objects
in code already prepared to deal with IOError exceptions.
also a minor fix:
urllib2 - fix a couple places where IOError was raised rather than URLError.
for better or worse, URLError already inherits from IOError so
this won't break any existing code.
test_urllib2net - replace bad ftp urls.
The exact behaviour of omitted and negative indices for the Pointer type may
need a closer look (especially as it's subtly different from simple slices)
but there's time yet before 2.6, and not enough before 3.0a1 :-)
- Specialcase extended slices that amount to a shallow copy the same way as
is done for simple slices, in the tuple, string and unicode case.
- Specialcase step-1 extended slices to optimize the common case for all
involved types.
- For lists, allow extended slice assignment of differing lengths as long
as the step is 1. (Previously, 'l[:2:1] = []' failed even though
'l[:2] = []' and 'l[:2:None] = []' do not.)
- Implement extended slicing for buffer, array, structseq, mmap and
UserString.UserString.
- Implement slice-object support (but not non-step-1 slice assignment) for
UserString.MutableString.
- Add tests for all new functionality.
> returning NULL, and other pieces of the code call PySSL_SetError,
> which creates the error string. I think some of the places which set
> the string directly probably shouldn't; instead, they should call
> PySSL_SetError to cons up the error name directly from the err code.
> However, PySSL_SetError only works after the construction of an ssl
> object, which means it can't be used there... I'll take a longer look
> at it and see if there's a reasonable fix.
Here's a patch which addresses this. It also fixes the indentation in
PySSL_SetError, bringing it into line with PEP 7, fixes a compile warning
about one of the OpenSSL macros, and makes the namespace a bit more
consistent. I've tested it on FC 7 and OS X 10.4.
% ./python ./Lib/test/regrtest.py -R :1: -u all test_ssl
test_ssl
beginning 6 repetitions
123456
......
1 test OK.
[29244 refs]
%
[GvR: slightly edited to enforce 79-char line length, even if it required
violating the style guide.]
Neal Norwitz
Mark Dickinson
Thomas Heller
Raymond Hettinger
Georg Brandl
Jesus Cea
Gregory P. Smith
Amaury Forgeot d'Arc
Bob Ippolito
Nick Coghlan
Alexandre Vassalotti
Robert Schuppenies
Facundo Batista
Bill Janssen
Andrew M. Kuchling
Benjamin Peterson
Martin v. Löwis
Travis E. Oliphant
Ronald Oussoren
Walter Dörwald
Gerhard Häring
Christian Heimes
Brett Cannon
Skip Montanaro
Guido van Rossum
David Wolever
Eric Smith
Hye-Shik Chang
Andrew MacIntyre
Jeffrey Yasskin
Armin Rigo
Sean Reifscheider
Matthias Klose
Thomas Wouters