cpython/Python/marshal.c

1834 lines
50 KiB
C
Raw Normal View History

1991-06-04 16:42:30 -03:00
/* Write Python objects to files and read them back.
This is primarily intended for writing and reading compiled Python code,
even though dicts, lists, sets and frozensets, not commonly seen in
code objects, are supported.
Version 3 of this protocol properly supports circular links
and sharing. */
1991-06-04 16:42:30 -03:00
2006-03-01 19:49:13 -04:00
#define PY_SSIZE_T_CLEAN
#include "Python.h"
1991-06-04 16:42:30 -03:00
#include "longintrepr.h"
#include "code.h"
1991-06-04 16:42:30 -03:00
#include "marshal.h"
#include "../Modules/hashtable.h"
1991-06-04 16:42:30 -03:00
/* High water mark to determine when the marshalled object is dangerously deep
* and risks coring the interpreter. When the object stack gets this deep,
* raise an exception instead of continuing.
* On Windows debug builds, reduce this value.
*/
#if defined(MS_WINDOWS) && defined(_DEBUG)
#define MAX_MARSHAL_STACK_DEPTH 1000
#else
Merged revisions 55407-55513 via svnmerge from svn+ssh://pythondev@svn.python.org/python/branches/p3yk ................ r55413 | fred.drake | 2007-05-17 12:30:10 -0700 (Thu, 17 May 2007) | 1 line fix argument name in documentation; match the implementation ................ r55430 | jack.diederich | 2007-05-18 06:39:59 -0700 (Fri, 18 May 2007) | 1 line Implements class decorators, PEP 3129. ................ r55432 | guido.van.rossum | 2007-05-18 08:09:41 -0700 (Fri, 18 May 2007) | 2 lines obsubmit. ................ r55434 | guido.van.rossum | 2007-05-18 09:39:10 -0700 (Fri, 18 May 2007) | 3 lines Fix bug in test_inspect. (I presume this is how it should be fixed; Jack Diedrich, please verify.) ................ r55460 | brett.cannon | 2007-05-20 00:31:57 -0700 (Sun, 20 May 2007) | 4 lines Remove the imageop module. With imgfile already removed in Python 3.0 and rgbimg gone in Python 2.6 the unit tests themselves were made worthless. Plus third-party libraries perform the same function much better. ................ r55469 | neal.norwitz | 2007-05-20 11:28:20 -0700 (Sun, 20 May 2007) | 118 lines Merged revisions 55324-55467 via svnmerge from svn+ssh://pythondev@svn.python.org/python/trunk ........ r55348 | georg.brandl | 2007-05-15 13:19:34 -0700 (Tue, 15 May 2007) | 4 lines HTML-escape the plain traceback in cgitb's HTML output, to prevent the traceback inadvertently or maliciously closing the comment and injecting HTML into the error page. ........ r55372 | neal.norwitz | 2007-05-15 21:33:50 -0700 (Tue, 15 May 2007) | 6 lines Port rev 55353 from Guido: Add what looks like a necessary call to PyErr_NoMemory() when PyMem_MALLOC() fails. Will backport. ........ r55377 | neal.norwitz | 2007-05-15 22:06:33 -0700 (Tue, 15 May 2007) | 1 line Mention removal of some directories for obsolete platforms ........ r55380 | brett.cannon | 2007-05-15 22:50:03 -0700 (Tue, 15 May 2007) | 2 lines Change the maintainer of the BeOS port. ........ r55383 | georg.brandl | 2007-05-16 06:44:18 -0700 (Wed, 16 May 2007) | 2 lines Bug #1719995: don't use deprecated method in sets example. ........ r55386 | neal.norwitz | 2007-05-16 13:05:11 -0700 (Wed, 16 May 2007) | 5 lines Fix bug in marshal where bad data would cause a segfault due to lack of an infinite recursion check. Contributed by Damien Miller at Google. ........ r55389 | brett.cannon | 2007-05-16 15:42:29 -0700 (Wed, 16 May 2007) | 6 lines Remove the gopherlib module. It has been raising a DeprecationWarning since Python 2.5. Also remove gopher support from urllib/urllib2. As both imported gopherlib the usage of the support would have raised a DeprecationWarning. ........ r55394 | raymond.hettinger | 2007-05-16 18:08:04 -0700 (Wed, 16 May 2007) | 1 line calendar.py gets no benefit from xrange() instead of range() ........ r55395 | brett.cannon | 2007-05-16 19:02:56 -0700 (Wed, 16 May 2007) | 3 lines Complete deprecation of BaseException.message. Some subclasses were directly accessing the message attribute instead of using the descriptor. ........ r55396 | neal.norwitz | 2007-05-16 23:11:36 -0700 (Wed, 16 May 2007) | 4 lines Reduce the max stack depth to see if this fixes the segfaults on Windows and some other boxes. If this is successful, this rev should be backported. I'm not sure how close to the limit we should push this. ........ r55397 | neal.norwitz | 2007-05-16 23:23:50 -0700 (Wed, 16 May 2007) | 4 lines Set the depth to something very small to try to determine if the crashes on Windows are really due to the stack size or possibly some other problem. ........ r55398 | neal.norwitz | 2007-05-17 00:04:46 -0700 (Thu, 17 May 2007) | 4 lines Last try for tweaking the max stack depth. 5000 was the original value, 4000 didn't work either. 1000 does work on Windows. If 2000 works, that will hopefully be a reasonable balance. ........ r55412 | fred.drake | 2007-05-17 12:29:58 -0700 (Thu, 17 May 2007) | 1 line fix argument name in documentation; match the implementation ........ r55427 | neal.norwitz | 2007-05-17 22:47:16 -0700 (Thu, 17 May 2007) | 1 line Verify neither dumps or loads overflow the stack and segfault. ........ r55446 | collin.winter | 2007-05-18 16:11:24 -0700 (Fri, 18 May 2007) | 1 line Backport PEP 3110's new 'except' syntax to 2.6. ........ r55448 | raymond.hettinger | 2007-05-18 18:11:16 -0700 (Fri, 18 May 2007) | 1 line Improvements to NamedTuple's implementation, tests, and documentation ........ r55449 | raymond.hettinger | 2007-05-18 18:50:11 -0700 (Fri, 18 May 2007) | 1 line Fix beginner mistake -- don't mix spaces and tabs. ........ r55450 | neal.norwitz | 2007-05-18 20:48:47 -0700 (Fri, 18 May 2007) | 1 line Clear data so random memory does not get freed. Will backport. ........ r55452 | neal.norwitz | 2007-05-18 21:34:55 -0700 (Fri, 18 May 2007) | 3 lines Whoops, need to pay attention to those test failures. Move the clear to *before* the first use, not after. ........ r55453 | neal.norwitz | 2007-05-18 21:35:52 -0700 (Fri, 18 May 2007) | 1 line Give some clue as to what happened if the test fails. ........ r55455 | georg.brandl | 2007-05-19 11:09:26 -0700 (Sat, 19 May 2007) | 2 lines Fix docstring for add_package in site.py. ........ r55458 | brett.cannon | 2007-05-20 00:09:50 -0700 (Sun, 20 May 2007) | 2 lines Remove the rgbimg module. It has been deprecated since Python 2.5. ........ r55465 | nick.coghlan | 2007-05-20 04:12:49 -0700 (Sun, 20 May 2007) | 1 line Fix typo in example (should be backported, but my maintenance branch is woefully out of date) ........ ................ r55472 | brett.cannon | 2007-05-20 12:06:18 -0700 (Sun, 20 May 2007) | 2 lines Remove imageop from the Windows build process. ................ r55486 | neal.norwitz | 2007-05-20 23:59:52 -0700 (Sun, 20 May 2007) | 1 line Remove callable() builtin ................ r55506 | neal.norwitz | 2007-05-22 00:43:29 -0700 (Tue, 22 May 2007) | 78 lines Merged revisions 55468-55505 via svnmerge from svn+ssh://pythondev@svn.python.org/python/trunk ........ r55468 | neal.norwitz | 2007-05-20 11:06:27 -0700 (Sun, 20 May 2007) | 1 line rotor is long gone. ........ r55470 | neal.norwitz | 2007-05-20 11:43:00 -0700 (Sun, 20 May 2007) | 1 line Update directories/files at the top-level. ........ r55471 | brett.cannon | 2007-05-20 12:05:06 -0700 (Sun, 20 May 2007) | 2 lines Try to remove rgbimg from Windows builds. ........ r55474 | brett.cannon | 2007-05-20 16:17:38 -0700 (Sun, 20 May 2007) | 4 lines Remove the macfs module. This led to the deprecation of macostools.touched(); it completely relied on macfs and is a no-op on OS X according to code comments. ........ r55476 | brett.cannon | 2007-05-20 16:56:18 -0700 (Sun, 20 May 2007) | 3 lines Move imgfile import to the global namespace to trigger an import error ASAP to prevent creation of a test file. ........ r55477 | brett.cannon | 2007-05-20 16:57:38 -0700 (Sun, 20 May 2007) | 3 lines Cause posixfile to raise a DeprecationWarning. Documented as deprecated since Ptyhon 1.5. ........ r55479 | andrew.kuchling | 2007-05-20 17:03:15 -0700 (Sun, 20 May 2007) | 1 line Note removed modules ........ r55481 | martin.v.loewis | 2007-05-20 21:35:47 -0700 (Sun, 20 May 2007) | 2 lines Add Alexandre Vassalotti. ........ r55482 | george.yoshida | 2007-05-20 21:41:21 -0700 (Sun, 20 May 2007) | 4 lines fix against r55474 [Remove the macfs module] Remove "libmacfs.tex" from Makefile.deps and mac/mac.tex. ........ r55487 | raymond.hettinger | 2007-05-21 01:13:35 -0700 (Mon, 21 May 2007) | 1 line Replace assertion with straight error-checking. ........ r55489 | raymond.hettinger | 2007-05-21 09:40:10 -0700 (Mon, 21 May 2007) | 1 line Allow all alphanumeric and underscores in type and field names. ........ r55490 | facundo.batista | 2007-05-21 10:32:32 -0700 (Mon, 21 May 2007) | 5 lines Added timeout support to HTTPSConnection, through the socket.create_connection function. Also added a small test for this, and updated NEWS file. ........ r55495 | georg.brandl | 2007-05-21 13:34:16 -0700 (Mon, 21 May 2007) | 2 lines Patch #1686487: you can now pass any mapping after '**' in function calls. ........ r55502 | neal.norwitz | 2007-05-21 23:03:36 -0700 (Mon, 21 May 2007) | 1 line Document new params to HTTPSConnection ........ r55504 | neal.norwitz | 2007-05-22 00:16:10 -0700 (Tue, 22 May 2007) | 1 line Stop using METH_OLDARGS ........ r55505 | neal.norwitz | 2007-05-22 00:16:44 -0700 (Tue, 22 May 2007) | 1 line Stop using METH_OLDARGS implicitly ........ ................
2007-05-22 15:11:13 -03:00
#define MAX_MARSHAL_STACK_DEPTH 2000
#endif
#define TYPE_NULL '0'
#define TYPE_NONE 'N'
#define TYPE_FALSE 'F'
#define TYPE_TRUE 'T'
#define TYPE_STOPITER 'S'
#define TYPE_ELLIPSIS '.'
#define TYPE_INT 'i'
#define TYPE_FLOAT 'f'
#define TYPE_BINARY_FLOAT 'g'
#define TYPE_COMPLEX 'x'
#define TYPE_BINARY_COMPLEX 'y'
#define TYPE_LONG 'l'
#define TYPE_STRING 's'
#define TYPE_INTERNED 't'
#define TYPE_REF 'r'
#define TYPE_TUPLE '('
#define TYPE_LIST '['
#define TYPE_DICT '{'
#define TYPE_CODE 'c'
#define TYPE_UNICODE 'u'
#define TYPE_UNKNOWN '?'
#define TYPE_SET '<'
#define TYPE_FROZENSET '>'
#define FLAG_REF '\x80' /* with a type, add obj to index */
1991-06-04 16:42:30 -03:00
#define TYPE_ASCII 'a'
#define TYPE_ASCII_INTERNED 'A'
#define TYPE_SMALL_TUPLE ')'
#define TYPE_SHORT_ASCII 'z'
#define TYPE_SHORT_ASCII_INTERNED 'Z'
#define WFERR_OK 0
#define WFERR_UNMARSHALLABLE 1
#define WFERR_NESTEDTOODEEP 2
#define WFERR_NOMEMORY 3
typedef struct {
FILE *fp;
int error; /* see WFERR_* values */
int depth;
PyObject *str;
char *ptr;
char *end;
char *buf;
_Py_hashtable_t *hashtable;
int version;
} WFILE;
1991-06-04 16:42:30 -03:00
#define w_byte(c, p) do { \
if ((p)->ptr != (p)->end || w_reserve((p), 1)) \
*(p)->ptr++ = (c); \
} while(0)
static void
w_flush(WFILE *p)
{
assert(p->fp != NULL);
fwrite(p->buf, 1, p->ptr - p->buf, p->fp);
p->ptr = p->buf;
}
static int
w_reserve(WFILE *p, Py_ssize_t needed)
{
Py_ssize_t pos, size, delta;
if (p->ptr == NULL)
return 0; /* An error already occurred */
if (p->fp != NULL) {
w_flush(p);
return needed <= p->end - p->ptr;
}
assert(p->str != NULL);
pos = p->ptr - p->buf;
size = PyBytes_Size(p->str);
if (size > 16*1024*1024)
delta = (size >> 3); /* 12.5% overallocation */
else
delta = size + 1024;
delta = Py_MAX(delta, needed);
if (delta > PY_SSIZE_T_MAX - size) {
p->error = WFERR_NOMEMORY;
return 0;
}
size += delta;
if (_PyBytes_Resize(&p->str, size) != 0) {
p->ptr = p->buf = p->end = NULL;
return 0;
}
else {
p->buf = PyBytes_AS_STRING(p->str);
p->ptr = p->buf + pos;
p->end = p->buf + size;
return 1;
}
}
static void
w_string(const char *s, Py_ssize_t n, WFILE *p)
{
Py_ssize_t m;
if (!n || p->ptr == NULL)
return;
m = p->end - p->ptr;
if (p->fp != NULL) {
if (n <= m) {
Py_MEMCPY(p->ptr, s, n);
p->ptr += n;
}
else {
w_flush(p);
fwrite(s, 1, n, p->fp);
}
}
else {
if (n <= m || w_reserve(p, n - m)) {
Py_MEMCPY(p->ptr, s, n);
p->ptr += n;
}
}
}
static void
w_short(int x, WFILE *p)
1991-06-04 16:42:30 -03:00
{
w_byte((char)( x & 0xff), p);
w_byte((char)((x>> 8) & 0xff), p);
1991-06-04 16:42:30 -03:00
}
static void
w_long(long x, WFILE *p)
1991-06-04 16:42:30 -03:00
{
w_byte((char)( x & 0xff), p);
w_byte((char)((x>> 8) & 0xff), p);
w_byte((char)((x>>16) & 0xff), p);
w_byte((char)((x>>24) & 0xff), p);
1991-06-04 16:42:30 -03:00
}
#define SIZE32_MAX 0x7FFFFFFF
#if SIZEOF_SIZE_T > 4
# define W_SIZE(n, p) do { \
if ((n) > SIZE32_MAX) { \
(p)->depth--; \
(p)->error = WFERR_UNMARSHALLABLE; \
return; \
} \
w_long((long)(n), p); \
} while(0)
#else
# define W_SIZE w_long
#endif
static void
w_pstring(const char *s, Py_ssize_t n, WFILE *p)
{
W_SIZE(n, p);
w_string(s, n, p);
}
static void
w_short_pstring(const char *s, Py_ssize_t n, WFILE *p)
{
w_byte(Py_SAFE_DOWNCAST(n, Py_ssize_t, unsigned char), p);
w_string(s, n, p);
}
/* We assume that Python ints are stored internally in base some power of
2**15; for the sake of portability we'll always read and write them in base
exactly 2**15. */
#define PyLong_MARSHAL_SHIFT 15
#define PyLong_MARSHAL_BASE ((short)1 << PyLong_MARSHAL_SHIFT)
#define PyLong_MARSHAL_MASK (PyLong_MARSHAL_BASE - 1)
#if PyLong_SHIFT % PyLong_MARSHAL_SHIFT != 0
#error "PyLong_SHIFT must be a multiple of PyLong_MARSHAL_SHIFT"
#endif
#define PyLong_MARSHAL_RATIO (PyLong_SHIFT / PyLong_MARSHAL_SHIFT)
#define W_TYPE(t, p) do { \
w_byte((t) | flag, (p)); \
} while(0)
static void
w_PyLong(const PyLongObject *ob, char flag, WFILE *p)
{
Py_ssize_t i, j, n, l;
digit d;
W_TYPE(TYPE_LONG, p);
if (Py_SIZE(ob) == 0) {
w_long((long)0, p);
return;
}
/* set l to number of base PyLong_MARSHAL_BASE digits */
n = Py_ABS(Py_SIZE(ob));
l = (n-1) * PyLong_MARSHAL_RATIO;
d = ob->ob_digit[n-1];
assert(d != 0); /* a PyLong is always normalized */
do {
d >>= PyLong_MARSHAL_SHIFT;
l++;
} while (d != 0);
if (l > SIZE32_MAX) {
p->depth--;
p->error = WFERR_UNMARSHALLABLE;
return;
}
w_long((long)(Py_SIZE(ob) > 0 ? l : -l), p);
for (i=0; i < n-1; i++) {
d = ob->ob_digit[i];
for (j=0; j < PyLong_MARSHAL_RATIO; j++) {
w_short(d & PyLong_MARSHAL_MASK, p);
d >>= PyLong_MARSHAL_SHIFT;
}
assert (d == 0);
}
d = ob->ob_digit[n-1];
do {
w_short(d & PyLong_MARSHAL_MASK, p);
d >>= PyLong_MARSHAL_SHIFT;
} while (d != 0);
}
static int
w_ref(PyObject *v, char *flag, WFILE *p)
{
_Py_hashtable_entry_t *entry;
int w;
if (p->version < 3 || p->hashtable == NULL)
return 0; /* not writing object references */
/* if it has only one reference, it definitely isn't shared */
if (Py_REFCNT(v) == 1)
return 0;
entry = _Py_hashtable_get_entry(p->hashtable, v);
if (entry != NULL) {
/* write the reference index to the stream */
_Py_HASHTABLE_ENTRY_READ_DATA(p->hashtable, &w, sizeof(w), entry);
/* we don't store "long" indices in the dict */
assert(0 <= w && w <= 0x7fffffff);
w_byte(TYPE_REF, p);
w_long(w, p);
return 1;
} else {
size_t s = p->hashtable->entries;
/* we don't support long indices */
if (s >= 0x7fffffff) {
PyErr_SetString(PyExc_ValueError, "too many objects");
goto err;
}
w = (int)s;
Py_INCREF(v);
if (_Py_HASHTABLE_SET(p->hashtable, v, w) < 0) {
Py_DECREF(v);
goto err;
}
*flag |= FLAG_REF;
return 0;
}
err:
p->error = WFERR_UNMARSHALLABLE;
return 1;
}
static void
w_complex_object(PyObject *v, char flag, WFILE *p);
static void
w_object(PyObject *v, WFILE *p)
1991-06-04 16:42:30 -03:00
{
char flag = '\0';
p->depth++;
if (p->depth > MAX_MARSHAL_STACK_DEPTH) {
p->error = WFERR_NESTEDTOODEEP;
}
else if (v == NULL) {
w_byte(TYPE_NULL, p);
}
else if (v == Py_None) {
w_byte(TYPE_NONE, p);
}
else if (v == PyExc_StopIteration) {
w_byte(TYPE_STOPITER, p);
}
else if (v == Py_Ellipsis) {
w_byte(TYPE_ELLIPSIS, p);
}
else if (v == Py_False) {
w_byte(TYPE_FALSE, p);
}
else if (v == Py_True) {
w_byte(TYPE_TRUE, p);
}
else if (!w_ref(v, &flag, p))
w_complex_object(v, flag, p);
p->depth--;
}
static void
w_complex_object(PyObject *v, char flag, WFILE *p)
{
Py_ssize_t i, n;
if (PyLong_CheckExact(v)) {
long x = PyLong_AsLong(v);
if ((x == -1) && PyErr_Occurred()) {
PyLongObject *ob = (PyLongObject *)v;
PyErr_Clear();
w_PyLong(ob, flag, p);
}
else {
#if SIZEOF_LONG > 4
long y = Py_ARITHMETIC_RIGHT_SHIFT(long, x, 31);
if (y && y != -1) {
/* Too large for TYPE_INT */
w_PyLong((PyLongObject*)v, flag, p);
}
else
#endif
{
W_TYPE(TYPE_INT, p);
w_long(x, p);
}
}
}
else if (PyFloat_CheckExact(v)) {
if (p->version > 1) {
unsigned char buf[8];
if (_PyFloat_Pack8(PyFloat_AsDouble(v),
buf, 1) < 0) {
p->error = WFERR_UNMARSHALLABLE;
return;
}
W_TYPE(TYPE_BINARY_FLOAT, p);
w_string((char*)buf, 8, p);
}
else {
char *buf = PyOS_double_to_string(PyFloat_AS_DOUBLE(v),
'g', 17, 0, NULL);
if (!buf) {
p->error = WFERR_NOMEMORY;
return;
}
n = strlen(buf);
W_TYPE(TYPE_FLOAT, p);
w_byte((int)n, p);
w_string(buf, n, p);
PyMem_Free(buf);
}
}
else if (PyComplex_CheckExact(v)) {
if (p->version > 1) {
unsigned char buf[8];
if (_PyFloat_Pack8(PyComplex_RealAsDouble(v),
buf, 1) < 0) {
p->error = WFERR_UNMARSHALLABLE;
return;
}
W_TYPE(TYPE_BINARY_COMPLEX, p);
w_string((char*)buf, 8, p);
if (_PyFloat_Pack8(PyComplex_ImagAsDouble(v),
buf, 1) < 0) {
p->error = WFERR_UNMARSHALLABLE;
return;
}
w_string((char*)buf, 8, p);
}
else {
char *buf;
W_TYPE(TYPE_COMPLEX, p);
buf = PyOS_double_to_string(PyComplex_RealAsDouble(v),
'g', 17, 0, NULL);
if (!buf) {
p->error = WFERR_NOMEMORY;
return;
}
n = strlen(buf);
w_byte((int)n, p);
w_string(buf, n, p);
PyMem_Free(buf);
buf = PyOS_double_to_string(PyComplex_ImagAsDouble(v),
'g', 17, 0, NULL);
if (!buf) {
p->error = WFERR_NOMEMORY;
return;
}
n = strlen(buf);
w_byte((int)n, p);
w_string(buf, n, p);
PyMem_Free(buf);
}
}
else if (PyBytes_CheckExact(v)) {
W_TYPE(TYPE_STRING, p);
w_pstring(PyBytes_AS_STRING(v), PyBytes_GET_SIZE(v), p);
}
else if (PyUnicode_CheckExact(v)) {
if (p->version >= 4 && PyUnicode_IS_ASCII(v)) {
int is_short = PyUnicode_GET_LENGTH(v) < 256;
if (is_short) {
if (PyUnicode_CHECK_INTERNED(v))
W_TYPE(TYPE_SHORT_ASCII_INTERNED, p);
else
W_TYPE(TYPE_SHORT_ASCII, p);
w_short_pstring((char *) PyUnicode_1BYTE_DATA(v),
PyUnicode_GET_LENGTH(v), p);
}
else {
if (PyUnicode_CHECK_INTERNED(v))
W_TYPE(TYPE_ASCII_INTERNED, p);
else
W_TYPE(TYPE_ASCII, p);
w_pstring((char *) PyUnicode_1BYTE_DATA(v),
PyUnicode_GET_LENGTH(v), p);
}
}
else {
PyObject *utf8;
utf8 = PyUnicode_AsEncodedString(v, "utf8", "surrogatepass");
if (utf8 == NULL) {
p->depth--;
p->error = WFERR_UNMARSHALLABLE;
return;
}
if (p->version >= 3 && PyUnicode_CHECK_INTERNED(v))
W_TYPE(TYPE_INTERNED, p);
else
W_TYPE(TYPE_UNICODE, p);
w_pstring(PyBytes_AS_STRING(utf8), PyBytes_GET_SIZE(utf8), p);
Py_DECREF(utf8);
}
}
else if (PyTuple_CheckExact(v)) {
n = PyTuple_Size(v);
if (p->version >= 4 && n < 256) {
W_TYPE(TYPE_SMALL_TUPLE, p);
w_byte((unsigned char)n, p);
}
else {
W_TYPE(TYPE_TUPLE, p);
W_SIZE(n, p);
}
for (i = 0; i < n; i++) {
w_object(PyTuple_GET_ITEM(v, i), p);
}
}
else if (PyList_CheckExact(v)) {
W_TYPE(TYPE_LIST, p);
n = PyList_GET_SIZE(v);
W_SIZE(n, p);
for (i = 0; i < n; i++) {
w_object(PyList_GET_ITEM(v, i), p);
}
}
else if (PyDict_CheckExact(v)) {
Py_ssize_t pos;
PyObject *key, *value;
W_TYPE(TYPE_DICT, p);
/* This one is NULL object terminated! */
pos = 0;
while (PyDict_Next(v, &pos, &key, &value)) {
w_object(key, p);
w_object(value, p);
}
w_object((PyObject *)NULL, p);
}
else if (PyAnySet_CheckExact(v)) {
PyObject *value, *it;
if (PyObject_TypeCheck(v, &PySet_Type))
W_TYPE(TYPE_SET, p);
else
W_TYPE(TYPE_FROZENSET, p);
n = PyObject_Size(v);
if (n == -1) {
p->depth--;
p->error = WFERR_UNMARSHALLABLE;
return;
}
W_SIZE(n, p);
it = PyObject_GetIter(v);
if (it == NULL) {
p->depth--;
p->error = WFERR_UNMARSHALLABLE;
return;
}
while ((value = PyIter_Next(it)) != NULL) {
w_object(value, p);
Py_DECREF(value);
}
Py_DECREF(it);
if (PyErr_Occurred()) {
p->depth--;
p->error = WFERR_UNMARSHALLABLE;
return;
}
}
else if (PyCode_Check(v)) {
PyCodeObject *co = (PyCodeObject *)v;
W_TYPE(TYPE_CODE, p);
w_long(co->co_argcount, p);
w_long(co->co_kwonlyargcount, p);
w_long(co->co_nlocals, p);
w_long(co->co_stacksize, p);
w_long(co->co_flags, p);
w_object(co->co_code, p);
w_object(co->co_consts, p);
w_object(co->co_names, p);
w_object(co->co_varnames, p);
w_object(co->co_freevars, p);
w_object(co->co_cellvars, p);
w_object(co->co_filename, p);
w_object(co->co_name, p);
w_long(co->co_firstlineno, p);
w_object(co->co_lnotab, p);
}
else if (PyObject_CheckBuffer(v)) {
/* Write unknown bytes-like objects as a byte string */
Py_buffer view;
if (PyObject_GetBuffer(v, &view, PyBUF_SIMPLE) != 0) {
w_byte(TYPE_UNKNOWN, p);
p->depth--;
p->error = WFERR_UNMARSHALLABLE;
return;
}
W_TYPE(TYPE_STRING, p);
w_pstring(view.buf, view.len, p);
PyBuffer_Release(&view);
}
else {
W_TYPE(TYPE_UNKNOWN, p);
p->error = WFERR_UNMARSHALLABLE;
}
1991-06-04 16:42:30 -03:00
}
static int
w_init_refs(WFILE *wf, int version)
{
if (version >= 3) {
wf->hashtable = _Py_hashtable_new(sizeof(int), _Py_hashtable_hash_ptr,
_Py_hashtable_compare_direct);
if (wf->hashtable == NULL) {
PyErr_NoMemory();
return -1;
}
}
return 0;
}
static int
w_decref_entry(_Py_hashtable_entry_t *entry, void *Py_UNUSED(data))
{
Py_XDECREF(entry->key);
return 0;
}
static void
w_clear_refs(WFILE *wf)
{
if (wf->hashtable != NULL) {
_Py_hashtable_foreach(wf->hashtable, w_decref_entry, NULL);
_Py_hashtable_destroy(wf->hashtable);
}
}
/* version currently has no effect for writing ints. */
void
PyMarshal_WriteLongToFile(long x, FILE *fp, int version)
{
char buf[4];
WFILE wf;
memset(&wf, 0, sizeof(wf));
wf.fp = fp;
wf.ptr = wf.buf = buf;
wf.end = wf.ptr + sizeof(buf);
wf.error = WFERR_OK;
wf.version = version;
w_long(x, &wf);
w_flush(&wf);
}
1991-06-04 16:42:30 -03:00
void
PyMarshal_WriteObjectToFile(PyObject *x, FILE *fp, int version)
{
char buf[BUFSIZ];
WFILE wf;
memset(&wf, 0, sizeof(wf));
wf.fp = fp;
wf.ptr = wf.buf = buf;
wf.end = wf.ptr + sizeof(buf);
wf.error = WFERR_OK;
wf.version = version;
if (w_init_refs(&wf, version))
return; /* caller mush check PyErr_Occurred() */
w_object(x, &wf);
w_clear_refs(&wf);
w_flush(&wf);
}
typedef struct {
FILE *fp;
int depth;
PyObject *readable; /* Stream-like object being read from */
PyObject *current_filename;
char *ptr;
char *end;
char *buf;
Py_ssize_t buf_size;
PyObject *refs; /* a list */
} RFILE;
static const char *
r_string(Py_ssize_t n, RFILE *p)
{
Py_ssize_t read = -1;
if (p->ptr != NULL) {
/* Fast path for loads() */
char *res = p->ptr;
Py_ssize_t left = p->end - p->ptr;
if (left < n) {
PyErr_SetString(PyExc_EOFError,
"marshal data too short");
return NULL;
}
p->ptr += n;
return res;
}
if (p->buf == NULL) {
p->buf = PyMem_MALLOC(n);
if (p->buf == NULL) {
PyErr_NoMemory();
return NULL;
}
p->buf_size = n;
}
else if (p->buf_size < n) {
p->buf = PyMem_REALLOC(p->buf, n);
if (p->buf == NULL) {
PyErr_NoMemory();
return NULL;
}
p->buf_size = n;
}
if (!p->readable) {
assert(p->fp != NULL);
read = fread(p->buf, 1, n, p->fp);
}
else {
_Py_IDENTIFIER(readinto);
PyObject *res, *mview;
Py_buffer buf;
if (PyBuffer_FillInfo(&buf, NULL, p->buf, n, 0, PyBUF_CONTIG) == -1)
return NULL;
mview = PyMemoryView_FromBuffer(&buf);
if (mview == NULL)
return NULL;
res = _PyObject_CallMethodId(p->readable, &PyId_readinto, "N", mview);
if (res != NULL) {
read = PyNumber_AsSsize_t(res, PyExc_ValueError);
Py_DECREF(res);
}
}
if (read != n) {
if (!PyErr_Occurred()) {
if (read > n)
PyErr_Format(PyExc_ValueError,
"read() returned too much data: "
"%zd bytes requested, %zd returned",
n, read);
else
PyErr_SetString(PyExc_EOFError,
"EOF read where not expected");
}
return NULL;
}
return p->buf;
}
static int
r_byte(RFILE *p)
{
int c = EOF;
if (p->ptr != NULL) {
if (p->ptr < p->end)
c = (unsigned char) *p->ptr++;
return c;
}
if (!p->readable) {
assert(p->fp);
c = getc(p->fp);
}
else {
const char *ptr = r_string(1, p);
if (ptr != NULL)
c = *(unsigned char *) ptr;
}
return c;
}
static int
r_short(RFILE *p)
1991-06-04 16:42:30 -03:00
{
short x = -1;
const unsigned char *buffer;
buffer = (const unsigned char *) r_string(2, p);
if (buffer != NULL) {
x = buffer[0];
x |= buffer[1] << 8;
/* Sign-extension, in case short greater than 16 bits */
x |= -(x & 0x8000);
}
return x;
1991-06-04 16:42:30 -03:00
}
static long
r_long(RFILE *p)
1991-06-04 16:42:30 -03:00
{
long x = -1;
const unsigned char *buffer;
buffer = (const unsigned char *) r_string(4, p);
if (buffer != NULL) {
x = buffer[0];
x |= (long)buffer[1] << 8;
x |= (long)buffer[2] << 16;
x |= (long)buffer[3] << 24;
#if SIZEOF_LONG > 4
/* Sign extension for 64-bit machines */
x |= -(x & 0x80000000L);
#endif
}
return x;
}
static PyObject *
r_PyLong(RFILE *p)
{
PyLongObject *ob;
long n, size, i;
int j, md, shorts_in_top_digit;
digit d;
n = r_long(p);
if (PyErr_Occurred())
return NULL;
if (n == 0)
return (PyObject *)_PyLong_New(0);
if (n < -SIZE32_MAX || n > SIZE32_MAX) {
PyErr_SetString(PyExc_ValueError,
"bad marshal data (long size out of range)");
return NULL;
}
size = 1 + (Py_ABS(n) - 1) / PyLong_MARSHAL_RATIO;
shorts_in_top_digit = 1 + (Py_ABS(n) - 1) % PyLong_MARSHAL_RATIO;
ob = _PyLong_New(size);
if (ob == NULL)
return NULL;
Py_SIZE(ob) = n > 0 ? size : -size;
for (i = 0; i < size-1; i++) {
d = 0;
for (j=0; j < PyLong_MARSHAL_RATIO; j++) {
md = r_short(p);
if (PyErr_Occurred()) {
Py_DECREF(ob);
return NULL;
}
if (md < 0 || md > PyLong_MARSHAL_BASE)
goto bad_digit;
d += (digit)md << j*PyLong_MARSHAL_SHIFT;
}
ob->ob_digit[i] = d;
}
d = 0;
for (j=0; j < shorts_in_top_digit; j++) {
md = r_short(p);
if (PyErr_Occurred()) {
Py_DECREF(ob);
return NULL;
}
if (md < 0 || md > PyLong_MARSHAL_BASE)
goto bad_digit;
/* topmost marshal digit should be nonzero */
if (md == 0 && j == shorts_in_top_digit - 1) {
Py_DECREF(ob);
PyErr_SetString(PyExc_ValueError,
"bad marshal data (unnormalized long data)");
return NULL;
}
d += (digit)md << j*PyLong_MARSHAL_SHIFT;
}
if (PyErr_Occurred()) {
Py_DECREF(ob);
return NULL;
}
/* top digit should be nonzero, else the resulting PyLong won't be
normalized */
ob->ob_digit[size-1] = d;
return (PyObject *)ob;
bad_digit:
Py_DECREF(ob);
PyErr_SetString(PyExc_ValueError,
"bad marshal data (digit out of range in long)");
return NULL;
}
/* allocate the reflist index for a new object. Return -1 on failure */
static Py_ssize_t
r_ref_reserve(int flag, RFILE *p)
{
if (flag) { /* currently only FLAG_REF is defined */
Py_ssize_t idx = PyList_GET_SIZE(p->refs);
if (idx >= 0x7ffffffe) {
PyErr_SetString(PyExc_ValueError, "bad marshal data (index list too large)");
return -1;
}
if (PyList_Append(p->refs, Py_None) < 0)
return -1;
return idx;
} else
return 0;
}
/* insert the new object 'o' to the reflist at previously
* allocated index 'idx'.
* 'o' can be NULL, in which case nothing is done.
* if 'o' was non-NULL, and the function succeeds, 'o' is returned.
* if 'o' was non-NULL, and the function fails, 'o' is released and
* NULL returned. This simplifies error checking at the call site since
* a single test for NULL for the function result is enough.
*/
static PyObject *
r_ref_insert(PyObject *o, Py_ssize_t idx, int flag, RFILE *p)
{
if (o != NULL && flag) { /* currently only FLAG_REF is defined */
PyObject *tmp = PyList_GET_ITEM(p->refs, idx);
Py_INCREF(o);
PyList_SET_ITEM(p->refs, idx, o);
Py_DECREF(tmp);
}
return o;
}
/* combination of both above, used when an object can be
* created whenever it is seen in the file, as opposed to
* after having loaded its sub-objects.
*/
static PyObject *
r_ref(PyObject *o, int flag, RFILE *p)
{
assert(flag & FLAG_REF);
if (o == NULL)
return NULL;
if (PyList_Append(p->refs, o) < 0) {
Py_DECREF(o); /* release the new object */
return NULL;
}
return o;
}
static PyObject *
r_object(RFILE *p)
1991-06-04 16:42:30 -03:00
{
/* NULL is a valid return value, it does not necessarily means that
an exception is set. */
PyObject *v, *v2;
2013-03-20 02:40:07 -03:00
Py_ssize_t idx = 0;
long i, n;
int type, code = r_byte(p);
int flag, is_interned = 0;
PyObject *retval = NULL;
if (code == EOF) {
PyErr_SetString(PyExc_EOFError,
"EOF read where object expected");
return NULL;
}
p->depth++;
if (p->depth > MAX_MARSHAL_STACK_DEPTH) {
p->depth--;
PyErr_SetString(PyExc_ValueError, "recursion limit exceeded");
return NULL;
}
flag = code & FLAG_REF;
type = code & ~FLAG_REF;
#define R_REF(O) do{\
if (flag) \
O = r_ref(O, flag, p);\
} while (0)
switch (type) {
case TYPE_NULL:
break;
case TYPE_NONE:
Py_INCREF(Py_None);
retval = Py_None;
break;
case TYPE_STOPITER:
Py_INCREF(PyExc_StopIteration);
retval = PyExc_StopIteration;
break;
case TYPE_ELLIPSIS:
Py_INCREF(Py_Ellipsis);
retval = Py_Ellipsis;
break;
case TYPE_FALSE:
Py_INCREF(Py_False);
retval = Py_False;
break;
case TYPE_TRUE:
Py_INCREF(Py_True);
retval = Py_True;
break;
case TYPE_INT:
n = r_long(p);
retval = PyErr_Occurred() ? NULL : PyLong_FromLong(n);
R_REF(retval);
break;
case TYPE_LONG:
retval = r_PyLong(p);
R_REF(retval);
break;
case TYPE_FLOAT:
{
char buf[256];
const char *ptr;
double dx;
n = r_byte(p);
if (n == EOF) {
PyErr_SetString(PyExc_EOFError,
"EOF read where object expected");
break;
}
ptr = r_string(n, p);
if (ptr == NULL)
break;
memcpy(buf, ptr, n);
buf[n] = '\0';
dx = PyOS_string_to_double(buf, NULL, NULL);
if (dx == -1.0 && PyErr_Occurred())
break;
retval = PyFloat_FromDouble(dx);
R_REF(retval);
break;
}
case TYPE_BINARY_FLOAT:
{
const unsigned char *buf;
double x;
buf = (const unsigned char *) r_string(8, p);
if (buf == NULL)
break;
x = _PyFloat_Unpack8(buf, 1);
if (x == -1.0 && PyErr_Occurred())
break;
retval = PyFloat_FromDouble(x);
R_REF(retval);
break;
}
case TYPE_COMPLEX:
{
char buf[256];
const char *ptr;
Py_complex c;
n = r_byte(p);
if (n == EOF) {
PyErr_SetString(PyExc_EOFError,
"EOF read where object expected");
break;
}
ptr = r_string(n, p);
if (ptr == NULL)
break;
memcpy(buf, ptr, n);
buf[n] = '\0';
c.real = PyOS_string_to_double(buf, NULL, NULL);
if (c.real == -1.0 && PyErr_Occurred())
break;
n = r_byte(p);
if (n == EOF) {
PyErr_SetString(PyExc_EOFError,
"EOF read where object expected");
break;
}
ptr = r_string(n, p);
if (ptr == NULL)
break;
memcpy(buf, ptr, n);
buf[n] = '\0';
c.imag = PyOS_string_to_double(buf, NULL, NULL);
if (c.imag == -1.0 && PyErr_Occurred())
break;
retval = PyComplex_FromCComplex(c);
R_REF(retval);
break;
}
case TYPE_BINARY_COMPLEX:
{
const unsigned char *buf;
Py_complex c;
buf = (const unsigned char *) r_string(8, p);
if (buf == NULL)
break;
c.real = _PyFloat_Unpack8(buf, 1);
if (c.real == -1.0 && PyErr_Occurred())
break;
buf = (const unsigned char *) r_string(8, p);
if (buf == NULL)
break;
c.imag = _PyFloat_Unpack8(buf, 1);
if (c.imag == -1.0 && PyErr_Occurred())
break;
retval = PyComplex_FromCComplex(c);
R_REF(retval);
break;
}
case TYPE_STRING:
{
const char *ptr;
n = r_long(p);
if (PyErr_Occurred())
break;
if (n < 0 || n > SIZE32_MAX) {
PyErr_SetString(PyExc_ValueError, "bad marshal data (string size out of range)");
break;
}
v = PyBytes_FromStringAndSize((char *)NULL, n);
if (v == NULL)
break;
ptr = r_string(n, p);
if (ptr == NULL) {
Py_DECREF(v);
break;
}
memcpy(PyBytes_AS_STRING(v), ptr, n);
retval = v;
R_REF(retval);
break;
}
case TYPE_ASCII_INTERNED:
is_interned = 1;
case TYPE_ASCII:
n = r_long(p);
if (PyErr_Occurred())
break;
if (n < 0 || n > SIZE32_MAX) {
PyErr_SetString(PyExc_ValueError, "bad marshal data (unicode size out of range)");
break;
}
goto _read_ascii;
case TYPE_SHORT_ASCII_INTERNED:
is_interned = 1;
case TYPE_SHORT_ASCII:
n = r_byte(p);
if (n == EOF) {
PyErr_SetString(PyExc_EOFError,
"EOF read where object expected");
break;
}
_read_ascii:
{
const char *ptr;
ptr = r_string(n, p);
if (ptr == NULL)
break;
v = PyUnicode_FromKindAndData(PyUnicode_1BYTE_KIND, ptr, n);
if (v == NULL)
break;
if (is_interned)
PyUnicode_InternInPlace(&v);
retval = v;
R_REF(retval);
break;
}
case TYPE_INTERNED:
is_interned = 1;
case TYPE_UNICODE:
{
const char *buffer;
n = r_long(p);
if (PyErr_Occurred())
break;
if (n < 0 || n > SIZE32_MAX) {
PyErr_SetString(PyExc_ValueError, "bad marshal data (unicode size out of range)");
break;
}
if (n != 0) {
buffer = r_string(n, p);
if (buffer == NULL)
break;
v = PyUnicode_DecodeUTF8(buffer, n, "surrogatepass");
}
else {
v = PyUnicode_New(0, 0);
}
if (v == NULL)
break;
if (is_interned)
PyUnicode_InternInPlace(&v);
retval = v;
R_REF(retval);
break;
}
case TYPE_SMALL_TUPLE:
n = (unsigned char) r_byte(p);
if (PyErr_Occurred())
break;
goto _read_tuple;
case TYPE_TUPLE:
n = r_long(p);
if (PyErr_Occurred())
break;
if (n < 0 || n > SIZE32_MAX) {
PyErr_SetString(PyExc_ValueError, "bad marshal data (tuple size out of range)");
break;
}
_read_tuple:
v = PyTuple_New(n);
R_REF(v);
if (v == NULL)
break;
for (i = 0; i < n; i++) {
v2 = r_object(p);
if ( v2 == NULL ) {
if (!PyErr_Occurred())
PyErr_SetString(PyExc_TypeError,
"NULL object in marshal data for tuple");
Py_DECREF(v);
v = NULL;
break;
}
PyTuple_SET_ITEM(v, i, v2);
}
retval = v;
break;
case TYPE_LIST:
n = r_long(p);
if (PyErr_Occurred())
break;
if (n < 0 || n > SIZE32_MAX) {
PyErr_SetString(PyExc_ValueError, "bad marshal data (list size out of range)");
break;
}
v = PyList_New(n);
R_REF(v);
if (v == NULL)
break;
for (i = 0; i < n; i++) {
v2 = r_object(p);
if ( v2 == NULL ) {
if (!PyErr_Occurred())
PyErr_SetString(PyExc_TypeError,
"NULL object in marshal data for list");
Py_DECREF(v);
v = NULL;
break;
}
PyList_SET_ITEM(v, i, v2);
}
retval = v;
break;
case TYPE_DICT:
v = PyDict_New();
R_REF(v);
if (v == NULL)
break;
for (;;) {
PyObject *key, *val;
key = r_object(p);
if (key == NULL)
break;
val = r_object(p);
if (val == NULL) {
Py_DECREF(key);
break;
}
if (PyDict_SetItem(v, key, val) < 0) {
Py_DECREF(key);
Py_DECREF(val);
break;
}
Py_DECREF(key);
Py_DECREF(val);
}
if (PyErr_Occurred()) {
Py_DECREF(v);
v = NULL;
}
retval = v;
break;
case TYPE_SET:
case TYPE_FROZENSET:
n = r_long(p);
if (PyErr_Occurred())
break;
if (n < 0 || n > SIZE32_MAX) {
PyErr_SetString(PyExc_ValueError, "bad marshal data (set size out of range)");
break;
}
if (n == 0 && type == TYPE_FROZENSET) {
/* call frozenset() to get the empty frozenset singleton */
v = PyObject_CallFunction((PyObject*)&PyFrozenSet_Type, NULL);
if (v == NULL)
break;
R_REF(v);
retval = v;
}
else {
v = (type == TYPE_SET) ? PySet_New(NULL) : PyFrozenSet_New(NULL);
if (type == TYPE_SET) {
R_REF(v);
} else {
/* must use delayed registration of frozensets because they must
* be init with a refcount of 1
*/
idx = r_ref_reserve(flag, p);
if (idx < 0)
Py_CLEAR(v); /* signal error */
}
if (v == NULL)
break;
for (i = 0; i < n; i++) {
v2 = r_object(p);
if ( v2 == NULL ) {
if (!PyErr_Occurred())
PyErr_SetString(PyExc_TypeError,
"NULL object in marshal data for set");
Py_DECREF(v);
v = NULL;
break;
}
if (PySet_Add(v, v2) == -1) {
Py_DECREF(v);
Py_DECREF(v2);
v = NULL;
break;
}
Py_DECREF(v2);
}
if (type != TYPE_SET)
v = r_ref_insert(v, idx, flag, p);
retval = v;
}
break;
case TYPE_CODE:
{
int argcount;
int kwonlyargcount;
int nlocals;
int stacksize;
int flags;
PyObject *code = NULL;
PyObject *consts = NULL;
PyObject *names = NULL;
PyObject *varnames = NULL;
PyObject *freevars = NULL;
PyObject *cellvars = NULL;
PyObject *filename = NULL;
PyObject *name = NULL;
int firstlineno;
PyObject *lnotab = NULL;
idx = r_ref_reserve(flag, p);
if (idx < 0)
break;
v = NULL;
/* XXX ignore long->int overflows for now */
argcount = (int)r_long(p);
if (PyErr_Occurred())
goto code_error;
kwonlyargcount = (int)r_long(p);
if (PyErr_Occurred())
goto code_error;
nlocals = (int)r_long(p);
if (PyErr_Occurred())
goto code_error;
stacksize = (int)r_long(p);
if (PyErr_Occurred())
goto code_error;
flags = (int)r_long(p);
if (PyErr_Occurred())
goto code_error;
code = r_object(p);
if (code == NULL)
goto code_error;
consts = r_object(p);
if (consts == NULL)
goto code_error;
names = r_object(p);
if (names == NULL)
goto code_error;
varnames = r_object(p);
if (varnames == NULL)
goto code_error;
freevars = r_object(p);
if (freevars == NULL)
goto code_error;
cellvars = r_object(p);
if (cellvars == NULL)
goto code_error;
filename = r_object(p);
if (filename == NULL)
goto code_error;
if (PyUnicode_CheckExact(filename)) {
if (p->current_filename != NULL) {
if (!PyUnicode_Compare(filename, p->current_filename)) {
Py_DECREF(filename);
Py_INCREF(p->current_filename);
filename = p->current_filename;
}
}
else {
p->current_filename = filename;
}
}
name = r_object(p);
if (name == NULL)
goto code_error;
firstlineno = (int)r_long(p);
if (firstlineno == -1 && PyErr_Occurred())
break;
lnotab = r_object(p);
if (lnotab == NULL)
goto code_error;
v = (PyObject *) PyCode_New(
argcount, kwonlyargcount,
nlocals, stacksize, flags,
code, consts, names, varnames,
freevars, cellvars, filename, name,
firstlineno, lnotab);
v = r_ref_insert(v, idx, flag, p);
code_error:
Py_XDECREF(code);
Py_XDECREF(consts);
Py_XDECREF(names);
Py_XDECREF(varnames);
Py_XDECREF(freevars);
Py_XDECREF(cellvars);
Py_XDECREF(filename);
Py_XDECREF(name);
Py_XDECREF(lnotab);
}
retval = v;
break;
case TYPE_REF:
n = r_long(p);
if (n < 0 || n >= PyList_GET_SIZE(p->refs)) {
if (n == -1 && PyErr_Occurred())
break;
PyErr_SetString(PyExc_ValueError, "bad marshal data (invalid reference)");
break;
}
v = PyList_GET_ITEM(p->refs, n);
if (v == Py_None) {
PyErr_SetString(PyExc_ValueError, "bad marshal data (invalid reference)");
break;
}
Py_INCREF(v);
retval = v;
break;
default:
/* Bogus data got written, which isn't ideal.
This will let you keep working and recover. */
PyErr_SetString(PyExc_ValueError, "bad marshal data (unknown type code)");
break;
}
p->depth--;
return retval;
1991-06-04 16:42:30 -03:00
}
2004-06-13 17:31:49 -03:00
static PyObject *
2004-03-26 11:09:27 -04:00
read_object(RFILE *p)
{
PyObject *v;
if (PyErr_Occurred()) {
fprintf(stderr, "XXX readobject called with exception set\n");
return NULL;
}
v = r_object(p);
if (v == NULL && !PyErr_Occurred())
PyErr_SetString(PyExc_TypeError, "NULL object in marshal data for object");
return v;
2004-03-26 11:09:27 -04:00
}
int
PyMarshal_ReadShortFromFile(FILE *fp)
{
RFILE rf;
int res;
assert(fp);
rf.readable = NULL;
rf.fp = fp;
rf.current_filename = NULL;
rf.end = rf.ptr = NULL;
rf.buf = NULL;
res = r_short(&rf);
if (rf.buf != NULL)
PyMem_FREE(rf.buf);
return res;
}
long
PyMarshal_ReadLongFromFile(FILE *fp)
{
RFILE rf;
long res;
rf.fp = fp;
rf.readable = NULL;
rf.current_filename = NULL;
rf.ptr = rf.end = NULL;
rf.buf = NULL;
res = r_long(&rf);
if (rf.buf != NULL)
PyMem_FREE(rf.buf);
return res;
}
/* Return size of file in bytes; < 0 if unknown or INT_MAX if too big */
static off_t
getfilesize(FILE *fp)
{
struct _Py_stat_struct st;
if (_Py_fstat_noraise(fileno(fp), &st) != 0)
return -1;
#if SIZEOF_OFF_T == 4
else if (st.st_size >= INT_MAX)
return (off_t)INT_MAX;
#endif
else
return (off_t)st.st_size;
}
/* If we can get the size of the file up-front, and it's reasonably small,
* read it in one gulp and delegate to ...FromString() instead. Much quicker
* than reading a byte at a time from file; speeds .pyc imports.
* CAUTION: since this may read the entire remainder of the file, don't
* call it unless you know you're done with the file.
*/
PyObject *
PyMarshal_ReadLastObjectFromFile(FILE *fp)
{
/* REASONABLE_FILE_LIMIT is by defn something big enough for Tkinter.pyc. */
#define REASONABLE_FILE_LIMIT (1L << 18)
off_t filesize;
filesize = getfilesize(fp);
if (filesize > 0 && filesize <= REASONABLE_FILE_LIMIT) {
char* pBuf = (char *)PyMem_MALLOC(filesize);
if (pBuf != NULL) {
size_t n = fread(pBuf, 1, (size_t)filesize, fp);
PyObject* v = PyMarshal_ReadObjectFromString(pBuf, n);
PyMem_FREE(pBuf);
return v;
}
}
/* We don't have fstat, or we do but the file is larger than
* REASONABLE_FILE_LIMIT or malloc failed -- read a byte at a time.
*/
return PyMarshal_ReadObjectFromFile(fp);
#undef REASONABLE_FILE_LIMIT
}
PyObject *
PyMarshal_ReadObjectFromFile(FILE *fp)
{
RFILE rf;
PyObject *result;
rf.fp = fp;
rf.readable = NULL;
rf.current_filename = NULL;
rf.depth = 0;
rf.ptr = rf.end = NULL;
rf.buf = NULL;
rf.refs = PyList_New(0);
if (rf.refs == NULL)
return NULL;
result = r_object(&rf);
Py_DECREF(rf.refs);
if (rf.buf != NULL)
PyMem_FREE(rf.buf);
return result;
}
PyObject *
PyMarshal_ReadObjectFromString(const char *str, Py_ssize_t len)
{
RFILE rf;
PyObject *result;
rf.fp = NULL;
rf.readable = NULL;
rf.current_filename = NULL;
rf.ptr = (char *)str;
rf.end = (char *)str + len;
rf.buf = NULL;
rf.depth = 0;
rf.refs = PyList_New(0);
if (rf.refs == NULL)
return NULL;
result = r_object(&rf);
Py_DECREF(rf.refs);
if (rf.buf != NULL)
PyMem_FREE(rf.buf);
return result;
}
PyObject *
PyMarshal_WriteObjectToString(PyObject *x, int version)
{
WFILE wf;
memset(&wf, 0, sizeof(wf));
wf.str = PyBytes_FromStringAndSize((char *)NULL, 50);
if (wf.str == NULL)
return NULL;
wf.ptr = wf.buf = PyBytes_AS_STRING((PyBytesObject *)wf.str);
wf.end = wf.ptr + PyBytes_Size(wf.str);
wf.error = WFERR_OK;
wf.version = version;
if (w_init_refs(&wf, version)) {
Py_DECREF(wf.str);
return NULL;
}
w_object(x, &wf);
w_clear_refs(&wf);
if (wf.str != NULL) {
char *base = PyBytes_AS_STRING((PyBytesObject *)wf.str);
if (wf.ptr - base > PY_SSIZE_T_MAX) {
Py_DECREF(wf.str);
PyErr_SetString(PyExc_OverflowError,
"too much marshal data for a string");
return NULL;
}
if (_PyBytes_Resize(&wf.str, (Py_ssize_t)(wf.ptr - base)) < 0)
return NULL;
}
if (wf.error != WFERR_OK) {
Py_XDECREF(wf.str);
if (wf.error == WFERR_NOMEMORY)
PyErr_NoMemory();
else
PyErr_SetString(PyExc_ValueError,
(wf.error==WFERR_UNMARSHALLABLE)?"unmarshallable object"
:"object too deeply nested to marshal");
return NULL;
}
2012-03-02 13:22:23 -04:00
return wf.str;
}
1991-06-07 10:58:22 -03:00
/* And an interface for Python programs... */
1991-06-04 16:42:30 -03:00
static PyObject *
marshal_dump(PyObject *self, PyObject *args)
1991-06-04 16:42:30 -03:00
{
/* XXX Quick hack -- need to do this differently */
PyObject *x;
PyObject *f;
int version = Py_MARSHAL_VERSION;
PyObject *s;
PyObject *res;
_Py_IDENTIFIER(write);
if (!PyArg_ParseTuple(args, "OO|i:dump", &x, &f, &version))
return NULL;
s = PyMarshal_WriteObjectToString(x, version);
if (s == NULL)
return NULL;
res = _PyObject_CallMethodId(f, &PyId_write, "O", s);
Py_DECREF(s);
return res;
1991-06-04 16:42:30 -03:00
}
PyDoc_STRVAR(dump_doc,
"dump(value, file[, version])\n\
\n\
Write the value on the open file. The value must be a supported type.\n\
The file must be an open file object such as sys.stdout or returned by\n\
open() or os.popen(). It must be opened in binary mode ('wb' or 'w+b').\n\
\n\
If the value has (or contains an object that has) an unsupported type, a\n\
ValueError exception is raised - but garbage data will also be written\n\
to the file. The object will not be properly read back by load()\n\
\n\
The version argument indicates the data format that dump should use.");
static PyObject *
Partially merge trunk into p3yk. The removal of Mac/Tools is confusing svn merge in bad ways, so I'll have to merge that extra-carefully (probably manually.) Merged revisions 46495-46605 via svnmerge from svn+ssh://pythondev@svn.python.org/python/trunk ........ r46495 | tim.peters | 2006-05-28 03:52:38 +0200 (Sun, 28 May 2006) | 2 lines Added missing svn:eol-style property to text files. ........ r46497 | tim.peters | 2006-05-28 12:41:29 +0200 (Sun, 28 May 2006) | 3 lines PyErr_Display(), PyErr_WriteUnraisable(): Coverity found a cut-and-paste bug in both: `className` was referenced before being checked for NULL. ........ r46499 | fredrik.lundh | 2006-05-28 14:06:46 +0200 (Sun, 28 May 2006) | 5 lines needforspeed: added Py_MEMCPY macro (currently tuned for Visual C only), and use it for string copy operations. this gives a 20% speedup on some string benchmarks. ........ r46501 | michael.hudson | 2006-05-28 17:51:40 +0200 (Sun, 28 May 2006) | 26 lines Quality control, meet exceptions.c. Fix a number of problems with the need for speed code: One is doing this sort of thing: Py_DECREF(self->field); self->field = newval; Py_INCREF(self->field); without being very sure that self->field doesn't start with a value that has a __del__, because that almost certainly can lead to segfaults. As self->args is constrained to be an exact tuple we may as well exploit this fact consistently. This leads to quite a lot of simplification (and, hey, probably better performance). Add some error checking in places lacking it. Fix some rather strange indentation in the Unicode code. Delete some trailing whitespace. More to come, I haven't fixed all the reference leaks yet... ........ r46502 | george.yoshida | 2006-05-28 18:39:09 +0200 (Sun, 28 May 2006) | 3 lines Patch #1080727: add "encoding" parameter to doctest.DocFileSuite Contributed by Bjorn Tillenius. ........ r46503 | martin.v.loewis | 2006-05-28 18:57:38 +0200 (Sun, 28 May 2006) | 4 lines Rest of patch #1490384: Commit icon source, remove claim that Erik von Blokland is the author of the installer picture. ........ r46504 | michael.hudson | 2006-05-28 19:40:29 +0200 (Sun, 28 May 2006) | 16 lines Quality control, meet exceptions.c, round two. Make some functions that should have been static static. Fix a bunch of refleaks by fixing the definition of MiddlingExtendsException. Remove all the __new__ implementations apart from BaseException_new. Rewrite most code that needs it to cope with NULL fields (such code could get excercised anyway, the __new__-removal just makes it more likely). This involved editing the code for WindowsError, which I can't test. This fixes all the refleaks in at least the start of a regrtest -R :: run. ........ r46505 | marc-andre.lemburg | 2006-05-28 19:46:58 +0200 (Sun, 28 May 2006) | 10 lines Initial version of systimes - a module to provide platform dependent performance measurements. The module is currently just a proof-of-concept implementation, but will integrated into pybench once it is stable enough. License: pybench license. Author: Marc-Andre Lemburg. ........ r46507 | armin.rigo | 2006-05-28 21:13:17 +0200 (Sun, 28 May 2006) | 15 lines ("Forward-port" of r46506) Remove various dependencies on dictionary order in the standard library tests, and one (clearly an oversight, potentially critical) in the standard library itself - base64.py. Remaining open issues: * test_extcall is an output test, messy to make robust * tarfile.py has a potential bug here, but I'm not familiar enough with this code. Filed in as SF bug #1496501. * urllib2.HTTPPasswordMgr() returns a random result if there is more than one matching root path. I'm asking python-dev for clarification... ........ r46508 | georg.brandl | 2006-05-28 22:11:45 +0200 (Sun, 28 May 2006) | 4 lines The empty string is a valid import path. (fixes #1496539) ........ r46509 | georg.brandl | 2006-05-28 22:23:12 +0200 (Sun, 28 May 2006) | 3 lines Patch #1496206: urllib2 PasswordMgr ./. default ports ........ r46510 | georg.brandl | 2006-05-28 22:57:09 +0200 (Sun, 28 May 2006) | 3 lines Fix refleaks in UnicodeError get and set methods. ........ r46511 | michael.hudson | 2006-05-28 23:19:03 +0200 (Sun, 28 May 2006) | 3 lines use the UnicodeError traversal and clearing functions in UnicodeError subclasses. ........ r46512 | thomas.wouters | 2006-05-28 23:32:12 +0200 (Sun, 28 May 2006) | 4 lines Make last patch valid C89 so Windows compilers can deal with it. ........ r46513 | georg.brandl | 2006-05-28 23:42:54 +0200 (Sun, 28 May 2006) | 3 lines Fix ref-antileak in _struct.c which eventually lead to deallocating None. ........ r46514 | georg.brandl | 2006-05-28 23:57:35 +0200 (Sun, 28 May 2006) | 4 lines Correct None refcount issue in Mac modules. (Are they still used?) ........ r46515 | armin.rigo | 2006-05-29 00:07:08 +0200 (Mon, 29 May 2006) | 3 lines A clearer error message when passing -R to regrtest.py with release builds of Python. ........ r46516 | georg.brandl | 2006-05-29 00:14:04 +0200 (Mon, 29 May 2006) | 3 lines Fix C function calling conventions in _sre module. ........ r46517 | georg.brandl | 2006-05-29 00:34:51 +0200 (Mon, 29 May 2006) | 3 lines Convert audioop over to METH_VARARGS. ........ r46518 | georg.brandl | 2006-05-29 00:38:57 +0200 (Mon, 29 May 2006) | 3 lines METH_NOARGS functions do get called with two args. ........ r46519 | georg.brandl | 2006-05-29 11:46:51 +0200 (Mon, 29 May 2006) | 4 lines Fix refleak in socketmodule. Replace bogus Py_BuildValue calls. Fix refleak in exceptions. ........ r46520 | nick.coghlan | 2006-05-29 14:43:05 +0200 (Mon, 29 May 2006) | 7 lines Apply modified version of Collin Winter's patch #1478788 Renames functional extension module to _functools and adds a Python functools module so that utility functions like update_wrapper can be added easily. ........ r46522 | georg.brandl | 2006-05-29 15:53:16 +0200 (Mon, 29 May 2006) | 3 lines Convert fmmodule to METH_VARARGS. ........ r46523 | georg.brandl | 2006-05-29 16:13:21 +0200 (Mon, 29 May 2006) | 3 lines Fix #1494605. ........ r46524 | georg.brandl | 2006-05-29 16:28:05 +0200 (Mon, 29 May 2006) | 3 lines Handle PyMem_Malloc failure in pystrtod.c. Closes #1494671. ........ r46525 | georg.brandl | 2006-05-29 16:33:55 +0200 (Mon, 29 May 2006) | 3 lines Fix compiler warning. ........ r46526 | georg.brandl | 2006-05-29 16:39:00 +0200 (Mon, 29 May 2006) | 3 lines Fix #1494787 (pyclbr counts whitespace as superclass name) ........ r46527 | bob.ippolito | 2006-05-29 17:47:29 +0200 (Mon, 29 May 2006) | 1 line simplify the struct code a bit (no functional changes) ........ r46528 | armin.rigo | 2006-05-29 19:59:47 +0200 (Mon, 29 May 2006) | 2 lines Silence a warning. ........ r46529 | georg.brandl | 2006-05-29 21:39:45 +0200 (Mon, 29 May 2006) | 3 lines Correct some value converting strangenesses. ........ r46530 | nick.coghlan | 2006-05-29 22:27:44 +0200 (Mon, 29 May 2006) | 1 line When adding a module like functools, it helps to let SVN know about the file. ........ r46531 | georg.brandl | 2006-05-29 22:52:54 +0200 (Mon, 29 May 2006) | 4 lines Patches #1497027 and #972322: try HTTP digest auth first, and watch out for handler name collisions. ........ r46532 | georg.brandl | 2006-05-29 22:57:01 +0200 (Mon, 29 May 2006) | 3 lines Add News entry for last commit. ........ r46533 | georg.brandl | 2006-05-29 23:04:52 +0200 (Mon, 29 May 2006) | 4 lines Make use of METH_O and METH_NOARGS where possible. Use Py_UnpackTuple instead of PyArg_ParseTuple where possible. ........ r46534 | georg.brandl | 2006-05-29 23:58:42 +0200 (Mon, 29 May 2006) | 3 lines Convert more modules to METH_VARARGS. ........ r46535 | georg.brandl | 2006-05-30 00:00:30 +0200 (Tue, 30 May 2006) | 3 lines Whoops. ........ r46536 | fredrik.lundh | 2006-05-30 00:42:07 +0200 (Tue, 30 May 2006) | 4 lines fixed "abc".count("", 100) == -96 error (hopefully, nobody's relying on the current behaviour ;-) ........ r46537 | bob.ippolito | 2006-05-30 00:55:48 +0200 (Tue, 30 May 2006) | 1 line struct: modulo math plus warning on all endian-explicit formats for compatibility with older struct usage (ugly) ........ r46539 | bob.ippolito | 2006-05-30 02:26:01 +0200 (Tue, 30 May 2006) | 1 line Add a length check to aifc to ensure it doesn't write a bogus file ........ r46540 | tim.peters | 2006-05-30 04:25:25 +0200 (Tue, 30 May 2006) | 10 lines deprecated_err(): Stop bizarre warning messages when the tests are run in the order: test_genexps (or any other doctest-based test) test_struct test_doctest The `warnings` module needs an advertised way to save/restore its internal filter list. ........ r46541 | tim.peters | 2006-05-30 04:26:46 +0200 (Tue, 30 May 2006) | 2 lines Whitespace normalization. ........ r46542 | tim.peters | 2006-05-30 04:30:30 +0200 (Tue, 30 May 2006) | 2 lines Set a binary svn:mime-type property on this UTF-8 encoded file. ........ r46543 | neal.norwitz | 2006-05-30 05:18:50 +0200 (Tue, 30 May 2006) | 1 line Simplify further by using AddStringConstant ........ r46544 | tim.peters | 2006-05-30 06:16:25 +0200 (Tue, 30 May 2006) | 6 lines Convert relevant dict internals to Py_ssize_t. I don't have a box with nearly enough RAM, or an OS, that could get close to tickling this, though (requires a dict w/ at least 2**31 entries). ........ r46545 | neal.norwitz | 2006-05-30 06:19:21 +0200 (Tue, 30 May 2006) | 1 line Remove stray | in comment ........ r46546 | neal.norwitz | 2006-05-30 06:25:05 +0200 (Tue, 30 May 2006) | 1 line Use Py_SAFE_DOWNCAST for safety. Fix format strings. Remove 2 more stray | in comment ........ r46547 | neal.norwitz | 2006-05-30 06:43:23 +0200 (Tue, 30 May 2006) | 1 line No DOWNCAST is required since sizeof(Py_ssize_t) >= sizeof(int) and Py_ReprEntr returns an int ........ r46548 | tim.peters | 2006-05-30 07:04:59 +0200 (Tue, 30 May 2006) | 3 lines dict_print(): Explicitly narrow the return value from a (possibly) wider variable. ........ r46549 | tim.peters | 2006-05-30 07:23:59 +0200 (Tue, 30 May 2006) | 5 lines dict_print(): So that Neal & I don't spend the rest of our lives taking turns rewriting code that works ;-), get rid of casting illusions by declaring a new variable with the obvious type. ........ r46550 | georg.brandl | 2006-05-30 09:04:55 +0200 (Tue, 30 May 2006) | 3 lines Restore exception pickle support. #1497319. ........ r46551 | georg.brandl | 2006-05-30 09:13:29 +0200 (Tue, 30 May 2006) | 3 lines Add a test case for exception pickling. args is never NULL. ........ r46552 | neal.norwitz | 2006-05-30 09:21:10 +0200 (Tue, 30 May 2006) | 1 line Don't fail if the (sub)pkgname already exist. ........ r46553 | georg.brandl | 2006-05-30 09:34:45 +0200 (Tue, 30 May 2006) | 3 lines Disallow keyword args for exceptions. ........ r46554 | neal.norwitz | 2006-05-30 09:36:54 +0200 (Tue, 30 May 2006) | 5 lines I'm impatient. I think this will fix a few more problems with the buildbots. I'm not sure this is the best approach, but I can't think of anything better. If this creates problems, feel free to revert, but I think it's safe and should make things a little better. ........ r46555 | georg.brandl | 2006-05-30 10:17:00 +0200 (Tue, 30 May 2006) | 4 lines Do the check for no keyword arguments in __init__ so that subclasses of Exception can be supplied keyword args ........ r46556 | georg.brandl | 2006-05-30 10:47:19 +0200 (Tue, 30 May 2006) | 3 lines Convert test_exceptions to unittest. ........ r46557 | andrew.kuchling | 2006-05-30 14:52:01 +0200 (Tue, 30 May 2006) | 1 line Add SoC name, and reorganize this section a bit ........ r46559 | tim.peters | 2006-05-30 17:53:34 +0200 (Tue, 30 May 2006) | 11 lines PyLong_FromString(): Continued fraction analysis (explained in a new comment) suggests there are almost certainly large input integers in all non-binary input bases for which one Python digit too few is initally allocated to hold the final result. Instead of assert-failing when that happens, allocate more space. Alas, I estimate it would take a few days to find a specific such case, so this isn't backed up by a new test (not to mention that such a case may take hours to run, since conversion time is quadratic in the number of digits, and preliminary attempts suggested that the smallest such inputs contain at least a million digits). ........ r46560 | fredrik.lundh | 2006-05-30 19:11:48 +0200 (Tue, 30 May 2006) | 3 lines changed find/rfind to return -1 for matches outside the source string ........ r46561 | bob.ippolito | 2006-05-30 19:37:54 +0200 (Tue, 30 May 2006) | 1 line Change wrapping terminology to overflow masking ........ r46562 | fredrik.lundh | 2006-05-30 19:39:58 +0200 (Tue, 30 May 2006) | 3 lines changed count to return 0 for slices outside the source string ........ r46568 | tim.peters | 2006-05-31 01:28:02 +0200 (Wed, 31 May 2006) | 2 lines Whitespace normalization. ........ r46569 | brett.cannon | 2006-05-31 04:19:54 +0200 (Wed, 31 May 2006) | 5 lines Clarify wording on default values for strptime(); defaults are used when better values cannot be inferred. Closes bug #1496315. ........ r46572 | neal.norwitz | 2006-05-31 09:43:27 +0200 (Wed, 31 May 2006) | 1 line Calculate smallest properly (it was off by one) and use proper ssize_t types for Win64 ........ r46573 | neal.norwitz | 2006-05-31 10:01:08 +0200 (Wed, 31 May 2006) | 1 line Revert last checkin, it is better to do make distclean ........ r46574 | neal.norwitz | 2006-05-31 11:02:44 +0200 (Wed, 31 May 2006) | 3 lines On 64-bit platforms running test_struct after test_tarfile would fail since the deprecation warning wouldn't be raised. ........ r46575 | thomas.heller | 2006-05-31 13:37:58 +0200 (Wed, 31 May 2006) | 3 lines PyTuple_Pack is not available in Python 2.3, but ctypes must stay compatible with that. ........ r46576 | andrew.kuchling | 2006-05-31 15:18:56 +0200 (Wed, 31 May 2006) | 1 line 'functional' module was renamed to 'functools' ........ r46577 | kristjan.jonsson | 2006-05-31 15:35:41 +0200 (Wed, 31 May 2006) | 1 line Fixup the PCBuild8 project directory. exceptions.c have moved to Objects, and the functionalmodule.c has been replaced with _functoolsmodule.c. Other minor changes to .vcproj files and .sln to fix compilation ........ r46578 | andrew.kuchling | 2006-05-31 16:08:48 +0200 (Wed, 31 May 2006) | 15 lines [Bug #1473048] SimpleXMLRPCServer and DocXMLRPCServer don't look at the path of the HTTP request at all; you can POST or GET from / or /RPC2 or /blahblahblah with the same results. Security scanners that look for /cgi-bin/phf will therefore report lots of vulnerabilities. Fix: add a .rpc_paths attribute to the SimpleXMLRPCServer class, and report a 404 error if the path isn't on the allowed list. Possibly-controversial aspect of this change: the default makes only '/' and '/RPC2' legal. Maybe this will break people's applications (though I doubt it). We could just set the default to an empty tuple, which would exactly match the current behaviour. ........ r46579 | andrew.kuchling | 2006-05-31 16:12:47 +0200 (Wed, 31 May 2006) | 1 line Mention SimpleXMLRPCServer change ........ r46580 | tim.peters | 2006-05-31 16:28:07 +0200 (Wed, 31 May 2006) | 2 lines Trimmed trailing whitespace. ........ r46581 | tim.peters | 2006-05-31 17:33:22 +0200 (Wed, 31 May 2006) | 4 lines _range_error(): Speed and simplify (there's no real need for loops here). Assert that size_t is actually big enough, and that f->size is at least one. Wrap a long line. ........ r46582 | tim.peters | 2006-05-31 17:34:37 +0200 (Wed, 31 May 2006) | 2 lines Repaired error in new comment. ........ r46584 | neal.norwitz | 2006-06-01 07:32:49 +0200 (Thu, 01 Jun 2006) | 4 lines Remove ; at end of macro. There was a compiler recently that warned about extra semi-colons. It may have been the HP C compiler. This file will trigger a bunch of those warnings now. ........ r46585 | georg.brandl | 2006-06-01 08:39:19 +0200 (Thu, 01 Jun 2006) | 3 lines Correctly unpickle 2.4 exceptions via __setstate__ (patch #1498571) ........ r46586 | georg.brandl | 2006-06-01 10:27:32 +0200 (Thu, 01 Jun 2006) | 3 lines Correctly allocate complex types with tp_alloc. (bug #1498638) ........ r46587 | georg.brandl | 2006-06-01 14:30:46 +0200 (Thu, 01 Jun 2006) | 2 lines Correctly dispatch Faults in loads (patch #1498627) ........ r46588 | georg.brandl | 2006-06-01 15:00:49 +0200 (Thu, 01 Jun 2006) | 3 lines Some code style tweaks, and remove apply. ........ r46589 | armin.rigo | 2006-06-01 15:19:12 +0200 (Thu, 01 Jun 2006) | 5 lines [ 1497053 ] Let dicts propagate the exceptions in user __eq__(). [ 1456209 ] dictresize() vulnerability ( <- backport candidate ). ........ r46590 | tim.peters | 2006-06-01 15:41:46 +0200 (Thu, 01 Jun 2006) | 2 lines Whitespace normalization. ........ r46591 | tim.peters | 2006-06-01 15:49:23 +0200 (Thu, 01 Jun 2006) | 2 lines Record bugs 1275608 and 1456209 as being fixed. ........ r46592 | tim.peters | 2006-06-01 15:56:26 +0200 (Thu, 01 Jun 2006) | 5 lines Re-enable a new empty-string test added during the NFS sprint, but disabled then because str and unicode strings gave different results. The implementations were repaired later during the sprint, but the new test remained disabled. ........ r46594 | tim.peters | 2006-06-01 17:50:44 +0200 (Thu, 01 Jun 2006) | 7 lines Armin committed his patch while I was reviewing it (I'm sure he didn't know this), so merged in some changes I made during review. Nothing material apart from changing a new `mask` local from int to Py_ssize_t. Mostly this is repairing comments that were made incorrect, and adding new comments. Also a few minor code rewrites for clarity or helpful succinctness. ........ r46599 | neal.norwitz | 2006-06-02 06:45:53 +0200 (Fri, 02 Jun 2006) | 1 line Convert docstrings to comments so regrtest -v prints method names ........ r46600 | neal.norwitz | 2006-06-02 06:50:49 +0200 (Fri, 02 Jun 2006) | 2 lines Fix memory leak found by valgrind. ........ r46601 | neal.norwitz | 2006-06-02 06:54:52 +0200 (Fri, 02 Jun 2006) | 1 line More memory leaks from valgrind ........ r46602 | neal.norwitz | 2006-06-02 08:23:00 +0200 (Fri, 02 Jun 2006) | 11 lines Patch #1357836: Prevent an invalid memory read from test_coding in case the done flag is set. In that case, the loop isn't entered. I wonder if rather than setting the done flag in the cases before the loop, if they should just exit early. This code looks like it should be refactored. Backport candidate (also the early break above if decoding_fgets fails) ........ r46603 | martin.blais | 2006-06-02 15:03:43 +0200 (Fri, 02 Jun 2006) | 1 line Fixed struct test to not use unittest. ........ r46605 | tim.peters | 2006-06-03 01:22:51 +0200 (Sat, 03 Jun 2006) | 10 lines pprint functions used to sort a dict (by key) if and only if the output required more than one line. "Small" dicts got displayed in seemingly random order (the hash-induced order produced by dict.__repr__). None of this was documented. Now pprint functions always sort dicts by key, and the docs promise it. This was proposed and agreed to during the PyCon 2006 core sprint -- I just didn't have time for it before now. ........
2006-06-08 11:42:34 -03:00
marshal_load(PyObject *self, PyObject *f)
1991-06-04 16:42:30 -03:00
{
PyObject *data, *result;
_Py_IDENTIFIER(read);
RFILE rf;
/*
* Make a call to the read method, but read zero bytes.
* This is to ensure that the object passed in at least
* has a read method which returns bytes.
* This can be removed if we guarantee good error handling
* for r_string()
*/
data = _PyObject_CallMethodId(f, &PyId_read, "i", 0);
if (data == NULL)
return NULL;
if (!PyBytes_Check(data)) {
PyErr_Format(PyExc_TypeError,
"f.read() returned not bytes but %.100s",
data->ob_type->tp_name);
result = NULL;
}
else {
rf.depth = 0;
rf.fp = NULL;
rf.readable = f;
rf.current_filename = NULL;
rf.ptr = rf.end = NULL;
rf.buf = NULL;
if ((rf.refs = PyList_New(0)) != NULL) {
result = read_object(&rf);
Py_DECREF(rf.refs);
if (rf.buf != NULL)
PyMem_FREE(rf.buf);
} else
result = NULL;
}
Py_DECREF(data);
return result;
}
PyDoc_STRVAR(load_doc,
"load(file)\n\
\n\
Read one value from the open file and return it. If no valid value is\n\
read (e.g. because the data has a different Python version's\n\
incompatible marshal format), raise EOFError, ValueError or TypeError.\n\
The file must be an open file object opened in binary mode ('rb' or\n\
'r+b').\n\
\n\
Note: If an object containing an unsupported type was marshalled with\n\
dump(), load() will substitute None for the unmarshallable type.");
static PyObject *
marshal_dumps(PyObject *self, PyObject *args)
{
PyObject *x;
int version = Py_MARSHAL_VERSION;
if (!PyArg_ParseTuple(args, "O|i:dumps", &x, &version))
return NULL;
return PyMarshal_WriteObjectToString(x, version);
}
PyDoc_STRVAR(dumps_doc,
"dumps(value[, version])\n\
\n\
Return the string that would be written to a file by dump(value, file).\n\
The value must be a supported type. Raise a ValueError exception if\n\
value has (or contains an object that has) an unsupported type.\n\
\n\
The version argument indicates the data format that dumps should use.");
static PyObject *
marshal_loads(PyObject *self, PyObject *args)
{
RFILE rf;
Py_buffer p;
char *s;
Py_ssize_t n;
PyObject* result;
if (!PyArg_ParseTuple(args, "y*:loads", &p))
return NULL;
s = p.buf;
n = p.len;
rf.fp = NULL;
rf.readable = NULL;
rf.current_filename = NULL;
rf.ptr = s;
rf.end = s + n;
rf.depth = 0;
if ((rf.refs = PyList_New(0)) == NULL)
return NULL;
result = read_object(&rf);
PyBuffer_Release(&p);
Py_DECREF(rf.refs);
return result;
1991-06-04 16:42:30 -03:00
}
PyDoc_STRVAR(loads_doc,
"loads(bytes)\n\
\n\
Convert the bytes object to a value. If no valid value is found, raise\n\
EOFError, ValueError or TypeError. Extra characters in the input are\n\
ignored.");
static PyMethodDef marshal_methods[] = {
{"dump", marshal_dump, METH_VARARGS, dump_doc},
{"load", marshal_load, METH_O, load_doc},
{"dumps", marshal_dumps, METH_VARARGS, dumps_doc},
{"loads", marshal_loads, METH_VARARGS, loads_doc},
{NULL, NULL} /* sentinel */
1991-06-04 16:42:30 -03:00
};
PyDoc_STRVAR(module_doc,
"This module contains functions that can read and write Python values in\n\
a binary format. The format is specific to Python, but independent of\n\
machine architecture issues.\n\
\n\
Not all Python object types are supported; in general, only objects\n\
whose value is independent from a particular invocation of Python can be\n\
written and read by this module. The following types are supported:\n\
None, integers, floating point numbers, strings, bytes, bytearrays,\n\
tuples, lists, sets, dictionaries, and code objects, where it\n\
should be understood that tuples, lists and dictionaries are only\n\
supported as long as the values contained therein are themselves\n\
supported; and recursive lists and dictionaries should not be written\n\
(they will cause infinite loops).\n\
\n\
Variables:\n\
\n\
version -- indicates the format that the module uses. Version 0 is the\n\
historical format, version 1 shares interned strings and version 2\n\
uses a binary format for floating point numbers.\n\
Version 3 shares common object references (New in version 3.4).\n\
\n\
Functions:\n\
\n\
dump() -- write value to a file\n\
load() -- read value from a file\n\
dumps() -- write value to a string\n\
loads() -- read value from a string");
static struct PyModuleDef marshalmodule = {
PyModuleDef_HEAD_INIT,
"marshal",
module_doc,
0,
marshal_methods,
NULL,
NULL,
NULL,
NULL
};
PyMODINIT_FUNC
PyMarshal_Init(void)
1991-06-04 16:42:30 -03:00
{
PyObject *mod = PyModule_Create(&marshalmodule);
if (mod == NULL)
return NULL;
PyModule_AddIntConstant(mod, "version", Py_MARSHAL_VERSION);
return mod;
1991-06-04 16:42:30 -03:00
}