- Issue #10181: New memoryview implementation fixes multiple ownership

and lifetime issues of dynamically allocated Py_buffer members (#9990)
  as well as crashes (#8305, #7433). Many new features have been added
  (See whatsnew/3.3), and the documentation has been updated extensively.
  The ndarray test object from _testbuffer.c implements all aspects of
  PEP-3118, so further development towards the complete implementation
  of the PEP can proceed in a test-driven manner.

  Thanks to Nick Coghlan, Antoine Pitrou and Pauli Virtanen for review
  and many ideas.

- Issue #12834: Fix incorrect results of memoryview.tobytes() for
  non-contiguous arrays.

- Issue #5231: Introduce memoryview.cast() method that allows changing
  format and shape without making a copy of the underlying memory.

Doc/c-api/buffer.rst

@@ -7,6 +7,7 @@ Buffer Protocol
 
 .. sectionauthor:: Greg Stein <gstein@lyra.org>
 .. sectionauthor:: Benjamin Peterson
+.. sectionauthor:: Stefan Krah
 
 
 .. index::
@@ -20,7 +21,7 @@ as image processing or numeric analysis.
 
 While each of these types have their own semantics, they share the common
 characteristic of being backed by a possibly large memory buffer.  It is
-then desireable, in some situations, to access that buffer directly and
+then desirable, in some situations, to access that buffer directly and
 without intermediate copying.
 
 Python provides such a facility at the C level in the form of the *buffer
@@ -60,8 +61,10 @@ isn't needed anymore.  Failure to do so could lead to various issues such as
 resource leaks.
 
 
-The buffer structure
+.. _buffer-structure:
-====================
+
+Buffer structure
+================
 
 Buffer structures (or simply "buffers") are useful as a way to expose the
 binary data from another object to the Python programmer.  They can also be
@@ -81,93 +84,330 @@ can be created.
 
 .. c:type:: Py_buffer
 
-   .. c:member:: void *buf
+   .. c:member:: void \*obj
 
-      A pointer to the start of the memory for the object.
+      A new reference to the exporting object or *NULL*. The reference is owned
+      by the consumer and automatically decremented and set to *NULL* by
+      :c:func:`PyBuffer_Release`.
+
+      For temporary buffers that are wrapped by :c:func:`PyMemoryView_FromBuffer`
+      this field must be *NULL*.
+
+   .. c:member:: void \*buf
+
+      A pointer to the start of the logical structure described by the buffer
+      fields. This can be any location within the underlying physical memory
+      block of the exporter. For example, with negative :c:member:`~Py_buffer.strides`
+      the value may point to the end of the memory block.
+
+      For contiguous arrays, the value points to the beginning of the memory
+      block.
 
    .. c:member:: Py_ssize_t len
-      :noindex:
 
-      The total length of the memory in bytes.
+      ``product(shape) * itemsize``. For contiguous arrays, this is the length
+      of the underlying memory block. For non-contiguous arrays, it is the length
+      that the logical structure would have if it were copied to a contiguous
+      representation.
+
+      Accessing ``((char *)buf)[0] up to ((char *)buf)[len-1]`` is only valid
+      if the buffer has been obtained by a request that guarantees contiguity. In
+      most cases such a request will be :c:macro:`PyBUF_SIMPLE` or :c:macro:`PyBUF_WRITABLE`.
 
    .. c:member:: int readonly
 
-      An indicator of whether the buffer is read only.
+      An indicator of whether the buffer is read-only. This field is controlled
-
+      by the :c:macro:`PyBUF_WRITABLE` flag.
-   .. c:member:: const char *format
-      :noindex:
-
-      A *NULL* terminated string in :mod:`struct` module style syntax giving
-      the contents of the elements available through the buffer.  If this is
-      *NULL*, ``"B"`` (unsigned bytes) is assumed.
-
-   .. c:member:: int ndim
-
-      The number of dimensions the memory represents as a multi-dimensional
-      array.  If it is 0, :c:data:`strides` and :c:data:`suboffsets` must be
-      *NULL*.
-
-   .. c:member:: Py_ssize_t *shape
-
-      An array of :c:type:`Py_ssize_t`\s the length of :c:data:`ndim` giving the
-      shape of the memory as a multi-dimensional array.  Note that
-      ``((*shape)[0] * ... * (*shape)[ndims-1])*itemsize`` should be equal to
-      :c:data:`len`.
-
-   .. c:member:: Py_ssize_t *strides
-
-      An array of :c:type:`Py_ssize_t`\s the length of :c:data:`ndim` giving the
-      number of bytes to skip to get to a new element in each dimension.
-
-   .. c:member:: Py_ssize_t *suboffsets
-
-      An array of :c:type:`Py_ssize_t`\s the length of :c:data:`ndim`.  If these
-      suboffset numbers are greater than or equal to 0, then the value stored
-      along the indicated dimension is a pointer and the suboffset value
-      dictates how many bytes to add to the pointer after de-referencing. A
-      suboffset value that it negative indicates that no de-referencing should
-      occur (striding in a contiguous memory block).
-
-      Here is a function that returns a pointer to the element in an N-D array
-      pointed to by an N-dimensional index when there are both non-NULL strides
-      and suboffsets::
-
-          void *get_item_pointer(int ndim, void *buf, Py_ssize_t *strides,
-              Py_ssize_t *suboffsets, Py_ssize_t *indices) {
-              char *pointer = (char*)buf;
-              int i;
-              for (i = 0; i < ndim; i++) {
-                  pointer += strides[i] * indices[i];
-                  if (suboffsets[i] >=0 ) {
-                      pointer = *((char**)pointer) + suboffsets[i];
-                  }
-              }
-              return (void*)pointer;
-           }
-
 
    .. c:member:: Py_ssize_t itemsize
 
-      This is a storage for the itemsize (in bytes) of each element of the
+      Item size in bytes of a single element. Same as the value of :func:`struct.calcsize`
-      shared memory. It is technically un-necessary as it can be obtained
+      called on non-NULL :c:member:`~Py_buffer.format` values.
-      using :c:func:`PyBuffer_SizeFromFormat`, however an exporter may know
-      this information without parsing the format string and it is necessary
-      to know the itemsize for proper interpretation of striding. Therefore,
-      storing it is more convenient and faster.
 
-   .. c:member:: void *internal
+      Important exception: If a consumer requests a buffer without the
+      :c:macro:`PyBUF_FORMAT` flag, :c:member:`~Py_Buffer.format` will
+      be set to  *NULL*,  but :c:member:`~Py_buffer.itemsize` still has
+      the value for the original format.
+
+      If :c:member:`~Py_Buffer.shape` is present, the equality
+      ``product(shape) * itemsize == len`` still holds and the consumer
+      can use :c:member:`~Py_buffer.itemsize` to navigate the buffer.
+
+      If :c:member:`~Py_Buffer.shape` is *NULL* as a result of a :c:macro:`PyBUF_SIMPLE`
+      or a :c:macro:`PyBUF_WRITABLE` request, the consumer must disregard
+      :c:member:`~Py_buffer.itemsize` and assume ``itemsize == 1``.
+
+   .. c:member:: const char \*format
+
+      A *NUL* terminated string in :mod:`struct` module style syntax describing
+      the contents of a single item. If this is *NULL*, ``"B"`` (unsigned bytes)
+      is assumed.
+
+      This field is controlled by the :c:macro:`PyBUF_FORMAT` flag.
+
+   .. c:member:: int ndim
+
+      The number of dimensions the memory represents as an n-dimensional array.
+      If it is 0, :c:member:`~Py_Buffer.buf` points to a single item representing
+      a scalar. In this case, :c:member:`~Py_buffer.shape`, :c:member:`~Py_buffer.strides`
+      and :c:member:`~Py_buffer.suboffsets` MUST be *NULL*.
+
+      The macro :c:macro:`PyBUF_MAX_NDIM` limits the maximum number of dimensions
+      to 64. Exporters MUST respect this limit, consumers of multi-dimensional
+      buffers SHOULD be able to handle up to :c:macro:`PyBUF_MAX_NDIM` dimensions.
+
+   .. c:member:: Py_ssize_t \*shape
+
+      An array of :c:type:`Py_ssize_t` of length :c:member:`~Py_buffer.ndim`
+      indicating the shape of the memory as an n-dimensional array. Note that
+      ``shape[0] * ... * shape[ndim-1] * itemsize`` MUST be equal to
+      :c:member:`~Py_buffer.len`.
+
+      Shape values are restricted to ``shape[n] >= 0``. The case
+      ``shape[n] == 0`` requires special attention. See `complex arrays`_
+      for further information.
+
+      The shape array is read-only for the consumer.
+
+   .. c:member:: Py_ssize_t \*strides
+
+      An array of :c:type:`Py_ssize_t` of length :c:member:`~Py_buffer.ndim`
+      giving the number of bytes to skip to get to a new element in each
+      dimension.
+
+      Stride values can be any integer. For regular arrays, strides are
+      usually positive, but a consumer MUST be able to handle the case
+      ``strides[n] <= 0``. See `complex arrays`_ for further information.
+
+      The strides array is read-only for the consumer.
+
+   .. c:member:: Py_ssize_t \*suboffsets
+
+      An array of :c:type:`Py_ssize_t` of length :c:member:`~Py_buffer.ndim`.
+      If ``suboffsets[n] >= 0``, the values stored along the nth dimension are
+      pointers and the suboffset value dictates how many bytes to add to each
+      pointer after de-referencing. A suboffset value that is negative
+      indicates that no de-referencing should occur (striding in a contiguous
+      memory block).
+
+      This type of array representation is used by the Python Imaging Library
+      (PIL). See `complex arrays`_ for further information how to access elements
+      of such an array.
+
+      The suboffsets array is read-only for the consumer.
+
+   .. c:member:: void \*internal
 
       This is for use internally by the exporting object. For example, this
       might be re-cast as an integer by the exporter and used to store flags
       about whether or not the shape, strides, and suboffsets arrays must be
-      freed when the buffer is released. The consumer should never alter this
+      freed when the buffer is released. The consumer MUST NOT alter this
       value.
 
+.. _buffer-request-types:
+
+Buffer request types
+====================
+
+Buffers are usually obtained by sending a buffer request to an exporting
+object via :c:func:`PyObject_GetBuffer`. Since the complexity of the logical
+structure of the memory can vary drastically, the consumer uses the *flags*
+argument to specify the exact buffer type it can handle.
+
+All :c:data:`Py_buffer` fields are unambiguously defined by the request
+type.
+
+request-independent fields
+~~~~~~~~~~~~~~~~~~~~~~~~~~
+The following fields are not influenced by *flags* and must always be filled in
+with the correct values: :c:member:`~Py_buffer.obj`, :c:member:`~Py_buffer.buf`,
+:c:member:`~Py_buffer.len`, :c:member:`~Py_buffer.itemsize`, :c:member:`~Py_buffer.ndim`.
+
+
+readonly, format
+~~~~~~~~~~~~~~~~
+
+   .. c:macro:: PyBUF_WRITABLE
+
+      Controls the :c:member:`~Py_buffer.readonly` field. If set, the exporter
+      MUST provide a writable buffer or else report failure. Otherwise, the
+      exporter MAY provide either a read-only or writable buffer, but the choice
+      MUST be consistent for all consumers.
+
+   .. c:macro:: PyBUF_FORMAT
+
+      Controls the :c:member:`~Py_buffer.format` field. If set, this field MUST
+      be filled in correctly. Otherwise, this field MUST be *NULL*.
+
+
+:c:macro:`PyBUF_WRITABLE` can be \|'d to any of the flags in the next section.
+Since :c:macro:`PyBUF_SIMPLE` is defined as 0, :c:macro:`PyBUF_WRITABLE`
+can be used as a stand-alone flag to request a simple writable buffer.
+
+:c:macro:`PyBUF_FORMAT` can be \|'d to any of the flags except :c:macro:`PyBUF_SIMPLE`.
+The latter already implies format ``B`` (unsigned bytes).
+
+
+shape, strides, suboffsets
+~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+The flags that control the logical structure of the memory are listed
+in decreasing order of complexity. Note that each flag contains all bits
+of the flags below it.
+
+
++-----------------------------+-------+---------+------------+
+|  Request                    | shape | strides | suboffsets |
++=============================+=======+=========+============+
+| .. c:macro:: PyBUF_INDIRECT |  yes  |   yes   | if needed  |
++-----------------------------+-------+---------+------------+
+| .. c:macro:: PyBUF_STRIDES  |  yes  |   yes   |    NULL    |
++-----------------------------+-------+---------+------------+
+| .. c:macro:: PyBUF_ND       |  yes  |   NULL  |    NULL    |
++-----------------------------+-------+---------+------------+
+| .. c:macro:: PyBUF_SIMPLE   |  NULL |   NULL  |    NULL    |
++-----------------------------+-------+---------+------------+
+
+
+contiguity requests
+~~~~~~~~~~~~~~~~~~~
+
+C or Fortran contiguity can be explicitly requested, with and without stride
+information. Without stride information, the buffer must be C-contiguous.
+
++-----------------------------------+-------+---------+------------+--------+
+|  Request                          | shape | strides | suboffsets | contig |
++===================================+=======+=========+============+========+
+| .. c:macro:: PyBUF_C_CONTIGUOUS   |  yes  |   yes   |    NULL    |   C    |
++-----------------------------------+-------+---------+------------+--------+
+| .. c:macro:: PyBUF_F_CONTIGUOUS   |  yes  |   yes   |    NULL    |   F    |
++-----------------------------------+-------+---------+------------+--------+
+| .. c:macro:: PyBUF_ANY_CONTIGUOUS |  yes  |   yes   |    NULL    | C or F |
++-----------------------------------+-------+---------+------------+--------+
+| .. c:macro:: PyBUF_ND             |  yes  |   NULL  |    NULL    |   C    |
++-----------------------------------+-------+---------+------------+--------+
+
+
+compound requests
+~~~~~~~~~~~~~~~~~
+
+All possible requests are fully defined by some combination of the flags in
+the previous section. For convenience, the buffer protocol provides frequently
+used combinations as single flags.
+
+In the following table *U* stands for undefined contiguity. The consumer would
+have to call :c:func:`PyBuffer_IsContiguous` to determine contiguity.
+
+
+
++-------------------------------+-------+---------+------------+--------+----------+--------+
+|  Request                      | shape | strides | suboffsets | contig | readonly | format |
++===============================+=======+=========+============+========+==========+========+
+| .. c:macro:: PyBUF_FULL       |  yes  |   yes   | if needed  |   U    |     0    |  yes   |
++-------------------------------+-------+---------+------------+--------+----------+--------+
+| .. c:macro:: PyBUF_FULL_RO    |  yes  |   yes   | if needed  |   U    |  1 or 0  |  yes   |
++-------------------------------+-------+---------+------------+--------+----------+--------+
+| .. c:macro:: PyBUF_RECORDS    |  yes  |   yes   |    NULL    |   U    |     0    |  yes   |
++-------------------------------+-------+---------+------------+--------+----------+--------+
+| .. c:macro:: PyBUF_RECORDS_RO |  yes  |   yes   |    NULL    |   U    |  1 or 0  |  yes   |
++-------------------------------+-------+---------+------------+--------+----------+--------+
+| .. c:macro:: PyBUF_STRIDED    |  yes  |   yes   |    NULL    |   U    |     0    |  NULL  |
++-------------------------------+-------+---------+------------+--------+----------+--------+
+| .. c:macro:: PyBUF_STRIDED_RO |  yes  |   yes   |    NULL    |   U    |  1 or 0  |  NULL  |
++-------------------------------+-------+---------+------------+--------+----------+--------+
+| .. c:macro:: PyBUF_CONTIG     |  yes  |   NULL  |    NULL    |   C    |     0    |  NULL  |
++-------------------------------+-------+---------+------------+--------+----------+--------+
+| .. c:macro:: PyBUF_CONTIG_RO  |  yes  |   NULL  |    NULL    |   C    |  1 or 0  |  NULL  |
++-------------------------------+-------+---------+------------+--------+----------+--------+
+
+
+Complex arrays
+==============
+
+NumPy-style: shape and strides
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+The logical structure of NumPy-style arrays is defined by :c:member:`~Py_buffer.itemsize`,
+:c:member:`~Py_buffer.ndim`, :c:member:`~Py_buffer.shape` and :c:member:`~Py_buffer.strides`.
+
+If ``ndim == 0``, the memory location pointed to by :c:member:`~Py_buffer.buf` is
+interpreted as a scalar of size :c:member:`~Py_buffer.itemsize`. In that case,
+both :c:member:`~Py_buffer.shape` and :c:member:`~Py_buffer.strides` are *NULL*.
+
+If :c:member:`~Py_buffer.strides` is *NULL*, the array is interpreted as
+a standard n-dimensional C-array. Otherwise, the consumer must access an
+n-dimensional array as follows:
+
+   ``ptr = (char *)buf + indices[0] * strides[0] + ... + indices[n-1] * strides[n-1]``
+   ``item = *((typeof(item) *)ptr);``
+
+
+As noted above, :c:member:`~Py_buffer.buf` can point to any location within
+the actual memory block. An exporter can check the validity of a buffer with
+this function:
+
+.. code-block:: python
+
+   def verify_structure(memlen, itemsize, ndim, shape, strides, offset):
+       """Verify that the parameters represent a valid array within
+          the bounds of the allocated memory:
+              char *mem: start of the physical memory block
+              memlen: length of the physical memory block
+              offset: (char *)buf - mem
+       """
+       if offset % itemsize:
+           return False
+       if offset < 0 or offset+itemsize > memlen:
+           return False
+       if any(v % itemsize for v in strides):
+           return False
+
+       if ndim <= 0:
+           return ndim == 0 and not shape and not strides
+       if 0 in shape:
+           return True
+
+       imin = sum(strides[j]*(shape[j]-1) for j in range(ndim)
+                  if strides[j] <= 0)
+       imax = sum(strides[j]*(shape[j]-1) for j in range(ndim)
+                  if strides[j] > 0)
+
+       return 0 <= offset+imin and offset+imax+itemsize <= memlen
+
+
+PIL-style: shape, strides and suboffsets
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+In addition to the regular items, PIL-style arrays can contain pointers
+that must be followed in order to get to the next element in a dimension.
+For example, the regular three-dimensional C-array ``char v[2][2][3]`` can
+also be viewed as an array of 2 pointers to 2 two-dimensional arrays:
+``char (*v[2])[2][3]``. In suboffsets representation, those two pointers
+can be embedded at the start of :c:member:`~Py_buffer.buf`, pointing
+to two ``char x[2][3]`` arrays that can be located anywhere in memory.
+
+
+Here is a function that returns a pointer to the element in an N-D array
+pointed to by an N-dimensional index when there are both non-NULL strides
+and suboffsets::
+
+   void *get_item_pointer(int ndim, void *buf, Py_ssize_t *strides,
+                          Py_ssize_t *suboffsets, Py_ssize_t *indices) {
+       char *pointer = (char*)buf;
+       int i;
+       for (i = 0; i < ndim; i++) {
+           pointer += strides[i] * indices[i];
+           if (suboffsets[i] >=0 ) {
+               pointer = *((char**)pointer) + suboffsets[i];
+           }
+       }
+       return (void*)pointer;
+   }
+
 
 Buffer-related functions
 ========================
 
-
 .. c:function:: int PyObject_CheckBuffer(PyObject *obj)
 
    Return 1 if *obj* supports the buffer interface otherwise 0.  When 1 is
@@ -175,152 +415,69 @@ Buffer-related functions
    succeed.
 
 
-.. c:function:: int PyObject_GetBuffer(PyObject *obj, Py_buffer *view, int flags)
+.. c:function:: int PyObject_GetBuffer(PyObject *exporter, Py_buffer *view, int flags)
 
-      Export a view over some internal data from the target object *obj*.
+   Send a request to *exporter* to fill in *view* as specified by  *flags*.
-      *obj* must not be NULL, and *view* must point to an existing
+   If the exporter cannot provide a buffer of the exact type, it MUST raise
-      :c:type:`Py_buffer` structure allocated by the caller (most uses of
+   :c:data:`PyExc_BufferError`, set :c:member:`view->obj` to *NULL* and
-      this function will simply declare a local variable of type
+   return -1.
-      :c:type:`Py_buffer`).  The *flags* argument is a bit field indicating
-      what kind of buffer is requested.  The buffer interface allows
-      for complicated memory layout possibilities; however, some callers
-      won't want to handle all the complexity and instead request a simple
-      view of the target object (using :c:macro:`PyBUF_SIMPLE` for a read-only
-      view and :c:macro:`PyBUF_WRITABLE` for a read-write view).
 
-      Some exporters may not be able to share memory in every possible way and
+   On success, fill in *view*, set :c:member:`view->obj` to a new reference
-      may need to raise errors to signal to some consumers that something is
+   to *exporter* and return 0.
-      just not possible. These errors should be a :exc:`BufferError` unless
-      there is another error that is actually causing the problem. The
-      exporter can use flags information to simplify how much of the
-      :c:data:`Py_buffer` structure is filled in with non-default values and/or
-      raise an error if the object can't support a simpler view of its memory.
 
-      On success, 0 is returned and the *view* structure is filled with useful
+   Successful calls to :c:func:`PyObject_GetBuffer` must be paired with calls
-      values.  On error, -1 is returned and an exception is raised; the *view*
+   to :c:func:`PyBuffer_Release`, similar to :c:func:`malloc` and :c:func:`free`.
-      is left in an undefined state.
+   Thus, after the consumer is done with the buffer, :c:func:`PyBuffer_Release`
-
+   must be called exactly once.
-      The following are the possible values to the *flags* arguments.
-
-      .. c:macro:: PyBUF_SIMPLE
-
-         This is the default flag.  The returned buffer exposes a read-only
-         memory area.  The format of data is assumed to be raw unsigned bytes,
-         without any particular structure.  This is a "stand-alone" flag
-         constant.  It never needs to be '|'d to the others.  The exporter will
-         raise an error if it cannot provide such a contiguous buffer of bytes.
-
-      .. c:macro:: PyBUF_WRITABLE
-
-         Like :c:macro:`PyBUF_SIMPLE`, but the returned buffer is writable.  If
-         the exporter doesn't support writable buffers, an error is raised.
-
-      .. c:macro:: PyBUF_STRIDES
-
-         This implies :c:macro:`PyBUF_ND`.  The returned buffer must provide
-         strides information (i.e. the strides cannot be NULL).  This would be
-         used when the consumer can handle strided, discontiguous arrays.
-         Handling strides automatically assumes you can handle shape.  The
-         exporter can raise an error if a strided representation of the data is
-         not possible (i.e. without the suboffsets).
-
-      .. c:macro:: PyBUF_ND
-
-         The returned buffer must provide shape information.  The memory will be
-         assumed C-style contiguous (last dimension varies the fastest).  The
-         exporter may raise an error if it cannot provide this kind of
-         contiguous buffer.  If this is not given then shape will be *NULL*.
-
-      .. c:macro:: PyBUF_C_CONTIGUOUS
-                  PyBUF_F_CONTIGUOUS
-                  PyBUF_ANY_CONTIGUOUS
-
-         These flags indicate that the contiguity returned buffer must be
-         respectively, C-contiguous (last dimension varies the fastest), Fortran
-         contiguous (first dimension varies the fastest) or either one.  All of
-         these flags imply :c:macro:`PyBUF_STRIDES` and guarantee that the
-         strides buffer info structure will be filled in correctly.
-
-      .. c:macro:: PyBUF_INDIRECT
-
-         This flag indicates the returned buffer must have suboffsets
-         information (which can be NULL if no suboffsets are needed).  This can
-         be used when the consumer can handle indirect array referencing implied
-         by these suboffsets. This implies :c:macro:`PyBUF_STRIDES`.
-
-      .. c:macro:: PyBUF_FORMAT
-
-         The returned buffer must have true format information if this flag is
-         provided.  This would be used when the consumer is going to be checking
-         for what 'kind' of data is actually stored.  An exporter should always
-         be able to provide this information if requested.  If format is not
-         explicitly requested then the format must be returned as *NULL* (which
-         means ``'B'``, or unsigned bytes).
-
-      .. c:macro:: PyBUF_STRIDED
-
-         This is equivalent to ``(PyBUF_STRIDES | PyBUF_WRITABLE)``.
-
-      .. c:macro:: PyBUF_STRIDED_RO
-
-         This is equivalent to ``(PyBUF_STRIDES)``.
-
-      .. c:macro:: PyBUF_RECORDS
-
-         This is equivalent to ``(PyBUF_STRIDES | PyBUF_FORMAT |
-         PyBUF_WRITABLE)``.
-
-      .. c:macro:: PyBUF_RECORDS_RO
-
-         This is equivalent to ``(PyBUF_STRIDES | PyBUF_FORMAT)``.
-
-      .. c:macro:: PyBUF_FULL
-
-         This is equivalent to ``(PyBUF_INDIRECT | PyBUF_FORMAT |
-         PyBUF_WRITABLE)``.
-
-      .. c:macro:: PyBUF_FULL_RO
-
-         This is equivalent to ``(PyBUF_INDIRECT | PyBUF_FORMAT)``.
-
-      .. c:macro:: PyBUF_CONTIG
-
-         This is equivalent to ``(PyBUF_ND | PyBUF_WRITABLE)``.
-
-      .. c:macro:: PyBUF_CONTIG_RO
-
-         This is equivalent to ``(PyBUF_ND)``.
 
 
 .. c:function:: void PyBuffer_Release(Py_buffer *view)
 
-   Release the buffer *view*.  This should be called when the buffer is no
+   Release the buffer *view* and decrement the reference count for
-   longer being used as it may free memory from it.
+   :c:member:`view->obj`. This function MUST be called when the buffer
+   is no longer being used, otherwise reference leaks may occur.
+
+   It is an error to call this function on a buffer that was not obtained via
+   :c:func:`PyObject_GetBuffer`.
 
 
 .. c:function:: Py_ssize_t PyBuffer_SizeFromFormat(const char *)
 
-   Return the implied :c:data:`~Py_buffer.itemsize` from the struct-stype
+   Return the implied :c:data:`~Py_buffer.itemsize` from :c:data:`~Py_buffer.format`.
-   :c:data:`~Py_buffer.format`.
+   This function is not yet implemented.
 
 
-.. c:function:: int PyBuffer_IsContiguous(Py_buffer *view, char fortran)
+.. c:function:: int PyBuffer_IsContiguous(Py_buffer *view, char order)
 
-   Return 1 if the memory defined by the *view* is C-style (*fortran* is
+   Return 1 if the memory defined by the *view* is C-style (*order* is
-   ``'C'``) or Fortran-style (*fortran* is ``'F'``) contiguous or either one
+   ``'C'``) or Fortran-style (*order* is ``'F'``) contiguous or either one
-   (*fortran* is ``'A'``).  Return 0 otherwise.
+   (*order* is ``'A'``).  Return 0 otherwise.
 
 
-.. c:function:: void PyBuffer_FillContiguousStrides(int ndim, Py_ssize_t *shape, Py_ssize_t *strides, Py_ssize_t itemsize, char fortran)
+.. c:function:: void PyBuffer_FillContiguousStrides(int ndim, Py_ssize_t *shape, Py_ssize_t *strides, Py_ssize_t itemsize, char order)
 
    Fill the *strides* array with byte-strides of a contiguous (C-style if
-   *fortran* is ``'C'`` or Fortran-style if *fortran* is ``'F'``) array of the
+   *order* is ``'C'`` or Fortran-style if *order* is ``'F'``) array of the
    given shape with the given number of bytes per element.
 
 
-.. c:function:: int PyBuffer_FillInfo(Py_buffer *view, PyObject *obj, void *buf, Py_ssize_t len, int readonly, int infoflags)
+.. c:function:: int PyBuffer_FillInfo(Py_buffer *view, PyObject *exporter, void *buf, Py_ssize_t len, int readonly, int flags)
+
+   Handle buffer requests for an exporter that wants to expose *buf* of size *len*
+   with writability set according to *readonly*. *buf* is interpreted as a sequence
+   of unsigned bytes.
+
+   The *flags* argument indicates the request type. This function always fills in
+   *view* as specified by flags, unless *buf* has been designated as read-only
+   and :c:macro:`PyBUF_WRITABLE` is set in *flags*.
+
+   On success, set :c:member:`view->obj` to a new reference to *exporter* and
+   return 0. Otherwise, raise :c:data:`PyExc_BufferError`, set
+   :c:member:`view->obj` to *NULL* and return -1;
+
+   If this function is used as part of a :ref:`getbufferproc <buffer-structs>`,
+   *exporter* MUST be set to the exporting object. Otherwise, *exporter* MUST
+   be NULL.
+
 
-   Fill in a buffer-info structure, *view*, correctly for an exporter that can
-   only share a contiguous chunk of memory of "unsigned bytes" of the given
-   length.  Return 0 on success and -1 (with raising an error) on error.
 

Doc/c-api/memoryview.rst

@@ -17,16 +17,19 @@ any other object.
 
    Create a memoryview object from an object that provides the buffer interface.
    If *obj* supports writable buffer exports, the memoryview object will be
-   readable and writable, otherwise it will be read-only.
+   read/write, otherwise it may be either read-only or read/write at the
+   discretion of the exporter.
 
+.. c:function:: PyObject *PyMemoryView_FromMemory(char *mem, Py_ssize_t size, int flags)
+
+   Create a memoryview object using *mem* as the underlying buffer.
+   *flags* can be one of :c:macro:`PyBUF_READ` or :c:macro:`PyBUF_WRITE`.
 
 .. c:function:: PyObject *PyMemoryView_FromBuffer(Py_buffer *view)
 
    Create a memoryview object wrapping the given buffer structure *view*.
-   The memoryview object then owns the buffer represented by *view*, which
+   For simple byte buffers, :c:func:`PyMemoryView_FromMemory` is the preferred
-   means you shouldn't try to call :c:func:`PyBuffer_Release` yourself: it
+   function.
-   will be done on deallocation of the memoryview object.
-
 
 .. c:function:: PyObject *PyMemoryView_GetContiguous(PyObject *obj, int buffertype, char order)
 
@@ -43,10 +46,16 @@ any other object.
    currently allowed to create subclasses of :class:`memoryview`.
 
 
-.. c:function:: Py_buffer *PyMemoryView_GET_BUFFER(PyObject *obj)
+.. c:function:: Py_buffer *PyMemoryView_GET_BUFFER(PyObject *mview)
 
-   Return a pointer to the buffer structure wrapped by the given
+   Return a pointer to the memoryview's private copy of the exporter's buffer.
-   memoryview object.  The object **must** be a memoryview instance;
+   *mview* **must** be a memoryview instance; this macro doesn't check its type,
-   this macro doesn't check its type, you must do it yourself or you
+   you must do it yourself or you will risk crashes.
-   will risk crashes.
+
+.. c:function:: Py_buffer *PyMemoryView_GET_BASE(PyObject *mview)
+
+   Return either a pointer to the exporting object that the memoryview is based
+   on or *NULL* if the memoryview has been created by one of the functions
+   :c:func:`PyMemoryView_FromMemory` or :c:func:`PyMemoryView_FromBuffer`.
+   *mview* **must** be a memoryview instance.
 

Doc/c-api/typeobj.rst

@@ -1198,46 +1198,74 @@ Buffer Object Structures
 
 .. sectionauthor:: Greg J. Stein <greg@lyra.org>
 .. sectionauthor:: Benjamin Peterson
-
+.. sectionauthor:: Stefan Krah
-
-The :ref:`buffer interface <bufferobjects>` exports a model where an object can expose its internal
-data.
-
-If an object does not export the buffer interface, then its :attr:`tp_as_buffer`
-member in the :c:type:`PyTypeObject` structure should be *NULL*.  Otherwise, the
-:attr:`tp_as_buffer` will point to a :c:type:`PyBufferProcs` structure.
-
 
 .. c:type:: PyBufferProcs
 
-   Structure used to hold the function pointers which define an implementation of
+   This structure holds pointers to the functions required by the
-   the buffer protocol.
+   :ref:`Buffer protocol <bufferobjects>`. The protocol defines how
+   an exporter object can expose its internal data to consumer objects.
 
-   .. c:member:: getbufferproc bf_getbuffer
+.. c:member:: getbufferproc PyBufferProcs.bf_getbuffer
 
-      This should fill a :c:type:`Py_buffer` with the necessary data for
+   The signature of this function is::
-      exporting the type.  The signature of :data:`getbufferproc` is ``int
+
-      (PyObject *obj, Py_buffer *view, int flags)``.  *obj* is the object to
+       int (PyObject *exporter, Py_buffer *view, int flags);
-      export, *view* is the :c:type:`Py_buffer` struct to fill, and *flags* gives
+
-      the conditions the caller wants the memory under.  (See
+   Handle a request to *exporter* to fill in *view* as specified by *flags*.
-      :c:func:`PyObject_GetBuffer` for all flags.)  :c:member:`bf_getbuffer` is
+   A standard implementation of this function will take these steps:
-      responsible for filling *view* with the appropriate information.
+
-      (:c:func:`PyBuffer_FillView` can be used in simple cases.)  See
+   - Check if the request can be met. If not, raise :c:data:`PyExc_BufferError`,
-      :c:type:`Py_buffer`\s docs for what needs to be filled in.
+     set :c:data:`view->obj` to *NULL* and return -1.
+
+   - Fill in the requested fields.
+
+   - Increment an internal counter for the number of exports.
+
+   - Set :c:data:`view->obj` to *exporter* and increment :c:data:`view->obj`.
+
+   - Return 0.
+
+   The individual fields of *view* are described in section
+   :ref:`Buffer structure <buffer-structure>`, the rules how an exporter
+   must react to specific requests are in section
+   :ref:`Buffer request types <buffer-request-types>`.
+
+   All memory pointed to in the :c:type:`Py_buffer` structure belongs to
+   the exporter and must remain valid until there are no consumers left.
+   :c:member:`~Py_buffer.shape`, :c:member:`~Py_buffer.strides`,
+   :c:member:`~Py_buffer.suboffsets` and :c:member:`~Py_buffer.internal`
+   are read-only for the consumer.
+
+   :c:func:`PyBuffer_FillInfo` provides an easy way of exposing a simple
+   bytes buffer while dealing correctly with all request types.
+
+   :c:func:`PyObject_GetBuffer` is the interface for the consumer that
+   wraps this function.
+
+.. c:member:: releasebufferproc PyBufferProcs.bf_releasebuffer
+
+   The signature of this function is::
+
+       void (PyObject *exporter, Py_buffer *view);
+
+   Handle a request to release the resources of the buffer. If no resources
+   need to be released, this field may be *NULL*. A standard implementation
+   of this function will take these steps:
+
+   - Decrement an internal counter for the number of exports.
+
+   - If the counter is 0, free all memory associated with *view*.
+
+   The exporter MUST use the :c:member:`~Py_buffer.internal` field to keep
+   track of buffer-specific resources (if present). This field is guaranteed
+   to remain constant, while a consumer MAY pass a copy of the original buffer
+   as the *view* argument.
 
 
-   .. c:member:: releasebufferproc bf_releasebuffer
+   This function MUST NOT decrement :c:data:`view->obj`, since that is
+   done automatically in :c:func:`PyBuffer_Release`.
 
-      This should release the resources of the buffer.  The signature of
-      :c:data:`releasebufferproc` is ``void (PyObject *obj, Py_buffer *view)``.
-      If the :c:data:`bf_releasebuffer` function is not provided (i.e. it is
-      *NULL*), then it does not ever need to be called.
 
-      The exporter of the buffer interface must make sure that any memory
+   :c:func:`PyBuffer_Release` is the interface for the consumer that
-      pointed to in the :c:type:`Py_buffer` structure remains valid until
+   wraps this function.
-      releasebuffer is called.  Exporters will need to define a
-      :c:data:`bf_releasebuffer` function if they can re-allocate their memory,
-      strides, shape, suboffsets, or format variables which they might share
-      through the struct bufferinfo.
-
-      See :c:func:`PyBuffer_Release`.

Doc/library/stdtypes.rst

@@ -2377,7 +2377,7 @@ memoryview type
 
 :class:`memoryview` objects allow Python code to access the internal data
 of an object that supports the :ref:`buffer protocol <bufferobjects>` without
-copying.  Memory is generally interpreted as simple bytes.
+copying.
 
 .. class:: memoryview(obj)
 
@@ -2391,52 +2391,88 @@ copying.  Memory is generally interpreted as simple bytes.
    is a single byte, but other types such as :class:`array.array` may have
    bigger elements.
 
-   ``len(view)`` returns the total number of elements in the memoryview,
+   ``len(view)`` is equal to the length of :class:`~memoryview.tolist`.
-   *view*.  The :class:`~memoryview.itemsize` attribute will give you the
+   If ``view.ndim = 0``, the length is 1. If ``view.ndim = 1``, the length
+   is equal to the number of elements in the view. For higher dimensions,
+   the length is equal to the length of the nested list representation of
+   the view. The :class:`~memoryview.itemsize` attribute will give you the
    number of bytes in a single element.
 
-   A :class:`memoryview` supports slicing to expose its data.  Taking a single
+   A :class:`memoryview` supports slicing to expose its data. If
-   index will return a single element as a :class:`bytes` object.  Full
+   :class:`~memoryview.format` is one of the native format specifiers
-   slicing will result in a subview::
+   from the :mod:`struct` module, indexing will return a single element
+   with the correct type. Full slicing will result in a subview::
 
-      >>> v = memoryview(b'abcefg')
+    >>> v = memoryview(b'abcefg')
-      >>> v[1]
+    >>> v[1]
-      b'b'
+    98
-      >>> v[-1]
+    >>> v[-1]
-      b'g'
+    103
-      >>> v[1:4]
+    >>> v[1:4]
-      <memory at 0x77ab28>
+    <memory at 0x7f3ddc9f4350>
-      >>> bytes(v[1:4])
+    >>> bytes(v[1:4])
-      b'bce'
+    b'bce'
 
-   If the object the memoryview is over supports changing its data, the
+   Other native formats::
-   memoryview supports slice assignment::
+
+      >>> import array
+      >>> a = array.array('l', [-11111111, 22222222, -33333333, 44444444])
+      >>> a[0]
+      -11111111
+      >>> a[-1]
+      44444444
+      >>> a[2:3].tolist()
+      [-33333333]
+      >>> a[::2].tolist()
+      [-11111111, -33333333]
+      >>> a[::-1].tolist()
+      [44444444, -33333333, 22222222, -11111111]
+
+   .. versionadded:: 3.3
+
+   If the underlying object is writable, the memoryview supports slice
+   assignment. Resizing is not allowed::
 
       >>> data = bytearray(b'abcefg')
       >>> v = memoryview(data)
       >>> v.readonly
       False
-      >>> v[0] = b'z'
+      >>> v[0] = ord(b'z')
       >>> data
       bytearray(b'zbcefg')
       >>> v[1:4] = b'123'
       >>> data
       bytearray(b'z123fg')
-      >>> v[2] = b'spam'
+      >>> v[2:3] = b'spam'
       Traceback (most recent call last):
-        File "<stdin>", line 1, in <module>
+      File "<stdin>", line 1, in <module>
-      ValueError: cannot modify size of memoryview object
+      ValueError: memoryview assignment: lvalue and rvalue have different structures
+      >>> v[2:6] = b'spam'
+      >>> data
+      bytearray(b'z1spam')
 
-   Notice how the size of the memoryview object cannot be changed.
+   Memoryviews of hashable (read-only) types are also hashable. The hash
-
+   is defined as ``hash(m) == hash(m.tobytes())``::
-   Memoryviews of hashable (read-only) types are also hashable and their
-   hash value matches the corresponding bytes object::
 
       >>> v = memoryview(b'abcefg')
       >>> hash(v) == hash(b'abcefg')
       True
       >>> hash(v[2:4]) == hash(b'ce')
       True
+      >>> hash(v[::-2]) == hash(b'abcefg'[::-2])
+      True
+
+   Hashing of multi-dimensional objects is supported::
+
+      >>> buf = bytes(list(range(12)))
+      >>> x = memoryview(buf)
+      >>> y = x.cast('B', shape=[2,2,3])
+      >>> x.tolist()
+      [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11]
+      >>> y.tolist()
+      [[[0, 1, 2], [3, 4, 5]], [[6, 7, 8], [9, 10, 11]]]
+      >>> hash(x) == hash(y) == hash(y.tobytes())
+      True
 
    .. versionchanged:: 3.3
       Memoryview objects are now hashable.
@@ -2455,12 +2491,20 @@ copying.  Memory is generally interpreted as simple bytes.
          >>> bytes(m)
          b'abc'
 
+      For non-contiguous arrays the result is equal to the flattened list
+      representation with all elements converted to bytes.
+
    .. method:: tolist()
 
-      Return the data in the buffer as a list of integers. ::
+      Return the data in the buffer as a list of elements. ::
 
          >>> memoryview(b'abc').tolist()
          [97, 98, 99]
+         >>> import array
+         >>> a = array.array('d', [1.1, 2.2, 3.3])
+         >>> m = memoryview(a)
+         >>> m.tolist()
+         [1.1, 2.2, 3.3]
 
    .. method:: release()
 
@@ -2487,7 +2531,7 @@ copying.  Memory is generally interpreted as simple bytes.
          >>> with memoryview(b'abc') as m:
          ...     m[0]
          ...
-         b'a'
+         97
          >>> m[0]
          Traceback (most recent call last):
            File "<stdin>", line 1, in <module>
@@ -2495,45 +2539,219 @@ copying.  Memory is generally interpreted as simple bytes.
 
       .. versionadded:: 3.2
 
+   .. method:: cast(format[, shape])
+
+      Cast a memoryview to a new format or shape. *shape* defaults to
+      ``[byte_length//new_itemsize]``, which means that the result view
+      will be one-dimensional. The return value is a new memoryview, but
+      the buffer itself is not copied. Supported casts are 1D -> C-contiguous
+      and C-contiguous -> 1D. One of the formats must be a byte format
+      ('B', 'b' or 'c'). The byte length of the result must be the same
+      as the original length.
+
+      Cast 1D/long to 1D/unsigned bytes::
+
+         >>> import array
+         >>> a = array.array('l', [1,2,3])
+         >>> x = memoryview(a)
+         >>> x.format
+         'l'
+         >>> x.itemsize
+         8
+         >>> len(x)
+         3
+         >>> x.nbytes
+         24
+         >>> y = x.cast('B')
+         >>> y.format
+         'B'
+         >>> y.itemsize
+         1
+         >>> len(y)
+         24
+         >>> y.nbytes
+         24
+
+      Cast 1D/unsigned bytes to 1D/char::
+
+         >>> b = bytearray(b'zyz')
+         >>> x = memoryview(b)
+         >>> x[0] = b'a'
+         Traceback (most recent call last):
+           File "<stdin>", line 1, in <module>
+         ValueError: memoryview: invalid value for format "B"
+         >>> y = x.cast('c')
+         >>> y[0] = b'a'
+         >>> b
+         bytearray(b'ayz')
+
+      Cast 1D/bytes to 3D/ints to 1D/signed char::
+
+         >>> import struct
+         >>> buf = struct.pack("i"*12, *list(range(12)))
+         >>> x = memoryview(buf)
+         >>> y = x.cast('i', shape=[2,2,3])
+         >>> y.tolist()
+         [[[0, 1, 2], [3, 4, 5]], [[6, 7, 8], [9, 10, 11]]]
+         >>> y.format
+         'i'
+         >>> y.itemsize
+         4
+         >>> len(y)
+         2
+         >>> y.nbytes
+         48
+         >>> z = y.cast('b')
+         >>> z.format
+         'b'
+         >>> z.itemsize
+         1
+         >>> len(z)
+         48
+         >>> z.nbytes
+         48
+
+      Cast 1D/unsigned char to to 2D/unsigned long::
+
+         >>> buf = struct.pack("L"*6, *list(range(6)))
+         >>> x = memoryview(buf)
+         >>> y = x.cast('L', shape=[2,3])
+         >>> len(y)
+         2
+         >>> y.nbytes
+         48
+         >>> y.tolist()
+         [[0, 1, 2], [3, 4, 5]]
+
+      .. versionadded:: 3.3
+
    There are also several readonly attributes available:
 
+   .. attribute:: obj
+
+      The underlying object of the memoryview::
+
+         >>> b  = bytearray(b'xyz')
+         >>> m = memoryview(b)
+         >>> m.obj is b
+         True
+
+      .. versionadded:: 3.3
+
+   .. attribute:: nbytes
+
+      ``nbytes == product(shape) * itemsize == len(m.tobytes())``. This is
+      the amount of space in bytes that the array would use in a contiguous
+      representation. It is not necessarily equal to len(m)::
+
+         >>> import array
+         >>> a = array.array('i', [1,2,3,4,5])
+         >>> m = memoryview(a)
+         >>> len(m)
+         5
+         >>> m.nbytes
+         20
+         >>> y = m[::2]
+         >>> len(y)
+         3
+         >>> y.nbytes
+         12
+         >>> len(y.tobytes())
+         12
+
+      Multi-dimensional arrays::
+
+         >>> import struct
+         >>> buf = struct.pack("d"*12, *[1.5*x for x in range(12)])
+         >>> x = memoryview(buf)
+         >>> y = x.cast('d', shape=[3,4])
+         >>> y.tolist()
+         [[0.0, 1.5, 3.0, 4.5], [6.0, 7.5, 9.0, 10.5], [12.0, 13.5, 15.0, 16.5]]
+         >>> len(y)
+         3
+         >>> y.nbytes
+         96
+
+      .. versionadded:: 3.3
+
+   .. attribute:: readonly
+
+      A bool indicating whether the memory is read only.
+
    .. attribute:: format
 
       A string containing the format (in :mod:`struct` module style) for each
-      element in the view.  This defaults to ``'B'``, a simple bytestring.
+      element in the view. A memoryview can be created from exporters with
+      arbitrary format strings, but some methods (e.g. :meth:`tolist`) are
+      restricted to native single element formats. Special care must be taken
+      when comparing memoryviews. Since comparisons are required to return a
+      value for ``==`` and ``!=``, two memoryviews referencing the same
+      exporter can compare as not-equal if the exporter's format is not
+      understood::
+
+         >>> from ctypes import BigEndianStructure, c_long
+         >>> class BEPoint(BigEndianStructure):
+         ...     _fields_ = [("x", c_long), ("y", c_long)]
+         ...
+         >>> point = BEPoint(100, 200)
+         >>> a = memoryview(point)
+         >>> b = memoryview(point)
+         >>> a == b
+         False
+         >>> a.tolist()
+         Traceback (most recent call last):
+           File "<stdin>", line 1, in <module>
+         NotImplementedError: memoryview: unsupported format T{>l:x:>l:y:}
 
    .. attribute:: itemsize
 
       The size in bytes of each element of the memoryview::
 
-         >>> m = memoryview(array.array('H', [1,2,3]))
+         >>> import array, struct
+         >>> m = memoryview(array.array('H', [32000, 32001, 32002]))
          >>> m.itemsize
          2
          >>> m[0]
-         b'\x01\x00'
+         32000
-         >>> len(m[0]) == m.itemsize
+         >>> struct.calcsize('H') == m.itemsize
          True
 
-   .. attribute:: shape
-
-      A tuple of integers the length of :attr:`ndim` giving the shape of the
-      memory as a N-dimensional array.
-
    .. attribute:: ndim
 
       An integer indicating how many dimensions of a multi-dimensional array the
       memory represents.
 
+   .. attribute:: shape
+
+      A tuple of integers the length of :attr:`ndim` giving the shape of the
+      memory as a N-dimensional array.
+
    .. attribute:: strides
 
       A tuple of integers the length of :attr:`ndim` giving the size in bytes to
       access each element for each dimension of the array.
 
-   .. attribute:: readonly
+   .. attribute:: suboffsets
 
-      A bool indicating whether the memory is read only.
+      Used internally for PIL-style arrays. The value is informational only.
 
-   .. memoryview.suboffsets isn't documented because it only seems useful for C
+   .. attribute:: c_contiguous
+
+      A bool indicating whether the memory is C-contiguous.
+
+      .. versionadded:: 3.3
+
+   .. attribute:: f_contiguous
+
+      A bool indicating whether the memory is Fortran contiguous.
+
+      .. versionadded:: 3.3
+
+   .. attribute:: contiguous
+
+      A bool indicating whether the memory is contiguous.
+
+      .. versionadded:: 3.3
 
 
 .. _typecontextmanager:

Doc/whatsnew/3.3.rst

@@ -49,6 +49,62 @@
 This article explains the new features in Python 3.3, compared to 3.2.
 
 
+.. _pep-3118:
+
+PEP 3118: New memoryview implementation and buffer protocol documentation
+=========================================================================
+
+:issue:`10181` - memoryview bug fixes and features.
+  Written by Stefan Krah.
+
+The new memoryview implementation comprehensively fixes all ownership and
+lifetime issues of dynamically allocated fields in the Py_buffer struct
+that led to multiple crash reports. Additionally, several functions that
+crashed or returned incorrect results for non-contiguous or multi-dimensional
+input have been fixed.
+
+The memoryview object now has a PEP-3118 compliant getbufferproc()
+that checks the consumer's request type. Many new features have been
+added, most of them work in full generality for non-contiguous arrays
+and arrays with suboffsets.
+
+The documentation has been updated, clearly spelling out responsibilities
+for both exporters and consumers. Buffer request flags are grouped into
+basic and compound flags. The memory layout of non-contiguous and
+multi-dimensional NumPy-style arrays is explained.
+
+Features
+--------
+
+* All native single character format specifiers in struct module syntax
+  (optionally prefixed with '@') are now supported.
+
+* With some restrictions, the cast() method allows changing of format and
+  shape of C-contiguous arrays.
+
+* Multi-dimensional list representations are supported for any array type.
+
+* Multi-dimensional comparisons are supported for any array type.
+
+* All array types are hashable if the exporting object is hashable
+  and the view is read-only.
+
+* Arbitrary slicing of any 1-D arrays type is supported. For example, it
+  is now possible to reverse a memoryview in O(1) by using a negative step.
+
+API changes
+-----------
+
+* The maximum number of dimensions is officially limited to 64.
+
+* The representation of empty shape, strides and suboffsets is now
+  an empty tuple instead of None.
+
+* Accessing a memoryview element with format 'B' (unsigned bytes)
+  now returns an integer (in accordance with the struct module syntax).
+  For returning a bytes object the view must be cast to 'c' first.
+
+
 .. _pep-393:
 
 PEP 393: Flexible String Representation

Include/abstract.h

@@ -559,7 +559,7 @@ xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx*/
     /* Copy the data from the src buffer to the buffer of destination
      */
 
-     PyAPI_FUNC(int) PyBuffer_IsContiguous(Py_buffer *view, char fort);
+     PyAPI_FUNC(int) PyBuffer_IsContiguous(const Py_buffer *view, char fort);
 
 
      PyAPI_FUNC(void) PyBuffer_FillContiguousStrides(int ndims,

Include/memoryobject.h

@@ -6,70 +6,64 @@
 extern "C" {
 #endif
 
+#ifndef Py_LIMITED_API
+PyAPI_DATA(PyTypeObject) _PyManagedBuffer_Type;
+#endif
 PyAPI_DATA(PyTypeObject) PyMemoryView_Type;
 
 #define PyMemoryView_Check(op) (Py_TYPE(op) == &PyMemoryView_Type)
 
 #ifndef Py_LIMITED_API
-/* Get a pointer to the underlying Py_buffer of a memoryview object. */
+/* Get a pointer to the memoryview's private copy of the exporter's buffer. */
 #define PyMemoryView_GET_BUFFER(op) (&((PyMemoryViewObject *)(op))->view)
-/* Get a pointer to the PyObject from which originates a memoryview object. */
+/* Get a pointer to the exporting object (this may be NULL!). */
 #define PyMemoryView_GET_BASE(op) (((PyMemoryViewObject *)(op))->view.obj)
 #endif
 
-
-PyAPI_FUNC(PyObject *) PyMemoryView_GetContiguous(PyObject *base, 
-						  int buffertype, 
-						  char fort);
-
-    /* Return a contiguous chunk of memory representing the buffer
-       from an object in a memory view object.  If a copy is made then the
-       base object for the memory view will be a *new* bytes object. 
-       
-       Otherwise, the base-object will be the object itself and no 
-       data-copying will be done. 
-
-       The buffertype argument can be PyBUF_READ, PyBUF_WRITE,
-       PyBUF_SHADOW to determine whether the returned buffer
-       should be READONLY, WRITABLE, or set to update the
-       original buffer if a copy must be made.  If buffertype is
-       PyBUF_WRITE and the buffer is not contiguous an error will
-       be raised.  In this circumstance, the user can use
-       PyBUF_SHADOW to ensure that a a writable temporary
-       contiguous buffer is returned.  The contents of this
-       contiguous buffer will be copied back into the original
-       object after the memoryview object is deleted as long as
-       the original object is writable and allows setting an
-       exclusive write lock. If this is not allowed by the
-       original object, then a BufferError is raised.
-       
-       If the object is multi-dimensional and if fortran is 'F',
-       the first dimension of the underlying array will vary the
-       fastest in the buffer.  If fortran is 'C', then the last
-       dimension will vary the fastest (C-style contiguous).  If
-       fortran is 'A', then it does not matter and you will get
-       whatever the object decides is more efficient.  
-
-       A new reference is returned that must be DECREF'd when finished.
-    */
-
 PyAPI_FUNC(PyObject *) PyMemoryView_FromObject(PyObject *base);
-
+PyAPI_FUNC(PyObject *) PyMemoryView_FromMemory(char *mem, Py_ssize_t size,
+                                               int flags);
 #ifndef Py_LIMITED_API
 PyAPI_FUNC(PyObject *) PyMemoryView_FromBuffer(Py_buffer *info);
-    /* create new if bufptr is NULL 
-        will be a new bytesobject in base */
 #endif
+PyAPI_FUNC(PyObject *) PyMemoryView_GetContiguous(PyObject *base,
+                                                  int buffertype,
+                                                  char order);
 
 
-/* The struct is declared here so that macros can work, but it shouldn't
+/* The structs are declared here so that macros can work, but they shouldn't
-   be considered public. Don't access those fields directly, use the macros
+   be considered public. Don't access their fields directly, use the macros
    and functions instead! */
 #ifndef Py_LIMITED_API
+#define _Py_MANAGED_BUFFER_RELEASED    0x001  /* access to exporter blocked */
+#define _Py_MANAGED_BUFFER_FREE_FORMAT 0x002  /* free format */
 typedef struct {
     PyObject_HEAD
-    Py_buffer view;
+    int flags;          /* state flags */
-    Py_hash_t hash;
+    Py_ssize_t exports; /* number of direct memoryview exports */
+    Py_buffer master; /* snapshot buffer obtained from the original exporter */
+} _PyManagedBufferObject;
+
+
+/* static storage used for casting between formats */
+#define _Py_MEMORYVIEW_MAX_FORMAT 3 /* must be >= 3 */
+
+/* memoryview state flags */
+#define _Py_MEMORYVIEW_RELEASED    0x001  /* access to master buffer blocked */
+#define _Py_MEMORYVIEW_C           0x002  /* C-contiguous layout */
+#define _Py_MEMORYVIEW_FORTRAN     0x004  /* Fortran contiguous layout */
+#define _Py_MEMORYVIEW_SCALAR      0x008  /* scalar: ndim = 0 */
+#define _Py_MEMORYVIEW_PIL         0x010  /* PIL-style layout */
+
+typedef struct {
+    PyObject_VAR_HEAD
+    _PyManagedBufferObject *mbuf; /* managed buffer */
+    Py_hash_t hash;               /* hash value for read-only views */
+    int flags;                    /* state flags */
+    Py_ssize_t exports;           /* number of buffer re-exports */
+    Py_buffer view;               /* private copy of the exporter's view */
+    char format[_Py_MEMORYVIEW_MAX_FORMAT]; /* used for casting */
+    Py_ssize_t ob_array[1];       /* shape, strides, suboffsets */
 } PyMemoryViewObject;
 #endif
 

Include/object.h

@@ -186,15 +186,16 @@ typedef struct bufferinfo {
     Py_ssize_t *shape;
     Py_ssize_t *strides;
     Py_ssize_t *suboffsets;
-    Py_ssize_t smalltable[2];  /* static store for shape and strides of
-                                  mono-dimensional buffers. */
     void *internal;
 } Py_buffer;
 
 typedef int (*getbufferproc)(PyObject *, Py_buffer *, int);
 typedef void (*releasebufferproc)(PyObject *, Py_buffer *);
 
-    /* Flags for getting buffers */
+/* Maximum number of dimensions */
+#define PyBUF_MAX_NDIM 64
+
+/* Flags for getting buffers */
 #define PyBUF_SIMPLE 0
 #define PyBUF_WRITABLE 0x0001
 /*  we used to include an E, backwards compatible alias  */

Lib/ctypes/test/test_pep3118.py

@@ -25,14 +25,17 @@ class Test(unittest.TestCase):
             v = memoryview(ob)
             try:
                 self.assertEqual(normalize(v.format), normalize(fmt))
-                if shape is not None:
+                if shape:
                     self.assertEqual(len(v), shape[0])
                 else:
                     self.assertEqual(len(v) * sizeof(itemtp), sizeof(ob))
                 self.assertEqual(v.itemsize, sizeof(itemtp))
                 self.assertEqual(v.shape, shape)
-                # ctypes object always have a non-strided memory block
+                # XXX Issue #12851: PyCData_NewGetBuffer() must provide strides
-                self.assertEqual(v.strides, None)
+                #     if requested. memoryview currently reconstructs missing
+                #     stride information, so this assert will fail.
+                # self.assertEqual(v.strides, ())
+
                 # they are always read/write
                 self.assertFalse(v.readonly)
 
@@ -52,14 +55,15 @@ class Test(unittest.TestCase):
             v = memoryview(ob)
             try:
                 self.assertEqual(v.format, fmt)
-                if shape is not None:
+                if shape:
                     self.assertEqual(len(v), shape[0])
                 else:
                     self.assertEqual(len(v) * sizeof(itemtp), sizeof(ob))
                 self.assertEqual(v.itemsize, sizeof(itemtp))
                 self.assertEqual(v.shape, shape)
-                # ctypes object always have a non-strided memory block
+                # XXX Issue #12851
-                self.assertEqual(v.strides, None)
+                # self.assertEqual(v.strides, ())
+
                 # they are always read/write
                 self.assertFalse(v.readonly)
 
@@ -110,34 +114,34 @@ native_types = [
 
     ## simple types
 
-    (c_char,                    "<c",                   None,           c_char),
+    (c_char,                    "<c",                   (),           c_char),
-    (c_byte,                    "<b",                   None,           c_byte),
+    (c_byte,                    "<b",                   (),           c_byte),
-    (c_ubyte,                   "<B",                   None,           c_ubyte),
+    (c_ubyte,                   "<B",                   (),           c_ubyte),
-    (c_short,                   "<h",                   None,           c_short),
+    (c_short,                   "<h",                   (),           c_short),
-    (c_ushort,                  "<H",                   None,           c_ushort),
+    (c_ushort,                  "<H",                   (),           c_ushort),
 
     # c_int and c_uint may be aliases to c_long
-    #(c_int,                     "<i",                   None,           c_int),
+    #(c_int,                     "<i",                   (),           c_int),
-    #(c_uint,                    "<I",                   None,           c_uint),
+    #(c_uint,                    "<I",                   (),           c_uint),
 
-    (c_long,                    "<l",                   None,           c_long),
+    (c_long,                    "<l",                   (),           c_long),
-    (c_ulong,                   "<L",                   None,           c_ulong),
+    (c_ulong,                   "<L",                   (),           c_ulong),
 
     # c_longlong and c_ulonglong are aliases on 64-bit platforms
     #(c_longlong,                "<q",                   None,           c_longlong),
     #(c_ulonglong,               "<Q",                   None,           c_ulonglong),
 
-    (c_float,                   "<f",                   None,           c_float),
+    (c_float,                   "<f",                   (),           c_float),
-    (c_double,                  "<d",                   None,           c_double),
+    (c_double,                  "<d",                   (),           c_double),
     # c_longdouble may be an alias to c_double
 
-    (c_bool,                    "<?",                   None,           c_bool),
+    (c_bool,                    "<?",                   (),           c_bool),
-    (py_object,                 "<O",                   None,           py_object),
+    (py_object,                 "<O",                   (),           py_object),
 
     ## pointers
 
-    (POINTER(c_byte),           "&<b",                  None,           POINTER(c_byte)),
+    (POINTER(c_byte),           "&<b",                  (),           POINTER(c_byte)),
-    (POINTER(POINTER(c_long)),  "&&<l",                 None,           POINTER(POINTER(c_long))),
+    (POINTER(POINTER(c_long)),  "&&<l",                 (),           POINTER(POINTER(c_long))),
 
     ## arrays and pointers
 
@@ -145,32 +149,32 @@ native_types = [
     (c_float * 4 * 3 * 2,       "(2,3,4)<f",            (2,3,4),        c_float),
     (POINTER(c_short) * 2,      "(2)&<h",               (2,),           POINTER(c_short)),
     (POINTER(c_short) * 2 * 3,  "(3,2)&<h",             (3,2,),         POINTER(c_short)),
-    (POINTER(c_short * 2),      "&(2)<h",               None,           POINTER(c_short)),
+    (POINTER(c_short * 2),      "&(2)<h",               (),           POINTER(c_short)),
 
     ## structures and unions
 
-    (Point,                     "T{<l:x:<l:y:}",        None,           Point),
+    (Point,                     "T{<l:x:<l:y:}",        (),           Point),
     # packed structures do not implement the pep
-    (PackedPoint,               "B",                    None,           PackedPoint),
+    (PackedPoint,               "B",                    (),           PackedPoint),
-    (Point2,                    "T{<l:x:<l:y:}",        None,           Point2),
+    (Point2,                    "T{<l:x:<l:y:}",        (),           Point2),
-    (EmptyStruct,               "T{}",                  None,           EmptyStruct),
+    (EmptyStruct,               "T{}",                  (),           EmptyStruct),
     # the pep does't support unions
-    (aUnion,                    "B",                    None,           aUnion),
+    (aUnion,                    "B",                    (),           aUnion),
 
     ## pointer to incomplete structure
-    (Incomplete,                "B",                    None,           Incomplete),
+    (Incomplete,                "B",                    (),           Incomplete),
-    (POINTER(Incomplete),       "&B",                   None,           POINTER(Incomplete)),
+    (POINTER(Incomplete),       "&B",                   (),           POINTER(Incomplete)),
 
     # 'Complete' is a structure that starts incomplete, but is completed after the
     # pointer type to it has been created.
-    (Complete,                  "T{<l:a:}",             None,           Complete),
+    (Complete,                  "T{<l:a:}",             (),           Complete),
     # Unfortunately the pointer format string is not fixed...
-    (POINTER(Complete),         "&B",                   None,           POINTER(Complete)),
+    (POINTER(Complete),         "&B",                   (),           POINTER(Complete)),
 
     ## other
 
     # function signatures are not implemented
-    (CFUNCTYPE(None),           "X{}",                  None,           CFUNCTYPE(None)),
+    (CFUNCTYPE(None),           "X{}",                  (),           CFUNCTYPE(None)),
 
     ]
 
@@ -186,10 +190,10 @@ class LEPoint(LittleEndianStructure):
 # and little endian machines.
 #
 endian_types = [
-    (BEPoint,                   "T{>l:x:>l:y:}",        None,           BEPoint),
+    (BEPoint,                   "T{>l:x:>l:y:}",        (),           BEPoint),
-    (LEPoint,                   "T{<l:x:<l:y:}",        None,           LEPoint),
+    (LEPoint,                   "T{<l:x:<l:y:}",        (),           LEPoint),
-    (POINTER(BEPoint),          "&T{>l:x:>l:y:}",       None,           POINTER(BEPoint)),
+    (POINTER(BEPoint),          "&T{>l:x:>l:y:}",       (),           POINTER(BEPoint)),
-    (POINTER(LEPoint),          "&T{<l:x:<l:y:}",       None,           POINTER(LEPoint)),
+    (POINTER(LEPoint),          "&T{<l:x:<l:y:}",       (),           POINTER(LEPoint)),
     ]
 
 if __name__ == "__main__":

Lib/test/test_memoryview.py

@@ -24,15 +24,14 @@ class AbstractMemoryTests:
         return filter(None, [self.ro_type, self.rw_type])
 
     def check_getitem_with_type(self, tp):
-        item = self.getitem_type
         b = tp(self._source)
         oldrefcount = sys.getrefcount(b)
         m = self._view(b)
-        self.assertEqual(m[0], item(b"a"))
+        self.assertEqual(m[0], ord(b"a"))
-        self.assertIsInstance(m[0], bytes)
+        self.assertIsInstance(m[0], int)
-        self.assertEqual(m[5], item(b"f"))
+        self.assertEqual(m[5], ord(b"f"))
-        self.assertEqual(m[-1], item(b"f"))
+        self.assertEqual(m[-1], ord(b"f"))
-        self.assertEqual(m[-6], item(b"a"))
+        self.assertEqual(m[-6], ord(b"a"))
         # Bounds checking
         self.assertRaises(IndexError, lambda: m[6])
         self.assertRaises(IndexError, lambda: m[-7])
@@ -76,7 +75,9 @@ class AbstractMemoryTests:
         b = self.rw_type(self._source)
         oldrefcount = sys.getrefcount(b)
         m = self._view(b)
-        m[0] = tp(b"0")
+        m[0] = ord(b'1')
+        self._check_contents(tp, b, b"1bcdef")
+        m[0:1] = tp(b"0")
         self._check_contents(tp, b, b"0bcdef")
         m[1:3] = tp(b"12")
         self._check_contents(tp, b, b"012def")
@@ -102,10 +103,17 @@ class AbstractMemoryTests:
         # Wrong index/slice types
         self.assertRaises(TypeError, setitem, 0.0, b"a")
         self.assertRaises(TypeError, setitem, (0,), b"a")
+        self.assertRaises(TypeError, setitem, (slice(0,1,1), 0), b"a")
+        self.assertRaises(TypeError, setitem, (0, slice(0,1,1)), b"a")
+        self.assertRaises(TypeError, setitem, (0,), b"a")
         self.assertRaises(TypeError, setitem, "a", b"a")
+        # Not implemented: multidimensional slices
+        slices = (slice(0,1,1), slice(0,1,2))
+        self.assertRaises(NotImplementedError, setitem, slices, b"a")
         # Trying to resize the memory object
-        self.assertRaises(ValueError, setitem, 0, b"")
+        exc = ValueError if m.format == 'c' else TypeError
-        self.assertRaises(ValueError, setitem, 0, b"ab")
+        self.assertRaises(exc, setitem, 0, b"")
+        self.assertRaises(exc, setitem, 0, b"ab")
         self.assertRaises(ValueError, setitem, slice(1,1), b"a")
         self.assertRaises(ValueError, setitem, slice(0,2), b"a")
 
@@ -175,7 +183,7 @@ class AbstractMemoryTests:
         self.assertEqual(m.shape, (6,))
         self.assertEqual(len(m), 6)
         self.assertEqual(m.strides, (self.itemsize,))
-        self.assertEqual(m.suboffsets, None)
+        self.assertEqual(m.suboffsets, ())
         return m
 
     def test_attributes_readonly(self):
@@ -209,12 +217,16 @@ class AbstractMemoryTests:
                 # If tp is a factory rather than a plain type, skip
                 continue
 
+            class MyView():
+                def __init__(self, base):
+                    self.m = memoryview(base)
             class MySource(tp):
                 pass
             class MyObject:
                 pass
 
-            # Create a reference cycle through a memoryview object
+            # Create a reference cycle through a memoryview object.
+            # This exercises mbuf_clear().
             b = MySource(tp(b'abc'))
             m = self._view(b)
             o = MyObject()
@@ -226,6 +238,17 @@ class AbstractMemoryTests:
             gc.collect()
             self.assertTrue(wr() is None, wr())
 
+            # This exercises memory_clear().
+            m = MyView(tp(b'abc'))
+            o = MyObject()
+            m.x = m
+            m.o = o
+            wr = weakref.ref(o)
+            m = o = None
+            # The cycle must be broken
+            gc.collect()
+            self.assertTrue(wr() is None, wr())
+
     def _check_released(self, m, tp):
         check = self.assertRaisesRegex(ValueError, "released")
         with check: bytes(m)
@@ -283,9 +306,12 @@ class AbstractMemoryTests:
         i = io.BytesIO(b'ZZZZ')
         self.assertRaises(TypeError, i.readinto, m)
 
+    def test_getbuf_fail(self):
+        self.assertRaises(TypeError, self._view, {})
+
     def test_hash(self):
         # Memoryviews of readonly (hashable) types are hashable, and they
-        # hash as the corresponding object.
+        # hash as hash(obj.tobytes()).
         tp = self.ro_type
         if tp is None:
             self.skipTest("no read-only type to test")

Lib/test/test_sys.py

@@ -773,8 +773,8 @@ class SizeofTest(unittest.TestCase):
         check(int(PyLong_BASE), size(vh) + 2*self.longdigit)
         check(int(PyLong_BASE**2-1), size(vh) + 2*self.longdigit)
         check(int(PyLong_BASE**2), size(vh) + 3*self.longdigit)
-        # memory (Py_buffer + hash value)
+        # memoryview
-        check(memoryview(b''), size(h + 'PP2P2i7P' + 'P'))
+        check(memoryview(b''), size(h + 'PPiP4P2i5P3cP'))
         # module
         check(unittest, size(h + '3P'))
         # None

Misc/ACKS

@@ -1041,6 +1041,7 @@ John Viega
 Kannan Vijayan
 Kurt Vile
 Norman Vine
+Pauli Virtanen
 Frank Visser
 Johannes Vogel
 Sjoerd de Vries

Misc/NEWS

@@ -10,6 +10,23 @@ What's New in Python 3.3 Alpha 1?
 Core and Builtins
 -----------------
 
+- Issue #10181: New memoryview implementation fixes multiple ownership
+  and lifetime issues of dynamically allocated Py_buffer members (#9990)
+  as well as crashes (#8305, #7433). Many new features have been added
+  (See whatsnew/3.3), and the documentation has been updated extensively.
+  The ndarray test object from _testbuffer.c implements all aspects of
+  PEP-3118, so further development towards the complete implementation
+  of the PEP can proceed in a test-driven manner.
+
+  Thanks to Nick Coghlan, Antoine Pitrou and Pauli Virtanen for review
+  and many ideas.
+
+- Issue #12834: Fix incorrect results of memoryview.tobytes() for
+  non-contiguous arrays.
+
+- Issue #5231: Introduce memoryview.cast() method that allows changing
+  format and shape without making a copy of the underlying memory.
+
 - Issue #14084: Fix a file descriptor leak when importing a module with a
   bad encoding.
 

Misc/valgrind-python.supp

@@ -412,4 +412,15 @@
    fun:SHA1_Update
 }
 
+{
+   test_buffer_non_debug
+   Memcheck:Addr4
+   fun:PyUnicodeUCS2_FSConverter
+}
+
+{
+   test_buffer_non_debug
+   Memcheck:Addr4
+   fun:PyUnicode_FSConverter
+}
 

Modules/_testcapimodule.c

@@ -275,95 +275,6 @@ test_lazy_hash_inheritance(PyObject* self)
 }
 
 
-/* Issue #7385: Check that memoryview() does not crash
- *   when bf_getbuffer returns an error
- */
-
-static int
-broken_buffer_getbuffer(PyObject *self, Py_buffer *view, int flags)
-{
-    PyErr_SetString(
-        TestError,
-        "test_broken_memoryview: expected error in bf_getbuffer");
-    return -1;
-}
-
-static PyBufferProcs memoryviewtester_as_buffer = {
-    (getbufferproc)broken_buffer_getbuffer,     /* bf_getbuffer */
-    0,          /* bf_releasebuffer */
-};
-
-static PyTypeObject _MemoryViewTester_Type = {
-    PyVarObject_HEAD_INIT(NULL, 0)
-    "memoryviewtester",         /* Name of this type */
-    sizeof(PyObject),           /* Basic object size */
-    0,                          /* Item size for varobject */
-    (destructor)PyObject_Del, /* tp_dealloc */
-    0,                          /* tp_print */
-    0,                          /* tp_getattr */
-    0,                          /* tp_setattr */
-    0,                          /* tp_compare */
-    0,                          /* tp_repr */
-    0,                          /* tp_as_number */
-    0,                          /* tp_as_sequence */
-    0,                          /* tp_as_mapping */
-    0,                          /* tp_hash */
-    0,                          /* tp_call */
-    0,                          /* tp_str */
-    PyObject_GenericGetAttr,  /* tp_getattro */
-    0,                          /* tp_setattro */
-    &memoryviewtester_as_buffer,                        /* tp_as_buffer */
-    Py_TPFLAGS_DEFAULT,         /* tp_flags */
-    0,                          /* tp_doc */
-    0,                          /* tp_traverse */
-    0,                          /* tp_clear */
-    0,                          /* tp_richcompare */
-    0,                          /* tp_weaklistoffset */
-    0,                          /* tp_iter */
-    0,                          /* tp_iternext */
-    0,                          /* tp_methods */
-    0,                          /* tp_members */
-    0,                          /* tp_getset */
-    0,                          /* tp_base */
-    0,                          /* tp_dict */
-    0,                          /* tp_descr_get */
-    0,                          /* tp_descr_set */
-    0,                          /* tp_dictoffset */
-    0,                          /* tp_init */
-    0,                          /* tp_alloc */
-    PyType_GenericNew,                  /* tp_new */
-};
-
-static PyObject*
-test_broken_memoryview(PyObject* self)
-{
-    PyObject *obj = PyObject_New(PyObject, &_MemoryViewTester_Type);
-    PyObject *res;
-
-    if (obj == NULL) {
-        PyErr_Clear();
-        PyErr_SetString(
-            TestError,
-            "test_broken_memoryview: failed to create object");
-        return NULL;
-    }
-
-    res = PyMemoryView_FromObject(obj);
-    if (res || !PyErr_Occurred()){
-        PyErr_SetString(
-            TestError,
-            "test_broken_memoryview: memoryview() didn't raise an Exception");
-        Py_XDECREF(res);
-        Py_DECREF(obj);
-        return NULL;
-    }
-
-    PyErr_Clear();
-    Py_DECREF(obj);
-    Py_RETURN_NONE;
-}
-
-
 /* Tests of PyLong_{As, From}{Unsigned,}Long(), and (#ifdef HAVE_LONG_LONG)
    PyLong_{As, From}{Unsigned,}LongLong().
 
@@ -2421,7 +2332,6 @@ static PyMethodDef TestMethods[] = {
     {"test_list_api",           (PyCFunction)test_list_api,      METH_NOARGS},
     {"test_dict_iteration",     (PyCFunction)test_dict_iteration,METH_NOARGS},
     {"test_lazy_hash_inheritance",      (PyCFunction)test_lazy_hash_inheritance,METH_NOARGS},
-    {"test_broken_memoryview",          (PyCFunction)test_broken_memoryview,METH_NOARGS},
     {"test_long_api",           (PyCFunction)test_long_api,      METH_NOARGS},
     {"test_long_and_overflow", (PyCFunction)test_long_and_overflow,
      METH_NOARGS},
@@ -2684,7 +2594,6 @@ PyInit__testcapi(void)
         return NULL;
 
     Py_TYPE(&_HashInheritanceTester_Type)=&PyType_Type;
-    Py_TYPE(&_MemoryViewTester_Type)=&PyType_Type;
 
     Py_TYPE(&test_structmembersType)=&PyType_Type;
     Py_INCREF(&test_structmembersType);

Objects/abstract.c

@@ -340,7 +340,7 @@ PyObject_GetBuffer(PyObject *obj, Py_buffer *view, int flags)
 }
 
 static int
-_IsFortranContiguous(Py_buffer *view)
+_IsFortranContiguous(const Py_buffer *view)
 {
     Py_ssize_t sd, dim;
     int i;
@@ -361,7 +361,7 @@ _IsFortranContiguous(Py_buffer *view)
 }
 
 static int
-_IsCContiguous(Py_buffer *view)
+_IsCContiguous(const Py_buffer *view)
 {
     Py_ssize_t sd, dim;
     int i;
@@ -382,16 +382,16 @@ _IsCContiguous(Py_buffer *view)
 }
 
 int
-PyBuffer_IsContiguous(Py_buffer *view, char fort)
+PyBuffer_IsContiguous(const Py_buffer *view, char order)
 {
 
     if (view->suboffsets != NULL) return 0;
 
-    if (fort == 'C')
+    if (order == 'C')
         return _IsCContiguous(view);
-    else if (fort == 'F')
+    else if (order == 'F')
         return _IsFortranContiguous(view);
-    else if (fort == 'A')
+    else if (order == 'A')
         return (_IsCContiguous(view) || _IsFortranContiguous(view));
     return 0;
 }
@@ -651,7 +651,7 @@ int
 PyBuffer_FillInfo(Py_buffer *view, PyObject *obj, void *buf, Py_ssize_t len,
               int readonly, int flags)
 {
-    if (view == NULL) return 0;
+    if (view == NULL) return 0; /* XXX why not -1? */
     if (((flags & PyBUF_WRITABLE) == PyBUF_WRITABLE) &&
         (readonly == 1)) {
         PyErr_SetString(PyExc_BufferError,

Objects/object.c

@@ -1650,6 +1650,9 @@ _Py_ReadyTypes(void)
     if (PyType_Ready(&PyProperty_Type) < 0)
         Py_FatalError("Can't initialize property type");
 
+    if (PyType_Ready(&_PyManagedBuffer_Type) < 0)
+        Py_FatalError("Can't initialize managed buffer type");
+
     if (PyType_Ready(&PyMemoryView_Type) < 0)
         Py_FatalError("Can't initialize memoryview type");
 

PCbuild/_testbuffer.vcproj

@@ -0,0 +1,521 @@
+<?xml version="1.0" encoding="Windows-1252"?>
+<VisualStudioProject
+	ProjectType="Visual C++"
+	Version="9.00"
+	Name="_testbuffer"
+	ProjectGUID="{A2697BD3-28C1-4AEC-9106-8B748639FD16}"
+	RootNamespace="_testbuffer"
+	Keyword="Win32Proj"
+	TargetFrameworkVersion="196613"
+	>
+	<Platforms>
+		<Platform
+			Name="Win32"
+		/>
+		<Platform
+			Name="x64"
+		/>
+	</Platforms>
+	<ToolFiles>
+	</ToolFiles>
+	<Configurations>
+		<Configuration
+			Name="Debug|Win32"
+			ConfigurationType="2"
+			InheritedPropertySheets=".\pyd_d.vsprops"
+			CharacterSet="0"
+			>
+			<Tool
+				Name="VCPreBuildEventTool"
+			/>
+			<Tool
+				Name="VCCustomBuildTool"
+			/>
+			<Tool
+				Name="VCXMLDataGeneratorTool"
+			/>
+			<Tool
+				Name="VCWebServiceProxyGeneratorTool"
+			/>
+			<Tool
+				Name="VCMIDLTool"
+			/>
+			<Tool
+				Name="VCCLCompilerTool"
+			/>
+			<Tool
+				Name="VCManagedResourceCompilerTool"
+			/>
+			<Tool
+				Name="VCResourceCompilerTool"
+			/>
+			<Tool
+				Name="VCPreLinkEventTool"
+			/>
+			<Tool
+				Name="VCLinkerTool"
+				BaseAddress="0x1e1F0000"
+			/>
+			<Tool
+				Name="VCALinkTool"
+			/>
+			<Tool
+				Name="VCManifestTool"
+			/>
+			<Tool
+				Name="VCXDCMakeTool"
+			/>
+			<Tool
+				Name="VCBscMakeTool"
+			/>
+			<Tool
+				Name="VCFxCopTool"
+			/>
+			<Tool
+				Name="VCAppVerifierTool"
+			/>
+			<Tool
+				Name="VCPostBuildEventTool"
+			/>
+		</Configuration>
+		<Configuration
+			Name="Debug|x64"
+			ConfigurationType="2"
+			InheritedPropertySheets=".\pyd_d.vsprops;.\x64.vsprops"
+			CharacterSet="0"
+			>
+			<Tool
+				Name="VCPreBuildEventTool"
+			/>
+			<Tool
+				Name="VCCustomBuildTool"
+			/>
+			<Tool
+				Name="VCXMLDataGeneratorTool"
+			/>
+			<Tool
+				Name="VCWebServiceProxyGeneratorTool"
+			/>
+			<Tool
+				Name="VCMIDLTool"
+				TargetEnvironment="3"
+			/>
+			<Tool
+				Name="VCCLCompilerTool"
+			/>
+			<Tool
+				Name="VCManagedResourceCompilerTool"
+			/>
+			<Tool
+				Name="VCResourceCompilerTool"
+			/>
+			<Tool
+				Name="VCPreLinkEventTool"
+			/>
+			<Tool
+				Name="VCLinkerTool"
+				BaseAddress="0x1e1F0000"
+			/>
+			<Tool
+				Name="VCALinkTool"
+			/>
+			<Tool
+				Name="VCManifestTool"
+			/>
+			<Tool
+				Name="VCXDCMakeTool"
+			/>
+			<Tool
+				Name="VCBscMakeTool"
+			/>
+			<Tool
+				Name="VCFxCopTool"
+			/>
+			<Tool
+				Name="VCAppVerifierTool"
+			/>
+			<Tool
+				Name="VCPostBuildEventTool"
+			/>
+		</Configuration>
+		<Configuration
+			Name="Release|Win32"
+			ConfigurationType="2"
+			InheritedPropertySheets=".\pyd.vsprops"
+			CharacterSet="0"
+			WholeProgramOptimization="1"
+			>
+			<Tool
+				Name="VCPreBuildEventTool"
+			/>
+			<Tool
+				Name="VCCustomBuildTool"
+			/>
+			<Tool
+				Name="VCXMLDataGeneratorTool"
+			/>
+			<Tool
+				Name="VCWebServiceProxyGeneratorTool"
+			/>
+			<Tool
+				Name="VCMIDLTool"
+			/>
+			<Tool
+				Name="VCCLCompilerTool"
+			/>
+			<Tool
+				Name="VCManagedResourceCompilerTool"
+			/>
+			<Tool
+				Name="VCResourceCompilerTool"
+			/>
+			<Tool
+				Name="VCPreLinkEventTool"
+			/>
+			<Tool
+				Name="VCLinkerTool"
+				BaseAddress="0x1e1F0000"
+			/>
+			<Tool
+				Name="VCALinkTool"
+			/>
+			<Tool
+				Name="VCManifestTool"
+			/>
+			<Tool
+				Name="VCXDCMakeTool"
+			/>
+			<Tool
+				Name="VCBscMakeTool"
+			/>
+			<Tool
+				Name="VCFxCopTool"
+			/>
+			<Tool
+				Name="VCAppVerifierTool"
+			/>
+			<Tool
+				Name="VCPostBuildEventTool"
+			/>
+		</Configuration>
+		<Configuration
+			Name="Release|x64"
+			ConfigurationType="2"
+			InheritedPropertySheets=".\pyd.vsprops;.\x64.vsprops"
+			CharacterSet="0"
+			WholeProgramOptimization="1"
+			>
+			<Tool
+				Name="VCPreBuildEventTool"
+			/>
+			<Tool
+				Name="VCCustomBuildTool"
+			/>
+			<Tool
+				Name="VCXMLDataGeneratorTool"
+			/>
+			<Tool
+				Name="VCWebServiceProxyGeneratorTool"
+			/>
+			<Tool
+				Name="VCMIDLTool"
+				TargetEnvironment="3"
+			/>
+			<Tool
+				Name="VCCLCompilerTool"
+			/>
+			<Tool
+				Name="VCManagedResourceCompilerTool"
+			/>
+			<Tool
+				Name="VCResourceCompilerTool"
+			/>
+			<Tool
+				Name="VCPreLinkEventTool"
+			/>
+			<Tool
+				Name="VCLinkerTool"
+				BaseAddress="0x1e1F0000"
+			/>
+			<Tool
+				Name="VCALinkTool"
+			/>
+			<Tool
+				Name="VCManifestTool"
+			/>
+			<Tool
+				Name="VCXDCMakeTool"
+			/>
+			<Tool
+				Name="VCBscMakeTool"
+			/>
+			<Tool
+				Name="VCFxCopTool"
+			/>
+			<Tool
+				Name="VCAppVerifierTool"
+			/>
+			<Tool
+				Name="VCPostBuildEventTool"
+			/>
+		</Configuration>
+		<Configuration
+			Name="PGInstrument|Win32"
+			ConfigurationType="2"
+			InheritedPropertySheets=".\pyd.vsprops;.\pginstrument.vsprops"
+			CharacterSet="0"
+			WholeProgramOptimization="1"
+			>
+			<Tool
+				Name="VCPreBuildEventTool"
+			/>
+			<Tool
+				Name="VCCustomBuildTool"
+			/>
+			<Tool
+				Name="VCXMLDataGeneratorTool"
+			/>
+			<Tool
+				Name="VCWebServiceProxyGeneratorTool"
+			/>
+			<Tool
+				Name="VCMIDLTool"
+			/>
+			<Tool
+				Name="VCCLCompilerTool"
+			/>
+			<Tool
+				Name="VCManagedResourceCompilerTool"
+			/>
+			<Tool
+				Name="VCResourceCompilerTool"
+			/>
+			<Tool
+				Name="VCPreLinkEventTool"
+			/>
+			<Tool
+				Name="VCLinkerTool"
+				BaseAddress="0x1e1F0000"
+			/>
+			<Tool
+				Name="VCALinkTool"
+			/>
+			<Tool
+				Name="VCManifestTool"
+			/>
+			<Tool
+				Name="VCXDCMakeTool"
+			/>
+			<Tool
+				Name="VCBscMakeTool"
+			/>
+			<Tool
+				Name="VCFxCopTool"
+			/>
+			<Tool
+				Name="VCAppVerifierTool"
+			/>
+			<Tool
+				Name="VCPostBuildEventTool"
+			/>
+		</Configuration>
+		<Configuration
+			Name="PGInstrument|x64"
+			ConfigurationType="2"
+			InheritedPropertySheets=".\pyd.vsprops;.\x64.vsprops;.\pginstrument.vsprops"
+			CharacterSet="0"
+			WholeProgramOptimization="1"
+			>
+			<Tool
+				Name="VCPreBuildEventTool"
+			/>
+			<Tool
+				Name="VCCustomBuildTool"
+			/>
+			<Tool
+				Name="VCXMLDataGeneratorTool"
+			/>
+			<Tool
+				Name="VCWebServiceProxyGeneratorTool"
+			/>
+			<Tool
+				Name="VCMIDLTool"
+				TargetEnvironment="3"
+			/>
+			<Tool
+				Name="VCCLCompilerTool"
+			/>
+			<Tool
+				Name="VCManagedResourceCompilerTool"
+			/>
+			<Tool
+				Name="VCResourceCompilerTool"
+			/>
+			<Tool
+				Name="VCPreLinkEventTool"
+			/>
+			<Tool
+				Name="VCLinkerTool"
+				BaseAddress="0x1e1F0000"
+				TargetMachine="17"
+			/>
+			<Tool
+				Name="VCALinkTool"
+			/>
+			<Tool
+				Name="VCManifestTool"
+			/>
+			<Tool
+				Name="VCXDCMakeTool"
+			/>
+			<Tool
+				Name="VCBscMakeTool"
+			/>
+			<Tool
+				Name="VCFxCopTool"
+			/>
+			<Tool
+				Name="VCAppVerifierTool"
+			/>
+			<Tool
+				Name="VCPostBuildEventTool"
+			/>
+		</Configuration>
+		<Configuration
+			Name="PGUpdate|Win32"
+			ConfigurationType="2"
+			InheritedPropertySheets=".\pyd.vsprops;.\pgupdate.vsprops"
+			CharacterSet="0"
+			WholeProgramOptimization="1"
+			>
+			<Tool
+				Name="VCPreBuildEventTool"
+			/>
+			<Tool
+				Name="VCCustomBuildTool"
+			/>
+			<Tool
+				Name="VCXMLDataGeneratorTool"
+			/>
+			<Tool
+				Name="VCWebServiceProxyGeneratorTool"
+			/>
+			<Tool
+				Name="VCMIDLTool"
+			/>
+			<Tool
+				Name="VCCLCompilerTool"
+			/>
+			<Tool
+				Name="VCManagedResourceCompilerTool"
+			/>
+			<Tool
+				Name="VCResourceCompilerTool"
+			/>
+			<Tool
+				Name="VCPreLinkEventTool"
+			/>
+			<Tool
+				Name="VCLinkerTool"
+				BaseAddress="0x1e1F0000"
+			/>
+			<Tool
+				Name="VCALinkTool"
+			/>
+			<Tool
+				Name="VCManifestTool"
+			/>
+			<Tool
+				Name="VCXDCMakeTool"
+			/>
+			<Tool
+				Name="VCBscMakeTool"
+			/>
+			<Tool
+				Name="VCFxCopTool"
+			/>
+			<Tool
+				Name="VCAppVerifierTool"
+			/>
+			<Tool
+				Name="VCPostBuildEventTool"
+			/>
+		</Configuration>
+		<Configuration
+			Name="PGUpdate|x64"
+			ConfigurationType="2"
+			InheritedPropertySheets=".\pyd.vsprops;.\x64.vsprops;.\pgupdate.vsprops"
+			CharacterSet="0"
+			WholeProgramOptimization="1"
+			>
+			<Tool
+				Name="VCPreBuildEventTool"
+			/>
+			<Tool
+				Name="VCCustomBuildTool"
+			/>
+			<Tool
+				Name="VCXMLDataGeneratorTool"
+			/>
+			<Tool
+				Name="VCWebServiceProxyGeneratorTool"
+			/>
+			<Tool
+				Name="VCMIDLTool"
+				TargetEnvironment="3"
+			/>
+			<Tool
+				Name="VCCLCompilerTool"
+			/>
+			<Tool
+				Name="VCManagedResourceCompilerTool"
+			/>
+			<Tool
+				Name="VCResourceCompilerTool"
+			/>
+			<Tool
+				Name="VCPreLinkEventTool"
+			/>
+			<Tool
+				Name="VCLinkerTool"
+				BaseAddress="0x1e1F0000"
+				TargetMachine="17"
+			/>
+			<Tool
+				Name="VCALinkTool"
+			/>
+			<Tool
+				Name="VCManifestTool"
+			/>
+			<Tool
+				Name="VCXDCMakeTool"
+			/>
+			<Tool
+				Name="VCBscMakeTool"
+			/>
+			<Tool
+				Name="VCFxCopTool"
+			/>
+			<Tool
+				Name="VCAppVerifierTool"
+			/>
+			<Tool
+				Name="VCPostBuildEventTool"
+			/>
+		</Configuration>
+	</Configurations>
+	<References>
+	</References>
+	<Files>
+		<Filter
+			Name="Source Files"
+			>
+			<File
+				RelativePath="..\Modules\_testbuffer.c"
+				>
+			</File>
+		</Filter>
+	</Files>
+	<Globals>
+	</Globals>
+</VisualStudioProject>

PCbuild/pcbuild.sln

@@ -142,6 +142,11 @@ Project("{8BC9CEB8-8B4A-11D0-8D11-00A0C91BC942}") = "python3dll", "python3dll.vc
 EndProject
 Project("{8BC9CEB8-8B4A-11D0-8D11-00A0C91BC942}") = "xxlimited", "xxlimited.vcproj", "{F749B822-B489-4CA5-A3AD-CE078F5F338A}"
 EndProject
+Project("{8BC9CEB8-8B4A-11D0-8D11-00A0C91BC942}") = "_testbuffer", "_testbuffer.vcproj", "{A2697BD3-28C1-4AEC-9106-8B748639FD16}"
+	ProjectSection(ProjectDependencies) = postProject
+		{CF7AC3D1-E2DF-41D2-BEA6-1E2556CDEA26} = {CF7AC3D1-E2DF-41D2-BEA6-1E2556CDEA26}
+	EndProjectSection
+EndProject
 Global
 	GlobalSection(SolutionConfigurationPlatforms) = preSolution
 		Debug|Win32 = Debug|Win32
@@ -609,6 +614,22 @@ Global
 		{F749B822-B489-4CA5-A3AD-CE078F5F338A}.Release|Win32.Build.0 = Release|Win32
 		{F749B822-B489-4CA5-A3AD-CE078F5F338A}.Release|x64.ActiveCfg = Release|x64
 		{F749B822-B489-4CA5-A3AD-CE078F5F338A}.Release|x64.Build.0 = Release|x64
+		{A2697BD3-28C1-4AEC-9106-8B748639FD16}.Debug|Win32.ActiveCfg = Debug|Win32
+		{A2697BD3-28C1-4AEC-9106-8B748639FD16}.Debug|Win32.Build.0 = Debug|Win32
+		{A2697BD3-28C1-4AEC-9106-8B748639FD16}.Debug|x64.ActiveCfg = Debug|x64
+		{A2697BD3-28C1-4AEC-9106-8B748639FD16}.Debug|x64.Build.0 = Debug|x64
+		{A2697BD3-28C1-4AEC-9106-8B748639FD16}.PGInstrument|Win32.ActiveCfg = PGInstrument|Win32
+		{A2697BD3-28C1-4AEC-9106-8B748639FD16}.PGInstrument|Win32.Build.0 = PGInstrument|Win32
+		{A2697BD3-28C1-4AEC-9106-8B748639FD16}.PGInstrument|x64.ActiveCfg = PGInstrument|x64
+		{A2697BD3-28C1-4AEC-9106-8B748639FD16}.PGInstrument|x64.Build.0 = PGInstrument|x64
+		{A2697BD3-28C1-4AEC-9106-8B748639FD16}.PGUpdate|Win32.ActiveCfg = PGUpdate|Win32
+		{A2697BD3-28C1-4AEC-9106-8B748639FD16}.PGUpdate|Win32.Build.0 = PGUpdate|Win32
+		{A2697BD3-28C1-4AEC-9106-8B748639FD16}.PGUpdate|x64.ActiveCfg = PGUpdate|x64
+		{A2697BD3-28C1-4AEC-9106-8B748639FD16}.PGUpdate|x64.Build.0 = PGUpdate|x64
+		{A2697BD3-28C1-4AEC-9106-8B748639FD16}.Release|Win32.ActiveCfg = Release|Win32
+		{A2697BD3-28C1-4AEC-9106-8B748639FD16}.Release|Win32.Build.0 = Release|Win32
+		{A2697BD3-28C1-4AEC-9106-8B748639FD16}.Release|x64.ActiveCfg = Release|x64
+		{A2697BD3-28C1-4AEC-9106-8B748639FD16}.Release|x64.Build.0 = Release|x64
 	EndGlobalSection
 	GlobalSection(SolutionProperties) = preSolution
 		HideSolutionNode = FALSE

PCbuild/readme.txt

@@ -92,6 +92,9 @@ _socket
 _testcapi
     tests of the Python C API, run via Lib/test/test_capi.py, and
     implemented by module Modules/_testcapimodule.c
+_testbuffer
+    buffer protocol tests, run via Lib/test/test_buffer.py, and
+    implemented by module Modules/_testbuffer.c
 pyexpat
     Python wrapper for accelerated XML parsing, which incorporates stable
     code from the Expat project:  http://sourceforge.net/projects/expat/

setup.py

@@ -530,6 +530,8 @@ class PyBuildExt(build_ext):
         # Python C API test module
         exts.append( Extension('_testcapi', ['_testcapimodule.c'],
                                depends=['testcapi_long.h']) )
+        # Python PEP-3118 (buffer protocol) test module
+        exts.append( Extension('_testbuffer', ['_testbuffer.c']) )
         # profiler (_lsprof is for cProfile.py)
         exts.append( Extension('_lsprof', ['_lsprof.c', 'rotatingtree.c']) )
         # static Unicode character database