traverseda
/
cpython
1
0
Fork 0
Code Issues Pull Requests Releases Wiki Activity

Add doctests to the descriptor HowTo (GH-23500)

This commit is contained in:
Raymond Hettinger 2020-11-24 20:57:02 -08:00 committed by GitHub
parent ed1a5a5bac
commit 2d44a6bc4f
No known key found for this signature in database
GPG Key ID: 4AEE18F83AFDEB23
1 changed files with 397 additions and 57 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

454
Doc/howto/descriptor.rst
View File

@ -43,21 +43,26 @@ Simple example: A descriptor that returns a constant
----------------------------------------------------
The :class:`Ten` class is a descriptor that always returns the constant ``10``
from its :meth:`__get__` method::
from its :meth:`__get__` method:
.. testcode::
class Ten:
def __get__(self, obj, objtype=None):
return 10
To use the descriptor, it must be stored as a class variable in another class::
To use the descriptor, it must be stored as a class variable in another class:
.. testcode::
class A:
x = 5 # Regular class attribute
y = Ten() # Descriptor instance
An interactive session shows the difference between normal attribute lookup
and descriptor lookup::
and descriptor lookup:
.. doctest::
>>> a = A() # Make an instance of class A
>>> a.x # Normal attribute lookup
@ -83,7 +88,9 @@ Dynamic lookups
---------------
Interesting descriptors typically run computations instead of returning
constants::
constants:
.. testcode::
import os
@ -131,7 +138,9 @@ the public attribute is accessed.
In the following example, *age* is the public attribute and *_age* is the
private attribute. When the public attribute is accessed, the descriptor logs
the lookup or update::
the lookup or update:
.. testcode::
import logging
@ -201,7 +210,9 @@ variable name was used.
In this example, the :class:`Person` class has two descriptor instances,
*name* and *age*. When the :class:`Person` class is defined, it makes a
callback to :meth:`__set_name__` in *LoggedAccess* so that the field names can
be recorded, giving each descriptor its own *public_name* and *private_name*::
be recorded, giving each descriptor its own *public_name* and *private_name*:
.. testcode::
import logging
@ -236,7 +247,9 @@ be recorded, giving each descriptor its own *public_name* and *private_name*::
An interactive session shows that the :class:`Person` class has called
:meth:`__set_name__` so that the field names would be recorded. Here
we call :func:`vars` to look up the descriptor without triggering it::
we call :func:`vars` to look up the descriptor without triggering it:
.. doctest::
>>> vars(vars(Person)['name'])
{'public_name': 'name', 'private_name': '_name'}
@ -307,7 +320,9 @@ restrictions. If those restrictions aren't met, it raises an exception to
prevent data corruption at its source.
This :class:`Validator` class is both an :term:`abstract base class` and a
managed attribute descriptor::
managed attribute descriptor:
.. testcode::
from abc import ABC, abstractmethod
@ -347,7 +362,7 @@ Here are three practical data validation utilities:
user-defined `predicate
<https://en.wikipedia.org/wiki/Predicate_(mathematical_logic)>`_ as well.
::
.. testcode::
class OneOf(Validator):
@ -400,10 +415,12 @@ Here are three practical data validation utilities:
)
Practical use
-------------
Practical application
---------------------
Here's how the data validators can be used in a real class::
Here's how the data validators can be used in a real class:
.. testcode::
class Component:
@ -418,11 +435,26 @@ Here's how the data validators can be used in a real class::
The descriptors prevent invalid instances from being created::
Component('WIDGET', 'metal', 5) # Allowed.
Component('Widget', 'metal', 5) # Blocked: 'Widget' is not all uppercase
Component('WIDGET', 'metle', 5) # Blocked: 'metle' is misspelled
Component('WIDGET', 'metal', -5) # Blocked: -5 is negative
Component('WIDGET', 'metal', 'V') # Blocked: 'V' isn't a number
>>> Component('Widget', 'metal', 5) # Blocked: 'Widget' is not all uppercase
Traceback (most recent call last):
...
ValueError: Expected <method 'isupper' of 'str' objects> to be true for 'Widget'
>>> Component('WIDGET', 'metle', 5) # Blocked: 'metle' is misspelled
Traceback (most recent call last):
...
ValueError: Expected 'metle' to be one of {'metal', 'plastic', 'wood'}
>>> Component('WIDGET', 'metal', -5) # Blocked: -5 is negative
Traceback (most recent call last):
...
ValueError: Expected -5 to be at least 0
>>> Component('WIDGET', 'metal', 'V') # Blocked: 'V' isn't a number
Traceback (most recent call last):
...
TypeError: Expected 'V' to be an int or float
>>> c = Component('WIDGET', 'metal', 5) # Allowed: The inputs are valid
Technical Tutorial
@ -526,7 +558,9 @@ If a descriptor is found for ``a.x``, then it is invoked with:
``desc.__get__(a, type(a))``.
The logic for a dotted lookup is in :meth:`object.__getattribute__`. Here is
a pure Python equivalent::
a pure Python equivalent:
.. testcode::
def object_getattribute(obj, name):
"Emulate PyObject_GenericGetAttr() in Objects/object.c"
@ -546,9 +580,108 @@ a pure Python equivalent::
return cls_var # class variable
raise AttributeError(name)
.. testcode::
:hide:
# Test the fidelity of object_getattribute() by comparing it with the
# normal object.__getattribute__(). The former will be accessed by
# square brackets and the latter by the dot operator.
class Object:
def __getitem__(obj, name):
try:
return object_getattribute(obj, name)
except AttributeError:
if not hasattr(type(obj), '__getattr__'):
raise
return type(obj).__getattr__(obj, name) # __getattr__
class DualOperator(Object):
x = 10
def __init__(self, z):
self.z = z
@property
def p2(self):
return 2 * self.x
@property
def p3(self):
return 3 * self.x
def m5(self, y):
return 5 * y
def m7(self, y):
return 7 * y
def __getattr__(self, name):
return ('getattr_hook', self, name)
class DualOperatorWithSlots:
__getitem__ = Object.__getitem__
__slots__ = ['z']
x = 15
def __init__(self, z):
self.z = z
@property
def p2(self):
return 2 * self.x
def m5(self, y):
return 5 * y
def __getattr__(self, name):
return ('getattr_hook', self, name)
.. doctest::
:hide:
>>> a = DualOperator(11)
>>> vars(a).update(p3 = '_p3', m7 = '_m7')
>>> a.x == a['x'] == 10
True
>>> a.z == a['z'] == 11
True
>>> a.p2 == a['p2'] == 20
True
>>> a.p3 == a['p3'] == 30
True
>>> a.m5(100) == a.m5(100) == 500
True
>>> a.m7 == a['m7'] == '_m7'
True
>>> a.g == a['g'] == ('getattr_hook', a, 'g')
True
>>> b = DualOperatorWithSlots(22)
>>> b.x == b['x'] == 15
True
>>> b.z == b['z'] == 22
True
>>> b.p2 == b['p2'] == 30
True
>>> b.m5(200) == b['m5'](200) == 1000
True
>>> b.g == b['g'] == ('getattr_hook', b, 'g')
True
Interestingly, attribute lookup doesn't call :meth:`object.__getattribute__`
directly. Instead, both the dot operator and the :func:`getattr` function
perform attribute lookup by way of a helper function::
perform attribute lookup by way of a helper function:
.. testcode::
def getattr_hook(obj, name):
"Emulate slot_tp_getattr_hook() in Objects/typeobject.c"
@ -650,7 +783,9 @@ be used to implement an `object relational mapping
The essential idea is that the data is stored in an external database. The
Python instances only hold keys to the database's tables. Descriptors take
care of lookups or updates::
care of lookups or updates:
.. testcode::
class Field:
@ -665,8 +800,11 @@ care of lookups or updates::
conn.execute(self.store, [value, obj.key])
conn.commit()
We can use the :class:`Field` class to define "models" that describe the schema
for each table in a database::
We can use the :class:`Field` class to define `models
<https://en.wikipedia.org/wiki/Database_model>`_ that describe the schema for
each table in a database:
.. testcode::
class Movie:
table = 'Movies' # Table name
@ -687,12 +825,41 @@ for each table in a database::
def __init__(self, key):
self.key = key
An interactive session shows how data is retrieved from the database and how
it can be updated::
To use the models, first connect to the database::
>>> import sqlite3
>>> conn = sqlite3.connect('entertainment.db')
An interactive session shows how data is retrieved from the database and how
it can be updated:
.. testsetup::
song_data = [
('Country Roads', 'John Denver', 1972),
('Me and Bobby McGee', 'Janice Joplin', 1971),
('Coal Miners Daughter', 'Loretta Lynn', 1970),
]
movie_data = [
('Star Wars', 'George Lucas', 1977),
('Jaws', 'Steven Spielberg', 1975),
('Aliens', 'James Cameron', 1986),
]
import sqlite3
conn = sqlite3.connect(':memory:')
conn.execute('CREATE TABLE Music (title text, artist text, year integer);')
conn.execute('CREATE INDEX MusicNdx ON Music (title);')
conn.executemany('INSERT INTO Music VALUES (?, ?, ?);', song_data)
conn.execute('CREATE TABLE Movies (title text, director text, year integer);')
conn.execute('CREATE INDEX MovieNdx ON Music (title);')
conn.executemany('INSERT INTO Movies VALUES (?, ?, ?);', movie_data)
conn.commit()
.. doctest::
>>> Movie('Star Wars').director
'George Lucas'
>>> jaws = Movie('Jaws')
@ -724,7 +891,9 @@ triggers a function call upon access to an attribute. Its signature is::
property(fget=None, fset=None, fdel=None, doc=None) -> property
The documentation shows a typical use to define a managed attribute ``x``::
The documentation shows a typical use to define a managed attribute ``x``:
.. testcode::
class C:
def getx(self): return self.__x
@ -733,7 +902,9 @@ The documentation shows a typical use to define a managed attribute ``x``::
x = property(getx, setx, delx, "I'm the 'x' property.")
To see how :func:`property` is implemented in terms of the descriptor protocol,
here is a pure Python equivalent::
here is a pure Python equivalent:
.. testcode::
class Property:
"Emulate PyProperty_Type() in Objects/descrobject.c"
@ -772,6 +943,57 @@ here is a pure Python equivalent::
def deleter(self, fdel):
return type(self)(self.fget, self.fset, fdel, self.__doc__)
.. testcode::
:hide:
# Verify the Property() emulation
class CC:
def getx(self):
return self.__x
def setx(self, value):
self.__x = value
def delx(self):
del self.__x
x = Property(getx, setx, delx, "I'm the 'x' property.")
# Now do it again but use the decorator style
class CCC:
@Property
def x(self):
return self.__x
@x.setter
def x(self, value):
self.__x = value
@x.deleter
def x(self):
del self.__x
.. doctest::
:hide:
>>> cc = CC()
>>> hasattr(cc, 'x')
False
>>> cc.x = 33
>>> cc.x
33
>>> del cc.x
>>> hasattr(cc, 'x')
False
>>> ccc = CCC()
>>> hasattr(ccc, 'x')
False
>>> ccc.x = 333
>>> ccc.x == 333
True
>>> del ccc.x
>>> hasattr(ccc, 'x')
False
The :func:`property` builtin helps whenever a user interface has granted
attribute access and then subsequent changes require the intervention of a
method.
@ -780,7 +1002,9 @@ For instance, a spreadsheet class may grant access to a cell value through
``Cell('b10').value``. Subsequent improvements to the program require the cell
to be recalculated on every access; however, the programmer does not want to
affect existing client code accessing the attribute directly. The solution is
to wrap access to the value attribute in a property data descriptor::
to wrap access to the value attribute in a property data descriptor:
.. testcode::
class Cell:
...
@ -791,6 +1015,9 @@ to wrap access to the value attribute in a property data descriptor::
self.recalc()
return self._value
Either the built-in :func:`property` or our :func:`Property` equivalent would
work in this example.
Functions and methods
---------------------
@ -804,7 +1031,9 @@ prepended to the other arguments. By convention, the instance is called
*self* but could be called *this* or any other variable name.
Methods can be created manually with :class:`types.MethodType` which is
roughly equivalent to::
roughly equivalent to:
.. testcode::
class MethodType:
"Emulate Py_MethodType in Objects/classobject.c"
@ -821,7 +1050,9 @@ roughly equivalent to::
To support automatic creation of methods, functions include the
:meth:`__get__` method for binding methods during attribute access. This
means that functions are non-data descriptors that return bound methods
during dotted lookup from an instance. Here's how it works::
during dotted lookup from an instance. Here's how it works:
.. testcode::
class Function:
...
@ -833,13 +1064,17 @@ during dotted lookup from an instance. Here's how it works::
return MethodType(self, obj)
Running the following class in the interpreter shows how the function
descriptor works in practice::
descriptor works in practice:
.. testcode::
class D:
def f(self, x):
return x
The function has a :term:`qualified name` attribute to support introspection::
The function has a :term:`qualified name` attribute to support introspection:
.. doctest::
>>> D.f.__qualname__
'D.f'
@ -867,7 +1102,7 @@ Internally, the bound method stores the underlying function and the bound
instance::
>>> d.f.__func__
<function D.f at 0x1012e5ae8>
<function D.f at 0x00C45070>
>>> d.f.__self__
<__main__.D object at 0x1012e1f98>
@ -919,20 +1154,26 @@ It can be called either from an object or the class: ``s.erf(1.5) --> .9332`` o
``Sample.erf(1.5) --> .9332``.
Since static methods return the underlying function with no changes, the
example calls are unexciting::
example calls are unexciting:
.. testcode::
class E:
@staticmethod
def f(x):
print(x)
.. doctest::
>>> E.f(3)
3
>>> E().f(3)
3
Using the non-data descriptor protocol, a pure Python version of
:func:`staticmethod` would look like this::
:func:`staticmethod` would look like this:
.. doctest::
class StaticMethod:
"Emulate PyStaticMethod_Type() in Objects/funcobject.c"
@ -949,27 +1190,31 @@ Class methods
Unlike static methods, class methods prepend the class reference to the
argument list before calling the function. This format is the same
for whether the caller is an object or a class::
for whether the caller is an object or a class:
.. testcode::
class F:
@classmethod
def f(cls, x):
return cls.__name__, x
>>> print(F.f(3))
.. doctest::
>>> F.f(3)
('F', 3)
>>> print(F().f(3))
>>> F().f(3)
('F', 3)
This behavior is useful whenever the method only needs to have a class
reference and does rely on data stored in a specific instance. One use for
class methods is to create alternate class constructors. For example, the
classmethod :func:`dict.fromkeys` creates a new dictionary from a list of
keys. The pure Python equivalent is::
keys. The pure Python equivalent is:
class Dict:
...
.. testcode::
class Dict(dict):
@classmethod
def fromkeys(cls, iterable, value=None):
"Emulate dict_fromkeys() in Objects/dictobject.c"
@ -978,13 +1223,17 @@ keys. The pure Python equivalent is::
d[key] = value
return d
Now a new dictionary of unique keys can be constructed like this::
Now a new dictionary of unique keys can be constructed like this:
.. doctest::
>>> Dict.fromkeys('abracadabra')
{'a': None, 'r': None, 'b': None, 'c': None, 'd': None}
{'a': None, 'b': None, 'r': None, 'c': None, 'd': None}
Using the non-data descriptor protocol, a pure Python version of
:func:`classmethod` would look like this::
:func:`classmethod` would look like this:
.. testcode::
class ClassMethod:
"Emulate PyClassMethod_Type() in Objects/funcobject.c"
@ -999,9 +1248,31 @@ Using the non-data descriptor protocol, a pure Python version of
return self.f.__get__(cls)
return MethodType(self.f, cls)
.. testcode::
:hide:
# Verify the emulation works
class T:
@ClassMethod
def cm(cls, x, y):
return (cls, x, y)
.. doctest::
:hide:
>>> T.cm(11, 22)
(<class 'T'>, 11, 22)
# Also call it from an instance
>>> t = T()
>>> t.cm(11, 22)
(<class 'T'>, 11, 22)
The code path for ``hasattr(obj, '__get__')`` was added in Python 3.9 and
makes it possible for :func:`classmethod` to support chained decorators.
For example, a classmethod and property could be chained together::
For example, a classmethod and property could be chained together:
.. testcode::
class G:
@classmethod
@ -1009,6 +1280,12 @@ For example, a classmethod and property could be chained together::
def __doc__(cls):
return f'A doc for {cls.__name__!r}'
.. doctest::
>>> G.__doc__
"A doc for 'G'"
Member objects and __slots__
----------------------------
@ -1017,11 +1294,15 @@ fixed-length array of slot values. From a user point of view that has
several effects:
1. Provides immediate detection of bugs due to misspelled attribute
assignments. Only attribute names specified in ``__slots__`` are allowed::
assignments. Only attribute names specified in ``__slots__`` are allowed:
.. testcode::
class Vehicle:
__slots__ = ('id_number', 'make', 'model')
.. doctest::
>>> auto = Vehicle()
>>> auto.id_nubmer = 'VYE483814LQEX'
Traceback (most recent call last):
@ -1029,7 +1310,9 @@ assignments. Only attribute names specified in ``__slots__`` are allowed::
AttributeError: 'Vehicle' object has no attribute 'id_nubmer'
2. Helps create immutable objects where descriptors manage access to private
attributes stored in ``__slots__``::
attributes stored in ``__slots__``:
.. testcode::
class Immutable:
@ -1047,7 +1330,19 @@ attributes stored in ``__slots__``::
def name(self): # Read-only descriptor
return self._name
mark = Immutable('Botany', 'Mark Watney') # Create an immutable instance
.. doctest::
>>> mark = Immutable('Botany', 'Mark Watney')
>>> mark.dept
'Botany'
>>> mark.dept = 'Space Pirate'
Traceback (most recent call last):
...
AttributeError: can't set attribute
>>> mark.location = 'Mars'
Traceback (most recent call last):
...
AttributeError: 'Immutable' object has no attribute 'location'
3. Saves memory. On a 64-bit Linux build, an instance with two attributes
takes 48 bytes with ``__slots__`` and 152 bytes without. This `flyweight
@ -1055,7 +1350,9 @@ design pattern <https://en.wikipedia.org/wiki/Flyweight_pattern>`_ likely only
matters when a large number of instances are going to be created.
4. Blocks tools like :func:`functools.cached_property` which require an
instance dictionary to function correctly::
instance dictionary to function correctly:
.. testcode::
from functools import cached_property
@ -1067,17 +1364,21 @@ instance dictionary to function correctly::
return 4 * sum((-1.0)**n / (2.0*n + 1.0)
for n in reversed(range(100_000)))
.. doctest::
>>> CP().pi
Traceback (most recent call last):
...
TypeError: No '__dict__' attribute on 'CP' instance to cache 'pi' property.
It's not possible to create an exact drop-in pure Python version of
It is not possible to create an exact drop-in pure Python version of
``__slots__`` because it requires direct access to C structures and control
over object memory allocation. However, we can build a mostly faithful
simulation where the actual C structure for slots is emulated by a private
``_slotvalues`` list. Reads and writes to that private structure are managed
by member descriptors::
by member descriptors:
.. testcode::
null = object()
@ -1114,7 +1415,9 @@ by member descriptors::
return f'<Member {self.name!r} of {self.clsname!r}>'
The :meth:`type.__new__` method takes care of adding member objects to class
variables::
variables:
.. testcode::
class Type(type):
'Simulate how the type metaclass adds member objects for slots'
@ -1129,7 +1432,9 @@ variables::
The :meth:`object.__new__` method takes care of creating instances that have
slots instead of an instance dictionary. Here is a rough simulation in pure
Python::
Python:
.. testcode::
class Object:
'Simulate how object.__new__() allocates memory for __slots__'
@ -1161,7 +1466,9 @@ Python::
super().__delattr__(name)
To use the simulation in a real class, just inherit from :class:`Object` and
set the :term:`metaclass` to :class:`Type`::
set the :term:`metaclass` to :class:`Type`:
.. testcode::
class H(Object, metaclass=Type):
'Instance variables stored in slots'
@ -1174,8 +1481,8 @@ set the :term:`metaclass` to :class:`Type`::
At this point, the metaclass has loaded member objects for *x* and *y*::
>>> import pprint
>>> pprint.pp(dict(vars(H)))
>>> from pprint import pp
>>> pp(dict(vars(H)))
{'__module__': '__main__',
'__doc__': 'Instance variables stored in slots',
'slot_names': ['x', 'y'],
@ -1183,8 +1490,20 @@ At this point, the metaclass has loaded member objects for *x* and *y*::
'x': <Member 'x' of 'H'>,
'y': <Member 'y' of 'H'>}
.. doctest::
:hide:
# We test this separately because the preceding section is not
# doctestable due to the hex memory address for the __init__ function
>>> isinstance(vars(H)['x'], Member)
True
>>> isinstance(vars(H)['y'], Member)
True
When instances are created, they have a ``slot_values`` list where the
attributes are stored::
attributes are stored:
.. doctest::
>>> h = H(10, 20)
>>> vars(h)
@ -1193,9 +1512,30 @@ attributes are stored::
>>> vars(h)
{'_slotvalues': [55, 20]}
Misspelled or unassigned attributes will raise an exception::
Misspelled or unassigned attributes will raise an exception:
.. doctest::
>>> h.xz
Traceback (most recent call last):
...
AttributeError: 'H' object has no attribute 'xz'
.. doctest::
:hide:
# Examples for deleted attributes are not shown because this section
# is already a bit lengthy. We still test that code here.
>>> del h.x
>>> hasattr(h, 'x')
False
# Also test the code for uninitialized slots
>>> class HU(Object, metaclass=Type):
... slot_names = ['x', 'y']
...
>>> hu = HU()
>>> hasattr(hu, 'x')
False
>>> hasattr(hu, 'y')
False