#8696: add documentation for logging.config.dictConfig (PEP 391)

Doc/library/logging.rst

@@ -323,24 +323,34 @@ order::
 Configuring Logging
 ^^^^^^^^^^^^^^^^^^^
 
-Programmers can configure logging either by creating loggers, handlers, and
+Programmers can configure logging in three ways:
-formatters explicitly in a main module with the configuration methods listed
+
-above (using Python code), or by creating a logging config file.  The following
+1. Creating loggers, handlers, and formatters explicitly using Python
-code is an example of configuring a very simple logger, a console handler, and a
+   code that calls the configuration methods listed above.
-simple formatter in a Python module::
+2. Creating a logging config file and reading it using the :func:`fileConfig`
+   function.
+3. Creating a dictionary of configuration information and passing it
+   to the :func:`dictConfig` function.
+
+The following example configures a very simple logger, a console
+handler, and a simple formatter using Python code:
 
     import logging
 
     # create logger
     logger = logging.getLogger("simple_example")
     logger.setLevel(logging.DEBUG)
+
     # create console handler and set level to debug
     ch = logging.StreamHandler()
     ch.setLevel(logging.DEBUG)
+
     # create formatter
     formatter = logging.Formatter("%(asctime)s - %(name)s - %(levelname)s - %(message)s")
+
     # add formatter to ch
     ch.setFormatter(formatter)
+
     # add ch to logger
     logger.addHandler(ch)
 
@@ -2620,17 +2630,57 @@ logging module using these functions or by making calls to the main API (defined
 in :mod:`logging` itself) and defining handlers which are declared either in
 :mod:`logging` or :mod:`logging.handlers`.
 
+.. function:: dictConfig(config)
+
+    Takes the logging configuration from a dictionary.  The contents of
+    this dictionary are described in :ref:`logging-config-dictschema`
+    below.
+
+    If an error is encountered during configuration, this function will
+    raise a :exc:`ValueError`, :exc:`TypeError`, :exc:`AttributeError`
+    or :exc:`ImportError` with a suitably descriptive message.  The
+    following is a (possibly incomplete) list of conditions which will
+    raise an error:
+
+    * A ``level`` which is not a string or which is a string not
+      corresponding to an actual logging level.
+    * A ``propagate`` value which is not a boolean.
+    * An id which does not have a corresponding destination.
+    * A non-existent handler id found during an incremental call.
+    * An invalid logger name.
+    * Inability to resolve to an internal or external object.
+
+    Parsing is performed by the :class:`DictConfigurator` class, whose
+    constructor is passed the dictionary used for configuration, and
+    has a :meth:`configure` method.  The :mod:`logging.config` module
+    has a callable attribute :attr:`dictConfigClass`
+    which is initially set to :class:`DictConfigurator`.
+    You can replace the value of :attr:`dictConfigClass` with a
+    suitable implementation of your own.
+
+    :func:`dictConfig` calls :attr:`dictConfigClass` passing
+    the specified dictionary, and then calls the :meth:`configure` method on
+    the returned object to put the configuration into effect::
+
+          def dictConfig(config):
+              dictConfigClass(config).configure()
+
+    For example, a subclass of :class:`DictConfigurator` could call
+    ``DictConfigurator.__init__()`` in its own :meth:`__init__()`, then
+    set up custom prefixes which would be usable in the subsequent
+    :meth:`configure` call. :attr:`dictConfigClass` would be bound to
+    this new subclass, and then :func:`dictConfig` could be called exactly as
+    in the default, uncustomized state.
 
 .. function:: fileConfig(fname[, defaults])
 
    Reads the logging configuration from a :mod:`ConfigParser`\-format file named
    *fname*. This function can be called several times from an application,
-   allowing an end user the ability to select from various pre-canned
+   allowing an end user to select from various pre-canned
    configurations (if the developer provides a mechanism to present the choices
    and load the chosen configuration). Defaults to be passed to the ConfigParser
    can be specified in the *defaults* argument.
 
-
 .. function:: listen([port])
 
    Starts up a socket server on the specified port, and listens for new
@@ -2653,6 +2703,402 @@ in :mod:`logging` itself) and defining handlers which are declared either in
    :func:`listen`.
 
 
+.. _logging-config-dictschema:
+
+Configuration dictionary schema
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+
+Describing a logging configuration requires listing the various
+objects to create and the connections between them; for example, you
+may create a handler named "console" and then say that the logger
+named "startup" will send its messages to the "console" handler.
+These objects aren't limited to those provided by the :mod:`logging`
+module because you might write your own formatter or handler class.
+The parameters to these classes may also need to include external
+objects such as ``sys.stderr``.  The syntax for describing these
+objects and connections is defined in :ref:`logging-config-dict-connections`
+below.
+
+Dictionary Schema Details
+"""""""""""""""""""""""""
+
+The dictionary passed to :func:`dictConfig` must contain the following
+keys:
+
+* `version` - to be set to an integer value representing the schema
+  version.  The only valid value at present is 1, but having this key
+  allows the schema to evolve while still preserving backwards
+  compatibility.
+
+All other keys are optional, but if present they will be interpreted
+as described below.  In all cases below where a 'configuring dict' is
+mentioned, it will be checked for the special ``'()'`` key to see if a
+custom instantiation is required.  If so, the mechanism described
+above is used to instantiate; otherwise, the context is used to
+determine how to instantiate.
+
+* `formatters` - the corresponding value will be a dict in which each
+  key is a formatter id and each value is a dict describing how to
+  configure the corresponding Formatter instance.
+
+  The configuring dict is searched for keys ``format`` and ``datefmt``
+  (with defaults of ``None``) and these are used to construct a
+  :class:`logging.Formatter` instance.
+
+* `filters` - the corresponding value will be a dict in which each key
+  is a filter id and each value is a dict describing how to configure
+  the corresponding Filter instance.
+
+  The configuring dict is searched for the key ``name`` (defaulting to the
+  empty string) and this is used to construct a :class:`logging.Filter`
+  instance.
+
+* `handlers` - the corresponding value will be a dict in which each
+  key is a handler id and each value is a dict describing how to
+  configure the corresponding Handler instance.
+
+  The configuring dict is searched for the following keys:
+
+  * ``class`` (mandatory).  This is the fully qualified name of the
+    handler class.
+
+  * ``level`` (optional).  The level of the handler.
+
+  * ``formatter`` (optional).  The id of the formatter for this
+    handler.
+
+  * ``filters`` (optional).  A list of ids of the filters for this
+    handler.
+
+  All *other* keys are passed through as keyword arguments to the
+  handler's constructor.  For example, given the snippet::
+
+      handlers:
+        console:
+          class : logging.StreamHandler
+          formatter: brief
+          level   : INFO
+          filters: [allow_foo]
+          stream  : ext://sys.stdout
+        file:
+          class : logging.handlers.RotatingFileHandler
+          formatter: precise
+          filename: logconfig.log
+          maxBytes: 1024
+          backupCount: 3
+
+  the handler with id ``console`` is instantiated as a
+  :class:`logging.StreamHandler`, using ``sys.stdout`` as the underlying
+  stream.  The handler with id ``file`` is instantiated as a
+  :class:`logging.handlers.RotatingFileHandler` with the keyword arguments
+  ``filename='logconfig.log', maxBytes=1024, backupCount=3``.
+
+* `loggers` - the corresponding value will be a dict in which each key
+  is a logger name and each value is a dict describing how to
+  configure the corresponding Logger instance.
+
+  The configuring dict is searched for the following keys:
+
+  * ``level`` (optional).  The level of the logger.
+
+  * ``propagate`` (optional).  The propagation setting of the logger.
+
+  * ``filters`` (optional).  A list of ids of the filters for this
+    logger.
+
+  * ``handlers`` (optional).  A list of ids of the handlers for this
+    logger.
+
+  The specified loggers will be configured according to the level,
+  propagation, filters and handlers specified.
+
+* `root` - this will be the configuration for the root logger.
+  Processing of the configuration will be as for any logger, except
+  that the ``propagate`` setting will not be applicable.
+
+* `incremental` - whether the configuration is to be interpreted as
+  incremental to the existing configuration.  This value defaults to
+  ``False``, which means that the specified configuration replaces the
+  existing configuration with the same semantics as used by the
+  existing :func:`fileConfig` API.
+
+  If the specified value is ``True``, the configuration is processed
+  as described in the section on :ref:`logging-config-dict-incremental`.
+
+* `disable_existing_loggers` - whether any existing loggers are to be
+  disabled. This setting mirrors the parameter of the same name in
+  :func:`fileConfig`. If absent, this parameter defaults to ``True``.
+  This value is ignored if `incremental` is ``True``.
+
+.. _logging-config-dict-incremental:
+
+Incremental Configuration
+"""""""""""""""""""""""""
+
+It is difficult to provide complete flexibility for incremental
+configuration.  For example, because objects such as filters
+and formatters are anonymous, once a configuration is set up, it is
+not possible to refer to such anonymous objects when augmenting a
+configuration.
+
+Furthermore, there is not a compelling case for arbitrarily altering
+the object graph of loggers, handlers, filters, formatters at
+run-time, once a configuration is set up; the verbosity of loggers and
+handlers can be controlled just by setting levels (and, in the case of
+loggers, propagation flags).  Changing the object graph arbitrarily in
+a safe way is problematic in a multi-threaded environment; while not
+impossible, the benefits are not worth the complexity it adds to the
+implementation.
+
+Thus, when the ``incremental`` key of a configuration dict is present
+and is ``True``, the system will completely ignore any ``formatters`` and
+``filters`` entries, and process only the ``level``
+settings in the ``handlers`` entries, and the ``level`` and
+``propagate`` settings in the ``loggers`` and ``root`` entries.
+
+Using a value in the configuration dict lets configurations to be sent
+over the wire as pickled dicts to a socket listener. Thus, the logging
+verbosity of a long-running application can be altered over time with
+no need to stop and restart the application.
+
+.. _logging-config-dict-connections:
+
+Object connections
+""""""""""""""""""
+
+The schema describes a set of logging objects - loggers,
+handlers, formatters, filters - which are connected to each other in
+an object graph.  Thus, the schema needs to represent connections
+between the objects.  For example, say that, once configured, a
+particular logger has attached to it a particular handler.  For the
+purposes of this discussion, we can say that the logger represents the
+source, and the handler the destination, of a connection between the
+two.  Of course in the configured objects this is represented by the
+logger holding a reference to the handler.  In the configuration dict,
+this is done by giving each destination object an id which identifies
+it unambiguously, and then using the id in the source object's
+configuration to indicate that a connection exists between the source
+and the destination object with that id.
+
+So, for example, consider the following YAML snippet::
+
+    formatters:
+      brief:
+        # configuration for formatter with id 'brief' goes here
+      precise:
+        # configuration for formatter with id 'precise' goes here
+    handlers:
+      h1: #This is an id
+       # configuration of handler with id 'h1' goes here
+       formatter: brief
+      h2: #This is another id
+       # configuration of handler with id 'h2' goes here
+       formatter: precise
+    loggers:
+      foo.bar.baz:
+        # other configuration for logger 'foo.bar.baz'
+        handlers: [h1, h2]
+
+(Note: YAML used here because it's a little more readable than the
+equivalent Python source form for the dictionary.)
+
+The ids for loggers are the logger names which would be used
+programmatically to obtain a reference to those loggers, e.g.
+``foo.bar.baz``.  The ids for Formatters and Filters can be any string
+value (such as ``brief``, ``precise`` above) and they are transient,
+in that they are only meaningful for processing the configuration
+dictionary and used to determine connections between objects, and are
+not persisted anywhere when the configuration call is complete.
+
+The above snippet indicates that logger named ``foo.bar.baz`` should
+have two handlers attached to it, which are described by the handler
+ids ``h1`` and ``h2``. The formatter for ``h1`` is that described by id
+``brief``, and the formatter for ``h2`` is that described by id
+``precise``.
+
+
+.. _logging-config-dict-userdef:
+
+User-defined objects
+""""""""""""""""""""
+
+The schema supports user-defined objects for handlers, filters and
+formatters.  (Loggers do not need to have different types for
+different instances, so there is no support in this configuration
+schema for user-defined logger classes.)
+
+Objects to be configured are described by dictionaries
+which detail their configuration.  In some places, the logging system
+will be able to infer from the context how an object is to be
+instantiated, but when a user-defined object is to be instantiated,
+the system will not know how to do this.  In order to provide complete
+flexibility for user-defined object instantiation, the user needs
+to provide a 'factory' - a callable which is called with a
+configuration dictionary and which returns the instantiated object.
+This is signalled by an absolute import path to the factory being
+made available under the special key ``'()'``.  Here's a concrete
+example::
+
+    formatters:
+      brief:
+        format: '%(message)s'
+      default:
+        format: '%(asctime)s %(levelname)-8s %(name)-15s %(message)s'
+        datefmt: '%Y-%m-%d %H:%M:%S'
+      custom:
+          (): my.package.customFormatterFactory
+          bar: baz
+          spam: 99.9
+          answer: 42
+
+The above YAML snippet defines three formatters.  The first, with id
+``brief``, is a standard :class:`logging.Formatter` instance with the
+specified format string.  The second, with id ``default``, has a
+longer format and also defines the time format explicitly, and will
+result in a :class:`logging.Formatter` initialized with those two format
+strings.  Shown in Python source form, the ``brief`` and ``default``
+formatters have configuration sub-dictionaries::
+
+    {
+      'format' : '%(message)s'
+    }
+
+and::
+
+    {
+      'format' : '%(asctime)s %(levelname)-8s %(name)-15s %(message)s',
+      'datefmt' : '%Y-%m-%d %H:%M:%S'
+    }
+
+respectively, and as these dictionaries do not contain the special key
+``'()'``, the instantiation is inferred from the context: as a result,
+standard :class:`logging.Formatter` instances are created.  The
+configuration sub-dictionary for the third formatter, with id
+``custom``, is::
+
+  {
+    '()' : 'my.package.customFormatterFactory',
+    'bar' : 'baz',
+    'spam' : 99.9,
+    'answer' : 42
+  }
+
+and this contains the special key ``'()'``, which means that
+user-defined instantiation is wanted.  In this case, the specified
+factory callable will be used. If it is an actual callable it will be
+used directly - otherwise, if you specify a string (as in the example)
+the actual callable will be located using normal import mechanisms.
+The callable will be called with the **remaining** items in the
+configuration sub-dictionary as keyword arguments.  In the above
+example, the formatter with id ``custom`` will be assumed to be
+returned by the call::
+
+    my.package.customFormatterFactory(bar='baz', spam=99.9, answer=42)
+
+The key ``'()'`` has been used as the special key because it is not a
+valid keyword parameter name, and so will not clash with the names of
+the keyword arguments used in the call.  The ``'()'`` also serves as a
+mnemonic that the corresponding value is a callable.
+
+
+.. _logging-config-dict-externalobj:
+
+Access to external objects
+""""""""""""""""""""""""""
+
+There are times where a configuration needs to refer to objects
+external to the configuration, for example ``sys.stderr``.  If the
+configuration dict is constructed using Python code, this is
+straightforward, but a problem arises when the configuration is
+provided via a text file (e.g. JSON, YAML).  In a text file, there is
+no standard way to distinguish ``sys.stderr`` from the literal string
+``'sys.stderr'``.  To facilitate this distinction, the configuration
+system looks for certain special prefixes in string values and
+treat them specially.  For example, if the literal string
+``'ext://sys.stderr'`` is provided as a value in the configuration,
+then the ``ext://`` will be stripped off and the remainder of the
+value processed using normal import mechanisms.
+
+The handling of such prefixes is done in a way analogous to protocol
+handling: there is a generic mechanism to look for prefixes which
+match the regular expression ``^(?P<prefix>[a-z]+)://(?P<suffix>.*)$``
+whereby, if the ``prefix`` is recognised, the ``suffix`` is processed
+in a prefix-dependent manner and the result of the processing replaces
+the string value.  If the prefix is not recognised, then the string
+value will be left as-is.
+
+
+.. _logging-config-dict-internalobj:
+
+Access to internal objects
+""""""""""""""""""""""""""
+
+As well as external objects, there is sometimes also a need to refer
+to objects in the configuration.  This will be done implicitly by the
+configuration system for things that it knows about.  For example, the
+string value ``'DEBUG'`` for a ``level`` in a logger or handler will
+automatically be converted to the value ``logging.DEBUG``, and the
+``handlers``, ``filters`` and ``formatter`` entries will take an
+object id and resolve to the appropriate destination object.
+
+However, a more generic mechanism is needed for user-defined
+objects which are not known to the :mod:`logging` module.  For
+example, consider :class:`logging.handlers.MemoryHandler`, which takes
+a ``target`` argument which is another handler to delegate to. Since
+the system already knows about this class, then in the configuration,
+the given ``target`` just needs to be the object id of the relevant
+target handler, and the system will resolve to the handler from the
+id.  If, however, a user defines a ``my.package.MyHandler`` which has
+an ``alternate`` handler, the configuration system would not know that
+the ``alternate`` referred to a handler.  To cater for this, a generic
+resolution system allows the user to specify::
+
+    handlers:
+      file:
+        # configuration of file handler goes here
+
+      custom:
+        (): my.package.MyHandler
+        alternate: cfg://handlers.file
+
+The literal string ``'cfg://handlers.file'`` will be resolved in an
+analogous way to strings with the ``ext://`` prefix, but looking
+in the configuration itself rather than the import namespace.  The
+mechanism allows access by dot or by index, in a similar way to
+that provided by ``str.format``.  Thus, given the following snippet::
+
+    handlers:
+      email:
+        class: logging.handlers.SMTPHandler
+        mailhost: localhost
+        fromaddr: my_app@domain.tld
+        toaddrs:
+          - support_team@domain.tld
+          - dev_team@domain.tld
+        subject: Houston, we have a problem.
+
+in the configuration, the string ``'cfg://handlers'`` would resolve to
+the dict with key ``handlers``, the string ``'cfg://handlers.email``
+would resolve to the dict with key ``email`` in the ``handlers`` dict,
+and so on.  The string ``'cfg://handlers.email.toaddrs[1]`` would
+resolve to ``'dev_team.domain.tld'`` and the string
+``'cfg://handlers.email.toaddrs[0]'`` would resolve to the value
+``'support_team@domain.tld'``. The ``subject`` value could be accessed
+using either ``'cfg://handlers.email.subject'`` or, equivalently,
+``'cfg://handlers.email[subject]'``.  The latter form only needs to be
+used if the key contains spaces or non-alphanumeric characters.  If an
+index value consists only of decimal digits, access will be attempted
+using the corresponding integer value, falling back to the string
+value if needed.
+
+Given a string ``cfg://handlers.myhandler.mykey.123``, this will
+resolve to ``config_dict['handlers']['myhandler']['mykey']['123']``.
+If the string is specified as ``cfg://handlers.myhandler.mykey[123]``,
+the system will attempt to retrieve the value from
+``config_dict['handlers']['myhandler']['mykey'][123]``, and fall back
+to ``config_dict['handlers']['myhandler']['mykey']['123']`` if that
+fails.
+
 .. _logging-config-fileformat:
 
 Configuration file format