Issue #20226: Major improvements to Argument Clinic.

* You may now specify an expression as the default value for a
  parameter!  Example: "sys.maxsize - 1".  This support is
  intentionally quite limited; you may only use values that
  can be represented as static C values.
* Removed "doc_default", simplified support for "c_default"
  and "py_default".  (I'm not sure we still even need
  "py_default", but I'm leaving it in for now in case a
  use presents itself.)
* Parameter lines support a trailing '\\' as a line
  continuation character, allowing you to break up long lines.
* The argument parsing code generated when supporting optional
  groups now uses PyTuple_GET_SIZE instead of PyTuple_GetSize,
  leading to a 850% speedup in parsing.  (Just kidding, this
  is an unmeasurable difference.)
* A bugfix for the recent regression where the generated
  prototype from pydoc for builtins would be littered with
  unreadable "=<object ...>"" default values for parameters
  that had no default value.
* Converted some asserts into proper failure messages.
* Many doc improvements and fixes.

Doc/c-api/arg.rst

@@ -294,6 +294,8 @@ Other objects
    the object pointer is stored.  If the Python object does not have the required
    type, :exc:`TypeError` is raised.
 
+.. _o_ampersand:
+
 ``O&`` (object) [*converter*, *anything*]
    Convert a Python object to a C variable through a *converter* function.  This
    takes two arguments: the first is a function, the second is the address of a C

Doc/howto/clinic.rst

@@ -127,6 +127,12 @@ convert a function to work with it.  Let's dive in!
    margin, with no line wider than 80 characters.
    (Argument Clinic will preserve indents inside the docstring.)
 
+   If the old docstring had a first line that looked like a function
+   signature, throw that line away.  (The docstring doesn't need it
+   anymore--when you use ``help()`` on your builtin in the future,
+   the first line will be built automatically based on the function's
+   signature.)
+
    Sample::
 
     /*[clinic input]
@@ -196,6 +202,10 @@ convert a function to work with it.  Let's dive in!
 
        name_of_parameter: converter = default_value
 
+   Argument Clinic's support for "default values" is quite sophisticated;
+   please see :ref:`the section below on default values <default_values>`
+   for more information.
+
    Add a blank line below the parameters.
 
    What's a "converter"?  It establishes both the type
@@ -513,16 +523,6 @@ The base function would now be named ``pickler_dumper()``,
 and the impl function would now be named ``pickler_dumper_impl()``.
 
 
-The NULL default value
-----------------------
-
-For string and object parameters, you can set them to ``None`` to indicate
-that there is no default.  However, that means the C variable will be
-initialized to ``Py_None``.  For convenience's sakes, there's a special
-value called ``NULL`` for just this case: from Python's perspective it
-behaves like a default value of ``None``, but the C variable is initialized
-with ``NULL``.
-
 
 Converting functions using PyArg_UnpackTuple
 --------------------------------------------
@@ -654,35 +654,70 @@ the same converters.
 All arguments to Argument Clinic converters are keyword-only.
 All Argument Clinic converters accept the following arguments:
 
-``py_default``
+  ``c_default``
-  The default value for this parameter when defined in Python.
+    The default value for this parameter when defined in C.
-  Specifically, the value that will be used in the ``inspect.Signature``
+    Specifically, this will be the initializer for the variable declared
-  string.
+    in the "parse function".  See :ref:`the section on default values <default_values>`
-  If a default value is specified for the parameter, defaults to
+    for how to use this.
-  ``repr(default)``, else defaults to ``None``.
+    Specified as a string.
-  Specified as a string.
 
-``c_default``
+  ``annotation``
-  The default value for this parameter when defined in C.
+    The annotation value for this parameter.  Not currently supported,
-  Specifically, this will be the initializer for the variable declared
+    because PEP 8 mandates that the Python library may not use
-  in the "parse function".
+    annotations.
-  Specified as a string.
 
-``required``
+In addition, some converters accept additional arguments.  Here is a list
-  If a parameter takes a default value, Argument Clinic infers that the
+of these arguments, along with their meanings:
-  parameter is optional.  However, you may want a parameter to take a
-  default value in C, but not behave in Python as if the parameter is
-  optional.  Passing in ``required=True`` to a converter tells Argument
-  Clinic that this parameter is not optional, even if it has a default
-  value.
 
-  (The need for ``required`` may be obviated by ``c_default``, which is
+  ``bitwise``
-  newer but arguably a better solution.)
+    Only supported for unsigned integers.  The native integer value of this
+    Python argument will be written to the parameter without any range checking,
+    even for negative values.
 
-``annotation``
+  ``converter``
-  The annotation value for this parameter.  Not currently supported,
+    Only supported by the ``object`` converter.  Specifies the name of a
-  because PEP 8 mandates that the Python library may not use
+    :ref:`C "converter function" <o_ampersand>`
-  annotations.
+    to use to convert this object to a native type.
+
+  ``encoding``
+    Only supported for strings.  Specifies the encoding to use when converting
+    this string from a Python str (Unicode) value into a C ``char *`` value.
+
+  ``length``
+    Only supported for strings.  If true, requests that the length of the
+    string be passed in to the impl function, just after the string parameter,
+    in a parameter named ``<parameter_name>_length``.
+
+  ``nullable``
+    Only supported for strings.  If true, this parameter may also be set to
+    ``None``, in which case the C parameter will be set to ``NULL``.
+
+  ``subclass_of``
+    Only supported for the ``object`` converter.  Requires that the Python
+    value be a subclass of a Python type, as expressed in C.
+
+  ``types``
+    Only supported for the ``object`` (and ``self``) converter.  Specifies
+    the C type that will be used to declare the variable.  Default value is
+    ``"PyObject *"``.
+
+  ``types``
+    A string containing a list of Python types (and possibly pseudo-types);
+    this restricts the allowable Python argument to values of these types.
+    (This is not a general-purpose facility; as a rule it only supports
+    specific lists of types as shown in the legacy converter table.)
+
+  ``zeroes``
+    Only supported for strings.  If true, embedded NUL bytes (``'\\0'``) are
+    permitted inside the value.
+
+Please note, not every possible combination of arguments will work.
+Often these arguments are implemented internally by specific ``PyArg_ParseTuple``
+*format units*, with specific behavior.  For example, currently you cannot
+call ``str`` and pass in ``zeroes=True`` without also specifying an ``encoding``;
+although it's perfectly reasonable to think this would work, these semantics don't
+map to any existing format unit.  So Argument Clinic doesn't support it.  (Or, at
+least, not yet.)
 
 Below is a table showing the mapping of legacy converters into real
 Argument Clinic converters.  On the left is the legacy converter,
@@ -720,9 +755,9 @@ on the right is the text you'd replace it with.
 ``'u'``     ``Py_UNICODE``
 ``'U'``     ``unicode``
 ``'w*'``    ``Py_buffer(types='bytearray rwbuffer')``
-``'y#'``    ``str(type='bytes', length=True)``
+``'y#'``    ``str(types='bytes', length=True)``
 ``'Y'``     ``PyByteArrayObject``
-``'y'``     ``str(type='bytes')``
+``'y'``     ``str(types='bytes')``
 ``'y*'``    ``Py_buffer``
 ``'Z#'``    ``Py_UNICODE(nullable=True, length=True)``
 ``'z#'``    ``str(nullable=True, length=True)``
@@ -789,6 +824,90 @@ This restriction doesn't seem unreasonable; CPython itself always passes in stat
 hard-coded encoding strings for parameters whose format units start with ``e``.
 
 
+.. _default_values:
+
+Parameter default values
+------------------------
+
+Default values for parameters can be any of a number of values.
+At their simplest, they can be string, int, or float literals::
+
+    foo: str = "abc"
+    bar: int = 123
+    bat: float = 45.6
+
+They can also use any of Python's built-in constants::
+
+    yep:  bool = True
+    nope: bool = False
+    nada: object = None
+
+There's also special support for a default value of ``NULL``, and
+for simple expressions, documented in the following sections.
+
+
+The ``NULL`` default value
+--------------------------
+
+For string and object parameters, you can set them to ``None`` to indicate
+that there's no default.  However, that means the C variable will be
+initialized to ``Py_None``.  For convenience's sakes, there's a special
+value called ``NULL`` for just this reason: from Python's perspective it
+behaves like a default value of ``None``, but the C variable is initialized
+with ``NULL``.
+
+Expressions specified as default values
+---------------------------------------
+
+The default value for a parameter can be more than just a literal value.
+It can be an entire expression, using math operators and looking up attributes
+on objects.  However, this support isn't exactly simple, because of some
+non-obvious semantics.
+
+Consider the following example::
+
+    foo: Py_ssize_t = sys.maxsize - 1
+
+``sys.maxsize`` can have different values on different platforms.  Therefore
+Argument Clinic can't simply evaluate that expression locally and hard-code it
+in C.  So it stores the default in such a way that it will get evaluated at
+runtime, when the user asks for the function's signature.
+
+What namespace is available when the expression is evaluated?  It's evaluated
+in the context of the module the builtin came from.  So, if your module has an
+attribute called "``max_widgets``", you may simply use it::
+
+    foo: Py_ssize_t = max_widgets
+
+If the symbol isn't found in the current module, it fails over to looking in
+``sys.modules``.  That's how it can find ``sys.maxsize`` for example.  (Since you
+don't know in advance what modules the user will load into their interpreter,
+it's best to restrict yourself to modules that are preloaded by Python itself.)
+
+Evaluating default values only at runtime means Argument Clinic can't compute
+the correct equivalent C default value.  So you need to tell it explicitly.
+When you use an expression, you must also specify the equivalent expression
+in C, using the ``c_default`` parameter to the converter::
+
+    foo: Py_ssize_t(c_default="PY_SSIZE_T_MAX - 1") = sys.maxsize - 1
+
+Another complication: Argument Clinic can't know in advance whether or not the
+expression you supply is valid.  It parses it to make sure it looks legal, but
+it can't *actually* know.  You must be very careful when using expressions to
+specify values that are guaranteed to be valid at runtime!
+
+Finally, because expressions must be representable as static C values, there
+are many restrictions on legal expressions.  Here's a list of Python features
+you're not permitted to use:
+
+* Function calls.
+* Inline if statements (``3 if foo else 5``).
+* Automatic sequence unpacking (``*[1, 2, 3]``).
+* List/set/dict comprehensions and generator expressions.
+* Tuple/list/set/dict literals.
+
+
+
 Using a return converter
 ------------------------
 
@@ -1096,7 +1215,73 @@ any arguments.
 You can still use a self converter, a return converter, and specify
 a ``type`` argument to the object converter for ``METH_O``.
 
-Using Argument Clinic in Python files
+The #ifdef trick
+----------------------------------------------
+
+If you're converting a function that isn't available on all platforms,
+there's a trick you can use to make life a little easier.  The existing
+code probably looks like this::
+
+    #ifdef HAVE_FUNCTIONNAME
+    static module_functionname(...)
+    {
+    ...
+    }
+    #endif /* HAVE_FUNCTIONNAME */
+
+And then in the ``PyMethodDef`` structure at the bottom you'll have::
+
+    #ifdef HAVE_FUNCTIONNAME
+    {'functionname', ... },
+    #endif /* HAVE_FUNCTIONNAME */
+
+In this scenario, you should change the code to look like the following::
+
+    #ifdef HAVE_FUNCTIONNAME
+    /*[clinic input]
+    module.functionname
+    ...
+    [clinic start generated code]*/
+    static module_functionname(...)
+    {
+    ...
+    }
+    #endif /* HAVE_FUNCTIONNAME */
+
+Run Argument Clinic on the code in this state, then refresh the file in
+your editor.  Now you'll have the generated code, including the #define
+for the ``PyMethodDef``, like so::
+
+    #ifdef HAVE_FUNCTIONNAME
+    /*[clinic input]
+    ...
+    [clinic start generated code]*/
+    ...
+    #define MODULE_FUNCTIONNAME \
+        {'functionname', ... },
+    ...
+    /*[clinic end generated code: checksum=...]*/
+    static module_functionname(...)
+    {
+    ...
+    }
+    #endif /* HAVE_FUNCTIONNAME */
+
+Change the #endif at the bottom as follows::
+
+    #else
+    #define MODULE_FUNCTIONNAME
+    #endif /* HAVE_FUNCTIONNAME */
+
+Now you can remove the #ifdefs around the ``PyMethodDef`` structure
+at the end, and replace those three lines with ``MODULE_FUNCTIONNAME``.
+If the function is available, the macro turns into the ``PyMethodDef``
+static value, including the trailing comma; if the function isn't
+available, the macro turns into nothing.  Perfect!
+
+(This is the preferred approach for optional functions; in the future,
+Argument Clinic may generate the entire ``PyMethodDef`` structure.)
+
 -------------------------------------
 
 It's actually possible to use Argument Clinic to preprocess Python files.

Lib/inspect.py

@@ -1954,6 +1954,8 @@ class Signature:
         if not s:
             return None
 
+        Parameter = cls._parameter_cls
+
         if s.endswith("/)"):
             kind = Parameter.POSITIONAL_ONLY
             s = s[:-2] + ')'
@@ -1969,55 +1971,74 @@ class Signature:
         if not isinstance(module, ast.Module):
             return None
 
-        # ast.FunctionDef
         f = module.body[0]
 
         parameters = []
         empty = Parameter.empty
         invalid = object()
 
-        def parse_attribute(node):
+        module = None
-            if not isinstance(node.ctx, ast.Load):
+        module_dict = {}
-                return None
+        module_name = getattr(func, '__module__', None)
+        if module_name:
+            module = sys.modules.get(module_name, None)
+            if module:
+                module_dict = module.__dict__
+        sys_module_dict = sys.modules
 
-            value = node.value
+        def parse_name(node):
-            o = parse_node(value)
+            assert isinstance(node, ast.arg)
-            if o is invalid:
+            if node.annotation != None:
-                return invalid
+                raise ValueError("Annotations are not currently supported")
+            return node.arg
 
-            if isinstance(value, ast.Name):
+        def wrap_value(s):
-                name = o
+            try:
-                if name not in sys.modules:
+                value = eval(s, module_dict)
-                    return invalid
+            except NameError:
-                o = sys.modules[name]
+                try:
+                    value = eval(s, sys_module_dict)
+                except NameError:
+                    raise RuntimeError()
 
-            return getattr(o, node.attr, invalid)
+            if isinstance(value, str):
+                return ast.Str(value)
+            if isinstance(value, (int, float)):
+                return ast.Num(value)
+            if isinstance(value, bytes):
+                return ast.Bytes(value)
+            if value in (True, False, None):
+                return ast.NameConstant(value)
+            raise RuntimeError()
 
-        def parse_node(node):
+        class RewriteSymbolics(ast.NodeTransformer):
-            if isinstance(node, ast.arg):
+            def visit_Attribute(self, node):
-                if node.annotation != None:
+                a = []
-                    raise ValueError("Annotations are not currently supported")
+                n = node
-                return node.arg
+                while isinstance(n, ast.Attribute):
-            if isinstance(node, ast.Num):
+                    a.append(n.attr)
-                return node.n
+                    n = n.value
-            if isinstance(node, ast.Str):
+                if not isinstance(n, ast.Name):
-                return node.s
+                    raise RuntimeError()
-            if isinstance(node, ast.NameConstant):
+                a.append(n.id)
-                return node.value
+                value = ".".join(reversed(a))
-            if isinstance(node, ast.Attribute):
+                return wrap_value(value)
-                return parse_attribute(node)
+
-            if isinstance(node, ast.Name):
+            def visit_Name(self, node):
                 if not isinstance(node.ctx, ast.Load):
-                    return invalid
+                    raise ValueError()
-                return node.id
+                return wrap_value(node.id)
-            return invalid
 
         def p(name_node, default_node, default=empty):
-            name = parse_node(name_node)
+            name = parse_name(name_node)
             if name is invalid:
                 return None
             if default_node:
-                o = parse_node(default_node)
+                try:
+                    default_node = RewriteSymbolics().visit(default_node)
+                    o = ast.literal_eval(default_node)
+                except ValueError:
+                    o = invalid
                 if o is invalid:
                     return None
                 default = o if o is not invalid else default

Lib/test/test_inspect.py

@@ -1601,12 +1601,15 @@ class TestSignatureObject(unittest.TestCase):
         self.assertTrue(isinstance(signature, inspect.Signature))
         def p(name): return signature.parameters[name].default
         self.assertEqual(p('s'), 'avocado')
+        self.assertEqual(p('b'), b'bytes')
         self.assertEqual(p('d'), 3.14)
         self.assertEqual(p('i'), 35)
-        self.assertEqual(p('c'), sys.maxsize)
         self.assertEqual(p('n'), None)
         self.assertEqual(p('t'), True)
         self.assertEqual(p('f'), False)
+        self.assertEqual(p('local'), 3)
+        self.assertEqual(p('sys'), sys.maxsize)
+        self.assertEqual(p('exp'), sys.maxsize - 1)
 
     def test_signature_on_non_function(self):
         with self.assertRaisesRegex(TypeError, 'is not a callable object'):

Misc/NEWS

@@ -88,6 +88,9 @@ Tests
 Tools/Demos
 -----------
 
+- Issue #20226: Argument Clinic now permits simple expressions
+  (e.g. "sys.maxsize - 1") as default values for parameters.
+
 - Issue #19936: Added executable bits or shebang lines to Python scripts which
   requires them.  Disable executable bits and shebang lines in test and
   benchmark files in order to prevent using a random system python, and in

Modules/_cursesmodule.c

@@ -618,7 +618,7 @@ curses_window_addch(PyObject *self, PyObject *args)
     int group_right_1 = 0;
     long attr = 0;
 
-    switch (PyTuple_Size(args)) {
+    switch (PyTuple_GET_SIZE(args)) {
         case 1:
             if (!PyArg_ParseTuple(args, "O:addch", &ch))
                 return NULL;
@@ -650,7 +650,7 @@ curses_window_addch(PyObject *self, PyObject *args)
 
 static PyObject *
 curses_window_addch_impl(PyObject *self, int group_left_1, int x, int y, PyObject *ch, int group_right_1, long attr)
-/*[clinic end generated code: checksum=44ed958b891cde91205e584c766e048f3999714f]*/
+/*[clinic end generated code: checksum=b073327add8197b6ba7fb96c87062422c8312954]*/
 {
     PyCursesWindowObject *cwself = (PyCursesWindowObject *)self;
     int coordinates_group = group_left_1;

Modules/_dbmmodule.c

@@ -297,7 +297,7 @@ dbm_dbm_get(PyObject *self, PyObject *args)
     int group_right_1 = 0;
     PyObject *default_value = NULL;
 
-    switch (PyTuple_Size(args)) {
+    switch (PyTuple_GET_SIZE(args)) {
         case 1:
             if (!PyArg_ParseTuple(args, "s#:get", &key, &key_length))
                 return NULL;
@@ -318,7 +318,7 @@ dbm_dbm_get(PyObject *self, PyObject *args)
 
 static PyObject *
 dbm_dbm_get_impl(dbmobject *dp, const char *key, Py_ssize_clean_t key_length, int group_right_1, PyObject *default_value)
-/*[clinic end generated code: checksum=28cf8928811bde51e535d67ae98ea039d79df717]*/
+/*[clinic end generated code: checksum=2c3209571267017f1b9abbd19e1b521849fd5d4a]*/
 {
     datum dbm_key, val;
 
@@ -450,7 +450,7 @@ dbm.open as dbmopen
     flags: str="r"
         How to open the file.  "r" for reading, "w" for writing, etc.
 
-    mode: int(doc_default="0o666") = 0o666
+    mode: int(py_default="0o666") = 0o666
         If creating a new file, the mode bits for the new file
         (e.g. os.O_RDWR).
 

Modules/_opcode.c

@@ -39,7 +39,7 @@ _opcode_stack_effect(PyModuleDef *module, PyObject *args)
     int oparg = 0;
     int _return_value;
 
-    switch (PyTuple_Size(args)) {
+    switch (PyTuple_GET_SIZE(args)) {
         case 1:
             if (!PyArg_ParseTuple(args, "i:stack_effect", &opcode))
                 return NULL;
@@ -64,7 +64,7 @@ exit:
 
 static int
 _opcode_stack_effect_impl(PyModuleDef *module, int opcode, int group_right_1, int oparg)
-/*[clinic end generated code: checksum=e880e62dc7b0de73403026eaf4f8074aa106358b]*/
+/*[clinic end generated code: checksum=47e76ec27523da4ab192713642d32482cd743aa4]*/
 {
     int effect;
     if (HAS_ARG(opcode)) {

Modules/_testcapimodule.c

@@ -2870,7 +2870,7 @@ PyDoc_STRVAR(docstring_with_signature_and_extra_newlines,
 );
 
 PyDoc_STRVAR(docstring_with_signature_with_defaults,
-"docstring_with_signature_with_defaults(s='avocado', d=3.14, i=35, c=sys.maxsize, n=None, t=True, f=False)\n"
+"docstring_with_signature_with_defaults(s='avocado', b=b'bytes', d=3.14, i=35, n=None, t=True, f=False, local=the_number_three, sys=sys.maxsize, exp=sys.maxsize - 1)\n"
 "\n"
 "\n"
 "\n"
@@ -3317,6 +3317,8 @@ PyInit__testcapi(void)
     Py_INCREF(&PyInstanceMethod_Type);
     PyModule_AddObject(m, "instancemethod", (PyObject *)&PyInstanceMethod_Type);
 
+    PyModule_AddIntConstant(m, "the_number_three", 3);
+
     TestError = PyErr_NewException("_testcapi.error", NULL, NULL);
     Py_INCREF(TestError);
     PyModule_AddObject(m, "error", TestError);

Modules/posixmodule.c

@@ -2401,7 +2401,7 @@ class dir_fd_converter(CConverter):
 
 /*[clinic input]
 
-os.stat -> object(doc_default='stat_result')
+os.stat
 
     path : path_t(allow_fd=True)
         Path to be examined; can be string, bytes, or open-file-descriptor int.
@@ -2523,7 +2523,7 @@ posix_lstat(PyObject *self, PyObject *args, PyObject *kwargs)
     #define OS_ACCESS_DIR_FD_CONVERTER dir_fd_unavailable
 #endif
 /*[clinic input]
-os.access -> object(doc_default='True if granted, False otherwise')
+os.access
 
     path: path_t(allow_fd=True)
         Path to be tested; can be string, bytes, or open-file-descriptor int.

Modules/zlibmodule.c

@@ -202,7 +202,7 @@ zlib_compress(PyModuleDef *module, PyObject *args)
     int group_right_1 = 0;
     int level = 0;
 
-    switch (PyTuple_Size(args)) {
+    switch (PyTuple_GET_SIZE(args)) {
         case 1:
             if (!PyArg_ParseTuple(args, "y*:compress", &bytes))
                 return NULL;
@@ -227,7 +227,7 @@ zlib_compress(PyModuleDef *module, PyObject *args)
 
 static PyObject *
 zlib_compress_impl(PyModuleDef *module, Py_buffer *bytes, int group_right_1, int level)
-/*[clinic end generated code: checksum=2c59af563a4595c5ecea4011701f482ae350aa5f]*/
+/*[clinic end generated code: checksum=66c4d16d0b8b9dd423648d9ef00d6a89d3363665]*/
 {
     PyObject *ReturnVal = NULL;
     Byte *input, *output = NULL;

Tools/clinic/clinic.py

@@ -55,6 +55,13 @@ class Null:
 NULL = Null()
 
 
+class Unknown:
+    def __repr__(self):
+        return '<Unknown>'
+
+unknown = Unknown()
+
+
 def _text_accumulator():
     text = []
     def output():
@@ -197,7 +204,7 @@ def version_splitter(s):
     accumulator = []
     def flush():
         if not accumulator:
-            raise ValueError('Malformed version string: ' + repr(s))
+            raise ValueError('Unsupported version string: ' + repr(s))
         version.append(int(''.join(accumulator)))
         accumulator.clear()
 
@@ -596,7 +603,7 @@ static {impl_return_type}
         count_min = sys.maxsize
         count_max = -1
 
-        add("switch (PyTuple_Size(args)) {{\n")
+        add("switch (PyTuple_GET_SIZE(args)) {{\n")
         for subset in permute_optional_groups(left, required, right):
             count = len(subset)
             count_min = min(count_min, count)
@@ -1069,6 +1076,7 @@ class Clinic:
         self.filename = filename
         self.modules = collections.OrderedDict()
         self.classes = collections.OrderedDict()
+        self.functions = []
 
         global clinic
         clinic = self
@@ -1343,29 +1351,7 @@ class Parameter:
     def is_keyword_only(self):
         return self.kind == inspect.Parameter.KEYWORD_ONLY
 
-py_special_values = {
-    NULL: "None",
-}
 
-def py_repr(o):
-    special = py_special_values.get(o)
-    if special:
-        return special
-    return repr(o)
-
-
-c_special_values = {
-    NULL: "NULL",
-    None: "Py_None",
-}
-
-def c_repr(o):
-    special = c_special_values.get(o)
-    if special:
-        return special
-    if isinstance(o, str):
-        return '"' + quoted_for_c_string(o) + '"'
-    return repr(o)
 
 def add_c_converter(f, name=None):
     if not name:
@@ -1407,8 +1393,7 @@ class CConverter(metaclass=CConverterAutoRegister):
     """
     For the init function, self, name, function, and default
     must be keyword-or-positional parameters.  All other
-    parameters (including "required" and "doc_default")
+    parameters must be keyword-only.
-    must be keyword-only.
     """
 
     # The C type to use for this variable.
@@ -1418,23 +1403,23 @@ class CConverter(metaclass=CConverterAutoRegister):
 
     # The Python default value for this parameter, as a Python value.
     # Or the magic value "unspecified" if there is no default.
+    # Or the magic value "unknown" if this value is a cannot be evaluated
+    # at Argument-Clinic-preprocessing time (but is presumed to be valid
+    # at runtime).
     default = unspecified
 
     # If not None, default must be isinstance() of this type.
     # (You can also specify a tuple of types.)
     default_type = None
 
-    # "default" as it should appear in the documentation, as a string.
-    # Or None if there is no default.
-    doc_default = None
-
-    # "default" converted into a str for rendering into Python code.
-    py_default = None
-
     # "default" converted into a C value, as a string.
     # Or None if there is no default.
     c_default = None
 
+    # "default" converted into a Python value, as a string.
+    # Or None if there is no default.
+    py_default = None
+
     # The default value used to initialize the C variable when
     # there is no default, but not specifying a default may
     # result in an "uninitialized variable" warning.  This can
@@ -1485,12 +1470,12 @@ class CConverter(metaclass=CConverterAutoRegister):
     # Only used by format units ending with '#'.
     length = False
 
-    def __init__(self, name, function, default=unspecified, *, doc_default=None, c_default=None, py_default=None, required=False, annotation=unspecified, **kwargs):
+    def __init__(self, name, function, default=unspecified, *, c_default=None, py_default=None, annotation=unspecified, **kwargs):
         self.function = function
         self.name = name
 
         if default is not unspecified:
-            if self.default_type and not isinstance(default, self.default_type):
+            if self.default_type and not isinstance(default, (self.default_type, Unknown)):
                 if isinstance(self.default_type, type):
                     types_str = self.default_type.__name__
                 else:
@@ -1498,23 +1483,19 @@ class CConverter(metaclass=CConverterAutoRegister):
                 fail("{}: default value {!r} for field {} is not of type {}".format(
                     self.__class__.__name__, default, name, types_str))
             self.default = default
-            self.py_default = py_default if py_default is not None else py_repr(default)
+
-            self.doc_default = doc_default if doc_default is not None else self.py_default
+        self.c_default = c_default
-            self.c_default = c_default if c_default is not None else c_repr(default)
+        self.py_default = py_default
-        else:
+
-            self.py_default = py_default
-            self.doc_default = doc_default
-            self.c_default = c_default
         if annotation != unspecified:
             fail("The 'annotation' parameter is not currently permitted.")
-        self.required = required
         self.converter_init(**kwargs)
 
     def converter_init(self):
         pass
 
     def is_optional(self):
-        return (self.default is not unspecified) and (not self.required)
+        return (self.default is not unspecified)
 
     def render(self, parameter, data):
         """
@@ -1655,8 +1636,9 @@ class bool_converter(CConverter):
     c_ignored_default = '0'
 
     def converter_init(self):
-        self.default = bool(self.default)
+        if self.default is not unspecified:
-        self.c_default = str(int(self.default))
+            self.default = bool(self.default)
+            self.c_default = str(int(self.default))
 
 class char_converter(CConverter):
     type = 'char'
@@ -1665,7 +1647,7 @@ class char_converter(CConverter):
     c_ignored_default = "'\0'"
 
     def converter_init(self):
-        if len(self.default) != 1:
+        if isinstance(self.default, str) and (len(self.default) != 1):
             fail("char_converter: illegal default value " + repr(self.default))
 
 
@@ -1797,8 +1779,8 @@ class object_converter(CConverter):
 
 
 @add_legacy_c_converter('s#', length=True)
-@add_legacy_c_converter('y', type="bytes")
+@add_legacy_c_converter('y', types="bytes")
-@add_legacy_c_converter('y#', type="bytes", length=True)
+@add_legacy_c_converter('y#', types="bytes", length=True)
 @add_legacy_c_converter('z', nullable=True)
 @add_legacy_c_converter('z#', nullable=True, length=True)
 class str_converter(CConverter):
@@ -1993,8 +1975,8 @@ class CReturnConverter(metaclass=CReturnConverterAutoRegister):
     # Or the magic value "unspecified" if there is no default.
     default = None
 
-    def __init__(self, *, doc_default=None, **kwargs):
+    def __init__(self, *, py_default=None, **kwargs):
-        self.doc_default = doc_default
+        self.py_default = py_default
         try:
             self.return_converter_init(**kwargs)
         except TypeError as e:
@@ -2212,6 +2194,7 @@ class DSLParser:
         self.indent = IndentStack()
         self.kind = CALLABLE
         self.coexist = False
+        self.parameter_continuation = ''
 
     def directive_version(self, required):
         global version
@@ -2244,15 +2227,18 @@ class DSLParser:
         self.block.signatures.append(c)
 
     def at_classmethod(self):
-        assert self.kind is CALLABLE
+        if self.kind is not CALLABLE:
+            fail("Can't set @classmethod, function is not a normal callable")
         self.kind = CLASS_METHOD
 
     def at_staticmethod(self):
-        assert self.kind is CALLABLE
+        if self.kind is not CALLABLE:
+            fail("Can't set @staticmethod, function is not a normal callable")
         self.kind = STATIC_METHOD
 
     def at_coexist(self):
-        assert self.coexist == False
+        if self.coexist:
+            fail("Called @coexist twice!")
         self.coexist = True
 
 
@@ -2503,6 +2489,7 @@ class DSLParser:
         if not self.indent.infer(line):
             return self.next(self.state_function_docstring, line)
 
+        self.parameter_continuation = ''
         return self.next(self.state_parameter, line)
 
 
@@ -2516,6 +2503,10 @@ class DSLParser:
                 p.group = -p.group
 
     def state_parameter(self, line):
+        if self.parameter_continuation:
+            line = self.parameter_continuation + ' ' + line.lstrip()
+            self.parameter_continuation = ''
+
         if self.ignore_line(line):
             return
 
@@ -2529,6 +2520,11 @@ class DSLParser:
             # we indented, must be to new parameter docstring column
             return self.next(self.state_parameter_docstring_start, line)
 
+        line = line.rstrip()
+        if line.endswith('\\'):
+            self.parameter_continuation = line[:-1]
+            return
+
         line = line.lstrip()
 
         if line in ('*', '/', '[', ']'):
@@ -2547,48 +2543,123 @@ class DSLParser:
         else:
             fail("Function " + self.function.name + " has an unsupported group configuration. (Unexpected state " + str(self.parameter_state) + ")")
 
-        ast_input = "def x({}): pass".format(line)
+        base, equals, default = line.rpartition('=')
+        if not equals:
+            base = default
+            default = None
         module = None
         try:
+            ast_input = "def x({}): pass".format(base)
             module = ast.parse(ast_input)
         except SyntaxError:
-            pass
+            try:
+                default = None
+                ast_input = "def x({}): pass".format(line)
+                module = ast.parse(ast_input)
+            except SyntaxError:
+                pass
         if not module:
             fail("Function " + self.function.name + " has an invalid parameter declaration:\n\t" + line)
 
         function_args = module.body[0].args
         parameter = function_args.args[0]
 
-        py_default = None
-
         parameter_name = parameter.arg
         name, legacy, kwargs = self.parse_converter(parameter.annotation)
 
-        if function_args.defaults:
+        if not default:
-            expr = function_args.defaults[0]
+            value = unspecified
-            # mild hack: explicitly support NULL as a default value
+            if 'py_default' in kwargs:
-            if isinstance(expr, ast.Name) and expr.id == 'NULL':
+                fail("You can't specify py_default without specifying a default value!")
-                value = NULL
+        else:
-            elif isinstance(expr, ast.Attribute):
+            default = default.strip()
+            ast_input = "x = {}".format(default)
+            try:
+                module = ast.parse(ast_input)
+
+                # blacklist of disallowed ast nodes
+                class DetectBadNodes(ast.NodeVisitor):
+                    bad = False
+                    def bad_node(self, node):
+                        self.bad = True
+
+                    # inline function call
+                    visit_Call = bad_node
+                    # inline if statement ("x = 3 if y else z")
+                    visit_IfExp = bad_node
+
+                    # comprehensions and generator expressions
+                    visit_ListComp = visit_SetComp = bad_node
+                    visit_DictComp = visit_GeneratorExp = bad_node
+
+                    # literals for advanced types
+                    visit_Dict = visit_Set = bad_node
+                    visit_List = visit_Tuple = bad_node
+
+                    # "starred": "a = [1, 2, 3]; *a"
+                    visit_Starred = bad_node
+
+                    # allow ellipsis, for now
+                    # visit_Ellipsis = bad_node
+
+                blacklist = DetectBadNodes()
+                blacklist.visit(module)
+                if blacklist.bad:
+                    fail("Unsupported expression as default value: " + repr(default))
+
+                expr = module.body[0].value
+                # mild hack: explicitly support NULL as a default value
+                if isinstance(expr, ast.Name) and expr.id == 'NULL':
+                    value = NULL
+                    py_default = 'None'
+                    c_default = "NULL"
+                elif (isinstance(expr, ast.BinOp) or
+                    (isinstance(expr, ast.UnaryOp) and not isinstance(expr.operand, ast.Num))):
+                    c_default = kwargs.get("c_default")
+                    if not (isinstance(c_default, str) and c_default):
+                        fail("When you specify an expression (" + repr(default) + ") as your default value,\nyou MUST specify a valid c_default.")
+                    py_default = default
+                    value = unknown
+                elif isinstance(expr, ast.Attribute):
+                    a = []
+                    n = expr
+                    while isinstance(n, ast.Attribute):
+                        a.append(n.attr)
+                        n = n.value
+                    if not isinstance(n, ast.Name):
+                        fail("Unsupported default value " + repr(default) + " (looked like a Python constant)")
+                    a.append(n.id)
+                    py_default = ".".join(reversed(a))
+
+                    c_default = kwargs.get("c_default")
+                    if not (isinstance(c_default, str) and c_default):
+                        fail("When you specify a named constant (" + repr(py_default) + ") as your default value,\nyou MUST specify a valid c_default.")
+
+                    try:
+                        value = eval(py_default)
+                    except NameError:
+                        value = unknown
+                else:
+                    value = ast.literal_eval(expr)
+                    py_default = repr(value)
+                    if isinstance(value, (bool, None.__class__)):
+                        c_default = "Py_" + py_default
+                    elif isinstance(value, str):
+                        c_default = '"' + quoted_for_c_string(value) + '"'
+                    else:
+                        c_default = py_default
+
+            except SyntaxError as e:
+                fail("Syntax error: " + repr(e.text))
+            except (ValueError, AttributeError):
+                value = unknown
                 c_default = kwargs.get("c_default")
+                py_default = default
                 if not (isinstance(c_default, str) and c_default):
                     fail("When you specify a named constant (" + repr(py_default) + ") as your default value,\nyou MUST specify a valid c_default.")
 
-                a = []
+            kwargs.setdefault('c_default', c_default)
-                n = expr
+            kwargs.setdefault('py_default', py_default)
-                while isinstance(n, ast.Attribute):
-                    a.append(n.attr)
-                    n = n.value
-                if not isinstance(n, ast.Name):
-                    fail("Malformed default value (looked like a Python constant)")
-                a.append(n.id)
-                py_default = ".".join(reversed(a))
-                kwargs["py_default"] = py_default
-                value = eval(py_default)
-            else:
-                value = ast.literal_eval(expr)
-        else:
-            value = unspecified
 
         dict = legacy_converters if legacy else converters
         legacy_str = "legacy " if legacy else ""
@@ -2777,7 +2848,7 @@ class DSLParser:
             if p.converter.is_optional():
                 a.append('=')
                 value = p.converter.default
-                a.append(p.converter.doc_default)
+                a.append(p.converter.py_default)
             s = fix_right_bracket_count(p.right_bracket_count)
             s += "".join(a)
             if add_comma:
@@ -2788,9 +2859,18 @@ class DSLParser:
         add(fix_right_bracket_count(0))
         add(')')
 
-        # if f.return_converter.doc_default:
+        # PEP 8 says:
+        #
+        #     The Python standard library will not use function annotations
+        #     as that would result in a premature commitment to a particular
+        #     annotation style. Instead, the annotations are left for users
+        #     to discover and experiment with useful annotation styles.
+        #
+        # therefore this is commented out:
+        #
+        # if f.return_converter.py_default:
         #     add(' -> ')
-        #     add(f.return_converter.doc_default)
+        #     add(f.return_converter.py_default)
 
         docstring_first_line = output()
 
@@ -2998,8 +3078,8 @@ def main(argv):
 
                 # print("   ", short_name + "".join(parameters))
             print()
-        print("All converters also accept (doc_default=None, required=False, annotation=None).")
+        print("All converters also accept (c_default=None, py_default=None, annotation=None).")
-        print("All return converters also accept (doc_default=None).")
+        print("All return converters also accept (py_default=None).")
         sys.exit(0)
 
     if ns.make:

Tools/clinic/clinic_test.py

@@ -231,20 +231,20 @@ xyz
         self._test_clinic("""
     verbatim text here
     lah dee dah
-/*[copy]
+/*[copy input]
 def
-[copy]*/
+[copy start generated code]*/
 abc
-/*[copy checksum: 03cfd743661f07975fa2f1220c5194cbaff48451]*/
+/*[copy end generated code: checksum=03cfd743661f07975fa2f1220c5194cbaff48451]*/
 xyz
 """, """
     verbatim text here
     lah dee dah
-/*[copy]
+/*[copy input]
 def
-[copy]*/
+[copy start generated code]*/
 def
-/*[copy checksum: 7b18d017f89f61cf17d47f92749ea6930a3f1deb]*/
+/*[copy end generated code: checksum=7b18d017f89f61cf17d47f92749ea6930a3f1deb]*/
 xyz
 """)
 
@@ -292,17 +292,6 @@ os.access
         p = function.parameters['path']
         self.assertEqual(1, p.converter.args['allow_fd'])
 
-    def test_param_docstring(self):
-        function = self.parse_function("""
-module os
-os.stat as os_stat_fn -> object(doc_default='stat_result')
-
-   path: str
-       Path to be examined""")
-        p = function.parameters['path']
-        self.assertEqual("Path to be examined", p.docstring)
-        self.assertEqual(function.return_converter.doc_default, 'stat_result')
-
     def test_function_docstring(self):
         function = self.parse_function("""
 module os