cpython/Doc/tools/rstlint.py

233 lines
7.4 KiB
Python
Raw Normal View History

#!/usr/bin/env python
# -*- coding: utf-8 -*-
# Check for stylistic and formal issues in .rst and .py
# files included in the documentation.
#
# 01/2009, Georg Brandl
Merged revisions 68633,68648,68667,68706,68718,68720-68721,68724-68727,68739 via svnmerge from svn+ssh://pythondev@svn.python.org/python/trunk ........ r68633 | thomas.heller | 2009-01-16 12:53:44 -0600 (Fri, 16 Jan 2009) | 3 lines Change an example in the docs to avoid a mistake when the code is copy pasted and changed afterwards. ........ r68648 | benjamin.peterson | 2009-01-16 22:28:57 -0600 (Fri, 16 Jan 2009) | 1 line use enumerate ........ r68667 | amaury.forgeotdarc | 2009-01-17 14:18:59 -0600 (Sat, 17 Jan 2009) | 3 lines #4077: No need to append \n when calling Py_FatalError + fix a declaration to make it match the one in pythonrun.h ........ r68706 | benjamin.peterson | 2009-01-17 19:28:46 -0600 (Sat, 17 Jan 2009) | 1 line fix grammar ........ r68718 | georg.brandl | 2009-01-18 04:42:35 -0600 (Sun, 18 Jan 2009) | 1 line #4976: union() and intersection() take multiple args, but talk about "the other". ........ r68720 | georg.brandl | 2009-01-18 04:45:22 -0600 (Sun, 18 Jan 2009) | 1 line #4974: fix redundant mention of lists and tuples. ........ r68721 | georg.brandl | 2009-01-18 04:48:16 -0600 (Sun, 18 Jan 2009) | 1 line #4914: trunc is in math. ........ r68724 | georg.brandl | 2009-01-18 07:24:10 -0600 (Sun, 18 Jan 2009) | 1 line #4979: correct result range for some random functions. ........ r68725 | georg.brandl | 2009-01-18 07:47:26 -0600 (Sun, 18 Jan 2009) | 1 line #4857: fix augmented assignment target spec. ........ r68726 | georg.brandl | 2009-01-18 08:41:52 -0600 (Sun, 18 Jan 2009) | 1 line #4923: clarify what was added. ........ r68727 | georg.brandl | 2009-01-18 12:25:30 -0600 (Sun, 18 Jan 2009) | 1 line #4986: augassigns are not expressions. ........ r68739 | benjamin.peterson | 2009-01-18 15:11:38 -0600 (Sun, 18 Jan 2009) | 1 line fix test that wasn't working as expected #4990 ........
2009-01-18 18:27:04 -04:00
# TODO: - wrong versions in versionadded/changed
# - wrong markup after versionchanged directive
from __future__ import with_statement
import os
import re
import sys
import getopt
import subprocess
from os.path import join, splitext, abspath, exists
from collections import defaultdict
directives = [
# standard docutils ones
'admonition', 'attention', 'caution', 'class', 'compound', 'container',
'contents', 'csv-table', 'danger', 'date', 'default-role', 'epigraph',
'error', 'figure', 'footer', 'header', 'highlights', 'hint', 'image',
'important', 'include', 'line-block', 'list-table', 'meta', 'note',
'parsed-literal', 'pull-quote', 'raw', 'replace',
'restructuredtext-test-directive', 'role', 'rubric', 'sectnum', 'sidebar',
'table', 'target-notes', 'tip', 'title', 'topic', 'unicode', 'warning',
# Sphinx custom ones
'acks', 'attribute', 'autoattribute', 'autoclass', 'autodata',
'autoexception', 'autofunction', 'automethod', 'automodule', 'centered',
'cfunction', 'class', 'classmethod', 'cmacro', 'cmdoption', 'cmember',
'code-block', 'confval', 'cssclass', 'ctype', 'currentmodule', 'cvar',
'data', 'deprecated', 'describe', 'directive', 'doctest', 'envvar', 'event',
'exception', 'function', 'glossary', 'highlight', 'highlightlang', 'index',
'literalinclude', 'method', 'module', 'moduleauthor', 'productionlist',
'program', 'role', 'sectionauthor', 'seealso', 'sourcecode', 'staticmethod',
'tabularcolumns', 'testcode', 'testoutput', 'testsetup', 'toctree', 'todo',
'todolist', 'versionadded', 'versionchanged'
]
all_directives = '(' + '|'.join(directives) + ')'
seems_directive_re = re.compile(r'\.\. %s([^a-z:]|:(?!:))' % all_directives)
default_role_re = re.compile(r'(^| )`\w([^`]*?\w)?`($| )')
leaked_markup_re = re.compile(r'[a-z]::[^=]|:[a-z]+:|`|\.\.\s*\w+:')
checkers = {}
checker_props = {'severity': 1, 'falsepositives': False}
def checker(*suffixes, **kwds):
"""Decorator to register a function as a checker."""
def deco(func):
for suffix in suffixes:
checkers.setdefault(suffix, []).append(func)
for prop in checker_props:
setattr(func, prop, kwds.get(prop, checker_props[prop]))
return func
return deco
@checker('.py', severity=4)
def check_syntax(fn, lines):
"""Check Python examples for valid syntax."""
Merged revisions 68288-68291,68325-68326,68338,68388,68393,68423 via svnmerge from svn+ssh://pythondev@svn.python.org/python/trunk ................ r68288 | benjamin.peterson | 2009-01-03 18:39:07 -0600 (Sat, 03 Jan 2009) | 1 line only check the actual compile() call for a SyntaxError ................ r68289 | georg.brandl | 2009-01-04 02:26:10 -0600 (Sun, 04 Jan 2009) | 2 lines Test commit. ................ r68290 | georg.brandl | 2009-01-04 04:23:49 -0600 (Sun, 04 Jan 2009) | 4 lines Add "suspicious" builder which finds leftover markup in the HTML files. Patch by Gabriel Genellina. ................ r68291 | georg.brandl | 2009-01-04 04:24:09 -0600 (Sun, 04 Jan 2009) | 2 lines Fix two issues found by the suspicious builder. ................ r68325 | benjamin.peterson | 2009-01-04 16:00:18 -0600 (Sun, 04 Jan 2009) | 1 line use Jinja 2.1.1 ................ r68326 | georg.brandl | 2009-01-04 16:03:10 -0600 (Sun, 04 Jan 2009) | 2 lines Update make.bat. ................ r68338 | neal.norwitz | 2009-01-04 21:57:25 -0600 (Sun, 04 Jan 2009) | 1 line Make sure to checkout any new packages ................ r68388 | benjamin.peterson | 2009-01-07 21:39:46 -0600 (Wed, 07 Jan 2009) | 1 line string exceptions are gone ................ r68393 | benjamin.peterson | 2009-01-07 22:01:00 -0600 (Wed, 07 Jan 2009) | 1 line use new sphinx modules ................ r68423 | benjamin.peterson | 2009-01-08 20:13:34 -0600 (Thu, 08 Jan 2009) | 29 lines Merged revisions 68306-68308,68340,68368,68422 via svnmerge from svn+ssh://pythondev@svn.python.org/sandbox/trunk/2to3/lib2to3 ........ r68306 | benjamin.peterson | 2009-01-04 12:27:19 -0600 (Sun, 04 Jan 2009) | 1 line fix_urllib: add mappings for the url parsing functions ........ r68307 | benjamin.peterson | 2009-01-04 12:30:01 -0600 (Sun, 04 Jan 2009) | 1 line remove duplicated function ........ r68308 | benjamin.peterson | 2009-01-04 12:50:34 -0600 (Sun, 04 Jan 2009) | 1 line turtle is no longer renamed ........ r68340 | georg.brandl | 2009-01-05 02:11:39 -0600 (Mon, 05 Jan 2009) | 2 lines Fix undefined locals in parse_tokens(). ........ r68368 | benjamin.peterson | 2009-01-06 17:56:10 -0600 (Tue, 06 Jan 2009) | 1 line fix typo (thanks to Robert Lehmann) ........ r68422 | benjamin.peterson | 2009-01-08 20:01:03 -0600 (Thu, 08 Jan 2009) | 1 line run the imports fixers after fix_import, so fix_import doesn't try to make stdlib renames into relative imports #4876 ........ ................
2009-01-08 23:03:23 -04:00
code = ''.join(lines)
if '\r' in code:
if os.name != 'nt':
yield 0, '\\r in code file'
code = code.replace('\r', '')
try:
compile(code, fn, 'exec')
except SyntaxError as err:
yield err.lineno, 'not compilable: %s' % err
@checker('.rst', severity=2)
def check_suspicious_constructs(fn, lines):
"""Check for suspicious reST constructs."""
inprod = False
for lno, line in enumerate(lines):
if seems_directive_re.match(line):
yield lno+1, 'comment seems to be intended as a directive'
if '.. productionlist::' in line:
inprod = True
elif not inprod and default_role_re.search(line):
yield lno+1, 'default role used'
elif inprod and not line.strip():
inprod = False
@checker('.py', '.rst')
def check_whitespace(fn, lines):
"""Check for whitespace and line length issues."""
for lno, line in enumerate(lines):
if '\r' in line:
yield lno+1, '\\r in line'
if '\t' in line:
yield lno+1, 'OMG TABS!!!1'
if line[:-1].rstrip(' \t') != line[:-1]:
yield lno+1, 'trailing whitespace'
@checker('.rst', severity=0)
def check_line_length(fn, lines):
"""Check for line length; this checker is not run by default."""
for lno, line in enumerate(lines):
if len(line) > 81:
# don't complain about tables, links and function signatures
if line.lstrip()[0] not in '+|' and \
'http://' not in line and \
not line.lstrip().startswith(('.. function',
'.. method',
'.. cfunction')):
yield lno+1, "line too long"
@checker('.html', severity=2, falsepositives=True)
def check_leaked_markup(fn, lines):
"""Check HTML files for leaked reST markup; this only works if
the HTML files have been built.
"""
for lno, line in enumerate(lines):
if leaked_markup_re.search(line):
yield lno+1, 'possibly leaked markup: %r' % line
def main(argv):
usage = '''\
Usage: %s [-v] [-f] [-s sev] [-i path]* [path]
Options: -v verbose (print all checked file names)
-f enable checkers that yield many false positives
-s sev only show problems with severity >= sev
-i path ignore subdir or file path
''' % argv[0]
try:
gopts, args = getopt.getopt(argv[1:], 'vfs:i:')
except getopt.GetoptError:
print(usage)
return 2
verbose = False
severity = 1
ignore = []
falsepos = False
for opt, val in gopts:
if opt == '-v':
verbose = True
elif opt == '-f':
falsepos = True
elif opt == '-s':
severity = int(val)
elif opt == '-i':
ignore.append(abspath(val))
if len(args) == 0:
path = '.'
elif len(args) == 1:
path = args[0]
else:
print(usage)
return 2
if not exists(path):
print('Error: path %s does not exist' % path)
return 2
count = defaultdict(int)
for root, dirs, files in os.walk(path):
# ignore subdirs controlled by svn
if '.svn' in dirs:
dirs.remove('.svn')
# ignore subdirs in ignore list
if abspath(root) in ignore:
del dirs[:]
continue
for fn in files:
fn = join(root, fn)
if fn[:2] == './':
fn = fn[2:]
# ignore files in ignore list
if abspath(fn) in ignore:
continue
ext = splitext(fn)[1]
checkerlist = checkers.get(ext, None)
if not checkerlist:
continue
if verbose:
print('Checking %s...' % fn)
try:
with open(fn, 'r') as f:
lines = list(f)
except (IOError, OSError) as err:
print('%s: cannot open: %s' % (fn, err))
count[4] += 1
continue
for checker in checkerlist:
if checker.falsepositives and not falsepos:
continue
csev = checker.severity
if csev >= severity:
for lno, msg in checker(fn, lines):
Merged revisions 77236,77383,77399,77857,78238,78861-78862,78958 via svnmerge from svn+ssh://svn.python.org/python/branches/py3k ................ r77236 | georg.brandl | 2010-01-02 15:51:12 +0100 (Sa, 02 Jan 2010) | 1 line #7592: remove duplicate description. ................ r77383 | georg.brandl | 2010-01-09 10:48:46 +0100 (Sa, 09 Jan 2010) | 9 lines Merged revisions 77382 via svnmerge from svn+ssh://pythondev@svn.python.org/python/trunk ........ r77382 | georg.brandl | 2010-01-09 10:47:11 +0100 (Sa, 09 Jan 2010) | 1 line #7422: make it clear that getargspec() only works on Python functions. ........ ................ r77399 | georg.brandl | 2010-01-09 23:39:42 +0100 (Sa, 09 Jan 2010) | 1 line Remove redundant brackets in signatures. ................ r77857 | georg.brandl | 2010-01-30 18:54:04 +0100 (Sa, 30 Jan 2010) | 1 line #7814: fix wrong example function usage. ................ r78238 | georg.brandl | 2010-02-19 10:10:15 +0100 (Fr, 19 Feb 2010) | 1 line #5341: fix parenthesis placement. ................ r78861 | georg.brandl | 2010-03-12 11:04:37 +0100 (Fr, 12 Mär 2010) | 1 line Make tool compatible with 2.x and 3.x. ................ r78862 | georg.brandl | 2010-03-12 11:06:40 +0100 (Fr, 12 Mär 2010) | 13 lines Merged revisions 78859-78860 via svnmerge from svn+ssh://pythondev@svn.python.org/python/trunk ........ r78859 | georg.brandl | 2010-03-12 10:57:43 +0100 (Fr, 12 Mär 2010) | 1 line Get rid of backticks. ........ r78860 | georg.brandl | 2010-03-12 11:02:03 +0100 (Fr, 12 Mär 2010) | 1 line Fix warnings from "make check". ........ ................ r78958 | georg.brandl | 2010-03-14 11:51:01 +0100 (So, 14 Mär 2010) | 37 lines Merged revisions 78101,78115,78117,78182,78188,78245,78386,78496 via svnmerge from svn+ssh://pythondev@svn.python.org/python/trunk ........ r78101 | georg.brandl | 2010-02-08 01:04:54 +0100 (Mo, 08 Feb 2010) | 1 line Fix test_fnmatch. ........ r78115 | georg.brandl | 2010-02-08 23:40:51 +0100 (Mo, 08 Feb 2010) | 1 line Fix missing string formatting placeholder. ........ r78117 | georg.brandl | 2010-02-08 23:48:37 +0100 (Mo, 08 Feb 2010) | 1 line Convert test failure from output-producing to self.fail(). ........ r78182 | georg.brandl | 2010-02-14 09:18:23 +0100 (So, 14 Feb 2010) | 1 line #7926: fix stray parens. ........ r78188 | georg.brandl | 2010-02-14 14:38:12 +0100 (So, 14 Feb 2010) | 1 line #7926: fix-up wording. ........ r78245 | georg.brandl | 2010-02-19 20:36:08 +0100 (Fr, 19 Feb 2010) | 1 line #7967: PyXML is no more. ........ r78386 | georg.brandl | 2010-02-23 22:48:57 +0100 (Di, 23 Feb 2010) | 1 line #6544: fix refleak in kqueue, occurring in certain error conditions. ........ r78496 | georg.brandl | 2010-02-27 15:58:08 +0100 (Sa, 27 Feb 2010) | 1 line Link to http://www.python.org/dev/workflow/ from bugs page. ........ ................
2010-10-06 04:55:35 -03:00
print('[%d] %s:%d: %s' % (csev, fn, lno, msg))
count[csev] += 1
if verbose:
print()
if not count:
if severity > 1:
print('No problems with severity >= %d found.' % severity)
else:
print('No problems found.')
else:
for severity in sorted(count):
number = count[severity]
print('%d problem%s with severity %d found.' %
(number, number > 1 and 's' or '', severity))
return int(bool(count))
if __name__ == '__main__':
sys.exit(main(sys.argv))