traverseda
/
cpython
1
0
Fork 0
Code Issues Pull Requests Releases Wiki Activity

Wrap some long lines.

This commit is contained in:
Skip Montanaro 2008-04-28 03:25:37 +00:00
parent 141534e56f
commit dfd982715b
1 changed files with 12 additions and 9 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

21
Doc/library/robotparser.rst
View File

@ -3,7 +3,8 @@
============================================= =============================================
.. module:: robotparser .. module:: robotparser
:synopsis: Loads a robots.txt file and answers questions about fetchability of other URLs. :synopsis: Loads a robots.txt file and answers questions about
fetchability of other URLs.
.. sectionauthor:: Skip Montanaro <skip@pobox.com> .. sectionauthor:: Skip Montanaro <skip@pobox.com>
@ -21,8 +22,8 @@ structure of :file:`robots.txt` files, see http://www.robotstxt.org/orig.html.
.. class:: RobotFileParser() .. class:: RobotFileParser()
This class provides a set of methods to read, parse and answer questions about a This class provides a set of methods to read, parse and answer questions
single :file:`robots.txt` file. about a single :file:`robots.txt` file.
.. method:: set_url(url) .. method:: set_url(url)
@ -42,20 +43,22 @@ structure of :file:`robots.txt` files, see http://www.robotstxt.org/orig.html.
.. method:: can_fetch(useragent, url) .. method:: can_fetch(useragent, url)
Returns ``True`` if the *useragent* is allowed to fetch the *url* according to Returns ``True`` if the *useragent* is allowed to fetch the *url*
the rules contained in the parsed :file:`robots.txt` file. according to the rules contained in the parsed :file:`robots.txt`
file.
.. method:: mtime() .. method:: mtime()
Returns the time the ``robots.txt`` file was last fetched. This is useful for Returns the time the ``robots.txt`` file was last fetched. This is
long-running web spiders that need to check for new ``robots.txt`` files useful for long-running web spiders that need to check for new
periodically. ``robots.txt`` files periodically.
.. method:: modified() .. method:: modified()
Sets the time the ``robots.txt`` file was last fetched to the current time. Sets the time the ``robots.txt`` file was last fetched to the current
time.
The following example demonstrates basic use of the RobotFileParser class. :: The following example demonstrates basic use of the RobotFileParser class. ::