Merge with 3.3
This commit is contained in:
commit
b0dc5c3bc5
|
@ -19,10 +19,10 @@ Web site that published the :file:`robots.txt` file. For more details on the
|
||||||
structure of :file:`robots.txt` files, see http://www.robotstxt.org/orig.html.
|
structure of :file:`robots.txt` files, see http://www.robotstxt.org/orig.html.
|
||||||
|
|
||||||
|
|
||||||
.. class:: RobotFileParser()
|
.. class:: RobotFileParser(url='')
|
||||||
|
|
||||||
This class provides a set of methods to read, parse and answer questions
|
This class provides methods to read, parse and answer questions about the
|
||||||
about a single :file:`robots.txt` file.
|
:file:`robots.txt` file at *url*.
|
||||||
|
|
||||||
.. method:: set_url(url)
|
.. method:: set_url(url)
|
||||||
|
|
||||||
|
|
Loading…
Reference in New Issue