traverseda
/
cpython
1
0
Fork 0
Code Issues Pull Requests Releases Wiki Activity
30,672 Commits 9 Branches 456 Tags 1.7 GiB
Commit Graph

20 Commits

Author SHA1 Message Date
Martin v. Löwis 31bd529f53 Patch #1014237: Consistently return booleans throughout. 2004-08-23 20:42:35 +00:00
Raymond Hettinger bac788a3cd Replace str.find()!=1 with the more readable "in" operator. 2004-05-04 09:21:43 +00:00
Raymond Hettinger 2d95f1ad57 SF patch #911431: robot.txt must be robots.txt 
(Contributed by George Yoshida.)
 2004-03-13 20:27:23 +00:00
Guido van Rossum 68468eba63 Get rid of many apply() calls. 2003-02-27 20:14:51 +00:00
Neal Norwitz 5aee504ccb Remove import of re, it is not used 2002-05-31 14:14:06 +00:00
Raymond Hettinger aef22fb9cd Patch 560023 adding docstrings. 2.2 Candidate (after verifying modules were not updated after 2.2). 2002-05-29 16:18:42 +00:00
Tim Peters bc0e910826 Convert a pile of obvious "yes/no" functions to return bool. 2002-04-04 22:55:58 +00:00
Martin v. Löwis 73f570ba08 Correctly set default entry in all cases. 2002-03-18 10:43:18 +00:00
Martin v. Löwis d22368ffb3 Patch #499513: use readline() instead of readlines(). Removed the 
unnecessary redirection limit code which is already in FancyURLopener.
 2002-03-18 10:41:20 +00:00
Martin v. Löwis 1c63f6e489 Correct various errors: 
- Use substring search, not re search for user-agent and paths.
- Consider * entry last. Unquote, then requote URLs.
- Treat empty Disallow as "allow everything".
Add test cases. Fixes #523041
 2002-02-28 15:24:47 +00:00
Andrew M. Kuchling e7abf97903 Remove unused import (PyChecker) 2001-08-13 14:43:43 +00:00
Tim Peters 0e6d213177 Whitespace normalization. 2001-02-15 23:56:39 +00:00
Skip Montanaro 5bba231d1e The bulk of the credit for these changes goes to Bastian Kleineidam 
* restores urllib as the file fetcher (closes bug #132000)
* allows checking URLs with empty paths (closes patches #103511 and 103721)
* properly handle user agents with versions (e.g., SpamMeister/1.5)
* added several more tests
 2001-02-12 20:58:30 +00:00
Eric S. Raymond 141971f22a String method conversion. 2001-02-09 08:40:40 +00:00
Tim Peters dfc538acae Whitespace normalization. 2001-01-21 04:49:16 +00:00
Skip Montanaro e99d5ea25b added __all__ lists to a number of Python modules 
added test script and expected output file as well
this closes patch 103297.
__all__ attributes will be added to other modules without first submitting
a patch, just adding the necessary line to the test script to verify
more-or-less correct implementation.
 2001-01-20 19:54:20 +00:00
Skip Montanaro 663f6c2ad2 rewrite of robotparser.py by Bastian Kleineidam. Closes patch 102229. 2001-01-20 15:59:25 +00:00
Guido van Rossum dc8b7980e0 Skip Montanaro: 
The robotparser.py module currently lives in Tools/webchecker.  In
preparation for its migration to Lib, I made the following changes:

    * renamed the test() function _test
    * corrected the URLs in _test() so they refer to actual documents
    * added an "if __name__ == '__main__'" catcher to invoke _test()
      when run as a main program
    * added doc strings for the two main methods, parse and can_fetch
    * replaced usage of regsub and regex with corresponding re code
 2000-03-27 19:29:31 +00:00
Guido van Rossum 986abac1ba Give in to tabnanny 1998-04-06 14:29:28 +00:00
Guido van Rossum bbf8c2fafd Skip Montanaro's robots.txt parser. 1997-01-30 03:18:23 +00:00