Commit Graph

54 Commits

Author SHA1 Message Date
Guido van Rossum 3edbb35023 Added robots.txt support, using Skip Montanaro's parser.
Fixed occasional inclusion of unpicklable objects (Message in errors).
Changed indent of a few messages.
1997-01-30 03:19:41 +00:00
Guido van Rossum bbf8c2fafd Skip Montanaro's robots.txt parser. 1997-01-30 03:18:23 +00:00
Guido van Rossum 272b37d686 web tree checker 1997-01-30 02:44:48 +00:00
Guido van Rossum d7e4705d8f mime types guesser 1997-01-30 02:44:20 +00:00