Guido van Rossum
|
325a64f207
|
Catch I/O errors when parsing robots.txt file.
Add version number, printed at startup in non-quited mode.
|
1997-01-30 03:30:20 +00:00 |
|
Guido van Rossum
|
df47bafa1c
|
Basic README file
|
1997-01-30 03:24:00 +00:00 |
|
Guido van Rossum
|
3edbb35023
|
Added robots.txt support, using Skip Montanaro's parser.
Fixed occasional inclusion of unpicklable objects (Message in errors).
Changed indent of a few messages.
|
1997-01-30 03:19:41 +00:00 |
|
Guido van Rossum
|
bbf8c2fafd
|
Skip Montanaro's robots.txt parser.
|
1997-01-30 03:18:23 +00:00 |
|
Guido van Rossum
|
272b37d686
|
web tree checker
|
1997-01-30 02:44:48 +00:00 |
|
Guido van Rossum
|
d7e4705d8f
|
mime types guesser
|
1997-01-30 02:44:20 +00:00 |