c59a5d449f
Set proper User-agent header (Python-webchecker/<version>).
...
When -x is combined with -q, still do the checking, but don't print
the error in this phase -- they are reported by report_errors().
1997-01-30 06:04:00 +00:00
2739cd74b3
Some refinements of the external-link checking code: insert the errors
...
in the 'bad' dictionary (sanitize them so they are picklable; the
sanitation code is now a subroutine); don't check mailto: URLs; omit
colon in Error message.
1997-01-30 04:26:57 +00:00
de66268588
Added -x option to check external links. Slooooow!
1997-01-30 03:58:21 +00:00
325a64f207
Catch I/O errors when parsing robots.txt file.
...
Add version number, printed at startup in non-quited mode.
1997-01-30 03:30:20 +00:00
df47bafa1c
Basic README file
1997-01-30 03:24:00 +00:00
3edbb35023
Added robots.txt support, using Skip Montanaro's parser.
...
Fixed occasional inclusion of unpicklable objects (Message in errors).
Changed indent of a few messages.
1997-01-30 03:19:41 +00:00
bbf8c2fafd
Skip Montanaro's robots.txt parser.
1997-01-30 03:18:23 +00:00
272b37d686
web tree checker
1997-01-30 02:44:48 +00:00
d7e4705d8f
mime types guesser
1997-01-30 02:44:20 +00:00
Guido van Rossum