58 Commits

Author	SHA1	Message	Date
Guido van Rossum	2739cd74b3	Some refinements of the external-link checking code: insert the errors ... in the 'bad' dictionary (sanitize them so they are picklable; the sanitation code is now a subroutine); don't check mailto: URLs; omit colon in Error message.	1997-01-30 04:26:57 +00:00
Guido van Rossum	de66268588	Added -x option to check external links. Slooooow!	1997-01-30 03:58:21 +00:00
Guido van Rossum	325a64f207	Catch I/O errors when parsing robots.txt file. ... Add version number, printed at startup in non-quited mode.	1997-01-30 03:30:20 +00:00
Guido van Rossum	df47bafa1c	Basic README file	1997-01-30 03:24:00 +00:00
Guido van Rossum	3edbb35023	Added robots.txt support, using Skip Montanaro's parser. ... Fixed occasional inclusion of unpicklable objects (Message in errors). Changed indent of a few messages.	1997-01-30 03:19:41 +00:00
Guido van Rossum	bbf8c2fafd	Skip Montanaro's robots.txt parser.	1997-01-30 03:18:23 +00:00
Guido van Rossum	272b37d686	web tree checker	1997-01-30 02:44:48 +00:00
Guido van Rossum	d7e4705d8f	mime types guesser	1997-01-30 02:44:20 +00:00