73fd46d24e
The solution is to convert bytes to text via utf-8. I'm not entirely sure if this is safe, but it looks like robots.txt is expected to be ascii. |
||
---|---|---|
.. | ||
__init__.py | ||
error.py | ||
parse.py | ||
request.py | ||
response.py | ||
robotparser.py |