Issue #9360: Cleanup and improvements to the nntplib module. The API

now conforms to the philosophy of bytes and unicode separation in Python 3.
A test suite has also been added.
This commit is contained in:
Antoine Pitrou 2010-09-29 15:03:40 +00:00
parent 926f0da582
commit 69ab95105f
4 changed files with 2042 additions and 469 deletions

View File

@ -11,100 +11,99 @@
single: Network News Transfer Protocol
This module defines the class :class:`NNTP` which implements the client side of
the NNTP protocol. It can be used to implement a news reader or poster, or
automated news processors. For more information on NNTP (Network News Transfer
Protocol), see Internet :rfc:`977`.
the Network News Transfer Protocol. It can be used to implement a news reader
or poster, or automated news processors. It is compatible with :rfc:`3977`
as well as the older :rfc:`977` and :rfc:`2980`.
Here are two small examples of how it can be used. To list some statistics
about a newsgroup and print the subjects of the last 10 articles::
>>> s = NNTP('news.gmane.org')
>>> s = nntplib.NNTP('news.gmane.org')
>>> resp, count, first, last, name = s.group('gmane.comp.python.committers')
>>> print('Group', name, 'has', count, 'articles, range', first, 'to', last)
Group gmane.comp.python.committers has 1071 articles, range 1 to 1071
>>> resp, subs = s.xhdr('subject', first + '-' + last)
>>> for id, sub in subs[-10:]: print(id, sub)
Group gmane.comp.python.committers has 1096 articles, range 1 to 1096
>>> resp, overviews = s.over((last - 9, last))
>>> for id, over in overviews:
... print(id, nntplib.decode_header(over['subject']))
...
1062 Re: Mercurial Status?
1063 Re: [python-committers] (Windows) buildbots on 3.x
1064 Re: Mercurial Status?
1065 Re: Mercurial Status?
1066 Python 2.6.6 status
1067 Commit Privileges for Ask Solem
1068 Re: Commit Privileges for Ask Solem
1069 Re: Commit Privileges for Ask Solem
1070 Re: Commit Privileges for Ask Solem
1071 2.6.6 rc 2
1087 Re: Commit privileges for Łukasz Langa
1088 Re: 3.2 alpha 2 freeze
1089 Re: 3.2 alpha 2 freeze
1090 Re: Commit privileges for Łukasz Langa
1091 Re: Commit privileges for Łukasz Langa
1092 Updated ssh key
1093 Re: Updated ssh key
1094 Re: Updated ssh key
1095 Hello fellow committers!
1096 Re: Hello fellow committers!
>>> s.quit()
'205 Bye!'
To post an article from a file (this assumes that the article has valid
To post an article from a binary file (this assumes that the article has valid
headers, and that you have right to post on the particular newsgroup)::
>>> s = NNTP('news.gmane.org')
>>> f = open('/tmp/article')
>>> s = nntplib.NNTP('news.gmane.org')
>>> f = open('/tmp/article.txt', 'rb')
>>> s.post(f)
'240 Article posted successfully.'
>>> s.quit()
'205 Bye!'
The module itself defines the following items:
The module itself defines the following classes:
.. class:: NNTP(host[, port [, user[, password [, readermode][, usenetrc]]]])
.. class:: NNTP(host, port=119, user=None, password=None, readermode=None, usenetrc=True, [timeout])
Return a new instance of the :class:`NNTP` class, representing a connection
to the NNTP server running on host *host*, listening at port *port*. The
default *port* is 119. If the optional *user* and *password* are provided,
or if suitable credentials are present in :file:`/.netrc` and the optional
flag *usenetrc* is true (the default), the ``AUTHINFO USER`` and ``AUTHINFO
PASS`` commands are used to identify and authenticate the user to the server.
If the optional flag *readermode* is true, then a ``mode reader`` command is
sent before authentication is performed. Reader mode is sometimes necessary
if you are connecting to an NNTP server on the local machine and intend to
call reader-specific commands, such as ``group``. If you get unexpected
to the NNTP server running on host *host*, listening at port *port*.
An optional *timeout* can be specified for the socket connection.
If the optional *user* and *password* are provided, or if suitable
credentials are present in :file:`/.netrc` and the optional flag *usenetrc*
is true (the default), the ``AUTHINFO USER`` and ``AUTHINFO PASS`` commands
are used to identify and authenticate the user to the server. If the optional
flag *readermode* is true, then a ``mode reader`` command is sent before
authentication is performed. Reader mode is sometimes necessary if you are
connecting to an NNTP server on the local machine and intend to call
reader-specific commands, such as ``group``. If you get unexpected
:exc:`NNTPPermanentError`\ s, you might need to set *readermode*.
*readermode* defaults to ``None``. *usenetrc* defaults to ``True``.
.. exception:: NNTPError
Derived from the standard exception :exc:`Exception`, this is the base class for
all exceptions raised by the :mod:`nntplib` module.
Derived from the standard exception :exc:`Exception`, this is the base
class for all exceptions raised by the :mod:`nntplib` module. Instances
of this class have the following attribute:
.. attribute:: response
The response of the server if available, as a :class:`str` object.
.. exception:: NNTPReplyError
Exception raised when an unexpected reply is received from the server. For
backwards compatibility, the exception ``error_reply`` is equivalent to this
class.
Exception raised when an unexpected reply is received from the server.
.. exception:: NNTPTemporaryError
Exception raised when an error code in the range 400--499 is received. For
backwards compatibility, the exception ``error_temp`` is equivalent to this
class.
Exception raised when a response code in the range 400--499 is received.
.. exception:: NNTPPermanentError
Exception raised when an error code in the range 500--599 is received. For
backwards compatibility, the exception ``error_perm`` is equivalent to this
class.
Exception raised when a response code in the range 500--599 is received.
.. exception:: NNTPProtocolError
Exception raised when a reply is received from the server that does not begin
with a digit in the range 1--5. For backwards compatibility, the exception
``error_proto`` is equivalent to this class.
with a digit in the range 1--5.
.. exception:: NNTPDataError
Exception raised when there is some error in the response data. For backwards
compatibility, the exception ``error_data`` is equivalent to this class.
Exception raised when there is some error in the response data.
.. _nntp-objects:
@ -112,10 +111,29 @@ The module itself defines the following items:
NNTP Objects
------------
NNTP instances have the following methods. The *response* that is returned as
the first item in the return tuple of almost all methods is the server's
response: a string beginning with a three-digit code. If the server's response
indicates an error, the method raises one of the above exceptions.
:class:`NNTP` instances have the following methods. The *response* that is
returned as the first item in the return tuple of almost all methods is the
server's response: a string beginning with a three-digit code. If the server's
response indicates an error, the method raises one of the above exceptions.
.. note::
Many of the following methods take an optional keyword-only argument *file*.
When the *file* argument is supplied, it must be either a :term:`file object`
opened for binary writing, or the name of an on-disk file to be written to.
The method will then write any data returned by the server (except for the
response line and the terminating dot) to the file; any list of lines,
tuples or objects that the method normally returns will be empty.
.. versionchanged:: 3.2
Many of the following methods have been reworked and fixed, which makes
them incompatible with their 3.1 counterparts.
.. method:: NNTP.quit()
Send a ``QUIT`` command and close the connection. Once this method has been
called, no other methods of the NNTP object should be called.
.. method:: NNTP.getwelcome()
@ -125,62 +143,70 @@ indicates an error, the method raises one of the above exceptions.
that may be relevant to the user.)
.. method:: NNTP.set_debuglevel(level)
.. method:: NNTP.getcapabilities()
Set the instance's debugging level. This controls the amount of debugging
output printed. The default, ``0``, produces no debugging output. A value of
``1`` produces a moderate amount of debugging output, generally a single line
per request or response. A value of ``2`` or higher produces the maximum amount
of debugging output, logging each line sent and received on the connection
(including message text).
Return the :rfc:`3977` capabilities advertised by the server, as a
:class:`dict` instance mapping capability names to (possibly empty) lists
of values. On legacy servers which don't understand the ``CAPABILITIES``
command, an empty dictionary is returned instead.
>>> s = NNTP('news.gmane.org')
>>> 'POST' in s.getcapabilities()
True
.. versionadded:: 3.2
.. method:: NNTP.newgroups(date, time, [file])
.. method:: NNTP.newgroups(date, *, file=None)
Send a ``NEWGROUPS`` command. The *date* argument should be a string of the
form ``'yymmdd'`` indicating the date, and *time* should be a string of the form
``'hhmmss'`` indicating the time. Return a pair ``(response, groups)`` where
*groups* is a list of group names that are new since the given date and time. If
the *file* parameter is supplied, then the output of the ``NEWGROUPS`` command
is stored in a file. If *file* is a string, then the method will open a file
object with that name, write to it then close it. If *file* is a :term:`file
object`, then it will start calling :meth:`write` on it to store the lines of
the command output. If *file* is supplied, then the returned *list* is an empty list.
Send a ``NEWGROUPS`` command. The *date* argument should be a
:class:`datetime.date` or :class:`datetime.datetime` object.
Return a pair ``(response, groups)`` where *groups* is a list representing
the groups that are new since the given *date*. If *file* is supplied,
though, then *groups* will be empty.
>>> from datetime import date, timedelta
>>> resp, groups = s.newgroups(date.today() - timedelta(days=3))
>>> len(groups)
85
>>> groups[0]
GroupInfo(group='gmane.network.tor.devel', last='4', first='1', flag='m')
.. method:: NNTP.newnews(group, date, time, [file])
.. method:: NNTP.newnews(group, date, *, file=None)
Send a ``NEWNEWS`` command. Here, *group* is a group name or ``'*'``, and
*date* and *time* have the same meaning as for :meth:`newgroups`. Return a pair
``(response, articles)`` where *articles* is a list of message ids. If the
*file* parameter is supplied, then the output of the ``NEWNEWS`` command is
stored in a file. If *file* is a string, then the method will open a file
object with that name, write to it then close it. If *file* is a :term:`file
object`, then it will start calling :meth:`write` on it to store the lines of the
command output. If *file* is supplied, then the returned *list* is an empty list.
*date* has the same meaning as for :meth:`newgroups`. Return a pair
``(response, articles)`` where *articles* is a list of message ids.
This command is frequently disabled by NNTP server administrators.
.. method:: NNTP.list([file])
.. method:: NNTP.list(*, file=None)
Send a ``LIST`` command. Return a pair ``(response, list)`` where *list* is a
list of tuples. Each tuple has the form ``(group, last, first, flag)``, where
list of tuples representing all the groups available from this NNTP server.
Each tuple has the form ``(group, last, first, flag)``, where
*group* is a group name, *last* and *first* are the last and first article
numbers (as strings), and *flag* is ``'y'`` if posting is allowed, ``'n'`` if
not, and ``'m'`` if the newsgroup is moderated. (Note the ordering: *last*,
*first*.) If the *file* parameter is supplied, then the output of the ``LIST``
command is stored in a file. If *file* is a string, then the method will open
a file with that name, write to it then close it. If *file* is a :term:`file
object`, then it will start calling :meth:`write` on it to store the lines of
the command output. If *file* is supplied, then the returned *list* is an empty
list.
numbers, and *flag* is ``'y'`` if posting is allowed, ``'n'`` if not,
and ``'m'`` if the newsgroup is moderated. (Note the ordering: *last*, *first*.)
This command will often return very large results. It is best to cache the
results offline unless you really need to refresh them.
.. method:: NNTP.descriptions(grouppattern)
Send a ``LIST NEWSGROUPS`` command, where *grouppattern* is a wildmat string as
specified in RFC2980 (it's essentially the same as DOS or UNIX shell wildcard
strings). Return a pair ``(response, list)``, where *list* is a list of tuples
containing ``(name, title)``.
specified in :rfc:`3977` (it's essentially the same as DOS or UNIX shell wildcard
strings). Return a pair ``(response, descriptions)``, where *descriptions*
is a dictionary mapping group names to textual descriptions.
>>> resp, descs = s.descriptions('gmane.comp.python.*')
>>> len(descs)
295
>>> descs.popitem()
('gmane.comp.python.bio.general', 'BioPython discussion list (Moderated)')
.. method:: NNTP.description(group)
@ -195,30 +221,73 @@ indicates an error, the method raises one of the above exceptions.
.. method:: NNTP.group(name)
Send a ``GROUP`` command, where *name* is the group name. Return a tuple
``(response, count, first, last, name)`` where *count* is the (estimated) number
of articles in the group, *first* is the first article number in the group,
*last* is the last article number in the group, and *name* is the group name.
The numbers are returned as strings.
Send a ``GROUP`` command, where *name* is the group name. The group is
selected as the current group, if it exists. Return a tuple
``(response, count, first, last, name)`` where *count* is the (estimated)
number of articles in the group, *first* is the first article number in
the group, *last* is the last article number in the group, and *name*
is the group name.
.. method:: NNTP.help([file])
.. method:: NNTP.over(message_spec, *, file=None)
Send a ``OVER`` command, or a ``XOVER`` command on legacy servers.
*message_spec* can be either a string representing a message id, or
a ``(first, last)`` tuple of numbers indicating a range of articles in
the current group, or a ``(first, None)`` tuple indicating a range of
articles starting from *first* to the last article in the current group,
or :const:`None` to select the current article in the current group.
Return a pair ``(response, overviews)``. *overviews* is a list of
``(article_number, overview)`` tuples, one for each article selected
by *message_spec*. Each *overview* is a dictionary with the same number
of items, but this number depends on the server. These items are either
message headers (the key is then the lower-cased header name) or metadata
items (the key is then the metadata name prepended with ``":"``). The
following items are guaranteed to be present by the NNTP specification:
* the ``subject``, ``from``, ``date``, ``message-id`` and ``references``
headers
* the ``:bytes`` metadata: the number of bytes in the entire raw article
(including headers and body)
* the ``:lines`` metadata: the number of lines in the article body
It is advisable to use the :func:`decode_header` function on header
values when they may contain non-ASCII characters::
>>> _, _, first, last, _ = s.group('gmane.comp.python.devel')
>>> resp, overviews = s.over((last, last))
>>> art_num, over = overviews[0]
>>> art_num
117216
>>> list(over.keys())
['xref', 'from', ':lines', ':bytes', 'references', 'date', 'message-id', 'subject']
>>> over['from']
'=?UTF-8?B?Ik1hcnRpbiB2LiBMw7Z3aXMi?= <martin@v.loewis.de>'
>>> nntplib.decode_header(over['from'])
'"Martin v. Löwis" <martin@v.loewis.de>'
.. versionadded:: 3.2
.. method:: NNTP.help(*, file=None)
Send a ``HELP`` command. Return a pair ``(response, list)`` where *list* is a
list of help strings. If the *file* parameter is supplied, then the output of
the ``HELP`` command is stored in a file. If *file* is a string, then the
method will open a file with that name, write to it then close it. If *file*
is a :term:`file object`, then it will start calling :meth:`write` on it to store
the lines of the command output. If *file* is supplied, then the returned *list*
is an empty list.
list of help strings.
.. method:: NNTP.stat(id)
.. method:: NNTP.stat(message_spec=None)
Send a ``STAT`` command, where *id* is the message id (enclosed in ``'<'`` and
``'>'``) or an article number (as a string). Return a triple ``(response,
number, id)`` where *number* is the article number (as a string) and *id* is the
message id (enclosed in ``'<'`` and ``'>'``).
Send a ``STAT`` command, where *message_spec* is either a message id
(enclosed in ``'<'`` and ``'>'``) or an article number in the current group.
If *message_spec* is omitted or :const:`None`, the current article in the
current group is considered. Return a triple ``(response, number, id)``
where *number* is the article number and *id* is the message id.
>>> _, _, first, last, _ = s.group('gmane.comp.python.devel')
>>> resp, number, message_id = s.stat(first)
>>> number, message_id
(9099, '<20030112190404.GE29873@epoch.metaslash.com>')
.. method:: NNTP.next()
@ -231,28 +300,69 @@ indicates an error, the method raises one of the above exceptions.
Send a ``LAST`` command. Return as for :meth:`stat`.
.. method:: NNTP.head(id)
.. method:: NNTP.article(message_spec=None, *, file=None)
Send a ``HEAD`` command, where *id* has the same meaning as for :meth:`stat`.
Return a tuple ``(response, number, id, list)`` where the first three are the
same as for :meth:`stat`, and *list* is a list of the article's headers (an
uninterpreted list of lines, without trailing newlines).
Send an ``ARTICLE`` command, where *message_spec* has the same meaning as
for :meth:`stat`. Return a tuple ``(response, info)`` where *info*
is a :class:`~collections.namedtuple` with three members *number*,
*message_id* and *lines* (in that order). *number* is the article number
in the group (or 0 if the information is not available), *message_id* the
message id as a string, and *lines* a list of lines (without terminating
newlines) comprising the raw message including headers and body.
>>> resp, info = s.article('<20030112190404.GE29873@epoch.metaslash.com>')
>>> info.number
0
>>> info.message_id
'<20030112190404.GE29873@epoch.metaslash.com>'
>>> len(info.lines)
65
>>> info.lines[0]
b'Path: main.gmane.org!not-for-mail'
>>> info.lines[1]
b'From: Neal Norwitz <neal@metaslash.com>'
>>> info.lines[-3:]
[b'There is a patch for 2.3 as well as 2.2.', b'', b'Neal']
.. method:: NNTP.body(id,[file])
.. method:: NNTP.head(message_spec=None, *, file=None)
Send a ``BODY`` command, where *id* has the same meaning as for :meth:`stat`.
If the *file* parameter is supplied, then the body is stored in a file. If
*file* is a string, then the method will open a file with that name, write
to it then close it. If *file* is a :term:`file object`, then it will start
calling :meth:`write` on it to store the lines of the body. Return as for
:meth:`head`. If *file* is supplied, then the returned *list* is an empty list.
Same as :meth:`article()`, but sends a ``HEAD`` command. The *lines*
returned (or written to *file*) will only contain the message headers, not
the body.
.. method:: NNTP.article(id)
.. method:: NNTP.body(message_spec=None, *, file=None)
Send an ``ARTICLE`` command, where *id* has the same meaning as for
:meth:`stat`. Return as for :meth:`head`.
Same as :meth:`article()`, but sends a ``BODY`` command. The *lines*
returned (or written to *file*) will only contain the message body, not the
headers.
.. method:: NNTP.post(data)
Post an article using the ``POST`` command. The *data* argument is either
a :term:`file object` opened for binary reading, or any iterable of bytes
objects (representing raw lines of the article to be posted). It should
represent a well-formed news article, including the required headers. The
:meth:`post` method automatically escapes lines beginning with ``.`` and
appends the termination line.
If the method succeeds, the server's response is returned. If the server
refuses posting, a :class:`NNTPReplyError` is raised.
.. method:: NNTP.ihave(message_id, data)
Send an ``IHAVE`` command. *message_id* is the id of the message to send
to the server (enclosed in ``'<'`` and ``'>'``). The *data* parameter
and the return value are the same as for :meth:`post()`.
.. method:: NNTP.date()
Return a pair ``(response, date)``. *date* is a :class:`~datetime.datetime`
object containing the current date and time of the server.
.. method:: NNTP.slave()
@ -260,10 +370,23 @@ indicates an error, the method raises one of the above exceptions.
Send a ``SLAVE`` command. Return the server's *response*.
.. method:: NNTP.xhdr(header, string, [file])
.. method:: NNTP.set_debuglevel(level)
Send an ``XHDR`` command. This command is not defined in the RFC but is a
common extension. The *header* argument is a header keyword, e.g.
Set the instance's debugging level. This controls the amount of debugging
output printed. The default, ``0``, produces no debugging output. A value of
``1`` produces a moderate amount of debugging output, generally a single line
per request or response. A value of ``2`` or higher produces the maximum amount
of debugging output, logging each line sent and received on the connection
(including message text).
The following are optional NNTP extensions defined in :rfc:`2980`. Some of
them have been superseded by newer commands in :rfc:`3977`.
.. method:: NNTP.xhdr(header, string, *, file=None)
Send an ``XHDR`` command. The *header* argument is a header keyword, e.g.
``'subject'``. The *string* argument should have the form ``'first-last'``
where *first* and *last* are the first and last article numbers to search.
Return a pair ``(response, list)``, where *list* is a list of pairs ``(id,
@ -276,66 +399,55 @@ indicates an error, the method raises one of the above exceptions.
returned *list* is an empty list.
.. method:: NNTP.post(file)
.. method:: NNTP.xover(start, end, *, file=None)
Post an article using the ``POST`` command. The *file* argument is an open file
object which is read until EOF using its :meth:`readline` method. It should be
a well-formed news article, including the required headers. The :meth:`post`
method automatically escapes lines beginning with ``.``.
.. method:: NNTP.ihave(id, file)
Send an ``IHAVE`` command. *id* is a message id (enclosed in ``'<'`` and
``'>'``). If the response is not an error, treat *file* exactly as for the
:meth:`post` method.
.. method:: NNTP.date()
Return a triple ``(response, date, time)``, containing the current date and time
in a form suitable for the :meth:`newnews` and :meth:`newgroups` methods. This
is an optional NNTP extension, and may not be supported by all servers.
.. method:: NNTP.xgtitle(name, [file])
Process an ``XGTITLE`` command, returning a pair ``(response, list)``, where
*list* is a list of tuples containing ``(name, title)``. If the *file* parameter
is supplied, then the output of the ``XGTITLE`` command is stored in a file.
If *file* is a string, then the method will open a file with that name, write
to it then close it. If *file* is a :term:`file object`, then it will start
calling :meth:`write` on it to store the lines of the command output. If *file*
is supplied, then the returned *list* is an empty list. This is an optional NNTP
extension, and may not be supported by all servers.
RFC2980 says "It is suggested that this extension be deprecated". Use
:meth:`descriptions` or :meth:`description` instead.
.. method:: NNTP.xover(start, end, [file])
Return a pair ``(resp, list)``. *list* is a list of tuples, one for each
article in the range delimited by the *start* and *end* article numbers. Each
tuple is of the form ``(article number, subject, poster, date, id, references,
size, lines)``. If the *file* parameter is supplied, then the output of the
``XOVER`` command is stored in a file. If *file* is a string, then the method
will open a file with that name, write to it then close it. If *file* is a
:term:`file object`, then it will start calling :meth:`write` on it to store the
lines of the command output. If *file* is supplied, then the returned *list* is
an empty list. This is an optional NNTP extension, and may not be supported by
all servers.
Send an ``XOVER`` command. *start* and *end* are article numbers
delimiting the range of articles to select. The return value is the
same of for :meth:`over()`. It is recommended to use :meth:`over()`
instead, since it will automatically use the newer ``OVER`` command
if available.
.. method:: NNTP.xpath(id)
Return a pair ``(resp, path)``, where *path* is the directory path to the
article with message ID *id*. This is an optional NNTP extension, and may not
be supported by all servers.
article with message ID *id*. Most of the time, this extension is not
enabled by NNTP server administrators.
.. method:: NNTP.quit()
.. XXX deprecated:
Send a ``QUIT`` command and close the connection. Once this method has been
called, no other methods of the NNTP object should be called.
.. method:: NNTP.xgtitle(name, *, file=None)
Process an ``XGTITLE`` command, returning a pair ``(response, list)``, where
*list* is a list of tuples containing ``(name, title)``. If the *file* parameter
is supplied, then the output of the ``XGTITLE`` command is stored in a file.
If *file* is a string, then the method will open a file with that name, write
to it then close it. If *file* is a :term:`file object`, then it will start
calling :meth:`write` on it to store the lines of the command output. If *file*
is supplied, then the returned *list* is an empty list. This is an optional NNTP
extension, and may not be supported by all servers.
RFC2980 says "It is suggested that this extension be deprecated". Use
:meth:`descriptions` or :meth:`description` instead.
Utility functions
-----------------
The module also defines the following utility function:
.. function:: decode_header(header_str)
Decode a header value, un-escaping any escaped non-ASCII characters.
*header_str* must be a :class:`str` object. The unescaped value is
returned. Using this function is recommended to display some headers
in a human readable form::
>>> decode_header("Some subject")
'Some subject'
>>> decode_header("=?ISO-8859-15?Q?D=E9buter_en_Python?=")
'Débuter en Python'
>>> decode_header("Re: =?UTF-8?B?cHJvYmzDqG1lIGRlIG1hdHJpY2U=?=")
'Re: problème de matrice'

File diff suppressed because it is too large Load Diff

1091
Lib/test/test_nntplib.py Normal file

File diff suppressed because it is too large Load Diff

View File

@ -76,6 +76,10 @@ Core and Builtins
Library
-------
- Issue #9360: Cleanup and improvements to the nntplib module. The API
now conforms to the philosophy of bytes and unicode separation in Python 3.
A test suite has also been added.
- Issue #9962: GzipFile now has the peek() method.
- Issue #9090: When a socket with a timeout fails with EWOULDBLOCK or EAGAIN,