This mimics get_param's error handling for the most part. It is slightly
better in some regards as get_param can produce some really weird results for
duplicate *0* parts. It departs from get_param slightly in that if we have a
mix of non-extended and extended pieces for the same parameter name, the new
parser assumes they were all supposed to be extended and concatenates all the
values, whereas get_param always picks the non-extended parameter value. All
of this error recovery is pretty much arbitrary decisions...
It is unlikely anyone is using the fact that the dictionary returned
by the 'params' attribute was previously writable, but even if someone
is the API is provisional so this kind of change is acceptable (and
needed, to get the API "right" before it becomes official).
Patch by Stéphane Wirtel.
Since EmailMessage is a provisional API we can fix API bugs in a
maintenance release, but I used a trick suggested by Serhiy to
maintain backward compatibility with 3.4.0/1.
Adds `load_package_tests` function to test.support, uses it in test_asyncio,
test_email, test_json, test_tools, test_importlib and all test_importlib
sub-packages to implement test discovery.
More tests of the concrete policy methods would probably be
a good idea, but this is a start, and it brings line coverage
up to 100% for the policy module.
Patch by Milan Oberkirch.
This is more RFC compliant (see issue) and fixes a problem with
signature verifiers rejecting the part when signed. There is some
amount of backward compatibility concern here since it changes
the output, but the RFC issue coupled with fixing the problem
with signature verifiers seems worth the small risk of breaking
code that depends on the current incorrect output.
This applies only to the new parser. The old parser decodes encoded words
inside quoted strings already, although it gets the whitespace wrong
when it does so.
This version of the patch only handles the most common case (a single encoded
word surrounded by quotes), but I haven't seen any other variations of this in
the wild yet, so its good enough for now.
This is a bit of an ugly hack because of the way generator pieces together the
output message. The deepcopys aren't too expensive, though, because we know it
is only called on messages that are not multiparts, and the payload (the thing
that could be large) is an immutable object.
Test and preliminary work on patch by Vajrasky Kok.
This fixes an edge case (20206) where if the input ended in a character
needing encoding but there was no newline on the string, the last byte
of the encoded character would be dropped. The fix is to use a more
efficient algorithm, provided by Serhiy Storchaka (5803), that does not
have the bug.
This is a backward compatible partial fix, the complete fix requires raising
an error instead of accepting the invalid input, so the real fix is only
suitable for 3.4.
This adds EmailMessage and, MIMEPart subclasses of Message
with new API methods, and a ContentManager class used by
the new methods. Also a new policy setting, content_manager.
Patch was reviewed by Stephen J. Turnbull and Serhiy Storchaka,
and reflects their feedback.
I will ideally add some examples of using the new API to the
documentation before the final release.
This also backs out the previous fixes for for #14360, #1717, and #16564.
Those bugs were actually caused by the fact that set_payload didn't decode to
str, thus rendering the model inconsistent. This fix does mean the data
processed by the encoder functions goes through an extra encode/decode cycle,
but it means the model is always consistent. Future API updates will provide
a better way to encode payloads, which will bypass this minor de-optimization.
Tests by Vajrasky Kok.
This was triggered by wanting to make the doctest in email.policy.rst pass;
as_bytes and __bytes__ are clearly useful now that we have BytesGenerator.
Also updated the Message docs to document the policy keyword that was
added in 3.3.
There is more to be done here in terms of accepting RFC invalid
input that some mailers accept, but this covers the valid
RFC places where encoded words can occur in structured headers.
The problem was I was only checking for decimal digits after the third '?',
not for *hex* digits :(.
This changeset also fixes a couple of comment typos, deletes an unused
function relating to encoded word parsing, and removed an invalid
'if' test from the folding function that was revealed by the tests
written to validate this issue.
There were no tests for the encoders module. encode_base64 worked
because it is the default and so got tested implicitly elsewhere, and
we use encode_7or8bit internally, so that worked, too. I previously
fixed encode_noop, so this fix means that everythign in the encoders
module now works, hopefully correctly. Also added an explicit test
for encode_base64.
Previously the parts of the message retained whatever linesep they had on
read, which means if the messages weren't read in univeral newline mode, the
line endings could well be inconsistent. In general sending it via smtplib
would result in them getting fixed, but it is better to generate them
correctly to begin with. Also, the new send_message method of smtplib does
not do the fixup, so that method is producing rfc-invalid output without this
fix.
Previously the parts of the message retained whatever linesep they had on
read, which means if the messages weren't read in univeral newline mode, the
line endings could well be inconsistent. In general sending it via smtplib
would result in them getting fixed, but it is better to generate them
correctly to begin with. Also, the new send_message method of smtplib does
not do the fixup, so that method is producing rfc-invalid output without this
fix.
This code passes all the same tests that the existing RFC mime header
parser passes, plus a bunch of additional ones.
There are a couple of commented out tests where there are issues with the
folding. The folding doesn't normally get invoked for headers parsed from
source, and the cases are marginal anyway (headers with invalid binary data)
so I'm not worried about them, but will fix them after the beta.
There are things that can be done to make this API even more convenient, but I
think this is a solid foundation worth having. And the parser is a full RFC
parser, so it handles cases that the current parser doesn't. (There are also
probably cases where it fails when the current parser doesn't, but I haven't
found them yet ;)
Oh, yeah, and there are some really ugly bits in the parser for handling some
'postel' cases that are unfortunately common.
I hope/plan to to eventually refactor a lot of the code in the parser which
should reduce the line count...but there is no escaping the fact that the
error recovery is welter of special cases.
This is a behavior change: before this leading and trailing spaces were
stripped from ASCII parts, now they are preserved. Without this fix we didn't
parse the examples in the RFC correctly, so I think breaking backward
compatibility here is justified.
Patch by Ralf Schlatterbeck.
This commit also restores the news item for 167256 that it looks like
Terry inadvertently deleted. (Either that, or I don't understand
now merging works...which is equally possible.)
Which also means that it is now producing *something* for any base64
payload, which is what leads to the couple of older test changes in
test_email. This is a slightly backward incompatible behavior change,
but the new behavior is so much more useful than the old (you can now
*reliably* detect errors, and any program that was detecting errors by
sniffing for a base64 return from get_payload(decode=True) and then doing
its own error-recovery decode will just get the error-recovery decode
right away). So this seems to me to be worth the small risk inherent
in this behavior change.
This patch also refactors the defect tests into a separate test file,
since they are no longer just parser tests.
This patch also deprecates the MalformedHeaderDefect. My best guess is that
this defect was rendered obsolete by a refactoring of the parser, and the
corresponding defect for the new parser (which this patch introduces) was
overlooked.
R David Murray
Benjamin Peterson
Serhiy Storchaka
Berker Peksag
Ezio Melotti
Zachary Ware
Terry Jan Reedy
Andrew Svetlov
Victor Stinner
Georg Brandl
Alexander Belopolsky