This also backs out the previous fixes for for #14360, #1717, and #16564.
Those bugs were actually caused by the fact that set_payload didn't decode to
str, thus rendering the model inconsistent. This fix does mean the data
processed by the encoder functions goes through an extra encode/decode cycle,
but it means the model is always consistent. Future API updates will provide
a better way to encode payloads, which will bypass this minor de-optimization.
Tests by Vajrasky Kok.
There were no tests for the encoders module. encode_base64 worked
because it is the default and so got tested implicitly elsewhere, and
we use encode_7or8bit internally, so that worked, too. I previously
fixed encode_noop, so this fix means that everythign in the encoders
module now works, hopefully correctly. Also added an explicit test
for encode_base64.
Previously the parts of the message retained whatever linesep they had on
read, which means if the messages weren't read in univeral newline mode, the
line endings could well be inconsistent. In general sending it via smtplib
would result in them getting fixed, but it is better to generate them
correctly to begin with. Also, the new send_message method of smtplib does
not do the fixup, so that method is producing rfc-invalid output without this
fix.
This code passes all the same tests that the existing RFC mime header
parser passes, plus a bunch of additional ones.
There are a couple of commented out tests where there are issues with the
folding. The folding doesn't normally get invoked for headers parsed from
source, and the cases are marginal anyway (headers with invalid binary data)
so I'm not worried about them, but will fix them after the beta.
There are things that can be done to make this API even more convenient, but I
think this is a solid foundation worth having. And the parser is a full RFC
parser, so it handles cases that the current parser doesn't. (There are also
probably cases where it fails when the current parser doesn't, but I haven't
found them yet ;)
Oh, yeah, and there are some really ugly bits in the parser for handling some
'postel' cases that are unfortunately common.
I hope/plan to to eventually refactor a lot of the code in the parser which
should reduce the line count...but there is no escaping the fact that the
error recovery is welter of special cases.
This is a behavior change: before this leading and trailing spaces were
stripped from ASCII parts, now they are preserved. Without this fix we didn't
parse the examples in the RFC correctly, so I think breaking backward
compatibility here is justified.
Patch by Ralf Schlatterbeck.
Which also means that it is now producing *something* for any base64
payload, which is what leads to the couple of older test changes in
test_email. This is a slightly backward incompatible behavior change,
but the new behavior is so much more useful than the old (you can now
*reliably* detect errors, and any program that was detecting errors by
sniffing for a base64 return from get_payload(decode=True) and then doing
its own error-recovery decode will just get the error-recovery decode
right away). So this seems to me to be worth the small risk inherent
in this behavior change.
This patch also refactors the defect tests into a separate test file,
since they are no longer just parser tests.
This patch also deprecates the MalformedHeaderDefect. My best guess is that
this defect was rendered obsolete by a refactoring of the parser, and the
corresponding defect for the new parser (which this patch introduces) was
overlooked.
This patch primarily does two things: (1) it adds some internal-interface
methods to Policy that allow for Policy to control the parsing and folding of
headers in such a way that we can construct a backward compatibility policy
that is 100% compatible with the 3.2 API, while allowing a new policy to
implement the email6 API. (2) it adds that backward compatibility policy and
refactors the test suite so that the only differences between the 3.2
test_email.py file and the 3.3 test_email.py file is some small changes in
test framework and the addition of tests for bugs fixed that apply to the 3.2
API.
There are some additional teaks, such as moving just the code needed for the
compatibility policy into _policybase, so that the library code can import
only _policybase. That way the new code that will be added for email6
will only get imported when a non-compatibility policy is imported.
Éric pointed out that given that the default was documented as None, someone
would reasonably pass that to get the default behavior. In fixing the code to
use None, I noticed that the change to _charset was being done after it had
already been passed to MIMENonMultipart. The change to the test verifies that
the order is now correct.
Previously it would just accept the unicode, which would wind up as unicode in
the transfer-encoded message object, which is just wrong.
Patch by Jeff Knupp.
In Python2, if a unicode string was assigned as the value of a header,
email would automatically CTE encode it using the UTF8 charset.
This capability was lost in the Python3 translation, and this patch
restores it.
Patch by Ali Ikinci, assisted by R. David Murray.
I also added a fix for the mailbox test that was depending (with a comment
that it was a bad idea to so depend) on non-ASCII causing message_from_string
to raise an error. It now uses support.patch to induce an error during
message serialization.
This new interface will also allow for future planned enhancements
in control over the parser/generator without requiring any additional
complexity in the parser/generator API.
Patch reviewed by Éric Araujo and Barry Warsaw.
R David Murray
Ezio Melotti
Georg Brandl
Alexander Belopolsky