This also backs out the previous fixes for for #14360, #1717, and #16564.
Those bugs were actually caused by the fact that set_payload didn't decode to
str, thus rendering the model inconsistent. This fix does mean the data
processed by the encoder functions goes through an extra encode/decode cycle,
but it means the model is always consistent. Future API updates will provide
a better way to encode payloads, which will bypass this minor de-optimization.
Tests by Vajrasky Kok.
This was triggered by wanting to make the doctest in email.policy.rst pass;
as_bytes and __bytes__ are clearly useful now that we have BytesGenerator.
Also updated the Message docs to document the policy keyword that was
added in 3.3.
There is more to be done here in terms of accepting RFC invalid
input that some mailers accept, but this covers the valid
RFC places where encoded words can occur in structured headers.
The problem was I was only checking for decimal digits after the third '?',
not for *hex* digits :(.
This changeset also fixes a couple of comment typos, deletes an unused
function relating to encoded word parsing, and removed an invalid
'if' test from the folding function that was revealed by the tests
written to validate this issue.
There were no tests for the encoders module. encode_base64 worked
because it is the default and so got tested implicitly elsewhere, and
we use encode_7or8bit internally, so that worked, too. I previously
fixed encode_noop, so this fix means that everythign in the encoders
module now works, hopefully correctly. Also added an explicit test
for encode_base64.
The new _has_surrogates code was suggested by Serhiy Storchaka. See
the issue for timings, but it is far faster than any other alternative,
and also removes the load time that we previously incurred from compiling
the complex regex this replaces.
Previously the parts of the message retained whatever linesep they had on
read, which means if the messages weren't read in univeral newline mode, the
line endings could well be inconsistent. In general sending it via smtplib
would result in them getting fixed, but it is better to generate them
correctly to begin with. Also, the new send_message method of smtplib does
not do the fixup, so that method is producing rfc-invalid output without this
fix.
Previously the parts of the message retained whatever linesep they had on
read, which means if the messages weren't read in univeral newline mode, the
line endings could well be inconsistent. In general sending it via smtplib
would result in them getting fixed, but it is better to generate them
correctly to begin with. Also, the new send_message method of smtplib does
not do the fixup, so that method is producing rfc-invalid output without this
fix.
Previously the parts of the message retained whatever linesep they had on
read, which means if the messages weren't read in univeral newline mode, the
line endings could well be inconsistent. In general sending it via smtplib
would result in them getting fixed, but it is better to generate them
correctly to begin with. Also, the new send_message method of smtplib does
not do the fixup, so that method is producing rfc-invalid output without this
fix.
R David Murray
Ezio Melotti
Victor Stinner
Terry Jan Reedy
Andrew Svetlov
Philip Jenvey
Georg Brandl
Alexander Belopolsky