This makes Header.encode throw a HeaderParseError if it winds up
formatting a header such that a continuation line has no leading
whitespace and looks like a header. Since Header accepts values
containing newlines and preserves them (and this is by design), without
this fix any program that took user input (say, a subject in a web form)
and passed it to the email package as a header was vulnerable to header
injection attacks. (As far as we know this has never been exploited.)
Thanks to Jakub Wilk for reporting this vulnerability.
This applies only when generating strings from non-RFC compliant binary
input; it makes the existing recoding behavior more consistent (ie:
now no data is lost when recoding).
Before this patch if a value being encoded had an embedded newline,
the line following the newline would have no leading whitespace,
and the whitespace it did have was encoded into the word. Now
the existing whitespace gets turned into a blank, the way it does
in other header reformatting, and the _continuation_ws gets added
at the beginning of the encoded line.
The RFC is bit hard to understand on this point, but the examples
clearly show that parameter values that are encoded according
to its charset/language rules don't have surrounding quotes, and
the ABNF does not allow for quotes. So when we produce such
encoded values, we no longer add quotes.
Such addresses are not RFC compliant except under the 'obsolete syntax'
rules, but before this fix the whitespace was dropped from the input,
concatenating the pieces. That breaks one of the principles of the
email package, that of preserving the input as much as possible.
It also denies the application program the opportunity to apply its
own heuristics to interpretation of such non-compliant addresses.
It is possible users of the email package were depending on the local
part always being a single token, so this fix will not be backported.
The tests that were failing on (some) windows machines, where the
msg_XX.txt files used native \r\n lineseps are now also run on machines
that use \n natively, and conversely the \n tests are run on Windows.
The failing tests revealed one place where linesep needed to be added
to a flatten call in generator. There was also another that the tests
didn't catch, so I added a test for that case as well.
The work on this is not 100% complete, but everything is present to
allow real-world testing of the code. The only remaining major todo
item is to (hopefully!) enhance the handling of non-ASCII bytes in headers
converted to unicode by RFC2047 encoding them rather than replacing them with
'?'s.
Also made the doc string for email._parseaddr's 'quote' function more
accurate; I'd love to make the function match the old docstring instead,
but other code uses it according the existing semantics.
Two character years are now converted to four character years using
the Posix standard rule (<68 == 2000, >=68==1900). This makes the
parsed date RFC2822 compliant even if the input is not.
Patch and test by Jeffrey Finkelstein.
The RFCs encourage following Postel's law: be liberal in what you accept.
So if someone forgot to pad the base64 encoded word payload to an
even four bytes, we add the padding before handing it to base64mime.decode.
Previously, missing padding resulted in a HeaderParseError.
Patch by Jason Williams.
svn+ssh://pythondev@svn.python.org/python/trunk
........
r81675 | r.david.murray | 2010-06-03 11:43:20 -0400 (Thu, 03 Jun 2010) | 10 lines
#5610: use \Z not $ so we don't eat extra chars when body part ends with \r\n.
If a body part ended with \r\n, feedparser, using '$' to terminate its
search for the newline, would match on the \r\n, and think that it needed
to strip two characters in order to account for the line end before the
boundary. That made it chop one too many characters off the end of
the body part. Using \Z makes the match correct.
Patch and test by Tony Nelson.
........
This means that what gets encoded in base64 is the encoded version of the
unicode payload. This bug was revealed by a forward port of the tests from
Issue 1368247, but the fix was completely different.
Note that the merge is only of the tests, the doc changes were inappropriate
since email5 expects unicode, not bytes. I'm also not convinced that
quopri works correctly in email5, but that's a different issue.
Merged revisions 81658 via svnmerge from
svn+ssh://pythondev@svn.python.org/python/trunk
........
r81658 | r.david.murray | 2010-06-02 18:03:15 -0400 (Wed, 02 Jun 2010) | 9 lines
#1368247: make set_charset/MIMEText automatically encode unicode _payload.
Fixes (mysterious, to the end user) UnicodeErrors when using utf-8 as
the charset and unicode as the _text argument. Also makes the way in
which unicode gets encoded to quoted printable for other charsets more
sane (it only worked by accident previously). The _payload now is encoded
to the charset.output_charset if it is unicode.
........
svn+ssh://pythondev@svn.python.org/python/trunk
It turns out that email5 (py3k), because it is using unicode for the
payload, doesn't do the encoding to the output character set until later
in the process. Specifically, charset.body_encode no longer does the
input-to-output charset conversion. So the if test in the exception
clause in encoders.encode_7or8bit really is needed in email5.
So, this merge only merges the test, not the removal of the 'if'.
........
r80800 | r.david.murray | 2010-05-05 13:31:03 -0400 (Wed, 05 May 2010) | 9 lines
Issue #7472: remove unused code from email.encoders.encode_7or8bit.
Yukihiro Nakadaira noticed a typo in encode_7or8bit that was trying
to special case iso-2022 codecs. It turns out that the code in
question is never used, because whereas it was designed to trigger
if the payload encoding was eight bit but its output encoding was
7 bit, in practice the payload is always converted to the 7bit
encoding before encode_7or8bit is called. Patch by Shawat Anand.
........
the trunk patch, and removes the last trace of bdecode, which was
a commented out call in message.py.
Merged revisions 78778 via svnmerge from
svn+ssh://pythondev@svn.python.org/python/trunk
........
r78778 | r.david.murray | 2010-03-07 21:04:06 -0500 (Sun, 07 Mar 2010) | 9 lines
Issue #7143: get_payload used to strip any trailing newline from a
base64 transfer-encoded payload *after* decoding it; it no longer does.
email had a special method in utils, _bdecode, specifically to do this,
so it must have served a purpose at some point, yet it is clearly wrong
per RFC. Fixed with Barry's approval, but no backport. Email package
minor version number is bumped, now version 4.0.1.
Patch by Joaquin Cuenca Abela.
........
svn+ssh://pythondev@svn.python.org/python/trunk
........
r78274 | r.david.murray | 2010-02-20 23:23:00 -0500 (Sat, 20 Feb 2010) | 9 lines
Issue 7970: When email.Parser.Parser parses a MIME message of type
message/rfc822 it turns it into an object whose body consists of
a list containing a single Message object. HeaderParser, on the
other hand, just copies the body as a string. Generator.flatten
has a special handler for the message mime type that expected the
body to be the one item list. This fails if the message was parsed
by HeaderParser. So we now check to see if the body is a string
first, and if so just we just emit it.
........
svn+ssh://pythondev@svn.python.org/python/trunk
Merge adds an additional test for as_string with a maxheaderlen specified.
........
r77517 | r.david.murray | 2010-01-16 00:15:17 -0500 (Sat, 16 Jan 2010) | 6 lines
Issue #1670765: Prevent email.generator.Generator from re-wrapping
headers in multipart/signed MIME parts, which fixes one of the sources of
invalid modifications to such parts by Generator. Patch and tests by
Martin von Gagern.
........
r77525 | r.david.murray | 2010-01-16 11:08:32 -0500 (Sat, 16 Jan 2010) | 2 lines
Fix issue number in comment.
........
svn+ssh://pythondev@svn.python.org/python/trunk
........
r60054 | christian.heimes | 2008-01-18 20:12:56 +0100 (Fri, 18 Jan 2008) | 1 line
Silence Coverity false alerts with CIDs #172, #183, #184
........
r60057 | guido.van.rossum | 2008-01-18 21:56:30 +0100 (Fri, 18 Jan 2008) | 3 lines
Fix an edge case whereby the __del__() method of a classic class could
create a new weakref to the object.
........
r60058 | raymond.hettinger | 2008-01-18 22:14:58 +0100 (Fri, 18 Jan 2008) | 1 line
Better variable name in an example.
........
r60063 | guido.van.rossum | 2008-01-19 00:05:40 +0100 (Sat, 19 Jan 2008) | 3 lines
This got fixed for classic classes in r60057,
and backported to 2.5.2 in 60056.
........
r60068 | jeffrey.yasskin | 2008-01-19 10:56:06 +0100 (Sat, 19 Jan 2008) | 4 lines
Several tweaks: add construction from strings and .from_decimal(), change
__init__ to __new__ to enforce immutability, and remove "rational." from repr
and the parens from str.
........
r60069 | georg.brandl | 2008-01-19 11:11:27 +0100 (Sat, 19 Jan 2008) | 2 lines
Fix markup.
........
r60070 | georg.brandl | 2008-01-19 11:16:09 +0100 (Sat, 19 Jan 2008) | 3 lines
Amend curses docs by info how to write non-ascii characters.
Thanks to Jeroen Ruigrok van der Werven.
........
r60071 | georg.brandl | 2008-01-19 11:18:07 +0100 (Sat, 19 Jan 2008) | 2 lines
Indentation normalization.
........
r60073 | facundo.batista | 2008-01-19 13:32:27 +0100 (Sat, 19 Jan 2008) | 5 lines
Fix issue #1822: MIMEMultipart.is_multipart() behaves correctly for a
just-created (and empty) instance. Added tests for this. Thanks
Jonathan Share.
........
r60074 | andrew.kuchling | 2008-01-19 14:33:20 +0100 (Sat, 19 Jan 2008) | 1 line
Polish sentence
........
r60075 | christian.heimes | 2008-01-19 14:46:06 +0100 (Sat, 19 Jan 2008) | 1 line
Added unit test to verify that threading.local doesn't cause ref leaks. It seems that the thread local storage always keeps the storage of the last stopped thread alive. Can anybody comment on it, please?
........
r60076 | christian.heimes | 2008-01-19 16:06:09 +0100 (Sat, 19 Jan 2008) | 1 line
Update for threading.local test.
........
r60077 | andrew.kuchling | 2008-01-19 16:16:37 +0100 (Sat, 19 Jan 2008) | 1 line
Polish sentence
........
r60078 | georg.brandl | 2008-01-19 16:22:16 +0100 (Sat, 19 Jan 2008) | 2 lines
Fix typos.
........
No detailed change log; just check out the change log for the py3k-pep3137
branch. The most obvious changes:
- str8 renamed to bytes (PyString at the C level);
- bytes renamed to buffer (PyBytes at the C level);
- PyString and PyUnicode are no longer compatible.
I.e. we now have an immutable bytes type and a mutable bytes type.
The behavior of PyString was modified quite a bit, to make it more
bytes-like. Some changes are still on the to-do list.
line splitting examples don't split things the way they used to -- or should.
In these cases, change the test case and add an XXX.
The final failure was in Charset.body_encode() with euc-jp charset. These
return the original string unencoded, which isn't right. XXX and comment this
out for now; we'll fix it after a1.
Fix a couple of tests since .body_encode()'s arguments have changed. Also, I
think body_encode() should take a string not a byte array for consistency with
the rest of the api (but I'm not positive about this). In
quoprimime.body_encode(), body_check() must be passed an int.
Current status: 7F (no errors!)
MIMEApplication() requires a bytes object for its _data, so fix the tests.
We no longer need utils._identity() or utils._bdecode(). The former isn't
used anywhere AFAICT (where's "make test's" lint? <wink>) and the latter is a
kludge that is eliminated by base64.b64encode().
Current status: 5F/5E
This repairs the linear whitespace insertion between RFC 2047 encoded words
without leaving bogus trailing spaces at the end lines that end in encoded
words.
Current status: 7F/9E
email.base64mime, but test_smtplib still has failures for me. They are
timeout errors so think they're more related to my current wacky network setup
than bugs remaining in the code related to the email package.
This also r57693 that got clobbered with the sandbox sync, and fixes a couple
of other minor problems that cropped up. I will kill the sandbox branch next.
The email package now has 11F/11E.
This should restore the email package in the py3k branch to exactly what's in
the sandbox.
This wipes out 1-2 fixes made post-copy, which I'll re-apply shortly.
Use str.encode('raw-unicode-escape') consistently instead of bytes(string).
Remove the convert_eols argument from base64mime.decode(). This matches
previous API changes done to the quoprimime module.
R. David Murray
Ezio Melotti
Brett Cannon
Benjamin Peterson
Georg Brandl
Barry Warsaw
Martin v. Löwis
Christian Heimes
Guido van Rossum