This makes Header.encode throw a HeaderParseError if it winds up
formatting a header such that a continuation line has no leading
whitespace and looks like a header. Since Header accepts values
containing newlines and preserves them (and this is by design), without
this fix any program that took user input (say, a subject in a web form)
and passed it to the email package as a header was vulnerable to header
injection attacks. (As far as we know this has never been exploited.)
Thanks to Jakub Wilk for reporting this vulnerability.
This applies only when generating strings from non-RFC compliant binary
input; it makes the existing recoding behavior more consistent (ie:
now no data is lost when recoding).
Before this patch if a value being encoded had an embedded newline,
the line following the newline would have no leading whitespace,
and the whitespace it did have was encoded into the word. Now
the existing whitespace gets turned into a blank, the way it does
in other header reformatting, and the _continuation_ws gets added
at the beginning of the encoded line.
The RFC is bit hard to understand on this point, but the examples
clearly show that parameter values that are encoded according
to its charset/language rules don't have surrounding quotes, and
the ABNF does not allow for quotes. So when we produce such
encoded values, we no longer add quotes.
Such addresses are not RFC compliant except under the 'obsolete syntax'
rules, but before this fix the whitespace was dropped from the input,
concatenating the pieces. That breaks one of the principles of the
email package, that of preserving the input as much as possible.
It also denies the application program the opportunity to apply its
own heuristics to interpretation of such non-compliant addresses.
It is possible users of the email package were depending on the local
part always being a single token, so this fix will not be backported.
The rearranged code should do exactly what the old code did, but
the new code avoids a potentially costly re computation in the case
where a boundary already exists.
The tests that were failing on (some) windows machines, where the
msg_XX.txt files used native \r\n lineseps are now also run on machines
that use \n natively, and conversely the \n tests are run on Windows.
The failing tests revealed one place where linesep needed to be added
to a flatten call in generator. There was also another that the tests
didn't catch, so I added a test for that case as well.
The work on this is not 100% complete, but everything is present to
allow real-world testing of the code. The only remaining major todo
item is to (hopefully!) enhance the handling of non-ASCII bytes in headers
converted to unicode by RFC2047 encoding them rather than replacing them with
'?'s.
Also made the doc string for email._parseaddr's 'quote' function more
accurate; I'd love to make the function match the old docstring instead,
but other code uses it according the existing semantics.
Two character years are now converted to four character years using
the Posix standard rule (<68 == 2000, >=68==1900). This makes the
parsed date RFC2822 compliant even if the input is not.
Patch and test by Jeffrey Finkelstein.
The RFCs encourage following Postel's law: be liberal in what you accept.
So if someone forgot to pad the base64 encoded word payload to an
even four bytes, we add the padding before handing it to base64mime.decode.
Previously, missing padding resulted in a HeaderParseError.
Patch by Jason Williams.
svn+ssh://pythondev@svn.python.org/python/trunk
........
r81675 | r.david.murray | 2010-06-03 11:43:20 -0400 (Thu, 03 Jun 2010) | 10 lines
#5610: use \Z not $ so we don't eat extra chars when body part ends with \r\n.
If a body part ended with \r\n, feedparser, using '$' to terminate its
search for the newline, would match on the \r\n, and think that it needed
to strip two characters in order to account for the line end before the
boundary. That made it chop one too many characters off the end of
the body part. Using \Z makes the match correct.
Patch and test by Tony Nelson.
........
This means that what gets encoded in base64 is the encoded version of the
unicode payload. This bug was revealed by a forward port of the tests from
Issue 1368247, but the fix was completely different.
Note that the merge is only of the tests, the doc changes were inappropriate
since email5 expects unicode, not bytes. I'm also not convinced that
quopri works correctly in email5, but that's a different issue.
Merged revisions 81658 via svnmerge from
svn+ssh://pythondev@svn.python.org/python/trunk
........
r81658 | r.david.murray | 2010-06-02 18:03:15 -0400 (Wed, 02 Jun 2010) | 9 lines
#1368247: make set_charset/MIMEText automatically encode unicode _payload.
Fixes (mysterious, to the end user) UnicodeErrors when using utf-8 as
the charset and unicode as the _text argument. Also makes the way in
which unicode gets encoded to quoted printable for other charsets more
sane (it only worked by accident previously). The _payload now is encoded
to the charset.output_charset if it is unicode.
........
svn+ssh://pythondev@svn.python.org/python/trunk
It turns out that email5 (py3k), because it is using unicode for the
payload, doesn't do the encoding to the output character set until later
in the process. Specifically, charset.body_encode no longer does the
input-to-output charset conversion. So the if test in the exception
clause in encoders.encode_7or8bit really is needed in email5.
So, this merge only merges the test, not the removal of the 'if'.
........
r80800 | r.david.murray | 2010-05-05 13:31:03 -0400 (Wed, 05 May 2010) | 9 lines
Issue #7472: remove unused code from email.encoders.encode_7or8bit.
Yukihiro Nakadaira noticed a typo in encode_7or8bit that was trying
to special case iso-2022 codecs. It turns out that the code in
question is never used, because whereas it was designed to trigger
if the payload encoding was eight bit but its output encoding was
7 bit, in practice the payload is always converted to the 7bit
encoding before encode_7or8bit is called. Patch by Shawat Anand.
........
svn+ssh://pythondev@svn.python.org/python/trunk
........
r79994 | r.david.murray | 2010-04-12 10:26:06 -0400 (Mon, 12 Apr 2010) | 9 lines
Issue #7472: ISO-2022 charsets now consistently use 7bit CTE.
Fixed a typo in the email.encoders module so that messages output using
an ISO-2022 character set will use a content-transfer-encoding of
7bit consistently. Previously if the input data had any eight bit
characters the output data would get marked as 8bit even though it
was actually 7bit.
........
svn+ssh://pythondev@svn.python.org/python/trunk
........
r78018 | georg.brandl | 2010-02-06 11:08:21 +0100 (Sa, 06 Feb 2010) | 1 line
#7864: make deprecation notices a bit clearer.
........
r78035 | georg.brandl | 2010-02-06 23:44:17 +0100 (Sa, 06 Feb 2010) | 1 line
Fix duplicate import.
........
r78036 | georg.brandl | 2010-02-06 23:49:47 +0100 (Sa, 06 Feb 2010) | 1 line
Remove unused import.
........
r78037 | georg.brandl | 2010-02-06 23:59:15 +0100 (Sa, 06 Feb 2010) | 1 line
No need to assign the results of expressions used only for side effects.
........
r78038 | georg.brandl | 2010-02-07 00:02:29 +0100 (So, 07 Feb 2010) | 1 line
Add a missing import.
........
r78039 | georg.brandl | 2010-02-07 00:06:24 +0100 (So, 07 Feb 2010) | 1 line
Add missing imports.
........
r78040 | georg.brandl | 2010-02-07 00:08:00 +0100 (So, 07 Feb 2010) | 1 line
Fix a few UnboundLocalErrors in test_long.
........
r78042 | georg.brandl | 2010-02-07 00:12:12 +0100 (So, 07 Feb 2010) | 1 line
Add missing import.
........
r78043 | georg.brandl | 2010-02-07 00:12:19 +0100 (So, 07 Feb 2010) | 1 line
Remove duplicate test method.
........
r78046 | georg.brandl | 2010-02-07 00:18:00 +0100 (So, 07 Feb 2010) | 1 line
Fix various missing import/unbound name errors.
........
r78048 | georg.brandl | 2010-02-07 00:23:45 +0100 (So, 07 Feb 2010) | 1 line
We heard you like test failures so we put unbound locals in your test so that you can fail while you fail.
........
r78049 | georg.brandl | 2010-02-07 00:33:33 +0100 (So, 07 Feb 2010) | 1 line
Fix import/access for some identifiers. _TestSharedCTypes does not seem to be executed?
........
r78050 | georg.brandl | 2010-02-07 00:34:10 +0100 (So, 07 Feb 2010) | 1 line
Fix more unbound locals in code paths that do not seem to be used.
........
r78051 | georg.brandl | 2010-02-07 00:53:52 +0100 (So, 07 Feb 2010) | 1 line
Add missing import when running these tests standalone.
........
r78052 | georg.brandl | 2010-02-07 00:54:04 +0100 (So, 07 Feb 2010) | 1 line
Add missing import when running these tests standalone.
........
r78054 | georg.brandl | 2010-02-07 00:58:25 +0100 (So, 07 Feb 2010) | 1 line
Add missing import.
........
r78059 | georg.brandl | 2010-02-07 12:34:15 +0100 (So, 07 Feb 2010) | 1 line
Use "regexp" consistently.
........
r78075 | georg.brandl | 2010-02-07 13:16:12 +0100 (So, 07 Feb 2010) | 1 line
Fix another duplicated test method.
........
r78076 | georg.brandl | 2010-02-07 13:19:43 +0100 (So, 07 Feb 2010) | 1 line
Fix wrong usage of "except X, Y:".
........
r78077 | georg.brandl | 2010-02-07 13:25:50 +0100 (So, 07 Feb 2010) | 1 line
Fix two redefined test methods.
........
r78078 | georg.brandl | 2010-02-07 13:27:06 +0100 (So, 07 Feb 2010) | 1 line
Fix a redefined test method.
........
r78079 | georg.brandl | 2010-02-07 13:34:26 +0100 (So, 07 Feb 2010) | 1 line
Add a minimal test for fnmatchcase().
........
r78080 | georg.brandl | 2010-02-07 13:55:12 +0100 (So, 07 Feb 2010) | 1 line
Remove duplicate test method.
........
the trunk patch, and removes the last trace of bdecode, which was
a commented out call in message.py.
Merged revisions 78778 via svnmerge from
svn+ssh://pythondev@svn.python.org/python/trunk
........
r78778 | r.david.murray | 2010-03-07 21:04:06 -0500 (Sun, 07 Mar 2010) | 9 lines
Issue #7143: get_payload used to strip any trailing newline from a
base64 transfer-encoded payload *after* decoding it; it no longer does.
email had a special method in utils, _bdecode, specifically to do this,
so it must have served a purpose at some point, yet it is clearly wrong
per RFC. Fixed with Barry's approval, but no backport. Email package
minor version number is bumped, now version 4.0.1.
Patch by Joaquin Cuenca Abela.
........
svn+ssh://pythondev@svn.python.org/python/trunk
........
r78274 | r.david.murray | 2010-02-20 23:23:00 -0500 (Sat, 20 Feb 2010) | 9 lines
Issue 7970: When email.Parser.Parser parses a MIME message of type
message/rfc822 it turns it into an object whose body consists of
a list containing a single Message object. HeaderParser, on the
other hand, just copies the body as a string. Generator.flatten
has a special handler for the message mime type that expected the
body to be the one item list. This fails if the message was parsed
by HeaderParser. So we now check to see if the body is a string
first, and if so just we just emit it.
........
svn+ssh://pythondev@svn.python.org/python/trunk
Merge adds an additional test for as_string with a maxheaderlen specified.
........
r77517 | r.david.murray | 2010-01-16 00:15:17 -0500 (Sat, 16 Jan 2010) | 6 lines
Issue #1670765: Prevent email.generator.Generator from re-wrapping
headers in multipart/signed MIME parts, which fixes one of the sources of
invalid modifications to such parts by Generator. Patch and tests by
Martin von Gagern.
........
r77525 | r.david.murray | 2010-01-16 11:08:32 -0500 (Sat, 16 Jan 2010) | 2 lines
Fix issue number in comment.
........
svn+ssh://pythondev@svn.python.org/python/trunk
........
r75301 | r.david.murray | 2009-10-09 17:50:54 -0400 (Fri, 09 Oct 2009) | 5 lines
Issue #7082: When falling back to the MIME 'name' parameter, the
correct place to look for it is the Content-Type header.
Patch by Darren Worrall.
........
svn+ssh://pythondev@svn.python.org/python/trunk
........
r73004 | jeffrey.yasskin | 2009-05-28 22:44:31 -0500 (Thu, 28 May 2009) | 5 lines
Fix nearly all compilation warnings under Apple gcc-4.0. Tested with OPT="-g
-Wall -Wstrict-prototypes -Werror" in both --with-pydebug mode and --without.
There's still a batch of non-prototype warnings in Xlib.h that I don't know how
to fix.
........
r73439 | benjamin.peterson | 2009-06-15 19:29:31 -0500 (Mon, 15 Jun 2009) | 1 line
don't mask encoding errors when decoding a string #6289
........
r73496 | vinay.sajip | 2009-06-21 12:37:27 -0500 (Sun, 21 Jun 2009) | 1 line
Issue #6314: logging.basicConfig() performs extra checks on the "level" argument.
........
r73509 | amaury.forgeotdarc | 2009-06-22 14:33:48 -0500 (Mon, 22 Jun 2009) | 2 lines
#4490 Fix sample code run by "python -m xml.sax.xmlreader"
........
r73529 | r.david.murray | 2009-06-23 13:02:46 -0500 (Tue, 23 Jun 2009) | 4 lines
Fix issue 5230 by having pydoc's safeimport check to see if the import
error was thrown from itself in order to decide if the module can't be
found. Thanks to Lucas Prado Melo for collaborating on the fix and tests.
........
r73564 | amaury.forgeotdarc | 2009-06-25 17:29:29 -0500 (Thu, 25 Jun 2009) | 6 lines
#2016 Fix a crash in function call when the **kwargs dictionary is mutated
during the function call setup.
This even gives a slight speedup, probably because tuple allocation
is faster than PyMem_NEW.
........
r73576 | benjamin.peterson | 2009-06-26 18:37:06 -0500 (Fri, 26 Jun 2009) | 1 line
document is_declared_global()
........
r73577 | benjamin.peterson | 2009-06-27 09:16:23 -0500 (Sat, 27 Jun 2009) | 1 line
link to extensive generator docs in the reference manual
........
r73595 | ezio.melotti | 2009-06-27 18:45:39 -0500 (Sat, 27 Jun 2009) | 1 line
stmt and setup can contain multiple statements, see #5896
........
r73596 | ezio.melotti | 2009-06-27 19:07:45 -0500 (Sat, 27 Jun 2009) | 1 line
Fixed a wrong apostrophe
........
r73605 | georg.brandl | 2009-06-28 07:10:18 -0500 (Sun, 28 Jun 2009) | 1 line
Remove stray pychecker directive.
........
R. David Murray
Georg Brandl
Ezio Melotti
Brett Cannon
Florent Xicluna
Barry Warsaw
Benjamin Peterson
Amaury Forgeot d'Arc