This PR replaces GH-1977. The reason for the replacement is two-fold.
The fix itself is different is that if the CTE header doesn't exist in the original message, it is inserted. This is important because the new CTE could be quoted-printable whereas the original is implicit 8bit.
Also the tests are different. The test_nonascii_as_string_without_cte test in GH-1977 doesn't actually test the issue in that it passes without the fix. The test_nonascii_as_string_without_content_type_and_cte test is improved here, and even though it doesn't fail without the fix, it is included for completeness.
Automerge-Triggered-By: @warsaw
(cherry picked from commit bf838227c3)
Co-authored-by: Mark Sapiro <mark@msapiro.net>
* bpo-39040: Fix parsing of email headers with encoded-words inside a quoted string.
It is fairly common to find malformed mime headers (especially content-disposition
headers) where the parameter values, instead of being encoded to RFC
standards, are "encoded" by doing RFC 2047 "encoded word" encoding, and
then enclosing the whole thing in quotes. The processing of these malformed
headers was incorrectly leaving the spaces between encoded words in the decoded
text (whitespace between adjacent encoded words is supposed to be stripped on
decoding). This changeset fixes the encoded word processing inside quoted strings
(bare-quoted-string) to do correct RFC 2047 decoding by stripping that
whitespace.
(cherry picked from commit 21017ed904)
Co-authored-by: Abhilash Raj <maxking@users.noreply.github.com>
Disallow CR or LF in email.headerregistry.Address arguments to guard against header injection attacks.
(cherry picked from commit 614f17211c)
Co-authored-by: Ashwin Ramaswami <aramaswamis@gmail.com>
raw_data_manager (default for EmailPolicy, EmailMessage)
does correct wrapping of 'text' parts as long as the message contains
characters outside of 7bit US-ASCII set: base64 or qp
Content-Transfer-Encoding is applied if the lines would be too long
without it. It did not, however, do this for ascii-only text,
which could result in lines that were longer than
policy.max_line_length or even the rfc 998 maximum.
This changeset fixes the heuristic so that if lines are longer than
policy.max_line_length, it will always apply a
content-transfer-encoding so that the lines are wrapped correctly.
(cherry picked from commit 6f2f475d5a)
Co-authored-by: Arkadiusz Hiler <arek.l1@gmail.com>
Fix a potential IndexError when passing an empty value to the message-id
parser. Instead, HeaderParseError should be raised.
(cherry picked from commit 3ae4ea1931)
Co-authored-by: Abhilash Raj <maxking@users.noreply.github.com>
This adds a new InvalidMessageID token to the email header parser which can be
used to represent invalid message-id headers in the parse tree.
(cherry picked from commit 68157da8b4)
Co-authored-by: Abhilash Raj <maxking@users.noreply.github.com>
parse_message_id() was improperly using a token defined inside an exception
handler, which was raising `UnboundLocalError` on parsing an invalid value.
https://bugs.python.org/issue38698
(cherry picked from commit bb815499af)
Co-authored-by: Claudiu Popa <pcmanticore@gmail.com>
KeyError should cause a failure in parsing the encoded word and should be caught and raised as a _InvalidEWError instead.
(cherry picked from commit 65dcc8a8dc)
Co-authored-by: Andrei Troie <andreitroie90@gmail.com>
Fixes a case in which email._header_value_parser.get_unstructured hangs the system for some invalid headers. This covers the cases in which the header contains either:
- a case without trailing whitespace
- an invalid encoded word
https://bugs.python.org/issue37764
This fix should also be backported to 3.7 and 3.8
https://bugs.python.org/issue37764
(cherry picked from commit c5b242f87f)
Co-authored-by: Ashwin Ramaswami <aramaswamis@gmail.com>
* [bpo-21315](https://bugs.python.org/issue21315): Fix parsing of encoded words with missing leading ws.
Because of missing leading whitespace, encoded word would get parsed as
unstructured token. This patch fixes that by looking for encoded words when
splitting tokens with whitespace.
Missing trailing whitespace around encoded word now register a defect
instead.
Original patch suggestion by David R. Murray on [bpo-21315](https://bugs.python.org/issue21315).
(cherry picked from commit 66c4f3f38b)
Co-authored-by: Abhilash Raj <maxking@users.noreply.github.com>
(cherry picked from commit dc20fc4311)
Co-authored-by: Miss Islington (bot) <31488909+miss-islington@users.noreply.github.com>
https://bugs.python.org/issue21315
Special characters in email address header display names are normally
put within double quotes. However, encoded words (=?charset?x?...?=) are
not allowed withing double quotes. When the header contains a word with
special characters and another word that must be encoded, the first one
must also be encoded.
In the next example, the display name in the From header is quoted and
therefore the comma is allowed; in the To header, the comma is not
within quotes and not encoded, which is not allowed and therefore
rejected by some mail servers.
From: "Foo Bar, France" <foo@example.com>
To: Foo Bar, =?utf-8?q?Espa=C3=B1a?= <foo@example.com>
https://bugs.python.org/issue37482
(cherry picked from commit df0c21ff46)
Co-authored-by: bsiem <52461103+bsiem@users.noreply.github.com>
This should fix the IndexError trying to retrieve `DisplayName.display_name` and `DisplayName.value` when the `value` is basically an empty string.
https://bugs.python.org/issue32178
(cherry picked from commit 09a1872a80)
Co-authored-by: Abhilash Raj <maxking@users.noreply.github.com>
This exception was caused because the input ended unexpectedly with only one
single quote instead of a pair with some value inside it.
(cherry picked from commit 719a062bcb)
Co-authored-by: Abhilash Raj <maxking@users.noreply.github.com>
* bpo-37461: Fix infinite loop in parsing of specially crafted email headers.
Some crafted email header would cause the get_parameter method to run in an
infinite loop causing a DoS attack surface when parsing those headers. This
patch fixes that by making sure the DQUOTE character is handled to prevent
going into an infinite loop.
(cherry picked from commit a4a994bd3e)
Co-authored-by: Abhilash Raj <maxking@users.noreply.github.com>
As far as I can tell, this infinite loop would be triggered if:
1. The value being folded contains a single word (no spaces) longer than
max_line_length
2. The max_line_length is shorter than the encoding's name + 9
characters.
bpo-36564: https://bugs.python.org/issue36564
(cherry picked from commit f69d5c6198)
Co-authored-by: Paul Ganssle <pganssle@users.noreply.github.com>
* patched string index out of range error in get_word function of _header_value_parser.py and created tests in test__header_value_parser.py for CFWS.
* Raise HeaderParseError instead of continuing when parsing a word.
(cherry picked from commit 7213df7bbf)
Co-authored-by: Abhilash Raj <maxking@users.noreply.github.com>
When certain malformed messages have content-type set to 'mulitpart/*' but
still have a single part body, iter_attachments can raise AttributeError. This
patch fixes it by returning a None value instead when the body is single part.
(cherry picked from commit 02257012f6)
Co-authored-by: Abhilash Raj <maxking@users.noreply.github.com>
https://bugs.python.org/issue33972
* [bpo-36520](https://bugs.python.org/issue36520): reset the encoded word offset when starting a new
line during an email header folding operation
* 📜🤖 Added by blurb_it.
* [bpo-36520](https://bugs.python.org/issue36520): add an additional test case, and provide descriptive
comments for the test_folding_with_utf8_encoding_* tests
* [bpo-36520](https://bugs.python.org/issue36520): fix whitespace issue
* [bpo-36520](https://bugs.python.org/issue36520): changes per reviewer request -- remove extraneous
backslashes; add whitespace between terminating quotes and
line-continuation backslashes; use "bpo-" instead of
"issue GH-" in comments
(cherry picked from commit f6713e84af)
Co-authored-by: websurfer5 <49998481+websurfer5@users.noreply.github.com>
https://bugs.python.org/issue36520
* bpo-30835: email: Fix AttributeError when parsing invalid Content-Transfer-Encoding
Parsing an email containing a multipart Content-Type, along with a
Content-Transfer-Encoding containing an invalid (non-ASCII-decodable) byte
will fail. email.feedparser.FeedParser._parsegen() gets the header and
attempts to convert it to lowercase before comparing it with the accepted
encodings, but as the header contains an invalid byte, it's returned as a
Header object rather than a str.
Cast the Content-Transfer-Encoding header to a str to avoid this.
Found using the AFL fuzzer.
Reported-by: Daniel Axtens <dja@axtens.net>
Signed-off-by: Andrew Donnellan <andrew@donnellan.id.au>
* Add email and NEWS entry for the bugfix.
* bpo-35805: Add parser for Message-ID header.
This parser is based on the definition of Identification Fields from RFC 5322
Sec 3.6.4.
This should also prevent folding of Message-ID header using RFC 2047 encoded
words and hence fix bpo-35805.
* Prevent folding of non-ascii message-id headers.
* Add fold method to MsgID token to prevent folding.
Fix mktime() overflow error in test_email: run
test_localtime_daylight_true_dst_true() and
test_localtime_daylight_false_dst_true() with a specific timezone.
When attempting to base64-decode a payload of invalid length (1 mod 4),
properly recognize and handle it. The given data will be returned as-is,
i.e. not decoded, along with a new defect, InvalidBase64LengthDefect.
The original algorithm tried to delegate the folding to the tokens so
that those tokens whose folding rules differed could specify the
differences. However, this resulted in a lot of duplicated code because
most of the rules were the same.
The new algorithm moves all folding logic into a set of functions
external to the token classes, but puts the information about which
tokens can be folded in which ways on the tokens...with the exception of
mime-parameters, which are a special case (which was not even
implemented in the old folder).
This algorithm can still probably be improved and hopefully simplified
somewhat.
Note that some of the test expectations are changed. I believe the
changes are toward more desirable and consistent behavior: in general
when (re) folding a line the canonical version of the tokens is
generated, rather than preserving errors or extra whitespace.
Leading whitespace was incorrectly dropped during folding of certain lines in the _header_value_parser's folding algorithm. This makes the whitespace handling code consistent.
It turns out we can't depend on email.message getting imported every place
message_factory is needed, so to avoid a circular import we need to special
case Policy.message_factory=None in the parser instead of using monkey
patching. I had a feeling that was a bad idea when I did it.
Miss Skeleton (bot)
Abhilash Raj
Ashwin Ramaswami
bsiem
Victor Stinner
Batuhan Taşkaya
Krzysztof Wojcik
Jens Troeger
Dong-hee Na
Tal Einat
jayyyin
R. David Murray
Antoine Pitrou
Joel Hillacre
mircea-cosbuc
Steve Dower
Berker Peksag
Martin Panter