Commit Graph

785 Commits

Author SHA1 Message Date
Miss Islington (bot) 4aaa4259b5
[3.13] gh-121650: Encode newlines in headers, and verify headers are sound (GH-122233) (#122484)
gh-121650: Encode newlines in headers, and verify headers are sound (GH-122233)

GH-GH- Encode header parts that contain newlines

Per RFC 2047:

> [...] these encoding schemes allow the
> encoding of arbitrary octet values, mail readers that implement this
> decoding should also ensure that display of the decoded data on the
> recipient's terminal will not cause unwanted side-effects

It seems that the "quoted-word" scheme is a valid way to include
a newline character in a header value, just like we already allow
undecodable bytes or control characters.
They do need to be properly quoted when serialized to text, though.

GH-GH- Verify that email headers are well-formed

This should fail for custom fold() implementations that aren't careful
about newlines.

(cherry picked from commit 0976339818)

Co-authored-by: Petr Viktorin <encukou@gmail.com>
Co-authored-by: Bas Bloemsaat <bas@bloemsaat.org>
Co-authored-by: Serhiy Storchaka <storchaka@gmail.com>
2024-08-06 19:06:41 +02:00
Miss Islington (bot) 9a332f260d
[3.13] gh-120930: Remove extra blank occuring in wrapped encoded words in email headers (GH-121747) (GH-121963)
gh-120930: Remove extra blank occuring in wrapped encoded words in email headers (GH-121747)
(cherry picked from commit cecaceea31)

Co-authored-by: Matthieu Caneill <matthieucan@users.noreply.github.com>
2024-07-19 19:21:53 +02:00
Serhiy Storchaka a45d9051ed
[3.13] gh-121905: Consistently use "floating-point" instead of "floating point" (GH-121907) (GH-122012)
(cherry picked from commit 1a0c7b9ba4)
2024-07-19 09:13:08 +00:00
Miss Islington (bot) 6892b400dc
[3.13] gh-118643: Fix AttributeError in the email module (GH-119099) (GH-119389)
Fix regression introduced in gh-100884: AttributeError when re-fold a long
address list.

Also fix more cases of incorrect encoding of the address separator in the
address list missed in gh-100884.
(cherry picked from commit 858b9e85fc)

Co-authored-by: Serhiy Storchaka <storchaka@gmail.com>
2024-05-22 14:07:38 +03:00
Miss Islington (bot) 054f1af811
[3.13] gh-92081: Fix for email.generator.Generator with whitespace between encoded words. (GH-92281) (#119245)
* Fix for email.generator.Generator with whitespace between encoded words.

email.generator.Generator currently does not handle whitespace between
encoded words correctly when the encoded words span multiple lines.  The
current generator will create an encoded word for each line.  If the end
of the line happens to correspond with the end real word in the
plaintext, the generator will place an unencoded space at the start of
the subsequent lines to represent the whitespace between the plaintext
words.

A compliant decoder will strip all the whitespace from between two
encoded words which leads to missing spaces in the round-tripped
output.

The fix for this is to make sure that whitespace between two encoded
words ends up inside of one or the other of the encoded words.  This
fix places the space inside of the second encoded word.

A second problem happens with continuation lines.  A continuation line that
starts with whitespace and is followed by a non-encoded word is fine because
the newline between such continuation lines is defined as condensing to
a single space character.  When the continuation line starts with whitespace
followed by an encoded word, however, the RFCs specify that the word is run
together with the encoded word on the previous line.  This is because normal
words are filded on syntactic breaks by encoded words are not.

The solution to this is to add the whitespace to the start of the encoded word
on the continuation line.

Test cases are from GH-92081

* Rename a variable so it's not confused with the final variable.
(cherry picked from commit a6fdb31b67)

Co-authored-by: Toshio Kuratomi <a.badger@gmail.com>
2024-05-20 20:01:56 +00:00
wim glenn fed8d73fde
gh-118455: Fix mangle_from_ default value in email.policy.Policy.__doc__ (#118456)
* Fix mangle_from_ default value in email.policy.Policy.__doc__

The docstring says it defaults to True, but it actually defaults
to False. Only the Compat32 subclass overrides that.

---------

Co-authored-by: Nikita Sobolev <mail@sobolevn.me>
2024-05-05 09:18:04 +03:00
Serhiy Storchaka deaecb88fa
gh-80361: Fix TypeError in email.Message.get_payload() (GH-117994)
It was raised when the charset is rfc2231 encoded, e.g.:

   Content-Type: text/plain; charset*=ansi-x3.4-1968''utf-8
2024-04-17 19:31:26 +03:00
Ivan Savin 1aa8bbe62f
bpo-40944: Fix IndexError when parse emails with truncated Message-ID, address, routes, etc (GH-20790)
Co-authored-by: Serhiy Storchaka <storchaka@gmail.com>
2024-04-17 10:14:22 +00:00
Serhiy Storchaka aec1dac4ef
gh-117313: Fix re-folding email messages containing non-standard line separators (GH-117369)
Only treat '\n', '\r' and '\r\n' as line separators in re-folding the email
messages.  Preserve control characters '\v', '\f', '\x1c', '\x1d' and '\x1e'
and Unicode line separators '\x85', '\u2028' and '\u2029' as is.
2024-04-17 13:00:25 +03:00
Serhiy Storchaka f74e51229c
gh-86650: Fix IndexError when parse emails with invalid Message-ID (GH-117934)
In particularly, one-off addresses generated by Microsoft Outlook:
https://learn.microsoft.com/en-us/office/client-developer/outlook/mapi/one-off-addresses

Co-authored-by: fsc-eriker <72394365+fsc-eriker@users.noreply.github.com>
2024-04-17 10:44:41 +03:00
tsufeki 8cc9adbfdd
gh-75171: Fix parsing invalid email address headers starting or ending with a dot (GH-15600)
Co-authored-by: Tim Bell <timothybell@gmail.com>
Co-authored-by: Serhiy Storchaka <storchaka@gmail.com>
2024-04-17 10:39:15 +03:00
Serhiy Storchaka f97f25ef5d
gh-76511: Fix email.Message.as_string() for non-ASCII message with ASCII charset (GH-116125) 2024-03-05 17:49:01 +02:00
Thomas Weißschuh 09fab93c3d
gh-100884: email/_header_value_parser: don't encode list separators (GH-100885)
ListSeparator should not be encoded. This could happen when a long line
pushes its separator to the next line, which would have been encoded.
2024-02-17 10:13:46 +00:00
Shantanu 2124a3ddcc
gh-109653: Improve import time of importlib.metadata / email.utils (#114664)
My criterion for delayed imports is that they're only worth it if the
majority of users of the module would benefit from it, otherwise you're
just moving latency around unpredictably.

mktime_tz is not used anywhere in the standard library and grep.app
indicates it's not got much use in the ecosystem either.

Distribution.files is not nearly as widely used as other
importlib.metadata APIs, so we defer the csv import.

Before:
```
λ hyperfine -w 8 './python -c "import importlib.metadata"'
Benchmark 1: ./python -c "import importlib.metadata"
  Time (mean ± σ):      65.1 ms ±   0.5 ms    [User: 55.3 ms, System: 9.8 ms]
  Range (min … max):    64.4 ms …  66.4 ms    44 runs
```

After:
```
λ hyperfine -w 8 './python -c "import importlib.metadata"'
Benchmark 1: ./python -c "import importlib.metadata"
  Time (mean ± σ):      62.0 ms ±   0.3 ms    [User: 52.5 ms, System: 9.6 ms]
  Range (min … max):    61.3 ms …  62.8 ms    46 runs
```

for about a 3ms saving with warm disk cache, maybe 7-11ms with cold disk
cache.
2024-01-29 01:30:22 -08:00
Rito Takeuchi 504334c7be
gh-77749: Fix inconsistent behavior of non-ASCII handling in EmailPolicy.fold() (GH-6986)
It now always encodes non-ASCII characters in headers if utf8 is false.

Co-authored-by: Serhiy Storchaka <storchaka@gmail.com>
2024-01-26 15:19:41 +00:00
Serhiy Storchaka e9d5b6ea2d
gh-113594: Fix UnicodeEncodeError in TokenList.fold() (GH-113730)
It occurred when try to re-encode an unknown-8bit part combined with non-unknown-8bit part.
2024-01-10 14:54:36 +02:00
Victor Stinner 4a153a1d3b
[CVE-2023-27043] gh-102988: Reject malformed addresses in email.parseaddr() (#111116)
Detect email address parsing errors and return empty tuple to
indicate the parsing error (old API). Add an optional 'strict'
parameter to getaddresses() and parseaddr() functions. Patch by
Thomas Dwyer.

Co-Authored-By: Thomas Dwyer <github@tomd.tel>
2023-12-15 16:10:40 +01:00
Sidney Markowitz 27a5fd8cb8
gh-94606: Fix error when message with Unicode surrogate not surrogateescaped string (GH-94641)
Co-authored-by: Serhiy Storchaka <storchaka@gmail.com>
2023-12-11 18:21:18 +02:00
Alex Waygood aa3f419acb
gh-109653: Improve the import time of `email.utils` (#109824) 2023-10-12 15:03:20 -07:00
htsedebenham c65592c4d6
gh-106186: Don't report MultipartInvariantViolationDefect for valid multipart emails when parsing header only (#107016) 2023-07-23 12:25:18 +02:00
Gregory P. Smith a31dea1feb
gh-106669: Revert "gh-102988: Detect email address parsing errors ... (#105127)" (#106733)
This reverts commit 18dfbd0357.
Adds a regression test from the issue.

See https://github.com/python/cpython/issues/106669.
2023-07-20 20:30:52 -07:00
CF Bolz-Tereick 7e6ce48872
gh-106628: email parsing speedup (gh-106629) 2023-07-13 15:12:56 +09:00
Thomas Dwyer 18dfbd0357
gh-102988: Detect email address parsing errors and return empty tuple to indicate the parsing error (old API) (#105127)
Detect email address parsing errors and return empty tuple to indicate the parsing error (old API). This fixes or at least ameliorates CVE-2023-27043.

---------

Co-authored-by: Gregory P. Smith <greg@krypto.org>
2023-07-10 23:00:55 +00:00
JosephSBoyle 70e2a42647
gh-102542 Remove unused bytes object and bytes slicing (#106433)
Remove unused bytes object and bytes slicing

Co-authored-by: Shantanu <12621235+hauntsaninja@users.noreply.github.com>
2023-07-05 09:17:37 -07:00
Paul Ganssle 0b7fd8ffc5
GH-103857: Deprecate utcnow and utcfromtimestamp (#103858)
Using `datetime.datetime.utcnow()` and `datetime.datetime.utcfromtimestamp()` will now raise a `DeprecationWarning`.

We also have removed our internal uses of these functions and documented the change.
2023-04-27 11:32:30 -06:00
JosephSBoyle 04ea04807d
gh-102498 Clean up unused variables and imports in the email module (#102482)
* Clean up unused variables and imports in the email module

* Remove extra newline char

* Remove superflous dict+unpacking syntax

* Remove unused 'msg' var

* Clean up unused variables and imports in the email module

* Remove extra newline char

* Remove superflous dict+unpacking syntax

* Remove unused 'msg' var

---------

Co-authored-by: Barry Warsaw <barry@python.org>
2023-04-24 19:19:28 +00:00
Alan Williams 5e6661bce9
gh-72346: Added isdst deprecation warning to email.utils.localtime (GH-91450) 2023-03-19 19:20:20 -05:00
JosephSBoyle b097925858
gh-102507 Remove invisible pagebreak characters (#102531)
Co-authored-by: AlexWaygood <alex.waygood@gmail.com>
2023-03-08 13:58:14 +00:00
Bob Kline 49cae39ef0
gh-101021: Document binary parameters as bytes (#101024) 2023-01-14 11:01:27 -08:00
Nikita Sobolev 6746135b07
gh-100792: Make `email.message.Message.__contains__` twice as fast (#100793) 2023-01-07 13:26:05 -08:00
Nick Drozd 024ac542d7
bpo-45975: Simplify some while-loops with walrus operator (GH-29347) 2022-11-26 14:33:25 -08:00
Gary Donovan 5d4d83130c
Fix typo on inline comment for email.generator (GH-98210)
Trivial change to comment - no issue or new entry necessary
2022-11-25 10:03:20 -08:00
Serhiy Storchaka ea5ed0ba51
gh-95087: Fix IndexError in parsing invalid date in the email module (GH-95201)
Co-authored-by: wouter bolsterlee <wouter@bolsterl.ee>
2022-07-25 09:17:25 +03:00
oda-gitso 71abeb0895
gh-93010: InvalidHeaderError used but nonexistent (#93015)
* fix issue 93010

Co-authored-by: blurb-it[bot] <43283697+blurb-it[bot]@users.noreply.github.com>
2022-05-23 09:10:18 -07:00
slateny 8f29318079
gh-77630: Change Charset to charset (GH-92439) 2022-05-08 17:35:32 +03:00
Serhiy Storchaka e91dee87ed
bpo-43323: Fix UnicodeEncodeError in the email module (GH-32137)
It was raised if the charset itself contains characters not encodable
in UTF-8 (in particular \udcxx characters representing non-decodable
bytes in the source).
2022-04-30 13:17:23 +03:00
Brett Cannon 407c3afe19
gh-91217: deprecate uu (GH-92009)
Automerge-Triggered-By: GH:brettcannon
2022-04-27 20:26:33 -07:00
Barry Warsaw 4403320727
Rewrite audio.py to jive with image.py (#91886)
Similar to the rewrite of email/mime/image.py and associated test after the
deprecation of imghdr.py, thisrewrites email/mime/audio.py and associated
tests after the deprecation of sndhdr.py.

Closes #91885
2022-04-24 15:50:07 -07:00
Brett Cannon e7929cba16
gh-91217: deprecate-sndhdr (#91806)
Also inline necessary functionality from `sndhdr` into `email.mime.audio` for `MIMEAudio`.

Co-authored-by: Hugo van Kemenade <hugovk@users.noreply.github.com>
2022-04-22 15:48:03 -07:00
Barry Warsaw 1fcb39ea64
gh-91520: Rewrite imghdr inlining for clarity and completeness (#91521)
* Rewrite imghdr inlining for clarity and completeness

* Move MIMEImage class back closer to the top of the file since it's the
  important thing.
* Use a decorate to mark a given rule function and simplify the rule function
  names for clarity.
* Copy over all the imghdr test data files into the email package's test data
  directory.  This way when imghdr is actually removed, it won't affect the
  MIMEImage guessing tests.
* Rewrite and extend the MIMEImage tests to test for all supported
  auto-detected MIME image subtypes.
* Remove the now redundant PyBanner048.gif data file.

* See https://github.com/python/cpython/pull/91461#discussion_r850313336

Co-authored-by: Oleg Iarygin <dralife@yandex.ru>

Co-authored-by: Oleg Iarygin <dralife@yandex.ru>
2022-04-14 17:48:59 -07:00
Brett Cannon 3fc57e8f6f
gh-91217: deprecate imghdr (#91461)
* Deprecate imghdr

Co-authored-by: Hugo van Kemenade <hugovk@users.noreply.github.com>

* Update Doc/whatsnew/3.11.rst

Co-authored-by: Hugo van Kemenade <hugovk@users.noreply.github.com>

* Inline `imghdr` into `email.mime.image`

Co-authored-by: Hugo van Kemenade <hugovk@users.noreply.github.com>
Co-authored-by: Barry Warsaw <barry@python.org>
2022-04-13 10:47:41 -07:00
Serhiy Storchaka 884eba3c76
bpo-26579: Add object.__getstate__(). (GH-2821)
Copying and pickling instances of subclasses of builtin types
bytearray, set, frozenset, collections.OrderedDict, collections.deque,
weakref.WeakSet, and datetime.tzinfo now copies and pickles instance attributes
implemented as slots.
2022-04-06 20:00:14 +03:00
Nikita Sobolev 0cbdd21311
bpo-46565: `del` loop vars that are leaking into module namespaces (GH-30993) 2022-02-03 11:20:08 +02:00
Ben Hoyt b9e687618d
bpo-45239: Fix parsedate_tz when time has more than 2 dots in it (GH-28452)
Co-authored-by: Łukasz Langa <lukasz@langa.pl>
2021-10-13 18:21:27 +02:00
Christian Clauss 745c9d9dfc
Fix typos in the Lib directory (GH-28775)
Fix typos in the Lib directory as identified by codespell.

Co-authored-by: Terry Jan Reedy <tjreedy@udel.edu>
2021-10-06 16:13:48 -07:00
Serhiy Storchaka 3c65457156
bpo-45060: Get rid of few uses of the equality operators with None (GH-28087) 2021-08-31 16:59:52 +03:00
wouter bolsterlee 989f6a3800
bpo-45001: Make email date parsing more robust against malformed input (GH-27946)
Various date parsing utilities in the email module, such as
email.utils.parsedate(), are supposed to gracefully handle invalid
input, typically by raising an appropriate exception or by returning
None.

The internal email._parseaddr._parsedate_tz() helper used by some of
these date parsing routines tries to be robust against malformed input,
but unfortunately it can still crash ungracefully when a non-empty but
whitespace-only input is passed. This manifests as an unexpected
IndexError.

In practice, this can happen when parsing an email with only a newline
inside a ‘Date:’ header, which unfortunately happens occasionally in the
real world.

Here's a minimal example:

    $ python
    Python 3.9.6 (default, Jun 30 2021, 10:22:16)
    [GCC 11.1.0] on linux
    Type "help", "copyright", "credits" or "license" for more information.
    >>> import email.utils
    >>> email.utils.parsedate('foo')
    >>> email.utils.parsedate(' ')
    Traceback (most recent call last):
      File "<stdin>", line 1, in <module>
      File "/usr/lib/python3.9/email/_parseaddr.py", line 176, in parsedate
        t = parsedate_tz(data)
      File "/usr/lib/python3.9/email/_parseaddr.py", line 50, in parsedate_tz
        res = _parsedate_tz(data)
      File "/usr/lib/python3.9/email/_parseaddr.py", line 72, in _parsedate_tz
        if data[0].endswith(',') or data[0].lower() in _daynames:
    IndexError: list index out of range

The fix is rather straight-forward: guard against empty lists, after
splitting on whitespace, but before accessing the first element.
2021-08-26 16:49:03 +02:00
Johannes Reiff b33186bc43
bpo-41402: Fix email ContentManager calling .encode() on bytes (GH-21631) 2021-08-09 18:45:41 +02:00
andrei kulakov e3f877c32d
bpo-42892: fix email multipart attribute error (GH-26903) 2021-07-30 19:05:49 +02:00
Zackery Spytz 89f4c34797
bpo-27513: email.utils.getaddresses() now handles Header objects (#13797)
getaddresses() should be able to handle a Header object if passed
one.

Co-authored-by: Łukasz Langa <lukasz@langa.pl>
2021-07-19 18:07:54 +02:00