The header-folder of the new email API has a long standing known buglet where
if the first token is longer than max_line_length, it puts that token on the next
line. It turns out there is also a *parsing* bug when parsing such a header:
the space prefixing that first, non-empty line gets preserved and tacked on to
the start of the header value, which is not the expected behavior per the RFCs.
The bug arises from the fact that the parser assumed that there would be at
least one token on the line with the header, which is going to be true for
probably every email producer other than the python email library with its
folding buglet. Clearly, though, this is a case that needs to be handled
correctly. The fix is simple: strip the blanks off the start of the whole
value, not just the first physical line of the value.
Co-authored-by: blurb-it[bot] <43283697+blurb-it[bot]@users.noreply.github.com>
Co-authored-by: Bénédikt Tran <10796600+picnixz@users.noreply.github.com>
## Encode header parts that contain newlines
Per RFC 2047:
> [...] these encoding schemes allow the
> encoding of arbitrary octet values, mail readers that implement this
> decoding should also ensure that display of the decoded data on the
> recipient's terminal will not cause unwanted side-effects
It seems that the "quoted-word" scheme is a valid way to include
a newline character in a header value, just like we already allow
undecodable bytes or control characters.
They do need to be properly quoted when serialized to text, though.
## Verify that email headers are well-formed
This should fail for custom fold() implementations that aren't careful
about newlines.
Co-authored-by: Bas Bloemsaat <bas@bloemsaat.org>
Co-authored-by: Serhiy Storchaka <storchaka@gmail.com>
As reported in #117847 and #115366, an unpaired backtick in a docstring
tends to confuse e.g. Sphinx running on subclasses of standard library
objects, and the typographic style of using a backtick as an opening
quote is no longer in favor. Convert almost all uses of the form
The variable `foo' should do xyz
to
The variable 'foo' should do xyz
and also fix up miscellaneous other unpaired backticks (extraneous /
missing characters).
No functional change is intended here other than in human-readable
docstrings.
Fix regression introduced in gh-100884: AttributeError when re-fold a long
address list.
Also fix more cases of incorrect encoding of the address separator in the
address list missed in gh-100884.
* Fix for email.generator.Generator with whitespace between encoded words.
email.generator.Generator currently does not handle whitespace between
encoded words correctly when the encoded words span multiple lines. The
current generator will create an encoded word for each line. If the end
of the line happens to correspond with the end real word in the
plaintext, the generator will place an unencoded space at the start of
the subsequent lines to represent the whitespace between the plaintext
words.
A compliant decoder will strip all the whitespace from between two
encoded words which leads to missing spaces in the round-tripped
output.
The fix for this is to make sure that whitespace between two encoded
words ends up inside of one or the other of the encoded words. This
fix places the space inside of the second encoded word.
A second problem happens with continuation lines. A continuation line that
starts with whitespace and is followed by a non-encoded word is fine because
the newline between such continuation lines is defined as condensing to
a single space character. When the continuation line starts with whitespace
followed by an encoded word, however, the RFCs specify that the word is run
together with the encoded word on the previous line. This is because normal
words are filded on syntactic breaks by encoded words are not.
The solution to this is to add the whitespace to the start of the encoded word
on the continuation line.
Test cases are from #92081
* Rename a variable so it's not confused with the final variable.
* Fix mangle_from_ default value in email.policy.Policy.__doc__
The docstring says it defaults to True, but it actually defaults
to False. Only the Compat32 subclass overrides that.
---------
Co-authored-by: Nikita Sobolev <mail@sobolevn.me>
Only treat '\n', '\r' and '\r\n' as line separators in re-folding the email
messages. Preserve control characters '\v', '\f', '\x1c', '\x1d' and '\x1e'
and Unicode line separators '\x85', '\u2028' and '\u2029' as is.
My criterion for delayed imports is that they're only worth it if the
majority of users of the module would benefit from it, otherwise you're
just moving latency around unpredictably.
mktime_tz is not used anywhere in the standard library and grep.app
indicates it's not got much use in the ecosystem either.
Distribution.files is not nearly as widely used as other
importlib.metadata APIs, so we defer the csv import.
Before:
```
λ hyperfine -w 8 './python -c "import importlib.metadata"'
Benchmark 1: ./python -c "import importlib.metadata"
Time (mean ± σ): 65.1 ms ± 0.5 ms [User: 55.3 ms, System: 9.8 ms]
Range (min … max): 64.4 ms … 66.4 ms 44 runs
```
After:
```
λ hyperfine -w 8 './python -c "import importlib.metadata"'
Benchmark 1: ./python -c "import importlib.metadata"
Time (mean ± σ): 62.0 ms ± 0.3 ms [User: 52.5 ms, System: 9.6 ms]
Range (min … max): 61.3 ms … 62.8 ms 46 runs
```
for about a 3ms saving with warm disk cache, maybe 7-11ms with cold disk
cache.
Detect email address parsing errors and return empty tuple to
indicate the parsing error (old API). Add an optional 'strict'
parameter to getaddresses() and parseaddr() functions. Patch by
Thomas Dwyer.
Co-Authored-By: Thomas Dwyer <github@tomd.tel>
Detect email address parsing errors and return empty tuple to indicate the parsing error (old API). This fixes or at least ameliorates CVE-2023-27043.
---------
Co-authored-by: Gregory P. Smith <greg@krypto.org>
Using `datetime.datetime.utcnow()` and `datetime.datetime.utcfromtimestamp()` will now raise a `DeprecationWarning`.
We also have removed our internal uses of these functions and documented the change.
* Clean up unused variables and imports in the email module
* Remove extra newline char
* Remove superflous dict+unpacking syntax
* Remove unused 'msg' var
* Clean up unused variables and imports in the email module
* Remove extra newline char
* Remove superflous dict+unpacking syntax
* Remove unused 'msg' var
---------
Co-authored-by: Barry Warsaw <barry@python.org>
It was raised if the charset itself contains characters not encodable
in UTF-8 (in particular \udcxx characters representing non-decodable
bytes in the source).
Similar to the rewrite of email/mime/image.py and associated test after the
deprecation of imghdr.py, thisrewrites email/mime/audio.py and associated
tests after the deprecation of sndhdr.py.
Closes#91885
Also inline necessary functionality from `sndhdr` into `email.mime.audio` for `MIMEAudio`.
Co-authored-by: Hugo van Kemenade <hugovk@users.noreply.github.com>
* Rewrite imghdr inlining for clarity and completeness
* Move MIMEImage class back closer to the top of the file since it's the
important thing.
* Use a decorate to mark a given rule function and simplify the rule function
names for clarity.
* Copy over all the imghdr test data files into the email package's test data
directory. This way when imghdr is actually removed, it won't affect the
MIMEImage guessing tests.
* Rewrite and extend the MIMEImage tests to test for all supported
auto-detected MIME image subtypes.
* Remove the now redundant PyBanner048.gif data file.
* See https://github.com/python/cpython/pull/91461#discussion_r850313336
Co-authored-by: Oleg Iarygin <dralife@yandex.ru>
Co-authored-by: Oleg Iarygin <dralife@yandex.ru>
* Deprecate imghdr
Co-authored-by: Hugo van Kemenade <hugovk@users.noreply.github.com>
* Update Doc/whatsnew/3.11.rst
Co-authored-by: Hugo van Kemenade <hugovk@users.noreply.github.com>
* Inline `imghdr` into `email.mime.image`
Co-authored-by: Hugo van Kemenade <hugovk@users.noreply.github.com>
Co-authored-by: Barry Warsaw <barry@python.org>
Copying and pickling instances of subclasses of builtin types
bytearray, set, frozenset, collections.OrderedDict, collections.deque,
weakref.WeakSet, and datetime.tzinfo now copies and pickles instance attributes
implemented as slots.
RanKKI
Hugo van Kemenade
Damien
Petr Viktorin
Serhiy Storchaka
Matthieu Caneill
Geoffrey Thomas
Toshio Kuratomi
wim glenn
Ivan Savin
tsufeki
Thomas Weißschuh
Shantanu
Rito Takeuchi
Victor Stinner
Sidney Markowitz
Alex Waygood
htsedebenham
Gregory P. Smith
CF Bolz-Tereick
Thomas Dwyer
JosephSBoyle
Paul Ganssle
Alan Williams
Bob Kline
Nikita Sobolev
Nick Drozd
Gary Donovan
oda-gitso
slateny
Brett Cannon
Barry Warsaw
Ben Hoyt
Christian Clauss