This pull request fixes the newline conversion bug originally reported in bpo-1812. When that issue was originally submitted, the open builtin did not default to universal newline mode; now it does, which makes the issue fix simpler, since the only code path that needs to be changed is the one in doctest._load_testfile where the file is loaded from a package whose loader has a get_data method.
(cherry picked from commit e0b8101492)
Co-authored-by: Peter Donis <peterdonis@alum.mit.edu>
Hold reference of __bases__ tuple until tuple item is done with, because by
dropping the reference the item may be destroyed.
(cherry picked from commit 1c56f8ffad)
Co-authored-by: Yonatan Goldschmidt <yon.goldschmidt@gmail.com>
The regex http.cookiejar.LOOSE_HTTP_DATE_RE was vulnerable to regular
expression denial of service (REDoS).
LOOSE_HTTP_DATE_RE.match is called when using http.cookiejar.CookieJar
to parse Set-Cookie headers returned by a server.
Processing a response from a malicious HTTP server can lead to extreme
CPU usage and execution will be blocked for a long time.
The regex contained multiple overlapping \s* capture groups.
Ignoring the ?-optional capture groups the regex could be simplified to
\d+-\w+-\d+(\s*\s*\s*)$
Therefore, a long sequence of spaces can trigger bad performance.
Matching a malicious string such as
LOOSE_HTTP_DATE_RE.match("1-c-1" + (" " * 2000) + "!")
caused catastrophic backtracking.
The fix removes ambiguity about which \s* should match a particular
space.
You can create a malicious server which responds with Set-Cookie headers
to attack all python programs which access it e.g.
from http.server import BaseHTTPRequestHandler, HTTPServer
def make_set_cookie_value(n_spaces):
spaces = " " * n_spaces
expiry = f"1-c-1{spaces}!"
return f"b;Expires={expiry}"
class Handler(BaseHTTPRequestHandler):
def do_GET(self):
self.log_request(204)
self.send_response_only(204) GH- Don't bother sending Server and Date
n_spaces = (
int(self.path[1:]) GH- Can GET e.g. /100 to test shorter sequences
if len(self.path) > 1 else
65506 GH- Max header line length 65536
)
value = make_set_cookie_value(n_spaces)
for i in range(99): GH- Not necessary, but we can have up to 100 header lines
self.send_header("Set-Cookie", value)
self.end_headers()
if __name__ == "__main__":
HTTPServer(("", 44020), Handler).serve_forever()
This server returns 99 Set-Cookie headers. Each has 65506 spaces.
Extracting the cookies will pretty much never complete.
Vulnerable client using the example at the bottom of
https://docs.python.org/3/library/http.cookiejar.html :
import http.cookiejar, urllib.request
cj = http.cookiejar.CookieJar()
opener = urllib.request.build_opener(urllib.request.HTTPCookieProcessor(cj))
r = opener.open("http://localhost:44020/")
The popular requests library was also vulnerable without any additional
options (as it uses http.cookiejar by default):
import requests
requests.get("http://localhost:44020/")
* Regression test for http.cookiejar REDoS
If we regress, this test will take a very long time.
* Improve performance of http.cookiejar.ISO_DATE_RE
A string like
"444444" + (" " * 2000) + "A"
could cause poor performance due to the 2 overlapping \s* groups,
although this is not as serious as the REDoS in LOOSE_HTTP_DATE_RE was.
(cherry picked from commit 1b779bfb85)
Co-authored-by: bcaller <bcaller@users.noreply.github.com>
Relative imports use resolve_name to get the absolute target name,
which first seeks the current module's absolute package name from the globals:
If __package__ (and __spec__.parent) are missing then
import uses __name__, truncating the last segment if
the module is a submodule rather than a package __init__.py
(which it guesses from whether __path__ is defined).
The __name__ attempt should fail if there is no parent package (top level modules),
if __name__ is '__main__' (-m entry points), or both (scripts).
That is, if both __name__ has no subcomponents and the module does not seem
to be a package __init__ module then import should fail..
(cherry picked from commit 92420b3e67)
Co-authored-by: Ben Lewis <benjimin@users.noreply.github.com>
(cherry picked from commit 0a6693a469)
Co-authored-by: Brett Cannon <54418+brettcannon@users.noreply.github.com>
Modify the wheel event handler so it can also be used for module, path, and stack browsers.
Patch by George Zhang.
(cherry picked from commit 2cd9025858)
Co-authored-by: GeeTransit <geetransit@gmail.com>
…aders. (GH-15239)
Fixes a case in which email._header_value_parser.get_unstructured hangs the system for some invalid headers. This covers the cases in which the header contains either:
- a case without trailing whitespace
- an invalid encoded word
https://bugs.python.org/issue37764
This fix should also be backported to 3.7 and 3.8
https://bugs.python.org/issue37764
(cherry picked from commit c5b242f87f)
Co-authored-by: Ashwin Ramaswami <aramaswamis@gmail.com>
https://bugs.python.org/issue37764
This fixes an inconsistency between the Python and C implementations of
the datetime module. The pure python version of the code was not
accepting offsets greater than 23:59 but less than 24:00. This is an
accidental legacy of the original implementation, which was put in place
before tzinfo allowed sub-minute time zone offsets.
GH-14878
(cherry picked from commit 92c7e30adf)
FreeBSD implementation of poll(2) restricts the timeout argument to be
either zero, or positive, or equal to INFTIM (-1).
Unless otherwise overridden, socket timeout defaults to -1. This value
is then converted to milliseconds (-1000) and used as argument to the
poll syscall. poll returns EINVAL (22), and the connection fails.
This bug was discovered during the EINTR handling testing, and the
reproduction code can be found in
https://bugs.python.org/issue23618 (see connect_eintr.py,
attached). On GNU/Linux, the example runs as expected.
This change is trivial:
If the supplied timeout value is negative, truncate it to -1.
(cherry picked from commit 2814620657)
Co-authored-by: Artem Khramov <akhramov@pm.me>
Hi,
I've faced an issue w/ `mailbox.Maildir()`. The case is following:
1. I create a folder with `tempfile.TemporaryDirectory()`, so it's empty
2. I pass that folder path as an argument when instantiating `mailbox.Maildir()`
3. Then I receive an exception happening because "there's no such file or directory" (namely `cur`, `tmp` or `new`) during interaction with Maildir
**Expected result:** subdirs are created during `Maildir()` instance creation.
**Actual result:** subdirs are assumed as existing which leads to exceptions during use.
**Workaround:** remove the actual dir before passing the path to `Maildir()`. It will be created automatically with all subdirs needed.
**Fix:** This PR. Basically it adds creation of subdirs regardless of whether the base dir existed before.
https://bugs.python.org/issue30088
(cherry picked from commit e44184749c)
Co-authored-by: Sviatoslav Sydorenko <wk@sydorenko.org.ua>
* bpo-37014: Update docstring and Documentation of fileinput.FileInput()
* Explain the behavior of fileinput.FileInput() when reading stdin.
* Update blurb.
* bpo-37014: Fix typo in the docstring and documentation.
(cherry picked from commit aca273e240)
Co-authored-by: Michele Angrisano <michele.angrisano@gmail.com>
``_thread.interrupt_main()`` now avoids setting the Python error status if the ``SIGINT`` signal is ignored or not handled by Python.
(cherry picked from commit 608876b6b1)
Co-authored-by: Matěj Cepl <mcepl@cepl.eu>
* bpo-36929: Modify io/re tests to allow for missing mod name
For a vanishingly small number of internal types, CPython sets the
tp_name slot to mod_name.type_name, either in the PyTypeObject or the
PyType_Spec. There are a few minor places where this surfaces:
* Custom repr functions for those types (some of which ignore the
tp_name in favor of using a string literal, such as _io.TextIOWrapper)
* Pickling error messages
The test suite only tests the former. This commit modifies the test
suite to allow Python implementations to omit the module prefix.
https://bugs.python.org/issue36929
(cherry picked from commit ccb7ca728e)
Co-authored-by: Max Bernstein <tekknolagi@users.noreply.github.com>
This slightly expands an existing test case `test_popen_error` to trigger a `ResourceWarning` and fixes it.
https://bugs.python.org/issue35721
(cherry picked from commit 9932fd91e8)
Co-authored-by: Niklas Fiekas <niklas.fiekas@backscattering.de>
Extended attributes can only be set on user-writeable files, but shutil previously
first chmod()ed the destination file to the source's permissions and then tried to
copy xattrs. This will cause failures if attempting to copy read-only files with
xattrs, as occurs with Git clones on Lustre FS.
(cherry picked from commit 79efbb7193)
Co-authored-by: Olexa Bilaniuk <obilaniu@users.noreply.github.com>
Since bpo-30291 it is possible to specify the architecture of Python when using the launcher
(cherry picked from commit 1e2ad6c275)
Co-authored-by: mrh1997 <mrh1997@users.noreply.github.com>
This makes streamed zips compatible with MacOS Archive Utility and
other applications.
(cherry picked from commit 4ba3b50bfe)
Co-authored-by: Silas Sewell <silas@sewell.org>
Some methods of the SMTP class use mutable default arguments. Specially
`send_message` is affected as it mutates one of the args by appending items
to it, which has side effects on further calls.
(cherry picked from commit d5fbe9b1a3)
Co-authored-by: Pablo Aguiar <scorphus@gmail.com>
This code does not appear to be used anywhere in the python code base.
The use was removed in eb81795d7d.
(cherry picked from commit b91a3a0d61)
Co-authored-by: Eitan Adler <grimreaper@users.noreply.github.com>
Hangul composition check boundaries are wrong for the second character
([0x1161, 0x1176) instead of [0x1161, 0x1176]) and third character ((0x11A7, 0x11C3)
instead of [0x11A7, 0x11C3]).
(cherry picked from commit d134809cd3)
Co-authored-by: Wonsup Yoon <pusnow@me.com>
Before Python 3.6, os.path.abspath(None) used to report an AttributeError which was properly caught inside site.abs_paths, making it ignore __main__, one of sys.modules, which has __file__ and __cached__ set to None. With 3.6, os.path.abspath(None) raises TypeError instead which site.abs_path was not expecting. This resulted in an uncaught exception if a user had PYTHONSTARTUP set and the application called site.main() which a number of third-party programs do.
(cherry picked from commit 2487f30d55)
Co-authored-by: Steve Weber <steverweber@gmail.com>
uuid._ipconfig_getnode did not validate the maximum length of the value,
so long as the value had the same type of formatting as a MAC address.
This let it select DUIDs as MAC addresses. It now requires an exact
length match.
(cherry picked from commit c66c342cb4)
Co-authored-by: CtrlZvi <viz+github@flippedperspective.com>
The editline emulation needs to be initialized *after* the name is
defined. This fixes the long open issue.
(cherry picked from commit c2f082e9d1)
Co-authored-by: Zvezdan Petkovic <zpetkovic@acm.org>
This improves error message for situations when a non-class is
checked w.r.t. an abstract base class.
(cherry picked from commit 40472dd42d)
Co-authored-by: jab <jab@users.noreply.github.com>
Creating backup files with ~ suffix can be undesirable in some environment,
such as when building RPM packages. Instead of requiring the user to remove
those files manually, option -n was added, that simply disables this feature.
-n was selected because 2to3 has the same option with this behavior.
(cherry picked from commit 5affd5c29e)
Co-authored-by: Miro Hrončok <miro@hroncok.cz>
test_asyncio hangs indefinitely on macOS 10.13.2+ on `read_pty_output()`
using the KqueueSelector. Closing `proto.transport` (as is done in
`write_pty_output()`) seems to fix it.
(cherry picked from commit 12f74d8608)
Co-authored-by: Nathan Henrie <n8henrie@users.noreply.github.com>
* Prevent low-grade poplib REDOS (CVE-2018-1060)
The regex to test a mail server's timestamp is susceptible to
catastrophic backtracking on long evil responses from the server.
Happily, the maximum length of malicious inputs is 2K thanks
to a limit introduced in the fix for CVE-2013-1752.
A 2KB evil response from the mail server would result in small slowdowns
(milliseconds vs. microseconds) accumulated over many apop calls.
This is a potential DOS vector via accumulated slowdowns.
Replace it with a similar non-vulnerable regex.
The new regex is RFC compliant.
The old regex was non-compliant in edge cases.
* Prevent difflib REDOS (CVE-2018-1061)
The default regex for IS_LINE_JUNK is susceptible to
catastrophic backtracking.
This is a potential DOS vector.
Replace it with an equivalent non-vulnerable regex.
Also introduce unit and REDOS tests for difflib.
Co-authored-by: Tim Peters <tim.peters@gmail.com>
Co-authored-by: Christian Heimes <christian@python.org>
Co-authored-by: Jamie Davis <davisjam@vt.edu>
(cherry picked from commit 0e6c8ee235)
Make test.support.temp_cwd() fork-safe. The context manager test.support.temp_cwd() no longer removes the temporary directory when executing in a process other than the parent it entered from.
If a forked child exits the context manager it won't do the cleanup.
(cherry picked from commit 33dddac00b)
Co-authored-by: Anselm Kruis <a.kruis@science-computing.de>
Antoine Pitrou
Miss Islington (bot)
Éric Araujo
Ashwin Ramaswami
Paul Ganssle
Geoff Shannon
Steve Dower