parse_message_id() was improperly using a token defined inside an exception
handler, which was raising `UnboundLocalError` on parsing an invalid value.
https://bugs.python.org/issue38698
(cherry picked from commit bb815499af)
Co-authored-by: Claudiu Popa <pcmanticore@gmail.com>
Make ssl tests less strict and also accept TLSv3 as the default maximum
version. This change unbreaks test_min_max_version on Fedora 32.
https://bugs.python.org/issue38815
(cherry picked from commit 34864d1cff)
Co-authored-by: torsava <torsava@redhat.com>
SpooledTemporaryFile.rollback() might cause data corruption
when it is in text mode.
Co-Authored-By: Serhiy Storchaka <storchaka@gmail.com>
(cherry picked from commit ea9835c5d1)
Co-authored-by: Inada Naoki <songofacandy@gmail.com>
The regex http.cookiejar.LOOSE_HTTP_DATE_RE was vulnerable to regular
expression denial of service (REDoS).
LOOSE_HTTP_DATE_RE.match is called when using http.cookiejar.CookieJar
to parse Set-Cookie headers returned by a server.
Processing a response from a malicious HTTP server can lead to extreme
CPU usage and execution will be blocked for a long time.
The regex contained multiple overlapping \s* capture groups.
Ignoring the ?-optional capture groups the regex could be simplified to
\d+-\w+-\d+(\s*\s*\s*)$
Therefore, a long sequence of spaces can trigger bad performance.
Matching a malicious string such as
LOOSE_HTTP_DATE_RE.match("1-c-1" + (" " * 2000) + "!")
caused catastrophic backtracking.
The fix removes ambiguity about which \s* should match a particular
space.
You can create a malicious server which responds with Set-Cookie headers
to attack all python programs which access it e.g.
from http.server import BaseHTTPRequestHandler, HTTPServer
def make_set_cookie_value(n_spaces):
spaces = " " * n_spaces
expiry = f"1-c-1{spaces}!"
return f"b;Expires={expiry}"
class Handler(BaseHTTPRequestHandler):
def do_GET(self):
self.log_request(204)
self.send_response_only(204) GH- Don't bother sending Server and Date
n_spaces = (
int(self.path[1:]) GH- Can GET e.g. /100 to test shorter sequences
if len(self.path) > 1 else
65506 GH- Max header line length 65536
)
value = make_set_cookie_value(n_spaces)
for i in range(99): GH- Not necessary, but we can have up to 100 header lines
self.send_header("Set-Cookie", value)
self.end_headers()
if __name__ == "__main__":
HTTPServer(("", 44020), Handler).serve_forever()
This server returns 99 Set-Cookie headers. Each has 65506 spaces.
Extracting the cookies will pretty much never complete.
Vulnerable client using the example at the bottom of
https://docs.python.org/3/library/http.cookiejar.html :
import http.cookiejar, urllib.request
cj = http.cookiejar.CookieJar()
opener = urllib.request.build_opener(urllib.request.HTTPCookieProcessor(cj))
r = opener.open("http://localhost:44020/")
The popular requests library was also vulnerable without any additional
options (as it uses http.cookiejar by default):
import requests
requests.get("http://localhost:44020/")
* Regression test for http.cookiejar REDoS
If we regress, this test will take a very long time.
* Improve performance of http.cookiejar.ISO_DATE_RE
A string like
"444444" + (" " * 2000) + "A"
could cause poor performance due to the 2 overlapping \s* groups,
although this is not as serious as the REDoS in LOOSE_HTTP_DATE_RE was.
(cherry picked from commit 1b779bfb85)
Co-authored-by: bcaller <bcaller@users.noreply.github.com>
test_capi: trashcan tests now require the test "cpu" resource.
(cherry picked from commit 0127bb1c5c)
Co-authored-by: Victor Stinner <vstinner@python.org>
on platforms lacking a functional bind() for named unix domain sockets
https://bugs.python.org/issue38841
Automerge-Triggered-By: @asvetlov
(cherry picked from commit 559bad1a70)
Co-authored-by: xdegaye <xdegaye@gmail.com>
This PR implements a fix for `multiprocessing.Process` objects; the error occurs when Processes are created using either `fork` or `forkserver` as the `start_method`.
In these instances, the `MainThread` of the newly created `Process` object retains all attributes from its parent's `MainThread` object, including the `native_id` attribute. The resulting behavior is such that the new process' `MainThread` captures an incorrect/outdated `native_id` (the parent's instead of its own).
This change forces the Process object to update its `native_id` attribute during the bootstrap process.
cc @vstinner
https://bugs.python.org/issue38707
Automerge-Triggered-By: @pitrou
(cherry picked from commit c6b20be85c)
Co-authored-by: Jake Tesler <jake.tesler@gmail.com>
Ignore `GeneratorExit` exceptions when throwing an exception into the `aclose` coroutine of an asynchronous generator.
https://bugs.python.org/issue35409
(cherry picked from commit 8e0de2a480)
Co-authored-by: Vincent Michel <vxgmichel@gmail.com>
Ensure isabs() is always True for \\?\ prefixed paths
Avoid unnecessary usage of readlink() to avoid resolving broken links incorrectly
Ensure shutil tests run in test directory
if parent `__init__` is not called from a constructor of object derived from `asyncio.Future`
https://bugs.python.org/issue38785
(cherry picked from commit dad6be5ffe)
Co-authored-by: Andrew Svetlov <andrew.svetlov@gmail.com>
This change, which follows the behavior of C stdio's fdopen and Python 2's file object, allows pipes to be opened in append mode..
(cherry picked from commit 74fa9f723f)
* Add missing test class (mistake in GH-4455)
* Increase coverage with 4 more test cases
* Rename neg_uid to huge_uid in test_modified_uid_huge
* Replace test_main() with unittest.main()
* Update plistlib docs.
(cherry picked from commit d0d9f7cfa3)
Co-authored-by: Jon Janzen <jjjonjanzen@gmail.com>
bpo-34679, bpo-38563: asyncio.ProactorEventLoop.close() now only calls
signal.set_wakeup_fd() in the main thread.
(cherry picked from commit 1b53a24fb4)
Co-authored-by: Victor Stinner <vstinner@python.org>
The reverse iterator for empty dictionaries was not handling correctly shared-key dictionaries.
(cherry picked from commit 24dc2f8c56)
Co-authored-by: Dong-hee Na <donghee.na92@gmail.com>
* bpo-27657: Fix urlparse() with numeric paths
Revert parsing decision from bpo-754016 in favor of the documented
consensus in bpo-16932 of how to treat strings without a // to
designate the netloc.
* bpo-22891: Remove urlsplit() optimization for 'http' prefixed inputs.
(cherry picked from commit 5a88d50ff0)
Co-authored-by: Tim Graham <timograham@gmail.com>
Also updates the documentation to clarify the situation surrounding
the digestmod parameter that is required despite its position in the
argument list as of 3.8.0 as well as removing old python2 era
references to "binary strings".
We indavertently had this raise ValueError in 3.8.0 for the missing
arg. This is not considered an API change as no reasonable code would
be catching this missing argument error in order to handle it.
(cherry picked from commit f33c57d5c7)
Co-authored-by: Gregory P. Smith <greg@krypto.org>
main() is now responsible to send the ANSWER, rather than
ServerProto. main() now waits until it got the HELLO before sending
the ANSWER over the new transport.
Previously, there was a race condition between main() replacing the
protocol and the protocol sending the ANSWER once it gets the HELLO.
TLSv1.3 was disabled for the test: reenable it.
(cherry picked from commit fab4ef2df0)
Co-authored-by: Victor Stinner <vstinner@python.org>
* bpo-36389: _PyObject_CheckConsistency() available in release mode (GH-16612)
bpo-36389, bpo-38376: The _PyObject_CheckConsistency() function is
now also available in release mode. For example, it can be used to
debug a crash in the visit_decref() function of the GC.
Modify the following functions to also work in release mode:
* _PyDict_CheckConsistency()
* _PyObject_CheckConsistency()
* _PyType_CheckConsistency()
* _PyUnicode_CheckConsistency()
Other changes:
* _PyMem_IsPtrFreed(ptr) now also returns 1 if ptr is NULL
(equals to 0).
* _PyBytesWriter_CheckConsistency() now returns 1 and is only used
with assert().
* Reorder _PyObject_Dump() to write safe fields first, and only
attempt to render repr() at the end.
(cherry picked from commit 6876257eaa)
* bpo-36389: Fix _PyBytesWriter in release mode (GH-16624)
Fix _PyBytesWriter API when Python is built in release mode with
assertions.
(cherry picked from commit 60ec6efd96)
* bpo-38070: Enhance visit_decref() debug trace (GH-16631)
subtract_refs() now pass the parent object to visit_decref() which
pass it to _PyObject_ASSERT(). So if the "is freed" assertion fails,
the parent is used in debug trace, rather than the freed object. The
parent object is more likely to contain useful information. Freed
objects cannot be inspected are are displayed as "<object at xxx is
freed>" with no other detail.
(cherry picked from commit 4d5f94b8cd)
* Fix also a typo in PYMEM_DEADBYTE macro comment
* bpo-36389: Add newline to _PyObject_AssertFailed() (GH-16629)
Add a newline between the verbose object dump and the Py_FatalError()
logs for readability.
(cherry picked from commit 7775349895)
* [3.8] bpo-38456: Use /bin/true in test_subprocess (GH-16736)
Instead of sys.executable, "-c", "pass" or "import sys; sys.exit(0)"
use /bin/true when it is available. On a reasonable machine this
shaves up to two seconds wall time off the otherwise ~40sec execution
on a --with-pydebug build. It should be more notable on many
buildbots or overloaded slower I/O systems (CI, etc)..
(cherry picked from commit 67b93f80c7)
* Handle when there is no 'true' command
backport of 46113e0cf3 by Pablo Galindo.
The symbol table handing of PEP572's assignment expressions is not resolving correctly the scope of some variables in presence of global/nonlocal keywords in conjunction with comprehensions.
(cherry picked from commit fd5c414880)
Co-authored-by: Pablo Galindo <Pablogsal@gmail.com>
KeyError should cause a failure in parsing the encoded word and should be caught and raised as a _InvalidEWError instead.
(cherry picked from commit 65dcc8a8dc)
Co-authored-by: Andrei Troie <andreitroie90@gmail.com>
This reverts commit 87bd2071c7.
The reason for revert is a regression caused by the change in 3.8.0rc1, see bpo-38449 for more details.
https://bugs.python.org/issue38449
(cherry picked from commit 19a3d87300)
Co-authored-by: Abhilash Raj <maxking@users.noreply.github.com>
pathfix.py: Assume all files that end on '.py' are Python scripts when working recursively.
(cherry picked from commit 2b7dc40b2a)
Co-authored-by: Ruediger Pluem <r.pluem@gmx.de>
Add missing stat.S_IFDOOR, stat.S_IFPORT, stat.S_IFWHT,
stat.S_ISDOOR, stat.S_ISPORT, and stat.S_ISWHT values to
the Python implementation of the stat module.
(cherry picked from commit 7bb14316b8)
Co-authored-by: Ronan Lamy <ronan.lamy@gmail.com>
* [bpo-38379](https://bugs.python.org/issue38379): when a finalizer resurrects an object,
nothing is actually collected in this run of gc.
Change the stats to relect that truth.
(cherry picked from commit ecbf35f933)
Co-authored-by: Tim Peters <tim.peters@gmail.com>
https://bugs.python.org/issue38379
Automerge-Triggered-By: @pablogsal
bpo-37531, bpo-38207: On timeout, regrtest no longer attempts to call
`popen.communicate() again: it can hang until all child processes
using stdout and stderr pipes completes. Kill the worker process and
ignores its output.
Reenable test_regrtest.test_multiprocessing_timeout().
bpo-37531: Change also the faulthandler timeout of the main process
from 1 minute to 5 minutes, for Python slowest buildbots.
(cherry picked from commit 0ec618af98)
Co-authored-by: Victor Stinner <vstinner@python.org>
The implementation of weakref.proxy's methods call back into the Python
API using a borrowed references of the weakly referenced object
(acquired via PyWeakref_GET_OBJECT). This API call may delete the last
reference to the object (either directly or via GC), leaving a dangling
pointer, which can be subsequently dereferenced.
To fix this, claim a temporary ownership of the referenced object when
calling the appropriate method. Some functions because at the moment they
do not need to access the borrowed referent, but to protect against
future changes to these functions, ownership need to be fixed in
all potentially affected methods..
(cherry picked from commit 10cd00a9e3)
Co-authored-by: Pablo Galindo <Pablogsal@gmail.com>
On Windows use UTF-16 (or UTF-32 for 32-bit Tcl_UniChar) with the
"surrogatepass" error handler for converting to/from Tcl Unicode objects.
On Linux use UTF-8 with the "surrogateescape" error handler for converting
to/from Tcl String objects.
Converting strings from Tcl to Python and back now never fails
(except MemoryError).
(cherry picked from commit 06cb94bc84)
Co-authored-by: Serhiy Storchaka <storchaka@gmail.com>
Miss Islington (bot)
Steve Dower
Bruno P. Kinoshita
Benjamin Peterson
Jon Janzen
Senthil Kumaran
Victor Stinner
Gregory P. Smith
Abhilash Raj
Pablo Galindo