Each Python subinterpreter now has its own "small integer
singletons": numbers in [-5; 257] range.
It is no longer possible to change the number of small integers at
build time by overriding NSMALLNEGINTS and NSMALLPOSINTS macros:
macros should now be modified manually in pycore_pystate.h header
file.
For now, continue to share _PyLong_Zero and _PyLong_One singletons
between all subinterpreters.
Commit 6b5b013bcc ("bpo-26978: Implement pathlib.Path.link_to (Using
os.link) (GH-12990)") introduced a new link_to method in pathlib. However,
this makes pathlib crash when the 'os' module is missing a 'link' method.
Fix this by checking for the presence of the 'link' method on pathlib
module import, and if it's not present, turn it into a runtime error like
those emitted when there is no lchmod() or symlink().
Signed-off-by: Toke Høiland-Jørgensen <toke@redhat.com>
Provides a richer platform tag for AIX that we expect to be sufficient for PEP 425
binary distribution identification. Any backports to earlier Python versions will be
handled via setuptools.
Patch by Michael Felt.
When parsing an "elif" node, lineno and col_offset of the node now point to the "elif" keyword and not to its condition, making it consistent with the "if" node.
https://bugs.python.org/issue39031
Automerge-Triggered-By: @pablogsal
Replace hardcoded timeout constants in tests with SHORT_TIMEOUT of
test.support, so it's easier to ajdust this timeout for all tests at
once.
SHORT_TIMEOUT is 30 seconds by default, but it can be longer
depending on --timeout command line option.
The change makes almost all timeouts longer, except
test_reap_children() of test_support which is made 2x shorter:
SHORT_TIMEOUT should be enough. If this test starts to fail,
LONG_TIMEOUT should be used instead.
Uniformize also "from test import support" import in some test files.
* bpo-39022, bpo-38594: Sync with importlib_metadata 1.3 including improved docs for custom finders and better serialization support in EntryPoints.
* 📜🤖 Added by blurb_it.
* Correct module reference
In Python 3.9.0a1, sys.argv[0] was made an asolute path if a filename
was specified on the command line. Revert this change, since most
users expect sys.argv to be unmodified.
Fix test_pty: if the process is the session leader, closing the
master file descriptor raises a SIGHUP signal: simply ignore SIGHUP
when running the tests.
test_openssl_version now accepts version 3.0.0.
getpeercert() no longer returns IPv6 addresses with a trailing new line.
Signed-off-by: Christian Heimes <christian@python.org>
https://bugs.python.org/issue38820
On most platforms, the `environ` symbol is accessible everywhere.
In a dylib on OSX, it's not easily accessible, you need to find it with
_NSGetEnviron.
The code was caching the *value* of environ. But a setenv() can change the value,
leaving garbage at the old value. Fix: don't cache the value of environ, just
read it every time.
If waitpid() is called elsewhere, waitpid() call fails with
ChildProcessError: use return code 255 in this case, and log a
warning. It ensure that the pidfd file descriptor is closed if this
error occurs.
Break cycle generated when saving an exception in socket.py, codeop.py and dyld.py as they keep alive not only the exception but user objects through the ``__traceback__`` attribute.
https://bugs.python.org/issue36820
Automerge-Triggered-By: @pablogsal
parse_message_id() was improperly using a token defined inside an exception
handler, which was raising `UnboundLocalError` on parsing an invalid value.
https://bugs.python.org/issue38698
The readline module now detects if Python is linked to libedit at runtime
on all platforms. Previously, the check was only done on macOS.
If Python is used as a library by a binary linking to libedit, the linker
resolves the rl_initialize symbol required by the readline module against
libedit instead of libreadline, which leads to a segfault.
Take advantage of the existing supporting code to have readline module being
compatible with both situations.
This fixes the issue discussed in https://bugs.python.org/issue22377
and fixes it according to the comments made by Paul Ganssle @pganssle
* It clarifies which values are acceptable in the table
* It extends the note with a clearer information on the valid values
https://bugs.python.org/issue22377
* bpo-20928: bring elementtree's XInclude support en-par with the implementation in lxml by adding support for recursive includes and a base-URL.
* bpo-20928: Support xincluding the same file multiple times, just not recursively.
* bpo-20928: Add 'max_depth' parameter to xinclude that limits the maximum recursion depth to 6 by default.
* Add news entry for updated ElementInclude support
Add ast.unparse() as a function in the ast module that can be used to unparse an
ast.AST object and produce a string with code that would produce an equivalent ast.AST
object when parsed.
Extra newlines are removed at the end of non-shell files. If the file only has newlines after stripping other trailing whitespace, all are removed, as is done by patchcheck.py.
The previous code was raising a `KeyError` for both the Python and C implementation.
This was caused by the specified index of an invalid input which did not exist
in the memo structure, where the pickle stores what objects it has seen.
The malformed input would have caused either a `BINGET` or `LONG_BINGET` load
from the memo, leading to a `KeyError` as the determined index was bogus.
https://bugs.python.org/issue38876https://bugs.python.org/issue38876
This patch enables downstream projects inspecting a TypedDict subclass at runtime to tell which keys are optional.
This is essential for generating test data with Hypothesis or validating inputs with typeguard or pydantic.
* fix HTTP Digest handling in request.py
There is a bug triggered when server replies to a request with `WWW-Authenticate: Digest` where `qop="auth,auth-int"` rather than mere `qop="auth"`. Having both `auth` and `auth-int` is legitimate according to the `qop-options` rule in §3.2.1 of [[https://www.ietf.org/rfc/rfc2617.txt|RFC 2617]]:
> qop-options = "qop" "=" <"> 1#qop-value <">
> qop-value = "auth" | "auth-int" | token
> **qop-options**: [...] If present, it is a quoted string **of one or more** tokens indicating the "quality of protection" values supported by the server. The value `"auth"` indicates authentication; the value `"auth-int"` indicates authentication with integrity protection
This is description confirmed by the definition of the [//n//]`#`[//m//]//rule// extended-BNF pattern defined in §2.1 of [[https://www.ietf.org/rfc/rfc2616.txt|RFC 2616]] as 'a comma-separated list of //rule// with at least //n// and at most //m// items'.
When this reply is parsed by `get_authorization`, request.py only tests for identity with `'auth'`, failing to recognize it as one of the supported modes the server announced, and claims that `"qop 'auth,auth-int' is not supported"`.
* 📜🤖 Added by blurb_it.
* bpo-38686 review fix: remember why.
* fix trailing space in Lib/urllib/request.py
Co-Authored-By: Brandt Bucher <brandtbucher@gmail.com>
The regex http.cookiejar.LOOSE_HTTP_DATE_RE was vulnerable to regular
expression denial of service (REDoS).
LOOSE_HTTP_DATE_RE.match is called when using http.cookiejar.CookieJar
to parse Set-Cookie headers returned by a server.
Processing a response from a malicious HTTP server can lead to extreme
CPU usage and execution will be blocked for a long time.
The regex contained multiple overlapping \s* capture groups.
Ignoring the ?-optional capture groups the regex could be simplified to
\d+-\w+-\d+(\s*\s*\s*)$
Therefore, a long sequence of spaces can trigger bad performance.
Matching a malicious string such as
LOOSE_HTTP_DATE_RE.match("1-c-1" + (" " * 2000) + "!")
caused catastrophic backtracking.
The fix removes ambiguity about which \s* should match a particular
space.
You can create a malicious server which responds with Set-Cookie headers
to attack all python programs which access it e.g.
from http.server import BaseHTTPRequestHandler, HTTPServer
def make_set_cookie_value(n_spaces):
spaces = " " * n_spaces
expiry = f"1-c-1{spaces}!"
return f"b;Expires={expiry}"
class Handler(BaseHTTPRequestHandler):
def do_GET(self):
self.log_request(204)
self.send_response_only(204) # Don't bother sending Server and Date
n_spaces = (
int(self.path[1:]) # Can GET e.g. /100 to test shorter sequences
if len(self.path) > 1 else
65506 # Max header line length 65536
)
value = make_set_cookie_value(n_spaces)
for i in range(99): # Not necessary, but we can have up to 100 header lines
self.send_header("Set-Cookie", value)
self.end_headers()
if __name__ == "__main__":
HTTPServer(("", 44020), Handler).serve_forever()
This server returns 99 Set-Cookie headers. Each has 65506 spaces.
Extracting the cookies will pretty much never complete.
Vulnerable client using the example at the bottom of
https://docs.python.org/3/library/http.cookiejar.html :
import http.cookiejar, urllib.request
cj = http.cookiejar.CookieJar()
opener = urllib.request.build_opener(urllib.request.HTTPCookieProcessor(cj))
r = opener.open("http://localhost:44020/")
The popular requests library was also vulnerable without any additional
options (as it uses http.cookiejar by default):
import requests
requests.get("http://localhost:44020/")
* Regression test for http.cookiejar REDoS
If we regress, this test will take a very long time.
* Improve performance of http.cookiejar.ISO_DATE_RE
A string like
"444444" + (" " * 2000) + "A"
could cause poor performance due to the 2 overlapping \s* groups,
although this is not as serious as the REDoS in LOOSE_HTTP_DATE_RE was.
is_cgi() function of http.server library does not currently handle a
cgi script if one of the cgi_directories is located at the
sub-directory of given path. Since is_cgi() in CGIHTTPRequestHandler
class separates given path into (dir, rest) based on the first seen
'/', multi-level directories like /sub/dir/cgi-bin/hello.py is divided
into head=/sub, rest=dir/cgi-bin/hello.py then check whether '/sub'
exists in cgi_directories = [..., '/sub/dir/cgi-bin'].
This patch makes the is_cgi() keep expanding dir part to the next '/'
then checking if that expanded path exists in the cgi_directories.
Signed-off-by: Siwon Kang <kkangshawn@gmail.com>
https://bugs.python.org/issue38863
Skip the test_posix.test_pidfd_open() test if os.pidfd_open() fails
with a PermissionError. This situation can happen in a Linux sandbox
using a syscall whitelist which doesn't allow the pidfd_open()
syscall yet (like systemd-nspawn).
Remove BEGIN_FINALLY, END_FINALLY, CALL_FINALLY and POP_FINALLY bytecodes. Implement finally blocks by code duplication.
Reimplement frame.lineno setter using line numbers rather than bytecode offsets.
Note that the support is not actually enabled yet, and so we won't be publishing these packages. However, for those who want to build it themselves (even by reusing the Azure Pipelines definition), it's now relatively easy to enable.
Remove PyMethod_ClearFreeList() and PyCFunction_ClearFreeList()
functions: the free lists of bound method objects have been removed.
Remove also _PyMethod_Fini() and _PyCFunction_Fini() functions.
This exposes a Linux-specific syscall for sending a signal to a process
identified by a file descriptor rather than a pid.
For simplicity, we don't support the siginfo_t parameter to the syscall. This
parameter allows implementing a pidfd version of rt_sigqueueinfo(2), which
Python also doesn't support.
The PyFPE_START_PROTECT() and PyFPE_END_PROTECT() macros are empty:
they have been doing nothing for the last year (since commit
735ae8d139), so stop using them.
When building Python in some uncommon platforms there are some known tests that will fail. Right now, the test suite has the ability to ignore entire tests using the -x option and to receive a filter file using the --matchfile filter. The problem with the --matchfile option is that it receives a file with patterns to accept and when you want to ignore a couple of tests and subtests, is too cumbersome to lists ALL tests that are not the ones that you want to accept and he problem with -x is that is not easy to ignore just a subtests that fail and the whole test needs to be ignored.
For these reasons, add a new option to allow to ignore a list of test and subtests for these situations.
This PR implements a fix for `multiprocessing.Process` objects; the error occurs when Processes are created using either `fork` or `forkserver` as the `start_method`.
In these instances, the `MainThread` of the newly created `Process` object retains all attributes from its parent's `MainThread` object, including the `native_id` attribute. The resulting behavior is such that the new process' `MainThread` captures an incorrect/outdated `native_id` (the parent's instead of its own).
This change forces the Process object to update its `native_id` attribute during the bootstrap process.
cc @vstinner
https://bugs.python.org/issue38707
Automerge-Triggered-By: @pitrou
Ignore `GeneratorExit` exceptions when throwing an exception into the `aclose` coroutine of an asynchronous generator.
https://bugs.python.org/issue35409
The C-API docs are a bit sparse on the interplay between C `fork()` and the CPython runtime. This change adds some more information on the subject.
https://bugs.python.org/issue38816
Ensure isabs() is always True for \\?\ prefixed paths
Avoid unnecessary usage of readlink() to avoid resolving broken links incorrectly
Ensure shutil tests run in test directory
Small docs update for [bpo-34651](https://bugs.python.org/issue34651).
Other references to fork (e.g. the PyOS.*Fork functions or discussions of fork() when embedding Python) point back to os.fork, so I don't think any other updates are needed.
https://bugs.python.org/issue38778
Automerge-Triggered-By: @ericsnowcurrently
Additional note: the `method_check_args` function in `Objects/descrobject.c` is written in such a way that it applies to all kinds of descriptors. In particular, a future re-implementation of `wrapper_descriptor` could use that code.
CC @vstinner @encukou
https://bugs.python.org/issue37645
Automerge-Triggered-By: @encukou
After #9665, this moves the remaining types in posixmodule to be heap-allocated to make it compatible with PEP384 as well as modifying all the type accessors to fully make the type opaque.
The original PR that got messed up a rebase: https://github.com/python/cpython/pull/10854. All the issues in that commit have now been addressed since https://github.com/python/cpython/pull/11661 got committed.
This change also removes any state from the data segment and onto the module state itself.
https://bugs.python.org/issue35381
Automerge-Triggered-By: @encukou
Provide Py_EnterRecursiveCall() and Py_LeaveRecursiveCall() as
regular functions for the limited API. Previously, there were defined
as macros, but these macros didn't work with the limited API which
cannot access PyThreadState.recursion_depth field.
Remove _Py_CheckRecursionLimit from the stable ABI.
Add Include/cpython/ceval.h header file.
Whenever I use `path.suffix` I have to check again whether it includes the dot or not. I decided to add it to the docstring so I won't have to keep checking.
https://bugs.python.org/issue38422
Automerge-Triggered-By: @pitrou
open(), io.open(), codecs.open() and fileinput.FileInput no longer
accept "U" ("universal newline") in the file mode. This flag was
deprecated since Python 3.3.
This adds a "readlink" method to pathlib.Path objects that calls through
to os.readlink.
https://bugs.python.org/issue30618
Automerge-Triggered-By: @gpshead
Fix stdatomic.h header check for ICC compiler: the ICC implementation
lacks atomic_uintptr_t type which is needed by Python.
Test:
* atomic_int and atomic_uintptr_t types
* atomic_load_explicit() and atomic_store_explicit()
* memory_order_relaxed and memory_order_seq_cst constants
But don't test ATOMIC_VAR_INIT(): it's not used in Python.
test.regrtest now uses process groups in the multiprocessing mode
(-jN command line option) if process groups are available: if
os.setsid() and os.killpg() functions are available.
* bpo-27657: Fix urlparse() with numeric paths
Revert parsing decision from bpo-754016 in favor of the documented
consensus in bpo-16932 of how to treat strings without a // to
designate the netloc.
* bpo-22891: Remove urlsplit() optimization for 'http' prefixed inputs.
Also updates the documentation to clarify the situation surrounding
the digestmod parameter that is required despite its position in the
argument list as of 3.8.0 as well as removing old python2 era
references to "binary strings".
We indavertently had this raise ValueError in 3.8.0 for the missing
arg. This is not considered an API change as no reasonable code would
be catching this missing argument error in order to handle it.
main() is now responsible to send the ANSWER, rather than
ServerProto. main() now waits until it got the HELLO before sending
the ANSWER over the new transport.
Previously, there was a race condition between main() replacing the
protocol and the protocol sending the ANSWER once it gets the HELLO.
TLSv1.3 was disabled for the test: reenable it.
Add a total_nframe field to the traces collected by the tracemalloc module.
This field indicates the original number of frames before it was truncated.
Fix test_compile_dir_maxlevels() on Windows without long path
support: only create 3 subdirectories instead of between 20 and 100
subdirectories.
Fix also compile_dir() to use the current sys.getrecursionlimit()
value as the default maxlevels value, rather than using
sys.getrecursionlimit() value read at startup.
The symbol table handing of PEP572's assignment expressions is not resolving correctly the scope of some variables in presence of global/nonlocal keywords in conjunction with comprehensions.
Currently if any finalizer invoked during garbage collection resurrects any object, the gc gives up and aborts the collection. Although finalizers are assured to only run once per object, this behaviour of the gc can lead to an ever-increasing memory situation if new resurrecting objects are allocated in every new gc collection.
To avoid this, recompute what objects among the unreachable set need to be resurrected and what objects can be safely collected. In this way, resurrecting objects will not block the collection of other objects in the unreachable set.
Add missing stat.S_IFDOOR, stat.S_IFPORT, stat.S_IFWHT,
stat.S_ISDOOR, stat.S_ISPORT, and stat.S_ISWHT values to
the Python implementation of the stat module.
bpo-37531, bpo-38207: On timeout, regrtest no longer attempts to call
`popen.communicate() again: it can hang until all child processes
using stdout and stderr pipes completes. Kill the worker process and
ignores its output.
Reenable test_regrtest.test_multiprocessing_timeout().
bpo-37531: Change also the faulthandler timeout of the main process
from 1 minute to 5 minutes, for Python slowest buildbots.
The implementation of weakref.proxy's methods call back into the Python
API using a borrowed references of the weakly referenced object
(acquired via PyWeakref_GET_OBJECT). This API call may delete the last
reference to the object (either directly or via GC), leaving a dangling
pointer, which can be subsequently dereferenced.
To fix this, claim a temporary ownership of the referenced object when
calling the appropriate method. Some functions because at the moment they
do not need to access the borrowed referent, but to protect against
future changes to these functions, ownership need to be fixed in
all potentially affected methods.
Valgrind emits "Conditional jump or move depends on uninitialised
value(s)" false alarms on GCC builtin strcmp() function. The GCC code
is correct.
Valgrind bug: https://bugs.kde.org/show_bug.cgi?id=264936
In debug mode, PyObject_GC_Track() now calls tp_traverse() of the
object type to ensure that the object is valid: test that objects
visited by tp_traverse() are valid.
Fix pyexpat.c: only track the parser in the GC once the parser is
fully initialized.
Prior to 3.7, re.escape escaped many characters that don't have
special meaning in Python, but that use to require escaping in other
tools and languages. This commit aims to make it clear which characters
were, but are no longer escaped.
bpo-36389, bpo-38376: The _PyObject_CheckConsistency() function is
now also available in release mode. For example, it can be used to
debug a crash in the visit_decref() function of the GC.
Modify the following functions to also work in release mode:
* _PyDict_CheckConsistency()
* _PyObject_CheckConsistency()
* _PyType_CheckConsistency()
* _PyUnicode_CheckConsistency()
Other changes:
* _PyMem_IsPtrFreed(ptr) now also returns 1 if ptr is NULL
(equals to 0).
* _PyBytesWriter_CheckConsistency() now returns 1 and is only used
with assert().
* Reorder _PyObject_Dump() to write safe fields first, and only
attempt to render repr() at the end.
On Windows use UTF-16 (or UTF-32 for 32-bit Tcl_UniChar) with the
"surrogatepass" error handler for converting to/from Tcl Unicode objects.
On Linux use UTF-8 with the "surrogateescape" error handler for converting
to/from Tcl String objects.
Converting strings from Tcl to Python and back now never fails
(except MemoryError).
Victor Stinner
Batuhan Taşkaya
Toke Høiland-Jørgensen
Xtreak
Michael Felt
Lysandros Nikolaou
Jason R. Coombs
Pablo Galindo
JohnnyNajera
Steve Dower
Tim Gates
Yury Selivanov
Mark Dickinson
Kyle Stanley
Abhilash Raj
Elena Oat
AMIR
xdegaye
Christian Heimes
Anj-A
idomic
Andrew Svetlov
Benoit Hudson
Mario Corchero
wim glenn
Claudiu Popa
serge-sans-paille
Inada Naoki
Daniel Himmelstein
Matthew Rollings
Daniel Hillier
Tzu-ping Chung
Bruno P. Kinoshita
HongWeipeng
Terry Jan Reedy
Karl Dubost
Brandt Bucher
Stefan Behnel
Zac Hatfield-Dodds
Raymond Hettinger
Brett Cannon
PypeBros
bcaller
Siwon Kang
benedwards14
Mark Shannon
Jason Fried
Federico Bond
Benjamin Peterson
Łukasz Langa
Jake Tesler
Adam Johnson
Vincent Michel
Tomás Farías
jsnklln
Tal Einat
Victor Stinner
Zackery Spytz
Toke Høiland-Jørgensen
Andrey Doroschenko
Serhiy Storchaka
Eric Snow
Phil Connell
Kirill
Ilya Kulakov
Manjusaka
l0rb
Jeroen Demeyer
Eddie Elizondo
Alexandru Ardelean
Ram Rachum
Anthony Sottile
Lucas Cimon
Dong-hee Na
Marco Rougeth
Zsolt Dollenstein
Girts
Tyler Kieft
Hai Shi
Tim Graham
Gregory P. Smith
Joannah Nanjekye
Julien Danjou
Vinay Sajip
Stein Karlsen
Samuel Colvin
Ruediger Pluem
Ronan Lamy
Tim Peters
Antonio Gutierrez
James Abel
Ricardo Bánffy
Andrei Troie
nde