See 6b98b274b6 for an explanation of the problem and solution. Here I've applied the solution to channels.
(cherry picked from commit 8b209fd4f8, AKA gh-121805)
Co-authored-by: Eric Snow <ericsnowcurrently@gmail.com>
Any cross-interpreter mechanism for passing objects between interpreters must be very careful to respect isolation, even when the object is effectively immutable (e.g. int, str). Here this especially relates to when an interpreter sends one of its objects, and then is destroyed while the inter-interpreter machinery (e.g. queue) still holds a reference to the object.
When I added interpreters.Queue, I dealt with that case (using an atexit hook) by silently removing all items from the queue that were added by the finalizing interpreter.
Later, while working on concurrent.futures.InterpreterPoolExecutor (gh-116430), I noticed it was somewhat surprising when items were silently removed from the queue when the originating interpreter was destroyed. (See my comment on that PR.)
It took me a little while to realize what was going on. I expect that users, which much less context than I have, would experience the same pain.
My approach, here, to improving the situation is to give users three options:
1. return a singleton (interpreters.queues.UNBOUND) from Queue.get() in place of each removed item
2. raise an exception (interpreters.queues.ItemInterpreterDestroyed) from Queue.get() in place of each removed item
3. existing behavior: silently remove each item (i.e. Queue.get() skips each one)
The default will now be (1), but users can still explicitly opt in any of them, including to the silent removal behavior.
The behavior for each item may be set with the corresponding Queue.put() call. and a queue-wide default may be set when the queue is created. (This is the same as I did for "synconly".)
(cherry picked from commit 6b98b274b6, AKA gh-116431)
Co-authored-by: Eric Snow <ericsnowcurrently@gmail.com>
It is our general practice to make new optional parameters keyword-only,
even if the existing parameters are all positional-or-keyword. Passing
this parameter as positional would look confusing and could be error-prone
if additional parameters are added in the future.
(cherry picked from commit 50eec501fe)
Co-authored-by: Jelle Zijlstra <jelle.zijlstra@gmail.com>
We also need to close the `slave_fd` earlier so that reading from
`master_fd` won't block forever when the subprocess finishes.
(cherry picked from commit abc3aeebdb)
Co-authored-by: Sam Gross <colesbury@gmail.com>
gh-121497: Make Pyrepl respect correctly the history with input hook set (GH-121498)
(cherry picked from commit 4e36dd7d87)
Co-authored-by: Pablo Galindo Salgado <Pablogsal@gmail.com>
gh-121499: Fix multi-line history rendering in the REPL (GH-121531)
(cherry picked from commit 4b9e10d0ea)
Signed-off-by: Pablo Galindo <pablogsal@gmail.com>
Co-authored-by: Pablo Galindo Salgado <Pablogsal@gmail.com>
On POSIX systems, excluding macOS framework installs, the lib directory
for the free-threaded build now includes a "t" suffix to avoid conflicts
with a co-located default build installation.
(cherry picked from commit e8c91d90ba)
Co-authored-by: Sam Gross <colesbury@gmail.com>
When builtin static types are initialized for a subinterpreter, various "tp" slots have already been inherited (for the main interpreter). This was interfering with the logic in add_operators() (in Objects/typeobject.c), causing a wrapper to get created when it shouldn't. This change fixes that by preserving the original data from the static type struct and checking that.
(cherry picked from commit 5250a03133, AKA gh-121602)
Co-authored-by: Eric Snow <ericsnowcurrently@gmail.com>
This fixes a mistake in gh-113012 and adds a test that verifies the fix.
(cherry picked from commit 35a67e36aa, AKA gh-121597)
Co-authored-by: Eric Snow <ericsnowcurrently@gmail.com>
On heavily loaded machines, the subprocess may finish its sleep before
the parent process manages to synchronize with it via a failed lock.
This leads to errors like:
Exception: failed to sync child in 300.3 sec
Use pipes instead to mutually synchronize between parent and child.
(cherry picked from commit af9f6de6ea)
Co-authored-by: Sam Gross <colesbury@gmail.com>
gh-121571: Do not use `EnvironmentError` in tests, use `OSError` instead (GH-121572)
(cherry picked from commit e2822360da)
Co-authored-by: sobolevn <mail@sobolevn.me>
gh-117657: Skip test when running under TSan (GH-121549)
The ProcessPoolForkserver combined with resource_tracker starts a thread
after forking, which is not supported by TSan.
Also skip test_multiprocessing_fork for the same reason
(cherry picked from commit 04397434aa)
Co-authored-by: Sam Gross <colesbury@gmail.com>
The change in gh-118157 (b2cd54a) should have also updated clear_singlephase_extension() but didn't. We fix that here. Note that clear_singlephase_extension() (AKA _PyImport_ClearExtension()) is only used in tests.
(cherry picked from commit 15d48aea02, AKA gh-121503)
Co-authored-by: Eric Snow <ericsnowcurrently@gmail.com>
gh-121351: Skip test_not_wiping_history_file() if no readline (GH-121422)
(cherry picked from commit 68e279b37a)
Co-authored-by: Sergey B Kirpichev <skirpichev@gmail.com>
gh-121359: Run test_pyrepl in isolated mode (GH-121414)
run_repl() now pass the -I option (isolated mode) to Python if the
'env' parameter is not set.
(cherry picked from commit 6239d41527)
Co-authored-by: Victor Stinner <vstinner@python.org>
gh-121084: Fix test_typing random leaks (GH-121360)
Clear typing ABC caches when running tests for refleaks (-R option):
call _abc_caches_clear() on typing abstract classes and their
subclasses.
(cherry picked from commit 5f660e8e2c)
Co-authored-by: Victor Stinner <vstinner@python.org>
gh-117983: Defer import of threading for lazy module loading (GH-120233)
As noted in gh-117983, the import importlib.util can be triggered at
interpreter startup under some circumstances, so adding threading makes
it a potentially obligatory load.
Lazy loading is not used in the stdlib, so this removes an unnecessary
load for the majority of users and slightly increases the cost of the
first lazily loaded module.
An obligatory threading load breaks gevent, which monkeypatches the
stdlib. Although unsupported, there doesn't seem to be an offsetting
benefit to breaking their use case.
For reference, here are benchmarks for the current main branch:
```
❯ hyperfine -w 8 './python -c "import importlib.util"'
Benchmark 1: ./python -c "import importlib.util"
Time (mean ± σ): 9.7 ms ± 0.7 ms [User: 7.7 ms, System: 1.8 ms]
Range (min … max): 8.4 ms … 13.1 ms 313 runs
```
And with this patch:
```
❯ hyperfine -w 8 './python -c "import importlib.util"'
Benchmark 1: ./python -c "import importlib.util"
Time (mean ± σ): 8.4 ms ± 0.7 ms [User: 6.8 ms, System: 1.4 ms]
Range (min … max): 7.2 ms … 11.7 ms 352 runs
```
Compare to:
```
❯ hyperfine -w 8 './python -c pass'
Benchmark 1: ./python -c pass
Time (mean ± σ): 7.6 ms ± 0.6 ms [User: 5.9 ms, System: 1.6 ms]
Range (min … max): 6.7 ms … 11.3 ms 390 runs
```
This roughly halves the import time of importlib.util.
(cherry picked from commit 94f50f8ee6)
Co-authored-by: Chris Markiewicz <effigies@gmail.com>
gh-118714: Make the pdb post-mortem restart/quit behavior more reasonable (GH-118725)
(cherry picked from commit e245ed7d1e)
Co-authored-by: Tian Gao <gaogaotiantian@hotmail.com>
* [3.13] gh-120782: Update internal type cache when reloading datetime
When reloading _datetime module, the single-phase version did not invoke the PyInit__datetime function, whereas the current multi-phase version updates the static types through the module init. The outdated static type cache in the interpreter state needs to be invalidated at the end of reloading the multi-phase module.
This will allow Py_TRACE_REFS builds to pass the test suite, until the underlying issue can be resolved.
(cherry picked from commit 9bcb7d8c6f, AKA gh-121236)
Co-authored-by: Eric Snow <ericsnowcurrently@gmail.com>
gh-121200: Fix test_expanduser_pwd2() of test_posixpath (GH-121228)
Call getpwnam() to get pw_dir, since it can be different than
getpwall() pw_dir.
(cherry picked from commit 02cb5fdee3)
Co-authored-by: Victor Stinner <vstinner@python.org>
gh-121200: Log pwd entry in test_expanduser_pwd2() (GH-121207)
Use subTest() to log the pwd entry in test_expanduser_pwd2() of
test_posixpath to help debugging.
(cherry picked from commit 05a6f8da60)
Co-authored-by: Victor Stinner <vstinner@python.org>
gh-121084: Call _abc_registry_clear() when checking refleaks (GH-121191)
dash_R_cleanup() now calls _abc_registry_clear() before calling again
register().
(cherry picked from commit c766ad206e)
Co-authored-by: Victor Stinner <vstinner@python.org>
gh-121188: Sanitize invalid XML characters in regrtest (GH-121195)
When creating the JUnit XML file, regrtest now escapes characters
which are invalid in XML, such as the chr(27) control character used
in ANSI escape sequences.
(cherry picked from commit af8c3d7a26)
Co-authored-by: Victor Stinner <vstinner@python.org>
gh-87744: fix waitpid race while calling send_signal in asyncio (GH-121126)
asyncio earlier relied on subprocess module to send signals to the process, this has some drawbacks one being that subprocess module unnecessarily calls waitpid on child processes and hence it races with asyncio implementation which internally uses child watchers. To mitigate this, now asyncio sends signals directly to the process without going through the subprocess on non windows systems. On Windows it fallbacks to subprocess module handling but on windows there are no child watchers so this issue doesn't exists altogether.
(cherry picked from commit bd473aa598)
Co-authored-by: Kumar Aditya <kumaraditya@python.org>
* parse_intermixed_args() now raises ArgumentError instead of calling
error() if exit_on_error is false.
* Internal code now always raises ArgumentError instead of calling
error(). It is then caught at the higher level and error() is called if
exit_on_error is true.
(cherry picked from commit 81a654a342)
The check for whether the log file is a real file is expensive on NFS
filesystems. This commit reorders the rollover condition checking to
not do the file type check if the expected file size is less than the
rotation threshold.
(cherry picked from commit e9b4ec614b)
Co-authored-by: Craig Robson <craig@zhatt.com>
Co-authored-by: Oleg Iarygin <oleg@arhadthedev.net>
gh-119521: Remove _IncompleteInputError from the docs (GH-120993)
(cherry picked from commit 1167a9a30b)
Co-authored-by: Petr Viktorin <encukou@gmail.com>
Miss Islington (bot)
Kirill Podoprigora
Ken Jin
Jelle Zijlstra
neonene
Serhiy Storchaka
Victor Stinner
Miro Hrončok
Thomas Wouters