Joining a thread now ensures the underlying OS thread has exited. This is required for safer fork() in multi-threaded processes.
---------
Co-authored-by: blurb-it[bot] <43283697+blurb-it[bot]@users.noreply.github.com>
Fix test_unhandled_exceptions() of test_asyncio.test_streams: break
explicitly a reference cycle.
Fix also StreamTests.tearDown(): the loop must not be closed
explicitly, but using set_event_loop() which takes care of shutting
down the executor with executor.shutdown(wait=True).
BaseEventLoop.close() calls executor.shutdown(wait=False).
Copy the list of dangling threads to make sure that the list of
"Dangling thread" is complete. Previously, the list was incomplete if
threads completed just before the list was displayed.
Changes:
* Rewrite the warning to make it easier to understand.
* Use support.sleeping_retry().
* threading_cleanup() no longer copies threading._dangling,
but only counts the number of dangling thread.
* Remove support.gc_support() call.
When CPUs are isolated on Linux, os.process_cpu_count() is smaller
than os.cpu_count(). Fix the test for this case.
Example with "isolcpus=5,11 rcu_nocbs=5,11" options passed to a Linux
command line to isolated two logical CPUs:
$ ./python -c 'import os; print(os.process_cpu_count(), "/", os.cpu_count())'
10 / 12
Replace most of calls of _PyErr_WriteUnraisableMsg() and some
calls of PyErr_WriteUnraisable(NULL) with PyErr_FormatUnraisable().
Co-authored-by: Victor Stinner <vstinner@python.org>
This moves several general internal APIs out of _xxsubinterpretersmodule.c and into the new Python/crossinterp.c (and the corresponding internal headers).
Specifically:
* _Py_excinfo, etc.: the initial implementation for non-object exception snapshots (in pycore_pyerrors.h and Python/errors.c)
* _PyXI_exception_info, etc.: helpers for passing an exception beween interpreters (wraps _Py_excinfo)
* _PyXI_namespace, etc.: helpers for copying a dict of attrs between interpreters
* _PyXI_Enter(), _PyXI_Exit(): functions that abstract out the transitions between one interpreter and a second that will do some work temporarily
Again, these were all abstracted out of _xxsubinterpretersmodule.c as generalizations. I plan on proposing these as public API at some point.
- There is no longer a separate Python/executor.c file.
- Conventions in Python/bytecodes.c are slightly different -- don't use `goto error`,
you must use `GOTO_ERROR(error)` (same for others like `unused_local_error`).
- The `TIER_ONE` and `TIER_TWO` symbols are only valid in the generated (.c.h) files.
- In Lib/test/support/__init__.py, `Py_C_RECURSION_LIMIT` is imported from `_testcapi`.
- On Windows, in debug mode, stack allocation grows from 8MiB to 12MiB.
- **Beware!** This changes the env vars to enable uops and their debugging
to `PYTHON_UOPS` and `PYTHON_LLTRACE`.
This is partly to clear this stuff out of pystate.c, but also in preparation for moving some code out of _xxsubinterpretersmodule.c. This change also moves this stuff to the internal API (new: Include/internal/pycore_crossinterp.h). @vstinner did this previously and I undid it. Now I'm re-doing it. :/
* Add mimalloc v2.12
Modified src/alloc.c to remove include of alloc-override.c and not
compile new handler.
Did not include the following files:
- include/mimalloc-new-delete.h
- include/mimalloc-override.h
- src/alloc-override-osx.c
- src/alloc-override.c
- src/static.c
- src/region.c
mimalloc is thread safe and shares a single heap across all runtimes,
therefore finalization and getting global allocated blocks across all
runtimes is different.
* mimalloc: minimal changes for use in Python:
- remove debug spam for freeing large allocations
- use same bytes (0xDD) for freed allocations in CPython and mimalloc
This is important for the test_capi debug memory tests
* Don't export mimalloc symbol in libpython.
* Enable mimalloc as Python allocator option.
* Add mimalloc MIT license.
* Log mimalloc in Lib/test/pythoninfo.py.
* Document new mimalloc support.
* Use macro defs for exports as done in:
https://github.com/python/cpython/pull/31164/
Co-authored-by: Sam Gross <colesbury@gmail.com>
Co-authored-by: Christian Heimes <christian@python.org>
Co-authored-by: Victor Stinner <vstinner@python.org>
* Try to fix asyncio.Server.wait_closed() again
I identified the condition that `wait_closed()` is intended
to wait for: the server is closed *and* there are no more
active connections.
When this condition first becomes true, `_wakeup()` is called
(either from `close()` or from `_detach()`) and it sets `_waiters`
to `None`. So we just check for `self._waiters is None`; if it's
not `None`, we know we have to wait, and do so.
A problem was that the new test introduced in 3.12 explicitly
tested that `wait_closed()` returns immediately when the server
is *not* closed but there are currently no active connections.
This was a mistake (probably a misunderstanding of the intended
semantics). I've fixed the test, and added a separate test that
checks exactly for this scenario.
I also fixed an oddity where in `_wakeup()` the result of the
waiter was set to the waiter itself. This result is not used
anywhere and I changed this to `None`, to avoid a GC cycle.
* Update Lib/asyncio/base_events.py
---------
Co-authored-by: Carol Willing <carolcode@willingconsulting.com>
- `ThreadedChildWatcher.close()` is now *officially* a no-op; `_join_threads()` never did anything.
- Threads created by that class are now named `asyncio-waitpid-NNN`.
- `test.test_asyncio.utils.TestCase.close_loop()` now waits for the child watcher's threads, but not forever; if a thread hangs, it raises `RuntimeError`.
* Move existing tests for PySys_GetObject() and PySys_SetObject() into
specialized files.
* Add test for PySys_GetXOptions() using _testcapi.
* Add tests for PySys_FormatStdout(), PySys_FormatStderr(),
PySys_WriteStdout() and PySys_WriteStderr() using ctypes.
It now always returns an integer if one or less counting options are specified.
Previously it could return a single count as a 1-tuple, an integer (only if
option "update" was specified) or None if no items found.
The result is now the same if wantobjects is set to 0.
Fixes#109894
* set `interp.static_objects.last_resort_memory_error.args` to empty tuple to avoid crash on `PyErr_Display()` call
* allow `_PyExc_InitGlobalObjects()` to be called on subinterpreter init
---------
Co-authored-by: blurb-it[bot] <43283697+blurb-it[bot]@users.noreply.github.com>
Before, the '-s STRIPDIR' option on
compileall lead to some surprising results as it only strips away
path components that match, but leaves alone the non-matching ones
interspersed in between.
For example, with: python -m compileall -s/path/to/another/src
/path/to/build/src/file.py
The resulting written path will be: build/file.py
This fix only strips directories that are a fully matching prefix of the
source path. If a stripdir is provided that is not a valid prefix, a
warning will be displayed (which can be silenced with '-qq').
asyncio.TaskGroup and asyncio.Timeout classes now raise proper RuntimeError
if they are improperly used.
* When they are used without entering the context manager.
* When they are used after finishing.
* When the context manager is entered more than once (simultaneously or
sequentially).
* If there is no current task when entering the context manager.
They now remain in a consistent state after an exception is thrown,
so subsequent operations can be performed correctly (if they are allowed).
Co-authored-by: James Hilton-Balfe <gobot1234yt@gmail.com>
Test case matching patterns specified by options --match, --ignore,
--matchfile and --ignorefile are now tested in the order of
specification, and the last match determines whether the test case be run
or ignored.
Add PyUnicode_AsUTF8() function to the limited C API.
multiprocessing posixshmem now uses PyUnicode_AsUTF8() instead of
PyUnicode_AsUTF8AndSize(): the extension is built with the limited C
API. The function now raises an exception if the filename contains an
embedded null character instead of truncating silently the filename.
* PyUnicode_AsUTF8() now raises an exception if the string contains
embedded null characters.
* Update related C API tests (test_capi.test_unicode).
* type_new_set_doc() uses PyUnicode_AsUTF8AndSize() to silently
truncate doc containing null bytes.
Co-authored-by: Serhiy Storchaka <storchaka@gmail.com>
There were a few things I did in gh-110565 that need to be fixed. I also forgot to add tests in that PR.
(Note that this PR exposes a refleak introduced by gh-110246. I'll take care of that separately.)
Fix detection of gdb built without Python scripting support.
* check_usable_gdb() doesn't check gdb exit code when calling
run_gdb().
* Use shutil.which() to get the path to the gdb program.
The PySys_Audit() function was added in Python 3.8 by the PEP 578
"Python Runtime Audit Hooks".
Add also PySys_AuditTuple() to the limited C API, function added
to Python 3.13.
Move non-limited "PerfMap" C API from Include/sysmodule.h to
Include/cpython/sysmodule.h.
Add PyMem_RawMalloc(), PyMem_RawCalloc(), PyMem_RawRealloc() and
PyMem_RawFree() to the limited C API.
These functions were added by Python 3.4 and are needed to port
stdlib extensions to the limited C API, like grp and pwd.
Co-authored-by: Erlend E. Aasland <erlend@python.org>
zipinfo now supports the full range of values in the TZ string
determined by RFC 8536 and detects all invalid formats.
Both Python and C implementations now raise exceptions of the same
type on invalid data.
Now re.error is raised instead of OverflowError or RuntimeError for
too large width of look-behind pattern.
The limit is increased to 2**32-1 (was 2**31-1).
* tty.setraw() and tty.setcbreak() previously returned partially modified
list of the original tty attributes. Now they return the correct list of
the original tty attributes
* tty.cfmakeraw() and tty.cfmakecbreak() now make a copy of the list of
special characters before modifying it.
It already mostly worked, except in the case when invalid keyword
argument with non-ASCII name was passed to function with non-ASCII
parameter names. Then it crashed in the debug mode.
Replace select.epoll() with selectors.DefaultSelector to support
FreeBSD 14 and newer.
* Merge common code between test_timerfd_epoll()
test_timerfd_ns_epoll().
* Replace "_ = func()" with "func()".
* Add TimerfdTests.read_count_signaled() method.
Add write_all() helper function to test_pty to raise an exception on
short write: if os.writes() does not write all bytes. It should not
happen for a PTY.
The docs state that the space, tab, colon, and comma characters are
ignored in Py_BuildValue() format strings.
Co-authored-by: Serhiy Storchaka <storchaka@gmail.com>
It does not already work (because it locks only addCleanup(), not doCleanups()),
and it is no longer needed since the clean up procedure waits for all test threads to join.
Ensure that `PurePath('foo/a').with_name('.')` raises `ValueError`
Ensure that `PureWindowsPath('foo/a').with_name('a:b')` does not raise
`ValueError`.
multiprocessing test_terminate() and test_wait_socket_slow() no
longer test the CI performance: no longer check maximum elapsed time.
Add CLOCK_RES constant: tolerate a difference of 100 ms.
Deprecation warning about non-integer numbers in gettext now always refers
to the line in the user code where gettext function or method is used.
Previously, it could refer to a line in gettext code.
Also, increase test coverage for NullTranslations and domain-aware functions
like dngettext().
* test_timerfd_TFD_TIMER_ABSTIME() and
test_timerfd_ns_TFD_TIMER_ABSTIME() tolerate a difference of 50 us.
* test_timerfd_negative() checks if os.TFD_TIMER_CANCEL_ON_SET is
defined.
Add wrapper for timerfd_create, timerfd_settime, and timerfd_gettime to os module.
Co-authored-by: Serhiy Storchaka <storchaka@gmail.com>
Co-authored-by: Adam Turner <9087854+AA-Turner@users.noreply.github.com>
Co-authored-by: Erlend E. Aasland <erlend.aasland@protonmail.com>
Co-authored-by: Victor Stinner <vstinner@python.org>
sys.audit() now has assertions to check that the event argument is
not NULL and that the format argument does not use the "N" format.
Add tests on PySys_AuditTuple().
Replace os.kill() with proc.kill() which catchs PermissionError.
Rewrite _kill_with_event():
* Use subprocess context manager ("with proc:").
* Use sleeping_retry() to wait until the child process is ready.
* Replace SIGINT with proc.kill() on error.
* Replace 10 seconds with SHORT_TIMEOUT to wait until the process is
ready.
* Replace 0.5 seconds with SHORT_TIMEOUT to wait for the process
exit.
Irit Katriel
Anthony Shaw
Serhiy Storchaka
Sergey B Kirpichev
Mark Shannon
Gregory P. Smith
Dennis Sweeney
Masayuki Moriyama
Charles Machalow
Tian Gao
Davide Rizzo
CF Bolz-Tereick
Antoine Pitrou
Nikita Sobolev
Victor Stinner
AN Long
partev
Kumar Aditya
Eric Snow
Yilei Yang
Guido van Rossum
Thomas Grainger
Tomas R
Pablo Galindo Salgado
Sam Gross
Dino Viehland
zcxsythenew
Evan Kohilas
Barney Gale
Raymond Hettinger
gsallam
Batuhan Taskaya
Jokimax
Jochen Sprickerhof
Savannah Ostrowski
Radislav Chugunov
hetmankp
Furkan Onder
Shantanu
James
Pieter Eendebak
Alex Waygood
Lysandros Nikolaou
Zachary Ware
Hugo van Kemenade
Łukasz Langa
Kirill Podoprigora
Russell Keith-Magee
Filipe Laíns
Jelle Zijlstra
Zackery Spytz
Donghee Na
Masaru Tsuchiyama
Carl Meyer