The `@critical_section` directive instructs Argument Clinic to generate calls
to `Py_BEGIN_CRITICAL_SECTION()` and `Py_END_CRITICAL_SECTION()` around the
bound function. In `--disable-gil` builds, these calls will lock and unlock
the `self` object. They are no-ops in the default build.
This is used in one place (`_io._Buffered.close`) as a demonstration.
Subsequent PRs will use it more widely in the `_io.Buffered` bindings.
Most important: move how-to-quit sentence to the end and mention 'q'.
Re-group the other sentences and improve some wording.
---------
Co-authored-by: Hugo van Kemenade <hugovk@users.noreply.github.com>
Add `glob.translate()` function that converts a pathname with shell wildcards to a regular expression. The regular expression is used by pathlib to implement `match()` and `glob()`.
This function differs from `fnmatch.translate()` in that wildcards do not match path separators by default, and that a `*` pattern segment matches precisely one path segment. When *recursive* is set to true, `**` pattern segments match any number of path segments, and `**` cannot appear outside its own segment.
In pathlib, this change speeds up directory walking (because `_make_child_relpath()` does less work), makes path objects smaller (they don't need a `_lines` slot), and removes the need for some gnarly code.
Co-authored-by: Jason R. Coombs <jaraco@jaraco.com>
Co-authored-by: Adam Turner <9087854+AA-Turner@users.noreply.github.com>
* Split list_extend() into two sub-functions: list_extend_fast() and
list_extend_iter().
* list_inplace_concat() no longer has to call Py_DECREF() on the
list_extend() result, since list_extend() now returns an int.
New methods to access mailbox.Maildir message info and flags:
get_info, set_info, get_flags, set_flags, add_flag, remove_flag.
These methods speed up accessing a message's info and/or flags and are
useful when it is not necessary to access the message's contents,
as when iterating over a Maildir to find messages with specific flags.
---------
* Add more str type checking
* modernize to f-strings instead of %
Co-authored-by: Gregory P. Smith <greg@krypto.org>
Make `multiprocessing.managers.{DictProxy,ListProxy}` generic for type annotation use. `ListProxy[str]` for example.
Co-authored-by: Alex Waygood <Alex.Waygood@Gmail.com>
Co-authored-by: Gregory P. Smith <greg@krypto.org>
Now all results from worker processes are aggregated and
displayed together as a summary at the end of a regrtest run.
The traditional trace is left in place for use with sequential
in-process test runs but now raises a warning that those
numbers are not precise.
`-T -j` requires `--with-pydebug` as it relies on `-Xpresite=`.
The random module now imports the _sha2 module lazily in the
Random.seed() method for str, bytes and bytearray seeds. It also
imports lazily the warnings module in the _randbelow() method for
classes without getrandbits(). Lazy import makes Python startup
faster and reduces the number of imported modules at startup.
In most cases, doctest is not needed. So don't always import it at
startup. The change reduces the number of modules already
imported when a test is run.
* Remove mention of "vsapi" element type from the documentation.
* Add tests for element_create() and other ttk.Style methods.
* Add examples for element_create() in the documentation.
* Revert "gh-111089: Use PyUnicode_AsUTF8() in Argument Clinic (#111585)"
This reverts commit d9b606b3d0.
* Revert "gh-111089: Use PyUnicode_AsUTF8() in getargs.c (#111620)"
This reverts commit cde1071b2a.
* Revert "gh-111089: PyUnicode_AsUTF8() now raises on embedded NUL (#111091)"
This reverts commit d731579bfb.
* Revert "gh-111089: Add PyUnicode_AsUTF8() to the limited C API (#111121)"
This reverts commit d8f32be5b6.
* Revert "gh-111089: Use PyUnicode_AsUTF8() in sqlite3 (#111122)"
This reverts commit 37e4e20eaa.
Joining a thread now ensures the underlying OS thread has exited. This is required for safer fork() in multi-threaded processes.
---------
Co-authored-by: blurb-it[bot] <43283697+blurb-it[bot]@users.noreply.github.com>
Fix test_unhandled_exceptions() of test_asyncio.test_streams: break
explicitly a reference cycle.
Fix also StreamTests.tearDown(): the loop must not be closed
explicitly, but using set_event_loop() which takes care of shutting
down the executor with executor.shutdown(wait=True).
BaseEventLoop.close() calls executor.shutdown(wait=False).
Copy the list of dangling threads to make sure that the list of
"Dangling thread" is complete. Previously, the list was incomplete if
threads completed just before the list was displayed.
Changes:
* Rewrite the warning to make it easier to understand.
* Use support.sleeping_retry().
* threading_cleanup() no longer copies threading._dangling,
but only counts the number of dangling thread.
* Remove support.gc_support() call.
When CPUs are isolated on Linux, os.process_cpu_count() is smaller
than os.cpu_count(). Fix the test for this case.
Example with "isolcpus=5,11 rcu_nocbs=5,11" options passed to a Linux
command line to isolated two logical CPUs:
$ ./python -c 'import os; print(os.process_cpu_count(), "/", os.cpu_count())'
10 / 12
Replace most of calls of _PyErr_WriteUnraisableMsg() and some
calls of PyErr_WriteUnraisable(NULL) with PyErr_FormatUnraisable().
Co-authored-by: Victor Stinner <vstinner@python.org>
This moves several general internal APIs out of _xxsubinterpretersmodule.c and into the new Python/crossinterp.c (and the corresponding internal headers).
Specifically:
* _Py_excinfo, etc.: the initial implementation for non-object exception snapshots (in pycore_pyerrors.h and Python/errors.c)
* _PyXI_exception_info, etc.: helpers for passing an exception beween interpreters (wraps _Py_excinfo)
* _PyXI_namespace, etc.: helpers for copying a dict of attrs between interpreters
* _PyXI_Enter(), _PyXI_Exit(): functions that abstract out the transitions between one interpreter and a second that will do some work temporarily
Again, these were all abstracted out of _xxsubinterpretersmodule.c as generalizations. I plan on proposing these as public API at some point.
- There is no longer a separate Python/executor.c file.
- Conventions in Python/bytecodes.c are slightly different -- don't use `goto error`,
you must use `GOTO_ERROR(error)` (same for others like `unused_local_error`).
- The `TIER_ONE` and `TIER_TWO` symbols are only valid in the generated (.c.h) files.
- In Lib/test/support/__init__.py, `Py_C_RECURSION_LIMIT` is imported from `_testcapi`.
- On Windows, in debug mode, stack allocation grows from 8MiB to 12MiB.
- **Beware!** This changes the env vars to enable uops and their debugging
to `PYTHON_UOPS` and `PYTHON_LLTRACE`.
This is partly to clear this stuff out of pystate.c, but also in preparation for moving some code out of _xxsubinterpretersmodule.c. This change also moves this stuff to the internal API (new: Include/internal/pycore_crossinterp.h). @vstinner did this previously and I undid it. Now I'm re-doing it. :/
* Add mimalloc v2.12
Modified src/alloc.c to remove include of alloc-override.c and not
compile new handler.
Did not include the following files:
- include/mimalloc-new-delete.h
- include/mimalloc-override.h
- src/alloc-override-osx.c
- src/alloc-override.c
- src/static.c
- src/region.c
mimalloc is thread safe and shares a single heap across all runtimes,
therefore finalization and getting global allocated blocks across all
runtimes is different.
* mimalloc: minimal changes for use in Python:
- remove debug spam for freeing large allocations
- use same bytes (0xDD) for freed allocations in CPython and mimalloc
This is important for the test_capi debug memory tests
* Don't export mimalloc symbol in libpython.
* Enable mimalloc as Python allocator option.
* Add mimalloc MIT license.
* Log mimalloc in Lib/test/pythoninfo.py.
* Document new mimalloc support.
* Use macro defs for exports as done in:
https://github.com/python/cpython/pull/31164/
Co-authored-by: Sam Gross <colesbury@gmail.com>
Co-authored-by: Christian Heimes <christian@python.org>
Co-authored-by: Victor Stinner <vstinner@python.org>
* Try to fix asyncio.Server.wait_closed() again
I identified the condition that `wait_closed()` is intended
to wait for: the server is closed *and* there are no more
active connections.
When this condition first becomes true, `_wakeup()` is called
(either from `close()` or from `_detach()`) and it sets `_waiters`
to `None`. So we just check for `self._waiters is None`; if it's
not `None`, we know we have to wait, and do so.
A problem was that the new test introduced in 3.12 explicitly
tested that `wait_closed()` returns immediately when the server
is *not* closed but there are currently no active connections.
This was a mistake (probably a misunderstanding of the intended
semantics). I've fixed the test, and added a separate test that
checks exactly for this scenario.
I also fixed an oddity where in `_wakeup()` the result of the
waiter was set to the waiter itself. This result is not used
anywhere and I changed this to `None`, to avoid a GC cycle.
* Update Lib/asyncio/base_events.py
---------
Co-authored-by: Carol Willing <carolcode@willingconsulting.com>
- `ThreadedChildWatcher.close()` is now *officially* a no-op; `_join_threads()` never did anything.
- Threads created by that class are now named `asyncio-waitpid-NNN`.
- `test.test_asyncio.utils.TestCase.close_loop()` now waits for the child watcher's threads, but not forever; if a thread hangs, it raises `RuntimeError`.
Fix locale.LC_CTYPE documentation to no longer mention string.lower() et al. Those functions were removed in Python 3.0:
https://docs.python.org/2/library/string.html#deprecated-string-functions
Also, fix a comment in logging about locale-specific behavior of `str.lower()`.
Co-authored-by: Hugo van Kemenade <hugovk@users.noreply.github.com>
* Move existing tests for PySys_GetObject() and PySys_SetObject() into
specialized files.
* Add test for PySys_GetXOptions() using _testcapi.
* Add tests for PySys_FormatStdout(), PySys_FormatStderr(),
PySys_WriteStdout() and PySys_WriteStderr() using ctypes.
It now always returns an integer if one or less counting options are specified.
Previously it could return a single count as a 1-tuple, an integer (only if
option "update" was specified) or None if no items found.
The result is now the same if wantobjects is set to 0.
Fixes#109894
* set `interp.static_objects.last_resort_memory_error.args` to empty tuple to avoid crash on `PyErr_Display()` call
* allow `_PyExc_InitGlobalObjects()` to be called on subinterpreter init
---------
Co-authored-by: blurb-it[bot] <43283697+blurb-it[bot]@users.noreply.github.com>
Before, the '-s STRIPDIR' option on
compileall lead to some surprising results as it only strips away
path components that match, but leaves alone the non-matching ones
interspersed in between.
For example, with: python -m compileall -s/path/to/another/src
/path/to/build/src/file.py
The resulting written path will be: build/file.py
This fix only strips directories that are a fully matching prefix of the
source path. If a stripdir is provided that is not a valid prefix, a
warning will be displayed (which can be silenced with '-qq').
asyncio.TaskGroup and asyncio.Timeout classes now raise proper RuntimeError
if they are improperly used.
* When they are used without entering the context manager.
* When they are used after finishing.
* When the context manager is entered more than once (simultaneously or
sequentially).
* If there is no current task when entering the context manager.
They now remain in a consistent state after an exception is thrown,
so subsequent operations can be performed correctly (if they are allowed).
Co-authored-by: James Hilton-Balfe <gobot1234yt@gmail.com>
Test case matching patterns specified by options --match, --ignore,
--matchfile and --ignorefile are now tested in the order of
specification, and the last match determines whether the test case be run
or ignored.
Add PyUnicode_AsUTF8() function to the limited C API.
multiprocessing posixshmem now uses PyUnicode_AsUTF8() instead of
PyUnicode_AsUTF8AndSize(): the extension is built with the limited C
API. The function now raises an exception if the filename contains an
embedded null character instead of truncating silently the filename.
* PyUnicode_AsUTF8() now raises an exception if the string contains
embedded null characters.
* Update related C API tests (test_capi.test_unicode).
* type_new_set_doc() uses PyUnicode_AsUTF8AndSize() to silently
truncate doc containing null bytes.
Co-authored-by: Serhiy Storchaka <storchaka@gmail.com>
Add comments where .keys() is needed.
Leave debugger usages along because situation is unclear as indicated in expanded comment.
Most testing is manual.
There were a few things I did in gh-110565 that need to be fixed. I also forgot to add tests in that PR.
(Note that this PR exposes a refleak introduced by gh-110246. I'll take care of that separately.)
Fix detection of gdb built without Python scripting support.
* check_usable_gdb() doesn't check gdb exit code when calling
run_gdb().
* Use shutil.which() to get the path to the gdb program.
The PySys_Audit() function was added in Python 3.8 by the PEP 578
"Python Runtime Audit Hooks".
Add also PySys_AuditTuple() to the limited C API, function added
to Python 3.13.
Move non-limited "PerfMap" C API from Include/sysmodule.h to
Include/cpython/sysmodule.h.
Add PyMem_RawMalloc(), PyMem_RawCalloc(), PyMem_RawRealloc() and
PyMem_RawFree() to the limited C API.
These functions were added by Python 3.4 and are needed to port
stdlib extensions to the limited C API, like grp and pwd.
Co-authored-by: Erlend E. Aasland <erlend@python.org>
zipinfo now supports the full range of values in the TZ string
determined by RFC 8536 and detects all invalid formats.
Both Python and C implementations now raise exceptions of the same
type on invalid data.
Now re.error is raised instead of OverflowError or RuntimeError for
too large width of look-behind pattern.
The limit is increased to 2**32-1 (was 2**31-1).
* tty.setraw() and tty.setcbreak() previously returned partially modified
list of the original tty attributes. Now they return the correct list of
the original tty attributes
* tty.cfmakeraw() and tty.cfmakecbreak() now make a copy of the list of
special characters before modifying it.
It already mostly worked, except in the case when invalid keyword
argument with non-ASCII name was passed to function with non-ASCII
parameter names. Then it crashed in the debug mode.
Replace select.epoll() with selectors.DefaultSelector to support
FreeBSD 14 and newer.
* Merge common code between test_timerfd_epoll()
test_timerfd_ns_epoll().
* Replace "_ = func()" with "func()".
* Add TimerfdTests.read_count_signaled() method.
Sam Gross
Terry Jan Reedy
Barney Gale
Victor Stinner
Tian Gao
Sergey B Kirpichev
Serhiy Storchaka
Irit Katriel
Stephen Gildea
Nikita Sobolev
Łukasz Langa
Zac Hatfield-Dodds
Nicolas Tessore
Vinay Sajip
Mark Shannon
Donghee Na
Jelle Zijlstra
Kalyan
Pierre Ossman (ThinLinc team)
Stefan
Romuald Brunet
Anthony Shaw
Gregory P. Smith
Dennis Sweeney
Masayuki Moriyama
Charles Machalow
Davide Rizzo
CF Bolz-Tereick
Antoine Pitrou
AN Long
partev
Kumar Aditya
Eric Snow
Yilei Yang
Guido van Rossum
Thomas Grainger
Tomas R
Pablo Galindo Salgado
Dino Viehland
zcxsythenew
Evan Kohilas
c-bata
Raymond Hettinger
gsallam
Batuhan Taskaya
Hugo van Kemenade
Jokimax
Jochen Sprickerhof
Savannah Ostrowski
Radislav Chugunov
hetmankp
Furkan Onder
sc07kvm
Shantanu
James
Bar Harel
Pieter Eendebak
Alex Waygood
Lysandros Nikolaou
Zachary Ware
Kirill Podoprigora
Russell Keith-Magee
Thomas Wouters
Filipe Laíns