We make use of the same mechanism that we use for the static builtin types. This required a few tweaks.
This change is the final piece needed to make _datetime support multiple interpreters. I've updated the module slot accordingly.
(cherry picked from commit 105f22ea46, AKA gh-119929)
Co-authored-by: Eric Snow <ericsnowcurrently@gmail.com>
This makes the support official.
(cherry picked from commit dba7a167db, AKA gh-119991)
Co-authored-by: Eric Snow <ericsnowcurrently@gmail.com>
Co-authored-by: Kirill Podoprigora <kirill.bast9@mail.ru>
gh-119336: Restore removed _PyLong_NumBits() function (GH-119418)
It is used by the pywin32 project.
(cherry picked from commit e50fac96e8)
Co-authored-by: Ethan Smith <ethan@ethanhs.me>
Implement `shutil._rmtree_safe_fd()` using a list as a stack to avoid emitting recursion errors on deeply nested trees.
`shutil._rmtree_unsafe()` was fixed in a150679f90.
(cherry picked from commit 53b1981fb0)
Co-authored-by: Barney Gale <barney.gale@gmail.com>
Revert "[3.13] gh-69214: Fix fcntl.ioctl() request type (GH-119498) (#119504)"
This reverts commit 0bab0b3a53.
The change modified how negative values, like termios.TIOCSWINSZ, was
treated and is actually backward incompatible.
dSupport non-dict globals in LOAD_FROM_DICT_OR_GLOBALS
The implementation basically copies LOAD_GLOBAL. Possibly it could be deduplicated,
but that seems like it may get hairy since the two operations have different operands.
This is important to fix in 3.14 for PEP 649, but it's a bug in earlier versions too,
and we should backport to 3.13 and 3.12 if possible.
(cherry picked from commit 80a4e38994)
Release the GIL before calling `_Py_qsbr_unregister`.
The deadlock could occur when the GIL was enabled at runtime. The
`_Py_qsbr_unregister` call might block while holding the GIL because the
thread state was not active, but the GIL was still held.
(cherry picked from commit 078b8c8cf2)
Co-authored-by: Sam Gross <colesbury@gmail.com>
Make sure that `gilstate_counter` is not zero in when calling
`PyThreadState_Clear()`. A destructor called from `PyThreadState_Clear()` may
call back into `PyGILState_Ensure()` and `PyGILState_Release()`. If
`gilstate_counter` is zero, it will try to create a new thread state before
the current active thread state is destroyed, leading to an assertion failure
or crash.
(cherry picked from commit bcc1be39cb)
Co-authored-by: Sam Gross <colesbury@gmail.com>
Some of standard Tcl types were renamed, removed, or no longer
registered in Tcl 8.7/9.0. This change fixes automatic conversion of Tcl
values to Python values to avoid returning a Tcl_Obj where the primary
Python types (int, bool, str, bytes) were returned in older Tcl.
(cherry picked from commit 94e9585e99)
Co-authored-by: Christopher Chavez <chrischavez@gmx.us>
gh-119548: Add a 'clear' command to the REPL (GH-119549)
(cherry picked from commit e3bac04c37)
Co-authored-by: Pablo Galindo Salgado <Pablogsal@gmail.com>
For example, the free-threaded build now generates
`lib/pkgconfig/python-3.13t.pc` and the debug build generates
`lib/pkgconfig/python-3.13d.pc`.
(cherry picked from commit 1c04c63ced)
Co-authored-by: Sam Gross <colesbury@gmail.com>
GH-89727: Fix `os.fwalk()` recursion error on deep trees (GH-119638)
Implement `os.fwalk()` using a list as a stack to avoid emitting recursion
errors on deeply nested trees.
(cherry picked from commit 3c890b503c)
Co-authored-by: Barney Gale <barney.gale@gmail.com>
GH-89727: Partially fix `shutil.rmtree()` recursion error on deep trees (GH-119634)
Make `shutil._rmtree_unsafe()` call `os.walk()`, which is implemented
without recursion.
`shutil._rmtree_safe_fd()` is not affected and can still raise a recursion
error.
(cherry picked from commit a150679f90)
Co-authored-by: Barney Gale <barney.gale@gmail.com>
Co-authored-by: Jelle Zijlstra <jelle.zijlstra@gmail.com>
The deadlock only affected the free-threaded build and only occurred
when the GIL was enabled at runtime. The `Py_DECREF(old_name)` call
might temporarily release the GIL while holding the type seqlock.
Another thread may spin trying to acquire the seqlock while holding the
GIL.
The deadlock occurred roughly 1 in ~1,000 runs of `pool_in_threads.py`
from `test_multiprocessing_pool_circular_import`.
(cherry picked from commit c22323cd1c)
Co-authored-by: Sam Gross <colesbury@gmail.com>
gh-119443: Turn off from __future__ import annotations in REPL (GH-119493)
(cherry picked from commit a8e35e8eba)
Co-authored-by: Jelle Zijlstra <jelle.zijlstra@gmail.com>
This is a backport of 3 commits that go together.
(cherry picked from commit a895756)
(cherry picked from commit b30d30c)
(cherry picked from commit a89fc26)
- Cache line object to avoid creating a Unicode object
for all of the tokens in the same line.
- Speed up byte offset to column offset conversion by using the
smallest buffer possible to measure the difference.
(cherry picked from commit d87b015106)
Co-authored-by: Lysandros Nikolaou <lisandrosnik@gmail.com>
Co-authored-by: Pablo Galindo <pablogsal@gmail.com>
The assertion was added in gh-118532 but was based on the invalid assumption that PyState_FindModule() would only be called with an already-initialized module def. I've added a test to make sure we don't make that assumption again.
(cherry picked from commit 0c5ebe13e9)
Co-authored-by: Eric Snow <ericsnowcurrently@gmail.com>
gh-69214: Fix fcntl.ioctl() request type (GH-119498)
Use an 'unsigned long' instead of an 'unsigned int' for the request
parameter of fcntl.ioctl() to support requests larger than UINT_MAX.
(cherry picked from commit 92fab3356f)
Co-authored-by: Victor Stinner <vstinner@python.org>
ensurepip forks a subprocess to run pip itself, but that subprocess only inherits a -I isolated mode flag (see _run_pip() in Lib/ensurepip/__init__.py), not the "-E -s" flags that the installer has been using. This means that parts of ensurepip don't actually run in an isolated environment and can make incorrect decisions based on packages installed in the user site-packages.
(cherry picked from commit c9073eb1a9)
Co-authored-by: Michael Vincent <377567+Vynce@users.noreply.github.com>
Add `Py_BEGIN_CRITICAL_SECTION_SEQUENCE_FAST` and
`Py_END_CRITICAL_SECTION_SEQUENCE_FAST` macros and update `str.join` to use
them. Also add a regression test that would crash reliably without this
patch.
(cherry picked from commit baf347d916)
Co-authored-by: Josh {*()} Rosenberg <26495692+MojoVampire@users.noreply.github.com>
_PyArg_Parser holds static global data generated for modules by Argument Clinic. The _PyArg_Parser.kwtuple field is a tuple object, even though it's stored within a static global. In some cases the tuple is statically allocated and thus it's okay that it gets shared by multiple interpreters. However, in other cases the tuple is set lazily, allocated from the heap using the active interprepreter at the point the tuple is needed.
This is a problem once that interpreter is destroyed since _PyArg_Parser.kwtuple becomes at dangling pointer, leading to crashes. It isn't a problem if the tuple is allocated under the main interpreter, since its lifetime is bound to the lifetime of the runtime. The solution here is to temporarily switch to the main interpreter. The alternative would be to always statically allocate the tuple.
This change also fixes a bug where only the most recent parser was added to the global linked list.
(cherry picked from commit 81865002ae)
Co-authored-by: Eric Snow <ericsnowcurrently@gmail.com>
Fix regression introduced in gh-100884: AttributeError when re-fold a long
address list.
Also fix more cases of incorrect encoding of the address separator in the
address list missed in gh-100884.
(cherry picked from commit 858b9e85fc)
Co-authored-by: Serhiy Storchaka <storchaka@gmail.com>
gh-119050: Add XML support to libregrtest refleak checker (GH-119148)
regrtest test runner: Add XML support to the refleak checker
(-R option).
* run_unittest() now stores XML elements as string, rather than
objects, in support.junit_xml_list.
* runtest_refleak() now saves/restores XML strings before/after
checking for reference leaks. Save XML into a temporary file.
(cherry picked from commit 9257731f5d)
Co-authored-by: Victor Stinner <vstinner@python.org>
* Fix for email.generator.Generator with whitespace between encoded words.
email.generator.Generator currently does not handle whitespace between
encoded words correctly when the encoded words span multiple lines. The
current generator will create an encoded word for each line. If the end
of the line happens to correspond with the end real word in the
plaintext, the generator will place an unencoded space at the start of
the subsequent lines to represent the whitespace between the plaintext
words.
A compliant decoder will strip all the whitespace from between two
encoded words which leads to missing spaces in the round-tripped
output.
The fix for this is to make sure that whitespace between two encoded
words ends up inside of one or the other of the encoded words. This
fix places the space inside of the second encoded word.
A second problem happens with continuation lines. A continuation line that
starts with whitespace and is followed by a non-encoded word is fine because
the newline between such continuation lines is defined as condensing to
a single space character. When the continuation line starts with whitespace
followed by an encoded word, however, the RFCs specify that the word is run
together with the encoded word on the previous line. This is because normal
words are filded on syntactic breaks by encoded words are not.
The solution to this is to add the whitespace to the start of the encoded word
on the continuation line.
Test cases are from GH-92081
* Rename a variable so it's not confused with the final variable.
(cherry picked from commit a6fdb31b67)
Co-authored-by: Toshio Kuratomi <a.badger@gmail.com>
gh-119121: Fix and test `async.staggered.staggered_race` (GH-119173)
(cherry picked from commit 16b46ebd2b)
Co-authored-by: Nikita Sobolev <mail@sobolevn.me>
Restore behaviour from 3.12 when `path.with_suffix(None)` is called.
(cherry picked from commit 3c28510b98)
Co-authored-by: Barney Gale <barney.gale@gmail.com>
gh-119132: Update sys.version to identify free-threaded or not. (gh-119134)
(cherry picked from commit c141d43937)
Co-authored-by: Donghee Na <donghee.na@python.org>
The source line was not displayed if the warnings module had not yet
been imported.
(cherry picked from commit 100c7ab00a)
Co-authored-by: Kirill Podoprigora <kirill.bast9@mail.ru>
For unknown reasons some arguments for Tkinter binding can be created
as a 1-tuple containing a Tcl_Obj when wantobjects is 2.
(cherry picked from commit 5b88d95cc5)
Co-authored-by: Serhiy Storchaka <storchaka@gmail.com>
gh-58933: Make pdb return to caller frame correctly when f_trace is not set (GH-118979)
(cherry picked from commit f526314194)
Co-authored-by: Tian Gao <gaogaotiantian@hotmail.com>
gh-118789: Restore hidden `_PyWeakref_ClearRef` (GH-118797)
_PyWeakref_ClearRef was previously exposed in the public C-API, although
it begins with an underscore and is not documented. It's used by a few
C-API extensions. There is currently no alternative public API that can
replace its use.
_PyWeakref_ClearWeakRefsExceptCallbacks is the only thread-safe way to
use _PyWeakref_ClearRef in the free-threaded build. This exposes the C
symbol, but does not make the API public.
(cherry picked from commit db5af7da09)
Co-authored-by: Sam Gross <colesbury@gmail.com>
Some embedders and extensions include parts of the internal API. The
pycore_mimalloc.h file is transitively include by a number of other
internal headers. This avoids include errors for code that was
already including those headers.
(cherry picked from commit 71cc0651e7)
Co-authored-by: Sam Gross <colesbury@gmail.com>
The `list_preallocate_exact` function did not zero initialize array
contents. In the free-threaded build, this could expose uninitialized
memory to concurrent readers between the call to
`list_preallocate_exact` and the filling of the array contents with
items.
(cherry picked from commit 2402715e10)
Co-authored-by: Sam Gross <colesbury@gmail.com>
Print only filename with lineno if linecache.getline() returns an empty string.
(cherry picked from commit 7c87ce777b)
Co-authored-by: Radislav Chugunov <52372310+chgnrdv@users.noreply.github.com>
gh-118033: Fix `__weakref__` not set for generic dataclasses (GH-118099)
(cherry picked from commit fa9b9cb113)
Co-authored-by: Nikita Sobolev <mail@sobolevn.me>
Callbacks registered in the tkinter module now take arguments as
various Python objects (int, float, bytes, tuple), not just str.
To restore the previous behavior set tkinter module global wantobject to 1
before creating the Tk object or call the wantobject() method of the Tk object
with argument 1.
Calling it with argument 2 restores the current default behavior.
Fix an edge case in `binascii.a2b_base64` strict mode, where
excessive padding was not detected when no padding is necessary.
Co-authored-by: Terry Jan Reedy <tjreedy@udel.edu>
Co-authored-by: Pieter Eendebak <pieter.eendebak@gmail.com>
Rationale
=========
argparse performs a complex formatting of the usage for argument grouping
and for line wrapping to fit the terminal width. This formatting has been
a constant source of bugs for at least 10 years (see linked issues below)
where defensive assertion errors are triggered or brackets and paranthesis
are not properly handeled.
Problem
=======
The current implementation of argparse usage formatting relies on regular
expressions to group arguments usage only to separate them again later
with another set of regular expressions. This is a complex and error prone
approach that caused all the issues linked below. Special casing certain
argument formats has not solved the problem. The following are some of
the most common issues:
- empty `metavar`
- mutually exclusive groups with `SUPPRESS`ed arguments
- metavars with whitespace
- metavars with brackets or paranthesis
Solution
========
The following two comments summarize the solution:
- https://github.com/python/cpython/issues/82091#issuecomment-1093832187
- https://github.com/python/cpython/issues/77048#issuecomment-1093776995
Mainly, the solution is to rewrite the usage formatting to avoid the
group-then-separate approach. Instead, the usage parts are kept separate
and only joined together at the end. This allows for a much simpler
implementation that is easier to understand and maintain. It avoids the
regular expressions approach and fixes the corresponding issues.
This closes the following GitHub issues:
- #62090
- #62549
- #77048
- #82091
- #89743
- #96310
- #98666
These PRs become obsolete:
- #15372
- #96311
Add missing import to code that handles too large files and offsets.
Use list, not tuple, for a mutable sequence.
Add tests to prevent similar mistakes.
---------
Co-authored-by: Gregory P. Smith [Google LLC] <greg@krypto.org>
Co-authored-by: Kirill Podoprigora <kirill.bast9@mail.ru>
This change makes sure all extension/builtin modules have their init function run first by the main interpreter before proceeding with import in the original interpreter (main or otherwise). This means when the import of a single-phase init module fails in an isolated subinterpreter, it won't tie any global state/callbacks to the subinterpreter.
Add the ability to enable/disable the GIL at runtime, and use that in
the C module loading code.
We can't know before running a module init function if it supports
free-threading, so the GIL is temporarily enabled before doing so. If
the module declares support for running without the GIL, the GIL is
later disabled. Otherwise, the GIL is permanently enabled, and will
never be disabled again for the life of the current interpreter.
Now, such classes will no longer require changes in Python 3.13 in the normal case.
The test suite for robotframework passes with no DeprecationWarnings under this PR.
I also added a new DeprecationWarning for the case where `_field_types` exists
but is incomplete, since that seems likely to indicate a user mistake.
Add _PyType_LookupRef and use incref before setting attribute on type
Makes setting an attribute on a class and signaling type modified atomic
Avoid adding re-entrancy exposing the type cache in an inconsistent state by decrefing after type is updated
* Add PhotoImage.read() to read an image from a file.
* Add PhotoImage.data() to get the image data.
* Add background and grayscale parameters to PhotoImage.write().
* Add the PhotoImage method copy_replace() to copy a region
from one image to other image, possibly with pixel zooming and/or
subsampling.
* Add from_coords parameter to PhotoImage methods copy(), zoom() and subsample().
* Add zoom and subsample parameters to PhotoImage method copy().
The designated initializer syntax in static inline functions in pycore_backoff.h
causes problems for C++ or MSVC users who aren't yet using C++20.
While internal, pycore_backoff.h is included (indirectly, via pycore_code.h)
by some key 3rd party software that does so for speed.
This is *not* sufficient for the final 3.13 release, but it will do for beta 1:
- What's new entry
- Updated changelog entry (news blurb)
- Mention the proxy for f_globals in the datamodel and Python frame object docs
This doesn't have any C API details (what's new refers to the PEP).
For converting large ints to strings, CPython invokes a function in _pylong.py,
which uses the decimal module to implement an asymptotically waaaaay
sub-quadratic algorithm. But if the C decimal module isn't available, CPython
uses _pydecimal.py instead. Which in turn frequently does str(int). If the int
is very large, _pylong ends up doing the work, which in turn asks decimal to do
"big" arithmetic, which in turn calls str(big_int), which in turn ... it can
become infinite mutual recursion.
This change introduces a different int->str function that doesn't use decimal.
It's asymptotically worse, "Karatsuba time" instead of quadratic time, so
still a huge improvement. _pylong switches to that when the C decimal isn't
available. It is also used for not too large integers (less than 450_000 bits),
where it is faster (up to 2 times for 30_000 bits) than the asymptotically
better implementation that uses the C decimal.
Co-authored-by: Tim Peters <tim.peters@gmail.com>
* Initial stab.
* Test the tentative fix. Hangs "forever" without this change.
* Move the new test to a better spot.
* New comment to explain why _convert_to_str allows any poewr of 10.
* Fixed a comment, and fleshed out an existing test that appeared unfinished.
* Added temporary asserts. Or maybe permanent ;-)
* Update Lib/_pydecimal.py
Co-authored-by: Serhiy Storchaka <storchaka@gmail.com>
* Remove the new _convert_to_str().
Serhiy and I independently concluded that exact powers of 10
aren't possible in these contexts, so just checking the
string length is sufficient.
* At least for now, add the asserts to the other block too.
* 📜🤖 Added by blurb_it.
---------
Co-authored-by: Serhiy Storchaka <storchaka@gmail.com>
Co-authored-by: blurb-it[bot] <43283697+blurb-it[bot]@users.noreply.github.com>
This PR adds the ability to enable the GIL if it was disabled at
interpreter startup, and modifies the multi-phase module initialization
path to enable the GIL when loading a module, unless that module's spec
includes a slot indicating it can run safely without the GIL.
PEP 703 called the constant for the slot `Py_mod_gil_not_used`; I went
with `Py_MOD_GIL_NOT_USED` for consistency with gh-104148.
A warning will be issued up to once per interpreter for the first
GIL-using module that is loaded. If `-v` is given, a shorter message
will be printed to stderr every time a GIL-using module is loaded
(including the first one that issues a warning).
The function returns `True` or `False` depending on whether the GIL is
currently enabled. In the default build, it always returns `True`
because the GIL is always enabled.
Now inspect.signature() supports references to the module globals in
parameter defaults on methods in extension modules. Previously it was
only supported in functions. The workaround was to specify the fully
qualified name, including the module name.
Add "Raw" variant of PyTime functions:
* PyTime_MonotonicRaw()
* PyTime_PerfCounterRaw()
* PyTime_TimeRaw()
Changes:
* Add documentation and tests. Tests release the GIL while calling
raw clock functions.
* py_get_system_clock() and py_get_monotonic_clock() now check that
the GIL is hold by the caller if raise_exc is non-zero.
* Reimplement "Unchecked" functions with raw clock functions.
Co-authored-by: Petr Viktorin <encukou@gmail.com>
T. Wouters
Miss Islington (bot)
Victor Stinner
Jelle Zijlstra
Eric Snow
Lysandros Nikolaou
Hugo van Kemenade
Steve Dower
Serhiy Storchaka
Nikita Sobolev
Alex Waygood
chrysn
Youfu Zhang
Ali Hamdan
John Sirois
Tian Gao
Brett Simmers
James Hilton-Balfe
Jeong, YunWon
Erlend E. Aasland
Dino Viehland
Thomas Grainger
Petr Viktorin
Pieter Eendebak
Terry Jan Reedy
Pablo Galindo Salgado
Guido van Rossum
Ryan Batchelder
Tim Peters
Davide Rizzo
Irit Katriel
Sam Gross
Carl Meyer
Alexander Kanavin
infohash
Raphael Gaschignard
Sergey B Kirpichev
Crowthebird
Nice Zombies
Malcolm Smith
Harmen Stoppels
Anthony Shaw