Commit Graph

14448 Commits

Author SHA1 Message Date
Sam Gross c908d1f87d
gh-120974: Use common freelist code in asyncio (#122132)
This refactors asyncio to use the common freelist helper functions and
macros. As a side effect, the freelist for _asyncio.Future is now
re-enabled in the free-threaded build.
2024-07-23 21:54:24 +05:30
Sam Gross 47847aa8ef
gh-120974: Make _asyncio._enter_task atomic in the free-threaded build (#122138)
Use `PyDict_SetDefaultRef` to set the current task in a single operation
under the dictionary's lock.
2024-07-23 14:17:14 +05:30
Łukasz Langa dc93d1125f
gh-121957: Emit audit events for `python -i` and `python -m asyncio` (GH-121958)
Relatedly, emit the `cpython.run_startup` event from the Python version of
`PYTHONSTARTUP` handling.
2024-07-22 13:04:08 +02:00
Sam Gross 97248204a1
gh-121621: Disable asyncio freelist in free-threaded build (#122046)
The futureobj freelist isn't thread-safe. We intend to re-enable the
freelist in a thread-safe way for 3.14 (but not 3.13).
2024-07-19 15:26:29 -04:00
mpage e059aa6b01
gh-120973: Fix thread-safety issues with `threading.local` (#121655)
This is a small refactoring to the current design that allows us to
avoid manually iterating over threads.

This should also fix gh-118490.
2024-07-19 13:22:02 -04:00
Rodrigo Oliveira d66b06107b
gh-118830: Bump pickle.DEFAULT_PROTOCOL to 5 (GH-119340) 2024-07-19 16:47:10 +02:00
Serhiy Storchaka 1a0c7b9ba4
gh-121905: Consistently use "floating-point" instead of "floating point" (GH-121907) 2024-07-19 08:06:02 +00:00
Tian Gao eaf094c09b
gh-120289: Add external timer in traverse of _lsprof.Profiler (#121998) 2024-07-18 20:46:24 -07:00
Tian Gao 1ab1778283
gh-120289: Disallow disable() and clear() in external timer to prevent use-after-free (#120297) 2024-07-18 12:47:22 -07:00
Sam Gross 81fd625b5c
gh-121621: Move asyncio_running_loop to private struct (#121939)
This avoids changing the ABI and keeps the field in the private struct.
2024-07-17 15:21:24 -07:00
sobolevn f4bc84d261
gh-121925: Fix uninitialized variables in `main.c` (#121926) 2024-07-17 16:13:37 +00:00
Alex Waygood ac07451116
gh-120678: pyrepl: Include globals from modules passed with `-i` (GH-120904)
Co-authored-by: Łukasz Langa <lukasz@langa.pl>
2024-07-17 16:18:42 +02:00
Ken Jin 69c68de43a
gh-121621: Move asyncio running loop to thread state (GH-121695) 2024-07-17 01:09:58 +08:00
Petr Viktorin b4aedb23ae
gh-113993: Don't immortalize in PyUnicode_InternInPlace; keep immortalizing in other API (#121364)
* Switch PyUnicode_InternInPlace to _PyUnicode_InternMortal, clarify docs

* Document immortality in some functions that take `const char *`

This is PyUnicode_InternFromString;
PyDict_SetItemString, PyObject_SetAttrString;
PyObject_DelAttrString; PyUnicode_InternFromString;
and the PyModule_Add convenience functions.

Always point out a non-immortalizing alternative.

* Don't immortalize user-provided attr names in _ctypes
2024-07-16 15:36:21 +02:00
sobolevn 8b6d475581
gh-121791: Check for `NULL` in `MethodDescriptor2_new` in `_testcapi` (#121792) 2024-07-16 11:29:33 +03:00
Eric Snow 8b209fd4f8
gh-76785: Expand How Interpreter Channels Handle Interpreter Finalization (gh-121805)
See 6b98b274b6 for an explanation of the problem and solution.  Here I've applied the solution to channels.
2024-07-15 19:43:59 +00:00
Eric Snow 6b98b274b6
gh-76785: Expand How Interpreter Queues Handle Interpreter Finalization (gh-116431)
Any cross-interpreter mechanism for passing objects between interpreters must be very careful to respect isolation, even when the object is effectively immutable (e.g. int, str).  Here this especially relates to when an interpreter sends one of its objects, and then is destroyed while the inter-interpreter machinery (e.g. queue) still holds a reference to the object.

When I added interpreters.Queue, I dealt with that case (using an atexit hook) by silently removing all items from the queue that were added by the finalizing interpreter.

Later, while working on concurrent.futures.InterpreterPoolExecutor (gh-116430), I noticed it was somewhat surprising when items were silently removed from the queue when the originating interpreter was destroyed.  (See my comment on that PR.) 
 It took me a little while to realize what was going on.  I expect that users, which much less context than I have, would experience the same pain.

My approach, here, to improving the situation is to give users three options:

1. return a singleton (interpreters.queues.UNBOUND) from Queue.get() in place of each removed item
2. raise an exception (interpreters.queues.ItemInterpreterDestroyed) from Queue.get() in place of each removed item
3. existing behavior: silently remove each item (i.e. Queue.get() skips each one)

The default will now be (1), but users can still explicitly opt in any of them, including to the silent removal behavior.

The behavior for each item may be set with the corresponding Queue.put() call. and a queue-wide default may be set when the queue is created.  (This is the same as I did for "synconly".)
2024-07-15 12:49:23 -06:00
Kumar Aditya 48042c52a6
fix outdated comments in asyncio (#121783) 2024-07-15 05:59:19 +00:00
Ken Jin 5d6861ad06
gh-121621: Use PyMutex for writes to asyncio state (#121622)
Co-authored-by: Kumar Aditya <kumaraditya@python.org>
2024-07-14 12:22:56 +00:00
Savannah Ostrowski 65fededf9c
Update retroactive comments from GH-117741 (segfault in `FutureIter_dealloc`) (GH-121638)
Address comments
2024-07-12 10:34:30 +02:00
Sam Gross e8c91d90ba
gh-121103: Put free-threaded libraries in `lib/python3.14t` (#121293)
On POSIX systems, excluding macOS framework installs, the lib directory
for the free-threaded build now includes a "t" suffix to avoid conflicts
with a co-located default build installation.
2024-07-11 16:21:37 -04:00
Sam Gross 44937d11a6
gh-121592: Make select.poll() and related objects thread-safe (#121594)
This makes select.poll() and kqueue() objects thread-safe in the
free-threaded build. Note that calling close() concurrently with other
functions is still not thread-safe due to races on file descriptors
(gh-121544).
2024-07-11 10:21:09 -04:00
Irit Katriel 6557af6698
gh-121554: remove unnecessary internal functions in compile.c (#121555)
Co-authored-by: Erlend E. Aasland <erlend@python.org>
2024-07-10 23:48:37 +01:00
Eric Snow 35a67e36aa
gh-121596: Fix Sharing Interpreter Channels (gh-121597)
This fixes a mistake in gh-113012 and adds a test that verifies the fix.
2024-07-10 21:31:09 +00:00
Sam Gross 1d3cf79a50
gh-121368: Fix seq lock memory ordering in _PyType_Lookup (#121388)
The `_PySeqLock_EndRead` function needs an acquire fence to ensure that
the load of the sequence happens after any loads within the read side
critical section. The missing fence can trigger bugs on macOS arm64.

Additionally, we need a release fence in `_PySeqLock_LockWrite` to
ensure that the sequence update is visible before any modifications to
the cache entry.
2024-07-08 14:52:07 -04:00
Max Muoto 5289550b33
gh-121374: Correct docstrings in `_interpchannels` (gh-121418) 2024-07-08 10:32:17 -06:00
Serhiy Storchaka 8ecb8962e3
gh-121288: Make error message for index() methods consistent (GH-121395)
Make error message for index() methods consistent

Remove the repr of the searched value (which can be arbitrary large)
from ValueError messages for list.index(), range.index(), deque.index(),
deque.remove() and ShareableList.index().  Make the error messages
consistent with error messages for other index() and remove()
methods.
2024-07-05 10:50:45 -07:00
Cody Maloney 06a1c3fb24
gh-120754: Update estimated_size in C truncate (#121357)
Sometimes a large file is truncated (test_largefile). While
estimated_size is used as a estimate (the read will stil get the number
of bytes in the file), that it is much larger than the actual size of
data can result in a significant over allocation and sometimes lead to
a MemoryError / running out of memory.

This brings the C implementation to match the Python _pyio
implementation.
2024-07-04 12:59:18 +00:00
Cody Maloney 2f5f19e783
gh-120754: Reduce system calls in full-file FileIO.readall() case (#120755)
This reduces the system call count of a simple program[0] that reads all
the `.rst` files in Doc by over 10% (5706 -> 4734 system calls on my
linux system, 5813 -> 4875 on my macOS)

This reduces the number of `fstat()` calls always and seek calls most
the time. Stat was always called twice, once at open (to error early on
directories), and a second time to get the size of the file to be able
to read the whole file in one read. Now the size is cached with the
first call.

The code keeps an optimization that if the user had previously read a
lot of data, the current position is subtracted from the number of bytes
to read. That is somewhat expensive so only do it on larger files,
otherwise just try and read the extra bytes and resize the PyBytes as
needeed.

I built a little test program to validate the behavior + assumptions
around relative costs and then ran it under `strace` to get a log of the
system calls. Full samples below[1].

After the changes, this is everything in one `filename.read_text()`:

```python3
openat(AT_FDCWD, "cpython/Doc/howto/clinic.rst", O_RDONLY|O_CLOEXEC) = 3`
fstat(3, {st_mode=S_IFREG|0644, st_size=343, ...}) = 0`
ioctl(3, TCGETS, 0x7ffdfac04b40)        = -1 ENOTTY (Inappropriate ioctl for device)
lseek(3, 0, SEEK_CUR)                   = 0
read(3, ":orphan:\n\n.. This page is retain"..., 344) = 343
read(3, "", 1)                          = 0
close(3)                                = 0
```

This does make some tradeoffs
1. If the file size changes between open() and readall(), this will
still get all the data but might have more read calls.
2. I experimented with avoiding the stat + cached result for small files
in general, but on my dev workstation at least that tended to reduce
performance compared to using the fstat().

[0]

```python3
from pathlib import Path

nlines = []
for filename in Path("cpython/Doc").glob("**/*.rst"):
    nlines.append(len(filename.read_text()))
```

[1]
Before small file:

```
openat(AT_FDCWD, "cpython/Doc/howto/clinic.rst", O_RDONLY|O_CLOEXEC) = 3
fstat(3, {st_mode=S_IFREG|0644, st_size=343, ...}) = 0
ioctl(3, TCGETS, 0x7ffe52525930)        = -1 ENOTTY (Inappropriate ioctl for device)
lseek(3, 0, SEEK_CUR)                   = 0
lseek(3, 0, SEEK_CUR)                   = 0
fstat(3, {st_mode=S_IFREG|0644, st_size=343, ...}) = 0
read(3, ":orphan:\n\n.. This page is retain"..., 344) = 343
read(3, "", 1)                          = 0
close(3)                                = 0
```

After small file:

```
openat(AT_FDCWD, "cpython/Doc/howto/clinic.rst", O_RDONLY|O_CLOEXEC) = 3
fstat(3, {st_mode=S_IFREG|0644, st_size=343, ...}) = 0
ioctl(3, TCGETS, 0x7ffdfac04b40)        = -1 ENOTTY (Inappropriate ioctl for device)
lseek(3, 0, SEEK_CUR)                   = 0
read(3, ":orphan:\n\n.. This page is retain"..., 344) = 343
read(3, "", 1)                          = 0
close(3)                                = 0
```

Before large file:

```
openat(AT_FDCWD, "cpython/Doc/c-api/typeobj.rst", O_RDONLY|O_CLOEXEC) = 3
fstat(3, {st_mode=S_IFREG|0644, st_size=133104, ...}) = 0
ioctl(3, TCGETS, 0x7ffe52525930)        = -1 ENOTTY (Inappropriate ioctl for device)
lseek(3, 0, SEEK_CUR)                   = 0
lseek(3, 0, SEEK_CUR)                   = 0
fstat(3, {st_mode=S_IFREG|0644, st_size=133104, ...}) = 0
read(3, ".. highlight:: c\n\n.. _type-struc"..., 133105) = 133104
read(3, "", 1)                          = 0
close(3)                                = 0
```

After large file:

```
openat(AT_FDCWD, "cpython/Doc/c-api/typeobj.rst", O_RDONLY|O_CLOEXEC) = 3
fstat(3, {st_mode=S_IFREG|0644, st_size=133104, ...}) = 0
ioctl(3, TCGETS, 0x7ffdfac04b40)        = -1 ENOTTY (Inappropriate ioctl for device)
lseek(3, 0, SEEK_CUR)                   = 0
lseek(3, 0, SEEK_CUR)                   = 0
read(3, ".. highlight:: c\n\n.. _type-struc"..., 133105) = 133104
read(3, "", 1)                          = 0
close(3)                                = 0
```

Co-authored-by: Shantanu <12621235+hauntsaninja@users.noreply.github.com>
Co-authored-by: Erlend E. Aasland <erlend.aasland@protonmail.com>
Co-authored-by: Victor Stinner <vstinner@python.org>
2024-07-04 09:17:00 +02:00
Sergey B Kirpichev 51c4a324c0
gh-61103: Support float and long double complex types in ctypes module (#121248)
This amends 6988ff02a5: memory allocation for
stginfo->ffi_type_pointer.elements in PyCSimpleType_init() should be
more generic (perhaps someday fmt->pffi_type->elements will be not a
two-elements array).

It should finally resolve #61103.

Co-authored-by: Victor Stinner <vstinner@python.org>
Co-authored-by: Bénédikt Tran <10796600+picnixz@users.noreply.github.com>
2024-07-03 11:08:11 +02:00
AN Long 705a123898
gh-116181: Remove Py_BUILD_CORE_BUILTIN and Py_BUILD_CORE_MODULE in rotatingtree.c (#121260) 2024-07-03 13:05:05 +05:30
Serhiy Storchaka ff5806c78e
gh-121027: Make the functools.partial object a method descriptor (GH-121089)
Co-authored-by: d.grigonis <dgrigonis@users.noreply.github.com>
2024-07-03 09:02:15 +03:00
AN Long 294e724964
gh-117657: Fix data races reported by TSAN in some set methods (#120914)
Refactor the fast Unicode hash check into `_PyObject_HashFast` and use relaxed
atomic loads in the free-threaded build.

After this change, the TSAN doesn't report data races for this method.
2024-07-01 15:11:39 -04:00
AN Long 8a5176772c
gh-117657: Use critical section to make _socket.socket.close thread safe (GH-120490) 2024-07-01 16:38:30 +02:00
Will Childs-Klein 56a3ce2715
gh-117784: Only reference PHA functions ifndef SSL_VERIFY_POST_HANDSHAKE (GH-117785)
With this change, builds with OpenSSL forks that don't have this functionalty
(like AWS-LC or BoringSSL) will require less patching.
2024-07-01 15:28:35 +02:00
Sergey B Kirpichev 6988ff02a5
gh-61103: Support double complex (_Complex) type in ctypes (#120894)
Example:

```pycon
>>> import ctypes
>>> ctypes.__STDC_IEC_559_COMPLEX__
1
>>> libm = ctypes.CDLL('libm.so.6')
>>> libm.clog.argtypes = [ctypes.c_double_complex]
>>> libm.clog.restype = ctypes.c_double_complex
>>> libm.clog(1+1j)
(0.34657359027997264+0.7853981633974483j)
```

Co-authored-by: Nice Zombies <nineteendo19d0@gmail.com>
Co-authored-by: Bénédikt Tran <10796600+picnixz@users.noreply.github.com>
Co-authored-by: Victor Stinner <vstinner@python.org>
2024-07-01 10:54:33 +02:00
Victor Stinner a0b8b342c5
gh-121199: Use _Py__has_attribute() in timemodule.c (#121203)
Use the _Py__has_attribute() macro in timemodule.c and
bootstrap_hash.c to fix a build error on old GCC versions (GCC 4.8.5
on s390x).
2024-07-01 08:49:33 +00:00
Erlend E. Aasland f80376b129
gh-113565: Improve and harden detection of curses dependencies (#119816)
1. Use pkg-config to check for ncursesw/panelw. If that fails, use 
   pkg-config to check for ncurses/panel.
2. Regardless of pkg-config output, search for curses/panel headers, so
   we're sure we have all defines in pyconfig.h.
3. Regardless of pkg-config output, check if libncurses or libncursesw
   contains the 'initscr' symbol; if it does _and_ pkg-config failed
   earlier, add the resulting -llib linker option to CURSES_LIBS.
   Ditto for 'update_panels' and PANEL_LIBS.
4. Wrap the rest of the checks with WITH_SAVE_ENV and make sure we're 
   using updated LIBS and CPPFLAGS for those.

Add the PY_CHECK_CURSES convenience macro.
2024-07-01 08:10:03 +00:00
blhsing 6d34938dc8
gh-120713: Normalize year with century for datetime.strftime (GH-120820) 2024-06-29 09:32:42 +03:00
Justin Applegate 92893fd8dc
gh-121137: Add missing Py_DECREF calls for ADDITEMS opcode of _pickle.c (#121136)
PyObject_GetAttr returns a new reference, but this reference is never decremented using Py_DECREF, so Py_DECREF calls to this referece are added
2024-06-28 14:43:45 -07:00
Serhiy Storchaka db96edd6d1
gh-121027: Add a future warning in functools.partial.__get__ (#121086) 2024-06-27 11:47:20 +00:00
Victor Stinner 12af8ec864
gh-121040: Use __attribute__((fallthrough)) (#121044)
Fix warnings when using -Wimplicit-fallthrough compiler flag.

Annotate explicitly "fall through" switch cases with a new
_Py_FALLTHROUGH macro which uses __attribute__((fallthrough)) if
available. Replace "fall through" comments with _Py_FALLTHROUGH.

Add _Py__has_attribute() macro. No longer define __has_attribute()
macro if it's not defined. Move also _Py__has_builtin() at the top
of pyport.h.

Co-Authored-By: Nikita Sobolev <mail@sobolevn.me>
2024-06-27 09:58:44 +00:00
Victor Stinner 9e4a81f00f
gh-120642: Move private PyCode APIs to the internal C API (#120643)
* Move _Py_CODEUNIT and related functions to pycore_code.h.
* Move _Py_BackoffCounter to pycore_backoff.h.
* Move Include/cpython/optimizer.h content to pycore_optimizer.h.
* Remove Include/cpython/optimizer.h.
* Remove PyUnstable_Replace_Executor().

Rename functions:

* PyUnstable_GetExecutor() => _Py_GetExecutor()
* PyUnstable_GetOptimizer() => _Py_GetOptimizer()
* PyUnstable_SetOptimizer() => _Py_SetTier2Optimizer()
* PyUnstable_Optimizer_NewCounter() => _PyOptimizer_NewCounter()
* PyUnstable_Optimizer_NewUOpOptimizer() => _PyOptimizer_NewUOpOptimizer()
2024-06-26 13:54:03 +02:00
Kumar Aditya 82235449b8
gh-107803: fix thread safety issue in double linked list implementation (#121007) 2024-06-26 05:11:32 +00:00
Eric Snow a905721b9c
gh-120838: Add _PyThreadState_WHENCE_FINI (gh-121010)
We also add _PyThreadState_NewBound() and drop _PyThreadState_SetWhence().

This change only affects internal API.
2024-06-25 14:35:12 -06:00
Raymond Hettinger 9b32b89074
Add fast path in count_elements (gh-120983) 2024-06-25 03:10:00 -05:00
Steve Dower e731554337
Fixes loop variables to be the same types as their limit (GH-120958) 2024-06-24 17:11:47 +01:00
Victor Stinner 2e157851e3
gh-119182: Add PyUnicodeWriter_WriteUCS4() function (#120849) 2024-06-24 17:40:39 +02:00
Serhiy Storchaka c38e2f64d0
gh-119614: Fix truncation of strings with embedded null characters in Tkinter (GH-120909)
Now the null character is always represented as \xc0\x80 for
Tcl_NewStringObj().
2024-06-24 12:17:25 +03:00
Serhiy Storchaka 02df679574
Use _PyLong_IsNegative instead of _PyLong_Sign if appropriate. (GH-120493)
It is faster and more obvious.
2024-06-24 09:49:01 +03:00
Serhiy Storchaka f4ddaa3967
gh-101830: Fix Tcl_Obj to string conversion (GH-120884)
Accessing the Tkinter object's string representation no longer converts
the underlying Tcl object to a string on Windows.
2024-06-23 16:34:14 +03:00
Kumar Aditya 4717aaa1a7
GH-107803: double linked list implementation for asyncio tasks (GH-107804)
* linked list

* add tail optmiization to linked list

* wip

* wip

* wip

* more fixes

* finally it works

* add tests

* remove weakreflist

* add some comments

* reduce code duplication in _asynciomodule.c

* address some review comments

* add invariants about the state of the linked list

* add better explanation

* clinic regen

* reorder branches for better branch prediction

* Update Modules/_asynciomodule.c

* Apply suggestions from code review

Co-authored-by: Itamar Oren <itamarost@gmail.com>

* fix capturing of eager tasks

* add comment to task finalization

* fix tests and couple c implmentation to c task

improved linked-list logic and more comments

* fix test

---------

Co-authored-by: Itamar Oren <itamarost@gmail.com>
2024-06-22 10:58:35 -07:00
Sam Gross 8f17d69b7b
gh-119344: Make critical section API public (#119353)
This makes the following macros public as part of the non-limited C-API for
locking a single object or two objects at once.

* `Py_BEGIN_CRITICAL_SECTION(op)` / `Py_END_CRITICAL_SECTION()`
* `Py_BEGIN_CRITICAL_SECTION2(a, b)` / `Py_END_CRITICAL_SECTION2()`

The supporting functions and structs used by the macros are also exposed for
cases where C macros are not available.
2024-06-21 15:50:18 -04:00
Victor Stinner 913a956d85
gh-119182: Rewrite PyUnicodeWriter tests in Python (#120845) 2024-06-21 20:15:06 +02:00
Victor Stinner 4123226bbd
gh-119182: Add PyUnicodeWriter_DecodeUTF8Stateful() (#120639)
Add PyUnicodeWriter_WriteWideChar() and
PyUnicodeWriter_DecodeUTF8Stateful() functions.

Co-authored-by: Serhiy Storchaka <storchaka@gmail.com>
2024-06-21 19:33:15 +02:00
neonene a81d434c06
gh-120782: Update internal type cache when reloading datetime (#120829) 2024-06-21 22:39:33 +05:30
Petr Viktorin 6f1d448bc1
gh-113993: Allow interned strings to be mortal, and fix related issues (GH-120520)
* Add an InternalDocs file describing how interning should work and how to use it.

* Add internal functions to *explicitly* request what kind of interning is done:
  - `_PyUnicode_InternMortal`
  - `_PyUnicode_InternImmortal`
  - `_PyUnicode_InternStatic`

* Switch uses of `PyUnicode_InternInPlace` to those.

* Disallow using `_Py_SetImmortal` on strings directly.
  You should use `_PyUnicode_InternImmortal` instead:
  - Strings should be interned before immortalization, otherwise you're possibly
    interning a immortalizing copy.
  - `_Py_SetImmortal` doesn't handle the `SSTATE_INTERNED_MORTAL` to
    `SSTATE_INTERNED_IMMORTAL` update, and those flags can't be changed in
    backports, as they are now part of public API and version-specific ABI.

* Add private `_only_immortal` argument for `sys.getunicodeinternedsize`, used in refleak test machinery.

* Make sure the statically allocated string singletons are unique. This means these sets are now disjoint:
  - `_Py_ID`
  - `_Py_STR` (including the empty string)
  - one-character latin-1 singletons

  Now, when you intern a singleton, that exact singleton will be interned.

* Add a `_Py_LATIN1_CHR` macro, use it instead of `_Py_ID`/`_Py_STR` for one-character latin-1 singletons everywhere (including Clinic).

* Intern `_Py_STR` singletons at startup.

* For free-threaded builds, intern `_Py_LATIN1_CHR` singletons at startup.

* Beef up the tests. Cover internal details (marked with `@cpython_only`).

* Add lots of assertions

Co-Authored-By: Eric Snow <ericsnowcurrently@gmail.com>
2024-06-21 17:19:31 +02:00
Sam Gross 3af7263037
gh-117511: Make PyMutex public in the non-limited API (#117731) 2024-06-20 11:29:08 -04:00
Jelle Zijlstra e8e151d471
gh-120780: Show attribute name for LOAD_SPECIAL in dis output (#120781) 2024-06-20 07:07:24 -07:00
Mark Shannon 00257c746c
GH-119462: Enforce invariants of type versioning (GH-120731)
* Remove uses of Py_TPFLAGS_VALID_VERSION_TAG
2024-06-19 17:38:45 +01:00
Barney Gale 20d5b84f57
GH-73991: Add follow_symlinks argument to `pathlib.Path.copy()` (#120519)
Add support for not following symlinks in `pathlib.Path.copy()`.

On Windows we add the `COPY_FILE_COPY_SYMLINK` flag is following symlinks is disabled. If the source is symlink to a directory, this call will fail with `ERROR_ACCESS_DENIED`. In this case we add `COPY_FILE_DIRECTORY` to the flags and retry. This can fail on old Windowses, which we note in the docs.

No news as `copy()` was only just added.
2024-06-19 00:59:54 +00:00
Sam Gross e8752d7b80
gh-118789: Add `PyUnstable_Object_ClearWeakRefsNoCallbacks` (#118807)
This exposes `PyUnstable_Object_ClearWeakRefsNoCallbacks` as an unstable
C-API function to provide a thread-safe mechanism for clearing weakrefs
without executing callbacks.

Some C-API extensions need to clear weakrefs without calling callbacks,
such as after running finalizers like we do in subtype_dealloc.
Previously they could use `_PyWeakref_ClearRef` on each weakref, but
that's not thread-safe in the free-threaded build.

Co-authored-by: Petr Viktorin <encukou@gmail.com>
2024-06-18 09:57:23 -04:00
Bénédikt Tran 7c5da94b5d
gh-120674: Protect multi-line macros in _testbuffer.c and _testcapimodule.c (#120675)
Add do { ... } while (0) pattern.
2024-06-18 12:04:52 +00:00
Victor Stinner 5c4235cd8c
gh-119182: Add PyUnicodeWriter C API (#119184) 2024-06-17 17:10:52 +02:00
Bénédikt Tran 4bf17c381f
gh-119933: Improve ``SyntaxError`` message for invalid type parameters expressions (#119976)
Co-authored-by: Jelle Zijlstra <jelle.zijlstra@gmail.com>
2024-06-17 06:51:03 -07:00
Victor Stinner c2d5df5787
gh-83754: Use the Py_TYPE() macro (#120599)
Don't access directly PyObject.ob_type, but use the Py_TYPE() macro
instead.
2024-06-17 10:34:29 +02:00
Nikita Sobolev 3df2022931
gh-120586: Fix several "unused function" warnings in `posixmodule.c` (#120588) 2024-06-17 09:44:13 +03:00
Nikita Sobolev b337aefd3e
gh-120584: Fix "unused thread_critical_sections" warning in `test_critical_sections` (#120585) 2024-06-16 20:13:56 +03:00
Eric Snow b2e71ff4f8
gh-120161: Fix a Crash in the _datetime Module (gh-120182)
In gh-120009 I used an atexit hook to finalize the _datetime module's static types at interpreter shutdown.  However, atexit hooks are executed very early in finalization, which is a problem in the few cases where a subclass of one of those static types is still alive until the final GC collection.  The static builtin types don't have this probably because they are finalized toward the end, after the final GC collection.  To avoid the problem for _datetime, I have applied a similar approach here.

Also, credit goes to @mgorny and @neonene for the new tests.

FYI, I would have liked to take a slightly cleaner approach with managed static types, but wanted to get a smaller fix in first for the sake of backporting.  I'll circle back to the cleaner approach with a future change on the main branch.
2024-06-14 13:29:09 -06:00
neonene 50a389565a
gh-117398: Add datetime C-API type check test for subinterpreters (gh-119604)
Check if the DateTime C-API type matches the datetime.date type on main and shared/isolated subinterpreters.
2024-06-13 12:05:03 -06:00
neonene 127c1d2771
gh-71587: Drop local reference cache to `_strptime` module in `_datetime` (gh-120224)
The _strptime module object was cached in a static local variable (in the datetime.strptime() implementation).  That's a problem when it crosses isolation boundaries, such as reinitializing the runtme or between interpreters.  This change fixes the problem by dropping the static variable, instead always relying on the normal sys.modules cache (via PyImport_Import()).
2024-06-12 10:46:39 -06:00
Victor Stinner 42b25dd61f
gh-120155: Add assertion to sre.c match_getindex() (#120402)
Add an assertion to help static analyzers to detect that i*2 cannot
overflow.
2024-06-12 15:27:07 +02:00
Bénédikt Tran 32d3e05fe6
gh-120029: remove unused macros in ``symtable.c`` (#120222)
Co-authored-by: Carl Meyer <carl@oddbird.net>
Co-authored-by: Jelle Zijlstra <jelle.zijlstra@gmail.com>
2024-06-12 09:23:45 -04:00
Bénédikt Tran 755dab719d
gh-120029: make `symtable.Symbol.__repr__` correctly reflect the compiler's flags, add methods (#120099)
Expose :class:`symtable.Symbol` methods :meth:`~symtable.Symbol.is_free_class`,
:meth:`~symtable.Symbol.is_comp_iter` and :meth:`~symtable.Symbol.is_comp_cell`.

---------

Co-authored-by: Carl Meyer <carl@oddbird.net>
2024-06-12 05:14:50 -06:00
Nikita Sobolev f5a9c34f38
gh-120056: Add `IP_RECVERR`, `IP_RECVORIGDSTADDR`, `IP_RECVTTL` to `socket` module (#120058)
* gh-120056: Add `IP_RECVERR` and `IP_RECVTTL` to `socket` module

* Fix news

* Address review

* Update NEWS
2024-06-11 21:00:56 -04:00
Pablo Galindo Salgado ec3af291fe
gh-120346: Respect PYTHON_BASIC_REPL when running in interactive inspect mode (#120349) 2024-06-11 16:15:01 +00:00
Victor Stinner 7aff2de62b
gh-120057: Add os.environ.refresh() method (#120059) 2024-06-10 16:34:17 +00:00
Victor Stinner b90bd3e5bb
gh-120155: Fix Coverity issue in zoneinfo load_data() (#120232)
Declare the 'rv' varaible at the top of the load_data() function to
make sure that it's initialized before the first 'goto error' which
uses 'rv' (return rv).

Fix the Coverity issue:

Error: UNINIT (CWE-457):
Python-3.12.2/Modules/_zoneinfo.c:1233:5: skipped_decl: Jumping over declaration of ""rv"".
Python-3.12.2/Modules/_zoneinfo.c:1284:5: uninit_use: Using uninitialized value ""rv"".
  1282|       }
  1283|
  1284|->     return rv;
  1285|   }
  1286|
2024-06-10 11:54:35 +02:00
Clinton e5a7bc6f2e
gh-120296: Fix format string of fcntl.ioctl() audit (#120301) 2024-06-10 08:17:50 +00:00
neonene 38a25e9560
gh-120244: Fix re.sub() reference leak (GH-120245) 2024-06-08 10:22:07 +00:00
Saul Shanabrook 55402d3232
gh-119258: Eliminate Type Guards in Tier 2 Optimizer with Watcher (GH-119365)
Co-authored-by: parmeggiani <parmeggiani@spaziodati.eu>
Co-authored-by: dpdani <git@danieleparmeggiani.me>
Co-authored-by: blurb-it[bot] <43283697+blurb-it[bot]@users.noreply.github.com>
Co-authored-by: Brandt Bucher <brandtbucher@microsoft.com>
Co-authored-by: Ken Jin <kenjin@python.org>
2024-06-08 17:41:45 +08:00
Jacob Walls 6b606522ca
gh-119577: Adjust DeprecationWarning when testing element truth values in ElementTree (GH-119762)
Adjust DeprecationWarning when testing element truth values in ElementTree, we're planning to go with the more natural True return rather than a disruptive harder to code around exception raise, and are deferring the behavior change for a few more releases.
2024-06-06 20:18:30 -07:00
Sam Gross e21057b999
gh-117657: Fix TSAN race involving import lock (#118523)
This adds a `_PyRecursiveMutex` type based on `PyMutex` and uses that
for the import lock. This fixes some data races in the free-threaded
build and generally simplifies the import lock code.
2024-06-06 13:40:58 -04:00
Victor Stinner 78634cfa3d
gh-120155: Initialize variables in _tkinter.c (#120156)
Initialize variables in _tkinter.c to make static analyzers happy.
2024-06-06 17:31:33 +02:00
benchatt 14e3c7071b
gh-115225: Raise error on unsupported ISO 8601 time strings (#119339)
Some time strings that contain fractional hours or minutes are permitted
by ISO 8601, but such strings are very unlikely to be intentional. The
current parser does not parse such strings correctly or raise an error.
This change raises a ValueError when hours or minutes contain a decimal mark.

Co-authored-by: blurb-it[bot] <43283697+blurb-it[bot]@users.noreply.github.com>
2024-06-05 13:35:40 -04:00
Pablo Galindo Salgado d9095194dd
gh-119842: Honor PyOS_InputHook in the new REPL (GH-119843)
Signed-off-by: Pablo Galindo <pablogsal@gmail.com>
Co-authored-by: Łukasz Langa <lukasz@langa.pl>
Co-authored-by: Michael Droettboom <mdboom@gmail.com>
2024-06-04 20:32:43 +02:00
Christopher Chavez e079935282
gh-112672: Fix builtin Tkinter with Tcl 9.0 (GH-112681)
* Add declaration of Tcl_AppInit(), missing in Tcl 9.0.
* Use Tcl_Size instead of int where needed.

Co-authored-by: Serhiy Storchaka <storchaka@gmail.com>
2024-06-04 19:47:15 +03:00
Serhiy Storchaka 7111d9605f
gh-89928: Fix integer conversion of device numbers (GH-31794)
Fix os.major(), os.minor() and os.makedev().
Support device numbers larger than 2**63-1.
Support non-existent device number (NODEV).
2024-06-04 19:36:37 +03:00
Bénédikt Tran ff1857d6ed
gh-120029: export `DEF_TYPE_PARAM` compiler flag (#120028) 2024-06-04 07:24:22 -07:00
Eric Snow 105f22ea46
gh-117398: Use Per-Interpreter State for the _datetime Static Types (gh-119929)
We make use of the same mechanism that we use for the static builtin types.  This required a few tweaks.

The relevant code could use some cleanup but I opted to avoid the significant churn in this change.  I'll tackle that separately.

This change is the final piece needed to make _datetime support multiple interpreters.  I've updated the module slot accordingly.
2024-06-03 17:09:18 -06:00
Eric Snow dba7a167db
gh-117142: Support Importing ctypes in Isolated Interpreters (gh-119991)
This makes the support official.

Co-authored-by: Kirill Podoprigora <kirill.bast9@mail.ru>
2024-06-03 16:42:48 -06:00
Eric Snow d82a7ba041
gh-117398: Add datetime Module State (gh-119810)
I was able to make use of the existing datetime_state struct, but there was one tricky thing I had to sort out.  We mostly aren't converting to heap types, so we can't use things like PyType_GetModuleByDef() to look up the module state.  The solution I came up with is somewhat novel, but I consider it straightforward.  Also, it shouldn't have much impact on performance.

In summary, this main changes here are:

* I've added some macros to help hide how various objects relate to module state
* as a solution to the module state lookup problem, I've stored the last loaded module on the current interpreter's internal dict (actually a weakref)
* if the static type method is used after the module has been deleted, it is reloaded
* to avoid extra work when loading the module, we directly copy the objects (new refs only) from the old module state into the new state if the old module hasn't been deleted yet
* during module init we set various objects on the static types' __dict__s; to simplify things, we only do that the first time; once those static types have a separate __dict__ per interpreter, we'll do it every time
* we now clear the module state when the module is destroyed (before, we were leaking everything in _datetime_global_state)
2024-06-03 15:56:00 -06:00
Sam Gross 47fb4327b5
gh-117657: Fix race involving immortalizing objects (#119927)
The free-threaded build currently immortalizes objects that use deferred
reference counting (see gh-117783). This typically happens once the
first non-main thread is created, but the behavior can be suppressed for
tests, in subinterpreters, or during a compile() call.

This fixes a race condition involving the tracking of whether the
behavior is suppressed.
2024-06-03 20:58:41 +00:00
Sergey B Kirpichev 61d3ab32da
gh-116560: Add PyLong_GetSign() public function (#116561)
Co-authored-by: Victor Stinner <vstinner@python.org>
2024-06-03 14:06:31 +02:00
Radislav Chugunov 52586f930f
gh-119506: fix `_io.TextIOWrapper.write()` write during flush (#119507)
Co-authored-by: Inada Naoki <songofacandy@gmail.com>
2024-06-03 16:47:36 +09:00
Victor Stinner 64ff1e217d
gh-119770: Make termios ioctl() constants positive (#119840) 2024-05-31 17:18:40 +02:00
Sam Gross bcc1be39cb
gh-119585: Fix crash involving `PyGILState_Release()` and `PyThreadState_Clear()` (#119753)
Make sure that `gilstate_counter` is not zero in when calling
`PyThreadState_Clear()`. A destructor called from `PyThreadState_Clear()` may
call back into `PyGILState_Ensure()` and `PyGILState_Release()`. If
`gilstate_counter` is zero, it will try to create a new thread state before
the current active thread state is destroyed, leading to an assertion failure
or crash.
2024-05-31 10:50:52 -04:00
Christopher Chavez 94e9585e99
gh-103194: Fix Tkinter’s Tcl value type handling for Tcl 8.7/9.0 (GH-103846)
Some of standard Tcl types were renamed, removed, or no longer
registered in Tcl 8.7/9.0. This change fixes automatic conversion of Tcl
values to Python values to avoid returning a Tcl_Obj where the primary
Python types (int, bool, str, bytes) were returned in older Tcl.
2024-05-31 11:23:53 +03:00
Irit Katriel 13a5fdc72f
gh-119744: move a few functions from compile.c to flowgraph.c (#119745) 2024-05-30 21:55:06 +01:00
Steve Dower 78d697b7d5
gh-119690: Adds Unicode support for named pipes in _winapi (GH-119717) 2024-05-29 16:51:09 +01:00
Kirill Podoprigora 34f9b3e724
gh-119655: Fix reference leak in the ``_datetimemodule.c`` (gh-119713) 2024-05-29 09:43:03 -06:00
Matthias Görgens 18c1a8d3a8
gh-97588: Align ctypes struct layout to GCC/MSVC (GH-97702)
Structure layout, and especially bitfields, sometimes resulted in clearly
wrong behaviour like overlapping fields. This fixes

Co-authored-by: Gregory P. Smith <gps@python.org>
Co-authored-by: Petr Viktorin <encukou@gmail.com>
2024-05-29 12:02:53 +02:00
Victor Stinner 7ca74a760a
gh-119661: Add _Py_SINGLETON() include in Argumenet Clinic (#119712)
When the _Py_SINGLETON() is used, Argument Clinic now adds an
explicit "pycore_runtime.h" include to get the macro. Previously, the
macro may or may not be included indirectly by another include.
2024-05-29 11:37:04 +02:00
Sergey B Kirpichev cd11ff12ac
gh-119613: Use C99+ functions instead of Py_IS_NAN/INFINITY/FINITE (#119619) 2024-05-29 09:51:19 +02:00
Eric Snow 548a11d5cf
gh-117398: Convert datetime.IsoCalendarDate To A Heap Type (gh-119637)
This is the only static type in the module that we will not keep static.
2024-05-28 16:42:23 -06:00
Petr Viktorin a89fc26f4a
gh-117398: gh-119655: datetime: Init static state once & don't free it (GH-119662)
- While datetime uses global state, only initialize it once.
- While `capi` is static, don't free it (thanks @neonene in https://github.com/python/cpython/pull/119641/files#r1616710048)
2024-05-28 17:27:52 +00:00
Serhiy Storchaka b313cc68d5
gh-117557: Improve error messages when a string, bytes or bytearray of length 1 are expected (GH-117631) 2024-05-28 12:01:37 +03:00
Erlend E. Aasland 3e8b60905e
gh-117398: Add multiphase support to _datetime (gh-119373)
This is minimal support.  Subinterpreters are not supported yet.  That will be addressed in a later change.

Co-authored-by: Eric Snow <ericsnowcurrently@gmail.com>
2024-05-27 16:02:46 -06:00
Eric Snow ae7b17673f
gh-119584: Fix test_import Failed Assertion (gh-119623)
The fix in gh-119561 introduced an assertion that doesn't hold true if any of the three new test extension modules are loaded more than once.  This is fine normally but breaks if the new test_check_state_first() is run more than once, which happens for refleak checking and with the regrtest --forever flag.  We fix that here by clearing each of the three modules after loading them.  We also tweak a check in _modules_by_index_check().
2024-05-27 19:35:30 +00:00
Eric Snow 0c5ebe13e9
gh-119560: Drop an Invalid Assert in PyState_FindModule() (gh-119561)
The assertion was added in gh-118532 but was based on the invalid assumption that PyState_FindModule() would only be called with an already-initialized module def.  I've added a test to make sure we don't make that assumption again.
2024-05-25 19:30:48 +00:00
Nice Zombies 96b392df30
gh-118263: Add additional arguments to path_t (Argument Clinic type) in posixmodule (GH-118355) 2024-05-24 19:04:17 +01:00
Victor Stinner 92fab3356f
gh-69214: Fix fcntl.ioctl() request type (#119498)
Use an 'unsigned long' instead of an 'unsigned int' for the request
parameter of fcntl.ioctl() to support requests larger than UINT_MAX.
2024-05-24 14:31:40 +02:00
Victor Stinner e94dbe4ed8
gh-119461: Fix ThreadedVSOCKSocketStreamTest (#119465)
Add socket.VMADDR_CID_LOCAL constant.

Fix ThreadedVSOCKSocketStreamTest: if get_cid() returns the host
address or the "any" address, use the local communication address
(loopback): VMADDR_CID_LOCAL.

On Linux 6.9, apparently, the /dev/vsock device is now available but
get_cid() returns VMADDR_CID_ANY (-1).
2024-05-23 23:26:09 +02:00
Eric Snow b30d30c747
gh-117398: Statically Allocate the Datetime C-API (GH-119472) 2024-05-23 21:15:52 +02:00
Petr Viktorin a192547dfe
gh-117142: Slightly hacky fix for memory leak of StgInfo (GH-119424)
Add a funciton that inlines PyObject_GetTypeData and skips
type-checking, so it doesn't need access to the CType_Type object.
This will break if the memory layout changes, but should
be an acceptable solution to enable ctypes in subinterpreters in
Python 3.13.

Mark _ctypes as safe for multiple interpreters

Co-authored-by: neonene <53406459+neonene@users.noreply.github.com>
2024-05-23 16:01:37 +00:00
neonene e12a6780bb
gh-117142: ctypes: Clean up c-analyzer .tsv files (GH-117544)
Co-authored-by: Petr Viktorin <encukou@gmail.com>
2024-05-22 20:30:41 +00:00
Geoffrey Thomas ef172521a9
Remove almost all unpaired backticks in docstrings (#119231)
As reported in #117847 and #115366, an unpaired backtick in a docstring
tends to confuse e.g. Sphinx running on subclasses of standard library
objects, and the typographic style of using a backtick as an opening
quote is no longer in favor. Convert almost all uses of the form

    The variable `foo' should do xyz

to

    The variable 'foo' should do xyz

and also fix up miscellaneous other unpaired backticks (extraneous /
missing characters).

No functional change is intended here other than in human-readable
docstrings.
2024-05-22 12:35:18 -04:00
Eric Snow 81865002ae
gh-119213: Be More Careful About _PyArg_Parser.kwtuple Across Interpreters (gh-119331)
_PyArg_Parser holds static global data generated for modules by Argument Clinic.  The _PyArg_Parser.kwtuple field is a tuple object, even though it's stored within a static global.  In some cases the tuple is statically allocated and thus it's okay that it gets shared by multiple interpreters.  However, in other cases the tuple is set lazily, allocated from the heap using the active interprepreter at the point the tuple is needed.

This is a problem once that interpreter is destroyed since _PyArg_Parser.kwtuple becomes at dangling pointer, leading to crashes.  It isn't a problem if the tuple is allocated under the main interpreter, since its lifetime is bound to the lifetime of the runtime.  The solution here is to temporarily switch to the main interpreter.  The alternative would be to always statically allocate the tuple.

This change also fixes a bug where only the most recent parser was added to the global linked list.
2024-05-22 09:57:52 -06:00
Irit Katriel 6e9863d7a3
gh-118692: Avoid creating unnecessary StopIteration instances for monitoring (#119216) 2024-05-21 20:42:51 +00:00
Nice Zombies b64182550f
gh-118507 : Refactor `nt._path_is*` to improve applicability for other cases (GH-118755) 2024-05-21 21:36:36 +01:00
Arnon Yaari 87939bd579
gh-117657: Fix itertools.count thread safety (#119268)
Fix itertools.count in free-threading mode
2024-05-21 10:16:34 -07:00
Erlend E. Aasland af359cee75
gh-118928: sqlite3: correctly bail if sequences of params are used with named placeholders (#119197) 2024-05-20 09:44:42 -04:00
Kirill Podoprigora 100c7ab00a
gh-119049: Fix incorrect display of warning which is constructed by C API (GH-119063)
The source line was not displayed if the warnings module had not yet
been imported.
2024-05-16 20:27:59 +00:00
Steve Dower 94591dca51
gh-118486: Simplify test_win32_mkdir_700 to check the exact ACL (GH-119056) 2024-05-15 11:59:41 +01:00
Erlend E. Aasland d8e0e00919
gh-118928: sqlite3: disallow sequences of params with named placeholders (#118929)
Follow-up of gh-101693. The previous DeprecationWarning is replaced with 
raising sqlite3.ProgrammingError.

Co-authored-by: Hugo van Kemenade <1324225+hugovk@users.noreply.github.com>
2024-05-14 16:10:55 +00:00
Nikita Sobolev fc75792594
gh-118998: Handle errors correctly in `tmtotuple` in `timemodule` (#118999) 2024-05-14 00:20:59 +03:00
Victor Stinner 7d7eec595a
gh-117873: Revert _posixshmem.shm_open() change (#118901) 2024-05-13 16:03:52 +02:00
Hugo van Kemenade a019347947
gh-118924: Remove `sqlite3.version` and `sqlite3.version_info` (#118925) 2024-05-10 20:42:34 +00:00
Jelle Zijlstra 13d7cf997b
gh-118895: Call PyType_Ready() on typing.NoDefault (#118897) 2024-05-10 08:42:00 -07:00
Victor Stinner a895756aec
gh-117398: Move types to datetime state (#118606)
Move types to the datetime_state structure of the _datetime
extension.
2024-05-10 15:24:06 +02:00
Victor Stinner 7cc5e81a82
gh-117873: Use positional-only parameters in _posixshmem (#118012)
* shm_unlink() parameter becomes positional-only.
* shm_open() first parameter (path) becomes positional-only,
  the two following parameters remain positional-or-keyword.
2024-05-10 12:04:16 +02:00
Dobatymo e85e8deaf3
gh-118209: Add Windows structured exception handling to mmap module (GH-118213) 2024-05-10 10:47:30 +01:00
Sam Gross 46c808172f
Revert "gh-115432: Add critical section variant that handles a NULL object (#115433)" (#118861)
This reverts commit ad4f909e0e.

The API ended up not being used.
2024-05-09 18:33:53 -04:00
Steve Dower 8af84b503d
gh-118773: Use language-invariant SDDL string instead of aliases for ACLs. (GH-118800) 2024-05-09 17:43:21 +01:00
mpage c30d8e5d6c
gh-117657: Acquire a critical section around `SemLock.__{enter,exit}__` (#118812)
These methods are purely wrappers around `Semlock.{acquire,release}`,
which expect a critical section to be held.
2024-05-09 12:05:52 -04:00
Raymond Hettinger e8cbcf4955
GH-101588: Remove deprecated pickle/copy/deepcopy from itertools (gh-118816) 2024-05-09 03:31:47 -05:00
trag1c c4f9823be2
gh-118671: Updated dead ActiveState links (#118730)
Co-authored-by: blurb-it[bot] <43283697+blurb-it[bot]@users.noreply.github.com>
Co-authored-by: Kirill Podoprigora <kirill.bast9@mail.ru>
2024-05-08 01:06:38 -06:00
Abhinav Upadhyay 26bab423fb
bpo-115773: Use the right variable name based on the field we are trying read (#118591)
Co-authored-by: Pablo Galindo Salgado <Pablogsal@gmail.com>
2024-05-07 14:50:41 +00:00
Serhiy Storchaka 65f5e586a1
gh-66410: Do not stringify arguments of Tkinter callback (GH-98592)
Callbacks registered in the tkinter module now take arguments as
various Python objects (int, float, bytes, tuple), not just str.
To restore the previous behavior set tkinter module global wantobject to 1
before creating the Tk object or call the wantobject() method of the Tk object
with argument 1.
Calling it with argument 2 restores the current default behavior.
2024-05-07 12:07:32 +00:00
Youfu Zhang fe47d9bee3
gh-118314: Fix padding edge case in binascii.a2b_base64 strict mode (GH-118320)
Fix an edge case in `binascii.a2b_base64` strict mode, where
excessive padding was not detected when no padding is necessary.

Co-authored-by: Terry Jan Reedy <tjreedy@udel.edu>
Co-authored-by: Pieter Eendebak <pieter.eendebak@gmail.com>
2024-05-07 11:18:45 +02:00
Dino Viehland ff6cbb2503
gh-112075: use per-thread dict version pool (#118676)
use thread state set of dict versions
2024-05-07 00:22:26 +00:00
Sam Gross 723d4d2fe8
gh-118527: Intern code consts in free-threaded build (#118667)
We already intern and immortalize most string constants. In the
free-threaded build, other constants can be a source of reference count
contention because they are shared by all threads running the same code
objects.
2024-05-06 20:12:39 -04:00
Serhiy Storchaka 67ce820d51
gh-71592: Fix a leak in tkinter.Tk destructor when _debug is true (GH-118664) 2024-05-06 19:25:20 +00:00
Dino Viehland 5a1618a2c8
gh-118362: Fix thread safety around lookups from the type cache in the face of concurrent mutators (#118454)
Add _PyType_LookupRef and use incref before setting attribute on type
Makes setting an attribute on a class and signaling type modified atomic
Avoid adding re-entrancy exposing the type cache in an inconsistent state by decrefing after type is updated
2024-05-06 10:50:35 -07:00
Serhiy Storchaka 1ff626ebda
gh-71592: Add ability to trace Tcl commands executed by Tkinter (GH-118291)
This is an experimental feature, for internal use.

Setting tkinter._debug = True before creating the root window enables
printing every executed Tcl command (or a Tcl command equivalent to the
used Tcl C API).

This will help to convert a Tkinter example into Tcl script to check
whether the issue is caused by Tkinter or exists in the underlying Tcl/Tk
library.
2024-05-06 20:12:51 +03:00
Petr Viktorin 417dd3aca7
gh-116322: Rename PyModule_ExperimentalSetGIL to PyUnstable_Module_SetGIL (GH-118645) 2024-05-06 18:59:36 +02:00
Pieter Eendebak 05adfbba2a
gh-95382: Improve performance of json encoder with indent (GH-118105) 2024-05-06 11:04:39 +03:00
Pablo Galindo Salgado f27f8c790a
gh-111201: A new Python REPL (GH-111567)
Co-authored-by: Łukasz Langa <lukasz@langa.pl>
Co-authored-by: Marta Gómez Macías <mgmacias@google.com>
Co-authored-by: Lysandros Nikolaou <lisandrosnik@gmail.com>
Co-authored-by: Hugo van Kemenade <1324225+hugovk@users.noreply.github.com>
Co-authored-by: Jelle Zijlstra <jelle.zijlstra@gmail.com>
2024-05-05 21:32:23 +02:00
Victor Stinner aa61f8bfcf
gh-110850: Remove _PyTime_TimeUnchecked() function (#118552)
Use the new public Raw functions:

* _PyTime_PerfCounterUnchecked() with PyTime_PerfCounterRaw()
* _PyTime_TimeUnchecked() with PyTime_TimeRaw()
* _PyTime_MonotonicUnchecked() with PyTime_MonotonicRaw()

Remove internal functions:

* _PyTime_PerfCounterUnchecked()
* _PyTime_TimeUnchecked()
* _PyTime_MonotonicUnchecked()
2024-05-05 12:15:19 +02:00
Davide Rizzo 08d169f14a
gh-109617: fix ncurses incompatibility on macOS with Xcode 15 (#111258)
Co-authored-by: Hugo van Kemenade <1324225+hugovk@users.noreply.github.com>
Co-authored-by: Łukasz Langa <lukasz@langa.pl>
2024-05-04 23:41:47 +02:00
Irit Katriel 85af789961
gh-111997: C-API for signalling monitoring events (#116413) 2024-05-04 08:23:50 +00:00
Brett Simmers c2627d6eea
gh-116322: Add Py_mod_gil module slot (#116882)
This PR adds the ability to enable the GIL if it was disabled at
interpreter startup, and modifies the multi-phase module initialization
path to enable the GIL when loading a module, unless that module's spec
includes a slot indicating it can run safely without the GIL.

PEP 703 called the constant for the slot `Py_mod_gil_not_used`; I went
with `Py_MOD_GIL_NOT_USED` for consistency with gh-104148.

A warning will be issued up to once per interpreter for the first
GIL-using module that is loaded. If `-v` is given, a shorter message
will be printed to stderr every time a GIL-using module is loaded
(including the first one that issues a warning).
2024-05-03 11:30:55 -04:00
Alexander Kanavin 37ccf16786
gh-101732: Modules/_ssl.c: use Y2038 compatible openssl function when available (GH-118425) 2024-05-03 15:34:05 +02:00
Jelle Zijlstra ca269e58c2
gh-116126: Implement PEP 696 (#116129)
Co-authored-by: Alex Waygood <Alex.Waygood@Gmail.com>
Co-authored-by: Bénédikt Tran <10796600+picnixz@users.noreply.github.com>
Co-authored-by: Shantanu <12621235+hauntsaninja@users.noreply.github.com>
2024-05-03 06:17:32 -07:00
Eric Snow f201628073
gh-117953: Other Cleanups in the Extensions Machinery (gh-118206)
This change will make some later changes simpler.
2024-05-03 00:51:43 +00:00
Steve Dower 8ed5466795
gh-118486: Switch mkdir(mode=0o700) on Windows to use OWNER RIGHTS instead of CURRENT_USER (GH-118515) 2024-05-02 19:43:54 +01:00
Pablo Galindo Salgado 6bcbee09df
gh-93502: Add new C-API functions to trace object creation and destruction (#115945) 2024-05-02 19:30:00 +02:00
Sergey B Kirpichev 9789440de3
gh-82062: Fix support of parameter defaults on methods in extension modules (GH-115270)
Now inspect.signature() supports references to the module globals in
parameter defaults on methods in extension modules.  Previously it was
only supported in functions.  The workaround was to specify the fully
qualified name, including the module name.
2024-05-02 17:44:33 +03:00
Steve Dower 81939dad77
gh-118486: Support mkdir(mode=0o700) on Windows (GH-118488) 2024-05-02 15:20:43 +01:00
Victor Stinner b52c753e0f
gh-110850: Add PyTime_TimeRaw() function (#118394)
Add "Raw" variant of PyTime functions:

* PyTime_MonotonicRaw()
* PyTime_PerfCounterRaw()
* PyTime_TimeRaw()

Changes:

* Add documentation and tests. Tests release the GIL while calling
  raw clock functions.
* py_get_system_clock() and py_get_monotonic_clock() now check that
  the GIL is hold by the caller if raise_exc is non-zero.
* Reimplement "Unchecked" functions with raw clock functions.

Co-authored-by: Petr Viktorin <encukou@gmail.com>
2024-05-01 18:05:01 +00:00
Guido van Rossum 7d83f7bcc4
gh-118335: Configure Tier 2 interpreter at build time (#118339)
The code for Tier 2 is now only compiled when configured
with `--enable-experimental-jit[=yes|interpreter]`.

We drop support for `PYTHON_UOPS` and -`Xuops`,
but you can disable the interpreter or JIT
at runtime by setting `PYTHON_JIT=0`.
You can also build it without enabling it by default
using `--enable-experimental-jit=yes-off`;
enable with `PYTHON_JIT=1`.

On Windows, the `build.bat` script supports
`--experimental-jit`, `--experimental-jit-off`,
`--experimental-interpreter`.

In the C code, `_Py_JIT` is defined as before
when the JIT is enabled; the new variable
`_Py_TIER2` is defined when the JIT *or* the
interpreter is enabled. It is actually a bitmask:
1: JIT; 2: default-off; 4: interpreter.
2024-04-30 18:26:34 -07:00
Victor Stinner e93c39b47e
gh-118422: Fix run_fileexflags() test (#118429)
Don't test the undefined behavior of fileno()
on a closed file, but use fstat() as a reliable
test if the file was closed or not.
2024-04-30 22:32:55 +02:00
Shantanu 6999d68d28
gh-118218: Reuse return tuple in itertools.pairwise (GH-118219) 2024-04-30 23:16:52 +03:00
Erlend E. Aasland b568c2c1ff
gh-118406: Add signature for sqlite3.Connection objects (#118428) 2024-04-30 19:58:22 +00:00
Sam Gross b2c3b70c71
gh-118332: Fix deadlock involving stop the world (#118412)
Avoid detaching thread state when stopping the world. When re-attaching
the thread state, the thread would attempt to resume the top-most
critical section, which might now be held by a thread paused for our
stop-the-world request.
2024-04-30 15:01:28 -04:00
Nice Zombies 9a75d56d5d
gh-118379: Use PyTuple_Pack instead of Py_BuildValue if possible (GH-118381) 2024-04-30 17:55:15 +03:00
Serhiy Storchaka 17a8af9508
gh-118402: Fix inspect.signature() for functools.cmp_to_key() result (GH-118427) 2024-04-30 17:49:28 +03:00
Sam Gross 7ccacb220d
gh-117783: Immortalize objects that use deferred reference counting (#118112)
Deferred reference counting is not fully implemented yet. As a temporary
measure, we immortalize objects that would use deferred reference
counting to avoid multi-threaded scaling bottlenecks.

This is only performed in the free-threaded build once the first
non-main thread is started. Additionally, some tests, including refleak
tests, suppress this behavior.
2024-04-29 14:36:02 -04:00
Serhiy Storchaka 444ac0b7a6
gh-118285: Fix signatures of operator.{attrgetter,itemgetter,methodcaller} instances (GH-118316)
* Allow to specify the signature of custom callable instances of extension
  type by the __text_signature__ attribute.
* Specify signatures of operator.attrgetter, operator.itemgetter, and
  operator.methodcaller instances.
2024-04-29 19:30:48 +03:00
Erlend E. Aasland c7e7bfc4ca
gh-115119: Detect _decimal dependencies using pkg-config (#115406)
pkg-config is supported for libmpdec 4.0.0 and newer.
2024-04-29 08:58:57 +02:00
Henrik Tunedal 133c1a7cdb
gh-118293: Suppress mouse cursor feedback when launching Windows processes with multiprocessing (GH-118315) 2024-04-28 21:10:44 +00:00
Eric Snow 09c2947581
gh-110693: Pending Calls Machinery Cleanups (gh-118296)
This does some cleanup in preparation for later changes.
2024-04-26 01:05:51 +00:00
neonene 2c45148912
gh-117578: Introduce _PyType_GetModuleByDef2 private function (GH-117661)
Co-authored-by: Erlend E. Aasland <erlend.aasland@protonmail.com>
Co-authored-by: Petr Viktorin <encukou@gmail.com>
2024-04-25 13:51:31 +02:00
Nice Zombies 10bb90ed49
gh-102511: Speed up os.path.splitroot() with native helpers (GH-118089) 2024-04-25 10:07:38 +01:00
Serhiy Storchaka 546cbcfa0e
gh-117968: Make the test for closed file more safe in the C API tests (GH-118230)
The behavior of fileno() after fclose() is undefined, but it is the only
practical way to check whether the file was closed.
Only test this on the known platforms (Linux, Windows, macOS), where we
already tested that it works.
2024-04-25 08:00:42 +03:00
Steve Dower 4b10e209c7
gh-117786: Fix venv created from Windows Store install by restoring __PYVENV_LAUNCHER__ smuggling (GH-117814) 2024-04-24 23:00:55 +01:00
Eric Snow 03e3e31723
gh-76785: Rename _xxsubinterpreters to _interpreters (gh-117791)
See https://discuss.python.org/t/pep-734-multiple-interpreters-in-the-stdlib/41147/26.
2024-04-24 16:18:24 +00:00
morotti 8fa1248685
gh-117151: optimize BufferedWriter(), do not buffer writes that are the buffer size (GH-118037)
BufferedWriter() was buffering calls that are the exact same size as the buffer. it's a very common case to read/write in blocks of the exact buffer size.

it's pointless to copy a full buffer, it's costing extra memory copy and the full buffer will have to be written in the next call anyway.

Co-authored-by: rmorotti <romain.morotti@man.com>
2024-04-23 18:51:20 +03:00
Joe Jevnik eb927e9fc8
gh-68114: Fix handling for removed PyArg_ParseTuple 'w' formatters (GH-8204)
Co-authored-by: Joe Jevnik <joe@quantopian.com>
Co-authored-by: Petr Viktorin <encukou@gmail.com>
2024-04-23 13:15:15 +02:00
Seth Michael Larson c9829eec08
gh-116741: Upgrade libexpat to 2.6.2 (#117296)
Upgrade libexpat to 2.6.2
2024-04-22 18:15:08 -07:00
Erlend E. Aasland 550483b7e6
gh-117995: Don't raise DeprecationWarnings for indexed nameless params (#118001)
Filter out '?NNN' placeholders when looking for named params.

Co-authored-by: AN Long <aisk@users.noreply.github.com>
2024-04-22 08:43:20 +02:00
Savannah Ostrowski d8f350309d
GH-115874: Fix segfault in FutureIter_dealloc (GH-117741) 2024-04-19 22:30:52 +00:00
Russell Keith-Magee ab99438900
gh-114099: Modify preprocessor symbol usage to support older macOS SDKs (GH-118073)
Co-authored-by: Joshua Root jmr@macports.org
2024-04-19 14:56:33 -04:00
lit 7c6cc00211
gh-88035: update doc-string of `epoch` in timemodule.c (GH-118076)
Follow #88035, update doc-string of epoch in timemodule.c

The epoch is `January 1st, 1970 on all platforms`, according to
current documentation.
2024-04-19 13:28:46 +02:00
Donghee Na 94444ea45a
gh-112069: Add _PySet_NextEntryRef to be thread-safe. (gh-117990) 2024-04-19 00:18:22 +09:00
Serhiy Storchaka 6078f2033e
gh-117968: Add tests for the part of the PyRun family of the C API (GH-117982)
Co-authored-by: NGRsoftlab <78017794+NGRsoftlab@users.noreply.github.com>
Co-authored-by: Erlend E. Aasland <erlend.aasland@protonmail.com>
2024-04-17 23:45:35 +03:00
Irit Katriel c179c0e6cb
gh-117680: make _PyInstructionSequence a PyObject and use it in tests (#117629) 2024-04-17 16:42:04 +01:00
CF Bolz-Tereick 5a0209fc23
GH-100242: bring functools.py partial implementation more in line with C code (GH-100244)
in partial.__new__, before checking for the existence of the attribute
'func', first check whether the argument is an instance of partial.
2024-04-17 15:34:46 +02:00
Joshua Root 8515fd79fe
gh-117845: Detect libedit hook function signature in configure (#117870)
Older libedit versions (like Apple's) use a different type signature
for rl_startup_hook and rl_pre_input_hook. Add a configure check to
determine which signature is accepted by introducing the
Py_RL_STARTUP_HOOK_TAKES_ARGS macro in pyconfig.h.
2024-04-17 11:26:10 +02:00
neonene c520bf9bdf
gh-117613: Argument Clinic: ensure that 'defining_class' params are positional-only (#117781) 2024-04-16 09:52:45 +00:00
Victor Stinner 2cc916e147
gh-117613: Enhance test_clinic @defining_class tests (#117896) 2024-04-16 09:32:51 +02:00
Steve Dower 185999bb3a
gh-90329: Add _winapi.GetLongPathName and GetShortPathName and use in venv to reduce warnings (GH-117817) 2024-04-15 15:36:06 +01:00
Serhiy Storchaka 57bdb75975
gh-117694: Improve tests for PyEval_EvalCodeEx() (GH-117695) 2024-04-15 12:44:54 +03:00
Serhiy Storchaka 39a6b29756
gh-117764: Use Argument Clinic for signal.set_wakeup_fd() (GH-117777) 2024-04-12 11:21:00 +00:00
Serhiy Storchaka 245e426b55
gh-117764: Add signatures for functions in the faulthandler module (GH-117771) 2024-04-12 13:14:31 +03:00
Serhiy Storchaka 2e098abf95
gh-117764: Add signature for functools.partial() (GH-117775) 2024-04-12 10:00:38 +00:00
Serhiy Storchaka ffbd97428d
gh-117764: Add signatures and improve docstrings in the _thread module (GH-117772) 2024-04-12 09:49:59 +00:00
Serhiy Storchaka 49af71a04a
gh-117764: Add signatures for some objects in the itertools module (GH-117774) 2024-04-12 12:36:27 +03:00
Serhiy Storchaka 3a26e29215
gh-117764: Add signatures in the atexit module (GH-117776) 2024-04-12 12:35:56 +03:00
Serhiy Storchaka 35f60c3def
gh-117764: Add signatures for __reduce__ and __reduce_ex__ in the _io module (GH-117773)
__reduce__() does not have parameters, __reduce_ex__() has a single
parameter.
2024-04-12 12:22:17 +03:00