Commit Graph

8196 Commits

Author SHA1 Message Date
Dino Viehland afe8f376c0
gh-112075: Adapt more dict methods to Argument Clinic (#114256)
* Move more dict objects to argument clinic

* Improve doc strings

* More doc string improvements

* Update Objects/dictobject.c

Co-authored-by: Erlend E. Aasland <erlend.aasland@protonmail.com>

* Update Objects/dictobject.c

Co-authored-by: Erlend E. Aasland <erlend.aasland@protonmail.com>

* Update Objects/dictobject.c

Co-authored-by: Erlend E. Aasland <erlend.aasland@protonmail.com>

* Update Objects/dictobject.c

Co-authored-by: Erlend E. Aasland <erlend.aasland@protonmail.com>

* Update Objects/dictobject.c

Co-authored-by: Erlend E. Aasland <erlend.aasland@protonmail.com>

* Update Objects/dictobject.c

Co-authored-by: Erlend E. Aasland <erlend.aasland@protonmail.com>

* Update Objects/dictobject.c

Co-authored-by: Erlend E. Aasland <erlend.aasland@protonmail.com>

---------

Co-authored-by: Erlend E. Aasland <erlend.aasland@protonmail.com>
2024-01-23 22:10:04 +00:00
Sam Gross 412920a41e
gh-112532: Improve mimalloc page visiting (#114133)
This adds support for visiting abandoned pages in mimalloc and improves
the performance of the page visiting code. Abandoned pages contain
memory blocks from threads that have exited. At some point, they may be
later reclaimed by other threads. We still need to visit those pages in
the free-threaded GC because they contain live objects.

This also reduces the overhead of visiting mimalloc pages:

 * Special cases for full, empty, and pages containing only a single
   block.
 * Fix free_map to use one bit instead of one byte per block.
 * Use fast integer division by a constant algorithm when computing
   block offset from block size and index.
2024-01-22 13:10:21 -08:00
Sam Gross 1d6d5e854c
gh-112529: Use GC heaps for GC allocations in free-threaded builds (gh-114157)
* gh-112529: Use GC heaps for GC allocations in free-threaded builds

The free-threaded build's garbage collector implementation will need to
find GC objects by traversing mimalloc heaps. This hooks up the
allocation calls with the correct heaps by using a thread-local
"current_obj_heap" variable.

* Refactor out setting heap based on type
2024-01-21 01:14:45 +09:00
Donghee Na a34e4db28a
gh-111968: Fix --without-freelists build (gh-114270) 2024-01-18 23:25:13 +00:00
Donghee Na d5442851a6
gh-112087: Remove duplicated critical_section (gh-114268) 2024-01-18 23:03:28 +00:00
Donghee Na 7fa511ba57
gh-111968: Use per-thread freelists for generator in free-threading (gh-114189) 2024-01-18 18:15:00 +00:00
kcatss a571a2fd3f
gh-114050: Fix crash when more than two arguments are passed to int() (GH-114067)
Co-authored-by: Kirill Podoprigora <kirill.bast9@mail.ru>
2024-01-18 13:27:44 +02:00
Mano Sriram 60ca37fdee
gh-106293: Fix typos in Objects/object_layout.md (#106294)
Co-authored-by: Terry Jan Reedy <tjreedy@udel.edu>
2024-01-16 23:11:15 +00:00
Sam Gross b331381485
gh-112529: Track if debug allocator is used as underlying allocator (#113747)
* gh-112529: Track if debug allocator is used as underlying allocator

The GC implementation for free-threaded builds will need to accurately
detect if the debug allocator is used because it affects the offset of
the Python object from the beginning of the memory allocation. The
current implementation of `_PyMem_DebugEnabled` only considers if the
debug allocator is the outer-most allocator; it doesn't handle the case
of "hooks" like tracemalloc being used on top of the debug allocator.

This change enables more accurate detection of the debug allocator by
tracking when debug hooks are enabled.

* Simplify _PyMem_DebugEnabled
2024-01-16 13:42:15 -08:00
Donghee Na 42b90cf0d6
gh-112087: Update list impl to be thread-safe with manual CS (gh-113863) 2024-01-16 09:11:14 +09:00
Donghee Na 3eae76554b
gh-111968: Use per-thread slice_cache in free-threading (gh-113972) 2024-01-16 00:38:57 +09:00
Sam Gross 44e47dfba5
gh-112532: Fix memory block count for free-threaded build (gh-113995)
This fixes `_PyInterpreterState_GetAllocatedBlocks()` and
`_Py_GetGlobalAllocatedBlocks()` in the free-threaded builds. The
gh-113263 change that introduced multiple mimalloc heaps per-thread
broke the logic for counting the number of allocated blocks. For subtle
reasons, this led to reported reference count leaks in the refleaks
buildbots.
2024-01-16 00:37:36 +09:00
Sergey B Kirpichev 0f2fa6150b
gh-109598: make PyComplex_RealAsDouble/ImagAsDouble use __complex__ (GH-109647)
`PyComplex_RealAsDouble()`/`PyComplex_ImagAsDouble` now try to convert
an object to a `complex` instance using its `__complex__()` method
before falling back to the ``__float__()`` method.

PyComplex_ImagAsDouble() also will not silently return 0.0 for
non-complex types anymore.  Instead we try to call PyFloat_AsDouble()
and return 0.0 only if this call is successful.
2024-01-15 16:04:17 +01:00
Donghee Na efa738e862
gh-111968: Explicit handling for finalized freelist (gh-113929) 2024-01-12 00:31:28 +00:00
Donghee Na 2e7577b622
gh-111968: Use per-thread freelists for tuple in free-threading (gh-113921) 2024-01-12 03:46:28 +09:00
Nikita Sobolev 2ac4cf4743
gh-112640: Add `kwdefaults` parameter to `types.FunctionType.__new__` (#112641) 2024-01-11 00:42:30 -08:00
Donghee Na c65ae26f2b
gh-111968: Unify naming scheme for freelist (gh-113919) 2024-01-11 08:51:51 +09:00
Sam Gross 73ae2023a7
gh-113753: Clear finalized bit when putting PyAsyncGenASend back into free list (#113754) 2024-01-10 10:18:38 -08:00
Donghee Na f728f7242c
gh-111968: Use per-thread freelists for float in free-threading (gh-113886) 2024-01-10 15:47:13 +00:00
Donghee Na 57bdc6c30d
gh-111968: Introduce _PyFreeListState and _PyFreeListState_GET API (gh-113584) 2024-01-10 08:04:41 +09:00
Donghee Na a023bc252d
gh-112087: Update list.{pop,clear,reverse,remove} to use CS (gh-113764) 2024-01-09 09:00:55 +09:00
Jakub Kulík 10d3f04aec
gh-112808: Fix mimalloc build on Solaris (#112809) 2024-01-08 15:50:56 -08:00
Jakub Kulík ae968d1032
gh-112806: Remove unused function warnings during mimalloc build on Solaris (#112807) 2024-01-08 15:49:33 -08:00
Sam Gross d0f0308a37
gh-113750: Fix object resurrection in free-threaded builds (gh-113751)
gh-113750: Fix object resurrection on free-threaded builds

This avoids the undesired re-initializing of fields like `ob_gc_bits`,
`ob_mutex`, and `ob_tid` when an object is resurrected due to its
finalizer being called.

This change has no effect on the default (with GIL) build.
2024-01-06 12:12:26 +09:00
Sam Gross 0b7476080b
gh-112532: Tag mimalloc heaps and pages (#113742)
* gh-112532: Tag mimalloc heaps and pages

Mimalloc pages are data structures that contain contiguous allocations
of the same block size. Note that they are distinct from operating
system pages. Mimalloc pages are contained in segments.

When a thread exits, it abandons any segments and contained pages that
have live allocations. These segments and pages may be later reclaimed
by another thread. To support GC and certain thread-safety guarantees in
free-threaded builds, we want pages to only be reclaimed by the
corresponding heap in the claimant thread. For example, we want pages
containing GC objects to only be claimed by GC heaps.

This allows heaps and pages to be tagged with an integer tag that is
used to ensure that abandoned pages are only claimed by heaps with the
same tag. Heaps can be initialized with a tag (0-15); any page allocated
by that heap copies the corresponding tag.

* Fix conversion warning
2024-01-05 12:08:50 -08:00
Sam Gross fcb3c2a444
gh-112532: Isolate abandoned segments by interpreter (#113717)
* gh-112532: Isolate abandoned segments by interpreter

Mimalloc segments are data structures that contain memory allocations along
with metadata. Each segment is "owned" by a thread. When a thread exits,
it abandons its segments to a global pool to be later reclaimed by other
threads. This changes the pool to be per-interpreter instead of process-wide.

This will be important for when we use mimalloc to find GC objects in the
`--disable-gil` builds. We want heaps to only store Python objects from a
single interpreter. Absent this change, the abandoning and reclaiming process
could break this isolation.

* Add missing '&_mi_abandoned_default' to 'tld_empty'
2024-01-04 22:21:40 +00:00
Alex Waygood f1f8392432
Document the `co_lines` method on code objects (#113682)
Co-authored-by: Hugo van Kemenade <hugovk@users.noreply.github.com>
2024-01-03 19:29:24 +00:00
Christopher Chavez a1eea1d032
gh-111178: Avoid calling functions from incompatible pointer types in dictobject.c (#112892)
Fix undefined behavior warnings (UBSan  -fsanitize=function).
2024-01-02 15:32:37 +01:00
Christopher Chavez acf4cf5ca5
gh-111178: Avoid calling functions from incompatible pointer types in descrobject.c (GH-112861)
Fix undefined behavior warnings (UBSan  -fsanitize=function), for example:

Python/generated_cases.c.h:3315:13: runtime error: call to function mappingproxy_dealloc through pointer to incorrect function type 'void (*)(struct _object *)'
descrobject.c:1160: note: mappingproxy_dealloc defined here
SUMMARY: UndefinedBehaviorSanitizer: undefined-behavior Python/generated_cases.c.h:3315:13 in
2024-01-02 15:03:39 +01:00
Christopher Chavez 5d36a95e64
gh-111178: Avoid calling functions from incompatible pointer types in listobject.c (GH-112820)
Fix undefined behavior warnings (UBSan  -fsanitize=function), for example:

Objects/object.c:674:11: runtime error: call to function list_repr through pointer to incorrect function type 'struct _object *(*)(struct _object *)'
listobject.c:382: note: list_repr defined here
SUMMARY: UndefinedBehaviorSanitizer: undefined-behavior Objects/object.c:674:11 in
2024-01-02 14:41:32 +01:00
Sam Gross acf3bcc886
gh-112532: Use separate mimalloc heaps for GC objects (gh-113263)
* gh-112532: Use separate mimalloc heaps for GC objects

In `--disable-gil` builds, we now use four separate heaps in
anticipation of using mimalloc to find GC objects when the GIL is
disabled. To support this, we also make a few changes to mimalloc:

* `mi_heap_t` and `mi_tld_t` initialization is split from allocation.
  This allows us to have a `mi_tld_t` per-`PyThreadState`, which is
  important to keep interpreter isolation, since the same OS thread may
  run in multiple interpreters (using different PyThreadStates.)

* Heap abandoning (mi_heap_collect_ex) can now be called from a
  different thread than the one that created the heap. This is necessary
  because we may clear and delete the containing PyThreadStates from a
  different thread during finalization and after fork().

* Use enum instead of defines and guard mimalloc includes.

* The enum typedef will be convenient for future PRs that use the type.
* Guarding the mimalloc includes allows us to unconditionally include
  pycore_mimalloc.h from other header files that rely on things like
  `struct _mimalloc_thread_state`.

* Only define _mimalloc_thread_state in Py_GIL_DISABLED builds
2023-12-27 01:53:20 +09:00
Donghee Na 8f5b998706
gh-111971: Make _PyUnicode_FromId thread-safe in --disable-gil (gh-113489) 2023-12-26 16:48:33 +00:00
Yan Yanchii 4a3d2419bb
gh-113212: Improve error message & document zero-arg super inside nested functions and generator expressions (GH-113307) 2023-12-22 17:12:08 +02:00
Sam Gross 9afb0e1606
gh-112027: Don't print mimalloc warning after mmap() call (gh-113372)
gh-112027: Don't print mimalloc warning after mmap

This changes the warning to a "verbose"-level message in prim.c. The
address passed to mmap is only a hint -- it's normal for mmap() to
sometimes not respect the hint and return a different address.
2023-12-22 09:38:27 +09:00
Raymond Hettinger d058eaeed4
gh-113157 gh-89519: Fix method descriptors (gh-113233)
Restore behaviors before classmethod descriptor chaining was introduced.
2023-12-21 16:08:35 -06:00
Shantanu 61e8184095
gh-95754: Better AttributeError on partially initialised module (#112577)
Co-authored-by: Serhiy Storchaka <storchaka@gmail.com>
2023-12-21 13:24:10 -08:00
Erlend E. Aasland 526d0a9b6e
gh-110383: Improve accuracy of str.split() and str.rsplit() docstrings (#113355)
Clarify split direction in the docstring body,
instead of in the 'maxsplit' param docstring.
2023-12-21 15:22:39 +01:00
Carey Metcalfe a2dd0e7038
gh-111375: Use `NULL` rather than `None` in the exception stack to indicate that an exception was handled (#113302) 2023-12-21 01:46:41 +00:00
Christopher Chavez a545a86ec6
gh-111178: Make slot functions in typeobject.c have compatible types (GH-112752) 2023-12-20 15:13:44 +01:00
Sam Gross fdee7b7b3e
gh-112532: Require mimalloc in `--disable-gil` builds (gh-112883) 2023-12-12 09:04:48 +09:00
Christopher Chavez f4fe65e2dd
gh-111178: Avoid calling functions from incompatible pointer types in memoryobject.c (GH-112863)
* Make memory_clear() compatible with inquiry
* Make memory_traverse() compatible with traverseproc
* Make memory_dealloc() compatible with destructor
* Make memory_repr() compatible with reprfunc
* Make memory_hash() compatible with hashfunc
* Make memoryiter_next() compatible with iternextfunc
* Make memoryiter_traverse() compatible with traverseproc
* Make memoryiter_dealloc() compatible with destructor
* Make several functions compatible with getter
* Make a few functions compatible with getter
* Make memory_item() compatible with ssizeargfunc
* Make memory_subscript() compatible with binaryfunc
* Make memory_length() compatible with lenfunc
* Make memory_ass_sub() compatible with objobjargproc
* Make memory_releasebuf() compatible with releasebufferproc
* Make memory_getbuf() compatible with getbufferproc
* Make mbuf_clear() compatible with inquiry
* Make mbuf_traverse() compatible with traverseproc
* Make mbuf_dealloc() compatible with destructor
2023-12-11 17:43:07 +01:00
Sam Gross cf6110ba13
gh-111924: Use PyMutex for Runtime-global Locks. (gh-112207)
This replaces some usages of PyThread_type_lock with PyMutex, which does not require memory allocation to initialize.

This simplifies some of the runtime initialization and is also one step towards avoiding changing the default raw memory allocator during initialize/finalization, which can be non-thread-safe in some circumstances.
2023-12-07 12:33:40 -07:00
andrewluotechnologies 9c3458e058
gh-112125: Fix None.__ne__(None) returning NotImplemented instead of False (#112504) 2023-12-07 13:56:01 +01:00
Serhiy Storchaka 8660fb7fd7
gh-112660: Do not clear arbitrary errors on import (GH-112661)
Previously arbitrary errors could be cleared during formatting error
messages for ImportError or AttributeError for modules. Now all
unexpected errors are reported.
2023-12-07 12:19:43 +02:00
Sergey B Kirpichev 9f92b31339
Minor refactoring of Object/abstract.c (UNARY_FUNC macro and more cases for BINARY_FUNC) (GH-112145)
* Use BINARY_FUNC macro for some remaining ops
* Add UNARY_FUNC macro to define unary PyNumber_* functions
2023-12-05 09:34:13 +02:00
chilaxan 0e732d0997
gh-112625: Protect bytearray from being freed by misbehaving iterator inside bytearray.join (GH-112626) 2023-12-04 08:15:43 +00:00
Irit Katriel bfb576ee23
gh-111058: Change coro.cr_frame/gen.gi_frame to be None for a closed coroutine/generator. (#112428) 2023-12-01 12:57:31 +00:00
Kirill Podoprigora 0785c68559
gh-111972: Make Unicode name C APIcapsule initialization thread-safe (#112249) 2023-11-30 11:12:49 +01:00
Eric Snow 790db85c77
gh-76785: Add _PyType_GetModuleName() to the Internal C-API (gh-112323)
The new function corresponds to the existing (public) PyType_GetName() and PyType_GetQualName().
2023-11-22 15:03:33 -07:00
Hugo van Kemenade 3b3ec0d77f
gh-111863: Rename `Py_NOGIL` to `Py_GIL_DISABLED` (#111864)
Rename Py_NOGIL to Py_GIL_DISABLED
2023-11-20 15:52:00 +02:00
Nikita Sobolev f8129146ef
gh-112266: Remove `(if defined)` part from `__dict__` and `__weakref__` docstrings (#112268) 2023-11-19 18:30:07 +00:00
Victor Stinner bd2f1485b0
gh-112026: Restore removed _PyDict_GetItemStringWithError() (#112119)
Restore the removed _PyDict_GetItemStringWithError() function. It is
used by numpy.
2023-11-15 17:10:06 +00:00
Miro Hrončok ec05507367
gh-96954: Fix `make regen-unicodedata` in out-of-tree builds (#112118)
This avoids:

    python3.13 Tools/unicode/makeunicodedata.py
    python3.13: can't open file '.../build/debug/Tools/unicode/makeunicodedata.py': [Errno 2] No such file or directory
    make: *** [Makefile:1498: regen-unicodedata] Error 2

Re-run `make regen-unicodedata` to update the script path in generated files.
2023-11-15 16:42:17 +00:00
Victor Stinner 58469244be
gh-112026: Restore removed private C API (#112115)
Restore removed private C API functions, macros and structures which
have no simple replacement for now:

* _PyDict_GetItem_KnownHash()
* _PyDict_NewPresized()
* _PyHASH_BITS
* _PyHASH_IMAG
* _PyHASH_INF
* _PyHASH_MODULUS
* _PyHASH_MULTIPLIER
* _PyLong_Copy()
* _PyLong_FromDigits()
* _PyLong_New()
* _PyLong_Sign()
* _PyObject_CallMethodId()
* _PyObject_CallMethodNoArgs()
* _PyObject_CallMethodOneArg()
* _PyObject_CallOneArg()
* _PyObject_EXTRA_INIT
* _PyObject_FastCallDict()
* _PyObject_GetAttrId()
* _PyObject_Vectorcall()
* _PyObject_VectorcallMethod()
* _PyStack_AsDict()
* _PyThread_CurrentFrames()
* _PyUnicodeWriter structure
* _PyUnicodeWriter_Dealloc()
* _PyUnicodeWriter_Finish()
* _PyUnicodeWriter_Init()
* _PyUnicodeWriter_Prepare()
* _PyUnicodeWriter_PrepareKind()
* _PyUnicodeWriter_WriteASCIIString()
* _PyUnicodeWriter_WriteChar()
* _PyUnicodeWriter_WriteLatin1String()
* _PyUnicodeWriter_WriteStr()
* _PyUnicodeWriter_WriteSubstring()
* _PyUnicode_AsString()
* _PyUnicode_FromId()
* _PyVectorcall_Function()
* _Py_HashDouble()
* _Py_HashPointer()
* _Py_IDENTIFIER()
* _Py_c_abs()
* _Py_c_diff()
* _Py_c_neg()
* _Py_c_pow()
* _Py_c_prod()
* _Py_c_quot()
* _Py_c_sum()
* _Py_static_string()
* _Py_static_string_init()
2023-11-15 16:38:31 +00:00
Furkan Onder 0ff6368519
gh-111906: Fix warnings during mimalloc build on FreeBSD (#111907)
Fix `unused function` warnings during mimalloc build on FreeBSD.
2023-11-14 17:07:46 +01:00
Victor Stinner 4f04172c92
gh-111262: Add PyDict_Pop() function (#112028)
_PyDict_Pop_KnownHash(): remove the default value and the return type
becomes an int.

Co-authored-by: Stefan Behnel <stefan_ml@behnel.de>
Co-authored-by: Antoine Pitrou <pitrou@free.fr>
2023-11-14 12:51:00 +00:00
Serhiy Storchaka 18203a6bc9
gh-111789: Use PyDict_GetItemRef() in Objects/ (GH-111827) 2023-11-14 11:25:39 +02:00
Victor Stinner babb787047
gh-111138: Add PyList_Extend() and PyList_Clear() functions (#111862)
* Split list_extend() into two sub-functions: list_extend_fast() and
  list_extend_iter().
* list_inplace_concat() no longer has to call Py_DECREF() on the
  list_extend() result, since list_extend() now returns an int.
2023-11-13 16:14:56 +00:00
Serhiy Storchaka 1d75ef6b61
gh-111999: Add signatures and improve docstrings for builtins (GH-112000) 2023-11-13 09:13:49 +02:00
Sam Gross fa84e5fe0a
gh-110481: fix 'unused function' warning for `is_shared_refcnt_dead`. (gh-111974)
Fix 'unused function' warning for `is_shared_refcnt_dead`.

The `is_shared_refcnt_dead` function is only used if `Py_REF_DEBUG` is set.
2023-11-11 08:54:35 +09:00
Serhiy Storchaka 771bd3c94a
Add private _PyUnicode_AsUTF8NoNUL() function (GH-111957)
Like PyUnicode_AsUTF8(), but check for embedded null characters.
2023-11-10 21:31:36 +02:00
Pavel Ovchinnikov 0c61d028be
Improve error message for "float modulo by zero" (#111685) 2023-11-09 11:43:58 -08:00
Irit Katriel 30ec968bef
gh-111354: remove comparisons with enum values, variable reuse, unused imports in genobject.c (#111708) 2023-11-09 10:27:20 +00:00
Sam Gross 31c90d5838
gh-111569: Implement Python critical section API (gh-111571)
Critical sections are helpers to replace the global interpreter lock
with finer grained locking.  They provide similar guarantees to the GIL
and avoid the deadlock risk that plain locking involves.  Critical
sections are implicitly ended whenever the GIL would be released.  They
are resumed when the GIL would be acquired.  Nested critical sections
behave as if the sections were interleaved.
2023-11-08 15:39:29 -07:00
Jelle Zijlstra 0b718e6407
gh-110543: Fix CodeType.replace in presence of comprehensions (#110586) 2023-11-08 13:11:59 -07:00
Victor Stinner 11e83488c5
gh-111089: Revert PyUnicode_AsUTF8() changes (#111833)
* Revert "gh-111089: Use PyUnicode_AsUTF8() in Argument Clinic (#111585)"

This reverts commit d9b606b3d0.

* Revert "gh-111089: Use PyUnicode_AsUTF8() in getargs.c (#111620)"

This reverts commit cde1071b2a.

* Revert "gh-111089: PyUnicode_AsUTF8() now raises on embedded NUL (#111091)"

This reverts commit d731579bfb.

* Revert "gh-111089: Add PyUnicode_AsUTF8() to the limited C API (#111121)"

This reverts commit d8f32be5b6.

* Revert "gh-111089: Use PyUnicode_AsUTF8() in sqlite3 (#111122)"

This reverts commit 37e4e20eaa.
2023-11-07 22:36:13 +00:00
Irit Katriel 2f9cb7e095
gh-81137: deprecate assignment of code object to a function of a mismatched type (#111823) 2023-11-07 18:54:36 +00:00
Serhiy Storchaka f55cb44359
gh-106672: C API: Report indiscriminately ignored errors (GH-106674)
Functions which indiscriminately ignore all errors now report them as
unraisable errors.
2023-11-07 15:58:04 +02:00
Irit Katriel 13405ecffd
gh-79932: raise exception if frame.clear() is called on a suspended frame (#111792) 2023-11-07 08:49:30 +00:00
Irit Katriel c5063fd62a
simplify code to pop exception in frame_setlineno (#111702) 2023-11-06 15:45:33 +00:00
Nikita Sobolev a28a3967ab
gh-111666: Speed up `BaseExceptionGroup.{derive,split,subgroup}` (#111667) 2023-11-04 09:59:46 +00:00
Victor Stinner 20cfab903d
gh-111506: Implement Py_SET_REFCNT() as opaque function in limited C API (#111508)
In the limited C API version 3.13, Py_SET_REFCNT() function is now
implemented as an opaque function call.

Add _Py_SetRefcnt() to the stable ABI.
2023-11-03 18:18:57 +01:00
Tian Gao e0afed7e27
gh-103615: Use local events for opcode tracing (GH-109472)
* Use local monitoring for opcode trace

* Remove f_opcode_trace_set

* Add test for setting f_trace_opcodes after settrace
2023-11-03 16:39:50 +00:00
scoder 24ddaee5ca
gh-106168: Revert the "size before item" setting (#111683)
gh-106168: Update the size only after setting the item, to avoid temporary inconsistencies.
Also remove the "what's new" sentence regarding the size setting since tuples cannot grow after allocation.
2023-11-03 11:02:39 +00:00
Irit Katriel d49aba5a7a
gh-111354: Simplify _PyGen_yf by moving some of its work to the compiler and frame state (#111648) 2023-11-03 10:01:36 +00:00
Mark Shannon 0887b9ce8b
GH-109369 Add vectorcall to `PyLong_Type` (GH-111642) 2023-11-02 15:07:47 +00:00
Irit Katriel 52cc4af6ae
gh-111354: simplify detection of RESUME after YIELD_VALUE at except-depth 1 (#111459) 2023-11-02 10:18:43 +00:00
Serhiy Storchaka 970e719a7a
gh-108082: Use PyErr_FormatUnraisable() (GH-111580)
Replace most of calls of _PyErr_WriteUnraisableMsg() and some
calls of PyErr_WriteUnraisable(NULL) with PyErr_FormatUnraisable().

Co-authored-by: Victor Stinner <vstinner@python.org>
2023-11-02 09:16:34 +00:00
Ayappan Perumal 794dff2fb1
gh-111544: Fix mimalloc build on AIX (#111593) 2023-11-02 08:01:50 +01:00
Victor Stinner d9b606b3d0
gh-111089: Use PyUnicode_AsUTF8() in Argument Clinic (#111585)
Replace PyUnicode_AsUTF8AndSize() with PyUnicode_AsUTF8() to remove
the explicit check for embedded null characters.

The change avoids to have to include explicitly <string.h> to get the
strlen() function when using a recent version of the limited C API.
2023-11-01 16:34:42 +01:00
Dino Viehland c42347d025
gh-90815: Exclude mimalloc .c files from Windows build (#111532)
* Don't include mimalloc .c's in Windows build
* Fix warnings on Windows related to mimalloc
2023-10-31 11:54:35 -07:00
Eric Snow c6fe0869ab
gh-76785: Move the Cross-Interpreter Code to Its Own File (gh-111502)
This is partly to clear this stuff out of pystate.c, but also in preparation for moving some code out of _xxsubinterpretersmodule.c.  This change also moves this stuff to the internal API (new: Include/internal/pycore_crossinterp.h).  @vstinner did this previously and I undid it.  Now I'm re-doing it. :/
2023-10-30 16:53:10 -06:00
Victor Stinner 1673e44017
gh-90815: Fix mimalloc build on WASI (#111524)
Include <unistd.h> to get sbrk() function.
2023-10-30 23:10:22 +01:00
Victor Stinner da4d3142a2
gh-90815: Fix _PyMem_MimallocEnabled() compiler warning (#111522)
Don't declare _PyMem_MimallocEnabled() if WITH_PYMALLOC macro is not
defined (./configure --without-pymalloc).

Fix also a typo in _PyInterpreterState_FinalizeAllocatedBlocks().
2023-10-30 23:05:49 +01:00
Sam Gross 6dfb8fe023
gh-110481: Implement biased reference counting (gh-110764) 2023-10-30 16:06:09 +00:00
Dino Viehland 05f2f0ac92
gh-90815: Add mimalloc memory allocator (#109914)
* Add mimalloc v2.12

Modified src/alloc.c to remove include of alloc-override.c and not
compile new handler.

Did not include the following files:

 - include/mimalloc-new-delete.h
 - include/mimalloc-override.h
 - src/alloc-override-osx.c
 - src/alloc-override.c
 - src/static.c
 - src/region.c

mimalloc is thread safe and shares a single heap across all runtimes,
therefore finalization and getting global allocated blocks across all
runtimes is different.

* mimalloc: minimal changes for use in Python:

 - remove debug spam for freeing large allocations
 - use same bytes (0xDD) for freed allocations in CPython and mimalloc
   This is important for the test_capi debug memory tests

* Don't export mimalloc symbol in libpython.
* Enable mimalloc as Python allocator option.
* Add mimalloc MIT license.
* Log mimalloc in Lib/test/pythoninfo.py.
* Document new mimalloc support.
* Use macro defs for exports as done in:
  https://github.com/python/cpython/pull/31164/

Co-authored-by: Sam Gross <colesbury@gmail.com>
Co-authored-by: Christian Heimes <christian@python.org>
Co-authored-by: Victor Stinner <vstinner@python.org>
2023-10-30 15:43:11 +00:00
Savannah Ostrowski 4a929d432b
GH-111339: Fix initialization and finalization of static optimizer types (GH-111430) 2023-10-29 13:53:25 -07:00
Raymond Hettinger 7f9a99e854
gh-89519: Remove classmethod descriptor chaining, deprecated since 3.11 (gh-110163) 2023-10-27 00:24:56 -05:00
Irit Katriel a0c414c35d
gh-111354: define names for RESUME oparg values (#111365) 2023-10-26 16:30:18 +01:00
Irit Katriel 67a91f78e4
gh-109094: replace frame->prev_instr by frame->instr_ptr (#109095) 2023-10-26 13:43:10 +00:00
Irit Katriel 0db2517687
gh-100762: Fix optimization in gen_close (#111069) 2023-10-25 16:22:34 +01:00
Savannah Ostrowski 6640f1d8d2
GH-94438: Restore ability to jump over None tests (GH-111237) 2023-10-23 23:13:16 -07:00
Radislav Chugunov 47d3e2ed93
gh-109894: Fix initialization of static `MemoryError` in subinterpreter (gh-110911)
Fixes #109894

* set `interp.static_objects.last_resort_memory_error.args` to empty tuple to avoid crash on `PyErr_Display()` call
* allow `_PyExc_InitGlobalObjects()` to be called on subinterpreter init

---------

Co-authored-by: blurb-it[bot] <43283697+blurb-it[bot]@users.noreply.github.com>
2023-10-23 17:06:59 -06:00
Furkan Onder 32c37fe1ba
gh-67565: Remove redundant C-contiguity checks (GH-105521)
Co-authored-by: Stefan Krah <skrah@bytereef.org>
2023-10-23 12:54:46 +03:00
Victor Stinner f1e751e933
gh-111089: PyUnicode_AsUTF8AndSize() sets size on error (#111106)
On error, PyUnicode_AsUTF8AndSize() now sets the size argument to -1,
to avoid undefined value.
2023-10-20 20:03:11 +02:00
Victor Stinner d731579bfb
gh-111089: PyUnicode_AsUTF8() now raises on embedded NUL (#111091)
* PyUnicode_AsUTF8() now raises an exception if the string contains
  embedded null characters.
* Update related C API tests (test_capi.test_unicode).
* type_new_set_doc() uses PyUnicode_AsUTF8AndSize() to silently
  truncate doc containing null bytes.

Co-authored-by: Serhiy Storchaka <storchaka@gmail.com>
2023-10-20 17:59:29 +02:00
Victor Stinner be5e8a0103
gh-110964: Remove private _PyArg functions (#110966)
Move the following private functions and structures to
pycore_modsupport.h internal C API:

* _PyArg_BadArgument()
* _PyArg_CheckPositional()
* _PyArg_NoKeywords()
* _PyArg_NoPositional()
* _PyArg_ParseStack()
* _PyArg_ParseStackAndKeywords()
* _PyArg_Parser structure
* _PyArg_UnpackKeywords()
* _PyArg_UnpackKeywordsWithVararg()
* _PyArg_UnpackStack()
* _Py_ANY_VARARGS()

Changes:

* Python/getargs.h now includes pycore_modsupport.h to export
  functions.
* clinic.py now adds pycore_modsupport.h when one of these functions
  is used.
* Add pycore_modsupport.h includes when a C extension uses one of
  these functions.
* Define Py_BUILD_CORE_MODULE in C extensions which now include
  directly or indirectly (via code generated by Argument Clinic)
  pycore_modsupport.h:

  * _csv
  * _curses_panel
  * _dbm
  * _gdbm
  * _multiprocessing.posixshmem
  * _sqlite.row
  * _statistics
  * grp
  * resource
  * syslog

* _testcapi: bad_get() no longer uses METH_FASTCALL calling
  convention but METH_VARARGS. Replace _PyArg_UnpackStack() with
  PyArg_ParseTuple().
* _testcapi: add PYTESTCAPI_NEED_INTERNAL_API macro which is defined
  by _testcapi sub-modules which need the internal C API
  (pycore_modsupport.h): exceptions.c, float.c, vectorcall.c,
  watchers.c.
* Remove Include/cpython/modsupport.h header file.
  Include/modsupport.h no longer includes the removed header file.
* Fix mypy clinic.py
2023-10-17 14:30:31 +02:00
Nikita Sobolev 6a4528d70c
gh-110864: TypeVar constructor: Partially revert gh-110784, `constraints` cannot be `NULL` (#110922) 2023-10-16 15:01:04 +00:00
Nikita Sobolev bad7a35055
gh-110907: AC: Disallow using `*` with vararg (#110908) 2023-10-16 15:26:11 +01:00
Nikita Sobolev 989a253947
gh-110525: Delete `test_c_api` method from `set` object (#110688) 2023-10-13 10:06:25 +02:00
Jelle Zijlstra d2a536b170
gh-110782: Fix crash when TypeVar is constructed with keyword args (#110784) 2023-10-12 16:00:12 -07:00
Serhiy Storchaka eb50cd37ea
gh-110289: C API: Add PyUnicode_EqualToUTF8() and PyUnicode_EqualToUTF8AndSize() functions (GH-110297) 2023-10-11 16:41:58 +03:00
Brandt Bucher 13380da91e
GH-104584: Fix refleak when tracing through calls (GH-110593) 2023-10-10 08:29:48 +00:00
Eric Snow 7bd560ce8d
gh-76785: Add SendChannel.send_buffer() (#110246)
(This is still a test module.)
2023-10-09 07:39:51 -06:00
Xuehai Pan 3bbe3b7c82
gh-110222: Add support of PyStructSequence in copy.replace() (GH-110223) 2023-10-04 19:47:41 +03:00
Xuehai Pan 9561648f4a
gh-110235: Raise TypeError for duplicate/unknown fields in PyStructSequence constructor (GH-110258) 2023-10-04 19:44:17 +03:00
Victor Stinner fc2cb86d21
gh-107073: Make PyObject_VisitManagedDict() public (#108763)
Make PyObject_VisitManagedDict() and PyObject_ClearManagedDict()
functions public in Python 3.13 C API.

* Rename _PyObject_VisitManagedDict() to PyObject_VisitManagedDict().
* Rename _PyObject_ClearManagedDict() to PyObject_ClearManagedDict().
* Document these functions.
2023-10-02 19:24:08 +02:00
Victor Stinner 7513994c92
gh-110014: Include explicitly <unistd.h> header (#110155)
* Remove unused <locale.h> includes.
* Remove unused <fcntl.h> include in traceback.h.
* Remove redundant <assert.h> and <stddef.h> includes. They  are already
  included by "Python.h".
* Remove <object.h> include in faulthandler.c. Python.h already includes it.
* Add missing <stdbool.h> in pycore_pythread.h if HAVE_PTHREAD_STUBS
  is defined.
* Fix also warnings in pthread_stubs.h: don't redefine macros if they
  are already defined, like the __NEED_pthread_t macro.
2023-09-30 20:06:45 +00:00
Victor Stinner 8b626a47ba
gh-110079: Remove extern "C" { ...} in C code (#110080) 2023-09-29 10:56:49 +02:00
Serhiy Storchaka b8d1744e7b
gh-109611: Add convenient C API function _PyFile_Flush() (GH-109612) 2023-09-23 09:35:30 +03:00
Tian Gao d5611f2804
GH-107265: Add missing deoptimizations for ENTER_EXECUTOR's original opcode (GH-109420) 2023-09-22 14:13:31 -07:00
Sam Gross 2aceb21ae6
gh-109693: Remove pycore_atomic_funcs.h (#109694)
_PyUnicode_FromId() now uses pyatomic.h functions instead.
2023-09-21 22:57:20 +02:00
Heinz-Alexander Fuetterer ef6d475db3
Fix typos in docs and comments (#109619) 2023-09-20 16:58:23 +00:00
James Gerity def828995a
fixes gh-109559: Update `unicodedata` for Unicode 15.1.0 (GH-109560)
---------

Co-authored-by: Benjamin Peterson <benjamin@python.org>
2023-09-19 22:07:47 -07:00
Serhiy Storchaka add16f1a5e
gh-108511: Add C API functions which do not silently ignore errors (GH-109025)
Add the following functions:

* PyObject_HasAttrWithError()
* PyObject_HasAttrStringWithError()
* PyMapping_HasKeyWithError()
* PyMapping_HasKeyStringWithError()
2023-09-17 14:23:31 +03:00
Hood Chatham 6b179adb8c
gh-106213: Make Emscripten trampolines work with JSPI (GH-106219)
There is a WIP proposal to enable webassembly stack switching which have been
implemented in v8:

https://github.com/WebAssembly/js-promise-integration

It is not possible to switch stacks that contain JS frames so the Emscripten JS
trampolines that allow calling functions with the wrong number of arguments
don't work in this case. However, the js-promise-integration proposal requires
the [type reflection for Wasm/JS API](https://github.com/WebAssembly/js-types)
proposal, which allows us to actually count the number of arguments a function
expects.

For better compatibility with stack switching, this PR checks if type reflection
is available, and if so we use a switch block to decide the appropriate
signature. If type reflection is unavailable, we should use the current EMJS
trampoline.

We cache the function argument counts since when I didn't cache them performance
was negatively affected.

Co-authored-by: T. Wouters <thomas@python.org>
Co-authored-by: Brett Cannon <brett@python.org>
2023-09-15 15:04:21 -07:00
Nikita Sobolev 47af188593
Add missing `PyDoc_STR` calls (#109393)
In files:

* Modules/_ctypes/cfield.c
* Modules/_struct.c
* Objects/dictobject.c
* Objects/typevarobject.c
* Objects/unionobject.c
2023-09-15 15:10:48 +02:00
James Hilton-Balfe a806e920c4
Add missing PyDoc_STR to GenericAlias.__parameters__ (#108811) 2023-09-13 20:42:44 -07:00
Brandt Bucher 22e65eecaa
GH-105848: Replace KW_NAMES + CALL with LOAD_CONST + CALL_KW (GH-109300) 2023-09-13 10:25:45 -07:00
Serhiy Storchaka 92578919a6
gh-109174: Add support of SimpleNamespace in copy.replace() (GH-109175) 2023-09-10 08:09:25 +03:00
Tian Gao 057bc72490
gh-109052: Use the base opcode when comparing code objects (gh-109107) 2023-09-09 10:24:49 +09:00
Mark Shannon 15d4c9fabc
GH-108716: Turn off deep-freezing of code objects. (GH-108722) 2023-09-08 10:34:40 +01:00
Carl Meyer f2584eade3
gh-108732: include comprehension locals in frame.f_locals (#109026)
Co-authored-by: Radislav Chugunov <52372310+chgnrdv@users.noreply.github.com>
Co-authored-by: Jelle Zijlstra <jelle.zijlstra@gmail.com>
2023-09-07 08:56:43 -06:00
Mark Shannon 0858328ca2
GH-108614: Add `RESUME_CHECK` instruction (GH-108630) 2023-09-07 14:39:03 +01:00
Daniel Weiss e7d5433f94
gh-108915: Removes extra backslashes in str.split docstring (#109044) 2023-09-07 05:33:51 +00:00
Dong-hee Na 3bfa24e29f
gh-107265: Remove all ENTER_EXECUTOR when execute _Py_Instrument (gh-108539) 2023-09-07 09:53:54 +09:00
Serhiy Storchaka 6f3c138dfa
gh-108751: Add copy.replace() function (GH-108752)
It creates a modified copy of an object by calling the object's
__replace__() method.

It is a generalization of dataclasses.replace(), named tuple's _replace()
method and replace() methods in various classes, and supports all these
stdlib classes.
2023-09-06 23:55:42 +03:00
Serhiy Storchaka 3a08db8d13
gh-106307: Fix PyMapping_GetOptionalItemString() (GH-108797)
The resulting pointer was not set to NULL if the creation of a temporary
string object was failed.

The tests were also missed due to oversight.
2023-09-06 19:47:38 +00:00
Irit Katriel 39376cb93d
gh-108991: replace _PyFrame_GetState by two simpler functions (#108992) 2023-09-06 12:54:59 +01:00
Victor Stinner 03c4080c71
gh-108765: Python.h no longer includes <ctype.h> (#108831)
Remove <ctype.h> in C files which don't use it; only sre.c and
_decimal.c still use it.

Remove _PY_PORT_CTYPE_UTF8_ISSUE code from pyport.h:

* Code added by commit b5047fd019
  in 2004 for MacOSX and FreeBSD.
* Test removed by commit 52ddaefb6b
  in 2007, since Python str type now uses locale independent
  functions like Py_ISALPHA() and Py_TOLOWER() and the Unicode
  database.

Modules/_sre/sre.c replaces _PY_PORT_CTYPE_UTF8_ISSUE with new
functions: sre_isalnum(), sre_tolower(), sre_toupper().

Remove unused includes:

* _localemodule.c: remove <stdio.h>.
* getargs.c: remove <float.h>.
* dynload_win.c: remove <direct.h>, it no longer calls _getcwd()
  since commit fb1f68ed7c (in 2001).
2023-09-03 18:54:27 +02:00
Victor Stinner 578ebc5d5f
gh-108767: Replace ctype.h functions with pyctype.h functions (#108772)
Replace <ctype.h> locale dependent functions with Python "pyctype.h"
locale independent functions:

* Replace isalpha() with Py_ISALPHA().
* Replace isdigit() with Py_ISDIGIT().
* Replace isxdigit() with Py_ISXDIGIT().
* Replace tolower() with Py_TOLOWER().

Leave Modules/_sre/sre.c unchanged, it uses locale dependent
functions on purpose.

Include explicitly <ctype.h> in _decimal.c to get isascii().
2023-09-01 18:36:53 +02:00
Victor Stinner b936cf4fe0
gh-108634: PyInterpreterState_New() no longer calls Py_FatalError() (#108748)
pycore_create_interpreter() now returns a status, rather than
calling Py_FatalError().

* PyInterpreterState_New() now calls Py_ExitStatusException() instead
  of calling Py_FatalError() directly.
* Replace Py_FatalError() with PyStatus in init_interpreter() and
  _PyObject_InitState().
* _PyErr_SetFromPyStatus() now raises RuntimeError, instead of
  ValueError. It can now call PyErr_NoMemory(), raise MemoryError,
  if it detects _PyStatus_NO_MEMORY() error message.
2023-09-01 12:43:30 +02:00
Victor Stinner ad73674283
gh-107603: Argument Clinic: Only include pycore_gc.h if needed (#108726)
Argument Clinic now only includes pycore_gc.h if PyGC_Head is needed,
and only includes pycore_runtime.h if _Py_ID() is needed.

* Add 'condition' optional argument to Clinic.add_include().
* deprecate_keyword_use() includes pycore_runtime.h when using
  the _PyID() function.
* Fix rendering of includes: comments start at the column 35.
* Mark PC/clinic/_wmimodule.cpp.h and
  "Objects/stringlib/clinic/*.h.h" header files as generated in
  .gitattributes.

Effects:

* 42 header files generated by AC no longer include the internal C
  API, instead of 4 header files before. For example,
  Modules/clinic/_abc.c.h no longer includes the internal C API.
* Fix _testclinic_depr.c.h: it now always includes pycore_runtime.h
  to get _Py_ID().
2023-08-31 23:42:34 +02:00
Victor Stinner 13a00078b8
gh-108634: Py_TRACE_REFS uses a hash table (#108663)
Python built with "configure --with-trace-refs" (tracing references)
is now ABI compatible with Python release build and debug build.
Moreover, it now also supports the Limited API.

Change Py_TRACE_REFS build:

* Remove _PyObject_EXTRA_INIT macro.
* The PyObject structure no longer has two extra members (_ob_prev
  and _ob_next).
* Use a hash table (_Py_hashtable_t) to trace references (all
  objects): PyInterpreterState.object_state.refchain.
* Py_TRACE_REFS build is now ABI compatible with release build and
  debug build.
* Limited C API extensions can now be built with Py_TRACE_REFS:
  xxlimited, xxlimited_35, _testclinic_limited.
* No longer rename PyModule_Create2() and PyModule_FromDefAndSpec2()
  functions to PyModule_Create2TraceRefs() and
  PyModule_FromDefAndSpec2TraceRefs().
* _Py_PrintReferenceAddresses() is now called before
  finalize_interp_delete() which deletes the refchain hash table.
* test_tracemalloc find_trace() now also filters by size to ignore
  the memory allocated by _PyRefchain_Trace().

Test changes for Py_TRACE_REFS:

* Add test.support.Py_TRACE_REFS constant.
* Add test_sys.test_getobjects() to test sys.getobjects() function.
* test_exceptions skips test_recursion_normalizing_with_no_memory()
  and test_memory_error_in_PyErr_PrintEx() if Python is built with
  Py_TRACE_REFS.
* test_repl skips test_no_memory().
* test_capi skisp test_set_nomemory().
2023-08-31 18:33:34 +02:00
Victor Stinner 79823c103b
gh-106320: Remove private _PyErr_ChainExceptions() (#108713)
Remove _PyErr_ChainExceptions(), _PyErr_ChainExceptions1() and
_PyErr_SetFromPyStatus() functions from the public C API.

* Move the private _PyErr_ChainExceptions() and
  _PyErr_ChainExceptions1() function to the internal C API
  (pycore_pyerrors.h).
* Move the private _PyErr_SetFromPyStatus() to the internal C API
  (pycore_initconfig.h).
* No longer export the _PyErr_ChainExceptions() function.
* Move run_in_subinterp_with_config() from _testcapi to
  _testinternalcapi.
2023-08-31 13:53:19 +02:00
Serhiy Storchaka 9205dfeca5
gh-108635: Make parameters of some implementations of special methods positional-only (GH-108636) 2023-08-29 17:55:56 +03:00
Victor Stinner c9eefc77a7
gh-106320: Remove private _PyErr_SetKeyError() (#108607)
Move the private _PyErr_SetKeyError() function to the internal C API
(pycore_pyerrors.h).
2023-08-29 03:13:41 +00:00
Victor Stinner 301eb7e607
gh-106320: Remove _PyAnextAwaitable_Type from the public C API (#108597)
It's not needed to declare it in Include/iterobject.h: just use
"extern" where it's used (only in object.c).
2023-08-29 02:05:11 +00:00
Victor Stinner 39506ee565
gh-108240: Add pycore_capsule.h internal header file (#108596)
Move _PyCapsule_SetTraverse() definition to a new internal header
file: pycore_capsule.h.
2023-08-29 01:20:02 +00:00
Jelle Zijlstra 482fad7f01
gh-108295: Fix crashes with TypeVar weakrefs (#108517) 2023-08-26 17:13:33 -07:00
Serhiy Storchaka 2b15536fa9
gh-107913: Fix possible losses of OSError error codes (GH-107930)
Functions like PyErr_SetFromErrno() and SetFromWindowsErr() should be
called immediately after using the C API which sets errno or the Windows
error code.
2023-08-27 00:35:06 +03:00
Victor Stinner 8ba4714611
gh-106320: Remove private AC converter functions (#108505)
Move these private functions to the internal C API
(pycore_abstract.h):

* _Py_convert_optional_to_ssize_t()
* _PyNumber_Index()

Argument Clinic now emits #include "pycore_abstract.h" when these
functions are used.

The parser of the c-analyzer tool now uses a list of files which use
the limited C API, rather than a list of files using the internal C
API.
2023-08-26 04:05:17 +02:00
Victor Stinner b32d4cad15
gh-108444: Replace _PyLong_AsInt() with PyLong_AsInt() (#108459)
Change generated by the command:

sed -i -e 's!_PyLong_AsInt!PyLong_AsInt!g' \
    $(find -name "*.c" -o -name "*.h")
2023-08-25 01:01:30 +02:00
Victor Stinner 4e5a7284ee
gh-108444: Argument Clinic uses PyLong_AsInt() (#108458)
Argument Clinic now uses the new public PyLong_AsInt(), rather than
the old name _PyLong_AsInt().
2023-08-25 00:51:22 +02:00
Victor Stinner be436e08b8
gh-108444: Add PyLong_AsInt() public function (#108445)
* Rename _PyLong_AsInt() to PyLong_AsInt().
* Add documentation.
* Add test.
* For now, keep _PyLong_AsInt() as an alias to PyLong_AsInt().
2023-08-24 23:55:30 +02:00
Victor Stinner c494fb333b
gh-106320: Remove private _PyEval function (#108433)
Move private _PyEval functions to the internal C API
(pycore_ceval.h):

* _PyEval_GetBuiltin()
* _PyEval_GetBuiltinId()
* _PyEval_GetSwitchInterval()
* _PyEval_MakePendingCalls()
* _PyEval_SetProfile()
* _PyEval_SetSwitchInterval()
* _PyEval_SetTrace()

No longer export most of these functions.
2023-08-24 20:25:22 +02:00
Victor Stinner 773b803c02
gh-106320: Remove private _PyManagedBuffer_Type (#108431)
Remove private _PyManagedBuffer_Type variable. Move it to the
internal C API and no longer export it.
2023-08-24 19:07:54 +02:00
Victor Stinner 52c6a6e48a
gh-108308: Remove _PyDict_GetItemStringWithError() function (#108426)
Remove the internal _PyDict_GetItemStringWithError() function. It can
now be replaced with the new public PyDict_ContainsString() and
PyDict_GetItemStringRef() functions.

getargs.c now now uses a strong reference for current_arg.
find_keyword() returns a strong reference.
2023-08-24 17:34:22 +02:00
Victor Stinner a35d48d4bd
gh-108240: _PyCapsule_SetTraverse() rejects NULL callbacks (#108417) 2023-08-24 16:37:59 +02:00
Victor Stinner 6726626646
gh-108314: Add PyDict_ContainsString() function (#108323) 2023-08-24 15:59:12 +02:00
Guido van Rossum 3107b453bc
gh-108253: Fix reads of uninitialized memory in funcobject.c (#108383) 2023-08-23 22:36:19 +00:00
Victor Stinner 513c89d012
gh-108240: Add _PyCapsule_SetTraverse() internal function (#108339)
The _socket extension uses _PyCapsule_SetTraverse() to visit and clear
the socket type in the garbage collector. So the _socket.socket type
can be cleared in some corner cases when it wasn't possible before.
2023-08-24 00:19:11 +02:00
Victor Stinner 4890f65ecf
gh-108308: Use PyDict_GetItemRef() in moduleobject.c (#108381)
Replace PyDict_GetItemWithError() with PyDict_GetItemRef() which
returns a strong reference.

Cleanup also the function: move variable definition to their first
assignation, rename variable names to use name longer than 1
character.
2023-08-23 21:57:40 +00:00
Victor Stinner 4dc9f48930
gh-108308: Replace _PyDict_GetItemStringWithError() (#108372)
Replace _PyDict_GetItemStringWithError() calls with
PyDict_GetItemStringRef() which returns a strong reference to the
item.

Co-authored-by: Serhiy Storchaka <storchaka@gmail.com>
2023-08-23 22:59:00 +02:00
Irit Katriel 72119d16a5
gh-105481: remove regen-opcode. Generated _PyOpcode_Caches in regen-cases. (#108367) 2023-08-23 18:39:00 +01:00
Victor Stinner 615f6e946d
gh-106320: Remove _PyDict_GetItemStringWithError() function (#108313)
Remove private _PyDict_GetItemStringWithError() function of the
public C API: the new PyDict_GetItemStringRef() can be used instead.

* Move private _PyDict_GetItemStringWithError() to the internal C API.
* _testcapi get_code_extra_index() uses PyDict_GetItemStringRef().
  Avoid using private functions in _testcapi which tests the public C
  API.
2023-08-22 18:17:25 +00:00
Guido van Rossum b8f96b5eda
gh-108253: Fix bug in func version cache (#108296)
When a function object changed its version, a stale pointer might remain in the cache.
Zap these whenever `func_version` changes (even when set to 0).
2023-08-22 08:29:49 -07:00
Dong-hee Na e6db23f66d
gh-107265: Fix code_hash for ENTER_EXECUTOR case (#108188) 2023-08-21 14:22:18 -07:00
Dong-hee Na 4fdf3fda0f
gh-107265: Fix code_richcompare for ENTER_EXECUTOR case (gh-108165) 2023-08-21 05:50:09 +00:00
Guido van Rossum 61c7249759
gh-106581: Project through calls (#108067)
This finishes the work begun in gh-107760. When, while projecting a superblock, we encounter a call to a short, simple function, the superblock will now enter the function using `_PUSH_FRAME`, continue through it, and leave it using `_POP_FRAME`, and then continue through the original code. Multiple frame pushes and pops are even possible. It is also possible to stop appending to the superblock in the middle of a called function, when running out of space or encountering an unsupported bytecode.
2023-08-17 11:29:58 -07:00
Mark Shannon 006e44f950
GH-108035: Remove the `_PyCFrame` struct as it is no longer needed for performance. (GH-108036) 2023-08-17 11:16:03 +01:00
Irit Katriel 665a4391e1
gh-105481: generate op IDs from bytecode.c instead of hard coding them in opcode.py (#107971) 2023-08-16 22:25:18 +00:00
Martin DeMello 99b6ce56f8
Note that lnotab_notes.txt is only valid before 3.11 (#107961) 2023-08-15 20:27:35 +00:00
Joon Hwan 김준환 c3887b57a7
gh-107910: Remove not needing newline in error message (GH-107928) 2023-08-14 12:26:18 +03:00
Nikita Sobolev d93b4ac2ff
gh-101162: Forbid using issubclass() with GenericAlias as the 1st arg (GH-103369) 2023-08-11 22:12:11 +03:00
Mark Shannon 666b68e8f2
GH-106485: Create object's dict-values instead of creating __dict__, when we can. (GH-107843) 2023-08-11 20:05:56 +01:00
Serhiy Storchaka 3901c991e1
gh-84805: Autogenerate signature for METH_NOARGS and METH_O extension functions (GH-107794) 2023-08-11 18:08:38 +03:00
Marc Mueller 16dcce2176
gh-107810: Improve DeprecationWarning for metaclasses with custom tp_new (GH-107834)
Co-authored-by: Kirill Podoprigora <kirill.bast9@mail.ru>
2023-08-10 15:55:47 +00:00
Mark Shannon 1d976b2da2
GH-106485: Handle dict subclasses correctly when dematerializing `__dict__` (GH-107837) 2023-08-10 13:34:00 +01:00
Brandt Bucher 326f0ba1c5
GH-106485: Dematerialize instance dictionaries when possible (GH-106539) 2023-08-09 19:14:50 +00:00
Brandt Bucher a9caf9cf90
GH-105848: Simplify the arrangement of CALL's stack (GH-107788) 2023-08-09 18:19:39 +00:00
Serhiy Storchaka 0e6e32fb84
gh-86457: Fix signature for code.replace() (GH-23199)
Also add support of @text_signature in Argument Clinic.
2023-08-07 23:34:53 +03:00
Eric Snow 430632d6f7
gh-107630: Initialize Each Interpreter's refchain Properly (gh-107733)
This finishes fixing the crashes in Py_TRACE_REFS builds.  We missed this part in gh-107567.
2023-08-07 13:14:56 -06:00
Brandt Bucher 05a824f294
GH-84436: Skip refcounting for known immortals (GH-107605) 2023-08-04 16:24:50 -07:00
Eric Snow 58ef741867
gh-107080: Fix Py_TRACE_REFS Crashes Under Isolated Subinterpreters (gh-107567)
The linked list of objects was a global variable, which broke isolation between interpreters, causing crashes. To solve this, we've moved the linked list to each interpreter.
2023-08-03 19:51:08 +00:00
Mark Shannon 0d30a5a409
GH-100964: Break cycles involving exception state when returning from generator (GH-107563) 2023-08-02 18:44:20 +01:00
Anders Kaseorg 557b05c7a5
gh-106092: Fix use-after-free crash in frame_dealloc (#106875)
It was possible for the trashcan to delay the deallocation of a
PyFrameObject until after its corresponding _PyInterpreterFrame has
already been freed.  So frame_dealloc needs to avoid dereferencing the
f_frame pointer unless it first checks that the pointer still points
to the interpreter frame within the frame object.

Signed-off-by: Anders Kaseorg <andersk@mit.edu>
2023-08-01 10:32:18 +01:00
Illia Volochii fc130c47da
gh-102509: Start initializing `ob_digit` of `_PyLongValue` (GH-102510) 2023-07-28 14:39:54 +01:00
Eric Snow 8bdae1424b
gh-101524: Only Use Public C-API in the _xxsubinterpreters Module (gh-107359)
The _xxsubinterpreters module should not rely on internal API.  Some of the functions it uses were recently moved there however.  Here we move them back (and expose them properly).
2023-07-27 15:30:16 -06:00
Eric Snow b72947a8d2
gh-106931: Intern Statically Allocated Strings Globally (gh-107272)
We tried this before with a dict and for all interned strings.  That ran into problems due to interpreter isolation.  However, exclusively using a per-interpreter cache caused some inconsistency that can eliminate the benefit of interning.  Here we circle back to using a global cache, but only for statically allocated strings.  We also use a more-basic _Py_hashtable_t for that global cache instead of a dict.

Ideally we would only have the global cache, but the optional isolation of each interpreter's allocator means that a non-static string object must not outlive its interpreter.  Thus we would have to store a copy of each such interned string in the global cache, tied to the main interpreter.
2023-07-27 13:56:59 -06:00
Mark Shannon c6539b36c1
GH-106895: Raise a `ValueError` when attempting to disable events that cannot be disabled. (GH-107337) 2023-07-27 15:27:11 +01:00
Victor Stinner 1a3faba9f1
gh-106869: Use new PyMemberDef constant names (#106871)
* Remove '#include "structmember.h"'.
* If needed, add <stddef.h> to get offsetof() function.
* Update Parser/asdl_c.py to regenerate Python/Python-ast.c.
* Replace:

  * T_SHORT => Py_T_SHORT
  * T_INT => Py_T_INT
  * T_LONG => Py_T_LONG
  * T_FLOAT => Py_T_FLOAT
  * T_DOUBLE => Py_T_DOUBLE
  * T_STRING => Py_T_STRING
  * T_OBJECT => _Py_T_OBJECT
  * T_CHAR => Py_T_CHAR
  * T_BYTE => Py_T_BYTE
  * T_UBYTE => Py_T_UBYTE
  * T_USHORT => Py_T_USHORT
  * T_UINT => Py_T_UINT
  * T_ULONG => Py_T_ULONG
  * T_STRING_INPLACE => Py_T_STRING_INPLACE
  * T_BOOL => Py_T_BOOL
  * T_OBJECT_EX => Py_T_OBJECT_EX
  * T_LONGLONG => Py_T_LONGLONG
  * T_ULONGLONG => Py_T_ULONGLONG
  * T_PYSSIZET => Py_T_PYSSIZET
  * T_NONE => _Py_T_NONE
  * READONLY => Py_READONLY
  * PY_AUDIT_READ => Py_AUDIT_READ
  * READ_RESTRICTED => Py_AUDIT_READ
  * PY_WRITE_RESTRICTED => _Py_WRITE_RESTRICTED
  * RESTRICTED => (READ_RESTRICTED | _Py_WRITE_RESTRICTED)
2023-07-25 15:28:30 +02:00
Victor Stinner 8ebc9fc321
gh-102304: Rename _Py_IncRefTotal_DO_NOT_USE_THIS() (#107193)
* Rename _Py_IncRefTotal_DO_NOT_USE_THIS() to _Py_INCREF_IncRefTotal()
* Rename _Py_DecRefTotal_DO_NOT_USE_THIS() to _Py_DECREF_DecRefTotal()
* Remove temporary _Py_INC_REFTOTAL() and _Py_DEC_REFTOTAL() macros
2023-07-24 19:47:50 +00:00
Victor Stinner fd66baf34a
gh-106320: Remove private _PyDict C API (#107145)
Move private _PyDict functions to the internal C API (pycore_dict.h):

* _PyDict_Contains_KnownHash()
* _PyDict_DebugMallocStats()
* _PyDict_DelItemIf()
* _PyDict_GetItemWithError()
* _PyDict_HasOnlyStringKeys()
* _PyDict_MaybeUntrack()
* _PyDict_MergeEx()

No longer export these functions.
2023-07-24 14:02:03 +00:00
Victor Stinner 0d6dfd68d2
gh-106320: Remove private _PyObject C API (#107147)
Move private debug _PyObject functions to the internal C API
(pycore_object.h):

* _PyDebugAllocatorStats()
* _PyObject_CheckConsistency()
* _PyObject_DebugTypeStats()
* _PyObject_IsFreed()

No longer export most of these functions, except of
_PyObject_IsFreed().

Move test functions using _PyObject_IsFreed() from _testcapi to
_testinternalcapi. check_pyobject_is_freed() test no longer catch
_testcapi.error: the tested function cannot raise _testcapi.error.
2023-07-23 20:09:08 +00:00
Victor Stinner 889851ecc3
gh-106320: Remove _PyFunction_Vectorcall() API (#107071)
Move _PyFunction_Vectorcall() API to the internal C API.
No longer export the function.
2023-07-22 21:44:33 +00:00
Victor Stinner ee15844db8
gh-106320: Move _PyMethodWrapper_Type to internal C API (#107064) 2023-07-22 20:57:59 +00:00
Victor Stinner 22422e9d1a
gh-106320: Remove private _PyInterpreterID C API (#107053)
Move the private _PyInterpreterID C API to the internal C API: add a
new pycore_interp_id.h header file.

Remove Include/interpreteridobject.h and
Include/cpython/interpreteridobject.h header files.
2023-07-22 19:31:55 +00:00
Victor Stinner 6fbc717214
gh-106320: Remove _Py_SwappedOp from the C API (#107036)
Move _Py_SwappedOp to the internal C API (pycore_object.h).
2023-07-22 16:07:07 +00:00
Victor Stinner 5e4af2a3e9
gh-106320: Move private _PySet API to the internal API (#107041)
* Add pycore_setobject.h header file.
* Move the following API to the internal C API:

  * _PySet_Dummy
  * _PySet_NextEntry()
  * _PySet_Update()
2023-07-22 17:04:34 +02:00
James Hilton-Balfe cdeb1a6caa
gh-96663: Add a better error message for __dict__-less classes setattr (#103232) 2023-07-21 17:24:26 -07:00
Victor Stinner 41ca164551
gh-106004: Add PyDict_GetItemRef() function (#106005)
* Add PyDict_GetItemRef() and PyDict_GetItemStringRef() functions.
  Add these functions to the stable ABI version 3.13.
* Add unit tests on the PyDict C API in test_capi.
2023-07-21 23:10:51 +02:00
Eric Snow 0ba07b2108
gh-105699: Fix a Crasher Related to a Deprecated Global Variable (gh-106923)
There was a slight race in _Py_ClearFileSystemEncoding() (when called from _Py_SetFileSystemEncoding()), between freeing the value and setting the variable to NULL, which occasionally caused crashes when multiple isolated interpreters were used.  (Notably, I saw at least 10 different, seemingly unrelated spooky-action-at-a-distance, ways this crashed. Yay, free threading!)  We avoid the problem by only setting the global variables with the main interpreter (i.e. runtime init).
2023-07-21 08:34:09 -06:00
Eric Snow 87e7cb09e4
gh-105699: Fix an Interned Strings Crasher (gh-106930)
A static (process-global) str object must only have its "interned" state cleared when no longer interned in any interpreters.  They are the only ones that can be shared by interpreters so we don't have to worry about any other str objects.

We trigger clearing the state with the main interpreter, since no other interpreters may exist at that point and _PyUnicode_ClearInterned() is only called during interpreter finalization.

We do not address here the fact that a string will only be interned in the first interpreter that interns it.  In any subsequent interpreters str.state.interned is already set so _PyUnicode_InternInPlace() will skip it.  That needs to be addressed separately from fixing the crasher.
2023-07-21 08:32:42 -06:00
Serhiy Storchaka e1c295e3da
gh-106719: Fix __annotations__ getter and setter in the type and module types (GH-106720)
No longer suppress arbitrary errors. Simplify the code.
2023-07-18 08:56:58 +03:00
Dennis Sweeney ab86426a34
gh-105235: Prevent reading outside buffer during mmap.find() (#105252)
* Add a special case for s[-m:] == p in _PyBytes_Find

* Add tests for _PyBytes_Find

* Make sure that start <= end in mmap.find
2023-07-12 22:50:45 -04:00
Serhiy Storchaka be1b968dc1
gh-106521: Remove _PyObject_LookupAttr() function (GH-106642) 2023-07-12 08:57:10 +03:00
Serhiy Storchaka 4bf43710d1
gh-106307: C API: Add PyMapping_GetOptionalItem() function (GH-106308)
Also add PyMapping_GetOptionalItemString() function.
2023-07-11 23:04:12 +03:00
Serhiy Storchaka 579aa89e68
gh-106521: Add PyObject_GetOptionalAttr() function (GH-106522)
It is a new name of former _PyObject_LookupAttr().

Add also PyObject_GetOptionalAttrString().
2023-07-11 22:13:27 +03:00
Jelle Zijlstra 945d3cbf2e
gh-106403: Restore weakref support for TypeVar and friends (#106418) 2023-07-11 08:43:24 -07:00