Commit Graph

13137 Commits

Author SHA1 Message Date
Itamar Ostricher 3c137dc613
GH-91054: Add code object watchers API (GH-99859)
* Add API to allow extensions to set callback function on creation and destruction of PyCodeObject

Co-authored-by: Ye11ow-Flash <janshah@cs.stonybrook.edu>
2022-12-02 17:28:27 +00:00
Serhiy Storchaka f08e52ccb0
gh-99612: Fix PyUnicode_DecodeUTF8Stateful() for ASCII-only data (GH-99613)
Previously *consumed was not set in this case.
2022-12-01 14:54:51 +02:00
Victor Stinner 85dd6cb6df
gh-99845: Use size_t type in __sizeof__() methods (#99846)
The implementation of __sizeof__() methods using _PyObject_SIZE() now
use an unsigned type (size_t) to compute the size, rather than a signed
type (Py_ssize_t).

Cast explicitly signed (Py_ssize_t) values to unsigned type
(Py_ssize_t).
2022-11-30 17:22:52 +01:00
Dong-hee Na 8bb2303fd7
gh-99127: Allow some features of syslog to the main interpreter only (gh-99128) 2022-11-30 07:58:20 +09:00
Victor Stinner 74d5f61ebd
gh-99845: Clean up _PyObject_VAR_SIZE() usage (#99847)
* code_sizeof() now uses an unsigned type (size_t) to compute the result.
* Fix _PyObject_ComputedDictPointer(): cast _PyObject_VAR_SIZE() to
  Py_ssize_t, rather than long: it's a different type on 64-bit Windows.
* Clarify that _PyObject_VAR_SIZE() uses an unsigned type (size_t).
2022-11-29 12:15:21 +01:00
Kumar Aditya 4cfc1b8568
GH-91375: Port `_asyncio` static types to heap types and module state (#99122)
Co-authored-by: Erlend E. Aasland <erlend.aasland@protonmail.com>
2022-11-29 15:37:37 +05:30
Serhiy Storchaka deaa8dee48
gh-99593: Add tests for Unicode C API (part 1) (GH-99651)
Add tests for functions corresponding to the str class methods.
2022-11-29 09:59:56 +02:00
Zackery Spytz 53eef27133
bpo-31718: Fix io.IncrementalNewlineDecoder SystemErrors and segfaults (#18640)
Co-authored-by: Oren Milman <orenmn@gmail.com>
Co-authored-by: Kumar Aditya <59607654+kumaraditya303@users.noreply.github.com>
2022-11-28 16:16:40 +05:30
TheShermanTanker ec2b76aa8b
GH-95896: posixmodule.c: fix osdefs.h inclusion to not depend on compiler (#95897)
Co-authored-by: Steve Dower <steve.dower@python.org>
2022-11-26 15:01:42 +05:30
Victor Stinner 5556d3e02c
gh-98724: Fix warnings on Py_SETREF() usage (#99781)
Cast argument to the expected type.
2022-11-26 00:30:37 +01:00
Zackery Spytz 85c128e34d
bpo-40882: Fix a memory leak in SharedMemory on Windows (GH-20684)
In multiprocessing.shared_memory.SharedMemory(), the temporary view
returned by MapViewOfFile() should be unmapped when it is no longer
needed.
2022-11-25 17:39:48 +00:00
Zackery Spytz d386115039
bpo-38031: Fix a possible assertion failure in _io.FileIO() (#GH-5688) 2022-11-25 12:55:26 +00:00
Victor Stinner 3a803bcaac
Revert "gh-98724: Fix Py_CLEAR() macro side effects" (#99737)
Revert "gh-98724: Fix Py_CLEAR() macro side effects (#99100)"

This reverts commit c03e05c2e7.
2022-11-24 22:17:33 +01:00
colorfulappl 0da728387c
gh-64490: Fix bugs in argument clinic varargs processing (#32092) 2022-11-24 20:56:50 +01:00
colorfulappl 8dbe08eb7c
gh-99240: Fix double-free bug in Argument Clinic str_converter generated code (GH-99241)
Fix double-free bug mentioned at https://github.com/python/cpython/issues/99240,
by moving memory clean up out of "exit" label.

Automerge-Triggered-By: GH:erlend-aasland
2022-11-24 06:01:26 -08:00
colorfulappl 69f6cc77d0
gh-64490: Fix refcount error when arguments are packed to tuple in argument clinic (#99233) 2022-11-24 17:54:18 +05:30
Illia Volochii 9dc08361be
gh-96828: Add an `ssl.OP_ENABLE_KTLS` option (GH-96830)
Expose the constant when OpenSSL defines it.
2022-11-23 18:24:09 -08:00
Victor Stinner 81f7359f67
gh-99537: Use Py_SETREF(var, NULL) in C code (#99687)
Replace "Py_DECREF(var); var = NULL;" with "Py_SETREF(var, NULL);".
2022-11-23 14:57:50 +01:00
Victor Stinner f1a4a6a587
gh-99300: Replace Py_INCREF() with Py_NewRef() in _elementtree.c (#99696)
* Replace Py_INCREF() and Py_XINCREF() using a cast with Py_NewRef()
  and Py_XNewRef() in Modules/_elementtree.c.
* Make reference counting more explicit: don't steal implicitly a
  reference on PyList_SET_ITEM(), use Py_NewRef() instead.
* Replace PyModule_AddObject() with PyModule_AddObjectRef().
2022-11-22 22:17:06 +01:00
Charles Machalow 1b2de89bce
gh-99547: Add isjunction methods for checking if a path is a junction (GH-99548) 2022-11-22 17:19:34 +00:00
Victor Stinner 9a91182d4a
gh-99537: Use Py_CLEAR() function in C code (#99686)
Replace "Py_XDECREF(var); var = NULL;" with "Py_CLEAR(var);".

Don't replace "Py_DECREF(var); var = NULL;" with "Py_CLEAR(var);". It
would add an useless "if (var)" test in code path where var cannot be
NULL.
2022-11-22 15:22:55 +01:00
Victor Stinner 7e3f09cad9
gh-99537: Use Py_SETREF() function in C code (#99656)
Fix potential race condition in code patterns:

* Replace "Py_DECREF(var); var = new;" with "Py_SETREF(var, new);"
* Replace "Py_XDECREF(var); var = new;" with "Py_XSETREF(var, new);"
* Replace "Py_CLEAR(var); var = new;" with "Py_XSETREF(var, new);"

Other changes:

* Replace "old = var; var = new; Py_DECREF(var)"
  with "Py_SETREF(var, new);"
* Replace "old = var; var = new; Py_XDECREF(var)"
  with "Py_XSETREF(var, new);"
* And remove the "old" variable.
2022-11-22 14:22:22 +01:00
mpage 3db0a21f73
gh-91053: Add an optional callback that is invoked whenever a function is modified (#98175) 2022-11-22 13:06:44 +01:00
Victor Stinner 20d9749a0f
gh-99537: Use Py_SETREF() function in longobject C code (#99655)
Replace "Py_DECREF(var); var = new;" with "Py_SETREF(var, new);"
in longobject.c and _testcapi/long.c.
2022-11-22 13:04:19 +01:00
Ronald Oussoren 959ba45d75
GH-97001: Release GIL in termios extension (#99503)
Without releasing the GIL calls to termios APIs might block the entire interpreter.
2022-11-22 11:14:23 +01:00
Petr Viktorin 4d82f628c4
gh-47146: Soft-deprecate structmember.h, expose its contents via Python.h (GH-99014)
The ``structmember.h`` header is deprecated, though it continues to be available
and there are no plans to remove it. There are no deprecation warnings. Old code
can stay unchanged (unless the extra include and non-namespaced macros bother
you greatly). Specifically, no uses in CPython are updated -- that would just be
unnecessary churn.
The ``structmember.h`` header is deprecated, though it continues to be
available and there are no plans to remove it.

Its contents are now available just by including ``Python.h``,
with a ``Py`` prefix added if it was missing:

- `PyMemberDef`, `PyMember_GetOne` and`PyMember_SetOne`
- Type macros like `Py_T_INT`, `Py_T_DOUBLE`, etc.
  (previously ``T_INT``, ``T_DOUBLE``, etc.)
- The flags `Py_READONLY` (previously ``READONLY``) and
  `Py_AUDIT_READ` (previously all uppercase)

Several items are not exposed from ``Python.h``:

- `T_OBJECT` (use `Py_T_OBJECT_EX`)
- `T_NONE` (previously undocumented, and pretty quirky)
- The macro ``WRITE_RESTRICTED`` which does nothing.
- The macros ``RESTRICTED`` and ``READ_RESTRICTED``, equivalents of
  `Py_AUDIT_READ`.
- In some configurations, ``<stddef.h>`` is not included from ``Python.h``.
  It should be included manually when using ``offsetof()``.

The deprecated header continues to provide its original
contents under the original names.
Your old code can stay unchanged, unless the extra include and non-namespaced
macros bother you greatly.

There is discussion on the issue to rename `T_PYSSIZET` to `PY_T_SSIZE` or
similar. I chose not to do that -- users will probably copy/paste that with any
spelling, and not renaming it makes migration docs simpler.


Co-Authored-By: Alexander Belopolsky <abalkin@users.noreply.github.com>
Co-Authored-By: Matthias Braun <MatzeB@users.noreply.github.com>
2022-11-22 08:25:43 +01:00
colorfulappl c450c8c9ed
gh-96002: Add functional test for Argument Clinic (#96178)
Co-authored-by: Kumar Aditya <59607654+kumaraditya303@users.noreply.github.com>
Co-authored-by: Erlend E. Aasland <erlend.aasland@protonmail.com>
2022-11-21 15:08:45 +01:00
Ronald Oussoren cdde29dde9
gh-99337: Fix compile errors with gcc 12 on macOS (#99470)
Fix a number of compile errors with GCC-12 on macOS:

1. In pylifecycle.c the compile rejects _Pragma within a declaration
2. posixmodule.c was missing a number of ..._RUNTIME macros for non-clang on macOS
3. _ctypes assumed that __builtin_available is always present on macOS
2022-11-21 10:50:20 +01:00
Nikita Sobolev a3360facba
gh-99284: [ctypes] remove `_use_broken_old_ctypes_structure_semantics_` (GH-99285)
It was untested and undocumented. No code has been found in the wild that ever used it.
2022-11-18 22:25:32 -08:00
Erlend E. Aasland 12b5a3c5f5
gh-93649: Split float/long tests from _testcapimodule.c (GH-99549)
Automerge-Triggered-By: GH:erlend-aasland
2022-11-17 00:56:56 -08:00
Erlend E. Aasland 51d10354f4
gh-93649: Split watcher API tests from _testcapimodule.c (#99532) 2022-11-16 20:13:32 +01:00
Eric Snow 9db1e17c80
gh-81057: Move the global Dict-Related Versions to _PyRuntimeState (gh-99497)
We also move the global func version.

https://github.com/python/cpython/issues/81057
2022-11-16 10:37:29 -07:00
Steve Dower 19c1462e8d
gh-99377: Add audit events for thread creation and clear (GH-99378) 2022-11-16 17:15:52 +00:00
Eric Snow 5f55067e23
gh-81057: Move More Globals in Core Code to _PyRuntimeState (gh-99516)
https://github.com/python/cpython/issues/81057
2022-11-16 09:37:14 -07:00
Erlend E. Aasland 00437ad304
gh-93649: Split memory and docstring tests from _testcapimodule.c (#99517) 2022-11-16 14:09:10 +01:00
Christian Heimes 33f42c269f
gh-96269: static and shared ext need different deps (#96316) 2022-11-16 14:03:35 +01:00
Victor Stinner 3ed8803ef5
gh-99300: Replace Py_INCREF() with Py_NewRef() (#99513)
Replace Py_INCREF() and Py_XINCREF() using a cast with Py_NewRef()
and Py_XNewRef().
2022-11-16 10:39:47 +01:00
Eric Snow 3c57971a2d
gh-81057: Move Globals in Core Code to _PyRuntimeState (gh-99496)
This is the first of several changes to consolidate non-object globals in core code.

https://github.com/python/cpython/issues/81057
2022-11-15 09:45:11 -07:00
Erlend E. Aasland ee821dcd39
gh-93649: Split pytime and datetime tests from _testcapimodule.c (#99494) 2022-11-15 08:17:52 +01:00
Erlend E. Aasland 3d9431983a
gh-93649: Split getargs tests from _testcapimodule.c (#99346) 2022-11-14 22:23:41 +01:00
Victor Stinner 65dd745f1a
gh-99300: Use Py_NewRef() in Modules/ directory (#99473)
Replace Py_INCREF() and Py_XINCREF() with Py_NewRef() and
Py_XNewRef() in test C files of the Modules/ directory.
2022-11-14 16:21:40 +01:00
Victor Stinner 3e2f7135e6
gh-99300: Use Py_NewRef() in Modules/ directory (#99469)
Replace Py_INCREF() and Py_XINCREF() with Py_NewRef() and
Py_XNewRef() in test C files of the Modules/ directory.
2022-11-14 16:21:23 +01:00
Kai Zhang e3d4fed074
gh-99370: Calculate zip path from prefix when in a venv (GH-99371)
Before python3.11, when in a venv the zip path is calculated
from prefix on POSIX platforms. In python3.11 the behavior is
accidentally changed to calculating from default prefix. This
change will break venv created from a non-installed python
with a stdlib zip file. This commit restores the behavior back
to before python3.11.
2022-11-14 15:05:14 +00:00
Irit Katriel a3ac9232f8
gh-87092: expose the compiler's codegen to python for unit tests (GH-99111) 2022-11-14 13:56:40 +00:00
Serhiy Storchaka 3fe7d7c020
gh-99426: Use PyUnicode_FromFormat() and PyErr_Format() instead of sprintf (GH-99427) 2022-11-14 15:25:34 +02:00
Victor Stinner c340cbb7f7
gh-99300: Use Py_NewRef() in Modules/ directory (#99468)
Replace Py_INCREF() and Py_XINCREF() with Py_NewRef() and
Py_XNewRef() in test C files of the Modules/ directory.
2022-11-14 13:44:56 +01:00
Victor Stinner f15a0fcb1d
gh-99300: Use Py_NewRef() in Modules/_datetimemodule.c (#99465)
Replace Py_INCREF() and Py_XINCREF() with Py_NewRef() and
Py_XNewRef() in Modules/_datetimemodule.c and Modules/_zoneinfo.c
2022-11-14 13:09:12 +01:00
Victor Stinner 7e4dec02ac
gh-99300: Use Py_NewRef() in Modules/ directory (#99467)
Replace Py_INCREF() and Py_XINCREF() with Py_NewRef() and
Py_XNewRef() in test C files of the Modules/ directory.
2022-11-14 13:08:43 +01:00
Victor Stinner 3817607b8a
gh-99300: Use Py_NewRef() in Modules/ directory (#99466)
Replace Py_INCREF() and Py_XINCREF() with Py_NewRef() and
Py_XNewRef() in test C files of the Modules/ directory.
2022-11-14 13:08:15 +01:00
Victor Stinner 1455c516fc
gh-99300: Use Py_NewRef() in Modules/_asynciomodule.c (#99441)
Replace Py_INCREF() and Py_XINCREF() with Py_NewRef() and
Py_XNewRef() in Modules/_asynciomodule.c.
2022-11-14 00:31:21 +01:00
ram vikram singh 88385b8564
gh-98930: improve the docstring of signal.strsignal (#99290)
Improves the docstring on signal.strsignal to make it explain when it returns a message, None, or when it raises ValueError.

Closes #98930

Co-authored-by: Gregory P. Smith <greg@krypto.org>
2022-11-13 11:41:23 -08:00
Nikita Sobolev 343eb0f94b
gh-99275: Fix `SystemError` in `ctypes` during `__initsubclass__` (#99283) 2022-11-13 11:22:45 -08:00
Victor Stinner 50b0415127
gh-99300: Use Py_NewRef() in Modules/_ctypes/ (#99436)
Replace Py_INCREF() and Py_XINCREF() with Py_NewRef() and
Py_XNewRef() in C files of the Modules/_ctypes/ directory.
2022-11-13 16:04:54 +01:00
Victor Stinner bbba3f3f43
gh-99300: Use Py_NewRef() in Modules/ directory (#99440)
Replace Py_INCREF() and Py_XINCREF() with Py_NewRef() and
Py_XNewRef() in test C files of the Modules/ directory.
2022-11-13 16:04:33 +01:00
Victor Stinner 0bedc28d14
gh-99300: Use Py_NewRef() in Modules/itertoolsmodule.c (#99439)
Replace Py_INCREF() and Py_XINCREF() with Py_NewRef() and
Py_XNewRef() in Modules/itertoolsmodule.c.
2022-11-13 16:04:22 +01:00
Victor Stinner abd004e355
gh-99300: Use Py_NewRef() in Modules/_elementtree.c (#99438)
Replace Py_INCREF() and Py_XINCREF() with Py_NewRef() and
Py_XNewRef() in Modules/_elementtree.c.
2022-11-13 16:04:11 +01:00
Erlend E. Aasland c95f554a40
gh-83638: Add sqlite3.Connection.autocommit for PEP 249 compliant behaviour (#93823)
Introduce the autocommit attribute to Connection and the autocommit
parameter to connect() for PEP 249-compliant transaction handling.

Co-authored-by: Alex Waygood <Alex.Waygood@Gmail.com>
Co-authored-by: C.A.M. Gerlach <CAM.Gerlach@Gerlach.CAM>
Co-authored-by: Géry Ogam <gery.ogam@gmail.com>
2022-11-12 23:44:41 +01:00
Kumar Aditya 504e12272b
GH-81057: Remove static state from arraymodule (#99409) 2022-11-12 10:11:32 +01:00
Vincent Fazio c41b13d39c
gh-99204: Calculate base_executable by alternate names in POSIX venvs (GH-99206)
Check to see if `base_executable` exists. If it does not, attempt
to use known alternative names of the python binary to find an
executable in the path specified by `home`.

If no alternative is found, previous behavior is preserved.

Signed-off-by: Vincent Fazio <vfazio@gmail.com>

Signed-off-by: Vincent Fazio <vfazio@gmail.com>
2022-11-10 16:26:42 +00:00
Eric Snow 73679b13ca
gh-90110: Update the C-analyzer Tool (gh-99307) 2022-11-10 09:03:57 -07:00
Victor Stinner c03e05c2e7
gh-98724: Fix Py_CLEAR() macro side effects (#99100)
The Py_CLEAR(), Py_SETREF() and Py_XSETREF() macros now only evaluate
their argument once. If an argument has side effects, these side
effects are no longer duplicated.

Add test_py_clear() and test_py_setref() unit tests to _testcapi.
2022-11-09 14:06:36 +01:00
Eric Snow 52f91c642b
gh-90868: Adjust the Generated Objects (gh-99223)
We do the following:

* move the generated _PyUnicode_InitStaticStrings() to its own file
* move the generated _PyStaticObjects_CheckRefcnt() to its own file
* include pycore_global_objects.h in extension modules instead of pycore_runtime_init.h

These changes help us avoid including things that aren't needed.

https://github.com/python/cpython/issues/90868
2022-11-08 10:03:03 -07:00
Eric Snow d45cc80452
gh-98627: Add the _testsinglephase Module (gh-99039)
This makes it more clear that a given test is definitely testing against a single-phase init (legacy) extension module.  The new module is a companion to _testmultiphase.

https://github.com/python/cpython/issues/98627
2022-11-08 09:58:11 -07:00
Victor Stinner 4d5fcca273
gh-91248: Add PyFrame_GetVar() function (#95712)
Add PyFrame_GetVar() and PyFrame_GetVarString() functions to get a
frame variable by its name.

Move PyFrameObject C API tests from test_capi to test_frame.
2022-11-08 17:40:27 +01:00
Kumar Aditya a751bf565c
GH-90699: fix refleak in `_testcapimodule.c` (gh-99236) 2022-11-08 15:50:07 +09:00
Kamil Turek b9dedfe61d
gh-92119: ctypes: Print exception class name instead of its representation (#98302) 2022-11-07 20:53:59 -08:00
Kumar Aditya be0d5008b3
GH-90699: Remove remaining `_Py_IDENTIFIER` stdlib usage (GH-99067) 2022-11-07 12:06:23 -08:00
Noam Cohen 80c08d1cd6
gh-95389: expose popular ETHERTYPE_* constants in the socket module (#95390)
Co-authored-by: Erlend E. Aasland <erlend.aasland@protonmail.com>
Co-authored-by: CAM Gerlach <CAM.Gerlach@Gerlach.CAM>
2022-11-07 15:27:37 +01:00
Harshil cfec5b18bf
remove new line in pickle exception message (GH-31782) 2022-11-07 07:43:39 +00:00
Dong-hee Na f626b7b504
gh-96055: Update faulthandler to emit proper unexpect signal number (gh-99162) 2022-11-07 10:19:25 +09:00
jonasdlindner ede6cb2615
Correct some typos in comments (GH-98194)
Automerge-Triggered-By: GH:AlexWaygood
2022-11-06 08:54:44 -08:00
Shantanu d3b82b4463
gh-83004: Clean up refleak in _pickle initialisation (#98841) 2022-11-06 06:05:13 -08:00
Shantanu 31f2f6568d
gh-83004: Clean up refleak in _zoneinfo initialisation (#98842) 2022-11-06 06:04:36 -08:00
Nikita Sobolev 317acb8038
gh-94808: add tests covering `PyFunction_GetKwDefaults` and `PyFunction_SetKwDefaults` (GH-98809) 2022-11-05 17:08:47 +00:00
Nikita Sobolev c5c40773cb
gh-94808: add tests covering `PySequence_{Set,Del}Slice` (GH-99123) 2022-11-05 17:08:00 +00:00
Nikita Sobolev b5f711185b
gh-94808: add tests covering `PyEval_GetFuncDesc` function (GH-98300) 2022-11-05 13:07:59 +00:00
Gregory P. Smith bee1070289
gh-73691: Increase size limits in _xxtestfuzz (#99070)
Now that our int<->str conversions are size limited and we have the
_pylong module handling larger integers, we don't need to limit
everything just to avoid wasting time in the quadratic time DoS-like
case while fuzzing.

We can tweak these further after seeing how this goes.
2022-11-03 14:41:20 -07:00
Kumar Aditya 0ee59a9ca3
GH-90699: Remove `_Py_IDENTIFIER` usage from `_ctypes` (GH-99054) 2022-11-03 13:20:10 -07:00
Shantanu 1208037246
gh-83004: Clean up refleak in _io initialisation (#98840) 2022-11-03 07:29:11 -07:00
Serhiy Storchaka e9ac890c02
gh-98740: Fix validation of conditional expressions in RE (GH-98764)
In very rare circumstances the JUMP opcode could be confused with the
argument of the opcode in the "then" part which doesn't end with the
JUMP opcode. This led to incorrect detection of the final JUMP opcode
and incorrect calculation of the size of the subexpression.

NOTE: Changed return value of functions _validate_inner() and
_validate_charset() in Modules/_sre/sre.c.  Now they return 0 on success,
-1 on failure, and 1 if the last op is JUMP (which usually is a failure).
Previously they returned 1 on success and 0 on failure.
2022-11-03 09:23:46 +02:00
Kumar Aditya 908e81f6c8
GH-90699: Remove `_Py_IDENTIFIER` usage from `_elementtree` module (GH-99012) 2022-11-02 13:29:41 -07:00
Steve Dower 3d889dc0a0
gh-98790: When DLLs directory is missing on Windows, assume executable_dir contains PYD files instead (GH-98936) 2022-11-02 18:38:40 +00:00
Kumar Aditya 18fc232e07
GH-90699: Remove `_Py_IDENTIFIER` usage from `_asyncio` module (#99010) 2022-11-02 10:16:06 -07:00
Kumar Aditya 780757ac58
GH-90699: Remove `_Py_IDENTIFIER` usage from `_json` module (GH-98956) 2022-11-02 09:03:38 -07:00
Kumar Aditya 0a806f2684
GH-90699: Remove `_Py_IDENTIFIER` usage from `_curses` module (GH-98957) 2022-11-02 08:41:11 -07:00
Jelle Zijlstra 5cf317ade1
gh-98658: Add __class_getitem__ to array.array (#98661)
Closes #98658
2022-10-31 21:10:18 -07:00
Kumar Aditya ab57505070
GH-98897: fix memory leak if `math.dist` raises exception (GH-98898) 2022-10-31 21:18:32 -05:00
Eric Snow 4702552885
gh-98610: Adjust the Optional Restrictions on Subinterpreters (GH-98618)
Previously, the optional restrictions on subinterpreters were: disallow fork, subprocess, and threads.  By default, we were disallowing all three for "isolated" interpreters.  We always allowed all three for the main interpreter and those created through the legacy `Py_NewInterpreter()` API.

Those settings were a bit conservative, so here we've adjusted the optional restrictions to: fork, exec, threads, and daemon threads.  The default for "isolated" interpreters disables fork, exec, and daemon threads.  Regular threads are allowed by default.  We continue always allowing everything For the main interpreter and the legacy API.

In the code, we add `_PyInterpreterConfig.allow_exec` and  `_PyInterpreterConfig.allow_daemon_threads`.  We also add `Py_RTFLAGS_DAEMON_THREADS` and `Py_RTFLAGS_EXEC`.
2022-10-31 12:35:54 -07:00
Charlie Zhao 3ac8c0ab6e
gh-98793: Fix typecheck in `overlapped.c` (#98835)
Co-authored-by: Kumar Aditya <59607654+kumaraditya303@users.noreply.github.com>
2022-10-29 21:34:46 -07:00
David Buchanan 1561e708c7
Fix comment typos in `_operator.c` (#98853) 2022-10-29 19:25:09 +01:00
Shaun Walbridge 3e07f827b3
gh-98739: Update libexpat from 2.4.9 to 2.5.0 (#98742)
* Update libexpat from 2.4.9 to 2.5.0 to address CVE-2022-43680.

Co-authored-by: Shaun Walbridge <shaun.walbridge@gmail.com>
2022-10-27 13:45:12 -07:00
Wenzel Jakob e60892f9db
gh-98586: Add vector call APIs to the Limited API (GH-98587)
Expose the facilities for making vector calls through Python's limited API.
2022-10-27 11:45:42 +02:00
Nikita Sobolev 7b24333fff
gh-94808: cover `PyFunction_GetDefaults` and `PyFunction_SetDefaults` (#98449) 2022-10-26 17:47:29 -07:00
Benjamin Peterson 29b391b137
Fix readline.c compiler warning. (GH-98738)
```
Modules/readline.c:1260:37: warning: assigning to 'char *' from 'const char *' discards qualifiers [-Wincompatible-pointer-types-discards-qualifiers]
    completer_word_break_characters =
                                    ^
```
2022-10-26 15:53:25 -07:00
Eric Snow f32369480d
gh-98608: Change _Py_NewInterpreter() to _Py_NewInterpreterFromConfig() (gh-98609)
(see https://github.com/python/cpython/issues/98608)

This change does the following:

1. change the argument to a new `_PyInterpreterConfig` struct
2. rename the function to `_Py_NewInterpreterFromConfig()`, inspired by `Py_InitializeFromConfig()` (takes a `_PyInterpreterConfig`  instead of `isolated_subinterpreter`)
3. split up the boolean `isolated_subinterpreter` into the corresponding multiple granular settings
   * allow_fork
   * allow_subprocess
   * allow_threads
4. add `PyInterpreterState.feature_flags` to store those settings
5. add a function for checking if a feature is enabled on an opaque `PyInterpreterState *`
6. drop `PyConfig._isolated_interpreter`

The existing default (see `Py_NewInterpeter()` and `Py_Initialize*()`) allows fork, subprocess, and threads and the optional "isolated" interpreter (see the `_xxsubinterpreters` module) disables all three.  None of that changes here; the defaults are preserved.

Note that the given `_PyInterpreterConfig` will not be used outside `_Py_NewInterpreterFromConfig()`, nor preserved.  This contrasts with how `PyConfig` is currently preserved, used, and even modified outside `Py_InitializeFromConfig()`.  I'd rather just avoid that mess from the start for `_PyInterpreterConfig`.  We can preserve it later if we find an actual need.

This change allows us to follow up with a number of improvements (e.g. stop disallowing subprocess and support disallowing exec instead).

(Note that this PR adds "private" symbols.  We'll probably make them public, and add docs, in a separate change.)
2022-10-26 11:16:30 -06:00
Nikita Sobolev 5d30544485
gh-94808: cover `PyMapping_HasKeyString` and `PyMapping_HasKey` (#98486) 2022-10-25 21:33:32 -07:00
Serhiy Storchaka 75a6fadf36
gh-91524: Speed up the regular expression substitution (#91525)
Functions re.sub() and re.subn() and corresponding re.Pattern methods
are now 2-3 times faster for replacement strings containing group references.

Closes #91524

Primarily authored by serhiy-storchaka Serhiy Storchaka
Minor-cleanups-by: Gregory P. Smith [Google] <greg@krypto.org>
2022-10-23 15:57:30 -07:00
Raymond Hettinger 5871e19942
GH-98363: Slicing isn't necessary. A size reduction will suffice. (GH-98538) 2022-10-22 07:21:06 -05:00
Raymond Hettinger 8d574234d4
GH-98363: Use better accessor macro (#98535) 2022-10-21 18:44:49 -05:00
Furkan Onder 4c1145bb37
bpo-2716: add license for audioop module (#19972) 2022-10-21 15:43:20 -07:00
Raymond Hettinger a5ff80c8bc
GH-98363: Fix exception handling in batched() (GH-98523) 2022-10-21 12:31:52 -05:00
Carl Meyer 82ccbf69a8
gh-91051: allow setting a callback hook on PyType_Modified (GH-97875) 2022-10-21 14:41:51 +01:00
Noam Cohen a371a7e03e
gh-95023: Added os.setns and os.unshare functions (#95046)
Added os.setns and os.unshare to easily switch between namespaces
on Linux.

Co-authored-by: Christian Heimes <christian@python.org>
Co-authored-by: CAM Gerlach <CAM.Gerlach@Gerlach.CAM>
Co-authored-by: Victor Stinner <vstinner@python.org>
2022-10-20 11:08:54 +02:00
Raymond Hettinger c1e02d4e4e
GH-98363: Presize the list for batched() (GH-98419) 2022-10-20 02:28:17 -05:00
Victor Stinner db03c8066a
gh-98393: os module reject bytes-like, only accept bytes (#98394)
The os module and the PyUnicode_FSDecoder() function no longer accept
bytes-like paths, like bytearray and memoryview types: only the exact
bytes type is accepted for bytes strings.
2022-10-18 17:52:31 +02:00
Raymond Hettinger 9608bef84a
Fix typos in comments (GH-98375) 2022-10-17 21:01:28 -05:00
Raymond Hettinger de3ece769a
GH-98363: Add itertools.batched() (GH-98364) 2022-10-17 18:53:45 -05:00
Benjamin Peterson 0f156c1c56
Remove unused arrange_output_buffer function from zlibmodule.c. (GH-98358) 2022-10-17 09:38:34 -07:00
Victor Stinner 1863302d61
gh-97669: Create Tools/build/ directory (#97963)
Create Tools/build/ directory. Move the following scripts from
Tools/scripts/ to Tools/build/:

* check_extension_modules.py
* deepfreeze.py
* freeze_modules.py
* generate_global_objects.py
* generate_levenshtein_examples.py
* generate_opcode_h.py
* generate_re_casefix.py
* generate_sre_constants.py
* generate_stdlib_module_names.py
* generate_token.py
* parse_html5_entities.py
* smelly.py
* stable_abi.py
* umarshal.py
* update_file.py
* verify_ensurepip_wheels.py

Update references to these scripts.
2022-10-17 12:01:00 +02:00
Ruben Vorderman eae7dad402
gh-95534: Improve gzip reading speed by 10% (#97664)
Change summary:
+ There is now a `gzip.READ_BUFFER_SIZE` constant that is 128KB. Other programs that read in 128KB chunks: pigz and cat. So this seems best practice among good programs. Also it is faster than 8 kb chunks.
+ a zlib._ZlibDecompressor was added. This is the _bz2.BZ2Decompressor ported to zlib. Since the zlib.Decompress object is better for in-memory decompression, the _ZlibDecompressor is hidden. It only makes sense in file decompression, and that is already implemented now in the gzip library. No need to bother the users with this.
+ The ZlibDecompressor uses the older Cpython arrange_output_buffer functions, as those are faster and more appropriate for the use case. 
+ GzipFile.read has been optimized. There is no longer a `unconsumed_tail` member to write back to padded file. This is instead handled by the ZlibDecompressor itself, which has an internal buffer. `_add_read_data` has been inlined, as it was just two calls.

EDIT: While I am adding improvements anyway, I figured I could add another one-liner optimization now to the python -m gzip application. That read chunks in io.DEFAULT_BUFFER_SIZE previously, but has been updated now to use READ_BUFFER_SIZE chunks.
2022-10-16 19:10:58 -07:00
Nikita Sobolev 7b48d02933
gh-94808: Cover `PyFunction_GetCode`, `PyFunction_GetGlobals`, `PyFunction_GetModule` (#98158) 2022-10-15 21:18:59 -07:00
Nikita Sobolev f01b56c7bd
gh-94808: Cover `PyEval_GetFuncName` (#98246) 2022-10-15 08:20:39 -07:00
Victor Stinner d4b9166385
gh-98178: syslog() is not thread-safe on macOS (#98213)
On macOS, fix a crash in syslog.syslog() in multi-threaded
applications. On macOS, the libc syslog() function is not
thread-safe, so syslog.syslog() no longer releases the GIL to call
it.
2022-10-13 13:34:55 +02:00
Victor Stinner 342b1151ae
signalmodule.c uses _PyErr_WriteUnraisableMsg() (#98217)
Signal wakeup fd errors are now logged with
_PyErr_WriteUnraisableMsg(), rather than PySys_WriteStderr() and
PyErr_WriteUnraisable(), to pass the error message to
sys.unraisablehook. By default, it's still written into stderr (unless
sys.unraisablehook is overriden).
2022-10-12 17:26:58 +02:00
Shantanu f871e9a7bb
gh-44098: Release the GIL during mmap on Unix (GH-98146)
This seems pretty straightforward. The issue mentions other calls in mmapmodule that we could release the GIL on, but those are in methods where we'd need to be careful to ensure that something sensible happens if those are called concurrently. In prior art, note that #12073 released the GIL for munmap.  In a toy benchmark, I see the speedup you'd expect from doing this.

Automerge-Triggered-By: GH:gvanrossum
2022-10-10 15:14:31 -07:00
Matthias Görgens 553d3c1017
gh-96821: Fix undefined behaviour in `audioop.c` (#96923)
* gh-96821: Fix undefined behaviour in `audioop.c`

Left-shifting negative numbers is undefined behaviour.

Fortunately, multiplication works just as well, is defined behaviour,
and gets compiled to the same machine code as before by optimizing
compilers.

Co-authored-by: blurb-it[bot] <43283697+blurb-it[bot]@users.noreply.github.com>
2022-10-10 17:12:29 +01:00
Michael Droettboom dfcdee4a18
gh-94808: Add coverage for bytesarray_setitem (#95802) 2022-10-10 08:28:41 -07:00
Noam Cohen 5405537813
gh-95011: Migrate syslog module to Argument Clinic (GH-95012) 2022-10-08 21:31:57 +03:00
Pablo Galindo Salgado 83eb827247
gh-97922: Run the GC only on eval breaker (#97920) 2022-10-08 07:57:09 -07:00
Carl Meyer e82d977eb0
gh-91052: Add PyDict_Unwatch for unwatching a dictionary (#98055) 2022-10-07 17:37:46 -07:00
Hagai Helman Tov 1523c9e9d4
gh-96288: Add a sentence to `os.mkdir`'s docstring. (#96271) 2022-10-07 15:06:16 -07:00
Stanley 676d8ef380
gh-65496: Correct wording on csv's skipinitialspace argument (#96170) 2022-10-07 11:15:34 -07:00
Nikita Sobolev 24a6645894
gh-97955: Migrate `zoneinfo` to Argument Clinic (#97958) 2022-10-07 11:06:23 -07:00
Nikita Sobolev 83cbe84dc2
gh-64373: Convert `_functools` to Argument Clinic (#96640) 2022-10-07 10:36:40 -07:00
Serhiy Storchaka cae7d1d7a7
Add more syslog tests (GH-97953) 2022-10-07 20:17:08 +03:00
Carl Meyer be4099e55d
Fix memory leaks in test_capi (#98017) 2022-10-07 08:17:41 -07:00
Guido van Rossum 09de8d7aaf
GH-90985: Revert "Deprecate passing a message into cancel()" (#97999)
Reason: we were too hasty in deprecating this.
We shouldn't deprecate it before we have a replacement.
2022-10-06 17:30:27 -07:00
Carl Meyer a4b7794887
GH-91052: Add C API for watching dictionaries (GH-31787) 2022-10-07 01:08:00 +01:00
Dong-hee Na b9d2e81716
fixes gh-96078: os.sched_yield release the GIL while calling sched_yield(2). (gh-97965) 2022-10-06 15:57:37 -07:00
Erik Welch 1c4728cc29
gh-94590: add signatures to operator itemgetter, attrgetter, methodcaller (#94591)
These were intentionally skipped when operator was updated to use the argument clinic:
https://github.com/python/cpython/issues/64385#issuecomment-1093641466

However, by not using the argument clinic, they missed out on getting signatures.
This is a narrow PR to update the docstrings so that `__text_signature__` can be
extracted from them.  Updating to use the argument clinic is beyond scope.

`methodcaller` uses `*args, **kwargs` to match variadic names used elsewhere,
including in `operator.call`.
2022-10-06 13:35:53 -07:00
Ned Deily 6d0a0191a4
gh-97897: Prevent os.mkfifo and os.mknod segfaults with macOS 13 SDK (GH-97944)
The macOS 13 SDK includes support for the `mkfifoat` and `mknodat` system calls.
 Using the `dir_fd` option with either `os.mkfifo` or `os.mknod` could result in a
 segfault if cpython is built with the macOS 13 SDK but run on an earlier
 version of macOS. Prevent this by adding runtime support for detection of
 these system calls ("weaklinking") as is done for other newer syscalls on
 macOS.
2022-10-06 09:11:47 -07:00
Nikita Sobolev e63d7dae90
gh-94808: Cover `PyUnicode_Count` in CAPI (#96929) 2022-10-06 17:20:22 +02:00
Serhiy Storchaka f8cbd79d32
gh-97758: Fix a crash in getpath_joinpath() called without arguments (GH-97759) 2022-10-05 12:51:58 +03:00
Guido van Rossum 8079bef56f
GH-96704: Add {Task,Handle}.get_context(), use it in call_exception_handler() (#96756)
Co-authored-by: Kumar Aditya <59607654+kumaraditya303@users.noreply.github.com>
2022-10-04 23:49:10 -07:00
Mark Shannon 76449350b3
GH-91079: Decouple C stack overflow checks from Python recursion checks. (GH-96510) 2022-10-05 01:34:03 +01:00
MonadChains 9302e331c7
gh-94808: Add test coverage for PyObject_HasAttrString (#96627)
* gh-94808: Add test for HasAttrString

* Harmonize to Python C code style guidelines

* Add check to verify no exception thrown
2022-10-03 13:37:15 -07:00
Serhiy Storchaka 0ee9619a4c
gh-97728: Argument Clinic: Fix uninitialized variable in the Py_UNICODE converter (GH-97729)
It affects function os.system() on Windows and Windows-specific modules
winreg, _winapi, _overlapped, and _msi.
2022-10-03 10:42:54 +03:00
Guido van Rossum 63780f4599
GH-97592: Fix crash in C remove_done_callback due to evil code (#97660)
Evil code could cause fut_callbacks to be cleared when PyObject_RichCompareBool is called.
2022-09-30 12:57:09 -07:00
Victor Stinner 9f2f1dd131
gh-94526: getpath_dirname() no longer encodes the path (#97645)
Fix the Python path configuration used to initialized sys.path at
Python startup. Paths are no longer encoded to UTF-8/strict to avoid
encoding errors if it contains surrogate characters (bytes paths are
decoded with the surrogateescape error handler).

getpath_basename() and getpath_dirname() functions no longer encode
the path to UTF-8/strict, but work directly on Unicode strings. These
functions now use PyUnicode_FindChar() and PyUnicode_Substring() on
the Unicode path, rather than strrchr() on the encoded bytes string.
2022-09-30 14:58:30 +02:00
Ofey Chan 83a3de4e06
gh-96348: Deprecate the 3-arg signature of coroutine.throw and generator.throw (GH-96428) 2022-09-30 09:43:02 +01:00
Michael Curran d79dd929ac
bpo-38748: Add ctypes test for stack corruption due to misaligned arguments (GH-26204) 2022-09-26 17:27:44 +01:00
chgnrdv 85752decbf
Fix typo in docstring and remove duplicate signal.h include in faulthandler.c (#96720)
This fix corrects a typo in dump_traceback_later function docstring and removes duplicate signal.h include directive
2022-09-26 18:12:19 +02:00
Kumar Aditya c8c0afc713
GH-78724: Initialize struct.Struct in __new__ (GH-94532)
Closes https://github.com/python/cpython/issues/75960
Closes https://github.com/python/cpython/issues/78724
2022-09-25 14:32:48 +01:00
Mark Dickinson f5f047aa62
gh-96735: Fix undefined behaviour in struct unpacking functions (#96739)
This PR fixes undefined behaviour in the struct module unpacking support functions `bu_longlong`, `lu_longlong`, `bu_int` and `lu_int`; thanks to @kumaraditya303 for finding these.

The fix is to accumulate the bytes in an unsigned integer type instead of a signed integer type, then to convert to the appropriate signed type. In cases where the width matches, that conversion will typically be compiled away to a no-op.
(Evidence from Godbolt: https://godbolt.org/z/5zvxodj64 .)

To make the conversions efficient, I've specialised the relevant functions for their output size: for `bu_longlong` and `lu_longlong`, this only entails checking that the output size is indeed `8`. But `bu_int` and `lu_int` were used for format sizes `2` and `4` - I've split those into two separate functions each.

No tests, because all of the affected cases are already exercised by the test suite.
2022-09-25 10:55:33 +01:00
Dong-hee Na 10e3d398c3
gh-97005: Update libexpat from 2.4.7 to 2.4.9 (gh-97006)
Co-authored-by: Gregory P. Smith [Google] <greg@krypto.org>
2022-09-22 21:25:05 +09:00
Matthias Görgens cbdeda8ce7
gh-96821: Fix undefined behaviour in `_testcapimodule.c` (GH-96915)
* gh-96821: Assert for demonstrating undefined behaviour

* Fix UB

Co-authored-by: C.A.M. Gerlach <CAM.Gerlach@Gerlach.CAM>
2022-09-19 15:59:13 +08:00
adphrost a41ed975e8
GH-91049: Introduce set vectorcall field API for PyFunctionObject (GH-92257)
Co-authored-by: Andrew Frost <adfrost@fb.com>
Co-authored-by: Itamar Ostricher <itamarost@gmail.com>
2022-09-15 16:42:37 +01:00
Benjamin Peterson fd1e477f53
closes gh-96734: Update to Unicode 15.0.0. (GH-96809) 2022-09-13 15:45:12 -07:00
Mark Shannon 12c5f328d2
GH-96754: Check whether the interpreter frame is complete before creating frame object. (GH-96776) 2022-09-13 09:25:16 +01:00
Guido van Rossum df50938f58
GH-46412: More efficient bool() for ndbm/_gdbmmodule (#96692) 2022-09-08 19:32:40 -07:00