Commit Graph

12780 Commits

Author SHA1 Message Date
Dong-hee Na 1935e1cc28
bpo-46794: Bump up the libexpat version into 2.4.6 (GH-31487) 2022-02-23 10:40:30 +09:00
Eric Snow 1f455361ec
bpo-46765: Replace Locally Cached Strings with Statically Initialized Objects (gh-31366)
https://bugs.python.org/issue46765
2022-02-22 17:23:51 -07:00
Inada Naoki 74127b89a8
bpo-46606: Reduce stack usage of getgroups and setgroups (GH-31073)
NGROUPS_MAX was 32 before Linux 2.6.4 but 65536 since Linux 2.6.4.
2022-02-22 11:59:27 +09:00
Andrew Svetlov 4140bcb1cd
bpo-45390: Propagate CancelledError's message from cancelled task to its awaiter (GH-31383)
Co-authored-by: Serhiy Storchaka <storchaka@gmail.com>
2022-02-21 22:59:04 +02:00
Dong-hee Na 2b86616456
bpo-46541: Remove usage of _Py_IDENTIFIER from pyexpat (GH-31468) 2022-02-21 23:46:52 +09:00
Christian Heimes be095f6c32
bpo-46232: Fix parsing of certs with bit string in DN (GH-30351) 2022-02-20 21:42:31 +01:00
Yilei "Dolee" Yang 6312c1052c
bpo-46784: Add newly exported expat symbols to the namespace. (GH-31397)
The libexpat 2.4.1 upgrade from  introduced the following new exported symbols:

* `testingAccountingGetCountBytesDirect`
* `testingAccountingGetCountBytesIndirect`
* `unsignedCharToPrintable`
* `XML_SetBillionLaughsAttackProtectionActivationThreshold`
* `XML_SetBillionLaughsAttackProtectionMaximumAmplification`

We need to adjust [Modules/expat/pyexpatns.h](https://github.com/python/cpython/blob/master/Modules/expat/pyexpatns.h)

(The newer libexpat upgrade  has no new symbols).

Automerge-Triggered-By: GH:gpshead
2022-02-18 14:33:06 -08:00
Dong-hee Na 8cb5f707a8
bpo-46541: Remove usage of _Py_IDENTIFIER from array module (GH-31376) 2022-02-17 13:02:17 +09:00
Dong-hee Na e8a19b092f
bpo-46541: Remove usage of _Py_IDENTIFIER from mmap module (GH-31375) 2022-02-17 01:04:38 +09:00
Erlend Egeberg Aasland b2077117d1
bpo-46541: Replace _Py_IDENTIFIER with _Py_ID in sqlite3 (GH-31351) 2022-02-17 00:24:44 +09:00
Dong-hee Na d64f3caebe
bpo-46541: Remove usage of _Py_IDENTIFIER from csv module (GH-31372) 2022-02-17 00:24:03 +09:00
Dong-hee Na e59309b9d0
bpo-46541: Remove usage of _Py_IDENTIFIER from dbms modules (GH-31358) 2022-02-16 18:57:28 +09:00
Guido van Rossum 602630ac18
bpo-46752: Add TaskGroup; add Task..cancelled(),.uncancel() (GH-31270)
asyncio/taskgroups.py is an adaptation of taskgroup.py from EdgeDb, with the following key changes:

- Allow creating new tasks as long as the last task hasn't finished
- Raise [Base]ExceptionGroup (directly) rather than TaskGroupError deriving from MultiError
- Instead of monkey-patching the parent task's cancel() method,
  add a new public API to Task

The Task class has a new internal flag, `_cancel_requested`, which is set when `.cancel()` is called successfully. The `.cancelling()` method returns the value of this flag. Further `.cancel()` calls while this flag is set return False. To reset this flag, call `.uncancel()`.

Thus, a Task that catches and ignores `CancelledError` should call `.uncancel()` if it wants to be cancellable again; until it does so, it is deemed to be busy with uninterruptible cleanup.

This new Task API helps solve the problem where TaskGroup needs to distinguish between whether the parent task being cancelled "from the outside" vs. "from inside".

Co-authored-by: Yury Selivanov <yury@edgedb.com>
Co-authored-by: Andrew Svetlov <andrew.svetlov@gmail.com>
2022-02-15 15:42:04 -08:00
DongGeon Lee 278fdd3e3a
bpo-46728: fix docstring of combinations_with_replacement for consistency (GH-31293) 2022-02-15 09:02:21 +09:00
Cyril Jouve 8aaaf7e182
bpo-46400: Update libexpat from 2.4.1 to 2.4.4 (GH-31022) 2022-02-13 00:29:41 +09:00
Jacob Walls 168fd6453b
bpo-45948: Remove constructor discrepancy in C version of ElementTree.XMLParser (GH-31152)
Both implementations accept target=None now.
2022-02-12 10:27:02 +02:00
Petr Viktorin 204946986f
bpo-46613: Add PyType_GetModuleByDef to the public API (GH-31081)
* Make PyType_GetModuleByDef public (remove underscore)

Co-authored-by: Victor Stinner <vstinner@python.org>
2022-02-11 17:22:11 +01:00
Dong-hee Na 0ac5372bf6
bpo-46323: Fix double-free issue for borrowed refs (GH-31272) 2022-02-11 17:48:47 +09:00
Dennis Sweeney 0a145069e8
bpo-44953: Add vectorcall for itemgetter and attrgetter instances (GH-27828) 2022-02-10 16:57:47 -05:00
Dong-hee Na db052851a7
bpo-46323: Allow alloca(0) for python callback function of ctypes (GH-31249) 2022-02-10 19:10:07 +09:00
Dong-hee Na d18120cd67
bpo-46323: Reduce stack usage of ctypes python callback function. (GH-31224) 2022-02-10 03:10:11 +09:00
Eric Snow 81c72044a1
bpo-46541: Replace core use of _Py_IDENTIFIER() with statically initialized global objects. (gh-30928)
We're no longer using _Py_IDENTIFIER() (or _Py_static_string()) in any core CPython code.  It is still used in a number of non-builtin stdlib modules.

The replacement is: PyUnicodeObject (not pointer) fields under _PyRuntimeState, statically initialized as part of _PyRuntime.  A new _Py_GET_GLOBAL_IDENTIFIER() macro facilitates lookup of the fields (along with _Py_GET_GLOBAL_STRING() for non-identifier strings).

https://bugs.python.org/issue46541#msg411799 explains the rationale for this change.

The core of the change is in:

* (new) Include/internal/pycore_global_strings.h - the declarations for the global strings, along with the macros
* Include/internal/pycore_runtime_init.h - added the static initializers for the global strings
* Include/internal/pycore_global_objects.h - where the struct in pycore_global_strings.h is hooked into _PyRuntimeState
* Tools/scripts/generate_global_objects.py - added generation of the global string declarations and static initializers

I've also added a --check flag to generate_global_objects.py (along with make check-global-objects) to check for unused global strings.  That check is added to the PR CI config.

The remainder of this change updates the core code to use _Py_GET_GLOBAL_IDENTIFIER() instead of _Py_IDENTIFIER() and the related _Py*Id functions (likewise for _Py_GET_GLOBAL_STRING() instead of _Py_static_string()).  This includes adding a few functions where there wasn't already an alternative to _Py*Id(), replacing the _Py_Identifier * parameter with PyObject *.

The following are not changed (yet):

* stop using _Py_IDENTIFIER() in the stdlib modules
* (maybe) get rid of _Py_IDENTIFIER(), etc. entirely -- this may not be doable as at least one package on PyPI using this (private) API
* (maybe) intern the strings during runtime init

https://bugs.python.org/issue46541
2022-02-08 13:39:07 -07:00
Dong-hee Na b5527688aa
bpo-46323: Use PyObject_Vectorcall while calling ctypes callback function (GH-31138) 2022-02-08 22:09:17 +09:00
Dong-hee Na e959dd9f5c
bpo-46323 Fix ref leak if ctypes.CFuncPtr raises an error. (GH-31209) 2022-02-08 14:22:13 +09:00
Victor Stinner f20ca766fe
bpo-46670: Fix #ifdef in sha3module.c (GH-31180)
* Test if HAVE_ALIGNED_REQUIRED is defined, not its value.
* Define explicitly NOT_PYTHON macro to 0.

Fix "gcc -Wundef" warnings.
2022-02-07 16:22:24 +01:00
Victor Stinner 4f1d3f33dd
bpo-46670: Remove unused macros in the Modules directory (GH-31194)
* bpo-46670: Remove unused macros in the Modules directory

* Add again LINKAT_DIR_FD_CONVERTER: generated by Argument Clinic
2022-02-07 16:21:31 +01:00
Victor Stinner d3e53bc532
bpo-39277: Fix PY_TIMEOUT_MAX cast in _threadmodule.c (GH-31195)
Cast PY_TIMEOUT_MAX to double, not to _PyTime_t.

Fix the clang warning:

Modules/_threadmodule.c:1648:26: warning: implicit conversion from
'_PyTime_t' (aka 'long') to 'double' changes value from
9223372036854775 to 9223372036854776
[-Wimplicit-const-int-float-conversion]
    double timeout_max = (_PyTime_t)PY_TIMEOUT_MAX * 1e-6;
                         ^~~~~~~~~~~~~~~~~~~~~~~~~ ~
2022-02-07 16:21:09 +01:00
Victor Stinner 4cce1352bb
bpo-46323: _ctypes.CFuncPtr fails if _argtypes_ is too long (GH-31188)
ctypes.CFUNCTYPE() and ctypes.WINFUNCTYPE() now fail to create the
type if its "_argtypes_" member contains too many arguments.
Previously, the error was only raised when calling a function.

Change also how CFUNCTYPE() and WINFUNCTYPE() handle KeyError to
prevent creating a chain of exceptions if ctypes.CFuncPtr raises an
error.
2022-02-07 14:53:15 +01:00
Zackery Spytz 59e004af63
bpo-40479: Fix undefined behavior in Modules/_hashopenssl.c (GH-31153)
va_end() must be called before returning.
2022-02-06 23:40:54 -08:00
Victor Stinner b556f53785
bpo-46670: Test if a macro is defined, not its value (GH-31178)
* audioop.c: #ifdef WORDS_BIGENDIAN
* ctypes.h: #ifdef USING_MALLOC_CLOSURE_DOT_C
* _ctypes/malloc_closure.c: #ifdef HAVE_FFI_CLOSURE_ALLOC
  and #ifdef USING_APPLE_OS_LIBFFI
* pytime.c: #ifdef __APPLE__
* unicodeobject.c: #ifdef HAVE_NON_UNICODE_WCHAR_T_REPRESENTATION
2022-02-07 01:46:51 +01:00
Christian Heimes 96b344c2f1
bpo-40280: Address more test failures on Emscripten (GH-31050)
Co-authored-by: Brett Cannon <brett@python.org>
2022-02-05 20:52:01 +01:00
Benjamin Peterson 1aa6be06c4
closes bpo-46626: Expose IP_BIND_ADDRESS_NO_PORT socket option. (GH-31106) 2022-02-03 10:46:50 -08:00
Petr Viktorin 0ef0853012
bpo-46433: _PyType_GetModuleByDef: handle static types in MRO (GH-30696)
Automerge-Triggered-By: GH:encukou
2022-02-02 07:57:51 -08:00
Christian Heimes f66c857572
bpo-45459: Add Py_buffer to limited API (GH-29991)
- [x] ``Py_buffer`` struct
- [x] ``PyBuffer_*()`` API functions
- [x] ``PyBUF_*`` constants
- [x] ``Py_bf_getbuffer`` and ``Py_bf_releasebuffer`` type slots
- [x] ``PyMemoryView_FromBuffer()`` API
- [x] tests for limited API
- [x] ``make regen-limited-abi``
- [x] documentation update
- [ ] export ``PyPickleBuffer*()`` API ???
2022-02-02 07:03:10 -08:00
Kumar Aditya a05866ce3e
Remove Python 3.3 compatibility code from overlapped.c (GH-31049) 2022-02-02 10:16:36 +02:00
Victor Stinner 6c6a153dee
bpo-46417: signal: move siginfo_type to the module state (GH-30964) 2022-01-27 21:21:50 +01:00
Victor Stinner af32b3ef1f
bpo-40170: PyType_SUPPORTS_WEAKREFS() becomes a regular function (GH-30938)
Convert the PyType_SUPPORTS_WEAKREFS() macro to a regular function.
It no longer access the PyTypeObject.tp_weaklistoffset member
directly.

Add _PyType_SUPPORTS_WEAKREFS() static inline functions, used
internally by Python for best performance.
2022-01-27 03:00:55 +01:00
Erlend Egeberg Aasland 3eb3b4f270
bpo-43853: Expand test suite for SQLite UDF's (GH-27642) 2022-01-26 08:26:16 -08:00
Christian Heimes 6e5a193816
bpo-46513: Remove AC_C_CHAR_UNSIGNED / __CHAR_UNSIGNED__ (GH-30851) 2022-01-26 01:03:49 -08:00
Mark Dickinson 83a0ef2162
bpo-29882: Fix portability bug introduced in GH-30774 (#30794) 2022-01-23 09:59:34 +00:00
Victor Stinner 9c8e490b8f
bpo-46417: Clear _io module static objects at exit (GH-30807)
Add _PyIO_Fini() function, called by finalize_interp_clear(). It
clears static objects used by the _io extension module.
2022-01-22 23:22:20 +01:00
Kumar Aditya ea5b96842e
bpo-46469: Make asyncio generic classes return GenericAlias (GH-30777)
* bpo-46469: Make asyncio generic classes return GenericAlias

* 📜🤖 Added by blurb_it.

* Update Misc/NEWS.d/next/Library/2022-01-22-05-05-08.bpo-46469.plUab5.rst

Co-authored-by: Jelle Zijlstra <jelle.zijlstra@gmail.com>

Co-authored-by: blurb-it[bot] <43283697+blurb-it[bot]@users.noreply.github.com>
Co-authored-by: Jelle Zijlstra <jelle.zijlstra@gmail.com>
2022-01-22 13:28:53 +02:00
Erlend Egeberg Aasland 38afeb1a33
bpo-46249: Move set lastrowid out of the sqlite3 query loop (GH-30489) 2022-01-22 18:40:22 +09:00
Victor Stinner 8ee07dda13
bpo-46417: Add _PyType_GetSubclasses() function (GH-30761)
Add a new _PyType_GetSubclasses() function to get type's subclasses.

_PyType_GetSubclasses(type) returns a list which holds strong
refererences to subclasses. It is safer than iterating on
type->tp_subclasses which yields weak references and can be modified
in the loop.

_PyType_GetSubclasses(type) now holds a reference to the tp_subclasses
dict while creating the list of subclasses.

set_collection_flag_recursive() of _abc.c now uses
_PyType_GetSubclasses().
2022-01-21 23:29:10 +01:00
Thomas Klausner 40fcd16889
bpo-30512: Add CAN Socket support for NetBSD (GH-30066) 2022-01-21 09:44:05 +02:00
Victor Stinner d013b24135
bpo-46417: signal uses PyStructSequence_NewType() (GH-30735)
The signal module now creates its struct_siginfo type as a heap type
using PyStructSequence_NewType(), rather than using a static type.

Add 'siginfo_type' member to the global signal_state_t structure.
2022-01-21 04:02:38 +01:00
Victor Stinner 1781d55eb3
bpo-46417: _curses uses PyStructSequence_NewType() (GH-30736)
The _curses module now creates its ncurses_version type as a heap
type using PyStructSequence_NewType(), rather than using a static
type.

* Move _PyStructSequence_FiniType() definition to pycore_structseq.h.
* test.pythoninfo: log curses.ncurses_version.
2022-01-21 03:30:20 +01:00
Victor Stinner 17f268a4ae
bpo-46417: time module uses PyStructSequence_NewType() (GH-30734)
The time module now creates its struct_time type as a heap
type using PyStructSequence_NewType(), rather than using a static
type.

* Add a module state to the time module: add traverse, clear and free
  functions.
* Use PyModule_AddType().
* Remove the 'initialized' variable.
2022-01-21 02:52:43 +01:00
Victor Stinner f389b37fb1
bpo-46417: _thread uses PyStructSequence_NewType() (GH-30733)
The _thread module now creates its _ExceptHookArgs type as a heap
type using PyStructSequence_NewType(), rather than using a static
type.
2022-01-21 02:51:04 +01:00
Zane Bitter 27df7566bc
bpo-41857: mention timeout argument units in select.poll() and select.depoll() doc-strings (GH-22406) 2022-01-21 01:08:44 +02:00
Steve Dower 7407fe4c25
bpo-46028: Calculate base_executable by resolving symlinks in a venv (GH-30144) 2022-01-18 15:46:26 +00:00
Brandt Bucher 5cd9a162cd
bpo-46361: Fix "small" `int` caching (GH-30583) 2022-01-16 16:06:37 +00:00
Mark Dickinson d02c5e9b55
bpo-46258: Streamline isqrt fast path (#30333) 2022-01-15 09:58:04 +00:00
Christian Heimes cfbde65df3
bpo-46383: Fix signature of zoneinfo module_free function (GH-30607) 2022-01-15 09:52:19 +01:00
Victor Stinner 7c770d3350
bpo-46280: Fix tracemalloc_copy_domain() (GH-30591)
Test if tracemalloc_copy_traces() failed to allocated memory in
tracemalloc_copy_domain().
2022-01-14 05:11:38 +01:00
Eric Snow 322f962f3e
bpo-45953: Statically initialize all the non-object PyInterpreterState fields we can. (gh-30589)
https://bugs.python.org/issue45953
2022-01-13 17:17:28 -07:00
neonene d4e64cd4b0
bpo-46362: Ensure ntpath.abspath() uses the Windows API correctly (GH-30571)
This makes ntpath.abspath()/getpath_abspath() follow normpath(), since some WinAPIs such as PathCchSkipRoot() require backslashed paths.
2022-01-13 23:35:42 +00:00
Christian Heimes 276c234ce0
bpo-40479: Fix typo, flag must be set for OpenSSL < 3.0.0 (GH-30584) 2022-01-13 21:47:42 +01:00
Victor Stinner 1a4d1c1c9b
bpo-46070: _PyGC_Fini() untracks objects (GH-30577)
Py_EndInterpreter() now explicitly untracks all objects currently
tracked by the GC. Previously, if an object was used later by another
interpreter, calling PyObject_GC_UnTrack() on the object crashed if
the previous or the next object of the PyGC_Head structure became a
dangling pointer.
2022-01-13 19:28:32 +01:00
Christian Heimes 443b308fee
bpo-40479: Fix hashlib's usedforsecurity for OpenSSL 3.0.0 (GH-30455) 2022-01-13 09:46:38 +01:00
Christian Heimes a6ca8eee22
bpo-46315: Add ifdef HAVE_ feature checks for WASI compatibility (GH-30507) 2022-01-13 09:46:04 +01:00
Eric Snow ed57b36c32
bpo-45953: Statically allocate the main interpreter (and initial thread state). (gh-29883)
Previously, the main interpreter was allocated on the heap during runtime initialization.  Here we instead embed it into _PyRuntimeState, which means it is statically allocated as part of the _PyRuntime global.  The same goes for the initial thread state (of each interpreter, including the main one).  Consequently there are fewer allocations during runtime/interpreter init, fewer possible failures, and better memory locality.

FYI, this also helps efforts to consolidate globals, which in turns helps work on subinterpreter isolation.

https://bugs.python.org/issue45953
2022-01-12 16:28:46 -07:00
Christian Heimes e34c9367f8
bpo-40280: Allow to compile _testcapi as builtin module (GH-30559) 2022-01-12 20:27:37 +01:00
Christian Heimes 43839ba438
bpo-40280: Add --with-emscripten-target to build for browser or node (GH-30552)
Co-authored-by: Ethan Smith <ethan@ethanhs.me>
2022-01-12 16:08:19 +01:00
Victor Stinner 08bc1bad11
bpo-46303: Fix fileutils.h compiler warnings (GH-30550)
Add missing pycore_fileutils.h include in _tkinter.c and
_testconsole.c.
2022-01-12 00:35:26 +01:00
Victor Stinner ea1a54506b
bpo-46303: Move fileutils.h private functions to internal C API (GH-30484)
Move almost all private functions of Include/cpython/fileutils.h to
the internal C API Include/internal/pycore_fileutils.h.

Only keep _Py_fopen_obj() in Include/cpython/fileutils.h, since it's
used by _testcapi which must not use the internal C API.

Move EncodeLocaleEx() and DecodeLocaleEx() functions from _testcapi
to _testinternalcapi, since the C API moved to the internal C API.
2022-01-11 11:56:16 +01:00
Serhiy Storchaka 2d787971c6
bpo-37295: Use constant-time comb() and perm() for larger n depending on k (GH-30305) 2022-01-09 15:32:25 +02:00
Daniel c9dc1f491e
bpo-46297: Fix interpreter crash on startup with multiple PythonPaths set in registry (GH-30466) 2022-01-07 22:26:00 +00:00
Erlend Egeberg Aasland b127e70a8a
bpo-46070: Fix asyncio initialisation guard (GH-30423)
If init flag is set, exit successfully immediately.
If not, only set the flag after successful initialization.
2022-01-07 15:08:19 +01:00
Erlend Egeberg Aasland f1a58441ee
bpo-44092: Remove unused member `reset` from `sqlite3.Cursor` (GH-30377)
Automerge-Triggered-By: GH:pablogsal
2022-01-03 15:47:16 -08:00
Erlend Egeberg Aasland 9d6a239a34
bpo-44092: Don't reset statements/cursors before rollback (GH-26026)
In SQLite versions pre 3.7.11, pending statements would block a rollback.  This is no longer the case, so remove the workaround.
2022-01-03 19:02:39 +00:00
David CARLIER c960b191b8
bpo-46222: posixmodule sendfile FreeBSD's constants updates. (GH-30327)
* posixodule sendfile FreeBSD's constants updates.

* 📜🤖 Added by blurb_it.

Co-authored-by: blurb-it[bot] <43283697+blurb-it[bot]@users.noreply.github.com>
2022-01-03 15:01:04 +02:00
Irit Katriel a82baed0e9
bpo-45615: Add missing test for printing traceback for non-exception. Fix traceback.py (GH-30091) 2022-01-02 09:34:03 +00:00
Mark Dickinson 0b58bac3e7
bpo-37295: More direct computation of power-of-two factor in math.comb (GH-30313)
Co-authored-by: Serhiy Storchaka <storchaka@gmail.com>
2021-12-31 19:52:27 +00:00
Sebastian Pipping e18d81569f
bpo-45321: Add missing error codes to module `xml.parsers.expat.errors` (GH-30188)
The idea is to ensure that module `xml.parsers.expat.errors`
contains all known error codes and messages,
even when CPython is compiled or run with an outdated version of libexpat.

https://bugs.python.org/issue45321
2021-12-31 10:57:00 +01:00
David CARLIER 66c47b63a0
bpo-46176: mmap module adding MAP_STACK constant. (GH-30252) 2021-12-29 22:52:29 +09:00
Mark Dickinson 02b5417f11
bpo-37295: Speed up math.comb(n, k) for 0 <= k <= n <= 67 (GH-30275) 2021-12-28 12:26:40 +00:00
neonene 6214caafbe
bpo-40915: Avoid compiler warnings by fixing mmapmodule conversion from LARGE_INTEGER to Py_ssize_t (GH-30175) 2021-12-18 13:03:43 +00:00
Irit Katriel 396b58345f
bpo-45711: Remove type and traceback from exc_info (GH-30122)
* Do not PUSH/POP traceback or type to the stack as part of exc_info

* Remove exc_traceback and exc_type from _PyErr_StackItem

* Add to what's new, because this change breaks things like Cython
2021-12-17 14:46:22 +00:00
Mark Shannon 342b93f9f2
bpo-46072: Add --with-pystats configure option to simplify gathering of VM stats (GH-30116)
* Simplify specialization stats collection macros.

* Add --enable-pystats option to configure.

* Update specialization summary script to handle larger number of kinds
2021-12-15 15:32:32 +00:00
Christian Heimes 74821b3053
bpo-46023: Skip build if module is marked as DISABLED (GH-30100) 2021-12-14 15:42:46 +01:00
Gareth Rees a62be77266
bpo-45643: Add signal.SIGSTKFLT on platforms where this is defined (GH-29266) 2021-12-13 18:22:43 +01:00
Kumar Aditya 41026c3155
bpo-45855: Replaced deprecated `PyImport_ImportModuleNoBlock` with PyImport_ImportModule (GH-30046) 2021-12-12 10:45:20 +02:00
Steve Dower bfc59ed0a0
bpo-46049: Fixes ._pth support on non-Windows (GH-30051) 2021-12-11 15:06:17 +00:00
Steve Dower 971ece8e17
bpo-46048: Fix parsing of single character lines in getpath readlines() (GH-30048) 2021-12-11 13:43:40 +00:00
neonene 3f398a77d3
bpo-45582: Fix test_embed failure during a PGO build on Windows (GH-30014)
This defines VPATH differently in PGO instrumentation builds, to account for a different default output directory. It also adds sys._vpath on Windows to make the value available to sysconfig so that it can be used in tests.
2021-12-10 17:13:55 +00:00
Christian Heimes 036bbb1d1b
bpo-46023: Fix makesetup handling of disabled rule (GH-30001) 2021-12-10 14:11:55 +01:00
Thomas Klausner 2fb797e93c
bpo-46000: Improve NetBSD curses compatibility (GH-29947) 2021-12-10 11:41:47 +02:00
Gabriele N. Tornetta 50669083fe
bpo-43931: Export Python version as API data (GH-25577)
When Python is embedded in other applications, it is not easy to determine which version of Python is being used. This change exposes the Python version as part of the API data. Tools like Austin (https://github.com/P403n1x87/austin) can benefit from this data when targeting applications like uWSGI, as the Python version can then be inferred systematically by looking at the exported symbols rather than relying on unreliable pattern matching or other hacks (like remote code execution etc...).

Automerge-Triggered-By: GH:pablogsal
2021-12-09 17:52:05 -08:00
Steve Dower 3363e1cb05
bpo-46018: Ensure that math.expm1 does not raise on underflow (GH-29997) 2021-12-09 18:31:54 +00:00
Pablo Galindo Salgado f0d290d25c
bpo-46025: Fix a crash in the atexit module for auto-unregistering functions (GH-30002) 2021-12-09 13:53:44 +00:00
David CARLIER 267539bff7
bpo-46016: fcntl module add FreeBSD's F_DUP2FD_CLOEXEC flag support (GH-29993) 2021-12-08 23:28:51 +01:00
Steve Dower 7778116c2f
bpo-46015: Fixes calculation of sys.path in a venv on Windows (GH-29992)
Also ensures that pybuilddir.txt is written early enough in the build to be picked up by later steps.
2021-12-08 19:25:58 +00:00
Steve Dower b0b3086279
bpo-45582: Write empty pybuilddir.txt on Windows to allow relocatable build directories (GH-29979) 2021-12-08 02:18:21 +00:00
Eric Snow 758b74e71e
bpo-46008: Add _PyInterpreterState_Main(). (gh-29978)
PyInterpreterState_Main() is a plain function exposed in the public C-API. For internal usage we can take the more efficient approach in this PR.

https://bugs.python.org/issue46008
2021-12-07 18:56:06 -07:00
Eric Snow 1f384e3184
bpo-46008: Stop calling _PyThreadState_Init() in new_threadstate(). (gh-29973)
This simplifies new_threadstate().  We also rename _PyThreadState_Init() to _PyThreadState_SetCurrent() to reflect what it actually does.

https://bugs.python.org/issue46008
2021-12-07 17:26:29 -07:00
Eric Snow 313f92a57b
bpo-46008: Move thread-related interpreter state into a sub-struct. (gh-29971)
This parallels _PyRuntimeState.interpreters.  Doing this helps make it more clear what part of PyInterpreterState relates to its threads.

https://bugs.python.org/issue46008
2021-12-07 14:03:47 -07:00
Christian Heimes 91b59a3fcd
bpo-45847: Revert Port _ctypes partly to PY_STDLIB_MOD (GH-29747) (GH-29969) 2021-12-07 20:56:41 +01:00
Christian Heimes 06c4ae8b13
bpo-45582: Fix framework path and bootstrap build (GH-29954)
* Check NS API return values for NULL to prevent segfault in
  ``_bootstrap_python``.
* Set modPathInitialized to 1 so the ``decode_to_dict`` path is used.

Signed-off-by: Christian Heimes <christian@python.org>
2021-12-07 19:09:53 +01:00
Mark Shannon 8319114fee
bpo-45947: Place dict and values pointer at fixed (negative) offset just before GC header. (GH-29879)
* Place __dict__ immediately before GC header for plain Python objects.

* Fix up lazy dict creation logic to use managed dict pointers.

* Manage values pointer, placing them directly before managed dict pointers.

* Convert hint-based load/store attr specialization target managed dict classes.

* Specialize LOAD_METHOD for managed dict objects.

* Remove unsafe _PyObject_GC_Calloc function.

* Remove unsafe _PyObject_GC_Malloc() function.

* Add comment explaning use of Py_TPFLAGS_MANAGED_DICT.
2021-12-07 16:02:53 +00:00
Steve Dower b7ef27bc08
bpo-45582: Ensure PYTHONHOME still overrides detected build prefixes (GH-29948) 2021-12-07 00:07:35 +00:00
Christian Heimes f16f93e527
bpo-45582: framework build: modPath must not be const (GH-29944)
Co-authored-by: Ronald Oussoren <ronaldoussoren@mac.com>
2021-12-06 19:13:12 +01:00
neonene af1db4eb55
bpo-45582: Fix getpath_isxfile() and test_embed on Windows (GH-29930) 2021-12-06 17:25:19 +00:00
Christian Heimes 612e59b53f
bpo-45950: Fix macOS framework builds of _bootstrap_python (GH-29936) 2021-12-06 17:13:53 +01:00
Serhiy Storchaka 60c320c38e
bpo-37295: Optimize math.comb() and math.perm() (GH-29090)
For very large numbers use divide-and-conquer algorithm for getting
benefit of Karatsuba multiplication of large numbers.

Do calculations completely in C unsigned long long instead of Python
integers if possible.
2021-12-05 22:26:10 +02:00
Christian Heimes 628abe4463
bpo-45582: Fix signature of _Py_Get_Getpath_CodeObject (GH-29921) 2021-12-05 18:41:46 +01:00
Serhiy Storchaka d15cdb2f32
bpo-27946: Fix possible crash in ElementTree.Element (GH-29915)
Getting an attribute via attrib.get() simultaneously with replacing
the attrib dict can lead to access to deallocated dict.
2021-12-05 14:22:54 +02:00
Christian Heimes 64be8d369b
bpo-45847: Update whatsnew and add place holder entries for missing extensions (GH-29914) 2021-12-04 15:14:48 +01:00
Christian Heimes 4045392e0e
bpo-45847: Port _ctypes partly to PY_STDLIB_MOD (GH-29747)
Co-authored-by: Erlend Egeberg Aasland <erlend.aasland@innova.no>
2021-12-04 11:21:43 +01:00
neonene 7d7c91a8e8
bpo-45582: Add a NOT operator to the condition in getpath_isxfile (GH-29906) 2021-12-03 22:04:11 +00:00
Irit Katriel 2ff758bd1a
bpo-45711: [asyncio] Normalize exceptions immediately after Fetch, before they are stored as StackItem, which should be normalized (GH-29890) 2021-12-03 19:05:14 +00:00
Christian Heimes ccb73a0d50
bpo-45582: Fix out-of-tree build issues with new getpath (GH-29902) 2021-12-03 14:47:06 +01:00
Steve Dower 99fcf15052
bpo-45582: Port getpath[p].c to Python (GH-29041)
The getpath.py file is frozen at build time and executed as code over a namespace. It is never imported, nor is it meant to be importable or reusable. However, it should be easier to read, modify, and patch than the previous code.

This commit attempts to preserve every previously tested quirk, but these may be changed in the future to better align platforms.
2021-12-03 00:08:42 +00:00
Gideon 6266e4af87
bpo-45917: Add math.exp2() method - return 2 raised to the power of x (GH-29829)
Co-authored-by: Mark Dickinson <dickinsm@gmail.com>
2021-11-29 18:55:43 +00:00
Erlend Egeberg Aasland c4a69a4ad0
bpo-45828: Use unraisable exceptions within sqlite3 callbacks (FH-29591) 2021-11-29 15:22:32 +00:00
Mark Shannon 60929576e4
bpo-45786: Allocate space for frame in frame object. (GH-29729) 2021-11-29 12:34:59 +00:00
Christian Heimes 4dc198dedd
bpo-45847: Fix _multiprocessing on platforms without sem_open (GH-29833) 2021-11-29 10:36:10 +01:00
Erlend Egeberg Aasland aaf42222cf
bpo-45847: Port _multiprocessing to PY_STDLIB_MOD (GH-29768) 2021-11-28 13:00:51 +01:00
Ma Lin 7edb6270a7
bpo-41735: Fix thread lock in zlib.Decompress.flush() may go wrong (GH-29587)
* Fix thread lock in zlib.Decompress.flush() may go wrong

Getting `.unconsumed_tail` before acquiring the thread lock may mix up decompress state.
2021-11-26 16:18:17 -08:00
Christian Heimes 4ebde73b8e
bpo-40280: Move hard-coded feature checks to configure (GH-29789)
Co-authored-by: Erlend Egeberg Aasland <erlend.aasland@innova.no>
2021-11-26 19:36:48 +01:00
Christian Heimes 8caceb7a47
bpo-40280: Add configure check for socket shutdown (GH-29795) 2021-11-26 15:16:54 +01:00
Christian Heimes ee1e2c604c
bpo-40280: Use Setup.stdlib static for wasm builds (GH-29784)
``Modules/Setup.stdlib`` contains ``Setup`` lines for all stdlib extension modules for which ``configure`` has detected their dependencies. The file is not used yet and still under development. To use the file, do ``ln -sfr Modules/Setup.stdlib Modules/Setup.local``.
2021-11-26 08:26:49 +01:00
Christian Heimes dc19e86983
bpo-41498: Fix build on platforms without sigset_t (GH-29770) 2021-11-25 12:28:41 +01:00
Pablo Galindo Salgado abfc794bbf
bpo-45822: Minor cleanups to the test_Py_CompileString test (GH-29750) 2021-11-24 18:30:03 +00:00
Christian Heimes 8af6481f6b
bpo-45847: Port _uuid to PY_STDLIB_MOD (GH-29741) 2021-11-24 10:20:37 +01:00
Erlend Egeberg Aasland 324527012f
bpo-45847: Port _posixshmem to PY_STDLIB_MOD (GH-29738) 2021-11-24 08:19:17 +01:00
Christian Heimes 9cf5646bb4
bpo-45847: Port _gdbm to PY_STDLIB_MOD (GH-29720) 2021-11-23 22:58:38 +01:00
Christian Heimes b9e9292d75
bpo-45847: Port _ssl and _hashlib to PY_STDLIB_MOD (GH-29727) 2021-11-23 22:58:13 +01:00
Christian Heimes 095bc8f0d6
bpo-45847: Port _crypt to PY_STDLIB_MOD (GH-29725) 2021-11-23 22:26:50 +01:00
Erlend Egeberg Aasland d82f2caf94
bpo-45847: Port _socket to PY_STDLIB_MOD (GH-29713) 2021-11-23 08:52:05 +01:00
Erlend Egeberg Aasland 2dc7d3dda6
bpo-45847: Port xxlimited and xxlimited_35 to PY_STDLIB_MOD (GH-29707) 2021-11-22 21:27:05 +01:00
Christian Heimes 0e1c2f3ef8
bpo-45847: port _struct to PY_STDLIB_MOD (GH-29706) 2021-11-22 15:58:43 +01:00
Erlend Egeberg Aasland eee683cbde
bpo-45847: Port audioop, _csv, and _posixsubprocess to PY_STDLIB_MOD_SIMPLE (GH-29705)
Automerge-Triggered-By: GH:tiran
2021-11-22 06:37:25 -08:00
Christian Heimes c6dec7e27a
bpo-45847: Port nis module to PY_STDLIB_MOD (GH-29699) 2021-11-22 15:18:41 +01:00
Christian Heimes d9cedabeba
bpo-45847: Port compression libs to PY_STDLIB_MOD (GH-29702) 2021-11-22 14:52:29 +01:00
Erlend Egeberg Aasland b451673f93
bpo-45847: Port mmap, select, and _xxsubinterpreters to Py_STDLIB_MOD (GH-29703) 2021-11-22 14:49:58 +01:00
Erlend Egeberg Aasland 5b946cadaa
bpo-45847: Port fcntl to Py_STDLIB_MOD (GH-29696)
Co-authored-by: Christian Heimes <christian@python.org>
2021-11-22 14:02:27 +01:00
Erlend Egeberg Aasland 39f7d2ff01
bpo-45847: Port _lfprof, _opcode, _asyncio, _queue, _statistics, and _typing to PY_STDLIB_MOD_SIMPLE (GH-29690)
Automerge-Triggered-By: GH:tiran
2021-11-22 01:57:50 -08:00
Erlend Egeberg Aasland 718cee08cc
bpo-45847: Port _bisect, _heapq, _json, _pickle, _random, and _zoneinfo to PY_STDLIB_MOD_SIMPLE (GH-29689)
Automerge-Triggered-By: GH:tiran
2021-11-22 00:45:41 -08:00
Erlend Egeberg Aasland 133c65a870
bpo-45847: Port array, _contextvars, math, and cmath to PY_STDLIB_MOD_SIMPLE (GH-29688) 2021-11-22 09:05:36 +01:00
Christian Heimes 2afa1a1266
bpo-45847: Port codecs and unicodedata to PY_STDLIB_MOD (GH-29685) 2021-11-21 14:08:47 +01:00
Christian Heimes f201d261cf
bpo-45847: Port grp, spwd, termios, resource, syslog to PY_STDLIB_MOD (GH-29668) 2021-11-21 10:45:31 +01:00
Christian Heimes f36c69a26e
bpo-45847: Port test modules to PY_STDLIB_MOD (GH-29660) 2021-11-20 16:43:10 +01:00
Christian Heimes 5596909eac
bpo-45847: Port _scproxy to PY_STDLIB_MOD (GH-29644) 2021-11-20 10:18:48 +01:00
Christian Heimes c8c21bdd19
bpo-45847: Port builtin hashlib extensions to PY_STDLIB_MOD (GH-29642) 2021-11-19 20:20:32 +01:00
Christian Heimes 7e44dc0ba7
bpo-45573: Move mandatory core modules to Modules/Setup.bootstrap (GH-29616) 2021-11-19 16:40:57 +01:00
Erlend Egeberg Aasland 29e5874d5a
bpo-45774: Autoconfiscate SQLite detection (GH-29507)
Co-authored-by: Christian Heimes <christian@python.org>
2021-11-19 15:10:41 +01:00
Christian Heimes e4bb22fabb
bpo-45573: Add Modules/Setup.stdlib with conditional modules (GH-29615) 2021-11-18 14:40:01 +01:00
Erlend Egeberg Aasland 0920b61a0c
bpo-45512: Use Argument Clinic to set sqlite3 isolation level (GH-29593) 2021-11-18 18:18:09 +09:00
Christian Heimes 25ecc040d0
bpo-45573: Introduce extension module flags in Makefile (GH-29594)
``configure`` now uses a standardized format to forward state, compiler
flags, and linker flags to ``Makefile``, ``setup.py``, and
``Modules/Setup``. ``makesetup`` use the new variables by default if a
module line does not contain any compiler or linker flags. ``setup.py``
has a new function ``addext()``.

For a module ``egg``, configure adds:

* ``MODULE_EGG`` with value yes, missing, disabled, or n/a
* ``MODULE_EGG_CFLAGS``
* ``MODULE_EGG_LDFLAGS``

``Makefile.pre.in`` may also provide ``MODULE_EGG_DEPS`` that lists
dependencies such as header files and static libs.

Signed-off-by: Christian Heimes <christian@python.org>
2021-11-18 09:18:44 +01:00
Dong-hee Na e002bbc6cc
bpo-45512: Simplify manage isolation level (GH-29562) 2021-11-17 21:47:02 +09:00
Pablo Galindo Salgado da20d7401d
bpo-45822: Respect PEP 263's coding cookies in the parser even if flags are not provided (GH-29582) 2021-11-16 12:30:47 -08:00
Erlend Egeberg Aasland 9d6215a54c
bpo-45126: Harden `sqlite3` connection initialisation (GH-28227) 2021-11-16 15:53:35 +01:00
Dong-hee Na 55868f1a33
bpo-45429: Support CREATE_WAITABLE_TIMER_HIGH_RESOLUTION if possible (GH-29203) 2021-11-16 22:41:20 +09:00
Mark Shannon b931077375
bpo-45753: Make recursion checks more efficient. (GH-29524)
* Uses recursion remaining, instead of recursion depth to speed up check against recursion limit.
2021-11-16 11:01:57 +00:00
Erlend Egeberg Aasland 822c3dcce3
bpo-45512: Raise exception if sqlite3.Connection.__init__ is called with bad isolation level (#29561)
* bpo-45512: Raise sqlite3.Connection.__init__ is called with bad isolation level

* Also explicitly test allowed isolation levels

* Use subTest for better error messages if something goes wrong

* Update Lib/test/test_sqlite3/test_dbapi.py

Co-authored-by: Dong-hee Na <donghee.na92@gmail.com>

Co-authored-by: Dong-hee Na <donghee.na92@gmail.com>
2021-11-15 21:55:38 +09:00
Erlend Egeberg Aasland b567b9d74b
bpo-45512: Simplify isolation_level handling in `sqlite3` (GH-29053) 2021-11-15 16:50:59 +09:00
Christian Heimes 464e6616be
bpo-45800: Move pyexpat build setup into configure (GH-29547)
Settings for :mod:`pyexpat` C extension are now detected by ``configure``.
The bundled ``expat`` library is built in ``Makefile``.

Signed-off-by: Christian Heimes <christian@python.org>
2021-11-14 10:02:24 +01:00
Christian Heimes 0486570f7b
bpo-45798: Move _decimal build setup into configure (GH-29541)
Settings for :mod:`decimal` internal C extension are now detected by
:program:`configure`. The bundled `libmpdec` library is built in
``Makefile``.

Signed-off-by: Christian Heimes <christian@python.org>
2021-11-13 13:56:16 +01:00
Christian Heimes 0a9f69539b
bpo-45747: Detect gdbm/dbm dependencies in configure (GH-29467)
Co-authored-by: Erlend Egeberg Aasland <erlend.aasland@innova.no>
2021-11-10 20:26:55 +01:00
Erlend Egeberg Aasland c1323d4b8c
bpo-45754: Use correct SQLite limit when checking statement length (GH-29489) 2021-11-10 18:46:11 +00:00
Irit Katriel 05fbd60147
bpo-45711: Use _PyErr_ClearExcState instead of setting only exc_value to NULL (GH-29404) 2021-11-10 16:57:14 +00:00
Christian Heimes cbab997efb
bpo-45723: Prepare support for autoconf 2.71 (GH-29441) 2021-11-08 19:31:14 +01:00
Christian Heimes 24af9a40a8
bpo-45743: Move __APPLE_USE_RFC_3542 into socketmodule.c (GH-29456) 2021-11-08 12:48:34 +01:00
180909 122ca4d73f
remove the repeat 'the' (#29455) 2021-11-08 10:19:54 +02:00
Christian Heimes e9594f6747
bpo-45731: Handle --enable-loadable-sqlite-extensions in configure (GH-29434) 2021-11-06 10:30:37 +01:00
Erlend Egeberg Aasland c273986711
bpo-45613: Set `sqlite3.threadsafety` dynamically (GH-29227)
Use the compile-time selected default SQLite threaded mode to set the
DB-API 2.0 attribute 'threadsafety'

Mappings:
  - SQLITE_THREADSAFE=0 => threadsafety=0
  - SQLITE_THREADSAFE=1 => threadsafety=3
  - SQLITE_THREADSAFE=2 => threadsafety=1
2021-11-03 21:01:37 +00:00
Erlend Egeberg Aasland 456e27ac0a
bpo-24139: Add support for SQLite extended result codes (GH-28076) 2021-11-02 23:49:38 +00:00
Erlend Egeberg Aasland 401272e6e6
bpo-42064: Adapt `sqlite3` to multi-phase init (PEP 489) (GH-29234) 2021-11-02 16:35:51 +01:00
Erlend Egeberg Aasland b6b38a8226
bpo-45243: Add support for setting/getting `sqlite3` connection limits (GH-28463) 2021-11-01 22:50:53 +00:00
Erlend Egeberg Aasland 39b4d5938c
bpo-45634: Don't combine error checks when adding `sqlite3` int constants (GH-29251) 2021-10-31 19:01:19 +09:00
Erlend Egeberg Aasland e2e62b3808
bpo-45581: Raise `MemoryError` in `sqlite3.connect` if SQLite signals memory error (GH-29171) 2021-10-29 22:21:58 +02:00
Christian Heimes f0150ac94a
bpo-45548: Some test modules must be built as shared libs (GH-29268)
Some test cases don't work when test modules are static extensions.

Add dependency on Modules/config.c to trigger a rebuild whenever a
module build type is changed.

``makesetup`` puts shared extensions into ``Modules/`` directory. Create
symlinks from pybuilddir so the extensions can be imported.

Note: It is not possible to use the content of pybuilddir.txt as a build
target. Makefile evaluates target variables in the first pass. The
pybuilddir.txt file does not exist at that point.
2021-10-29 17:49:57 +02:00
Tim Golden 7bddd96982
bpo-45621: Small changes to mmap (GH-29247)
* Small tidy-ups / comments
* Use randomized names when testing tagged mmaps to avoid any risk of parallel tests treading on each others' toes
2021-10-29 09:20:21 +01:00
Mark Shannon f291404a80
bpo-45637: Store the frame pointer in the cframe (GH-29267)
* Rename 'frame' to 'current_frame'
2021-10-28 13:59:11 +01:00
Christian Heimes 4c95fb4640
bpo-45548: Fix out-of-tree and Debian builds (GH-29263)
Add Modules subdirs to SRCDIRS to generate directories for out-of-tree
object files.

Debian wants ncurses lib. Works on Fedora, too.

Debian also needs pkg-config to detect correct flags.

Remove more outdated comments. Makefile now tracks header dependencies
-lintl is injected by configure when needed. Build _dbm with
gdbm-compat.

Group some modules by purpose. socket, select, and mmap work on Windows,
too.
2021-10-28 11:57:38 +02:00
Erlend Egeberg Aasland 8f24b7dbcb
bpo-42064: Convert `sqlite3` global state to module state (GH-29073) 2021-10-27 13:12:21 +02:00
Benjamin Peterson 19a6c41e56
Remove unused variables. (GH-29231) 2021-10-26 16:22:34 -07:00
Tim Golden aea5ecc458
bpo-40915: Fix mmap resize bugs on Windows (GH-29213)
(original patch by eryksun)

Correctly hand various failure modes when resizing an mmap on Windows:

* Resizing a pagefile-backed mmap now creates a new mmap and copies data
* Attempting to resize when another mapping is held on the same file raises an OSError
* Attempting to resize a nametagged mmap raises an OSError if another mapping is held with the same nametag
2021-10-26 22:56:43 +01:00
Christian Heimes b5ee79494b
bpo-45548: makesetup improvements (GH-29225)
* record which modules are build as shared extensions
* put object files in same directory as source files
* remove dependency on deleted _math.c

Signed-off-by: Christian Heimes <christian@python.org>
2021-10-26 10:58:43 +02:00
Christian Heimes 2b8677a3cd
bpo-45548: FreeBSD doesn't like auto vars in makesetup (GH-29216) 2021-10-25 20:52:55 +02:00
Christian Heimes ece916e92c
bpo-45548: Add missing extensions to Modules/Setup (GH-29199)
Adds _ctypes, _decimal, _multiprocessing, _posixshmem, _scproxy, _sqlite3,
and _uuid.

Use Makefile variables to build OpenSSL and TCL/TK modules.

Drop ``-lm`` from math lines. Extensions are always linked against libm.

Signed-off-by: Christian Heimes <christian@python.org>
2021-10-25 20:01:52 +02:00
Christian Heimes 81669d1b7b
bpo-45595: Make extensions depend on header files (GH-29198)
``setup.py`` and ``makesetup`` now track build dependencies on all Python
header files and module specific header files.

Signed-off-by: Christian Heimes <christian@python.org>
2021-10-25 20:01:03 +02:00
Christian Heimes fa26245a1c
bpo-45548: Remove _math.c workarounds for pre-C99 libm (GH-29179)
The :mod:`math` and :mod:`cmath` implementation now require a C99 compatible
``libm`` and no longer ship with workarounds for missing acosh, asinh,
expm1, and log1p functions.

The changeset also removes ``_math.c`` and moves the last remaining
workaround into ``_math.h``. This simplifies static builds with
``Modules/Setup`` and resolves symbol conflicts.

Co-authored-by: Mark Dickinson <mdickinson@enthought.com>
Co-authored-by: Brett Cannon <brett@python.org>
Signed-off-by: Christian Heimes <christian@python.org>
2021-10-25 01:25:27 -07:00
Eric Snow 17c61045c5
bpo-45506: Normalize _PyPathConfig.stdlib_dir when calculated. (#29040)
The recently added PyConfig.stdlib_dir was being set with ".." entries. When __file__ was added for from modules this caused a problem on out-of-tree builds. This PR fixes that by normalizing "stdlib_dir" when it is calculated in getpath.c.

https://bugs.python.org/issue45506
2021-10-22 17:20:03 -06:00
Christian Heimes f6e8b80d20
bpo-45571: use PY_CFLAGS_NODIST for shared Modules/Setup (GH-29161) 2021-10-22 20:02:54 +02:00
Christian Heimes ec93721e00
bpo-45570: Simplify setup macros for pyexpat (GH-29159)
* ``HAVE_EXPAT_CONFIG_H`` is not used by our code and not used by
  system-wide expat header files
* ``USE_PYEXPAT_CAPI`` is no longer used by our code
* ``XML_POOR_ENTROPY`` should be defined in expat_config.h

Signed-off-by: Christian Heimes <christian@python.org>
2021-10-22 09:28:23 -07:00
Christian Heimes 03e9f5dc75
bpo-43974: Move Py_BUILD_CORE_MODULE into module code (GH-29157)
setup.py no longer defines Py_BUILD_CORE_MODULE. Instead every
module defines the macro before #include "Python.h" unless
Py_BUILD_CORE_BUILTIN is already defined.

Py_BUILD_CORE_BUILTIN is defined for every module that is built by
Modules/Setup.

The PR also simplifies Modules/Setup. Makefile and makesetup
already define Py_BUILD_CORE_BUILTIN and include Modules/internal
for us.

Signed-off-by: Christian Heimes <christian@python.org>
2021-10-22 15:36:28 +02:00
Brett Cannon 01cf4fb9c1
bpo-45548: Make `Modules/Setup` easier to read (GH-29143) 2021-10-21 17:07:26 -07:00
Petr Viktorin 8a310dd5f4
bpo-45315: PyType_FromSpec: Copy spec->name and have the type own the memory for its name (GH-29103) 2021-10-21 11:46:20 +02:00
Brett Cannon dd86f63b55
bpo-45548: add some missing entries to `Modules/Setup` (GH-29115)
Also remove a duplicate entry for `_weakref`.
2021-10-20 17:34:49 -07:00
Zackery Spytz d8e1819251
bpo-41374: Remove obsolete exclusion of netinet/tcp.h on Cygwin (GH-21649) 2021-10-20 18:14:59 +01:00
Jeong YunWon d2cd5eef0c
bpo-45532: Replace 'default' with 'main' as default in sys.version (GH-29100) 2021-10-21 02:09:03 +09:00
Erlend Egeberg Aasland 09c04e7f0d
bpo-42064: Add module backref to `sqlite3` callback context (GH-28242) 2021-10-19 15:44:45 +02:00
Victor Stinner 52af0756b2
bpo-45434: Only exclude <stdlib.h> in Python 3.11 limited C API (GH-29027)
The Python 3.11 limited C API no longer includes stdlib.h, stdio.h,
string.h and errno.h.

* Exclude Py_MEMCPY() from Python 3.11 limited C API.
* xxlimited C extension is now built with Python 3.11 limited C API.
2021-10-19 12:10:22 +02:00
Victor Stinner 5f09bb021a
bpo-35134: Add Include/cpython/longobject.h (GH-29044)
Move Include/longobject.h non-limited API to a new
Include/cpython/longobject.h header file.

Move the following definitions to the internal C API:

* _PyLong_DigitValue
* _PyLong_FormatAdvancedWriter()
* _PyLong_FormatWriter()
2021-10-19 02:04:52 +02:00
Victor Stinner 547d26aa08
bpo-43760: Add PyThreadState_EnterTracing() (GH-28542)
Add PyThreadState_EnterTracing() and PyThreadState_LeaveTracing()
functions to the limited C API to suspend and resume tracing and
profiling.

Add an unit test on the PyThreadState C API to _testcapi.

Add also internal _PyThreadState_DisableTracing() and
_PyThreadState_ResetTracing().
2021-10-15 16:06:30 +02:00
Victor Stinner 354c35220d
bpo-45482: Rename namespaceobject.h to pycore_namespace.h (GH-28975)
Rename Include/namespaceobject.h to
Include/internal/pycore_namespace.h.

The _testmultiphase extension is now built with the
Py_BUILD_CORE_MODULE macro defined to access _PyNamespace_Type.

object.c: remove unused "pycore_context.h" include.
2021-10-15 15:21:21 +02:00
Victor Stinner 063abd931f
bpo-35081: Move interpreteridobject.h to Include/internal/ (GH-28969)
Move the interpreteridobject.h header file from Include/ to
Include/internal/. It only provides private functions.
2021-10-15 11:56:34 +02:00
Victor Stinner 8e5de40f90
bpo-35134: Move classobject.h to Include/cpython/ (GH-28968)
Move classobject.h, context.h, genobject.h and longintrepr.h header
files from Include/ to Include/cpython/.

Remove redundant "#ifndef Py_LIMITED_API" in context.h.

Remove explicit #include "longintrepr.h" in C files. It's not needed,
Python.h already includes it.
2021-10-15 09:46:29 +02:00
Victor Stinner 0a883a76cd
bpo-35134: Add Include/cpython/floatobject.h (GH-28957)
Split Include/floatobject.h into sub-files: add
Include/cpython/floatobject.h and
Include/internal/pycore_floatobject.h.
2021-10-14 23:41:06 +02:00
Victor Stinner 3cc56c828d
bpo-45439: Move _PyObject_VectorcallTstate() to pycore_call.h (GH-28893)
* Move _PyObject_VectorcallTstate() and _PyObject_FastCallTstate() to
  pycore_call.h (internal C API).
* Convert PyObject_CallOneArg(), PyObject_Vectorcall(),
  _PyObject_FastCall() and PyVectorcall_Function() static inline
  functions to regular functions.
* Add _PyVectorcall_FunctionInline() static inline function.
* PyObject_Vectorcall(), _PyObject_FastCall(), and
  PyObject_CallOneArg() now call _PyThreadState_GET() rather
  than PyThreadState_Get().
2021-10-14 21:53:04 +02:00
Serhiy Storchaka 39aa98346d
bpo-45467: Fix IncrementalDecoder and StreamReader in the "raw-unicode-escape" codec (GH-28944)
They support now splitting escape sequences between input chunks.

Add the third parameter "final" in codecs.raw_unicode_escape_decode().
It is True by default to match the former behavior.
2021-10-14 20:04:19 +03:00
Dong-hee Na d413c50363
no-issue: Make silence about warning '_POSIX_C_SOURCE redefined' (GH-28948) 2021-10-15 00:59:56 +09:00
Serhiy Storchaka c96d1546b1
bpo-45461: Fix IncrementalDecoder and StreamReader in the "unicode-escape" codec (GH-28939)
They support now splitting escape sequences between input chunks.

Add the third parameter "final" in codecs.unicode_escape_decode().
It is True by default to match the former behavior.
2021-10-14 13:17:00 +03:00
Victor Stinner aac29af678
bpo-45434: pyport.h no longer includes <stdlib.h> (GH-28914)
Include <stdlib.h> explicitly in C files.

Python.h includes <wchar.h>.
2021-10-13 19:25:53 +02:00
Victor Stinner bbe7497c5a
bpo-45434: Remove pystrhex.h header file (GH-28923)
Move Include/pystrhex.h to Include/internal/pycore_strhex.h.
The header file only contains private functions.

The following C extensions are now built with Py_BUILD_CORE_MODULE
macro defined to get access to the internal C API:

* _blake2
* _hashopenssl
* _md5
* _sha1
* _sha3
* _ssl
* binascii
2021-10-13 15:22:35 +02:00
Victor Stinner 97308dfcdc
bpo-45434: Move _Py_BEGIN_SUPPRESS_IPH to pycore_fileutils.h (GH-28922) 2021-10-13 15:03:35 +02:00
Victor Stinner 7cdc2a0f4b
pycore_pystate.h no longer redefines PyThreadState_GET() (GH-28921)
Redefining the PyThreadState_GET() macro in pycore_pystate.h is
useless since it doesn't affect files not including it. Either use
_PyThreadState_GET() directly, or don't use pycore_pystate.h internal
C API. For example, the _testcapi extension don't use the internal C
API, but use the public PyThreadState_Get() function instead.

Replace PyThreadState_Get() with _PyThreadState_GET(). The
_PyThreadState_GET() macro is more efficient than PyThreadState_Get()
and PyThreadState_GET() function calls which call fail with a fatal
Python error.

posixmodule.c and _ctypes extension now include <windows.h> before
pycore header files (like pycore_call.h).

_PyTraceback_Add() now uses _PyErr_Fetch()/_PyErr_Restore() instead
of PyErr_Fetch()/PyErr_Restore().

The _decimal and _xxsubinterpreters extensions are now built with the
Py_BUILD_CORE_MODULE macro defined to get access to the internal C
API.
2021-10-13 14:09:13 +02:00
Victor Stinner c63623a0a6
bpo-45434: bytearrayobject.h no longer includes <stdarg.h> (GH-28913)
bytearrayobject.h and _lzmamodule.c don't use va_list and so don't
need to include <stdarg.h>.
2021-10-13 04:37:55 +02:00
Erlend Egeberg Aasland cfb1df3b71
bpo-44991: Normalise function and collation callback naming (GH-28209) 2021-10-12 13:38:49 +02:00
Victor Stinner d943d19172
bpo-45439: Move _PyObject_CallNoArgs() to pycore_call.h (GH-28895)
* Move _PyObject_CallNoArgs() to pycore_call.h (internal C API).
* _ssl, _sqlite and _testcapi extensions now call the public
  PyObject_CallNoArgs() function, rather than _PyObject_CallNoArgs().
* _lsprof extension is now built with Py_BUILD_CORE_MODULE macro
  defined to get access to internal _PyObject_CallNoArgs().
2021-10-12 08:38:19 +02:00
Victor Stinner ce3489cfdb
bpo-45439: Rename _PyObject_CallNoArg() to _PyObject_CallNoArgs() (GH-28891)
Fix typo in the private _PyObject_CallNoArg() function name: rename
it to _PyObject_CallNoArgs() to be consistent with the public
function PyObject_CallNoArgs().
2021-10-12 00:42:23 +02:00
Dong-hee Na ab62051152
bpo-20028: Empty escapechar/quotechar is not allowed for csv.Dialect (GH-28833) 2021-10-11 20:08:15 +09:00
Serhiy Storchaka b454e8e4df
bpo-27580: Add support of null characters in the csv module. (GH-28808) 2021-10-09 19:17:43 +03:00
Dong-hee Na ec04db74e2
bpo-20028: Keep original exception when PyUnicode_GetLength return -1 (GH-28832) 2021-10-10 00:16:12 +09:00
Dong-hee Na 34bbc87b2d
bpo-20028: Improve error message of csv.Dialect when initializing (GH-28705) 2021-10-09 23:50:12 +09:00
Jeong YunWon 9f7a94fd66
bpo-45407: Remove outdated XXX comment from Struct___init___impl (GH-28805) 2021-10-08 15:45:11 +09:00
Matthias Reichl 392a898353
bpo-45262, asyncio: Fix cache of the running loop holder (GH-28796)
Prevent use-after-free of running loop holder via cache.
2021-10-08 00:46:49 +02:00
Erlend Egeberg Aasland 3f2c433da5
bpo-45041: Restore `sqlite3` executescript behaviour for `SELECT` queries (GH-28509)
* bpo-45041: Restore sqlite3 executescript behaviour for select queries

* Add regression test
2021-10-07 10:16:45 +01:00
Christian Clauss dd02a696e5
Fix typos in the Modules directory (GH-28761) 2021-10-07 01:34:42 -07:00
Ammar Askar db72e58ea5
bpo-29505: Add fuzzer for ast.literal_eval (GH-28777)
This supercedes https://github.com/python/cpython/pull/3437 and fuzzes the method we recommend for unsafe inputs, `ast.literal_eval`. This should exercise the tokenizer and parser.
2021-10-06 16:22:09 -07:00
Mark Shannon a7252f88d3
bpo-40116: Add insertion order bit-vector to dict values to allow dicts to share keys more freely. (GH-28520) 2021-10-06 13:19:53 +01:00
Hai Shi b9bb74871b
bpo-44050: Extension modules can share state when they don't support sub-interpreters. (GH-27794)
Automerge-Triggered-By: GH:encukou
2021-10-05 06:19:32 -07:00
Serhiy Storchaka 60b9e040c9
bpo-45355: Use sizeof(_Py_CODEUNIT) instead of literal 2 for the size of the code unit (GH-28711) 2021-10-03 21:22:42 +03:00
Serhiy Storchaka a5a56154f1
Remove trailing spaces. (GH-28706) 2021-10-03 16:58:14 +03:00
TAGAMI Yukihiro 0742abdc48
bpo-45329: Fix freed memory access in pyexpat.c (GH-28649) 2021-10-02 12:57:13 +03:00
AngstyDuck a450398933
bpo-44687: Ensure BufferedReader objects with unread buffers can peek even when the underlying file is closed (GH-28457) 2021-10-01 21:11:08 +01:00
Dong-hee Na 9eed75fde2
bpo-45332: Fix broken Decimal test and benchmark (GH-28680) 2021-10-01 22:49:46 +09:00
Julien Palard 9ce0f48e91
hashlib: Fix old message about unicode objects. (GH-28653) 2021-10-01 14:38:49 +02:00
Victor Stinner 833fdf126c
bpo-41710: Add private _PyDeadline_Get() function (GH-28674)
Add a private C API for deadlines: add _PyDeadline_Init() and
_PyDeadline_Get() functions.

* Add _PyTime_Add() and _PyTime_Mul() functions which compute t1+t2
  and t1*t2 and clamp the result on overflow.
* _PyTime_MulDiv() now uses _PyTime_Add() and _PyTime_Mul().
2021-10-01 13:29:25 +02:00
Victor Stinner 54957f16a6
bpo-41710: gc_collect_main() uses _PyTime_GetPerfCounter() (GH-28676)
If the DEBUG_STATS debug flag is set, gc_collect_main() now uses
_PyTime_GetPerfCounter() instead of _PyTime_GetMonotonicClock() to
measure the elapsed time.

On Windows, _PyTime_GetMonotonicClock() only has a resolution of 15.6
ms, whereas _PyTime_GetPerfCounter() is closer to a resolution of 100
ns.
2021-10-01 13:29:00 +02:00
Victor Stinner 37b8294d62
bpo-41710: PyThread_acquire_lock_timed() clamps the timout (GH-28643)
PyThread_acquire_lock_timed() now clamps the timeout into the
[_PyTime_MIN; _PyTime_MAX] range (_PyTime_t type) if it is too large,
rather than calling Py_FatalError() which aborts the process.

PyThread_acquire_lock_timed() no longer uses
MICROSECONDS_TO_TIMESPEC() to compute sem_timedwait() argument, but
_PyTime_GetSystemClock() and _PyTime_AsTimespec_truncate().

Fix _thread.TIMEOUT_MAX value on Windows: the maximum timeout is
0x7FFFFFFF milliseconds (around 24.9 days), not 0xFFFFFFFF
milliseconds (around 49.7 days).

Set PY_TIMEOUT_MAX to 0x7FFFFFFF milliseconds, rather than 0xFFFFFFFF
milliseconds.

Fix PY_TIMEOUT_MAX overflow test: replace (us >= PY_TIMEOUT_MAX) with
(us > PY_TIMEOUT_MAX).
2021-09-30 10:16:51 +02:00
Victor Stinner 09796f2f14
bpo-41710: Add _PyTime_AsTimespec_clamp() (GH-28629)
Add the _PyTime_AsTimespec_clamp() function: similar to
_PyTime_AsTimespec(), but clamp to _PyTime_t min/max and don't raise
an exception.

PyThread_acquire_lock_timed() now uses _PyTime_AsTimespec_clamp() to
remove the Py_UNREACHABLE() code path.

* Add _PyTime_AsTime_t() function.
* Add PY_TIME_T_MIN and PY_TIME_T_MAX constants.
* Replace _PyTime_AsTimeval_noraise() with _PyTime_AsTimeval_clamp().
* Add pytime_divide_round_up() function.
* Fix integer overflow in pytime_divide().
* Add pytime_divmod() function.
2021-09-30 02:11:41 +02:00
Victor Stinner 8d3e7eff09
bpo-43753: _operator.is_() uses Py_Is() (GH-28641) 2021-09-30 01:28:10 +02:00
Eric Snow 0c50b8c0b8
bpo-45211: Remember the stdlib dir during startup. (gh-28586)
During runtime startup we figure out the stdlib dir but currently throw that information away. This change preserves it and exposes it via PyConfig.stdlib_dir, _Py_GetStdlibDir(), and sys._stdlib_dir.

https://bugs.python.org/issue45211
2021-09-28 12:18:28 -06:00
Pablo Galindo Salgado 84975146a7
bpo-35606: Fix math.prod tests using 'start' as keyword parameter (GH-28595) 2021-09-28 13:32:43 +01:00
Eric Snow ae7839bbe8
bpo-45211: Move helpers from getpath.c to internal API. (gh-28550)
This accomplishes 2 things:

* consolidates some common code between getpath.c and getpathp.c
* makes the helpers available to code in other files

FWIW, the signature of the join_relfile() function (in fileutils.c) intentionally mirrors that of Windows' PathCchCombineEx().

Note that this change is mostly moving code around. No behavior is meant to change.

https://bugs.python.org/issue45211
2021-09-27 10:00:32 -06:00
Erlend Egeberg Aasland 7b88f63e1d
bpo-44958: Revert GH-27844 (GH-28574)
This reverts commit 050d103595, but keeps
the tests.
2021-09-26 22:24:19 +01:00
Victor Stinner 7834ff26cb
bpo-21302: Add nanosleep() implementation for time.sleep() in Unix (GH-28545)
Co-authored-by: Livius <egyszeregy@freemail.hu>
2021-09-25 14:36:26 +02:00
Antony Lee 6587fc60d4
bpo-44019: Implement operator.call(). (GH-27888)
Having `operator.call(obj, arg)` mean `type(obj).__call__(obj, arg)` is
consistent with the other dunder operators.  The semantics with `*args,
**kwargs` then follow naturally from the single-arg semantics.
2021-09-24 16:22:49 +01:00
Victor Stinner 58f8adfda3
bpo-21302: time.sleep() uses waitable timer on Windows (GH-28483)
On Windows, time.sleep() now uses a waitable timer which has a
resolution of 100 ns (10^-7 sec). Previously, it had a solution of 1
ms (10^-3 sec).

* On Windows, time.sleep() now calls PyErr_CheckSignals() before
  resetting the SIGINT event.
* Add _PyTime_As100Nanoseconds() function.
* Complete and update time.sleep() documentation.

Co-authored-by: Livius <egyszeregy@freemail.hu>
2021-09-22 16:09:30 +02:00
Mohamad Mansour 8f943ca257
[codemod] Fix non-matching bracket pairs (GH-28473)
Co-authored-by: Terry Jan Reedy <tjreedy@udel.edu>
Co-authored-by: Serhiy Storchaka <storchaka@gmail.com>
Co-authored-by: Łukasz Langa <lukasz@langa.pl>
2021-09-22 01:09:00 +02:00
Erlend Egeberg Aasland 050d103595
bpo-44958: Only reset `sqlite3` statements when needed (GH-27844) 2021-09-21 12:20:34 +01:00
Erlend Egeberg Aasland a6779715c4
bpo-45041: Simplify `sqlite3.Cursor.executescript()` (GH-28020) 2021-09-19 23:52:36 +01:00
Erlend Egeberg Aasland 771a546713
bpo-45040: Simplify sqlite3 transaction control functions (GH-28019) 2021-09-19 23:51:36 +01:00
Serhiy Storchaka f25f2e2e8c
Clean up initialization __class_getitem__ with Py_GenericAlias. (GH-28450)
The cast to PyCFunction is redundant. Overuse of redundant casts
can hide actual bugs.
2021-09-19 18:05:30 +03:00
Serhiy Storchaka 773319545b
bpo-45228: Fix stack buffer overflow in parsing J1939 address (GH-28404) 2021-09-17 11:46:19 +03:00
Victor Stinner b49263b698
bpo-21302: Add _PyTime_AsNanoseconds() (GH-28350)
Refactor pytime.c:

* Add pytime_from_nanoseconds() and pytime_as_nanoseconds(),
  and use explicitly these functions
* Add two empty lines between functions
* PEP 7: add braces { ... }
* C99: declare variables where they are set
* Rename private functions to lowercase
* Rename error_time_t_overflow() to pytime_time_t_overflow()
* Rename win_perf_counter_frequency() to py_win_perf_counter_frequency()
* py_get_monotonic_clock(): add an assertion to detect overflow when
  mach_absolute_time() unsigned uint64_t is casted to _PyTime_t
  (signed int64_t).

_testcapi: use _PyTime_FromNanoseconds().
2021-09-15 14:26:43 +02:00
Benjamin Peterson 024fda47d4
closes bpo-45190: Update Unicode data to version 14.0.0. (GH-28336) 2021-09-14 11:00:38 -07:00