Commit Graph

12631 Commits

Author SHA1 Message Date
Christian Heimes 25ecc040d0
bpo-45573: Introduce extension module flags in Makefile (GH-29594)
``configure`` now uses a standardized format to forward state, compiler
flags, and linker flags to ``Makefile``, ``setup.py``, and
``Modules/Setup``. ``makesetup`` use the new variables by default if a
module line does not contain any compiler or linker flags. ``setup.py``
has a new function ``addext()``.

For a module ``egg``, configure adds:

* ``MODULE_EGG`` with value yes, missing, disabled, or n/a
* ``MODULE_EGG_CFLAGS``
* ``MODULE_EGG_LDFLAGS``

``Makefile.pre.in`` may also provide ``MODULE_EGG_DEPS`` that lists
dependencies such as header files and static libs.

Signed-off-by: Christian Heimes <christian@python.org>
2021-11-18 09:18:44 +01:00
Dong-hee Na e002bbc6cc
bpo-45512: Simplify manage isolation level (GH-29562) 2021-11-17 21:47:02 +09:00
Pablo Galindo Salgado da20d7401d
bpo-45822: Respect PEP 263's coding cookies in the parser even if flags are not provided (GH-29582) 2021-11-16 12:30:47 -08:00
Erlend Egeberg Aasland 9d6215a54c
bpo-45126: Harden `sqlite3` connection initialisation (GH-28227) 2021-11-16 15:53:35 +01:00
Dong-hee Na 55868f1a33
bpo-45429: Support CREATE_WAITABLE_TIMER_HIGH_RESOLUTION if possible (GH-29203) 2021-11-16 22:41:20 +09:00
Mark Shannon b931077375
bpo-45753: Make recursion checks more efficient. (GH-29524)
* Uses recursion remaining, instead of recursion depth to speed up check against recursion limit.
2021-11-16 11:01:57 +00:00
Erlend Egeberg Aasland 822c3dcce3
bpo-45512: Raise exception if sqlite3.Connection.__init__ is called with bad isolation level (#29561)
* bpo-45512: Raise sqlite3.Connection.__init__ is called with bad isolation level

* Also explicitly test allowed isolation levels

* Use subTest for better error messages if something goes wrong

* Update Lib/test/test_sqlite3/test_dbapi.py

Co-authored-by: Dong-hee Na <donghee.na92@gmail.com>

Co-authored-by: Dong-hee Na <donghee.na92@gmail.com>
2021-11-15 21:55:38 +09:00
Erlend Egeberg Aasland b567b9d74b
bpo-45512: Simplify isolation_level handling in `sqlite3` (GH-29053) 2021-11-15 16:50:59 +09:00
Christian Heimes 464e6616be
bpo-45800: Move pyexpat build setup into configure (GH-29547)
Settings for :mod:`pyexpat` C extension are now detected by ``configure``.
The bundled ``expat`` library is built in ``Makefile``.

Signed-off-by: Christian Heimes <christian@python.org>
2021-11-14 10:02:24 +01:00
Christian Heimes 0486570f7b
bpo-45798: Move _decimal build setup into configure (GH-29541)
Settings for :mod:`decimal` internal C extension are now detected by
:program:`configure`. The bundled `libmpdec` library is built in
``Makefile``.

Signed-off-by: Christian Heimes <christian@python.org>
2021-11-13 13:56:16 +01:00
Christian Heimes 0a9f69539b
bpo-45747: Detect gdbm/dbm dependencies in configure (GH-29467)
Co-authored-by: Erlend Egeberg Aasland <erlend.aasland@innova.no>
2021-11-10 20:26:55 +01:00
Erlend Egeberg Aasland c1323d4b8c
bpo-45754: Use correct SQLite limit when checking statement length (GH-29489) 2021-11-10 18:46:11 +00:00
Irit Katriel 05fbd60147
bpo-45711: Use _PyErr_ClearExcState instead of setting only exc_value to NULL (GH-29404) 2021-11-10 16:57:14 +00:00
Christian Heimes cbab997efb
bpo-45723: Prepare support for autoconf 2.71 (GH-29441) 2021-11-08 19:31:14 +01:00
Christian Heimes 24af9a40a8
bpo-45743: Move __APPLE_USE_RFC_3542 into socketmodule.c (GH-29456) 2021-11-08 12:48:34 +01:00
180909 122ca4d73f
remove the repeat 'the' (#29455) 2021-11-08 10:19:54 +02:00
Christian Heimes e9594f6747
bpo-45731: Handle --enable-loadable-sqlite-extensions in configure (GH-29434) 2021-11-06 10:30:37 +01:00
Erlend Egeberg Aasland c273986711
bpo-45613: Set `sqlite3.threadsafety` dynamically (GH-29227)
Use the compile-time selected default SQLite threaded mode to set the
DB-API 2.0 attribute 'threadsafety'

Mappings:
  - SQLITE_THREADSAFE=0 => threadsafety=0
  - SQLITE_THREADSAFE=1 => threadsafety=3
  - SQLITE_THREADSAFE=2 => threadsafety=1
2021-11-03 21:01:37 +00:00
Erlend Egeberg Aasland 456e27ac0a
bpo-24139: Add support for SQLite extended result codes (GH-28076) 2021-11-02 23:49:38 +00:00
Erlend Egeberg Aasland 401272e6e6
bpo-42064: Adapt `sqlite3` to multi-phase init (PEP 489) (GH-29234) 2021-11-02 16:35:51 +01:00
Erlend Egeberg Aasland b6b38a8226
bpo-45243: Add support for setting/getting `sqlite3` connection limits (GH-28463) 2021-11-01 22:50:53 +00:00
Erlend Egeberg Aasland 39b4d5938c
bpo-45634: Don't combine error checks when adding `sqlite3` int constants (GH-29251) 2021-10-31 19:01:19 +09:00
Erlend Egeberg Aasland e2e62b3808
bpo-45581: Raise `MemoryError` in `sqlite3.connect` if SQLite signals memory error (GH-29171) 2021-10-29 22:21:58 +02:00
Christian Heimes f0150ac94a
bpo-45548: Some test modules must be built as shared libs (GH-29268)
Some test cases don't work when test modules are static extensions.

Add dependency on Modules/config.c to trigger a rebuild whenever a
module build type is changed.

``makesetup`` puts shared extensions into ``Modules/`` directory. Create
symlinks from pybuilddir so the extensions can be imported.

Note: It is not possible to use the content of pybuilddir.txt as a build
target. Makefile evaluates target variables in the first pass. The
pybuilddir.txt file does not exist at that point.
2021-10-29 17:49:57 +02:00
Tim Golden 7bddd96982
bpo-45621: Small changes to mmap (GH-29247)
* Small tidy-ups / comments
* Use randomized names when testing tagged mmaps to avoid any risk of parallel tests treading on each others' toes
2021-10-29 09:20:21 +01:00
Mark Shannon f291404a80
bpo-45637: Store the frame pointer in the cframe (GH-29267)
* Rename 'frame' to 'current_frame'
2021-10-28 13:59:11 +01:00
Christian Heimes 4c95fb4640
bpo-45548: Fix out-of-tree and Debian builds (GH-29263)
Add Modules subdirs to SRCDIRS to generate directories for out-of-tree
object files.

Debian wants ncurses lib. Works on Fedora, too.

Debian also needs pkg-config to detect correct flags.

Remove more outdated comments. Makefile now tracks header dependencies
-lintl is injected by configure when needed. Build _dbm with
gdbm-compat.

Group some modules by purpose. socket, select, and mmap work on Windows,
too.
2021-10-28 11:57:38 +02:00
Erlend Egeberg Aasland 8f24b7dbcb
bpo-42064: Convert `sqlite3` global state to module state (GH-29073) 2021-10-27 13:12:21 +02:00
Benjamin Peterson 19a6c41e56
Remove unused variables. (GH-29231) 2021-10-26 16:22:34 -07:00
Tim Golden aea5ecc458
bpo-40915: Fix mmap resize bugs on Windows (GH-29213)
(original patch by eryksun)

Correctly hand various failure modes when resizing an mmap on Windows:

* Resizing a pagefile-backed mmap now creates a new mmap and copies data
* Attempting to resize when another mapping is held on the same file raises an OSError
* Attempting to resize a nametagged mmap raises an OSError if another mapping is held with the same nametag
2021-10-26 22:56:43 +01:00
Christian Heimes b5ee79494b
bpo-45548: makesetup improvements (GH-29225)
* record which modules are build as shared extensions
* put object files in same directory as source files
* remove dependency on deleted _math.c

Signed-off-by: Christian Heimes <christian@python.org>
2021-10-26 10:58:43 +02:00
Christian Heimes 2b8677a3cd
bpo-45548: FreeBSD doesn't like auto vars in makesetup (GH-29216) 2021-10-25 20:52:55 +02:00
Christian Heimes ece916e92c
bpo-45548: Add missing extensions to Modules/Setup (GH-29199)
Adds _ctypes, _decimal, _multiprocessing, _posixshmem, _scproxy, _sqlite3,
and _uuid.

Use Makefile variables to build OpenSSL and TCL/TK modules.

Drop ``-lm`` from math lines. Extensions are always linked against libm.

Signed-off-by: Christian Heimes <christian@python.org>
2021-10-25 20:01:52 +02:00
Christian Heimes 81669d1b7b
bpo-45595: Make extensions depend on header files (GH-29198)
``setup.py`` and ``makesetup`` now track build dependencies on all Python
header files and module specific header files.

Signed-off-by: Christian Heimes <christian@python.org>
2021-10-25 20:01:03 +02:00
Christian Heimes fa26245a1c
bpo-45548: Remove _math.c workarounds for pre-C99 libm (GH-29179)
The :mod:`math` and :mod:`cmath` implementation now require a C99 compatible
``libm`` and no longer ship with workarounds for missing acosh, asinh,
expm1, and log1p functions.

The changeset also removes ``_math.c`` and moves the last remaining
workaround into ``_math.h``. This simplifies static builds with
``Modules/Setup`` and resolves symbol conflicts.

Co-authored-by: Mark Dickinson <mdickinson@enthought.com>
Co-authored-by: Brett Cannon <brett@python.org>
Signed-off-by: Christian Heimes <christian@python.org>
2021-10-25 01:25:27 -07:00
Eric Snow 17c61045c5
bpo-45506: Normalize _PyPathConfig.stdlib_dir when calculated. (#29040)
The recently added PyConfig.stdlib_dir was being set with ".." entries. When __file__ was added for from modules this caused a problem on out-of-tree builds. This PR fixes that by normalizing "stdlib_dir" when it is calculated in getpath.c.

https://bugs.python.org/issue45506
2021-10-22 17:20:03 -06:00
Christian Heimes f6e8b80d20
bpo-45571: use PY_CFLAGS_NODIST for shared Modules/Setup (GH-29161) 2021-10-22 20:02:54 +02:00
Christian Heimes ec93721e00
bpo-45570: Simplify setup macros for pyexpat (GH-29159)
* ``HAVE_EXPAT_CONFIG_H`` is not used by our code and not used by
  system-wide expat header files
* ``USE_PYEXPAT_CAPI`` is no longer used by our code
* ``XML_POOR_ENTROPY`` should be defined in expat_config.h

Signed-off-by: Christian Heimes <christian@python.org>
2021-10-22 09:28:23 -07:00
Christian Heimes 03e9f5dc75
bpo-43974: Move Py_BUILD_CORE_MODULE into module code (GH-29157)
setup.py no longer defines Py_BUILD_CORE_MODULE. Instead every
module defines the macro before #include "Python.h" unless
Py_BUILD_CORE_BUILTIN is already defined.

Py_BUILD_CORE_BUILTIN is defined for every module that is built by
Modules/Setup.

The PR also simplifies Modules/Setup. Makefile and makesetup
already define Py_BUILD_CORE_BUILTIN and include Modules/internal
for us.

Signed-off-by: Christian Heimes <christian@python.org>
2021-10-22 15:36:28 +02:00
Brett Cannon 01cf4fb9c1
bpo-45548: Make `Modules/Setup` easier to read (GH-29143) 2021-10-21 17:07:26 -07:00
Petr Viktorin 8a310dd5f4
bpo-45315: PyType_FromSpec: Copy spec->name and have the type own the memory for its name (GH-29103) 2021-10-21 11:46:20 +02:00
Brett Cannon dd86f63b55
bpo-45548: add some missing entries to `Modules/Setup` (GH-29115)
Also remove a duplicate entry for `_weakref`.
2021-10-20 17:34:49 -07:00
Zackery Spytz d8e1819251
bpo-41374: Remove obsolete exclusion of netinet/tcp.h on Cygwin (GH-21649) 2021-10-20 18:14:59 +01:00
Jeong YunWon d2cd5eef0c
bpo-45532: Replace 'default' with 'main' as default in sys.version (GH-29100) 2021-10-21 02:09:03 +09:00
Erlend Egeberg Aasland 09c04e7f0d
bpo-42064: Add module backref to `sqlite3` callback context (GH-28242) 2021-10-19 15:44:45 +02:00
Victor Stinner 52af0756b2
bpo-45434: Only exclude <stdlib.h> in Python 3.11 limited C API (GH-29027)
The Python 3.11 limited C API no longer includes stdlib.h, stdio.h,
string.h and errno.h.

* Exclude Py_MEMCPY() from Python 3.11 limited C API.
* xxlimited C extension is now built with Python 3.11 limited C API.
2021-10-19 12:10:22 +02:00
Victor Stinner 5f09bb021a
bpo-35134: Add Include/cpython/longobject.h (GH-29044)
Move Include/longobject.h non-limited API to a new
Include/cpython/longobject.h header file.

Move the following definitions to the internal C API:

* _PyLong_DigitValue
* _PyLong_FormatAdvancedWriter()
* _PyLong_FormatWriter()
2021-10-19 02:04:52 +02:00
Victor Stinner 547d26aa08
bpo-43760: Add PyThreadState_EnterTracing() (GH-28542)
Add PyThreadState_EnterTracing() and PyThreadState_LeaveTracing()
functions to the limited C API to suspend and resume tracing and
profiling.

Add an unit test on the PyThreadState C API to _testcapi.

Add also internal _PyThreadState_DisableTracing() and
_PyThreadState_ResetTracing().
2021-10-15 16:06:30 +02:00
Victor Stinner 354c35220d
bpo-45482: Rename namespaceobject.h to pycore_namespace.h (GH-28975)
Rename Include/namespaceobject.h to
Include/internal/pycore_namespace.h.

The _testmultiphase extension is now built with the
Py_BUILD_CORE_MODULE macro defined to access _PyNamespace_Type.

object.c: remove unused "pycore_context.h" include.
2021-10-15 15:21:21 +02:00
Victor Stinner 063abd931f
bpo-35081: Move interpreteridobject.h to Include/internal/ (GH-28969)
Move the interpreteridobject.h header file from Include/ to
Include/internal/. It only provides private functions.
2021-10-15 11:56:34 +02:00
Victor Stinner 8e5de40f90
bpo-35134: Move classobject.h to Include/cpython/ (GH-28968)
Move classobject.h, context.h, genobject.h and longintrepr.h header
files from Include/ to Include/cpython/.

Remove redundant "#ifndef Py_LIMITED_API" in context.h.

Remove explicit #include "longintrepr.h" in C files. It's not needed,
Python.h already includes it.
2021-10-15 09:46:29 +02:00
Victor Stinner 0a883a76cd
bpo-35134: Add Include/cpython/floatobject.h (GH-28957)
Split Include/floatobject.h into sub-files: add
Include/cpython/floatobject.h and
Include/internal/pycore_floatobject.h.
2021-10-14 23:41:06 +02:00
Victor Stinner 3cc56c828d
bpo-45439: Move _PyObject_VectorcallTstate() to pycore_call.h (GH-28893)
* Move _PyObject_VectorcallTstate() and _PyObject_FastCallTstate() to
  pycore_call.h (internal C API).
* Convert PyObject_CallOneArg(), PyObject_Vectorcall(),
  _PyObject_FastCall() and PyVectorcall_Function() static inline
  functions to regular functions.
* Add _PyVectorcall_FunctionInline() static inline function.
* PyObject_Vectorcall(), _PyObject_FastCall(), and
  PyObject_CallOneArg() now call _PyThreadState_GET() rather
  than PyThreadState_Get().
2021-10-14 21:53:04 +02:00
Serhiy Storchaka 39aa98346d
bpo-45467: Fix IncrementalDecoder and StreamReader in the "raw-unicode-escape" codec (GH-28944)
They support now splitting escape sequences between input chunks.

Add the third parameter "final" in codecs.raw_unicode_escape_decode().
It is True by default to match the former behavior.
2021-10-14 20:04:19 +03:00
Dong-hee Na d413c50363
no-issue: Make silence about warning '_POSIX_C_SOURCE redefined' (GH-28948) 2021-10-15 00:59:56 +09:00
Serhiy Storchaka c96d1546b1
bpo-45461: Fix IncrementalDecoder and StreamReader in the "unicode-escape" codec (GH-28939)
They support now splitting escape sequences between input chunks.

Add the third parameter "final" in codecs.unicode_escape_decode().
It is True by default to match the former behavior.
2021-10-14 13:17:00 +03:00
Victor Stinner aac29af678
bpo-45434: pyport.h no longer includes <stdlib.h> (GH-28914)
Include <stdlib.h> explicitly in C files.

Python.h includes <wchar.h>.
2021-10-13 19:25:53 +02:00
Victor Stinner bbe7497c5a
bpo-45434: Remove pystrhex.h header file (GH-28923)
Move Include/pystrhex.h to Include/internal/pycore_strhex.h.
The header file only contains private functions.

The following C extensions are now built with Py_BUILD_CORE_MODULE
macro defined to get access to the internal C API:

* _blake2
* _hashopenssl
* _md5
* _sha1
* _sha3
* _ssl
* binascii
2021-10-13 15:22:35 +02:00
Victor Stinner 97308dfcdc
bpo-45434: Move _Py_BEGIN_SUPPRESS_IPH to pycore_fileutils.h (GH-28922) 2021-10-13 15:03:35 +02:00
Victor Stinner 7cdc2a0f4b
pycore_pystate.h no longer redefines PyThreadState_GET() (GH-28921)
Redefining the PyThreadState_GET() macro in pycore_pystate.h is
useless since it doesn't affect files not including it. Either use
_PyThreadState_GET() directly, or don't use pycore_pystate.h internal
C API. For example, the _testcapi extension don't use the internal C
API, but use the public PyThreadState_Get() function instead.

Replace PyThreadState_Get() with _PyThreadState_GET(). The
_PyThreadState_GET() macro is more efficient than PyThreadState_Get()
and PyThreadState_GET() function calls which call fail with a fatal
Python error.

posixmodule.c and _ctypes extension now include <windows.h> before
pycore header files (like pycore_call.h).

_PyTraceback_Add() now uses _PyErr_Fetch()/_PyErr_Restore() instead
of PyErr_Fetch()/PyErr_Restore().

The _decimal and _xxsubinterpreters extensions are now built with the
Py_BUILD_CORE_MODULE macro defined to get access to the internal C
API.
2021-10-13 14:09:13 +02:00
Victor Stinner c63623a0a6
bpo-45434: bytearrayobject.h no longer includes <stdarg.h> (GH-28913)
bytearrayobject.h and _lzmamodule.c don't use va_list and so don't
need to include <stdarg.h>.
2021-10-13 04:37:55 +02:00
Erlend Egeberg Aasland cfb1df3b71
bpo-44991: Normalise function and collation callback naming (GH-28209) 2021-10-12 13:38:49 +02:00
Victor Stinner d943d19172
bpo-45439: Move _PyObject_CallNoArgs() to pycore_call.h (GH-28895)
* Move _PyObject_CallNoArgs() to pycore_call.h (internal C API).
* _ssl, _sqlite and _testcapi extensions now call the public
  PyObject_CallNoArgs() function, rather than _PyObject_CallNoArgs().
* _lsprof extension is now built with Py_BUILD_CORE_MODULE macro
  defined to get access to internal _PyObject_CallNoArgs().
2021-10-12 08:38:19 +02:00
Victor Stinner ce3489cfdb
bpo-45439: Rename _PyObject_CallNoArg() to _PyObject_CallNoArgs() (GH-28891)
Fix typo in the private _PyObject_CallNoArg() function name: rename
it to _PyObject_CallNoArgs() to be consistent with the public
function PyObject_CallNoArgs().
2021-10-12 00:42:23 +02:00
Dong-hee Na ab62051152
bpo-20028: Empty escapechar/quotechar is not allowed for csv.Dialect (GH-28833) 2021-10-11 20:08:15 +09:00
Serhiy Storchaka b454e8e4df
bpo-27580: Add support of null characters in the csv module. (GH-28808) 2021-10-09 19:17:43 +03:00
Dong-hee Na ec04db74e2
bpo-20028: Keep original exception when PyUnicode_GetLength return -1 (GH-28832) 2021-10-10 00:16:12 +09:00
Dong-hee Na 34bbc87b2d
bpo-20028: Improve error message of csv.Dialect when initializing (GH-28705) 2021-10-09 23:50:12 +09:00
Jeong YunWon 9f7a94fd66
bpo-45407: Remove outdated XXX comment from Struct___init___impl (GH-28805) 2021-10-08 15:45:11 +09:00
Matthias Reichl 392a898353
bpo-45262, asyncio: Fix cache of the running loop holder (GH-28796)
Prevent use-after-free of running loop holder via cache.
2021-10-08 00:46:49 +02:00
Erlend Egeberg Aasland 3f2c433da5
bpo-45041: Restore `sqlite3` executescript behaviour for `SELECT` queries (GH-28509)
* bpo-45041: Restore sqlite3 executescript behaviour for select queries

* Add regression test
2021-10-07 10:16:45 +01:00
Christian Clauss dd02a696e5
Fix typos in the Modules directory (GH-28761) 2021-10-07 01:34:42 -07:00
Ammar Askar db72e58ea5
bpo-29505: Add fuzzer for ast.literal_eval (GH-28777)
This supercedes https://github.com/python/cpython/pull/3437 and fuzzes the method we recommend for unsafe inputs, `ast.literal_eval`. This should exercise the tokenizer and parser.
2021-10-06 16:22:09 -07:00
Mark Shannon a7252f88d3
bpo-40116: Add insertion order bit-vector to dict values to allow dicts to share keys more freely. (GH-28520) 2021-10-06 13:19:53 +01:00
Hai Shi b9bb74871b
bpo-44050: Extension modules can share state when they don't support sub-interpreters. (GH-27794)
Automerge-Triggered-By: GH:encukou
2021-10-05 06:19:32 -07:00
Serhiy Storchaka 60b9e040c9
bpo-45355: Use sizeof(_Py_CODEUNIT) instead of literal 2 for the size of the code unit (GH-28711) 2021-10-03 21:22:42 +03:00
Serhiy Storchaka a5a56154f1
Remove trailing spaces. (GH-28706) 2021-10-03 16:58:14 +03:00
TAGAMI Yukihiro 0742abdc48
bpo-45329: Fix freed memory access in pyexpat.c (GH-28649) 2021-10-02 12:57:13 +03:00
AngstyDuck a450398933
bpo-44687: Ensure BufferedReader objects with unread buffers can peek even when the underlying file is closed (GH-28457) 2021-10-01 21:11:08 +01:00
Dong-hee Na 9eed75fde2
bpo-45332: Fix broken Decimal test and benchmark (GH-28680) 2021-10-01 22:49:46 +09:00
Julien Palard 9ce0f48e91
hashlib: Fix old message about unicode objects. (GH-28653) 2021-10-01 14:38:49 +02:00
Victor Stinner 833fdf126c
bpo-41710: Add private _PyDeadline_Get() function (GH-28674)
Add a private C API for deadlines: add _PyDeadline_Init() and
_PyDeadline_Get() functions.

* Add _PyTime_Add() and _PyTime_Mul() functions which compute t1+t2
  and t1*t2 and clamp the result on overflow.
* _PyTime_MulDiv() now uses _PyTime_Add() and _PyTime_Mul().
2021-10-01 13:29:25 +02:00
Victor Stinner 54957f16a6
bpo-41710: gc_collect_main() uses _PyTime_GetPerfCounter() (GH-28676)
If the DEBUG_STATS debug flag is set, gc_collect_main() now uses
_PyTime_GetPerfCounter() instead of _PyTime_GetMonotonicClock() to
measure the elapsed time.

On Windows, _PyTime_GetMonotonicClock() only has a resolution of 15.6
ms, whereas _PyTime_GetPerfCounter() is closer to a resolution of 100
ns.
2021-10-01 13:29:00 +02:00
Victor Stinner 37b8294d62
bpo-41710: PyThread_acquire_lock_timed() clamps the timout (GH-28643)
PyThread_acquire_lock_timed() now clamps the timeout into the
[_PyTime_MIN; _PyTime_MAX] range (_PyTime_t type) if it is too large,
rather than calling Py_FatalError() which aborts the process.

PyThread_acquire_lock_timed() no longer uses
MICROSECONDS_TO_TIMESPEC() to compute sem_timedwait() argument, but
_PyTime_GetSystemClock() and _PyTime_AsTimespec_truncate().

Fix _thread.TIMEOUT_MAX value on Windows: the maximum timeout is
0x7FFFFFFF milliseconds (around 24.9 days), not 0xFFFFFFFF
milliseconds (around 49.7 days).

Set PY_TIMEOUT_MAX to 0x7FFFFFFF milliseconds, rather than 0xFFFFFFFF
milliseconds.

Fix PY_TIMEOUT_MAX overflow test: replace (us >= PY_TIMEOUT_MAX) with
(us > PY_TIMEOUT_MAX).
2021-09-30 10:16:51 +02:00
Victor Stinner 09796f2f14
bpo-41710: Add _PyTime_AsTimespec_clamp() (GH-28629)
Add the _PyTime_AsTimespec_clamp() function: similar to
_PyTime_AsTimespec(), but clamp to _PyTime_t min/max and don't raise
an exception.

PyThread_acquire_lock_timed() now uses _PyTime_AsTimespec_clamp() to
remove the Py_UNREACHABLE() code path.

* Add _PyTime_AsTime_t() function.
* Add PY_TIME_T_MIN and PY_TIME_T_MAX constants.
* Replace _PyTime_AsTimeval_noraise() with _PyTime_AsTimeval_clamp().
* Add pytime_divide_round_up() function.
* Fix integer overflow in pytime_divide().
* Add pytime_divmod() function.
2021-09-30 02:11:41 +02:00
Victor Stinner 8d3e7eff09
bpo-43753: _operator.is_() uses Py_Is() (GH-28641) 2021-09-30 01:28:10 +02:00
Eric Snow 0c50b8c0b8
bpo-45211: Remember the stdlib dir during startup. (gh-28586)
During runtime startup we figure out the stdlib dir but currently throw that information away. This change preserves it and exposes it via PyConfig.stdlib_dir, _Py_GetStdlibDir(), and sys._stdlib_dir.

https://bugs.python.org/issue45211
2021-09-28 12:18:28 -06:00
Pablo Galindo Salgado 84975146a7
bpo-35606: Fix math.prod tests using 'start' as keyword parameter (GH-28595) 2021-09-28 13:32:43 +01:00
Eric Snow ae7839bbe8
bpo-45211: Move helpers from getpath.c to internal API. (gh-28550)
This accomplishes 2 things:

* consolidates some common code between getpath.c and getpathp.c
* makes the helpers available to code in other files

FWIW, the signature of the join_relfile() function (in fileutils.c) intentionally mirrors that of Windows' PathCchCombineEx().

Note that this change is mostly moving code around. No behavior is meant to change.

https://bugs.python.org/issue45211
2021-09-27 10:00:32 -06:00
Erlend Egeberg Aasland 7b88f63e1d
bpo-44958: Revert GH-27844 (GH-28574)
This reverts commit 050d103595, but keeps
the tests.
2021-09-26 22:24:19 +01:00
Victor Stinner 7834ff26cb
bpo-21302: Add nanosleep() implementation for time.sleep() in Unix (GH-28545)
Co-authored-by: Livius <egyszeregy@freemail.hu>
2021-09-25 14:36:26 +02:00
Antony Lee 6587fc60d4
bpo-44019: Implement operator.call(). (GH-27888)
Having `operator.call(obj, arg)` mean `type(obj).__call__(obj, arg)` is
consistent with the other dunder operators.  The semantics with `*args,
**kwargs` then follow naturally from the single-arg semantics.
2021-09-24 16:22:49 +01:00
Victor Stinner 58f8adfda3
bpo-21302: time.sleep() uses waitable timer on Windows (GH-28483)
On Windows, time.sleep() now uses a waitable timer which has a
resolution of 100 ns (10^-7 sec). Previously, it had a solution of 1
ms (10^-3 sec).

* On Windows, time.sleep() now calls PyErr_CheckSignals() before
  resetting the SIGINT event.
* Add _PyTime_As100Nanoseconds() function.
* Complete and update time.sleep() documentation.

Co-authored-by: Livius <egyszeregy@freemail.hu>
2021-09-22 16:09:30 +02:00
Mohamad Mansour 8f943ca257
[codemod] Fix non-matching bracket pairs (GH-28473)
Co-authored-by: Terry Jan Reedy <tjreedy@udel.edu>
Co-authored-by: Serhiy Storchaka <storchaka@gmail.com>
Co-authored-by: Łukasz Langa <lukasz@langa.pl>
2021-09-22 01:09:00 +02:00
Erlend Egeberg Aasland 050d103595
bpo-44958: Only reset `sqlite3` statements when needed (GH-27844) 2021-09-21 12:20:34 +01:00
Erlend Egeberg Aasland a6779715c4
bpo-45041: Simplify `sqlite3.Cursor.executescript()` (GH-28020) 2021-09-19 23:52:36 +01:00
Erlend Egeberg Aasland 771a546713
bpo-45040: Simplify sqlite3 transaction control functions (GH-28019) 2021-09-19 23:51:36 +01:00
Serhiy Storchaka f25f2e2e8c
Clean up initialization __class_getitem__ with Py_GenericAlias. (GH-28450)
The cast to PyCFunction is redundant. Overuse of redundant casts
can hide actual bugs.
2021-09-19 18:05:30 +03:00
Serhiy Storchaka 773319545b
bpo-45228: Fix stack buffer overflow in parsing J1939 address (GH-28404) 2021-09-17 11:46:19 +03:00
Victor Stinner b49263b698
bpo-21302: Add _PyTime_AsNanoseconds() (GH-28350)
Refactor pytime.c:

* Add pytime_from_nanoseconds() and pytime_as_nanoseconds(),
  and use explicitly these functions
* Add two empty lines between functions
* PEP 7: add braces { ... }
* C99: declare variables where they are set
* Rename private functions to lowercase
* Rename error_time_t_overflow() to pytime_time_t_overflow()
* Rename win_perf_counter_frequency() to py_win_perf_counter_frequency()
* py_get_monotonic_clock(): add an assertion to detect overflow when
  mach_absolute_time() unsigned uint64_t is casted to _PyTime_t
  (signed int64_t).

_testcapi: use _PyTime_FromNanoseconds().
2021-09-15 14:26:43 +02:00
Benjamin Peterson 024fda47d4
closes bpo-45190: Update Unicode data to version 14.0.0. (GH-28336) 2021-09-14 11:00:38 -07:00
Livius 85a4748118
bpo-21302: Add clock_nanosleep() implementation for time.sleep() (GH-28111)
In Unix operating systems, time.sleep() now uses the clock_nanosleep() function,
if available, which allows to sleep for an interval specified with nanosecond precision.

Co-authored-by: Victor Stinner <vstinner@python.org>
2021-09-13 14:37:38 +02:00
Erlend Egeberg Aasland c78d5ca380
bpo-45126: Fix ref. leak in `sqlite3.Connection.__init__` (GH-28231) 2021-09-12 21:27:42 +09:00
Serhiy Storchaka 92bf8691fb
bpo-43413: Fix handling keyword arguments in subclasses of some buitin classes (GH-26456)
* Constructors of subclasses of some buitin classes (e.g. tuple, list,
  frozenset) no longer accept arbitrary keyword arguments.
* Subclass of set can now define a __new__() method with additional
  keyword parameters without overriding also __init__().
2021-09-12 13:27:50 +03:00
Dong-hee Na 9abd07e596
bpo-44987: Speed up unicode normalization of ASCII strings (GH-28283) 2021-09-11 18:04:38 +03:00
David CARLIER 5f5b7d0c65
bpo-45163: Haiku build fix. (GH-28269)
linkage issues mainly for shared libs and missing system library,
also little nit into the signal extension as strsignal returns
a constant in this platform.
2021-09-11 11:46:22 +03:00
Henry-Joseph Audéoud 707137b863
bpo-40563: Support pathlike objects on dbm/shelve (GH-21849)
Co-authored-by: Hakan Çelik <hakancelik96@outlook.com>
2021-09-10 15:26:16 +03:00
Senthil Kumaran 794430700d
bpo-45067 - Verify the version of ncurses for extended color support feature usage. (GH-28260)
* issue45067 - Fix _curses compilation in CentOS 7. Verify the version of ncurses
for extended color support feature usage.

The function extended_color_content was introduced in 2017.  The ncurses-devel
package in CentOS 7 had a older version ncurses resulted in compilation error.
For compiling ncurses with extended color support, we verify the version of the
ncurses library.

Co-authored-by: blurb-it[bot] <43283697+blurb-it[bot]@users.noreply.github.com>
Co-authored-by: Serhiy Storchaka <storchaka@gmail.com>
2021-09-09 11:32:11 -07:00
Vincent Michel 06148b1870
bpo-44219: Release the GIL during isatty syscalls (GH-28250)
Release the GIL while performing isatty() system calls on arbitrary
file descriptors. In particular, this affects os.isatty(),
os.device_encoding() and io.TextIOWrapper. By extension,
io.open() in text mode is also affected.
2021-09-09 15:12:03 +02:00
Erlend Egeberg Aasland f235dd0784
bpo-38371: Remove deprecated `tkinter` split() method (GH-28237) 2021-09-08 23:02:19 +03:00
Victor Stinner cb15afcccf
bpo-39573: Py_TYPE becomes a static inline function (GH-28128)
Convert the Py_TYPE() and Py_SIZE() macros to static inline
functions. The Py_SET_TYPE() and Py_SET_SIZE() functions must now be
used to set an object type and size.
2021-09-08 11:59:13 +02:00
Matti Picus 34d82ce7cb
Remove documentation for non-existing socket class attributes (GH-28029)
The functions in question are available on the module-level only.
2021-09-07 20:09:11 +02:00
Stanisław Skonieczny 9dc363ee7c
bpo-45012: Release GIL around stat in os.scandir (GH-28085)
Releasing GIL allows other threads to continue
its work when os.scandir is fetching DirEntry.stat
info from file system.
2021-09-07 19:55:20 +02:00
Erlend Egeberg Aasland 979336de34
bpo-42064: Pass module state to trace, progress, and authorizer callbacks (GH-27940)
- add print-or-clear traceback helper
- add helpers to clear and visit saved contexts
- modify callbacks to use the new callback_context struct
2021-09-07 15:06:17 +02:00
Nikita Sobolev 8ca6b61e3f
bpo-45034: Fix how upper limit is formatted for `struct.pack("H", ...)` (GH-28178)
Co-authored-by: Mark Dickinson <dickinsm@gmail.com>
Co-authored-by: Serhiy Storchaka <storchaka@gmail.com>
2021-09-07 13:18:46 +01:00
Erlend Egeberg Aasland 0474d06008
bpo-44991: Normalise `sqlite3` callback naming (GH-28088)
- all callbacks are now named xxx_callback
- normalise callable naming in set_*() functions
- normalise context argument naming in callbacks

The sqlite code is being "touched" in bpo-42064 (and related issues);
this style change makes it easier to work with and review.
2021-09-07 13:43:44 +02:00
Erlend Egeberg Aasland 65c5756be9
Remove unused macros from Modules/_sqlite/microprotocols.h (GH-28171) 2021-09-05 12:59:30 +09:00
Victor Stinner 7974c30b9f
bpo-45094: Add Py_NO_INLINE macro (GH-28140)
* Rename _Py_NO_INLINE macro to Py_NO_INLINE: make it public and
  document it.
* Sort macros in the C API documentation.
2021-09-03 16:44:02 +02:00
Victor Stinner a1e15a7a60
bpo-45082: Cleanup ctypes.c_buffer alias (GH-28129)
* Remove commented deprecation of ctypes.c_buffer.
* Remove references to ctypes.c_string which doesn't exist.
* Remove StringTestCase: it only had skipped test methods.
2021-09-02 19:02:03 +02:00
Ruben Vorderman ea23e7820f
bpo-43613: Faster implementation of gzip.compress and gzip.decompress (GH-27941)
Co-authored-by: Łukasz Langa <lukasz@langa.pl>
2021-09-02 17:02:59 +02:00
Victor Stinner 19ba2122ac
bpo-37330: open() no longer accept 'U' in file mode (GH-28118)
open(), io.open(), codecs.open() and fileinput.FileInput no longer
accept "U" ("universal newline") in the file mode. This flag was
deprecated since Python 3.3.
2021-09-02 12:58:00 +02:00
Victor Stinner a806608705
bpo-45085: Remove the binhex module (GH-28117)
The binhex module, deprecated in Python 3.9, is now removed. The
following binascii functions, deprecated in Python 3.9, are now also
removed:

* a2b_hqx(), b2a_hqx();
* rlecode_hqx(), rledecode_hqx().

The binascii.crc_hqx() function remains available.
2021-09-02 12:10:08 +02:00
Serhiy Storchaka 3c65457156
bpo-45060: Get rid of few uses of the equality operators with None (GH-28087) 2021-08-31 16:59:52 +03:00
Petr Viktorin 01dea5f12b
bpo-42064: Offset arguments for PyObject_Vectorcall in the _sqlite module (GH-27931)
This allows e.g. methods to be called efficiently by providing
space for a "self" argument; see PY_VECTORCALL_ARGUMENTS_OFFSET docs.
2021-08-31 14:34:44 +02:00
Erlend Egeberg Aasland 001ef4600f
bpo-44991: Make GIL handling more explicit in `sqlite3` callbacks (GH-27934)
- acquire the GIL at the very start[1]
- release the GIL at the very end

[1] The trace callback performs a sanity check before acquiring the GIL

Automerge-Triggered-By: GH:encukou
2021-08-31 05:18:43 -07:00
Erlend Egeberg Aasland 86d8b46523
bpo-16379: expose SQLite error codes and error names in `sqlite3` (GH-27786) 2021-08-30 19:32:21 +01:00
Tobias Bergkvist 71853a7302
bpo-44689: ctypes.util.find_library() now finds macOS 11+ system libraries when built on older macOS systems (#27251)
Previously, when built on older macOS systems, `find_library` was not able to find macOS system libraries when running on Big Sur due to changes in how system libraries are stored.
2021-08-30 02:55:47 -04:00
Victor Stinner 3fc5d84046
bpo-44394: Update libexpat copy to 2.4.1 (GH-26945)
Update the vendored copy of libexpat to 2.4.1 (from 2.2.8) to get the
fix for the CVE-2013-0340 "Billion Laughs" vulnerability. This copy
is most used on Windows and macOS.

Co-authored-by: Łukasz Langa <lukasz@langa.pl>
2021-08-29 16:08:24 +02:00
Adam Dangoor 28db1f61f2
Fix typo in a comment in Modules/_ssl.c: s/validata/validate/ (GH-27993) 2021-08-27 13:38:24 +02:00
Soumendra Ganguly ae224bb566
bpo-41818: Add termios.tcgetwinsize(), termios.tcsetwinsize(). (GH-23686)
* Add termios.tcgetwinsize(), termios.tcsetwinsize(). Update docs.
* Add TIOCGSIZE support to termios.tcgetwinsize()
* Add TIOCSSIZE support to termios.tcsetwinsize()

Authored-by: Soumendra Ganguly <soumendraganguly@gmail.com>

* termios.tcgetwinsize() and termios.tcsetwinsize() should return/accept two-item tuples instead of lists.
* Refactor tcsetwinsize to share common code and accept any two item sequence, with overflow checking.

Co-authored-by: Gregory P. Smith <greg@krypto.org> [Google]
2021-08-26 19:56:26 -07:00
Erlend Egeberg Aasland a3c11cebf1
bpo-27334: Fix reference leak introduced by GH-26202 (GH-27942) 2021-08-25 15:57:54 +02:00
Erlend Egeberg Aasland 7ecd3425d4
bpo-27334: roll back transaction if sqlite3 context manager fails to commit (GH-26202)
Co-authored-by: Luca Citi
Co-authored-by: Berker Peksag <berker.peksag@gmail.com>
2021-08-25 11:59:42 +01:00
Erlend Egeberg Aasland 3df0fc89bc
bpo-44976: Lazy creation of sqlite3 result rows (GH-27884) 2021-08-25 11:28:47 +01:00
Pablo Galindo Salgado a24676bedc
Add tests for the C tokenizer and expose it as a private module (GH-27924) 2021-08-24 17:50:05 +01:00
Erlend Egeberg Aasland 9ed523159c
bpo-42064: Pass module state to `sqlite3` UDF callbacks (GH-27456)
- Establish common callback context struct
- Convert UDF callbacks to fetch module state from callback context
2021-08-24 05:24:09 -07:00
Serhiy Storchaka 4d68917386
bpo-44984: Rewrite test_null_strings in _testcapi (GH-27904)
Test also PyObject_Repr(NULL) and PyObject_Bytes(NULL).
2021-08-23 20:05:07 +02:00
Erlend Egeberg Aasland 6dd1cdb0cf
Remove unused UNKNOWN macros from cursor.h and prepare_protocol.h (GH-27885) 2021-08-22 22:23:45 +09:00
Erlend Egeberg Aasland 878e726701
bpo-44965: Early exit for non-DML statements in sqlite3.Cursor.executemany() (GH-27865) 2021-08-21 19:58:58 +01:00
Erlend Egeberg Aasland 243b6c3b8f
bpo-44079: Strip superfluous statement cache from sqlite3.Connection (GH-25998) 2021-08-19 00:37:53 +01:00
Hai Shi fcd651d16f
bpo-42035: Enhance test_get_type_name() of _testcapi (GH-27649) 2021-08-17 16:50:33 +02:00
Hai Shi 3e2c643ae0
bpo-42035: Add PyType_GetQualName() to get a type's qualified name. (GH-27551) 2021-08-17 15:39:34 +02:00
Ken Jin d84d2c4985
bpo-44914: Add tests for some invariants of tp_version_tag (GH-27774) 2021-08-16 20:18:36 +01:00
Irit Katriel 8ac0886091
bpo-44890: collect specialization stats if Py_DEBUG (GH-27731) 2021-08-12 12:15:06 +01:00
Serhiy Storchaka 058fb35b57
bpo-44854: Remove trailing whitespaces (GH-27689) 2021-08-09 21:32:54 +03:00
Serhiy Storchaka 0eec6276fd
bpo-44859: Improve error handling in sqlite3 and and raise more accurate exceptions. (GH-27654)
* MemoryError is now raised instead of sqlite3.Warning when
  memory is not enough for encoding a statement to UTF-8
  in Connection.__call__() and Cursor.execute().
* UnicodEncodeError is now raised instead of sqlite3.Warning when
  the statement contains surrogate characters
  in Connection.__call__() and Cursor.execute().
* TypeError is now raised instead of ValueError for non-string
  script argument in Cursor.executescript().
* ValueError is now raised for script containing the null
  character instead of truncating it in Cursor.executescript().
* Correctly handle exceptions raised when getting boolean value
  of the result of the progress handler.
* Add many tests covering different corner cases.

Co-authored-by: Erlend Egeberg Aasland <erlend.aasland@innova.no>
2021-08-08 08:49:44 +03:00
Ronald Oussoren 17c2316794
bpo-42971: Add errno.EQFULL (macOS) (GH-24419)
Co-authored-by: Łukasz Langa <lukasz@langa.pl>
2021-08-06 23:35:13 +02:00
Serhiy Storchaka 7d747f26e6
bpo-44839: Raise more specific errors in sqlite3 (GH-27613)
MemoryError raised in user-defined functions will now preserve
its type. OverflowError will now be converted to DataError.
Previously both were converted to OperationalError.
2021-08-06 21:28:47 +03:00
Erlend Egeberg Aasland 8f010dc920
bpo-44822: Don't truncate `str`s with embedded NULL chars returned by `sqlite3` UDF callbacks (GH-27588) 2021-08-05 10:22:08 +03:00
Victor Stinner c34fa2bb06
bpo-41117: Cleanup subtract_refs() (GH-27593)
subtract_refs() reuses the 'op' variable rather than calling the
FROM_GC() macro twice.

Issue reported by William Pickard.
2021-08-04 18:09:14 +02:00
Noah Kantrowitz be42c06bb0
Update URLs in comments and metadata to use HTTPS (GH-27458) 2021-07-30 15:54:46 +02:00
Erlend Egeberg Aasland 2d34664051
bpo-31746: Fix broken call in GH-27431 (GH-27464) 2021-07-29 21:47:23 +01:00
Erlend Egeberg Aasland 7e311e496b
bpo-31746: Prevent segfaults when sqlite3.Connection is uninitialised (GH-27431) 2021-07-29 20:45:32 +01:00
Irit Katriel ddd1c418c0
bpo-44725 : expose specialization stats in python (GH-27192) 2021-07-29 17:26:53 +01:00
Erlend Egeberg Aasland d542742128
bpo-42064: Optimise `sqlite3` state access, part 1 (GH-27273)
Prepare for module state:

- Add "get state by defining class" and "get state by module def" stubs
- Add AC defining class when needed
- Add state pointer to connection context
- Pass state as argument to utility functions

Automerge-Triggered-By: GH:encukou
2021-07-29 02:21:45 -07:00
Hai Shi a390ebea17
bpo-42035: Add a PyType_GetName() to get type's short name. (GH-23903) 2021-07-29 09:57:02 +02:00
Erlend Egeberg Aasland 5269c09145
bpo-44688: Remove ASCII limitation from `sqlite3` collation names (GH-27395) 2021-07-29 09:47:56 +02:00
Ray Donnelly 92b5dc780d
bpo-40263: Fixes an off-by-one error in _winapi_WaitForMultipleObjects_impl (GH-19501) 2021-07-28 20:58:05 +01:00
Geoffrey Thomas 196998e220
closes bpo-44751: Move crypt.h include from public header to _cryptmodule (GH-27394)
Automerge-Triggered-By: GH:benjaminp
2021-07-27 15:58:12 -07:00
Erlend Egeberg Aasland 890e22957d
bpo-42064: Migrate to `sqlite3_create_collation_v2` (GH-27156)
This implies that SQLite now takes care of destroying the callback
context (the PyObject callable it has been passed), so we can strip the
collation dict from the connection object.
2021-07-27 15:54:20 +02:00
Mariusz Felisiak 11749e2dc2
bpo-44740: Lowercase "internet" and "web" where appropriate. (#27378)
Co-authored-by: Łukasz Langa <lukasz@langa.pl>
2021-07-27 00:11:55 +02:00
Mark Shannon ae0a2b7562
bpo-44590: Lazily allocate frame objects (GH-27077)
* Convert "specials" array to InterpreterFrame struct, adding f_lasti, f_state and other non-debug FrameObject fields to it.

* Refactor, calls pushing the call to the interpreter upward toward _PyEval_Vector.

* Compute f_back when on thread stack, only filling in value when frame object outlives stack invocation.

* Move ownership of InterpreterFrame in generator from frame object to generator object.

* Do not create frame objects for Python calls.

* Do not create frame objects for generators.
2021-07-26 11:22:16 +01:00
Pablo Galindo Salgado 83d1430ee5
bpo-42854: Correctly use size_t for _ssl._SSLSocket.read and _ssl._SSLSocket.write (GH-27271) 2021-07-23 16:05:53 +01:00
Petr Viktorin a4760cc32d
bpo-42747: Remove Py_TPFLAGS_HAVE_AM_SEND and make Py_TPFLAGS_HAVE_VERSION_TAG no-op (GH-27260)
* Remove code that checks Py_TPFLAGS_HAVE_VERSION_TAG
    
    The field is always present in the type struct, as explained
    in the added comment.

* Remove Py_TPFLAGS_HAVE_AM_SEND
    
    The flag is not needed, and since it was added in 3.10 it can be removed now.
2021-07-23 06:21:11 -07:00
Yurii Karabas 96c4cbd96c
bpo-44353: Implement typing.NewType __call__ method in C (#27262)
Co-authored-by: Ken Jin <28750310+Fidget-Spinner@users.noreply.github.com>
Co-authored-by: Denis Laxalde <denis@laxalde.org>
2021-07-22 23:06:54 +02:00
Erlend Egeberg Aasland 4c0deb25ac
bpo-42064: Finalise establishing sqlite3 global state (GH-27155)
With this, all sqlite3 static globals have been moved to the global state.
There are a couple of global static strings left, but there should be no need for adding them to the state.

https://bugs.python.org/issue42064
2021-07-20 12:59:18 +02:00
Idan Moral 366fcbac18
bpo-44678: Separate error message for discontinuous padding in binascii.a2b_base64 strict mode (GH-27249)
* Renamed assertLeadingPadding function to match logic
* Added a separate error message for discontinuous padding
* Updated the tests for discontinuous padding
2021-07-19 15:42:19 -07:00
Idan Moral 35b98e38b6
bpo-43086: Add handling for out-of-spec data in a2b_base64 (GH-24402)
binascii.a2b_base64 gains a strict_mode= parameter. When enabled it will raise an
error on input that deviates from the base64 spec in any way.  The default remains
False for backward compatibility.

Code reviews and minor tweaks by: Gregory P. Smith <greg@krypto.org> [Google]
2021-07-18 17:45:19 -07:00
Erlend Egeberg Aasland 5007a4f23c
bpo-44641: Use vectorcall in sqlite3 collation callback (GH-27158) 2021-07-15 16:49:14 +01:00
Shane Harvey d59d7374a3
bpo-34932: Add socket.TCP_KEEPALIVE for macOS (GH-25079) 2021-07-14 23:53:15 +01:00
Erlend Egeberg Aasland 81b8c0a385
Fix docstring typo in sqlite3.Connection.executescript/sqlite3.Cursor.executescript (GH-27147)
Both `executescript` methods contain the same docstring typo:
_"Executes a multiple SQL statements at once."_ => _"Executes multiple SQL statements at once."_

Automerge-Triggered-By: GH:pablogsal
2021-07-14 14:54:37 -07:00
Erlend Egeberg Aasland 05162993fe
bpo-42064: Move `sqlite3` exceptions to global state, part 2 of 2 (GH-26884)
Automerge-Triggered-By: GH:encukou
2021-07-14 04:26:44 -07:00
Serhiy Storchaka f572cbf1fa
bpo-44608: Fix memory leak in _tkinter._flatten() (GH-27107)
if it is called with a sequence or set, but not list or tuple.
2021-07-14 08:19:18 +03:00
T. Wouters 0093876328
bpo-44630: Fix assertion errors in csv module (GH-27127)
Fix incorrect handling of exceptions when interpreting dialect objects in
the csv module. Not clearing exceptions between calls to
PyObject_GetAttrString() causes assertion failures in pydebug mode (or with
assertions enabled).

Add a minimal test that would've caught this (passing None as dialect, or
any object that isn't a csv.Dialect subclass, which the csv module allows
and caters to, even though it is not documented.) In pydebug mode, the test
triggers the assertion failure in the old code.

Contributed-By: T. Wouters [Google]
2021-07-13 15:56:45 -07:00
Filipe Laíns e14d5ae544
bpo-29753: revert 0d7ad9f (GH-19850) (GH-27085)
This reverts commit 0d7ad9fb38 as it has a regression.

See https://github.com/python/cpython/pull/19850#issuecomment-869410686
2021-07-11 17:43:50 +01:00
Srinivas Reddy Thatiparthy (శ్రీనివాస్ రెడ్డి తాటిపర్తి) 7b21108445
Remove irrelevant comment which was added in 2a70a3a (GH-27044) 2021-07-08 21:57:25 -07:00
Steve Dower bbf2fb6c7a
bpo-44582: Accelerate mimetypes.init on Windows with a native accelerator (GH-27059) 2021-07-08 16:48:42 +01:00
Rupert Tombs 6bd3ecfc27
bpo-44558: Match countOf `is`/`==` treatment to c (GH-27007) 2021-07-07 22:28:09 +09:00
Serhiy Storchaka f64de53ff0
bpo-44563: Fix error handling in tee.fromiterable() (GH-27020)
In debug build failed tee.fromiterable() corrupted the linked list of all GC objects.
2021-07-05 23:19:35 +01:00
Ma Lin a9a69bb3ea
bpo-41486: zlib uses an UINT32_MAX sliding window for the output buffer (GH-26143)
* zlib uses an UINT32_MAX sliding window for the output buffer

These funtions have an initial output buffer size parameter:
- zlib.decompress(data, /, wbits=MAX_WBITS, bufsize=DEF_BUF_SIZE)
- zlib.Decompress.flush([length])

If the initial size > UINT32_MAX, use an UINT32_MAX sliding window, instead of clamping to UINT32_MAX.
Speed up when (the initial size == the actual size).

This fixes a memory consumption and copying performance regression in earlier 3.10 beta releases if someone used an output buffer larger than 4GiB with zlib.decompress.

Reviewed-by: Gregory P. Smith
2021-07-04 18:10:44 -07:00
Victor Stinner 48e3a1d95a
bpo-44434: Remove useless calls to PyThread_exit_thread() (GH-26943)
Remove useless calls to PyThread_exit_thread() in two unit tests of
_testcapi and _testembed modules.

Co-authored-by: Alexey Izbyshev <izbyshev@ispras.ru>
2021-06-29 02:03:30 +02:00
Brandt Bucher ca2009d72a
bpo-43977: Properly update the tp_flags of existing subclasses when their parents are registered (GH-26864) 2021-06-25 08:20:43 -07:00
Samuel Marks 19459f8ce6
bpo-44321: Adds `os.EX_OK` for Windows (GH-26559) 2021-06-24 17:45:18 +01:00
Erlend Egeberg Aasland b19f455339
bpo-44491: Allow clearing the sqlite3 authoriser callback (GH-26863) 2021-06-24 15:35:57 +01:00
Erlend Egeberg Aasland a50e28377b
bpo-42064: Move `sqlite3` exceptions to global state, part 1 of 2 (GH-26745)
Also adds a test to verify the (borrowed) exceptions in `sqlite3.Connection`.
2021-06-23 05:56:40 -07:00
Erlend Egeberg Aasland 019ad62afd
bpo-42064: Remove stale extern declarations in `sqlite3` headers (GH-26840) 2021-06-23 21:06:53 +09:00
Dennis Sweeney d1ae57027f
bpo-41621: Document defaultdict's default_factory parameter (GH-21945)
It defaults to None and is positional only.
2021-06-22 10:19:24 -04:00
Victor Stinner 45a78f906d
bpo-44434: Don't call PyThread_exit_thread() explicitly (GH-26758)
_thread.start_new_thread() no longer calls PyThread_exit_thread()
explicitly at the thread exit, the call was redundant.

On Linux with the glibc, pthread_cancel() loads dynamically the
libgcc_s.so.1 library. dlopen() can fail if there is no more
available file descriptor to open the file. In this case, the process
aborts with the error message:

"libgcc_s.so.1 must be installed for pthread_cancel to work"

pthread_cancel() unwinds back to the thread's wrapping function that
calls the thread entry point.

The unwind function is dynamically loaded from the libgcc_s library
since it is tightly coupled to the C compiler (GCC). The unwinder
depends on DWARF, the compiler generates DWARF, so the unwinder
belongs to the compiler.

Thanks Florian Weimer and Carlos O'Donell for their help on
investigating this issue.
2021-06-21 13:16:18 +02:00
Georg Sauthoff a317778fd5
bpo-44077: Expose IP_RECVTOS in the socket module (GH-25992)
Co-authored-by: Pablo Galindo <pablogsal@gmail.com>
2021-06-20 21:08:07 +01:00
Erlend Egeberg Aasland 7d0a47e1af
bpo-44087: Disallow instantiation of sqlite3.Statement (GH-26567) 2021-06-20 20:24:32 +01:00
Erlend Egeberg Aasland 185ecdc146
bpo-40956: Convert sqlite3.connect and sqlite3.Connection.__init__ to AC (GH-24421) 2021-06-20 20:24:00 +01:00
Erlend Egeberg Aasland 00710e6346
bpo-43908: Make heap types converted during 3.10 alpha immutable (GH-26351)
* Make functools types immutable

* Multibyte codec types are now immutable

* pyexpat.xmlparser is now immutable

* array.arrayiterator is now immutable

* _thread types are now immutable

* _csv types are now immutable

* _queue.SimpleQueue is now immutable

* mmap.mmap is now immutable

* unicodedata.UCD is now immutable

* sqlite3 types are now immutable

* _lsprof.Profiler is now immutable

* _overlapped.Overlapped is now immutable

* _operator types are now immutable

* winapi__overlapped.Overlapped is now immutable

* _lzma types are now immutable

* _bz2 types are now immutable

* _dbm.dbm and _gdbm.gdbm are now immutable
2021-06-17 11:06:09 +01:00
Joe c544393b89
bpo-44389: Fix typo in ssl deprecation warning message (GH-26754)
`ssl.SSL_NO_TLS` should be `ssl.OP_NO_TLS`.
2021-06-17 02:39:57 -07:00
Victor Stinner 1cd3d859a4
bpo-42972: _thread.RLock implements the GH protocol (GH-26734)
The _thread.RLock type now fully implement the GC protocol: add a
traverse function and the Py_TPFLAGS_HAVE_GC flag.
2021-06-15 15:09:24 +02:00
Erlend Egeberg Aasland 10a5c806d4
bpo-42064: Move sqlite3 types to global state (GH-26537)
* Move connection type to global state
* Move cursor type to global state
* Move prepare protocol type to global state
* Move row type to global state
* Move statement type to global state
* ADD_TYPE takes a pointer
* pysqlite_get_state is now static inline
2021-06-15 14:47:34 +02:00
Christian Heimes bf527277d4
bpo-44389: Fix deprecation of OP_NO_TLSv1_3 (GH-26700)
Signed-off-by: Christian Heimes <christian@python.org>
2021-06-13 13:46:07 +02:00
Erlend Egeberg Aasland cb7230c7a7
bpo-44389: Remove duplicate SSL_OP_NO_TLSv1_2 flag (GH-26680) 2021-06-12 17:17:58 -03:00
Serhiy Storchaka be8b631b7a
Add more const modifiers. (GH-26691) 2021-06-12 16:11:59 +03:00
Mark Dickinson 4a42cebf6d
bpo-44339: Fix math.pow corner case to comply with IEEE 754 (GH-26606)
Change the behaviour of `math.pow(0.0, -math.inf)` and `math.pow(-0.0, -math.inf)` to return positive infinity instead of raising `ValueError`. This makes `math.pow` consistent with the built-in `pow` (and the `**` operator) for this particular special case, and brings the `math.pow` special-case handling into compliance with IEEE 754.
2021-06-12 10:23:02 +01:00
Christian Heimes e26014f1c4
bpo-44362: ssl: improve deprecation warnings and docs (GH-26646)
Signed-off-by: Christian Heimes <christian@python.org>
2021-06-11 09:15:48 +02:00
Ajith Ramachandran ac867f10b4
bpo-44357:Add `math.cbrt()` function: Cube Root (GH-26622)
* Add math.cbrt() function: Cube Root

Co-authored-by: blurb-it[bot] <43283697+blurb-it[bot]@users.noreply.github.com>
Co-authored-by: Mark Dickinson <dickinsm@gmail.com>
2021-06-10 17:42:09 +01:00
Mark Shannon 31aa0dbff4
bpo-44363: Get test_capi passing with address sanitizer (GH-26639) 2021-06-10 12:37:22 +01:00
Erlend Egeberg Aasland 1c02655fb0
bpo-44329: Refactor sqlite3 statement creation (GH-26566)
Call SQLite API's first, and return early in case of error. At the end,
allocate the object and initialise members. We now avoid unneeded
alloc/dealloc's in case the statement creation fails.
2021-06-08 16:00:56 +01:00
Pablo Galindo 781dc76577
Fix compiler errors for unused variables (GH-26601) 2021-06-08 13:16:24 +01:00
Pablo Galindo 6d518bb3a1
bpo-44348: Revert "bpo-39573: Py_TYPE becomes a static inline function (GH-26493)" (GH-26596)
This reverts commit f3fa63ec75 as is
causing crashes in some Windows tests in the buildbots.
2021-06-08 12:24:40 +01:00
Erlend Egeberg Aasland 0d12f24552
bpo-44326: Remove unused members from pysqlite_Statement (GH-26564)
* Remove unused db member of pysqlite_Statement

* Remove unused sql method from statement object
2021-06-06 23:12:07 +01:00
Erlend Egeberg Aasland 505624e917
bpo-44327: Remove unused members from pysqlite_Connection (GH-26565)
* Remove timeout_started

* Remove timeout member
2021-06-06 23:11:44 +01:00
hrchu 18e9edb7b3
Update bisect docstrings (GH-26548) 2021-06-06 11:22:48 -07:00
Pablo Galindo 6e3b7cf3af
bpo-44304: Ensure the sqlite3 destructor callback is always called with the GIL held (GH-26551) 2021-06-05 23:41:11 +01:00
Pablo Galindo fa106a685c
bpo-44304: Fix crash in the sqlite3 module when the GC clears Statement objects (GH-26545) 2021-06-05 03:50:39 +01:00
Kazantcev Andrey 2780df4781
Align comment for better readability. (GH-26192) 2021-06-04 18:41:23 -03:00
Erlend Egeberg Aasland 7459208de1
bpo-44315: Remove unused connection argument from pysqlite_step() (GH-26535) 2021-06-04 21:42:20 +01:00
Erlend Egeberg Aasland 006fd869e4
bpo-43853: Handle sqlite3_value_text() errors (GH-25422) 2021-06-04 19:34:00 +01:00
Erlend Egeberg Aasland 3446516ffa
bpo-44042: Optimize sqlite3 begin transaction (GH-25908) 2021-06-03 22:24:25 +01:00
Erlend Egeberg Aasland f461a7fc3f
bpo-42862: Use functools.lru_cache iso. _sqlite.Cache in sqlite3 module (GH-24203)
Co-authored-by: Pablo Galindo <Pablogsal@gmail.com>
2021-06-03 20:59:26 +01:00
Victor Stinner f3fa63ec75
bpo-39573: Py_TYPE becomes a static inline function (GH-26493)
Convert the Py_TYPE() and Py_SIZE() macros to static inline
functions. The Py_SET_TYPE() and Py_SET_SIZE() functions must now be
used to set an object type and size.
2021-06-03 18:42:59 +02:00
Erlend Egeberg Aasland d88b47b5a3
bpo-42213: Remove redundant cyclic GC hack in sqlite3 (GH-26517)
The sqlite3 module now fully implements the GC protocol, so there's no
need for this workaround anymore.

- Add and use managed resource helper for connections using TESTFN
- Sort test imports
- Split open-tests into their own test case

Automerge-Triggered-By: GH:vstinner
2021-06-03 09:38:19 -07:00
Erlend Egeberg Aasland 82ad22a97d
bpo-42213: Check connection in sqlite3.Connection.__enter__ (GH-26512)
Try to harden connection close:

- add tests that exercise stuff against a closed database
- add wrapper for sqlite3_close_v2()
- check connection on __enter__
- explicitly free pending statements before close()
- sqlite3_close_v2() always returns SQLITE_OK
2021-06-03 17:53:47 +02:00
stratakis bdb56902a3
bpo-44285: getpath.c: Assert that env_file is NULL during an error check (GH-26486)
This was flagged by a static analyzer, but the logic of why this is NULL on error is hard to follow for humans as well.
2021-06-02 16:54:33 +02:00
Erlend Egeberg Aasland fbf25b8c0d
bpo-44165: pysqlite_statement_create now returns a Py object, not an int (GH-26484)
GH-26206 was broken by GH-26475.
2021-06-02 14:22:15 +01:00
Erlend Egeberg Aasland a384b6c040
bpo-44165: Optimise sqlite3 statement preparation by passing string size (GH-26206) 2021-06-02 13:26:06 +01:00
Erlend Egeberg Aasland fffa0f92ad
bpo-42972: Track sqlite3 statement objects (GH-26475)
Allocate and track statement objects in pysqlite_statement_create.

By allocating and tracking creation of statement object in
pysqlite_statement_create(), the caller does not need to worry about GC
syncronization, and eliminates the possibility of getting a badly
created object. All related fault handling is moved to
pysqlite_statement_create().

Co-authored-by: Victor Stinner <vstinner@python.org>
2021-06-01 12:47:37 +02:00
Serhiy Storchaka a6a2065881
bpo-44260: Do not read system entropy without need in Random() (GH-26455) 2021-05-31 23:24:20 +03:00
Victor Stinner 142e5c5445
bpo-44263: Fix _decimal and _testcapi GC protocol (GH-26464)
* _testcapi.heapgctype: implement a traverse function since the type
  is defined with Py_TPFLAGS_HAVE_GC.
* _decimal: PyDecSignalDictMixin_Type is no longer defined with
  Py_TPFLAGS_HAVE_GC since it has no traverse function.
2021-05-31 13:10:31 +02:00
Erlend Egeberg Aasland d1124b09e8
bpo-42972: Fix sqlite3 traverse/clear functions (GH-26452) 2021-05-31 10:24:56 +02:00
Hai Shi 4b20f2574d
bpo-42972: Fully implement GC protocol for xxlimited (GH-26451) 2021-05-31 09:51:19 +02:00
Pablo Galindo 8b4312b909
bpo-44252: Correctly implement gc support for SSLError objects (GH-26439) 2021-05-29 00:29:52 +01:00
Ken Jin 490b638e63
bpo-42972: Fix GC assertion error in _winapi by untracking Overlapped earlier (GH(26429) 2021-05-28 18:29:19 +02:00
Ken Jin 0fa282c55f
bpo-42972: Fully support GC for _winapi.Overlapped (GH-26381) 2021-05-28 11:06:46 +02:00
Erlend Egeberg Aasland 3f8d332527
bpo-42972: Fully implement GC protocol for functools LRU cache (GH-26423) 2021-05-28 11:02:42 +02:00
Erlend Egeberg Aasland 8994e9c2cd
bpo-42972: Fully implement GC protocol for functools keywrapper and partial types (GH-26363) 2021-05-28 10:41:16 +02:00
Inada Naoki 28be3191a9
bpo-44256: Do not expose _functools._list_elem_type (GH-26416)
It is internal use only type.
2021-05-28 14:09:32 +09:00
Erlend Egeberg Aasland f4b70c22c8
bpo-42972: Fully support GC protocol for _operator heap types (GH-26371) 2021-05-27 22:59:07 +02:00
Erlend Egeberg Aasland fba42d1188
bpo-42972: Fully implement GC protocol for re types (GH-26368) 2021-05-27 19:23:07 +02:00
Erlend Egeberg Aasland 318adeba78
bpo-42972: Fully support GC for mmap heap types (GH-26373) 2021-05-27 17:53:59 +02:00
Erlend Egeberg Aasland 4d7f8f9f7f
bpo-42972: Fully support GC protocol for _queue.SimpleQueue (GH-26372) 2021-05-27 17:50:12 +02:00
Erlend Egeberg Aasland dcb8786a98
bpo-42972: Fully implement GC protocol for ssl heap types (GH-26370) 2021-05-27 09:50:09 +02:00
Erlend Egeberg Aasland 6ef5ba391d
bpo-42972: Fully support GC for hashlib heap types (GH-26374) 2021-05-27 09:48:19 +02:00
Erlend Egeberg Aasland 59af59c2df
bpo-42972: Fully support GC for pyexpat, unicodedata, and dbm/gdbm heap types (GH-26376)
* bpo-42972: pyexpat
* bpo-42972: unicodedata
* bpo-42972: dbm/gdbm
2021-05-27 17:29:00 +10:00
Victor Stinner 3e7ee02327
bpo-42161: mathmodule.c: move _PyLong_GetOne() loop invariant (GH-26391)
Move _PyLong_GetZero() and _PyLong_GetOne() loop invariants outside
loops in functions:

* math.comb()
* math.gcd()
* math.lcm()
* math.perm()
2021-05-27 00:51:07 +02:00
Gabriele N. Tornetta 90a6c07cb2
bpo-43879: Add native_thread_id field to PyThreadState (GH-25458) 2021-05-26 16:40:14 +02:00
Erlend Egeberg Aasland bd404ccac0
bpo-42972: Fully implement GC protocol for arraymodule types (GH-26114) 2021-05-25 19:26:44 +01:00
Erlend Egeberg Aasland d3c277a59c
bpo-42972: Fully implement GC protocol for sqlite3 heap types (GH-26104) 2021-05-25 18:43:56 +01:00
Zackery Spytz 6cc8ac9499
bpo-40736: Improve the error message for re.search() TypeError (GH-23312)
Include the invalid type in the error message.
2021-05-21 22:02:42 +01:00
Pablo Galindo 95d04710c5
Fix compiler warning for misleading guarding in the tkinter (GH-26244)
The newest gcc emmits this warning:

```
/Modules/_tkinter.c:272:9: warning: this ‘if’ clause does not guard... [-Wmisleading-indentation]
  272 |         if(tcl_lock)PyThread_acquire_lock(tcl_lock, 1); tcl_tstate = tstate; }
      |         ^~
/Modules/_tkinter.c:2869:5: note: in expansion of macro ‘LEAVE_PYTHON’
 2869 |     LEAVE_PYTHON
      |     ^~~~~~~~~~~~
/Modules/_tkinter.c:243:5: note: ...this statement, but the latter is misleadingly indented as if it were guarded by the ‘if’
  243 |     (*(PyThreadState**)Tcl_GetThreadData(&state_key, sizeof(PyThreadState*)))
      |     ^
/Modules/_tkinter.c:272:57: note: in expansion of macro ‘tcl_tstate’
  272 |         if(tcl_lock)PyThread_acquire_lock(tcl_lock, 1); tcl_tstate = tstate; }
      |                                                         ^~~~~~~~~~
/Modules/_tkinter.c:2869:5: note: in expansion of macro ‘LEAVE_PYTHON’
 2869 |     LEAVE_PYTHON

```

that's because the macro packs together two statements at the same level
as the "if". The warning is misleading but is very noisy so it makes
sense to fix it.
2021-05-19 19:10:23 +01:00
Pablo Galindo be93f81e58
Fix compiler warning in the xml module (GH-26245)
The newest version of gcc complains about passing un-initialized arrays
as constant pointers:

```
/Modules/expat/xmltok_ns.c: In function ‘findEncodingNS’:
/Modules/expat/xmltok.h:272:10: warning: ‘buf’ may be used uninitialized [-Wmaybe-uninitialized]
  272 |   (((enc)->utf8Convert)(enc, fromP, fromLim, toP, toLim))
      |   ~~~~~~~^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
/Modules/expat/xmltok_ns.c:95:3: note: in expansion of macro ‘XmlUtf8Convert’
   95 |   XmlUtf8Convert(enc, &ptr, end, &p, p + ENCODING_MAX - 1);
      |   ^~~~~~~~~~~~~~
/Modules/expat/xmltok.h:272:10: note: by argument 5 of type ‘const char *’ to ‘enum XML_Convert_Result(const ENCODING *, const char **, const char *, char **, const char *)’ {aka ‘enum XML_Convert_Result(const struct encoding *, const char **, const char *, char **, const char *)’}
  272 |   (((enc)->utf8Convert)(enc, fromP, fromLim, toP, toLim))
      |   ~~~~~~~^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
/Modules/expat/xmltok_ns.c:95:3: note: in expansion of macro ‘XmlUtf8Convert’
   95 |   XmlUtf8Convert(enc, &ptr, end, &p, p + ENCODING_MAX - 1);
      |   ^~~~~~~~~~~~~~
In file included from /Modules/expat/xmltok.c:1657:
/Modules/expat/xmltok_ns.c:92:8: note: ‘buf’ declared here
   92 |   char buf[ENCODING_MAX];

```
2021-05-19 19:05:40 +01:00
Gregory P. Smith c10392e7dd
bpo-44145: Release the GIL around HMAC_Update. (GH-26157)
It was always meant to be released for parallelization.
This now matches the other similar code in the module.

Thanks michaelforney for noticing!
2021-05-17 00:35:16 -07:00
Dong-hee Na a0ccc404ca
bpo-44113: Update __xxtestfuzz not to use Py_SetProgramName (GH-26083) 2021-05-13 08:22:18 +09:00
Erlend Egeberg Aasland 504ffdae4e
bpo-40645: Fix ref leaks in _hashopenssl (GH-26079) 2021-05-12 19:20:41 +01:00
Erlend Egeberg Aasland e5ba1fe995
bpo-44116: Add GC support to _csv heap types (GH-26074) 2021-05-12 19:18:58 +01:00
Petr Viktorin 3611db53f5
bpo-40645: Fix reference leak in the _hashopenssl extension (GH-26072)
The `PyModule_AddObjectRef` function doesn't steal a reference, so an extra `Py_DECREF` is needed.

Automerge-Triggered-By: GH:tiran
2021-05-12 09:04:34 -07:00