* bpo-9566: Fix compiler warnings in gcmodule.c (GH-11010)
Change PyDTrace_GC_DONE() argument type from int to Py_ssize_t.
(cherry picked from commit edad38e3e0)
* bpo-30465: Fix C downcast warning on Windows in ast.c (#6593)
ast.c: fstring_fix_node_location() downcasts a pointer difference to
a C int. Replace int with Py_ssize_t to fix the compiler warning.
(cherry picked from commit fb7e7992be)
* bpo-9566: Fix compiler warnings in peephole.c (GH-10652)
(cherry picked from commit 028f0ef4f3)
* bpo-27645, sqlite: Fix integer overflow on sleep (#6594)
Use the _PyTime_t type and round away from zero (ROUND_UP,
_PyTime_ROUND_TIMEOUT) the sleep duration, when converting a Python
object to seconds and then to milliseconds. Raise an OverflowError in
case of overflow.
Previously the (int)double conversion rounded towards zero
(ROUND_DOWN).
(cherry picked from commit ca405017d5)
* bpo-36389: _PyObject_IsFreed() now also detects uninitialized memory (GH-12770)
Replace _PyMem_IsFreed() function with _PyMem_IsPtrFreed() inline
function. The function is now way more efficient, it became a simple
comparison on integers, rather than a short loop. It detects also
uninitialized bytes and "forbidden bytes" filled by debug hooks
on memory allocators.
Add unit tests on _PyObject_IsFreed().
(cherry picked from commit 2b00db6855)
* bpo-36389: Change PyMem_SetupDebugHooks() constants (GH-12782)
Modify CLEANBYTE, DEADDYTE and FORBIDDENBYTE constants: use 0xCD,
0xDD and 0xFD, rather than 0xCB, 0xBB and 0xFB, to use the same byte
patterns than Windows CRT debug malloc() and free().
(cherry picked from commit 4c409beb4c)
Use crypt_r() when available instead of crypt() in the crypt module.
As a nice side effect: This also avoids a memory sanitizer flake as clang msan doesn't know about crypt's internal libc allocated buffer.
(cherry picked from commit 387512c7ec)
Co-authored-by: Gregory P. Smith <greg@krypto.org> [Google]
Fix str.format(), float.__format__() and complex.__format__() methods
for non-ASCII decimal point when using the "n" formatter.
Rewrite _PyUnicode_InsertThousandsGrouping(): it now requires
a _PyUnicodeWriter object for the buffer and a Python str object
for digits.
(cherry picked from commit 59423e3ddd)
bpo-34523, bpo-35290: C locale coercion now resets the Python
internal "force ASCII" mode. This change fix the filesystem encoding
on FreeBSD CURRENT, which has a new "C.UTF-8" locale, when
the UTF-8 mode is disabled.
Add _Py_ResetForceASCII(): _Py_SetLocaleFromEnv() now calls it.
(cherry picked from commit 353933e712)
* bpo-9263: _PyObject_Dump() detects freed memory (GH-10061)
_PyObject_Dump() now uses an heuristic to check if the object memory
has been freed: log "<freed object>" in that case.
The heuristic rely on the debug hooks on Python memory allocators
which fills the memory with DEADBYTE (0xDB) when memory is
deallocated. Use PYTHONMALLOC=debug to always enable these debug
hooks.
(cherry picked from commit 82af0b63b0)
* bpo-9263: Fix _PyObject_Dump() for freed object (#10661)
If _PyObject_Dump() detects that the object is freed, don't try to
dump it (exit immediately).
Enhance also _PyObject_IsFreed(): it now detects if the pointer
itself looks like freed memory.
(cherry picked from commit 2cf5d32fd9)
test_embed.InitConfigTests tests more configuration variables.
Changes:
* InitConfigTests tests more core configuration variables:
* base_exec_prefix
* base_prefix
* exec_prefix
* home
* module_search_path_env
* prefix
* "_testembed init_from_config" tests more variables:
* argv
* warnoptions
* xoptions
* Py_HasFileSystemDefaultEncoding value is no longer tested since it
depends on the LC_CTYPE locale and the platform.
* InitConfigTests: add check_global_config(), check_core_config() and
check_main_config() subfunctions to cleanup the code. Move also
constants at the class level (ex: COPY_MAIN_CONFIG).
* Use more macros in _PyCoreConfig_AsDict() and
_PyMainInterpreterConfig_AsDict() to reduce code duplication.
* Other minor cleanups.
(cherry picked from commit 01de89cb59)
* Add C functions:
* _Py_GetGlobalVariablesAsDict()
* _PyCoreConfig_AsDict()
* _PyMainInterpreterConfig_AsDict()
* Add new _testcapi methods:
* get_global_config()
* get_core_config()
* get_main_config()
* test.pythoninfo: get global, core and main configuration
* _testembed now serializes global, core and main configurations
using JSON to reuse _Py_GetGlobalVariablesAsDict(),
_PyCoreConfig_AsDict() and _PyMainInterpreterConfig_AsDict(),
rather than duplicating code.
* test_embed.InitConfigTests now test much more configuration
variables
Rename our new MEMORY_SANITIZER define to _Py_MEMORY_SANITIZER.
Project based C Preprocessor namespacing at its finest. :P
(cherry picked from commit 3015fb8ce4)
Co-authored-by: Gregory P. Smith <greg@krypto.org>
Adds configure flags for msan and ubsan builds to make it easier to enable.
These also encode the detail that address sanitizer and memory sanitizer
should disable pymalloc.
Define MEMORY_SANITIZER when appropriate at build time and adds workarounds
to existing code to mark things as initialized where the sanitizer is otherwise unable to
determine that. This lets our build succeed under the memory sanitizer. not all tests
pass without sanitizer failures yet but we're in pretty good shape after this.
(cherry picked from commit 1584a00815)
Co-authored-by: Gregory P. Smith <greg@krypto.org> [Google LLC]
* Add _Py_GetForceASCII(): check if Python forces the usage of ASCII
in Py_DecodeLocale() and Py_EncodeLocale().
* initfsencoding() now uses ASCII if _Py_GetForceASCII() is true.
On macOS, fix reading from and writing into a file with a size larger than 2 GiB.
(cherry picked from commit 74a8b6ea7e)
Co-authored-by: Stéphane Wirtel <stephane@wirtel.be>
The C accelerated _elementtree module now initializes hash randomization
salt from _Py_HashSecret instead of libexpat's default CPRNG.
Signed-off-by: Christian Heimes <christian@python.org>
https://bugs.python.org/issue34623
(cherry picked from commit cb5778f00c)
Co-authored-by: Christian Heimes <christian@python.org>
* bpo-34589: Make _PyCoreConfig.coerce_c_locale private (GH-9371)
_PyCoreConfig:
* Rename coerce_c_locale to _coerce_c_locale
* Rename coerce_c_locale_warn to _coerce_c_locale_warn
These fields are now private (name prefixed by "_").
(cherry picked from commit 188ebfa475)
* bpo-34589: C locale coercion off by default (GH-9073)
Py_Initialize() and Py_Main() cannot enable the C locale coercion
(PEP 538) anymore: it is always disabled. It can now only be enabled
by the Python program ("python3).
test_embed: get_filesystem_encoding() doesn't have to set PYTHONUTF8
nor PYTHONCOERCECLOCALE, these variables are already set in the
parent.
(cherry picked from commit 7a0791b699)
* bpo-34589: Add -X coerce_c_locale command line option (GH-9378)
Add a new -X coerce_c_locale command line option to control C locale
coercion (PEP 538).
(cherry picked from commit dbdee0073c)
* -X dev: it is now possible to override the memory allocator using
PYTHONMALLOC even if the developer mode is enabled.
* Add _Py_InitializeFromConfig()
* Add _Py_Initialize_ReadEnvVars() to set global configuration
variables from environment variables
* Fix the code to initialize Python: Py_Initialize() now also reads
environment variables
* _Py_InitializeCore() can now be called twice: the second call
only replaces the configuration.
* Write unit tests on Py_Initialize() and the different ways to
configure Python
* The isolated mode now always sets Py_IgnoreEnvironmentFlag and
Py_NoUserSiteDirectory to 1.
* pymain_read_conf() now saves/restores the configuration
if the encoding changed
Ned Deily
Inada Naoki
Gregory P. Smith
Victor Stinner
Rémi Lapeyre
Miss Islington (bot)
Serhiy Storchaka
Steve Dower
Alexey Izbyshev