Fix race in PyThread_release_lock that was leading to memory corruption and
deadlocks. The fix applies to POSIX systems where Python locks are implemented
with mutex and condition variable because POSIX semaphores are either not
provided, or are known to be broken. One particular example of such system is
macOS.
On Darwin, even though this is considered as POSIX, Python uses
mutex+condition variable to implement its lock, and, as of 2019-08-28, Py2.7
implementation, even though similar issue was fixed for Py3 in 2012, contains
synchronization bug: the condition is signalled after mutex unlock while the
correct protocol is to signal condition from under mutex:
https://github.com/python/cpython/blob/v2.7.16-127-g0229b56d8c0/Python/thread_pthread.h#L486-L506https://github.com/python/cpython/commit/187aa545165d (py3 fix)
PyPy has the same bug for both pypy2 and pypy3:
https://bitbucket.org/pypy/pypy/src/578667b3fef9/rpython/translator/c/src/thread_pthread.c#lines-443:465https://bitbucket.org/pypy/pypy/src/5b42890d48c3/rpython/translator/c/src/thread_pthread.c#lines-443:465
Signalling condition outside of corresponding mutex is considered OK by
POSIX, but in Python context it can lead to at least memory corruption if we
consider the whole lifetime of python level lock. For example the following
logical scenario:
T1 T2
sema = Lock()
sema.acquire()
sema.release()
sema.acquire()
free(sema)
...
can translate to the next C-level calls:
T1 T2
# sema = Lock()
sema = malloc(...)
sema.locked = 0
pthread_mutex_init(&sema.mut)
pthread_cond_init (&sema.lock_released)
# sema.acquire()
pthread_mutex_lock(&sema.mut)
# sees sema.locked == 0
sema.locked = 1
pthread_mutex_unlock(&sema.mut)
# sema.release()
pthread_mutex_lock(&sema.mut)
sema.locked = 0
pthread_mutex_unlock(&sema.mut)
# OS scheduler gets in and relinquishes control from T2
# to another process
...
# second sema.acquire()
pthread_mutex_lock(&sema.mut)
# sees sema.locked == 0
sema.locked = 1
pthread_mutex_unlock(&sema.mut)
# free(sema)
pthread_mutex_destroy(&sema.mut)
pthread_cond_destroy (&sema.lock_released)
free(sema)
# ...
e.g. malloc() which returns memory where sema was
...
# OS scheduler returns control to T2
# sema.release() continues
#
# BUT sema was already freed and writing to anywhere
# inside sema block CORRUPTS MEMORY. In particular if
# _another_ python-level lock was allocated where sema
# block was, writing into the memory can have effect on
# further synchronization correctness and in particular
# lead to deadlock on lock that was next allocated.
pthread_cond_signal(&sema.lock_released)
Note that T2.pthread_cond_signal(&sema.lock_released) CORRUPTS MEMORY as it
is called when sema memory was already freed and is potentially
reallocated for another object.
The fix is to move pthread_cond_signal to be done under corresponding mutex:
# sema.release()
pthread_mutex_lock(&sema.mut)
sema.locked = 0
pthread_cond_signal(&sema.lock_released)
pthread_mutex_unlock(&sema.mut)
To do so this patch cherry-picks thread_pthread.h part of the following 3.2 commit:
commit 187aa54516
Author: Kristján Valur Jónsson <kristjan@ccpgames.com>
Date: Tue Jun 5 22:17:42 2012 +0000
Signal condition variables with the mutex held. Destroy condition variables
before their mutexes.
Python/ceval_gil.h | 9 +++++----
Python/thread_pthread.h | 15 +++++++++------
2 files changed, 14 insertions(+), 10 deletions(-)
(ceval_gil.h is Python3 specific and does not apply to Python2.7)
The bug was there since 1994 - since at least [1]. It was discussed in 2001
with original code author[2], but the code was still considered to be
race-free. In 2010 the place where pthread_cond_signal should be - before or
after pthread_mutex_unlock - was discussed with the rationale to avoid
threads bouncing[3,4,5], and in 2012 pthread_cond_signal was moved to be
called from under mutex, but only for CPython3[6,7].
In 2019 the bug was (re-)discovered while testing Pygolang[8] on macOS with
CPython2 and PyPy2 and PyPy3.
[1] https://github.com/python/cpython/commit/2c8cb9f3d240
[2] https://bugs.python.org/issue433625
[3] https://bugs.python.org/issue8299#msg103224
[4] https://bugs.python.org/issue8410#msg103313
[5] https://bugs.python.org/issue8411#msg113301
[6] https://bugs.python.org/issue15038#msg163187
[7] https://github.com/python/cpython/commit/187aa545165d
[8] https://pypi.org/project/pygolang
(cherry picked from commit 187aa54516)
Co-Authored-By: Kristján Valur Jónsson <kristjan@ccpgames.com>
Fix an unlikely memory leak on conversion from string to float in the
function _Py_dg_strtod() used by float(str), complex(str),
pickle.load(), marshal.load(), etc.
Fix an unlikely memory leak in _Py_dg_strtod() on "undfl:" label:
rewrite memory management in this function to always release all
memory before exiting the function. Initialize variables to NULL, and
set them to NULL after calling Bfree() at the "cont:" label.
Note: Bfree(NULL) is well defined: it does nothing.
(cherry picked from commit 9776b0636a)
In _localemodule.c and selectmodule.c, remove dead code that would
cause double decrefs if run.
In addition, replace PyList_SetItem() with PyList_SET_ITEM() in cases
where a new list is populated and there is no possibility of an error.
In addition, check if the list changed size in the loop in array_array_fromlist().
(cherry picked from commit 99d56b5356)
Co-authored-by: Zackery Spytz <zspytz@gmail.com>
The pthread implementation of PyThread_start_new_thread() now uses
malloc/free rather than PyMem_Malloc/PyMem_Free, since the latters
are not thread-safe.
Fix an undefined behaviour in the pthread implementation of
PyThread_start_new_thread(): add a function wrapper to always return
NULL.
Add pythread_callback struct and pythread_wrapper() to thread_pthread.h.
(cherry picked from commit 9eea6eaf23)
This missed PyErr_NoMemory() could cause a SystemError when calling
_symtable.symtable().
(cherry picked from commit ad65f15581)
Co-authored-by: Zackery Spytz <zspytz@gmail.com>
Two kind of mistakes:
1. Missed space. After concatenating there is no space between words.
2. Missed comma. Causes unintentional concatenating in a list of strings.
(cherry picked from commit 34fd4c2019)
(cherry picked from commit 7054e5c80b)
Fix the following warning:
Python/pystrtod.c: In function 'format_float_short':
Python/pystrtod.c:1007:13: warning: 'strncpy' output truncated before terminating nul copying 3 bytes from a string of the same length [-Wstringop-truncation]
strncpy(p, "ERR", 3);
(cherry picked from commit 9fb8415759)
Python 2 never checked for I/O error when reading .py files and
thus could mistake an I/O error for EOF and create incorrect .pyc
files.
This adds an check for this and aborts on an error.
Clarify that the level argument is used to determine whether to
perform absolute or relative imports: 0 is absolute, while a positive number
is the number of parent directories to search relative to the current module..
(cherry picked from commit 461d225b19)
Co-authored-by: oldk <oldk1331@users.noreply.github.com>
When PyGILState_Ensure() is called in a non-Python thread before
PyEval_InitThreads(), only call PyEval_InitThreads() after calling
PyThreadState_New() to fix a crash.
(cherry picked from commit b4d1e1f7c1)
GCC says:
../cpython/Python/marshal.c: In function ‘PyMarshal_WriteLongToFile’:
../cpython/Python/marshal.c:70:35: warning: ‘wf.ptr’ may be used uninitialized in this function [-Wmaybe-uninitialized]
else if ((p)->ptr != (p)->end) *(p)->ptr++ = (c); \
^~
../cpython/Python/marshal.c:70:47: warning: ‘wf.end’ may be used uninitialized in this function [-Wmaybe-uninitialized]
else if ((p)->ptr != (p)->end) *(p)->ptr++ = (c); \
^~
../cpython/Python/marshal.c:77:10: warning: ‘wf.str’ may be used uninitialized in this function [-Wmaybe-uninitialized]
if (p->str == NULL)
~^~~~~
This isn't a real problem because if the file pointer is not NULL, the
string-related fields are never touched. But, it doesn't hurt to set the unused
fields to NULL.
bpo-31692, bpo-19527:
* Add a new PYTHONSHOWALLOCCOUNT environment variable, similar to
the Python 3 "-X showalloccount" option
* When Python is compiled with COUNT_ALLOCS, the new
PYTHONSHOWALLOCCOUNT environment variable now has to be set to dump
allocation counts into stderr on shutdown. Moreover, allocations
statistics are now dumped into stderr rather than stdout.
* Add @test.support.requires_type_collecting decorator: skip test if
COUNT_ALLOCS is defined
* Fix tests for COUNT_ALLOCS: decorate some methods with
@requires_type_collecting
* test_sys.test_objecttypes(): update object type when COUNT_ALLOCS
is defined
* [2.7] bpo-30765: Avoid blocking when PyThread_acquire_lock() is asked not to (GH-2403)
* bpo-30765: Avoid blocking when PyThread_acquire_lock() is asked not to lock
This is especially important if PyThread_acquire_lock() is called reentrantly
(for example from a signal handler).
* Update 2017-06-26-14-29-50.bpo-30765.Q5iBmf.rst
* Avoid core logic when taking the mutex failed.
(cherry picked from commit f84ac420c2)
* Remove test undef
Based on commit 5c4b0d063a by Ned
Deily, which is based on original patches by Brett Cannon and Steve
Dower.
Remove also the private _Py_svnversion() function and SVNVERSION
variable.
Note: Py_SubversionRevision() and Py_SubversionShortBranch() are
unchanged, they are part of the public API.
Issue #29188: Support glibc 2.24 on Linux: don't use getentropy() function but
read from /dev/urandom to get random bytes, for example in os.urandom(). On
Linux, getentropy() is implemented which getrandom() is blocking mode, whereas
os.urandom() should not block.
Serhiy Storchaka
Kirill Smelkov
Tony Flury
Inada Naoki
Victor Stinner
Benjamin Peterson
Miss Islington (bot)
tzickel
Alexey Izbyshev
Xiang Zhang
INADA Naoki
Mariatta
Victor Stinner
Oren Milman
Yuan Chao Chou
Antoine Pitrou
Mark Dickinson
Ned Deily