diff --git a/Doc/glossary.rst b/Doc/glossary.rst index 2f1277c22b0..21b92a99a1d 100644 --- a/Doc/glossary.rst +++ b/Doc/glossary.rst @@ -385,7 +385,7 @@ Glossary :meth:`str.lower` method can serve as a key function for case insensitive sorts. Alternatively, an ad-hoc key function can be built from a :keyword:`lambda` expression such as ``lambda r: (r[0], r[2])``. Also, - the :mod:`operator` module provides three key function constuctors: + the :mod:`operator` module provides three key function constructors: :func:`~operator.attrgetter`, :func:`~operator.itemgetter`, and :func:`~operator.methodcaller`. See the :ref:`Sorting HOW TO ` for examples of how to create and use key functions. diff --git a/Doc/library/argparse.rst b/Doc/library/argparse.rst index 79a98cbcfdc..0123b5cbac9 100644 --- a/Doc/library/argparse.rst +++ b/Doc/library/argparse.rst @@ -1642,8 +1642,8 @@ Argument groups --bar BAR bar help - Note that any arguments not your user defined groups will end up back in the - usual "positional arguments" and "optional arguments" sections. + Note that any arguments not in your user-defined groups will end up back + in the usual "positional arguments" and "optional arguments" sections. Mutual exclusion @@ -1833,9 +1833,10 @@ A partial upgrade path from :mod:`optparse` to :mod:`argparse`: * Replace all :meth:`optparse.OptionParser.add_option` calls with :meth:`ArgumentParser.add_argument` calls. -* Replace ``options, args = parser.parse_args()`` with ``args = +* Replace ``(options, args) = parser.parse_args()`` with ``args = parser.parse_args()`` and add additional :meth:`ArgumentParser.add_argument` - calls for the positional arguments. + calls for the positional arguments. Keep in mind that what was previously + called ``options``, now in :mod:`argparse` context is called ``args``. * Replace callback actions and the ``callback_*`` keyword arguments with ``type`` or ``action`` arguments. diff --git a/Doc/library/logging.handlers.rst b/Doc/library/logging.handlers.rst index c4dd438f5b2..ef65cfa5594 100644 --- a/Doc/library/logging.handlers.rst +++ b/Doc/library/logging.handlers.rst @@ -654,7 +654,7 @@ event of a certain severity or greater is seen. :class:`BufferingHandler`, which is an abstract class. This buffers logging records in memory. Whenever each record is added to the buffer, a check is made by calling :meth:`shouldFlush` to see if the buffer should be flushed. If it -should, then :meth:`flush` is expected to do the needful. +should, then :meth:`flush` is expected to do the flushing. .. class:: BufferingHandler(capacity) diff --git a/Doc/library/signal.rst b/Doc/library/signal.rst index 698b1e74f49..d1cae13d62e 100644 --- a/Doc/library/signal.rst +++ b/Doc/library/signal.rst @@ -5,46 +5,58 @@ :synopsis: Set handlers for asynchronous events. -This module provides mechanisms to use signal handlers in Python. Some general -rules for working with signals and their handlers: +This module provides mechanisms to use signal handlers in Python. -* A handler for a particular signal, once set, remains installed until it is - explicitly reset (Python emulates the BSD style interface regardless of the - underlying implementation), with the exception of the handler for - :const:`SIGCHLD`, which follows the underlying implementation. -* There is no way to "block" signals temporarily from critical sections (since - this is not supported by all Unix flavors). +General rules +------------- -* Although Python signal handlers are called asynchronously as far as the Python - user is concerned, they can only occur between the "atomic" instructions of the - Python interpreter. This means that signals arriving during long calculations - implemented purely in C (such as regular expression matches on large bodies of - text) may be delayed for an arbitrary amount of time. +The :func:`signal.signal` function allows to define custom handlers to be +executed when a signal is received. A small number of default handlers are +installed: :const:`SIGPIPE` is ignored (so write errors on pipes and sockets +can be reported as ordinary Python exceptions) and :const:`SIGINT` is +translated into a :exc:`KeyboardInterrupt` exception. -* When a signal arrives during an I/O operation, it is possible that the I/O - operation raises an exception after the signal handler returns. This is - dependent on the underlying Unix system's semantics regarding interrupted system - calls. +A handler for a particular signal, once set, remains installed until it is +explicitly reset (Python emulates the BSD style interface regardless of the +underlying implementation), with the exception of the handler for +:const:`SIGCHLD`, which follows the underlying implementation. -* Because the C signal handler always returns, it makes little sense to catch - synchronous errors like :const:`SIGFPE` or :const:`SIGSEGV`. +There is no way to "block" signals temporarily from critical sections (since +this is not supported by all Unix flavors). -* Python installs a small number of signal handlers by default: :const:`SIGPIPE` - is ignored (so write errors on pipes and sockets can be reported as ordinary - Python exceptions) and :const:`SIGINT` is translated into a - :exc:`KeyboardInterrupt` exception. All of these can be overridden. -* Some care must be taken if both signals and threads are used in the same - program. The fundamental thing to remember in using signals and threads - simultaneously is: always perform :func:`signal` operations in the main thread - of execution. Any thread can perform an :func:`alarm`, :func:`getsignal`, - :func:`pause`, :func:`setitimer` or :func:`getitimer`; only the main thread - can set a new signal handler, and the main thread will be the only one to - receive signals (this is enforced by the Python :mod:`signal` module, even - if the underlying thread implementation supports sending signals to - individual threads). This means that signals can't be used as a means of - inter-thread communication. Use locks instead. +Execution of Python signal handlers +^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ + +A Python signal handler does not get executed inside the low-level (C) signal +handler. Instead, the low-level signal handler sets a flag which tells the +:term:`virtual machine` to execute the corresponding Python signal handler +at a later point(for example at the next :term:`bytecode` instruction). +This has consequences: + +* It makes little sense to catch synchronous errors like :const:`SIGFPE` or + :const:`SIGSEGV`. + +* A long-running calculation implemented purely in C (such as regular + expression matching on a large body of text) may run uninterrupted for an + arbitrary amount of time, regardless of any signals received. The Python + signal handlers will be called when the calculation finishes. + + +Signals and threads +^^^^^^^^^^^^^^^^^^^ + +Python signal handlers are always executed in the main Python thread, +even if the signal was received in another thread. This means that signals +can't be used as a means of inter-thread communication. You can use +the synchronization primitives from the :mod:`threading` module instead. + +Besides, only the main thread is allowed to set a new signal handler. + + +Module contents +--------------- The variables defined in the :mod:`signal` module are: diff --git a/Doc/library/syslog.rst b/Doc/library/syslog.rst index 795d66d99fd..645c326678f 100644 --- a/Doc/library/syslog.rst +++ b/Doc/library/syslog.rst @@ -78,7 +78,8 @@ Priority levels (high to low): Facilities: :const:`LOG_KERN`, :const:`LOG_USER`, :const:`LOG_MAIL`, :const:`LOG_DAEMON`, :const:`LOG_AUTH`, :const:`LOG_LPR`, :const:`LOG_NEWS`, :const:`LOG_UUCP`, - :const:`LOG_CRON` and :const:`LOG_LOCAL0` to :const:`LOG_LOCAL7`. + :const:`LOG_CRON`, :const:`LOG_SYSLOG` and :const:`LOG_LOCAL0` to + :const:`LOG_LOCAL7`. Log options: :const:`LOG_PID`, :const:`LOG_CONS`, :const:`LOG_NDELAY`, :const:`LOG_NOWAIT` diff --git a/Doc/library/unittest.rst b/Doc/library/unittest.rst index bdf07a40cb2..b130a8b0889 100644 --- a/Doc/library/unittest.rst +++ b/Doc/library/unittest.rst @@ -640,7 +640,7 @@ This is the output of running the example above in verbose mode: :: Classes can be skipped just like methods: :: - @skip("showing class skipping") + @unittest.skip("showing class skipping") class MySkippedTestCase(unittest.TestCase): def test_not_run(self): pass diff --git a/Doc/library/webbrowser.rst b/Doc/library/webbrowser.rst index 23ba6c54719..dfb09ee40e8 100644 --- a/Doc/library/webbrowser.rst +++ b/Doc/library/webbrowser.rst @@ -137,6 +137,8 @@ for the controller classes, all defined in this module. +-----------------------+-----------------------------------------+-------+ | ``'macosx'`` | :class:`MacOSX('default')` | \(4) | +-----------------------+-----------------------------------------+-------+ +| ``'safari'`` | :class:`MacOSX('safari')` | \(4) | ++-----------------------+-----------------------------------------+-------+ Notes: diff --git a/Lib/concurrent/futures/_base.py b/Lib/concurrent/futures/_base.py index 79b91d495fc..9f11f6977f1 100644 --- a/Lib/concurrent/futures/_base.py +++ b/Lib/concurrent/futures/_base.py @@ -112,12 +112,14 @@ class _AllCompletedWaiter(_Waiter): def __init__(self, num_pending_calls, stop_on_exception): self.num_pending_calls = num_pending_calls self.stop_on_exception = stop_on_exception + self.lock = threading.Lock() super().__init__() def _decrement_pending_calls(self): - self.num_pending_calls -= 1 - if not self.num_pending_calls: - self.event.set() + with self.lock: + self.num_pending_calls -= 1 + if not self.num_pending_calls: + self.event.set() def add_result(self, future): super().add_result(future) diff --git a/Lib/idlelib/NEWS.txt b/Lib/idlelib/NEWS.txt index 4482be2024a..a6b06b40e0a 100644 --- a/Lib/idlelib/NEWS.txt +++ b/Lib/idlelib/NEWS.txt @@ -1,6 +1,11 @@ What's New in IDLE 3.2.3? ========================= +- Issue #14409: IDLE now properly executes commands in the Shell window + when it cannot read the normal config files on startup and + has to use the built-in default key bindings. + There was previously a bug in one of the defaults. + - Issue #3573: IDLE hangs when passing invalid command line args (directory(ies) instead of file(s)). diff --git a/Lib/idlelib/configHandler.py b/Lib/idlelib/configHandler.py index 73b8db5b232..da927260ee8 100644 --- a/Lib/idlelib/configHandler.py +++ b/Lib/idlelib/configHandler.py @@ -595,7 +595,7 @@ class IdleConf: '<>': [''], '<>': [''], '<>': [''], - '<>': [' '], + '<>': ['', ''], '<>': [''], '<>': [''], '<>': [''], diff --git a/Lib/logging/handlers.py b/Lib/logging/handlers.py index fed8c9393d6..7689b040c60 100644 --- a/Lib/logging/handlers.py +++ b/Lib/logging/handlers.py @@ -519,11 +519,16 @@ class SocketHandler(logging.Handler): """ ei = record.exc_info if ei: - dummy = self.format(record) # just to get traceback text into record.exc_text - record.exc_info = None # to avoid Unpickleable error - s = pickle.dumps(record.__dict__, 1) - if ei: - record.exc_info = ei # for next handler + # just to get traceback text into record.exc_text ... + dummy = self.format(record) + # See issue #14436: If msg or args are objects, they may not be + # available on the receiving end. So we convert the msg % args + # to a string, save it as msg and zap the args. + d = dict(record.__dict__) + d['msg'] = record.getMessage() + d['args'] = None + d['exc_info'] = None + s = pickle.dumps(d, 1) slen = struct.pack(">L", len(s)) return slen + s diff --git a/Lib/multiprocessing/connection.py b/Lib/multiprocessing/connection.py index df00f1d9066..fa6f7b39019 100644 --- a/Lib/multiprocessing/connection.py +++ b/Lib/multiprocessing/connection.py @@ -94,6 +94,13 @@ def arbitrary_address(family): else: raise ValueError('unrecognized family') +def _validate_family(family): + ''' + Checks if the family is valid for the current environment. + ''' + if sys.platform != 'win32' and family == 'AF_PIPE': + raise ValueError('Family %s is not recognized.' % family) + def address_type(address): ''' @@ -126,6 +133,7 @@ class Listener(object): or default_family address = address or arbitrary_address(family) + _validate_family(family) if family == 'AF_PIPE': self._listener = PipeListener(address, backlog) else: @@ -163,6 +171,7 @@ def Client(address, family=None, authkey=None): Returns a connection to the address of a `Listener` ''' family = family or address_type(address) + _validate_family(family) if family == 'AF_PIPE': c = PipeClient(address) else: diff --git a/Lib/pydoc.py b/Lib/pydoc.py index f45d46170e2..89fd09b5567 100755 --- a/Lib/pydoc.py +++ b/Lib/pydoc.py @@ -1829,7 +1829,7 @@ has the same effect as typing a particular string at the help> prompt. Welcome to Python %s! This is the online help utility. If this is your first time using Python, you should definitely check out -the tutorial on the Internet at http://docs.python.org/tutorial/. +the tutorial on the Internet at http://docs.python.org/%s/tutorial/. Enter the name of any module, keyword, or topic to get help on writing Python programs and using Python modules. To quit this help utility and @@ -1839,7 +1839,7 @@ To get a list of available modules, keywords, or topics, type "modules", "keywords", or "topics". Each module also comes with a one-line summary of what it does; to list the modules whose summaries contain a given word such as "spam", type "modules spam". -''' % sys.version[:3]) +''' % tuple([sys.version[:3]]*2)) def list(self, items, columns=4, width=80): items = list(sorted(items)) diff --git a/Lib/rlcompleter.py b/Lib/rlcompleter.py index 8b74ffaadda..d3a443737a5 100644 --- a/Lib/rlcompleter.py +++ b/Lib/rlcompleter.py @@ -1,13 +1,11 @@ -"""Word completion for GNU readline 2.0. +"""Word completion for GNU readline. -This requires the latest extension to the readline module. The completer -completes keywords, built-ins and globals in a selectable namespace (which -defaults to __main__); when completing NAME.NAME..., it evaluates (!) the -expression up to the last dot and completes its attributes. +The completer completes keywords, built-ins and globals in a selectable +namespace (which defaults to __main__); when completing NAME.NAME..., it +evaluates (!) the expression up to the last dot and completes its attributes. -It's very cool to do "import sys" type "sys.", hit the -completion key (twice), and see the list of names defined by the -sys module! +It's very cool to do "import sys" type "sys.", hit the completion key (twice), +and see the list of names defined by the sys module! Tip: to use the tab key as the completion key, call @@ -15,21 +13,19 @@ Tip: to use the tab key as the completion key, call Notes: -- Exceptions raised by the completer function are *ignored* (and -generally cause the completion to fail). This is a feature -- since -readline sets the tty device in raw (or cbreak) mode, printing a -traceback wouldn't work well without some complicated hoopla to save, -reset and restore the tty state. +- Exceptions raised by the completer function are *ignored* (and generally cause + the completion to fail). This is a feature -- since readline sets the tty + device in raw (or cbreak) mode, printing a traceback wouldn't work well + without some complicated hoopla to save, reset and restore the tty state. -- The evaluation of the NAME.NAME... form may cause arbitrary -application defined code to be executed if an object with a -__getattr__ hook is found. Since it is the responsibility of the -application (or the user) to enable this feature, I consider this an -acceptable risk. More complicated expressions (e.g. function calls or -indexing operations) are *not* evaluated. +- The evaluation of the NAME.NAME... form may cause arbitrary application + defined code to be executed if an object with a __getattr__ hook is found. + Since it is the responsibility of the application (or the user) to enable this + feature, I consider this an acceptable risk. More complicated expressions + (e.g. function calls or indexing operations) are *not* evaluated. - When the original stdin is not a tty device, GNU readline is never -used, and this module (and the readline module) are silently inactive. + used, and this module (and the readline module) are silently inactive. """ diff --git a/Lib/socket.py b/Lib/socket.py index 1e285493c48..a93cd11248c 100644 --- a/Lib/socket.py +++ b/Lib/socket.py @@ -197,6 +197,17 @@ class socket(_socket.socket): if self._io_refs <= 0: self._real_close() + def detach(self): + """detach() -> file descriptor + + Close the socket object without closing the underlying file descriptor. + The object cannot be used after this call, but the file descriptor + can be reused for other purposes. The file descriptor is returned. + """ + self._closed = True + return super().detach() + + def fromfd(fd, family, type, proto=0): """ fromfd(fd, family, type[, proto]) -> socket object diff --git a/Lib/test/test_concurrent_futures.py b/Lib/test/test_concurrent_futures.py index 372da27dca3..2afa93802e2 100644 --- a/Lib/test/test_concurrent_futures.py +++ b/Lib/test/test_concurrent_futures.py @@ -183,7 +183,9 @@ class ProcessPoolShutdownTest(ProcessPoolMixin, ExecutorShutdownTest): for p in processes: p.join() + class WaitTests(unittest.TestCase): + def test_first_completed(self): future1 = self.executor.submit(mul, 21, 2) future2 = self.executor.submit(time.sleep, 1.5) @@ -284,7 +286,21 @@ class WaitTests(unittest.TestCase): class ThreadPoolWaitTests(ThreadPoolMixin, WaitTests): - pass + + def test_pending_calls_race(self): + # Issue #14406: multi-threaded race condition when waiting on all + # futures. + event = threading.Event() + def future_func(): + event.wait() + oldswitchinterval = sys.getswitchinterval() + sys.setswitchinterval(1e-6) + try: + fs = {self.executor.submit(future_func) for i in range(100)} + event.set() + futures.wait(fs, return_when=futures.ALL_COMPLETED) + finally: + sys.setswitchinterval(oldswitchinterval) class ProcessPoolWaitTests(ProcessPoolMixin, WaitTests): diff --git a/Lib/test/test_multiprocessing.py b/Lib/test/test_multiprocessing.py index 8edb4208640..8de7a8d1d0c 100644 --- a/Lib/test/test_multiprocessing.py +++ b/Lib/test/test_multiprocessing.py @@ -2319,8 +2319,20 @@ class TestStdinBadfiledescriptor(unittest.TestCase): flike.flush() assert sio.getvalue() == 'foo' + +# +# Issue 14151: Test invalid family on invalid environment +# + +class TestInvalidFamily(unittest.TestCase): + + @unittest.skipIf(WIN32, "skipped on Windows") + def test_invalid_family(self): + with self.assertRaises(ValueError): + multiprocessing.connection.Listener(r'\\.\test') + testcases_other = [OtherTest, TestInvalidHandle, TestInitializers, - TestStdinBadfiledescriptor] + TestStdinBadfiledescriptor, TestInvalidFamily] # # diff --git a/Lib/test/test_socket.py b/Lib/test/test_socket.py index d77b7dc9ed6..cce0d1b6eb4 100644 --- a/Lib/test/test_socket.py +++ b/Lib/test/test_socket.py @@ -951,6 +951,7 @@ class BasicTCPTest(SocketConnectedTest): f = self.cli_conn.detach() self.assertEqual(f, fileno) # cli_conn cannot be used anymore... + self.assertTrue(self.cli_conn._closed) self.assertRaises(socket.error, self.cli_conn.recv, 1024) self.cli_conn.close() # ...but we can create another socket using the (still open) diff --git a/Misc/NEWS b/Misc/NEWS index e3b110de98f..2fa911e4bba 100644 --- a/Misc/NEWS +++ b/Misc/NEWS @@ -10,6 +10,9 @@ What's New in Python 3.2.4 Core and Builtins ----------------- +- Issue #13019: Fix potential reference leaks in bytearray.extend(). Patch + by Suman Saha. + - Issue #14378: Fix compiling ast.ImportFrom nodes with a "__future__" string as the module name that was not interned. @@ -31,6 +34,21 @@ Core and Builtins Library ------- +- Issue #14151: Raise a ValueError, not a NameError, when trying to create + a multiprocessing Client or Listener with an AF_PIPE type address under + non-Windows platforms. Patch by Popa Claudiu. + +- Issue #13872: socket.detach() now marks the socket closed (as mirrored + in the socket repr()). Patch by Matt Joiner. + +- Issue #14406: Fix a race condition when using ``concurrent.futures.wait( + return_when=ALL_COMPLETED)``. Patch by Matt Joiner. + +- Issue #14409: IDLE now properly executes commands in the Shell window + when it cannot read the normal config files on startup and + has to use the built-in default key bindings. + There was previously a bug in one of the defaults. + - Issue #10340: asyncore - properly handle EINVAL in dispatcher constructor on OSX; avoid to call handle_connect in case of a disconnected socket which was not meant to connect. @@ -97,6 +115,8 @@ Extension Modules Build ----- +- Issue #14437: Fix building the _io module under Cygwin. + - Issue #14387: Do not include accu.h from Python.h. - Issue #14359: Only use O_CLOEXEC in _posixmodule.c if it is defined. diff --git a/Modules/_io/_iomodule.h b/Modules/_io/_iomodule.h index 925e4f2cc72..c198b43e782 100644 --- a/Modules/_io/_iomodule.h +++ b/Modules/_io/_iomodule.h @@ -67,7 +67,7 @@ typedef struct { PyObject *filename; /* Not used, but part of the IOError object */ Py_ssize_t written; } PyBlockingIOErrorObject; -PyAPI_DATA(PyObject *) PyExc_BlockingIOError; +extern PyObject *PyExc_BlockingIOError; /* * Offset type for positioning. diff --git a/Modules/python.c b/Modules/python.c index 935908af5dc..cf9383f444e 100644 --- a/Modules/python.c +++ b/Modules/python.c @@ -22,9 +22,9 @@ extern wchar_t* _Py_DecodeUTF8_surrogateescape(const char *s, Py_ssize_t size); int main(int argc, char **argv) { - wchar_t **argv_copy = (wchar_t **)PyMem_Malloc(sizeof(wchar_t*)*argc); + wchar_t **argv_copy = (wchar_t **)PyMem_Malloc(sizeof(wchar_t*)*(argc+1)); /* We need a second copies, as Python might modify the first one. */ - wchar_t **argv_copy2 = (wchar_t **)PyMem_Malloc(sizeof(wchar_t*)*argc); + wchar_t **argv_copy2 = (wchar_t **)PyMem_Malloc(sizeof(wchar_t*)*(argc+1)); int i, res; char *oldloc; /* 754 requires that FP exceptions run in "no stop" mode by default, @@ -58,6 +58,8 @@ main(int argc, char **argv) } argv_copy2[i] = argv_copy[i]; } + argv_copy2[argc] = argv_copy[argc] = NULL; + setlocale(LC_ALL, oldloc); free(oldloc); res = Py_Main(argc, argv_copy); diff --git a/Objects/bytearrayobject.c b/Objects/bytearrayobject.c index 4202ff28e44..55b4df638a8 100644 --- a/Objects/bytearrayobject.c +++ b/Objects/bytearrayobject.c @@ -2234,8 +2234,10 @@ bytearray_extend(PyByteArrayObject *self, PyObject *arg) } bytearray_obj = PyByteArray_FromStringAndSize(NULL, buf_size); - if (bytearray_obj == NULL) + if (bytearray_obj == NULL) { + Py_DECREF(it); return NULL; + } buf = PyByteArray_AS_STRING(bytearray_obj); while ((item = PyIter_Next(it)) != NULL) { @@ -2268,8 +2270,10 @@ bytearray_extend(PyByteArrayObject *self, PyObject *arg) return NULL; } - if (bytearray_setslice(self, Py_SIZE(self), Py_SIZE(self), bytearray_obj) == -1) + if (bytearray_setslice(self, Py_SIZE(self), Py_SIZE(self), bytearray_obj) == -1) { + Py_DECREF(bytearray_obj); return NULL; + } Py_DECREF(bytearray_obj); Py_RETURN_NONE;