Merge branch 'master' into unquote_plus_handle_bytes

Doc/c-api/code.rst

@@ -40,8 +40,6 @@ bound into a function.
    can bind you to a precise Python version since the definition of the bytecode
    changes often.
 
-   .. audit-event:: code.__new__ code,filename,name,argcount,posonlyargcount,kwonlyargcount,nlocals,stacksize,flags c.PyCode_New
-
 .. c:function:: PyCodeObject* PyCode_NewWithPosOnlyArgs(int argcount, int posonlyargcount, int kwonlyargcount, int nlocals, int stacksize, int flags, PyObject *code, PyObject *consts, PyObject *names, PyObject *varnames, PyObject *freevars, PyObject *cellvars, PyObject *filename, PyObject *name, int firstlineno, PyObject *lnotab)
 
    Similar to :c:func:`PyCode_New`, but with an extra "posonlyargcount" for positonal-only arguments.

Doc/c-api/list.rst

@@ -71,8 +71,9 @@ List Objects
 
 .. c:function:: int PyList_SetItem(PyObject *list, Py_ssize_t index, PyObject *item)
 
-   Set the item at index *index* in list to *item*.  Return ``0`` on success
-   or ``-1`` on failure.
+   Set the item at index *index* in list to *item*.  Return ``0`` on success.
+   If *index* is out of bounds, return ``-1`` and set an :exc:`IndexError`
+   exception.
 
    .. note::
 
@@ -111,8 +112,7 @@ List Objects
 
    Return a list of the objects in *list* containing the objects *between* *low*
    and *high*.  Return *NULL* and set an exception if unsuccessful.  Analogous
-   to ``list[low:high]``.  Negative indices, as when slicing from Python, are not
-   supported.
+   to ``list[low:high]``.  Indexing from the end of the list is not supported.
 
 
 .. c:function:: int PyList_SetSlice(PyObject *list, Py_ssize_t low, Py_ssize_t high, PyObject *itemlist)
@@ -120,8 +120,8 @@ List Objects
    Set the slice of *list* between *low* and *high* to the contents of
    *itemlist*.  Analogous to ``list[low:high] = itemlist``. The *itemlist* may
    be *NULL*, indicating the assignment of an empty list (slice deletion).
-   Return ``0`` on success, ``-1`` on failure.  Negative indices, as when
-   slicing from Python, are not supported.
+   Return ``0`` on success, ``-1`` on failure.  Indexing from the end of the
+   list is not supported.
 
 
 .. c:function:: int PyList_Sort(PyObject *list)

Doc/c-api/sys.rst

@@ -330,7 +330,16 @@ accessible to C code.  They all work with the current interpreter thread's
 
    See :pep:`578` for a detailed description of auditing. Functions in the
    runtime and standard library that raise events include the details in each
-   function's documentation.
+   function's documentation and listed in the :ref:`audit events table
+   <audit-events>`.
+
+   .. audit-event:: sys.addaudithook "" c.PySys_AddAuditHook
+
+      If the interpreter is initialized, this function raises a auditing event
+      ``sys.addaudithook`` with no arguments. If any existing hooks raise an
+      exception derived from :class:`Exception`, the new hook will not be
+      added and the exception is cleared. As a result, callers cannot assume
+      that their hook has been added unless they control all existing hooks.
 
    .. versionadded:: 3.8
 

Doc/c-api/tuple.rst

@@ -57,7 +57,7 @@ Tuple Objects
 .. c:function:: PyObject* PyTuple_GetItem(PyObject *p, Py_ssize_t pos)
 
    Return the object at position *pos* in the tuple pointed to by *p*.  If *pos* is
-   out of bounds, return *NULL* and sets an :exc:`IndexError` exception.
+   out of bounds, return *NULL* and set an :exc:`IndexError` exception.
 
 
 .. c:function:: PyObject* PyTuple_GET_ITEM(PyObject *p, Py_ssize_t pos)
@@ -67,18 +67,21 @@ Tuple Objects
 
 .. c:function:: PyObject* PyTuple_GetSlice(PyObject *p, Py_ssize_t low, Py_ssize_t high)
 
-   Take a slice of the tuple pointed to by *p* from *low* to *high* and return it
-   as a new tuple.
+   Return the slice of the tuple pointed to by *p* between *low* and *high*,
+   or *NULL* on failure.  This is the equivalent of the Python expression
+   ``p[low:high]``.  Indexing from the end of the list is not supported.
 
 
 .. c:function:: int PyTuple_SetItem(PyObject *p, Py_ssize_t pos, PyObject *o)
 
    Insert a reference to object *o* at position *pos* of the tuple pointed to by
-   *p*. Return ``0`` on success.
+   *p*.  Return ``0`` on success.  If *pos* is out of bounds, return ``-1``
+   and set an :exc:`IndexError` exception.
 
    .. note::
 
-      This function "steals" a reference to *o*.
+      This function "steals" a reference to *o* and discards a reference to
+      an item already in the tuple at the affected position.
 
 
 .. c:function:: void PyTuple_SET_ITEM(PyObject *p, Py_ssize_t pos, PyObject *o)
@@ -88,7 +91,10 @@ Tuple Objects
 
    .. note::
 
-      This function "steals" a reference to *o*.
+      This macro "steals" a reference to *o*, and, unlike
+      :c:func:`PyTuple_SetItem`, does *not* discard a reference to any item that
+      is being replaced; any reference in the tuple at position *pos* will be
+      leaked.
 
 
 .. c:function:: int _PyTuple_Resize(PyObject **p, Py_ssize_t newsize)

Doc/library/codecs.rst

@@ -197,6 +197,9 @@ wider range of codecs when working with binary files:
    *buffering* has the same meaning as for the built-in :func:`open` function.
    It defaults to -1 which means that the default buffer size will be used.
 
+   .. versionchanged:: 3.9
+      The ``'U'`` mode has been removed.
+
 
 .. function:: EncodedFile(file, data_encoding, file_encoding=None, errors='strict')
 

Doc/library/fcntl.rst

@@ -36,6 +36,8 @@ descriptor.
 .. versionchanged:: 3.9
    On macOS, the fcntl module exposes the ``F_GETPATH`` constant, which obtains
    the path of a file from a file descriptor.
+   On Linux(>=3.15), the fcntl module exposes the ``F_OFD_GETLK``, ``F_OFD_SETLK``
+   and ``F_OFD_SETLKW`` constants, which working with open file description locks.
 
 The module defines the following functions:
 

Doc/library/fileinput.rst

@@ -148,8 +148,8 @@ available for subclassing as well:
    The sequence must be accessed in strictly sequential order; random access
    and :meth:`~io.TextIOBase.readline` cannot be mixed.
 
-   With *mode* you can specify which file mode will be passed to :func:`open`. It
-   must be one of ``'r'``, ``'rU'``, ``'U'`` and ``'rb'``.
+   With *mode* you can specify which file mode will be passed to :func:`open`.
+   It must be ``'r'`` or ``'rb'``.
 
    The *openhook*, when given, must be a function that takes two arguments,
    *filename* and *mode*, and returns an accordingly opened file-like object. You
@@ -166,15 +166,14 @@ available for subclassing as well:
    .. versionchanged:: 3.2
       Can be used as a context manager.
 
-   .. deprecated:: 3.4
-      The ``'rU'`` and ``'U'`` modes.
-
    .. deprecated:: 3.8
       Support for :meth:`__getitem__` method is deprecated.
 
    .. versionchanged:: 3.8
       The keyword parameter *mode* and *openhook* are now keyword-only.
 
+   .. versionchanged:: 3.9
+      The ``'rU'`` and ``'U'`` modes have been removed.
 
 
 **Optional in-place filtering:** if the keyword argument ``inplace=True`` is

Doc/library/functions.rst

@@ -1085,12 +1085,6 @@ are always available.  They are listed here in alphabetical order.
    first decoded using a platform-dependent encoding or using the specified
    *encoding* if given.
 
-   There is an additional mode character permitted, ``'U'``, which no longer
-   has any effect, and is considered deprecated. It previously enabled
-   :term:`universal newlines` in text mode, which became the default behaviour
-   in Python 3.0. Refer to the documentation of the
-   :ref:`newline <open-newline-parameter>` parameter for further details.
-
    .. note::
 
       Python doesn't depend on the underlying operating system's notion of text
@@ -1247,10 +1241,6 @@ are always available.  They are listed here in alphabetical order.
 
          * The file is now non-inheritable.
 
-   .. deprecated-removed:: 3.4 4.0
-
-      The ``'U'`` mode.
-
    .. versionchanged::
       3.5
 
@@ -1266,6 +1256,10 @@ are always available.  They are listed here in alphabetical order.
          * On Windows, opening a console buffer may return a subclass of
            :class:`io.RawIOBase` other than :class:`io.FileIO`.
 
+   .. versionchanged:: 3.9
+      The ``'U'`` mode has been removed.
+
+
 .. function:: ord(c)
 
    Given a string representing one Unicode character, return an integer

Doc/library/pathlib.rst

@@ -944,6 +944,19 @@ call fails (for example because the path doesn't exist).
    .. versionadded:: 3.5
 
 
+.. method:: Path.readlink()
+
+   Return the path to which the symbolic link points (as returned by
+   :func:`os.readlink`)::
+
+      >>> p = Path('mylink')
+      >>> p.symlink_to('setup.py')
+      >>> p.readlink()
+      PosixPath('setup.py')
+
+   .. versionadded:: 3.9
+
+
 .. method:: Path.rename(target)
 
    Rename this file or directory to the given *target*, and return a new Path
@@ -1153,6 +1166,7 @@ os and os.path                         pathlib
 :func:`os.path.isdir`                  :meth:`Path.is_dir`
 :func:`os.path.isfile`                 :meth:`Path.is_file`
 :func:`os.path.islink`                 :meth:`Path.is_symlink`
+:func:`os.readlink`                    :meth:`Path.readlink`
 :func:`os.stat`                        :meth:`Path.stat`,
                                        :meth:`Path.owner`,
                                        :meth:`Path.group`

Doc/library/sys.rst

@@ -33,18 +33,21 @@ always available.
    tuple of arguments. Native hooks added by :c:func:`PySys_AddAuditHook` are
    called first, followed by hooks added in the current interpreter.
 
-   Calling this function will trigger an event for all existing hooks, and if
-   any raise an exception derived from :class:`Exception`, the add will be
-   silently ignored. As a result, callers cannot assume that their hook has been
-   added unless they control all existing hooks.
+   .. audit-event:: sys.addaudithook "" sys.addaudithook
+
+      Raises a auditing event ``sys.addaudithook`` with no arguments. If any
+      existing hooks raise an exception derived from :class:`Exception`, the
+      new hook will not be added and the exception suppressed. As a result,
+      callers cannot assume that their hook has been added unless they control
+      all existing hooks.
 
    .. versionadded:: 3.8
 
    .. impl-detail::
 
-      When tracing is enabled, Python hooks are only traced if the callable has
-      a ``__cantrace__`` member that is set to a true value. Otherwise, trace
-      functions will not see the hook.
+      When tracing is enabled (see :func:`settrace`), Python hooks are only
+      traced if the callable has a ``__cantrace__`` member that is set to a
+      true value. Otherwise, trace functions will skip the hook.
 
 
 .. data:: argv
@@ -87,7 +90,7 @@ always available.
    native function is preferred when possible.
 
    See the :ref:`audit events table <audit-events>` for all events raised by
-   ``CPython``.
+   CPython.
 
    .. versionadded:: 3.8
 

Doc/library/types.rst

@@ -138,6 +138,10 @@ Standard names are defined for the following types:
 
    The type for code objects such as returned by :func:`compile`.
 
+   .. audit-event:: code.__new__ code,filename,name,argcount,posonlyargcount,kwonlyargcount,nlocals,stacksize,flags CodeType
+
+   Note that the audited arguments may not match the names or positions
+   required by the initializer.
 
 .. data:: CellType
 

Doc/reference/datamodel.rst

@@ -1716,7 +1716,7 @@ the descriptor defines :meth:`__set__` and/or :meth:`__delete__`, it is a data
 descriptor; if it defines neither, it is a non-data descriptor.  Normally, data
 descriptors define both :meth:`__get__` and :meth:`__set__`, while non-data
 descriptors have just the :meth:`__get__` method.  Data descriptors with
-:meth:`__set__` and :meth:`__get__` defined always override a redefinition in an
+:meth:`__get__` and :meth:`__set__` (and/or :meth:`__delete__`) defined always override a redefinition in an
 instance dictionary.  In contrast, non-data descriptors can be overridden by
 instances.
 

Doc/reference/simple_stmts.rst

@@ -839,6 +839,7 @@ the :ref:`relativeimports` section.
 :func:`importlib.import_module` is provided to support applications that
 determine dynamically the modules to be loaded.
 
+.. audit-event:: import module,filename,sys.path,sys.meta_path,sys.path_hooks import
 
 .. _future:
 

Doc/tools/static/switchers.js

@@ -23,6 +23,7 @@
       'fr': 'French',
       'ja': 'Japanese',
       'ko': 'Korean',
+      'pt-br': 'Brazilian Portuguese',
       'zh-cn': 'Simplified Chinese',
   };
 

Doc/tools/susp-ignored.csv

@@ -2,6 +2,7 @@ c-api/arg,,:ref,"PyArg_ParseTuple(args, ""O|O:ref"", &object, &callback)"
 c-api/list,,:high,list[low:high]
 c-api/sequence,,:i2,del o[i1:i2]
 c-api/sequence,,:i2,o[i1:i2]
+c-api/tuple,,:high,p[low:high]
 c-api/unicode,,:end,str[start:end]
 c-api/unicode,,:start,unicode[start:start+length]
 distutils/examples,267,`,This is the description of the ``foobar`` package.

Doc/tutorial/datastructures.rst

@@ -675,10 +675,10 @@ to a variable.  For example, ::
    >>> non_null
    'Trondheim'
 
-Note that in Python, unlike C, assignment cannot occur inside expressions. C
-programmers may grumble about this, but it avoids a common class of problems
-encountered in C programs: typing ``=`` in an expression when ``==`` was
-intended.
+Note that in Python, unlike C, assignment inside expressions must be done
+explicitly with the walrus operator ``:=``. This avoids a common class of
+problems encountered in C programs: typing ``=`` in an expression when ``==``
+was intended.
 
 
 .. _tut-comparing:

Doc/whatsnew/3.8.rst

@@ -595,6 +595,40 @@ The :func:`ast.parse` function has some new flags:
 asyncio
 -------
 
+:func:`asyncio.run` has graduated from the provisional to stable API. This
+function can be used to execute a :term:`coroutine` and return the result while
+automatically managing the event loop. For example::
+
+    import asyncio
+
+    async def main():
+        await asyncio.sleep(0)
+        return 42
+
+    asyncio.run(main())
+
+This is *roughly* equivalent to::
+
+    import asyncio
+
+    async def main():
+        await asyncio.sleep(0)
+        return 42
+
+    loop = asyncio.new_event_loop()
+    asyncio.set_event_loop(loop)
+    try:
+        loop.run_until_complete(main())
+    finally:
+        asyncio.set_event_loop(None)
+        loop.close()
+
+
+The actual implementation is significantly more complex. Thus,
+:func:`asyncio.run` should be the preferred way of running asyncio programs.
+
+(Contributed by Yury Selivanov in :issue:`32314`.)
+
 Running ``python -m asyncio`` launches a natively async REPL.  This allows rapid
 experimentation with code that has a top-level :keyword:`await`.  There is no
 longer a need to directly call ``asyncio.run()`` which would spawn a new event
@@ -612,6 +646,10 @@ loop on every invocation:
 
 (Contributed by Yury Selivanov in :issue:`37028`.)
 
+The exception :class:`asyncio.CancelledError` now inherits from
+:class:`BaseException` rather than :class:`Exception`.
+(Contributed by Yury Selivanov in :issue:`32528`.)
+
 On Windows, the default event loop is now :class:`~asyncio.ProactorEventLoop`.
 (Contributed by Victor Stinner in :issue:`34687`.)
 
@@ -622,6 +660,26 @@ On Windows, the default event loop is now :class:`~asyncio.ProactorEventLoop`.
 :exc:`KeyboardInterrupt` ("CTRL+C").
 (Contributed by Vladimir Matveev in :issue:`23057`.)
 
+Added :meth:`asyncio.Task.get_coro` for getting the wrapped coroutine
+within an :class:`asyncio.Task`.
+(Contributed by Alex Grönholm in :issue:`36999`.)
+
+Asyncio tasks can now be named, either by passing the ``name`` keyword
+argument to :func:`asyncio.create_task` or
+the :meth:`~asyncio.loop.create_task` event loop method, or by
+calling the :meth:`~asyncio.Task.set_name` method on the task object. The
+task name is visible in the ``repr()`` output of :class:`asyncio.Task` and
+can also be retrieved using the :meth:`~asyncio.Task.get_name` method.
+(Contributed by Alex Grönholm in :issue:`34270`.)
+
+Added support for
+`Happy Eyeballs <https://en.wikipedia.org/wiki/Happy_Eyeballs>`_ to
+:func:`asyncio.loop.create_connection`. To specify the behavior, two new
+parameters have been added: *happy_eyeballs_delay* and *interleave*. The Happy
+Eyeballs algorithm improves responsiveness in applications that support IPv4
+and IPv6 by attempting to simultaneously connect using both.
+(Contributed by twisteroid ambassador in :issue:`33530`.)
+
 
 builtins
 --------
@@ -1575,7 +1633,7 @@ Deprecated
 
 * Passing an object that is not an instance of
   :class:`concurrent.futures.ThreadPoolExecutor` to
-  :meth:`asyncio.loop.set_default_executor()` is
+  :meth:`loop.set_default_executor() <asyncio.loop.set_default_executor>` is
   deprecated and will be prohibited in Python 3.9.
   (Contributed by Elvis Pranskevichus in :issue:`34075`.)
 
@@ -1608,6 +1666,20 @@ Deprecated
   :keyword:`async def` instead.
   (Contributed by Andrew Svetlov in :issue:`36921`.)
 
+* In :mod:`asyncio`, the explicit passing of a *loop* argument has been
+  deprecated and will be removed in version 3.10 for the following:
+  :func:`asyncio.sleep`, :func:`asyncio.gather`, :func:`asyncio.shield`,
+  :func:`asyncio.wait_for`, :func:`asyncio.wait`, :func:`asyncio.as_completed`,
+  :class:`asyncio.Task`, :class:`asyncio.Lock`, :class:`asyncio.Event`,
+  :class:`asyncio.Condition`, :class:`asyncio.Semaphore`,
+  :class:`asyncio.BoundedSemaphore`, :class:`asyncio.Queue`,
+  :func:`asyncio.create_subprocess_exec`, and
+  :func:`asyncio.create_subprocess_shell`.
+
+* The explicit passing of coroutine objects to :func:`asyncio.wait` has been
+  deprecated and will be removed in version 3.11.
+  (Contributed by Yury Selivanov in :issue:`34790`.)
+
 * The following functions and methods are deprecated in the :mod:`gettext`
   module: :func:`~gettext.lgettext`, :func:`~gettext.ldgettext`,
   :func:`~gettext.lngettext` and :func:`~gettext.ldngettext`.
@@ -1852,13 +1924,6 @@ Changes in the Python API
   you adjust (possibly including adding accessor functions to the
   public API).  (See :issue:`35886`.)
 
-* Asyncio tasks can now be named, either by passing the ``name`` keyword
-  argument to :func:`asyncio.create_task` or
-  the :meth:`~asyncio.loop.create_task` event loop method, or by
-  calling the :meth:`~asyncio.Task.set_name` method on the task object. The
-  task name is visible in the ``repr()`` output of :class:`asyncio.Task` and
-  can also be retrieved using the :meth:`~asyncio.Task.get_name` method.
-
 * The :meth:`mmap.flush() <mmap.mmap.flush>` method now returns ``None`` on
   success and raises an exception on error under all platforms.  Previously,
   its behavior was platform-dependent: a nonzero value was returned on success;
@@ -1881,8 +1946,19 @@ Changes in the Python API
   (Contributed by Anthony Sottile in :issue:`36264`.)
 
 * The exception :class:`asyncio.CancelledError` now inherits from
-  :class:`BaseException` rather than a :class:`Exception`.
-  (Contributed by Yury Selivanov in :issue:`13528`.)
+  :class:`BaseException` rather than :class:`Exception`.
+  (Contributed by Yury Selivanov in :issue:`32528`.)
+
+* The function :func:`asyncio.wait_for` now correctly waits for cancellation
+  when using an instance of :class:`asyncio.Task`. Previously, upon reaching
+  *timeout*, it was cancelled and immediately returned.
+  (Contributed by Elvis Pranskevichus in :issue:`32751`.)
+
+* The function :func:`asyncio.BaseTransport.get_extra_info` now returns a safe
+  to use socket object when 'socket' is passed to the *name* parameter.
+  (Contributed by Yury Selivanov in :issue:`37027`.)
+
+* :class:`asyncio.BufferedProtocol` has graduated to the stable API.
 
 .. _bpo-36085-whatsnew:
 
@@ -1895,7 +1971,7 @@ Changes in the Python API
   resolution. If your application relies on these mechanisms, you should check
   for :func:`~os.add_dll_directory` and if it exists, use it to add your DLLs
   directory while loading your library. Note that Windows 7 users will need to
-  ensure that Windows Update KB2533625 has been installed (this is also verified
+  ensure that Windows Update KB2533623 has been installed (this is also verified
   by the installer).
   (Contributed by Steve Dower in :issue:`36085`.)
 

Doc/whatsnew/3.9.rst

@@ -125,8 +125,15 @@ that schedules a shutdown for the default executor that waits on the
 :func:`asyncio.run` has been updated to use the new :term:`coroutine`.
 (Contributed by Kyle Stanley in :issue:`34037`.)
 
+fcntl
+-----
+
+Added constants :data:`~fcntl.F_OFD_GETLK`, :data:`~fcntl.F_OFD_SETLK`
+and :data:`~fcntl.F_OFD_SETLKW`.
+(Contributed by Dong-hee Na in :issue:`38602`.)
+
 os
-__
+--
 
 Added :data:`~os.CLD_KILLED` and :data:`~os.CLD_STOPPED` for :attr:`si_code`.
 (Contributed by Dong-hee Na in :issue:`38493`.)
@@ -150,6 +157,13 @@ customization consistently by always using the value specified by
 case), and one used ``__VENV_NAME__`` instead.
 (Contributed by Brett Cannon in :issue:`37663`.)
 
+pathlib
+-------
+
+Added :meth:`~pathlib.Path.readlink()` which acts similar to
+:func:`~os.readlink`.
+(Contributed by Girts Folkmanis in :issue:`30618`)
+
 pprint
 ------
 
@@ -270,6 +284,14 @@ that may require changes to your code.
 Changes in the Python API
 -------------------------
 
+* :func:`open`, :func:`io.open`, :func:`codecs.open` and
+  :class:`fileinput.FileInput` no longer accept ``'U'`` ("universal newline")
+  in the file mode. This flag was deprecated since Python 3.3. In Python 3, the
+  "universal newline" is used by default when a file is open in text mode.  The
+  :ref:`newline parameter <open-newline-parameter>` of :func:`open` controls
+  how universal newlines works.
+  (Contributed by Victor Stinner in :issue:`37330`.)
+
 * :func:`__import__` and :func:`importlib.util.resolve_name` now raise
   :exc:`ImportError` where it previously raised :exc:`ValueError`. Callers
   catching the specific exception type and supporting both Python 3.9 and

Lib/_pyio.py

@@ -71,7 +71,6 @@ def open(file, mode="r", buffering=-1, encoding=None, errors=None,
     'b'       binary mode
     't'       text mode (default)
     '+'       open a disk file for updating (reading and writing)
-    'U'       universal newline mode (deprecated)
     ========= ===============================================================
 
     The default mode is 'rt' (open for reading text). For binary random
@@ -87,10 +86,6 @@ def open(file, mode="r", buffering=-1, encoding=None, errors=None,
     returned as strings, the bytes having been first decoded using a
     platform-dependent encoding or using the specified encoding if given.
 
-    'U' mode is deprecated and will raise an exception in future versions
-    of Python.  It has no effect in Python 3.  Use newline to control
-    universal newlines mode.
-
     buffering is an optional integer used to set the buffering policy.
     Pass 0 to switch buffering off (only allowed in binary mode), 1 to select
     line buffering (only usable in text mode), and an integer > 1 to indicate
@@ -176,7 +171,7 @@ def open(file, mode="r", buffering=-1, encoding=None, errors=None,
     if errors is not None and not isinstance(errors, str):
         raise TypeError("invalid errors: %r" % errors)
     modes = set(mode)
-    if modes - set("axrwb+tU") or len(mode) > len(modes):
+    if modes - set("axrwb+t") or len(mode) > len(modes):
         raise ValueError("invalid mode: %r" % mode)
     creating = "x" in modes
     reading = "r" in modes
@@ -185,13 +180,6 @@ def open(file, mode="r", buffering=-1, encoding=None, errors=None,
     updating = "+" in modes
     text = "t" in modes
     binary = "b" in modes
-    if "U" in modes:
-        if creating or writing or appending or updating:
-            raise ValueError("mode U cannot be combined with 'x', 'w', 'a', or '+'")
-        import warnings
-        warnings.warn("'U' mode is deprecated",
-                      DeprecationWarning, 2)
-        reading = True
     if text and binary:
         raise ValueError("can't have text and binary mode at once")
     if creating + reading + writing + appending > 1:

Lib/bisect.py

@@ -29,6 +29,7 @@ def bisect_right(a, x, lo=0, hi=None):
         hi = len(a)
     while lo < hi:
         mid = (lo+hi)//2
+        # Use __lt__ to match the logic in list.sort() and in heapq
         if x < a[mid]: hi = mid
         else: lo = mid+1
     return lo
@@ -63,6 +64,7 @@ def bisect_left(a, x, lo=0, hi=None):
         hi = len(a)
     while lo < hi:
         mid = (lo+hi)//2
+        # Use __lt__ to match the logic in list.sort() and in heapq
         if a[mid] < x: lo = mid+1
         else: hi = mid
     return lo

Lib/fileinput.py

@@ -209,15 +209,10 @@ class FileInput:
         self._isstdin = False
         self._backupfilename = None
         # restrict mode argument to reading modes
-        if mode not in ('r', 'rU', 'U', 'rb'):
-            raise ValueError("FileInput opening mode must be one of "
-                             "'r', 'rU', 'U' and 'rb'")
-        if 'U' in mode:
-            import warnings
-            warnings.warn("'U' mode is deprecated",
-                          DeprecationWarning, 2)
+        if mode not in ('r', 'rb'):
+            raise ValueError("FileInput opening mode must be 'r' or 'rb'")
         self._mode = mode
-        self._write_mode = mode.replace('r', 'w') if 'U' not in mode else 'w'
+        self._write_mode = mode.replace('r', 'w')
         if openhook:
             if inplace:
                 raise ValueError("FileInput cannot use an opening hook in inplace mode")

Lib/idlelib/NEWS.txt

@@ -1,7 +1,24 @@
-What's New in IDLE 3.8.0 (since 3.7.0)
-Released on 2019-10-20?
+What's New in IDLE 3.9.0 (since 3.8.0)
+Released on 2020-10-05?
 ======================================
 
+
+bpo-38598: Do not try to compile IDLE shell or output windows.
+
+
+What's New in IDLE 3.8.0 (since 3.7.0)
+Released on 2019-10-14
+======================================
+
+bpo-36698: IDLE no longer fails when writing non-encodable characters
+to stderr.  It now escapes them with a backslash, like the regular
+Python interpreter.  Add an errors field to the standard streams.
+
+bpo-13153: Improve tkinter's handing of non-BMP (astral) unicode
+characters, such as 'rocket \U0001f680'.  Whether a proper glyph or
+replacement char is displayed depends on the OS and font.  For IDLE,
+astral chars in code interfere with editing.
+
 bpo-35379: When exiting IDLE, catch any AttributeError.  One happens
 when EditorWindow.close is called twice.  Printing a traceback, when
 IDLE is run from a terminal, is useless and annoying.

Lib/idlelib/runscript.py

@@ -19,6 +19,7 @@ from idlelib.config import idleConf
 from idlelib import macosx
 from idlelib import pyshell
 from idlelib.query import CustomRun
+from idlelib import outwin
 
 indent_message = """Error: Inconsistent indentation detected!
 
@@ -46,6 +47,9 @@ class ScriptBinding:
             self.editwin.text_frame.bind('<<run-module-event-2>>', self._run_module_event)
 
     def check_module_event(self, event):
+        if isinstance(self.editwin, outwin.OutputWindow):
+            self.editwin.text.bell()
+            return 'break'
         filename = self.getfilename()
         if not filename:
             return 'break'
@@ -129,6 +133,9 @@ class ScriptBinding:
         module being executed and also add that directory to its
         sys.path if not already included.
         """
+        if isinstance(self.editwin, outwin.OutputWindow):
+            self.editwin.text.bell()
+            return 'break'
         filename = self.getfilename()
         if not filename:
             return 'break'

Lib/imp.py

@@ -225,7 +225,7 @@ def load_module(name, file, filename, details):
 
     """
     suffix, mode, type_ = details
-    if mode and (not mode.startswith(('r', 'U')) or '+' in mode):
+    if mode and (not mode.startswith('r') or '+' in mode):
         raise ValueError('invalid file open mode {!r}'.format(mode))
     elif file is None and type_ in {PY_SOURCE, PY_COMPILED}:
         msg = 'file object required for import (type code {})'.format(type_)

Lib/lib2to3/Grammar.txt

@@ -138,8 +138,8 @@ arglist: argument (',' argument)* [',']
 # that precede iterable unpackings are blocked; etc.
 argument: ( test [comp_for] |
             test '=' test |
-	    '**' expr |
-	    star_expr )
+            '**' test |
+	        '*' test )
 
 comp_iter: comp_for | comp_if
 comp_for: [ASYNC] 'for' exprlist 'in' testlist_safe [comp_iter]

Lib/lib2to3/fixes/fix_apply.py

@@ -37,10 +37,8 @@ class FixApply(fixer_base.BaseFix):
         # I feel like we should be able to express this logic in the
         # PATTERN above but I don't know how to do it so...
         if args:
-            if args.type == self.syms.star_expr:
-                return  # Make no change.
             if (args.type == self.syms.argument and
-                args.children[0].value == '**'):
+                args.children[0].value in {'**', '*'}):
                 return  # Make no change.
         if kwds and (kwds.type == self.syms.argument and
                      kwds.children[0].value == '**'):

Lib/lib2to3/fixes/fix_intern.py

@@ -30,10 +30,8 @@ class FixIntern(fixer_base.BaseFix):
             # PATTERN above but I don't know how to do it so...
             obj = results['obj']
             if obj:
-                if obj.type == self.syms.star_expr:
-                    return  # Make no change.
                 if (obj.type == self.syms.argument and
-                    obj.children[0].value == '**'):
+                    obj.children[0].value in {'**', '*'}):
                     return  # Make no change.
         names = ('sys', 'intern')
         new = ImportAndCall(node, results, names)

Lib/lib2to3/fixes/fix_reload.py

@@ -27,10 +27,8 @@ class FixReload(fixer_base.BaseFix):
             # PATTERN above but I don't know how to do it so...
             obj = results['obj']
             if obj:
-                if obj.type == self.syms.star_expr:
-                    return  # Make no change.
                 if (obj.type == self.syms.argument and
-                    obj.children[0].value == '**'):
+                    obj.children[0].value in {'**', '*'}):
                     return  # Make no change.
         names = ('importlib', 'reload')
         new = ImportAndCall(node, results, names)

Lib/lib2to3/tests/test_parser.py

@@ -253,6 +253,13 @@ class TestUnpackingGeneralizations(GrammarTest):
     def test_double_star_dict_literal_after_keywords(self):
         self.validate("""func(spam='fried', **{'eggs':'scrambled'})""")
 
+    def test_double_star_expression(self):
+        self.validate("""func(**{'a':2} or {})""")
+        self.validate("""func(**() or {})""")
+
+    def test_star_expression(self):
+        self.validate("""func(*[] or [2])""")
+
     def test_list_display(self):
         self.validate("""[*{2}, 3, *[4]]""")
 

Lib/pathlib.py

@@ -1244,6 +1244,15 @@ class Path(PurePath):
         with self.open(mode='w', encoding=encoding, errors=errors) as f:
             return f.write(data)
 
+    def readlink(self):
+        """
+        Return the path to which the symbolic link points.
+        """
+        path = self._accessor.readlink(self)
+        obj = self._from_parts((path,), init=False)
+        obj._init(template=self)
+        return obj
+
     def touch(self, mode=0o666, exist_ok=True):
         """
         Create this file with the given access mode, if it doesn't exist.

Lib/test/test_ast.py

@@ -129,11 +129,11 @@ exec_tests = [
     # Asynchronous comprehensions
     "async def f():\n [i async for b in c]",
     # Decorated FunctionDef
-    "@deco1\n@deco2()\ndef f(): pass",
+    "@deco1\n@deco2()\n@deco3(1)\ndef f(): pass",
     # Decorated AsyncFunctionDef
-    "@deco1\n@deco2()\nasync def f(): pass",
+    "@deco1\n@deco2()\n@deco3(1)\nasync def f(): pass",
     # Decorated ClassDef
-    "@deco1\n@deco2()\nclass C: pass",
+    "@deco1\n@deco2()\n@deco3(1)\nclass C: pass",
     # Decorator with generator argument
     "@deco(a for a in b)\ndef f(): pass",
     # Simple assignment expression
@@ -1870,9 +1870,9 @@ exec_results = [
 ('Module', [('Expr', (1, 0), ('Dict', (1, 0), [None, ('Constant', (1, 10), 2, None)], [('Dict', (1, 3), [('Constant', (1, 4), 1, None)], [('Constant', (1, 6), 2, None)]), ('Constant', (1, 12), 3, None)]))], []),
 ('Module', [('Expr', (1, 0), ('Set', (1, 0), [('Starred', (1, 1), ('Set', (1, 2), [('Constant', (1, 3), 1, None), ('Constant', (1, 6), 2, None)]), ('Load',)), ('Constant', (1, 10), 3, None)]))], []),
 ('Module', [('AsyncFunctionDef', (1, 0), 'f', ('arguments', [], [], None, [], [], None, []), [('Expr', (2, 1), ('ListComp', (2, 1), ('Name', (2, 2), 'i', ('Load',)), [('comprehension', ('Name', (2, 14), 'b', ('Store',)), ('Name', (2, 19), 'c', ('Load',)), [], 1)]))], [], None, None)], []),
-('Module', [('FunctionDef', (3, 0), 'f', ('arguments', [], [], None, [], [], None, []), [('Pass', (3, 9))], [('Name', (1, 1), 'deco1', ('Load',)), ('Call', (2, 0), ('Name', (2, 1), 'deco2', ('Load',)), [], [])], None, None)], []),
-('Module', [('AsyncFunctionDef', (3, 0), 'f', ('arguments', [], [], None, [], [], None, []), [('Pass', (3, 15))], [('Name', (1, 1), 'deco1', ('Load',)), ('Call', (2, 0), ('Name', (2, 1), 'deco2', ('Load',)), [], [])], None, None)], []),
-('Module', [('ClassDef', (3, 0), 'C', [], [], [('Pass', (3, 9))], [('Name', (1, 1), 'deco1', ('Load',)), ('Call', (2, 0), ('Name', (2, 1), 'deco2', ('Load',)), [], [])])], []),
+('Module', [('FunctionDef', (4, 0), 'f', ('arguments', [], [], None, [], [], None, []), [('Pass', (4, 9))], [('Name', (1, 1), 'deco1', ('Load',)), ('Call', (2, 1), ('Name', (2, 1), 'deco2', ('Load',)), [], []), ('Call', (3, 1), ('Name', (3, 1), 'deco3', ('Load',)), [('Constant', (3, 7), 1, None)], [])], None, None)], []),
+('Module', [('AsyncFunctionDef', (4, 0), 'f', ('arguments', [], [], None, [], [], None, []), [('Pass', (4, 15))], [('Name', (1, 1), 'deco1', ('Load',)), ('Call', (2, 1), ('Name', (2, 1), 'deco2', ('Load',)), [], []), ('Call', (3, 1), ('Name', (3, 1), 'deco3', ('Load',)), [('Constant', (3, 7), 1, None)], [])], None, None)], []),
+('Module', [('ClassDef', (4, 0), 'C', [], [], [('Pass', (4, 9))], [('Name', (1, 1), 'deco1', ('Load',)), ('Call', (2, 1), ('Name', (2, 1), 'deco2', ('Load',)), [], []), ('Call', (3, 1), ('Name', (3, 1), 'deco3', ('Load',)), [('Constant', (3, 7), 1, None)], [])])], []),
 ('Module', [('FunctionDef', (2, 0), 'f', ('arguments', [], [], None, [], [], None, []), [('Pass', (2, 9))], [('Call', (1, 1), ('Name', (1, 1), 'deco', ('Load',)), [('GeneratorExp', (1, 5), ('Name', (1, 6), 'a', ('Load',)), [('comprehension', ('Name', (1, 12), 'a', ('Store',)), ('Name', (1, 17), 'b', ('Load',)), [], 0)])], [])], None, None)], []),
 ('Module', [('Expr', (1, 0), ('NamedExpr', (1, 1), ('Name', (1, 1), 'a', ('Store',)), ('Constant', (1, 6), 1, None)))], []),
 ('Module', [('FunctionDef', (1, 0), 'f', ('arguments', [('arg', (1, 6), 'a', None, None)], [], None, [], [], None, []), [('Pass', (1, 14))], [], None, None)], []),

Lib/test/test_codecs.py

@@ -712,11 +712,23 @@ class UTF16Test(ReadTest, unittest.TestCase):
         self.addCleanup(support.unlink, support.TESTFN)
         with open(support.TESTFN, 'wb') as fp:
             fp.write(s)
-        with support.check_warnings(('', DeprecationWarning)):
-            reader = codecs.open(support.TESTFN, 'U', encoding=self.encoding)
-        with reader:
+        with codecs.open(support.TESTFN, 'r',
+                         encoding=self.encoding) as reader:
             self.assertEqual(reader.read(), s1)
 
+    def test_invalid_modes(self):
+        for mode in ('U', 'rU', 'r+U'):
+            with self.assertRaises(ValueError) as cm:
+                codecs.open(support.TESTFN, mode, encoding=self.encoding)
+            self.assertIn('invalid mode', str(cm.exception))
+
+        for mode in ('rt', 'wt', 'at', 'r+t'):
+            with self.assertRaises(ValueError) as cm:
+                codecs.open(support.TESTFN, mode, encoding=self.encoding)
+            self.assertIn("can't have text and binary mode at once",
+                          str(cm.exception))
+
+
 class UTF16LETest(ReadTest, unittest.TestCase):
     encoding = "utf-16-le"
     ill_formed_sequence = b"\x80\xdc"

Lib/test/test_fileinput.py

@@ -226,19 +226,11 @@ class FileInputTests(BaseTests, unittest.TestCase):
         self.assertEqual(fi.fileno(), -1)
 
     def test_opening_mode(self):
-        try:
-            # invalid mode, should raise ValueError
-            fi = FileInput(mode="w")
-            self.fail("FileInput should reject invalid mode argument")
-        except ValueError:
-            pass
-        # try opening in universal newline mode
-        t1 = self.writeTmp(b"A\nB\r\nC\rD", mode="wb")
-        with check_warnings(('', DeprecationWarning)):
-            fi = FileInput(files=t1, mode="U")
-        with check_warnings(('', DeprecationWarning)):
-            lines = list(fi)
-        self.assertEqual(lines, ["A\n", "B\n", "C\n", "D"])
+        # invalid modes
+        for mode in ('w', 'rU', 'U'):
+            with self.subTest(mode=mode):
+                with self.assertRaises(ValueError):
+                    FileInput(mode=mode)
 
     def test_stdin_binary_mode(self):
         with mock.patch('sys.stdin') as m_stdin:
@@ -985,10 +977,6 @@ class Test_hook_encoded(unittest.TestCase):
             self.assertEqual(lines, expected_lines)
 
         check('r', ['A\n', 'B\n', 'C\n', 'D\u20ac'])
-        with self.assertWarns(DeprecationWarning):
-            check('rU', ['A\n', 'B\n', 'C\n', 'D\u20ac'])
-        with self.assertWarns(DeprecationWarning):
-            check('U', ['A\n', 'B\n', 'C\n', 'D\u20ac'])
         with self.assertRaises(ValueError):
             check('rb', ['A\n', 'B\r\n', 'C\r', 'D\u20ac'])
 

Lib/test/test_io.py

@@ -3886,16 +3886,6 @@ class MiscIOTest(unittest.TestCase):
         self.assertEqual(f.mode, "wb")
         f.close()
 
-        with support.check_warnings(('', DeprecationWarning)):
-            f = self.open(support.TESTFN, "U")
-        self.assertEqual(f.name,            support.TESTFN)
-        self.assertEqual(f.buffer.name,     support.TESTFN)
-        self.assertEqual(f.buffer.raw.name, support.TESTFN)
-        self.assertEqual(f.mode,            "U")
-        self.assertEqual(f.buffer.mode,     "rb")
-        self.assertEqual(f.buffer.raw.mode, "rb")
-        f.close()
-
         f = self.open(support.TESTFN, "w+")
         self.assertEqual(f.mode,            "w+")
         self.assertEqual(f.buffer.mode,     "rb+") # Does it really matter?
@@ -3909,6 +3899,13 @@ class MiscIOTest(unittest.TestCase):
         f.close()
         g.close()
 
+    def test_removed_u_mode(self):
+        # "U" mode has been removed in Python 3.9
+        for mode in ("U", "rU", "r+U"):
+            with self.assertRaises(ValueError) as cm:
+                self.open(support.TESTFN, mode)
+            self.assertIn('invalid mode', str(cm.exception))
+
     def test_io_after_close(self):
         for kwargs in [
                 {"mode": "w"},

Lib/test/test_pathlib.py

@@ -1812,6 +1812,16 @@ class _BasePathTest(object):
         self.assertEqual(os.stat(r).st_size, size)
         self.assertFileNotFound(q.stat)
 
+    @support.skip_unless_symlink
+    def test_readlink(self):
+        P = self.cls(BASE)
+        self.assertEqual((P / 'linkA').readlink(), self.cls('fileA'))
+        self.assertEqual((P / 'brokenLink').readlink(),
+                         self.cls('non-existing'))
+        self.assertEqual((P / 'linkB').readlink(), self.cls('dirB'))
+        with self.assertRaises(OSError):
+            (P / 'fileA').readlink()
+
     def test_touch_common(self):
         P = self.cls(BASE)
         p = P / 'newfileA'

Lib/test/test_urllib.py

@@ -1551,85 +1551,6 @@ class URLopener_Tests(FakeHTTPMixin, unittest.TestCase):
             self.assertRaises(OSError, DummyURLopener().retrieve, url)
 
 
-# Just commented them out.
-# Can't really tell why keep failing in windows and sparc.
-# Everywhere else they work ok, but on those machines, sometimes
-# fail in one of the tests, sometimes in other. I have a linux, and
-# the tests go ok.
-# If anybody has one of the problematic environments, please help!
-# .   Facundo
-#
-# def server(evt):
-#     import socket, time
-#     serv = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
-#     serv.settimeout(3)
-#     serv.setsockopt(socket.SOL_SOCKET, socket.SO_REUSEADDR, 1)
-#     serv.bind(("", 9093))
-#     serv.listen()
-#     try:
-#         conn, addr = serv.accept()
-#         conn.send("1 Hola mundo\n")
-#         cantdata = 0
-#         while cantdata < 13:
-#             data = conn.recv(13-cantdata)
-#             cantdata += len(data)
-#             time.sleep(.3)
-#         conn.send("2 No more lines\n")
-#         conn.close()
-#     except socket.timeout:
-#         pass
-#     finally:
-#         serv.close()
-#         evt.set()
-#
-# class FTPWrapperTests(unittest.TestCase):
-#
-#     def setUp(self):
-#         import ftplib, time, threading
-#         ftplib.FTP.port = 9093
-#         self.evt = threading.Event()
-#         threading.Thread(target=server, args=(self.evt,)).start()
-#         time.sleep(.1)
-#
-#     def tearDown(self):
-#         self.evt.wait()
-#
-#     def testBasic(self):
-#         # connects
-#         ftp = urllib.ftpwrapper("myuser", "mypass", "localhost", 9093, [])
-#         ftp.close()
-#
-#     def testTimeoutNone(self):
-#         # global default timeout is ignored
-#         import socket
-#         self.assertIsNone(socket.getdefaulttimeout())
-#         socket.setdefaulttimeout(30)
-#         try:
-#             ftp = urllib.ftpwrapper("myuser", "mypass", "localhost", 9093, [])
-#         finally:
-#             socket.setdefaulttimeout(None)
-#         self.assertEqual(ftp.ftp.sock.gettimeout(), 30)
-#         ftp.close()
-#
-#     def testTimeoutDefault(self):
-#         # global default timeout is used
-#         import socket
-#         self.assertIsNone(socket.getdefaulttimeout())
-#         socket.setdefaulttimeout(30)
-#         try:
-#             ftp = urllib.ftpwrapper("myuser", "mypass", "localhost", 9093, [])
-#         finally:
-#             socket.setdefaulttimeout(None)
-#         self.assertEqual(ftp.ftp.sock.gettimeout(), 30)
-#         ftp.close()
-#
-#     def testTimeoutValue(self):
-#         ftp = urllib.ftpwrapper("myuser", "mypass", "localhost", 9093, [],
-#                                 timeout=30)
-#         self.assertEqual(ftp.ftp.sock.gettimeout(), 30)
-#         ftp.close()
-
-
 class RequestTests(unittest.TestCase):
     """Unit tests for urllib.request.Request."""
 

Lib/test/test_zipfile.py

@@ -1,6 +1,7 @@
 import contextlib
 import importlib.util
 import io
+import itertools
 import os
 import pathlib
 import posixpath
@@ -824,6 +825,227 @@ class StoredTestZip64InSmallFiles(AbstractTestZip64InSmallFiles,
             zinfo = zipfp.getinfo("strfile")
             self.assertEqual(zinfo.extra, extra)
 
+    def make_zip64_file(
+        self, file_size_64_set=False, file_size_extra=False,
+        compress_size_64_set=False, compress_size_extra=False,
+        header_offset_64_set=False, header_offset_extra=False,
+    ):
+        """Generate bytes sequence for a zip with (incomplete) zip64 data.
+
+        The actual values (not the zip 64 0xffffffff values) stored in the file
+        are:
+        file_size: 8
+        compress_size: 8
+        header_offset: 0
+        """
+        actual_size = 8
+        actual_header_offset = 0
+        local_zip64_fields = []
+        central_zip64_fields = []
+
+        file_size = actual_size
+        if file_size_64_set:
+            file_size = 0xffffffff
+            if file_size_extra:
+                local_zip64_fields.append(actual_size)
+                central_zip64_fields.append(actual_size)
+        file_size = struct.pack("<L", file_size)
+
+        compress_size = actual_size
+        if compress_size_64_set:
+            compress_size = 0xffffffff
+            if compress_size_extra:
+                local_zip64_fields.append(actual_size)
+                central_zip64_fields.append(actual_size)
+        compress_size = struct.pack("<L", compress_size)
+
+        header_offset = actual_header_offset
+        if header_offset_64_set:
+            header_offset = 0xffffffff
+            if header_offset_extra:
+                central_zip64_fields.append(actual_header_offset)
+        header_offset = struct.pack("<L", header_offset)
+
+        local_extra = struct.pack(
+            '<HH' + 'Q'*len(local_zip64_fields),
+            0x0001,
+            8*len(local_zip64_fields),
+            *local_zip64_fields
+        )
+
+        central_extra = struct.pack(
+            '<HH' + 'Q'*len(central_zip64_fields),
+            0x0001,
+            8*len(central_zip64_fields),
+            *central_zip64_fields
+        )
+
+        central_dir_size = struct.pack('<Q', 58 + 8 * len(central_zip64_fields))
+        offset_to_central_dir = struct.pack('<Q', 50 + 8 * len(local_zip64_fields))
+
+        local_extra_length = struct.pack("<H", 4 + 8 * len(local_zip64_fields))
+        central_extra_length = struct.pack("<H", 4 + 8 * len(central_zip64_fields))
+
+        filename = b"test.txt"
+        content = b"test1234"
+        filename_length = struct.pack("<H", len(filename))
+        zip64_contents = (
+            # Local file header
+            b"PK\x03\x04\x14\x00\x00\x00\x00\x00\x00\x00!\x00\x9e%\xf5\xaf"
+            + compress_size
+            + file_size
+            + filename_length
+            + local_extra_length
+            + filename
+            + local_extra
+            + content
+            # Central directory:
+            + b"PK\x01\x02-\x03-\x00\x00\x00\x00\x00\x00\x00!\x00\x9e%\xf5\xaf"
+            + compress_size
+            + file_size
+            + filename_length
+            + central_extra_length
+            + b"\x00\x00\x00\x00\x00\x00\x00\x00\x80\x01"
+            + header_offset
+            + filename
+            + central_extra
+            # Zip64 end of central directory
+            + b"PK\x06\x06,\x00\x00\x00\x00\x00\x00\x00-\x00-"
+            + b"\x00\x00\x00\x00\x00\x00\x00\x00\x00\x01\x00\x00\x00\x00\x00"
+            + b"\x00\x00\x01\x00\x00\x00\x00\x00\x00\x00"
+            + central_dir_size
+            + offset_to_central_dir
+            # Zip64 end of central directory locator
+            + b"PK\x06\x07\x00\x00\x00\x00l\x00\x00\x00\x00\x00\x00\x00\x01"
+            + b"\x00\x00\x00"
+            # end of central directory
+            + b"PK\x05\x06\x00\x00\x00\x00\x01\x00\x01\x00:\x00\x00\x002\x00"
+            + b"\x00\x00\x00\x00"
+        )
+        return zip64_contents
+
+    def test_bad_zip64_extra(self):
+        """Missing zip64 extra records raises an exception.
+
+        There are 4 fields that the zip64 format handles (the disk number is
+        not used in this module and so is ignored here). According to the zip
+        spec:
+              The order of the fields in the zip64 extended
+              information record is fixed, but the fields MUST
+              only appear if the corresponding Local or Central
+              directory record field is set to 0xFFFF or 0xFFFFFFFF.
+
+        If the zip64 extra content doesn't contain enough entries for the
+        number of fields marked with 0xFFFF or 0xFFFFFFFF, we raise an error.
+        This test mismatches the length of the zip64 extra field and the number
+        of fields set to indicate the presence of zip64 data.
+        """
+        # zip64 file size present, no fields in extra, expecting one, equals
+        # missing file size.
+        missing_file_size_extra = self.make_zip64_file(
+            file_size_64_set=True,
+        )
+        with self.assertRaises(zipfile.BadZipFile) as e:
+            zipfile.ZipFile(io.BytesIO(missing_file_size_extra))
+        self.assertIn('file size', str(e.exception).lower())
+
+        # zip64 file size present, zip64 compress size present, one field in
+        # extra, expecting two, equals missing compress size.
+        missing_compress_size_extra = self.make_zip64_file(
+            file_size_64_set=True,
+            file_size_extra=True,
+            compress_size_64_set=True,
+        )
+        with self.assertRaises(zipfile.BadZipFile) as e:
+            zipfile.ZipFile(io.BytesIO(missing_compress_size_extra))
+        self.assertIn('compress size', str(e.exception).lower())
+
+        # zip64 compress size present, no fields in extra, expecting one,
+        # equals missing compress size.
+        missing_compress_size_extra = self.make_zip64_file(
+            compress_size_64_set=True,
+        )
+        with self.assertRaises(zipfile.BadZipFile) as e:
+            zipfile.ZipFile(io.BytesIO(missing_compress_size_extra))
+        self.assertIn('compress size', str(e.exception).lower())
+
+        # zip64 file size present, zip64 compress size present, zip64 header
+        # offset present, two fields in extra, expecting three, equals missing
+        # header offset
+        missing_header_offset_extra = self.make_zip64_file(
+            file_size_64_set=True,
+            file_size_extra=True,
+            compress_size_64_set=True,
+            compress_size_extra=True,
+            header_offset_64_set=True,
+        )
+        with self.assertRaises(zipfile.BadZipFile) as e:
+            zipfile.ZipFile(io.BytesIO(missing_header_offset_extra))
+        self.assertIn('header offset', str(e.exception).lower())
+
+        # zip64 compress size present, zip64 header offset present, one field
+        # in extra, expecting two, equals missing header offset
+        missing_header_offset_extra = self.make_zip64_file(
+            file_size_64_set=False,
+            compress_size_64_set=True,
+            compress_size_extra=True,
+            header_offset_64_set=True,
+        )
+        with self.assertRaises(zipfile.BadZipFile) as e:
+            zipfile.ZipFile(io.BytesIO(missing_header_offset_extra))
+        self.assertIn('header offset', str(e.exception).lower())
+
+        # zip64 file size present, zip64 header offset present, one field in
+        # extra, expecting two, equals missing header offset
+        missing_header_offset_extra = self.make_zip64_file(
+            file_size_64_set=True,
+            file_size_extra=True,
+            compress_size_64_set=False,
+            header_offset_64_set=True,
+        )
+        with self.assertRaises(zipfile.BadZipFile) as e:
+            zipfile.ZipFile(io.BytesIO(missing_header_offset_extra))
+        self.assertIn('header offset', str(e.exception).lower())
+
+        # zip64 header offset present, no fields in extra, expecting one,
+        # equals missing header offset
+        missing_header_offset_extra = self.make_zip64_file(
+            file_size_64_set=False,
+            compress_size_64_set=False,
+            header_offset_64_set=True,
+        )
+        with self.assertRaises(zipfile.BadZipFile) as e:
+            zipfile.ZipFile(io.BytesIO(missing_header_offset_extra))
+        self.assertIn('header offset', str(e.exception).lower())
+
+    def test_generated_valid_zip64_extra(self):
+        # These values are what is set in the make_zip64_file method.
+        expected_file_size = 8
+        expected_compress_size = 8
+        expected_header_offset = 0
+        expected_content = b"test1234"
+
+        # Loop through the various valid combinations of zip64 masks
+        # present and extra fields present.
+        params = (
+            {"file_size_64_set": True, "file_size_extra": True},
+            {"compress_size_64_set": True, "compress_size_extra": True},
+            {"header_offset_64_set": True, "header_offset_extra": True},
+        )
+
+        for r in range(1, len(params) + 1):
+            for combo in itertools.combinations(params, r):
+                kwargs = {}
+                for c in combo:
+                    kwargs.update(c)
+                with zipfile.ZipFile(io.BytesIO(self.make_zip64_file(**kwargs))) as zf:
+                    zinfo = zf.infolist()[0]
+                    self.assertEqual(zinfo.file_size, expected_file_size)
+                    self.assertEqual(zinfo.compress_size, expected_compress_size)
+                    self.assertEqual(zinfo.header_offset, expected_header_offset)
+                    self.assertEqual(zf.read(zinfo), expected_content)
+
+
 @requires_zlib
 class DeflateTestZip64InSmallFiles(AbstractTestZip64InSmallFiles,
                                    unittest.TestCase):
@@ -1934,6 +2156,44 @@ class DecryptionTests(unittest.TestCase):
         self.assertRaises(TypeError, self.zip.open, "test.txt", pwd="python")
         self.assertRaises(TypeError, self.zip.extract, "test.txt", pwd="python")
 
+    def test_seek_tell(self):
+        self.zip.setpassword(b"python")
+        txt = self.plain
+        test_word = b'encryption'
+        bloc = txt.find(test_word)
+        bloc_len = len(test_word)
+        with self.zip.open("test.txt", "r") as fp:
+            fp.seek(bloc, os.SEEK_SET)
+            self.assertEqual(fp.tell(), bloc)
+            fp.seek(-bloc, os.SEEK_CUR)
+            self.assertEqual(fp.tell(), 0)
+            fp.seek(bloc, os.SEEK_CUR)
+            self.assertEqual(fp.tell(), bloc)
+            self.assertEqual(fp.read(bloc_len), txt[bloc:bloc+bloc_len])
+
+            # Make sure that the second read after seeking back beyond
+            # _readbuffer returns the same content (ie. rewind to the start of
+            # the file to read forward to the required position).
+            old_read_size = fp.MIN_READ_SIZE
+            fp.MIN_READ_SIZE = 1
+            fp._readbuffer = b''
+            fp._offset = 0
+            fp.seek(0, os.SEEK_SET)
+            self.assertEqual(fp.tell(), 0)
+            fp.seek(bloc, os.SEEK_CUR)
+            self.assertEqual(fp.read(bloc_len), txt[bloc:bloc+bloc_len])
+            fp.MIN_READ_SIZE = old_read_size
+
+            fp.seek(0, os.SEEK_END)
+            self.assertEqual(fp.tell(), len(txt))
+            fp.seek(0, os.SEEK_SET)
+            self.assertEqual(fp.tell(), 0)
+
+            # Read the file completely to definitely call any eof integrity
+            # checks (crc) and make sure they still pass.
+            fp.read()
+
+
 class AbstractTestsWithRandomBinaryFiles:
     @classmethod
     def setUpClass(cls):

Lib/zipfile.py

@@ -480,14 +480,26 @@ class ZipInfo (object):
 
                 # ZIP64 extension (large files and/or large archives)
                 if self.file_size in (0xffffffffffffffff, 0xffffffff):
+                    if len(counts) <= idx:
+                        raise BadZipFile(
+                            "Corrupt zip64 extra field. File size not found."
+                        )
                     self.file_size = counts[idx]
                     idx += 1
 
                 if self.compress_size == 0xFFFFFFFF:
+                    if len(counts) <= idx:
+                        raise BadZipFile(
+                            "Corrupt zip64 extra field. Compress size not found."
+                        )
                     self.compress_size = counts[idx]
                     idx += 1
 
                 if self.header_offset == 0xffffffff:
+                    if len(counts) <= idx:
+                        raise BadZipFile(
+                            "Corrupt zip64 extra field. Header offset not found."
+                        )
                     old = self.header_offset
                     self.header_offset = counts[idx]
                     idx+=1
@@ -792,10 +804,10 @@ class ZipExtFile(io.BufferedIOBase):
     # Chunk size to read during seek
     MAX_SEEK_READ = 1 << 24
 
-    def __init__(self, fileobj, mode, zipinfo, decrypter=None,
+    def __init__(self, fileobj, mode, zipinfo, pwd=None,
                  close_fileobj=False):
         self._fileobj = fileobj
-        self._decrypter = decrypter
+        self._pwd = pwd
         self._close_fileobj = close_fileobj
 
         self._compress_type = zipinfo.compress_type
@@ -810,11 +822,6 @@ class ZipExtFile(io.BufferedIOBase):
 
         self.newlines = None
 
-        # Adjust read size for encrypted files since the first 12 bytes
-        # are for the encryption/password information.
-        if self._decrypter is not None:
-            self._compress_left -= 12
-
         self.mode = mode
         self.name = zipinfo.filename
 
@@ -835,6 +842,30 @@ class ZipExtFile(io.BufferedIOBase):
         except AttributeError:
             pass
 
+        self._decrypter = None
+        if pwd:
+            if zipinfo.flag_bits & 0x8:
+                # compare against the file type from extended local headers
+                check_byte = (zipinfo._raw_time >> 8) & 0xff
+            else:
+                # compare against the CRC otherwise
+                check_byte = (zipinfo.CRC >> 24) & 0xff
+            h = self._init_decrypter()
+            if h != check_byte:
+                raise RuntimeError("Bad password for file %r" % zipinfo.orig_filename)
+
+
+    def _init_decrypter(self):
+        self._decrypter = _ZipDecrypter(self._pwd)
+        # The first 12 bytes in the cypher stream is an encryption header
+        #  used to strengthen the algorithm. The first 11 bytes are
+        #  completely random, while the 12th contains the MSB of the CRC,
+        #  or the MSB of the file time depending on the header type
+        #  and is used to check the correctness of the password.
+        header = self._fileobj.read(12)
+        self._compress_left -= 12
+        return self._decrypter(header)[11]
+
     def __repr__(self):
         result = ['<%s.%s' % (self.__class__.__module__,
                               self.__class__.__qualname__)]
@@ -1061,6 +1092,8 @@ class ZipExtFile(io.BufferedIOBase):
             self._decompressor = _get_decompressor(self._compress_type)
             self._eof = False
             read_offset = new_pos
+            if self._decrypter is not None:
+                self._init_decrypter()
 
         while read_offset > 0:
             read_len = min(self.MAX_SEEK_READ, read_offset)
@@ -1524,32 +1557,16 @@ class ZipFile:
 
             # check for encrypted flag & handle password
             is_encrypted = zinfo.flag_bits & 0x1
-            zd = None
             if is_encrypted:
                 if not pwd:
                     pwd = self.pwd
                 if not pwd:
                     raise RuntimeError("File %r is encrypted, password "
                                        "required for extraction" % name)
-
-                zd = _ZipDecrypter(pwd)
-                # The first 12 bytes in the cypher stream is an encryption header
-                #  used to strengthen the algorithm. The first 11 bytes are
-                #  completely random, while the 12th contains the MSB of the CRC,
-                #  or the MSB of the file time depending on the header type
-                #  and is used to check the correctness of the password.
-                header = zef_file.read(12)
-                h = zd(header[0:12])
-                if zinfo.flag_bits & 0x8:
-                    # compare against the file type from extended local headers
-                    check_byte = (zinfo._raw_time >> 8) & 0xff
             else:
-                    # compare against the CRC otherwise
-                    check_byte = (zinfo.CRC >> 24) & 0xff
-                if h[11] != check_byte:
-                    raise RuntimeError("Bad password for file %r" % name)
+                pwd = None
 
-            return ZipExtFile(zef_file, mode, zinfo, zd, True)
+            return ZipExtFile(zef_file, mode, zinfo, pwd, True)
         except:
             zef_file.close()
             raise

Mac/BuildScript/resources/License.rtf

@@ -16,7 +16,7 @@ Python was created in the early 1990s by Guido van Rossum at Stichting Mathemati
 \
 In 1995, Guido continued his work on Python at the Corporation for National Research Initiatives (CNRI, see http://www.cnri.reston.va.us) in Reston, Virginia where he released several versions of the software.\
 \
-In May 2000, Guido and the Python core development team moved to BeOpen.com to form the BeOpen PythonLabs team. In October of the same year, the PythonLabs team moved to Digital Creations (now Zope Corporation, see http://www.zope.com). In 2001, the Python Software Foundation (PSF, see http://www.python.org/psf/) was formed, a non-profit organization created specifically to own Python-related Intellectual Property. Zope Corporation is a sponsoring member of the PSF.\
+In May 2000, Guido and the Python core development team moved to BeOpen.com to form the BeOpen PythonLabs team. In October of the same year, the PythonLabs team moved to Digital Creations (now Zope Corporation, see http://www.zope.org). In 2001, the Python Software Foundation (PSF, see http://www.python.org/psf/) was formed, a non-profit organization created specifically to own Python-related Intellectual Property. Zope Corporation is a sponsoring member of the PSF.\
 \
 All Python releases are Open Source (see http://www.opensource.org for the Open Source Definition). Historically, most, but not all, Python releases have also been GPL-compatible; the table below summarizes the various releases.\
 \

Misc/NEWS.d/next/Core and Builtins/2019-06-18-17-53-06.bpo-37330.wAvHmz.rst

@@ -0,0 +1,3 @@
+:func:`open`, :func:`io.open`, :func:`codecs.open` and
+:class:`fileinput.FileInput` no longer accept ``'U'`` ("universal newline") in
+the file mode. This flag was deprecated since Python 3.3.

Misc/NEWS.d/next/Core and Builtins/2019-10-20-12-43-48.bpo-38535.ESMkVN.rst

@@ -0,0 +1,2 @@
+Fixed line numbers and column offsets for AST nodes for calls without
+arguments in decorators.

Misc/NEWS.d/next/Documentation/2019-10-26-13-19-07.bpo-38592.Y96BYO.rst

@@ -0,0 +1 @@
+Add Brazilian Portuguese to the language switcher at Python Documentation website.

Misc/NEWS.d/next/IDLE/2019-10-26-18-16-24.bpo-38598.6kH9FY.rst

@@ -0,0 +1 @@
+Do not try to compile IDLE shell or output windows

Misc/NEWS.d/next/Library/2018-04-24-13-18-48.bpo-33348.XaJDei.rst

@@ -0,0 +1,2 @@
+lib2to3 now recognizes expressions after ``*`` and `**` like in ``f(*[] or
+[])``.

Misc/NEWS.d/next/Library/2018-07-14-13-48-56.bpo-30618.T5AUF6.rst

@@ -0,0 +1 @@
+Add :meth:`~pathlib.Path.readlink`. Patch by Girts Folkmanis.

Misc/NEWS.d/next/Library/2019-07-09-05-44-39.bpo-36993.4javqu.rst

@@ -0,0 +1,2 @@
+Improve error reporting for corrupt zip files with bad zip64 extra data. Patch
+by Daniel Hillier.

Misc/NEWS.d/next/Library/2019-10-27-00-08-49.bpo-38334.pfLLmc.rst

@@ -0,0 +1 @@
+Fixed seeking backward on an encrypted :class:`zipfile.ZipExtFile`.

Misc/NEWS.d/next/Library/2019-10-27-22-29-45.bpo-38602.7jvYFA.rst

@@ -0,0 +1,3 @@
+Added constants :data:`~fcntl.F_OFD_GETLK`, :data:`~fcntl.F_OFD_SETLK`
+and :data:`~fcntl.F_OFD_SETLKW` to the :mod:`fcntl` module.
+Patch by Dong-hee Na.

Misc/NEWS.d/next/Windows/2019-10-28-05-01-29.bpo-38519.dCkY66.rst

@@ -0,0 +1,2 @@
+Restores the internal C headers that were missing from the nuget.org and
+Microsoft Store packages.

Misc/NEWS.d/next/Windows/2019-10-28-10-48-16.bpo-38589.V69Q1a.rst

@@ -0,0 +1 @@
+Fixes HTML Help shortcut when Windows is not installed to C drive

Modules/_io/_iomodule.c

@@ -138,7 +138,6 @@ Character Meaning
 'b'       binary mode
 't'       text mode (default)
 '+'       open a disk file for updating (reading and writing)
-'U'       universal newline mode (deprecated)
 ========= ===============================================================
 
 The default mode is 'rt' (open for reading text). For binary random
@@ -154,10 +153,6 @@ bytes objects without any decoding. In text mode (the default, or when
 returned as strings, the bytes having been first decoded using a
 platform-dependent encoding or using the specified encoding if given.
 
-'U' mode is deprecated and will raise an exception in future versions
-of Python.  It has no effect in Python 3.  Use newline to control
-universal newlines mode.
-
 buffering is an optional integer used to set the buffering policy.
 Pass 0 to switch buffering off (only allowed in binary mode), 1 to select
 line buffering (only usable in text mode), and an integer > 1 to indicate
@@ -233,12 +228,12 @@ static PyObject *
 _io_open_impl(PyObject *module, PyObject *file, const char *mode,
               int buffering, const char *encoding, const char *errors,
               const char *newline, int closefd, PyObject *opener)
-/*[clinic end generated code: output=aefafc4ce2b46dc0 input=7295902222e6b311]*/
+/*[clinic end generated code: output=aefafc4ce2b46dc0 input=1543f4511d2356a5]*/
 {
     unsigned i;
 
     int creating = 0, reading = 0, writing = 0, appending = 0, updating = 0;
-    int text = 0, binary = 0, universal = 0;
+    int text = 0, binary = 0;
 
     char rawmode[6], *m;
     int line_buffering, is_number;
@@ -296,10 +291,6 @@ _io_open_impl(PyObject *module, PyObject *file, const char *mode,
         case 'b':
             binary = 1;
             break;
-        case 'U':
-            universal = 1;
-            reading = 1;
-            break;
         default:
             goto invalid_mode;
         }
@@ -322,18 +313,6 @@ _io_open_impl(PyObject *module, PyObject *file, const char *mode,
     *m = '\0';
 
     /* Parameters validation */
-    if (universal) {
-        if (creating || writing || appending || updating) {
-            PyErr_SetString(PyExc_ValueError,
-                            "mode U cannot be combined with 'x', 'w', 'a', or '+'");
-            goto error;
-        }
-        if (PyErr_WarnEx(PyExc_DeprecationWarning,
-                         "'U' mode is deprecated", 1) < 0)
-            goto error;
-        reading = 1;
-    }
-
     if (text && binary) {
         PyErr_SetString(PyExc_ValueError,
                         "can't have text and binary mode at once");

Modules/_io/clinic/_iomodule.c.h

@@ -36,7 +36,6 @@ PyDoc_STRVAR(_io_open__doc__,
 "\'b\'       binary mode\n"
 "\'t\'       text mode (default)\n"
 "\'+\'       open a disk file for updating (reading and writing)\n"
-"\'U\'       universal newline mode (deprecated)\n"
 "========= ===============================================================\n"
 "\n"
 "The default mode is \'rt\' (open for reading text). For binary random\n"
@@ -52,10 +51,6 @@ PyDoc_STRVAR(_io_open__doc__,
 "returned as strings, the bytes having been first decoded using a\n"
 "platform-dependent encoding or using the specified encoding if given.\n"
 "\n"
-"\'U\' mode is deprecated and will raise an exception in future versions\n"
-"of Python.  It has no effect in Python 3.  Use newline to control\n"
-"universal newlines mode.\n"
-"\n"
 "buffering is an optional integer used to set the buffering policy.\n"
 "Pass 0 to switch buffering off (only allowed in binary mode), 1 to select\n"
 "line buffering (only usable in text mode), and an integer > 1 to indicate\n"
@@ -323,4 +318,4 @@ _io_open_code(PyObject *module, PyObject *const *args, Py_ssize_t nargs, PyObjec
 exit:
     return return_value;
 }
-/*[clinic end generated code: output=3df6bc6d91697545 input=a9049054013a1b77]*/
+/*[clinic end generated code: output=680e4b488c7da8a1 input=a9049054013a1b77]*/

Modules/fcntlmodule.c

@@ -495,6 +495,15 @@ all_ins(PyObject* m)
 #ifdef F_SETLKW
     if (PyModule_AddIntMacro(m, F_SETLKW)) return -1;
 #endif
+#ifdef F_OFD_GETLK
+    if (PyModule_AddIntMacro(m, F_OFD_GETLK)) return -1;
+#endif
+#ifdef F_OFD_SETLK
+    if (PyModule_AddIntMacro(m, F_OFD_SETLK)) return -1;
+#endif
+#ifdef F_OFD_SETLKW
+    if (PyModule_AddIntMacro(m, F_OFD_SETLKW)) return -1;
+#endif
 #ifdef F_GETOWN
     if (PyModule_AddIntMacro(m, F_GETOWN)) return -1;
 #endif

Modules/itertoolsmodule.c

@@ -3076,12 +3076,15 @@ static PyTypeObject cwr_type = {
 /* permutations object ********************************************************
 
 def permutations(iterable, r=None):
-    'permutations(range(3), 2) --> (0,1) (0,2) (1,0) (1,2) (2,0) (2,1)'
+    # permutations('ABCD', 2) --> AB AC AD BA BC BD CA CB CD DA DB DC
+    # permutations(range(3)) --> 012 021 102 120 201 210
     pool = tuple(iterable)
     n = len(pool)
     r = n if r is None else r
-    indices = range(n)
-    cycles = range(n-r+1, n+1)[::-1]
+    if r > n:
+        return
+    indices = list(range(n))
+    cycles = list(range(n, n-r, -1))
     yield tuple(pool[i] for i in indices[:r])
     while n:
         for i in reversed(range(r)):

PC/layout/main.py

@@ -216,12 +216,7 @@ def get_layout(ns):
 
     if ns.include_dev:
 
-        def _c(d):
-            if d.is_dir():
-                return d.name != "internal"
-            return True
-
-        for dest, src in rglob(ns.source / "Include", "**/*.h", _c):
+        for dest, src in rglob(ns.source / "Include", "**/*.h"):
             yield "include/{}".format(dest), src
         src = ns.source / "PC" / "pyconfig.h"
         yield "include/pyconfig.h", src

Python/ast.c

@@ -1747,8 +1747,10 @@ ast_for_decorator(struct compiling *c, const node *n)
         name_expr = NULL;
     }
     else if (NCH(n) == 5) { /* Call with no arguments */
-        d = Call(name_expr, NULL, NULL, LINENO(n),
-                 n->n_col_offset, n->n_end_lineno, n->n_end_col_offset, c->c_arena);
+        d = Call(name_expr, NULL, NULL,
+                 name_expr->lineno, name_expr->col_offset,
+                 CHILD(n, 3)->n_end_lineno, CHILD(n, 3)->n_end_col_offset,
+                 c->c_arena);
         if (!d)
             return NULL;
         name_expr = NULL;

Python/formatter_unicode.c

@@ -590,7 +590,7 @@ calc_number_widths(NumberFieldWidths *spec, Py_ssize_t n_prefix,
         spec->n_remainder + spec->n_rpadding;
 }
 
-/* Fill in the digit parts of a numbers's string representation,
+/* Fill in the digit parts of a number's string representation,
    as determined in calc_number_widths().
    Return -1 on error, or 0 on success. */
 static int

Tools/msi/bundle/bootstrap/PythonBootstrapperApplication.cpp

@@ -3013,8 +3013,8 @@ private:
             } else if (IsWindowsVersionOrGreater(6, 1, 1)) {
                 HMODULE hKernel32 = GetModuleHandleW(L"kernel32");
                 if (hKernel32 && !GetProcAddress(hKernel32, "AddDllDirectory")) {
-                    BalLog(BOOTSTRAPPER_LOG_LEVEL_ERROR, "Detected Windows Server 2008 R2 without KB2533625");
-                    BalLog(BOOTSTRAPPER_LOG_LEVEL_ERROR, "KB2533625 update is required to continue.");
+                    BalLog(BOOTSTRAPPER_LOG_LEVEL_ERROR, "Detected Windows Server 2008 R2 without KB2533623");
+                    BalLog(BOOTSTRAPPER_LOG_LEVEL_ERROR, "KB2533623 update is required to continue.");
                     /* The "MissingSP1" error also specifies updates are required */
                     LocGetString(_wixLoc, L"#(loc.FailureWS2K8R2MissingSP1)", &pLocString);
                 } else {
@@ -3044,8 +3044,8 @@ private:
             } else if (IsWindows7SP1OrGreater()) {
                 HMODULE hKernel32 = GetModuleHandleW(L"kernel32");
                 if (hKernel32 && !GetProcAddress(hKernel32, "AddDllDirectory")) {
-                    BalLog(BOOTSTRAPPER_LOG_LEVEL_ERROR, "Detected Windows 7 SP1 without KB2533625");
-                    BalLog(BOOTSTRAPPER_LOG_LEVEL_ERROR, "KB2533625 update is required to continue.");
+                    BalLog(BOOTSTRAPPER_LOG_LEVEL_ERROR, "Detected Windows 7 SP1 without KB2533623");
+                    BalLog(BOOTSTRAPPER_LOG_LEVEL_ERROR, "KB2533623 update is required to continue.");
                     /* The "MissingSP1" error also specifies updates are required */
                     LocGetString(_wixLoc, L"#(loc.FailureWin7MissingSP1)", &pLocString);
                 } else {

Tools/msi/doc/doc.wxs

@@ -8,7 +8,7 @@
         <PropertyRef Id="REGISTRYKEY" />
         
         <Property Id="HHExe" Value="C:\Windows\hh.exe" />
-        <CustomAction Id="SetHHExe" Property="HHCExe" Value='[WindowsFolder]\hh.exe' Execute="immediate" />
+        <CustomAction Id="SetHHExe" Property="HHExe" Value='[WindowsFolder]\hh.exe' Execute="immediate" />
         <InstallExecuteSequence>
             <Custom Action="SetHHExe" Before="CostFinalize">1</Custom>
         </InstallExecuteSequence>

Tools/pynche/StripViewer.py

@@ -371,22 +371,6 @@ class StripViewer:
                              command=self.__togglehex)
         hexbtn.grid(row=1, column=1, sticky=W)
 
-        # XXX: ignore this feature for now; it doesn't work quite right yet
-
-##        gentypevar = self.__gentypevar = IntVar()
-##        self.__variations = Radiobutton(frame,
-##                                        text='Variations',
-##                                        variable=gentypevar,
-##                                        value=0,
-##                                        command=self.__togglegentype)
-##        self.__variations.grid(row=0, column=1, sticky=W)
-##        self.__constants = Radiobutton(frame,
-##                                       text='Constants',
-##                                       variable=gentypevar,
-##                                       value=1,
-##                                       command=self.__togglegentype)
-##        self.__constants.grid(row=1, column=1, sticky=W)
-
         # create the white button
         whitebtn = Button(frame2,
                           text='White',
@@ -402,26 +386,6 @@ class StripViewer:
         red, green, blue = self.__sb.current_rgb()
         self.update_yourself(red, green, blue)
 
-##    def __togglegentype(self, event=None):
-##        which = self.__gentypevar.get()
-##        if which == 0:
-##            self.__reds.set(label='Red Variations',
-##                            generator=constant_cyan_generator)
-##            self.__greens.set(label='Green Variations',
-##                              generator=constant_magenta_generator)
-##            self.__blues.set(label='Blue Variations',
-##                             generator=constant_yellow_generator)
-##        elif which == 1:
-##            self.__reds.set(label='Red Constant',
-##                            generator=constant_red_generator)
-##            self.__greens.set(label='Green Constant',
-##                              generator=constant_green_generator)
-##            self.__blues.set(label='Blue Constant',
-##                             generator=constant_blue_generator)
-##        else:
-##            assert 0
-##        self.__sb.update_views_current()
-
     def __toblack(self, event=None):
         self.__sb.update_views(0, 0, 0)