Merge.

Doc/c-api/import.rst

@@ -132,8 +132,14 @@ Importing Modules
    such modules have no way to know that the module object is an unknown (and
    probably damaged with respect to the module author's intents) state.
 
+   The module's :attr:`__spec__` and :attr:`__loader__` will be set, if
+   not set already, with the appropriate values.  The spec's loader will
+   be set to the module's ``__loader__`` (if set) and to an instance of
+   :class:`SourceFileLoader` otherwise.
+
    The module's :attr:`__file__` attribute will be set to the code object's
-   :c:member:`co_filename`.
+   :c:member:`co_filename`.  If applicable, :attr:`__cached__` will also
+   be set.
 
    This function will reload the module if it was already imported.  See
    :c:func:`PyImport_ReloadModule` for the intended way to reload a module.

Doc/library/asyncio-subprocess.rst

@@ -8,7 +8,7 @@ Operating system support
 
 On Windows, the default event loop uses :class:`selectors.SelectSelector`
 which only supports sockets. The :class:`ProactorEventLoop` should be used to
-support subprocesses.
+support subprocesses. However, the latter does not support SSL.
 
 On Mac OS X older than 10.9 (Mavericks), :class:`selectors.KqueueSelector`
 does not support character devices like PTY, whereas it is used by the
@@ -262,9 +262,7 @@ display the output::
         stdout = stdout.decode('ascii').rstrip()
         print("Platform: %s" % stdout)
     else:
-        print("Python failed with exit code %s:" % exitcode)
+        print("Python failed with exit code %s:" % exitcode, flush=True)
-        sys.stdout.flush()
-        sys.stdout.buffer.flush()
         sys.stdout.buffer.write(stdout)
         sys.stdout.buffer.flush()
     loop.close()

Doc/library/codecs.rst

@@ -22,9 +22,10 @@ manages the codec and error handling lookup process.
 
 It defines the following functions:
 
-.. function:: encode(obj, encoding='utf-8', errors='strict')
+.. function:: encode(obj, [encoding[, errors]])
 
-   Encodes *obj* using the codec registered for *encoding*.
+   Encodes *obj* using the codec registered for *encoding*. The default
+   encoding is ``utf-8``.
 
    *Errors* may be given to set the desired error handling scheme. The
    default error handler is ``strict`` meaning that encoding errors raise
@@ -32,9 +33,10 @@ It defines the following functions:
    :exc:`UnicodeEncodeError`). Refer to :ref:`codec-base-classes` for more
    information on codec error handling.
 
-.. function:: decode(obj, encoding='utf-8', errors='strict')
+.. function:: decode(obj, [encoding[, errors]])
 
-   Decodes *obj* using the codec registered for *encoding*.
+   Decodes *obj* using the codec registered for *encoding*. The default
+   encoding is ``utf-8``.
 
    *Errors* may be given to set the desired error handling scheme. The
    default error handler is ``strict`` meaning that decoding errors raise

Doc/library/email.generator.rst

@@ -112,7 +112,7 @@ formatted string representation of a message object.  For more detail, see
 :mod:`email.message`.
 
 .. class:: BytesGenerator(outfp, mangle_from_=True, maxheaderlen=78, *, \
-                          policy=policy.default)
+                          policy=None)
 
    The constructor for the :class:`BytesGenerator` class takes a binary
    :term:`file-like object` called *outfp* for an argument.  *outfp* must
@@ -134,9 +134,11 @@ formatted string representation of a message object.  For more detail, see
    wrapping.  The default is 78, as recommended (but not required) by
    :rfc:`2822`.
 
+
    The *policy* keyword specifies a :mod:`~email.policy` object that controls a
-   number of aspects of the generator's operation.  The default policy
+   number of aspects of the generator's operation.  If no *policy* is specified,
-   maintains backward compatibility.
+   then the *policy* attached to the message object passed to :attr:`flatten`
+   is used.
 
    .. versionchanged:: 3.3 Added the *policy* keyword.
 
@@ -174,7 +176,7 @@ formatted string representation of a message object.  For more detail, see
 
       Optional *linesep* specifies the line separator character used to
       terminate lines in the output.  If specified it overrides the value
-      specified by the ``Generator``\ 's ``policy``.
+      specified by the ``Generator``\ or *msg*\ 's ``policy``.
 
    .. method:: clone(fp)
 

Doc/library/email.message.rst

@@ -34,7 +34,7 @@ Here are the methods of the :class:`Message` class:
 .. class:: Message(policy=compat32)
 
    If *policy* is specified (it must be an instance of a :mod:`~email.policy`
-   class) use the rules it specifies to udpate and serialize the representation
+   class) use the rules it specifies to update and serialize the representation
    of the message.  If *policy* is not set, use the :class:`compat32
    <email.policy.Compat32>` policy, which maintains backward compatibility with
    the Python 3.2 version of the email package.  For more information see the

Doc/library/email.parser.rst

@@ -60,15 +60,18 @@ list of defects that it can find.
 Here is the API for the :class:`FeedParser`:
 
 
-.. class:: FeedParser(_factory=email.message.Message, *, policy=policy.default)
+.. class:: FeedParser(_factory=email.message.Message, *, policy=policy.compat32)
 
    Create a :class:`FeedParser` instance.  Optional *_factory* is a no-argument
    callable that will be called whenever a new message object is needed.  It
    defaults to the :class:`email.message.Message` class.
 
-   The *policy* keyword specifies a :mod:`~email.policy` object that controls a
+   If *policy* is specified (it must be an instance of a :mod:`~email.policy`
-   number of aspects of the parser's operation.  The default policy maintains
+   class) use the rules it specifies to update the representation of the
-   backward compatibility.
+   message.  If *policy* is not set, use the :class:`compat32
+   <email.policy.Compat32>` policy, which maintains backward compatibility with
+   the Python 3.2 version of the email package.  For more information see the
+   :mod:`~email.policy` documentation.
 
    .. versionchanged:: 3.3 Added the *policy* keyword.
 
@@ -113,7 +116,7 @@ have the same API as the :class:`Parser` and :class:`BytesParser` classes.
    The BytesHeaderParser class.
 
 
-.. class:: Parser(_class=email.message.Message, *, policy=policy.default)
+.. class:: Parser(_class=email.message.Message, *, policy=policy.compat32)
 
    The constructor for the :class:`Parser` class takes an optional argument
    *_class*.  This must be a callable factory (such as a function or a class), and
@@ -121,9 +124,12 @@ have the same API as the :class:`Parser` and :class:`BytesParser` classes.
    :class:`~email.message.Message` (see :mod:`email.message`).  The factory will
    be called without arguments.
 
-   The *policy* keyword specifies a :mod:`~email.policy` object that controls a
+   If *policy* is specified (it must be an instance of a :mod:`~email.policy`
-   number of aspects of the parser's operation.  The default policy maintains
+   class) use the rules it specifies to update the representation of the
-   backward compatibility.
+   message.  If *policy* is not set, use the :class:`compat32
+   <email.policy.Compat32>` policy, which maintains backward compatibility with
+   the Python 3.2 version of the email package.  For more information see the
+   :mod:`~email.policy` documentation.
 
    .. versionchanged:: 3.3
       Removed the *strict* argument that was deprecated in 2.4.  Added the
@@ -159,15 +165,18 @@ have the same API as the :class:`Parser` and :class:`BytesParser` classes.
       Optional *headersonly* is as with the :meth:`parse` method.
 
 
-.. class:: BytesParser(_class=email.message.Message, *, policy=policy.default)
+.. class:: BytesParser(_class=email.message.Message, *, policy=policy.compat32)
 
    This class is exactly parallel to :class:`Parser`, but handles bytes input.
    The *_class* and *strict* arguments are interpreted in the same way as for
    the :class:`Parser` constructor.
 
-   The *policy* keyword specifies a :mod:`~email.policy` object that
+   If *policy* is specified (it must be an instance of a :mod:`~email.policy`
-   controls a number of aspects of the parser's operation.  The default
+   class) use the rules it specifies to update the representation of the
-   policy maintains backward compatibility.
+   message.  If *policy* is not set, use the :class:`compat32
+   <email.policy.Compat32>` policy, which maintains backward compatibility with
+   the Python 3.2 version of the email package.  For more information see the
+   :mod:`~email.policy` documentation.
 
    .. versionchanged:: 3.3
       Removed the *strict* argument.  Added the *policy* keyword.
@@ -209,7 +218,7 @@ in the top-level :mod:`email` package namespace.
 .. currentmodule:: email
 
 .. function:: message_from_string(s, _class=email.message.Message, *, \
-                                  policy=policy.default)
+                                  policy=policy.compat32)
 
    Return a message object structure from a string.  This is exactly equivalent to
    ``Parser().parsestr(s)``.  *_class* and *policy* are interpreted as
@@ -219,7 +228,7 @@ in the top-level :mod:`email` package namespace.
       Removed the *strict* argument.  Added the *policy* keyword.
 
 .. function:: message_from_bytes(s, _class=email.message.Message, *, \
-                                 policy=policy.default)
+                                 policy=policy.compat32)
 
    Return a message object structure from a byte string.  This is exactly
    equivalent to ``BytesParser().parsebytes(s)``.  Optional *_class* and
@@ -231,7 +240,7 @@ in the top-level :mod:`email` package namespace.
       Removed the *strict* argument.  Added the *policy* keyword.
 
 .. function:: message_from_file(fp, _class=email.message.Message, *, \
-                                policy=policy.default)
+                                policy=policy.compat32)
 
    Return a message object structure tree from an open :term:`file object`.
    This is exactly equivalent to ``Parser().parse(fp)``.  *_class*
@@ -242,7 +251,7 @@ in the top-level :mod:`email` package namespace.
       Removed the *strict* argument.  Added the *policy* keyword.
 
 .. function:: message_from_binary_file(fp, _class=email.message.Message, *, \
-                                       policy=policy.default)
+                                       policy=policy.compat32)
 
    Return a message object structure tree from an open binary :term:`file
    object`.  This is exactly equivalent to ``BytesParser().parse(fp)``.

Doc/library/imp.rst

@@ -79,7 +79,9 @@ This module provides an interface to the mechanisms used to implement the
    When *P* itself has a dotted name, apply this recipe recursively.
 
    .. deprecated:: 3.3
-      Use :func:`importlib.find_loader` instead.
+      Use :func:`importlib.util.find_spec` instead unless Python 3.3
+      compatibility is required, in which case use
+      :func:`importlib.find_loader`.
 
 
 .. function:: load_module(name, file, pathname, description)
@@ -104,9 +106,11 @@ This module provides an interface to the mechanisms used to implement the
 
    .. deprecated:: 3.3
       If previously used in conjunction with :func:`imp.find_module` then
-      call ``load_module()`` on the returned loader. If you wish to load a
+      consider using :func:`importlib.import_module`, otherwise use the loader
-      module from a specific file, then use one of the file-based loaders found
+      returned by the replacement you chose for :func:`imp.find_module`. If you
-      in :mod:`importlib.machinery`.
+      called :func:`imp.load_module` and related functions directly then use the
+      classes in :mod:`importlib.machinery`, e.g.
+      ``importlib.machinery.SourceFileLoader(name, path).load_module()``.
 
 
 .. function:: new_module(name)

Doc/library/importlib.rst

@@ -887,6 +887,11 @@ find and load modules.
 
       Concrete implementation of :meth:`importlib.abc.SourceLoader.set_data`.
 
+   .. method:: load_module(name=None)
+
+      Concrete implementation of :meth:`importlib.abc.Loader.load_module` where
+      specifying the name of the module to load is optional.
+
 
 .. class:: SourcelessFileLoader(fullname, path)
 
@@ -921,6 +926,11 @@ find and load modules.
       Returns ``None`` as bytecode files have no source when this loader is
       used.
 
+   .. method:: load_module(name=None)
+
+   Concrete implementation of :meth:`importlib.abc.Loader.load_module` where
+   specifying the name of the module to load is optional.
+
 
 .. class:: ExtensionFileLoader(fullname, path)
 
@@ -940,7 +950,7 @@ find and load modules.
 
       Path to the extension module.
 
-   .. method:: load_module(fullname)
+   .. method:: load_module(name=None)
 
       Loads the extension module if and only if *fullname* is the same as
       :attr:`name` or is ``None``.

Doc/library/ssl.rst

@@ -1588,8 +1588,19 @@ the sockets in non-blocking mode and use an event loop).
 Notes on non-blocking sockets
 -----------------------------
 
-When working with non-blocking sockets, there are several things you need
+SSL sockets behave slightly different than regular sockets in
-to be aware of:
+non-blocking mode. When working with non-blocking sockets, there are
+thus several things you need to be aware of:
+
+- Most :class:`SSLSocket` methods will raise either
+  :exc:`SSLWantWriteError` or :exc:`SSLWantReadError` instead of
+  :exc:`BlockingIOError` if an I/O operation would
+  block. :exc:`SSLWantReadError` will be raised if a read operation on
+  the underlying socket is necessary, and :exc:`SSLWantWriteError` for
+  a write operation on the underlying socket. Note that attempts to
+  *write* to an SSL socket may require *reading* from the underlying
+  socket first, and attempts to *read* from the SSL socket may require
+  a prior *write* to the underlying socket.
 
 - Calling :func:`~select.select` tells you that the OS-level socket can be
   read from (or written to), but it does not imply that there is sufficient
@@ -1598,8 +1609,14 @@ to be aware of:
   and :meth:`SSLSocket.send` failures, and retry after another call to
   :func:`~select.select`.
 
+- Conversely, since the SSL layer has its own framing, a SSL socket may
+  still have data available for reading without :func:`~select.select`
+  being aware of it.  Therefore, you should first call
+  :meth:`SSLSocket.recv` to drain any potentially available data, and then
+  only block on a :func:`~select.select` call if still necessary.
+
   (of course, similar provisions apply when using other primitives such as
-  :func:`~select.poll`)
+  :func:`~select.poll`, or those in the :mod:`selectors` module)
 
 - The SSL handshake itself will be non-blocking: the
   :meth:`SSLSocket.do_handshake` method has to be retried until it returns

Doc/library/subprocess.rst

@@ -54,18 +54,12 @@ use cases, the underlying :class:`Popen` interface can be used directly.
       >>> subprocess.call("exit 1", shell=True)
       1
 
-   .. warning::
-
-      Invoking the system shell with ``shell=True`` can be a security hazard
-      if combined with untrusted input. See the warning under
-      :ref:`frequently-used-arguments` for details.
-
    .. note::
 
-      Do not use ``stdout=PIPE`` or ``stderr=PIPE`` with this function. As
+      Do not use ``stdout=PIPE`` or ``stderr=PIPE`` with this
-      the pipes are not being read in the current process, the child
+      function.  The child process will block if it generates enough
-      process may block if it generates enough output to a pipe to fill up
+      output to a pipe to fill up the OS pipe buffer as the pipes are
-      the OS pipe buffer.
+      not being read from.
 
    .. versionchanged:: 3.3
       *timeout* was added.
@@ -99,18 +93,12 @@ use cases, the underlying :class:`Popen` interface can be used directly.
          ...
       subprocess.CalledProcessError: Command 'exit 1' returned non-zero exit status 1
 
-   .. warning::
-
-      Invoking the system shell with ``shell=True`` can be a security hazard
-      if combined with untrusted input. See the warning under
-      :ref:`frequently-used-arguments` for details.
-
    .. note::
 
-      Do not use ``stdout=PIPE`` or ``stderr=PIPE`` with this function. As
+      Do not use ``stdout=PIPE`` or ``stderr=PIPE`` with this
-      the pipes are not being read in the current process, the child
+      function.  The child process will block if it generates enough
-      process may block if it generates enough output to a pipe to fill up
+      output to a pipe to fill up the OS pipe buffer as the pipes are
-      the OS pipe buffer.
+      not being read from.
 
    .. versionchanged:: 3.3
       *timeout* was added.
@@ -177,17 +165,12 @@ use cases, the underlying :class:`Popen` interface can be used directly.
       ...     shell=True)
       'ls: non_existent_file: No such file or directory\n'
 
-   .. warning::
-
-      Invoking the system shell with ``shell=True`` can be a security hazard
-      if combined with untrusted input. See the warning under
-      :ref:`frequently-used-arguments` for details.
-
    .. note::
 
-      Do not use ``stderr=PIPE`` with this function. As the pipe is not being
+      Do not use ``stdout=PIPE`` or ``stderr=PIPE`` with this
-      read in the current process, the child process may block if it
+      function.  The child process will block if it generates enough
-      generates enough output to the pipe to fill up the OS pipe buffer.
+      output to a pipe to fill up the OS pipe buffer as the pipes are
+      not being read from.
 
    .. versionadded:: 3.1
 
@@ -210,7 +193,7 @@ use cases, the underlying :class:`Popen` interface can be used directly.
 
    Special value that can be used as the *stdin*, *stdout* or *stderr* argument
    to :class:`Popen` and indicates that a pipe to the standard stream should be
-   opened.
+   opened.  Most useful with :meth:`Popen.communicate`.
 
 
 .. data:: STDOUT
@@ -336,28 +319,9 @@ default values. The arguments that are most commonly needed are:
       instead of ``locale.getpreferredencoding()``.  See the
       :class:`io.TextIOWrapper` class for more information on this change.
 
-   .. warning::
+   .. note::
 
-      Executing shell commands that incorporate unsanitized input from an
+      Read the `Security Considerations`_ section before using ``shell=True``.
-      untrusted source makes a program vulnerable to `shell injection
-      <http://en.wikipedia.org/wiki/Shell_injection#Shell_injection>`_,
-      a serious security flaw which can result in arbitrary command execution.
-      For this reason, the use of ``shell=True`` is **strongly discouraged**
-      in cases where the command string is constructed from external input::
-
-         >>> from subprocess import call
-         >>> filename = input("What file would you like to display?\n")
-         What file would you like to display?
-         non_existent; rm -rf / #
-         >>> call("cat " + filename, shell=True) # Uh-oh. This will end badly...
-
-      ``shell=False`` disables all shell based features, but does not suffer
-      from this vulnerability; see the Note in the :class:`Popen` constructor
-      documentation for helpful hints in getting ``shell=False`` to work.
-
-      When using ``shell=True``, :func:`shlex.quote` can be used to properly
-      escape whitespace and shell metacharacters in strings that are going to
-      be used to construct shell commands.
 
 These options, along with all of the other options, are described in more
 detail in the :class:`Popen` constructor documentation.
@@ -378,7 +342,7 @@ functions.
                  startupinfo=None, creationflags=0, restore_signals=True, \
                  start_new_session=False, pass_fds=())
 
-   Execute a child program in a new process.  On Unix, the class uses
+   Execute a child program in a new process.  On POSIX, the class uses
    :meth:`os.execvp`-like behavior to execute the child program.  On Windows,
    the class uses the Windows ``CreateProcess()`` function.  The arguments to
    :class:`Popen` are as follows.
@@ -390,7 +354,7 @@ functions.
    arguments for additional differences from the default behavior.  Unless
    otherwise stated, it is recommended to pass *args* as a sequence.
 
-   On Unix, if *args* is a string, the string is interpreted as the name or
+   On POSIX, if *args* is a string, the string is interpreted as the name or
    path of the program to execute.  However, this can only be done if not
    passing arguments to the program.
 
@@ -421,7 +385,7 @@ functions.
    the shell as the program to execute.  If *shell* is *True*, it is
    recommended to pass *args* as a string rather than as a sequence.
 
-   On Unix with ``shell=True``, the shell defaults to :file:`/bin/sh`.  If
+   On POSIX with ``shell=True``, the shell defaults to :file:`/bin/sh`.  If
    *args* is a string, the string specifies the command
    to execute through the shell.  This means that the string must be
    formatted exactly as it would be when typed at the shell prompt.  This
@@ -438,11 +402,9 @@ functions.
    into the shell (e.g. :command:`dir` or :command:`copy`).  You do not need
    ``shell=True`` to run a batch file or console-based executable.
 
-   .. warning::
+   .. note::
 
-      Passing ``shell=True`` can be a security hazard if combined with
+      Read the `Security Considerations`_ section before using ``shell=True``.
-      untrusted input.  See the warning under :ref:`frequently-used-arguments`
-      for details.
 
    *bufsize* will be supplied as the corresponding argument to the :func:`open`
    function when creating the stdin/stdout/stderr pipe file objects: :const:`0`
@@ -463,9 +425,9 @@ functions.
    program to execute specified by *args*.  However, the original *args* is
    still passed to the program.  Most programs treat the program specified
    by *args* as the command name, which can then be different from the program
-   actually executed.  On Unix, the *args* name
+   actually executed.  On POSIX, the *args* name
    becomes the display name for the executable in utilities such as
-   :program:`ps`.  If ``shell=True``, on Unix the *executable* argument
+   :program:`ps`.  If ``shell=True``, on POSIX the *executable* argument
    specifies a replacement shell for the default :file:`/bin/sh`.
 
    *stdin*, *stdout* and *stderr* specify the executed program's standard input,
@@ -481,7 +443,7 @@ functions.
 
    If *preexec_fn* is set to a callable object, this object will be called in the
    child process just before the child is executed.
-   (Unix only)
+   (POSIX only)
 
    .. warning::
 
@@ -499,8 +461,8 @@ functions.
       common use of *preexec_fn* to call os.setsid() in the child.
 
    If *close_fds* is true, all file descriptors except :const:`0`, :const:`1` and
-   :const:`2` will be closed before the child process is executed. (Unix only).
+   :const:`2` will be closed before the child process is executed. (POSIX only).
-   The default varies by platform:  Always true on Unix.  On Windows it is
+   The default varies by platform:  Always true on POSIX.  On Windows it is
    true when *stdin*/*stdout*/*stderr* are :const:`None`, false otherwise.
    On Windows, if *close_fds* is true then no handles will be inherited by the
    child process.  Note that on Windows, you cannot set *close_fds* to true and
@@ -512,7 +474,7 @@ functions.
 
    *pass_fds* is an optional sequence of file descriptors to keep open
    between the parent and child.  Providing any *pass_fds* forces
-   *close_fds* to be :const:`True`.  (Unix only)
+   *close_fds* to be :const:`True`.  (POSIX only)
 
    .. versionadded:: 3.2
       The *pass_fds* parameter was added.
@@ -525,13 +487,13 @@ functions.
    If *restore_signals* is true (the default) all signals that Python has set to
    SIG_IGN are restored to SIG_DFL in the child process before the exec.
    Currently this includes the SIGPIPE, SIGXFZ and SIGXFSZ signals.
-   (Unix only)
+   (POSIX only)
 
    .. versionchanged:: 3.2
       *restore_signals* was added.
 
    If *start_new_session* is true the setsid() system call will be made in the
-   child process prior to the execution of the subprocess.  (Unix only)
+   child process prior to the execution of the subprocess.  (POSIX only)
 
    .. versionchanged:: 3.2
       *start_new_session* was added.
@@ -598,14 +560,21 @@ Exceptions defined in this module all inherit from :exc:`SubprocessError`.
       The :exc:`SubprocessError` base class was added.
 
 
-Security
+Security Considerations
-^^^^^^^^
+-----------------------
 
-Unlike some other popen functions, this implementation will never call a
+Unlike some other popen functions, this implementation will never
-system shell implicitly.  This means that all characters, including shell
+implicitly call a system shell.  This means that all characters,
-metacharacters, can safely be passed to child processes. Obviously, if the
+including shell metacharacters, can safely be passed to child processes.
-shell is invoked explicitly, then it is the application's responsibility to
+If the shell is invoked explicitly, via ``shell=True``, it is the application's
-ensure that all whitespace and metacharacters are quoted appropriately.
+responsibility to ensure that all whitespace and metacharacters are
+quoted appropriately to avoid
+`shell injection <http://en.wikipedia.org/wiki/Shell_injection#Shell_injection>`_
+vulnerabilities.
+
+When using ``shell=True``, the :func:`shlex.quote` function can be
+used to properly escape whitespace and shell metacharacters in strings
+that are going to be used to construct shell commands.
 
 
 Popen Objects
@@ -629,27 +598,27 @@ Instances of the :class:`Popen` class have the following methods:
    :exc:`TimeoutExpired` exception.  It is safe to catch this exception and
    retry the wait.
 
+   .. note::
+
+      This will deadlock when using ``stdout=PIPE`` or ``stderr=PIPE``
+      and the child process generates enough output to a pipe such that
+      it blocks waiting for the OS pipe buffer to accept more data.
+      Use :meth:`Popen.communicate` when using pipes to avoid that.
+
    .. note::
 
       The function is implemented using a busy loop (non-blocking call and
       short sleeps). Use the :mod:`asyncio` module for an asynchronous wait:
       see :class:`asyncio.create_subprocess_exec`.
 
-   .. warning::
-
-      This will deadlock when using ``stdout=PIPE`` and/or
-      ``stderr=PIPE`` and the child process generates enough output to
-      a pipe such that it blocks waiting for the OS pipe buffer to
-      accept more data.  Use :meth:`communicate` to avoid that.
-
    .. versionchanged:: 3.3
       *timeout* was added.
 
    .. deprecated:: 3.4
 
-      Do not use the undocumented *endtime* parameter.  It is was
+      Do not use the *endtime* parameter.  It is was unintentionally
-      unintentionally exposed in 3.3 but was intended to be private
+      exposed in 3.3 but was left undocumented as it was intended to be
-      for internal use.  Use *timeout* instead.
+      private for internal use.  Use *timeout* instead.
 
 .. method:: Popen.communicate(input=None, timeout=None)
 
@@ -716,13 +685,6 @@ Instances of the :class:`Popen` class have the following methods:
 
 The following attributes are also available:
 
-.. warning::
-
-   Use :meth:`~Popen.communicate` rather than :attr:`.stdin.write <Popen.stdin>`,
-   :attr:`.stdout.read <Popen.stdout>` or :attr:`.stderr.read <Popen.stderr>` to avoid
-   deadlocks due to any of the other OS pipe buffers filling up and blocking the
-   child process.
-
 .. attribute:: Popen.args
 
    The *args* argument as it was passed to :class:`Popen` -- a
@@ -756,6 +718,13 @@ The following attributes are also available:
    ``True``, the stream is a text stream, otherwise it is a byte stream. If the
    *stderr* argument was not :data:`PIPE`, this attribute is ``None``.
 
+.. warning::
+
+   Use :meth:`~Popen.communicate` rather than :attr:`.stdin.write <Popen.stdin>`,
+   :attr:`.stdout.read <Popen.stdout>` or :attr:`.stderr.read <Popen.stderr>` to avoid
+   deadlocks due to any of the other OS pipe buffers filling up and blocking the
+   child process.
+
 
 .. attribute:: Popen.pid
 
@@ -772,7 +741,7 @@ The following attributes are also available:
    hasn't terminated yet.
 
    A negative value ``-N`` indicates that the child was terminated by signal
-   ``N`` (Unix only).
+   ``N`` (POSIX only).
 
 
 Windows Popen Helpers
@@ -1044,7 +1013,7 @@ Replacing functions from the :mod:`popen2` module
 
    (child_stdout, child_stdin) = popen2.popen2("somestring", bufsize, mode)
    ==>
-   p = Popen(["somestring"], shell=True, bufsize=bufsize,
+   p = Popen("somestring", shell=True, bufsize=bufsize,
              stdin=PIPE, stdout=PIPE, close_fds=True)
    (child_stdout, child_stdin) = (p.stdout, p.stdin)
 
@@ -1097,7 +1066,7 @@ handling consistency are valid for these functions.
       >>> subprocess.getstatusoutput('/bin/junk')
       (256, 'sh: /bin/junk: not found')
 
-   Availability: Unix & Windows
+   Availability: POSIX & Windows
 
    .. versionchanged:: 3.3.4
       Windows support added
@@ -1113,7 +1082,7 @@ handling consistency are valid for these functions.
       >>> subprocess.getoutput('ls /bin/ls')
       '/bin/ls'
 
-   Availability: Unix & Windows
+   Availability: POSIX & Windows
 
    .. versionchanged:: 3.3.4
       Windows support added

Doc/library/sys.rst

@@ -1066,8 +1066,9 @@ always available.
      statements and for the prompts of :func:`input`;
    * The interpreter's own prompts and its error messages go to ``stderr``.
 
-   By default, these streams are regular text streams as returned by the
+   These streams are regular :term:`text files <text file>` like those
-   :func:`open` function.  Their parameters are chosen as follows:
+   returned by the :func:`open` function.  Their parameters are chosen as
+   follows:
 
    * The character encoding is platform-dependent.  Under Windows, if the stream
      is interactive (that is, if its :meth:`isatty` method returns ``True``), the
@@ -1075,26 +1076,22 @@ always available.
      platforms, the locale encoding is used (see :meth:`locale.getpreferredencoding`).
 
      Under all platforms though, you can override this value by setting the
-     :envvar:`PYTHONIOENCODING` environment variable.
+     :envvar:`PYTHONIOENCODING` environment variable before starting Python.
 
    * When interactive, standard streams are line-buffered.  Otherwise, they
      are block-buffered like regular text files.  You can override this
      value with the :option:`-u` command-line option.
 
-   To write or read binary data from/to the standard streams, use the
+   .. note::
-   underlying binary :data:`~io.TextIOBase.buffer`.  For example, to write
-   bytes to :data:`stdout`, use ``sys.stdout.buffer.write(b'abc')``.  Using
-   :meth:`io.TextIOBase.detach`, streams can be made binary by default.  This
-   function sets :data:`stdin` and :data:`stdout` to binary::
 
-      def make_streams_binary():
+      To write or read binary data from/to the standard streams, use the
-          sys.stdin = sys.stdin.detach()
+      underlying binary :data:`~io.TextIOBase.buffer` object.  For example, to
-          sys.stdout = sys.stdout.detach()
+      write bytes to :data:`stdout`, use ``sys.stdout.buffer.write(b'abc')``.
 
-   Note that the streams may be replaced with objects (like :class:`io.StringIO`)
+      However, if you are writing a library (and do not control in which
-   that do not support the :attr:`~io.BufferedIOBase.buffer` attribute or the
+      context its code will be executed), be aware that the standard streams
-   :meth:`~io.BufferedIOBase.detach` method and can raise :exc:`AttributeError`
+      may be replaced with file-like objects like :class:`io.StringIO` which
-   or :exc:`io.UnsupportedOperation`.
+      do not support the :attr:`~io.BufferedIOBase.buffer` attribute.
 
 
 .. data:: __stdin__

Doc/library/venv.rst

@@ -40,11 +40,11 @@ Creating virtual environments
    A venv is a directory tree which contains Python executable files and
    other files which indicate that it is a venv.
 
-   Common installation tools such as ``Distribute`` and ``pip`` work as
+   Common installation tools such as ``Setuptools`` and ``pip`` work as
    expected with venvs - i.e. when a venv is active, they install Python
    packages into the venv without needing to be told to do so explicitly.
    Of course, you need to install them into the venv first: this could be
-   done by running ``distribute_setup.py`` with the venv activated,
+   done by running ``ez_setup.py`` with the venv activated,
    followed by running ``easy_install pip``. Alternatively, you could download
    the source tarballs and run ``python setup.py install`` after unpacking,
    with the venv activated.

Doc/reference/compound_stmts.rst

@@ -313,14 +313,14 @@ exception, the saved exception is set as the context of the new exception.
 If the :keyword:`finally` clause executes a :keyword:`return` or :keyword:`break`
 statement, the saved exception is discarded::
 
-    def f():
+   >>> def f():
-        try:
+   ...     try:
-            1/0
+   ...         1/0
-        finally:
+   ...     finally:
-            return 42
+   ...         return 42
-
+   ...
-    >>> f()
+   >>> f()
-    42
+   42
 
 The exception information is not available to the program during execution of
 the :keyword:`finally` clause.
@@ -337,6 +337,20 @@ statement, the :keyword:`finally` clause is also executed 'on the way out.' A
 reason is a problem with the current implementation --- this restriction may be
 lifted in the future).
 
+The return value of a function is determined by the last :keyword:`return`
+statement executed.  Since the :keyword:`finally` clause always executes, a
+:keyword:`return` statement executed in the :keyword:`finally` clause will
+always be the last one executed::
+
+   >>> def foo():
+   ...     try:
+   ...         return 'try'
+   ...     finally:
+   ...         return 'finally'
+   ...
+   >>> foo()
+   'finally'
+
 Additional information on exceptions can be found in section :ref:`exceptions`,
 and information on using the :keyword:`raise` statement to generate exceptions
 may be found in section :ref:`raise`.

Doc/reference/datamodel.rst

@@ -323,8 +323,6 @@ Sequences
          object: mutable sequence
          object: mutable
          pair: assignment; statement
-         single: delete
-         statement: del
          single: subscription
          single: slicing
 

Lib/asyncio/base_events.py

@@ -775,11 +775,7 @@ class BaseEventLoop(events.AbstractEventLoop):
         elif self._scheduled:
             # Compute the desired timeout.
             when = self._scheduled[0]._when
-            deadline = max(0, when - self.time())
+            timeout = max(0, when - self.time())
-            if timeout is None:
-                timeout = deadline
-            else:
-                timeout = min(timeout, deadline)
 
         # TODO: Instrumentation only in debug mode?
         if logger.isEnabledFor(logging.INFO):

Lib/asyncio/selector_events.py

@@ -87,10 +87,17 @@ class BaseSelectorEventLoop(base_events.BaseEventLoop):
             pass
 
     def _write_to_self(self):
-        try:
+        # This may be called from a different thread, possibly after
-            self._csock.send(b'x')
+        # _close_self_pipe() has been called or even while it is
-        except (BlockingIOError, InterruptedError):
+        # running.  Guard for self._csock being None or closed.  When
-            pass
+        # a socket is closed, send() raises OSError (with errno set to
+        # EBADF, but let's not rely on the exact error code).
+        csock = self._csock
+        if csock is not None:
+            try:
+                csock.send(b'x')
+            except OSError:
+                pass
 
     def _start_serving(self, protocol_factory, sock,
                        sslcontext=None, server=None):

Lib/asyncio/streams.py

@@ -419,12 +419,17 @@ class StreamReader:
             return b''
 
         if n < 0:
-            while not self._eof:
+            # This used to just loop creating a new waiter hoping to
-                self._waiter = self._create_waiter('read')
+            # collect everything in self._buffer, but that would
-                try:
+            # deadlock if the subprocess sends more than self.limit
-                    yield from self._waiter
+            # bytes.  So just call self.read(self._limit) until EOF.
-                finally:
+            blocks = []
-                    self._waiter = None
+            while True:
+                block = yield from self.read(self._limit)
+                if not block:
+                    break
+                blocks.append(block)
+            return b''.join(blocks)
         else:
             if not self._buffer and not self._eof:
                 self._waiter = self._create_waiter('read')

Lib/ctypes/test/test_pep3118.py

@@ -96,6 +96,9 @@ class EmptyStruct(Structure):
 class aUnion(Union):
     _fields_ = [("a", c_int)]
 
+class StructWithArrays(Structure):
+    _fields_ = [("x", c_long * 3 * 2), ("y", Point * 4)]
+
 class Incomplete(Structure):
     pass
 
@@ -145,10 +148,10 @@ native_types = [
 
     ## arrays and pointers
 
-    (c_double * 4,              "(4)<d",                (4,),           c_double),
+    (c_double * 4,              "<d",                   (4,),           c_double),
-    (c_float * 4 * 3 * 2,       "(2,3,4)<f",            (2,3,4),        c_float),
+    (c_float * 4 * 3 * 2,       "<f",                   (2,3,4),        c_float),
-    (POINTER(c_short) * 2,      "(2)&<h",               (2,),           POINTER(c_short)),
+    (POINTER(c_short) * 2,      "&<h",                  (2,),           POINTER(c_short)),
-    (POINTER(c_short) * 2 * 3,  "(3,2)&<h",             (3,2,),         POINTER(c_short)),
+    (POINTER(c_short) * 2 * 3,  "&<h",                  (3,2,),         POINTER(c_short)),
     (POINTER(c_short * 2),      "&(2)<h",               (),           POINTER(c_short)),
 
     ## structures and unions
@@ -160,6 +163,9 @@ native_types = [
     (EmptyStruct,               "T{}",                  (),           EmptyStruct),
     # the pep does't support unions
     (aUnion,                    "B",                    (),           aUnion),
+    # structure with sub-arrays
+    (StructWithArrays,          "T{(2,3)<l:x:(4)T{<l:x:<l:y:}:y:}", (),  StructWithArrays),
+    (StructWithArrays * 3,      "T{(2,3)<l:x:(4)T{<l:x:<l:y:}:y:}", (3,),  StructWithArrays),
 
     ## pointer to incomplete structure
     (Incomplete,                "B",                    (),           Incomplete),

Lib/distutils/tests/support.py

@@ -207,4 +207,4 @@ def fixup_build_ext(cmd):
                 cmd.library_dirs = []
             else:
                 name, equals, value = runshared.partition('=')
-                cmd.library_dirs = value.split(os.pathsep)
+                cmd.library_dirs = [d for d in value.split(os.pathsep) if d]

Lib/email/generator.py

@@ -51,8 +51,9 @@ class Generator:
         by RFC 2822.
 
         The policy keyword specifies a policy object that controls a number of
-        aspects of the generator's operation.  The default policy maintains
+        aspects of the generator's operation.  If no policy is specified,
-        backward compatibility.
+        the policy associated with the Message object passed to the
+        flatten method is used.
 
         """
         self._fp = outfp
@@ -76,7 +77,9 @@ class Generator:
         Note that for subobjects, no From_ line is printed.
 
         linesep specifies the characters used to indicate a new line in
-        the output.  The default value is determined by the policy.
+        the output.  The default value is determined by the policy specified
+        when the Generator instance was created or, if none was specified,
+        from the policy associated with the msg.
 
         """
         # We use the _XXX constants for operating on data that comes directly

Lib/ensurepip/__init__.py

@@ -10,7 +10,7 @@ __all__ = ["version", "bootstrap"]
 
 _SETUPTOOLS_VERSION = "2.1"
 
-_PIP_VERSION = "1.5.4"
+_PIP_VERSION = "1.5.6"
 
 # pip currently requires ssl support, so we try to provide a nicer
 # error message when that is missing (http://bugs.python.org/issue19744)

Lib/fileinput.py

@@ -320,7 +320,10 @@ class FileInput:
             self._backupfilename = 0
             if self._filename == '-':
                 self._filename = '<stdin>'
-                self._file = sys.stdin
+                if 'b' in self._mode:
+                    self._file = sys.stdin.buffer
+                else:
+                    self._file = sys.stdin
                 self._isstdin = True
             else:
                 if self._inplace:

Lib/idlelib/EditorWindow.py

@@ -79,6 +79,8 @@ class HelpDialog(object):
         self.parent = None
 
 helpDialog = HelpDialog()  # singleton instance
+def _Help_dialog(parent):  # wrapper for htest
+    helpDialog.show_dialog(parent)
 
 
 class EditorWindow(object):
@@ -1064,7 +1066,7 @@ class EditorWindow(object):
         try:
             try:
                 mod = importlib.import_module('.' + name, package=__package__)
-            except ImportError:
+            except (ImportError, TypeError):
                 mod = importlib.import_module(name)
         except ImportError:
             print("\nFailed to import extension: ", name)
@@ -1700,19 +1702,21 @@ def fixwordbreaks(root):
     tk.call('set', 'tcl_nonwordchars', '[^a-zA-Z0-9_]')
 
 
-def test():
+def _Editor_window(parent):
-    root = Tk()
+    root = parent
     fixwordbreaks(root)
     root.withdraw()
     if sys.argv[1:]:
         filename = sys.argv[1]
     else:
         filename = None
+    macosxSupport.setupApp(root, None)
     edit = EditorWindow(root=root, filename=filename)
     edit.set_close_hook(root.quit)
     edit.text.bind("<<close-all-windows>>", edit.close_event)
-    root.mainloop()
-    root.destroy()
 
 if __name__ == '__main__':
-    test()
+    from idlelib.idle_test.htest import run
+    if len(sys.argv) <= 1:
+        run(_Help_dialog)
+    run(_Editor_window)

Lib/idlelib/aboutDialog.py

@@ -12,7 +12,7 @@ class AboutDialog(Toplevel):
     """Modal about dialog for idle
 
     """
-    def __init__(self,parent,title):
+    def __init__(self, parent, title):
         Toplevel.__init__(self, parent)
         self.configure(borderwidth=5)
         self.geometry("+%d+%d" % (parent.winfo_rootx()+30,
@@ -136,10 +136,5 @@ class AboutDialog(Toplevel):
         self.destroy()
 
 if __name__ == '__main__':
-    # test the dialog
+    from idlelib.idle_test.htest import run
-    root = Tk()
+    run(AboutDialog)
-    def run():
-        from idlelib import aboutDialog
-        aboutDialog.AboutDialog(root, 'About')
-    Button(root, text='Dialog', command=run).pack()
-    root.mainloop()

Lib/idlelib/configSectionNameDialog.py

@@ -8,10 +8,11 @@ from tkinter import *
 import tkinter.messagebox as tkMessageBox
 
 class GetCfgSectionNameDialog(Toplevel):
-    def __init__(self, parent, title, message, used_names):
+    def __init__(self, parent, title, message, used_names, _htest=False):
         """
         message - string, informational message to display
         used_names - string collection, names already in use for validity check
+        _htest - bool, change box location when running htest
         """
         Toplevel.__init__(self, parent)
         self.configure(borderwidth=5)
@@ -30,11 +31,12 @@ class GetCfgSectionNameDialog(Toplevel):
         self.messageInfo.config(width=self.frameMain.winfo_reqwidth())
         self.geometry(
                 "+%d+%d" % (
-                parent.winfo_rootx() +
+                    parent.winfo_rootx() +
-                (parent.winfo_width()/2 - self.winfo_reqwidth()/2),
+                    (parent.winfo_width()/2 - self.winfo_reqwidth()/2),
-                parent.winfo_rooty() +
+                    parent.winfo_rooty() +
-                (parent.winfo_height()/2 - self.winfo_reqheight()/2)
+                    ((parent.winfo_height()/2 - self.winfo_reqheight()/2)
-                ) )  #centre dialog over parent
+                    if not _htest else 100)
+                ) )  #centre dialog over parent (or below htest box)
         self.deiconify()  #geometry set, unhide
         self.wait_window()
 
@@ -92,15 +94,5 @@ if __name__ == '__main__':
     import unittest
     unittest.main('idlelib.idle_test.test_config_name', verbosity=2, exit=False)
 
-    # also human test the dialog
+    from idlelib.idle_test.htest import run
-    root = Tk()
+    run(GetCfgSectionNameDialog)
-    def run():
-        dlg=GetCfgSectionNameDialog(root,'Get Name',
-                "After the text entered with [Ok] is stripped, <nothing>, "
-                "'abc', or more that 30 chars are errors. "
-                "Close with a valid entry (printed), [Cancel], or [X]",
-                {'abc'})
-        print(dlg.result)
-    Message(root, text='').pack()  # will be needed for oher dialog tests
-    Button(root, text='Click to begin dialog test', command=run).pack()
-    root.mainloop()

Lib/idlelib/idle_test/htest.py

@@ -0,0 +1,93 @@
+'''Run human tests of Idle's window, dialog, and popup widgets.
+
+run(test): run *test*, a callable that causes a widget to be displayed.
+runall(): run all tests defined in this file.
+
+Let X be a global name bound to a widget callable. End the module with
+
+if __name__ == '__main__':
+    <unittest, if there is one>
+    from idlelib.idle_test.htest import run
+    run(X)
+
+The X object must have a .__name__ attribute and a 'parent' parameter.
+X will often be a widget class, but a callable instance with .__name__
+or a wrapper function also work. The name of wrapper functions, like
+'_Editor_Window', should start with '_'.
+
+This file must contain a matching instance of the folling template,
+with X.__name__ prepended, as in '_Editor_window_spec ...'.
+
+_spec = {
+    'file': '',
+    'kwds': {'title': ''},
+    'msg': ""
+    }
+
+file (no .py): used in runall() to import the file and get X.
+kwds: passed to X (**kwds), after 'parent' is added, to initialize X.
+title: an example; used for some widgets, delete if not.
+msg: displayed in a master window. Hints as to how the user might
+  test the widget. Close the window to skip or end the test.
+'''
+from importlib import import_module
+import tkinter as tk
+
+
+_Editor_window_spec = {
+    'file': 'EditorWindow',
+    'kwds': {},
+    'msg': "Test editor functions of interest"
+    }
+
+_Help_dialog_spec = {
+    'file': 'EditorWindow',
+    'kwds': {},
+    'msg': "If the help text displays, this works"
+    }
+
+AboutDialog_spec = {
+    'file': 'aboutDialog',
+    'kwds': {'title': 'About test'},
+    'msg': "Try each button"
+    }
+
+
+GetCfgSectionNameDialog_spec = {
+    'file': 'configSectionNameDialog',
+    'kwds': {'title':'Get Name',
+                 'message':'Enter something',
+                 'used_names': {'abc'},
+                 '_htest': True},
+    'msg': "After the text entered with [Ok] is stripped, <nothing>, "
+              "'abc', or more that 30 chars are errors.\n"
+              "Close 'Get Name' with a valid entry (printed to Shell), [Cancel], or [X]",
+    }
+
+def run(test):
+    "Display a widget with callable *test* using a _spec dict"
+    root = tk.Tk()
+    test_spec = globals()[test.__name__ + '_spec']
+    test_kwds = test_spec['kwds']
+    test_kwds['parent'] = root
+
+    def run_test():
+        widget = test(**test_kwds)
+        try:
+            print(widget.result)
+        except AttributeError:
+            pass
+    tk.Label(root, text=test_spec['msg'], justify='left').pack()
+    tk.Button(root, text='Test ' + test.__name__, command=run_test).pack()
+    root.mainloop()
+
+def runall():
+    "Run all tests. Quick and dirty version."
+    for k, d in globals().items():
+        if k.endswith('_spec'):
+            mod = import_module('idlelib.' + d['file'])
+            test = getattr(mod, k[:-5])
+            run(test)
+
+if __name__ == '__main__':
+    runall()

Lib/importlib/_bootstrap.py

@@ -1220,6 +1220,29 @@ class _SpecMethods:
             return self._load_unlocked()
 
 
+def _fix_up_module(ns, name, pathname, cpathname=None):
+    # This function is used by PyImport_ExecCodeModuleObject().
+    loader = ns.get('__loader__')
+    spec = ns.get('__spec__')
+    if not loader:
+        if spec:
+            loader = spec.loader
+        elif pathname == cpathname:
+            loader = SourcelessFileLoader(name, pathname)
+        else:
+            loader = SourceFileLoader(name, pathname)
+    if not spec:
+        spec = spec_from_file_location(name, pathname, loader=loader)
+    try:
+        ns['__spec__'] = spec
+        ns['__loader__'] = loader
+        ns['__file__'] = pathname
+        ns['__cached__'] = cpathname
+    except Exception:
+        # Not important enough to report.
+        pass
+
+
 # Loaders #####################################################################
 
 class BuiltinImporter:

Lib/pydoc.py

@@ -1404,6 +1404,9 @@ class _PlainTextDoc(TextDoc):
 def pager(text):
     """The first time this is called, determine what kind of pager to use."""
     global pager
+    # Escape non-encodable characters to avoid encoding errors later
+    encoding = sys.getfilesystemencoding()
+    text = text.encode(encoding, 'backslashreplace').decode(encoding)
     pager = getpager()
     pager(text)
 

Lib/random.py

@@ -105,7 +105,9 @@ class Random(_random.Random):
 
         if a is None:
             try:
-                a = int.from_bytes(_urandom(32), 'big')
+                # Seed with enough bytes to span the 19937 bit
+                # state space for the Mersenne Twister
+                a = int.from_bytes(_urandom(2500), 'big')
             except NotImplementedError:
                 import time
                 a = int(time.time() * 256) # use fractional seconds

Lib/test/script_helper.py

@@ -78,7 +78,7 @@ def assert_python_failure(*args, **env_vars):
     """
     return _assert_python(False, *args, **env_vars)
 
-def spawn_python(*args, **kw):
+def spawn_python(*args, stdout=subprocess.PIPE, stderr=subprocess.STDOUT, **kw):
     """Run a Python subprocess with the given arguments.
 
     kw is extra keyword args to pass to subprocess.Popen. Returns a Popen
@@ -86,8 +86,16 @@ def spawn_python(*args, **kw):
     """
     cmd_line = [sys.executable, '-E']
     cmd_line.extend(args)
+    # Under Fedora (?), GNU readline can output junk on stderr when initialized,
+    # depending on the TERM setting.  Setting TERM=vt100 is supposed to disable
+    # that.  References:
+    # - http://reinout.vanrees.org/weblog/2009/08/14/readline-invisible-character-hack.html
+    # - http://stackoverflow.com/questions/15760712/python-readline-module-prints-escape-character-during-import
+    # - http://lists.gnu.org/archive/html/bug-readline/2007-08/msg00004.html
+    env = kw.setdefault('env', dict(os.environ))
+    env['TERM'] = 'vt100'
     return subprocess.Popen(cmd_line, stdin=subprocess.PIPE,
-                            stdout=subprocess.PIPE, stderr=subprocess.STDOUT,
+                            stdout=stdout, stderr=stderr,
                             **kw)
 
 def kill_python(p):

Lib/test/test_asyncio/test_selector_events.py

@@ -121,8 +121,9 @@ class BaseSelectorEventLoopTests(unittest.TestCase):
         self.assertIsNone(self.loop._write_to_self())
 
     def test_write_to_self_exception(self):
-        self.loop._csock.send.side_effect = OSError()
+        # _write_to_self() swallows OSError
-        self.assertRaises(OSError, self.loop._write_to_self)
+        self.loop._csock.send.side_effect = RuntimeError()
+        self.assertRaises(RuntimeError, self.loop._write_to_self)
 
     def test_sock_recv(self):
         sock = mock.Mock()

Lib/test/test_asyncio/test_streams.py

@@ -1,7 +1,9 @@
 """Tests for streams.py."""
 
 import gc
+import os
 import socket
+import sys
 import unittest
 from unittest import mock
 try:
@@ -583,6 +585,43 @@ class StreamReaderTests(unittest.TestCase):
             server.stop()
             self.assertEqual(msg, b"hello world!\n")
 
+    @unittest.skipIf(sys.platform == 'win32', "Don't have pipes")
+    def test_read_all_from_pipe_reader(self):
+        # See Tulip issue 168.  This test is derived from the example
+        # subprocess_attach_read_pipe.py, but we configure the
+        # StreamReader's limit so that twice it is less than the size
+        # of the data writter.  Also we must explicitly attach a child
+        # watcher to the event loop.
+
+        code = """\
+import os, sys
+fd = int(sys.argv[1])
+os.write(fd, b'data')
+os.close(fd)
+"""
+        rfd, wfd = os.pipe()
+        args = [sys.executable, '-c', code, str(wfd)]
+
+        pipe = open(rfd, 'rb', 0)
+        reader = asyncio.StreamReader(loop=self.loop, limit=1)
+        protocol = asyncio.StreamReaderProtocol(reader, loop=self.loop)
+        transport, _ = self.loop.run_until_complete(
+            self.loop.connect_read_pipe(lambda: protocol, pipe))
+
+        watcher = asyncio.SafeChildWatcher()
+        watcher.attach_loop(self.loop)
+        try:
+            asyncio.set_child_watcher(watcher)
+            proc = self.loop.run_until_complete(
+                asyncio.create_subprocess_exec(*args, pass_fds={wfd}, loop=self.loop))
+            self.loop.run_until_complete(proc.wait())
+        finally:
+            asyncio.set_child_watcher(None)
+
+        os.close(wfd)
+        data = self.loop.run_until_complete(reader.read(-1))
+        self.assertEqual(data, b'data')
+
 
 if __name__ == '__main__':
     unittest.main()

Lib/test/test_cmd_line_script.py

@@ -1,5 +1,6 @@
 # tests command line execution of scripts
 
+import contextlib
 import importlib
 import importlib.machinery
 import zipimport
@@ -8,6 +9,7 @@ import sys
 import os
 import os.path
 import py_compile
+import subprocess
 
 import textwrap
 from test import support
@@ -173,6 +175,53 @@ class CmdLineTest(unittest.TestCase):
         expected = repr(importlib.machinery.BuiltinImporter).encode("utf-8")
         self.assertIn(expected, out)
 
+    @contextlib.contextmanager
+    def interactive_python(self, separate_stderr=False):
+        if separate_stderr:
+            p = spawn_python('-i', bufsize=1, stderr=subprocess.PIPE)
+            stderr = p.stderr
+        else:
+            p = spawn_python('-i', bufsize=1, stderr=subprocess.STDOUT)
+            stderr = p.stdout
+        try:
+            # Drain stderr until prompt
+            while True:
+                data = stderr.read(4)
+                if data == b">>> ":
+                    break
+                stderr.readline()
+            yield p
+        finally:
+            kill_python(p)
+            stderr.close()
+
+    def check_repl_stdout_flush(self, separate_stderr=False):
+        with self.interactive_python(separate_stderr) as p:
+            p.stdin.write(b"print('foo')\n")
+            p.stdin.flush()
+            self.assertEqual(b'foo', p.stdout.readline().strip())
+
+    def check_repl_stderr_flush(self, separate_stderr=False):
+        with self.interactive_python(separate_stderr) as p:
+            p.stdin.write(b"1/0\n")
+            p.stdin.flush()
+            stderr = p.stderr if separate_stderr else p.stdout
+            self.assertIn(b'Traceback ', stderr.readline())
+            self.assertIn(b'File "<stdin>"', stderr.readline())
+            self.assertIn(b'ZeroDivisionError', stderr.readline())
+
+    def test_repl_stdout_flush(self):
+        self.check_repl_stdout_flush()
+
+    def test_repl_stdout_flush_separate_stderr(self):
+        self.check_repl_stdout_flush(True)
+
+    def test_repl_stderr_flush(self):
+        self.check_repl_stderr_flush()
+
+    def test_repl_stderr_flush_separate_stderr(self):
+        self.check_repl_stderr_flush(True)
+
     def test_basic_script(self):
         with temp_dir() as script_dir:
             script_name = _make_test_script(script_dir, 'script')

Lib/test/test_code_module.py

@@ -51,7 +51,7 @@ class TestInteractiveConsole(unittest.TestCase):
         self.infunc.side_effect = ["undefined", EOFError('Finished')]
         self.console.interact()
         for call in self.stderr.method_calls:
-            if 'NameError:' in ''.join(call[1]):
+            if 'NameError' in ''.join(call[1]):
                 break
         else:
             raise AssertionError("No syntax error from console")

Lib/test/test_faulthandler.py

@@ -591,6 +591,31 @@ sys.exit(exitcode)
     def test_register_chain(self):
         self.check_register(chain=True)
 
+    @contextmanager
+    def check_stderr_none(self):
+        stderr = sys.stderr
+        try:
+            sys.stderr = None
+            with self.assertRaises(RuntimeError) as cm:
+                yield
+            self.assertEqual(str(cm.exception), "sys.stderr is None")
+        finally:
+            sys.stderr = stderr
+
+    def test_stderr_None(self):
+        # Issue #21497: provide an helpful error if sys.stderr is None,
+        # instead of just an attribute error: "None has no attribute fileno".
+        with self.check_stderr_none():
+            faulthandler.enable()
+        with self.check_stderr_none():
+            faulthandler.dump_traceback()
+        if hasattr(faulthandler, 'dump_traceback_later'):
+            with self.check_stderr_none():
+                faulthandler.dump_traceback_later(1e-3)
+        if hasattr(faulthandler, "register"):
+            with self.check_stderr_none():
+                faulthandler.register(signal.SIGUSR1)
+
 
 if __name__ == "__main__":
     unittest.main()

Lib/test/test_fileinput.py

@@ -19,11 +19,12 @@ try:
 except ImportError:
     gzip = None
 
-from io import StringIO
+from io import BytesIO, StringIO
 from fileinput import FileInput, hook_encoded
 
 from test.support import verbose, TESTFN, run_unittest, check_warnings
 from test.support import unlink as safe_unlink
+from unittest import mock
 
 
 # The fileinput module has 2 interfaces: the FileInput class which does
@@ -232,6 +233,13 @@ class FileInputTests(unittest.TestCase):
         finally:
             remove_tempfiles(t1)
 
+    def test_stdin_binary_mode(self):
+        with mock.patch('sys.stdin') as m_stdin:
+            m_stdin.buffer = BytesIO(b'spam, bacon, sausage, and spam')
+            fi = FileInput(files=['-'], mode='rb')
+            lines = list(fi)
+            self.assertEqual(lines, [b'spam, bacon, sausage, and spam'])
+
     def test_file_opening_hook(self):
         try:
             # cannot use openhook and inplace mode

Lib/test/test_gc.py

@@ -580,6 +580,38 @@ class GCTests(unittest.TestCase):
             # would be damaged, with an empty __dict__.
             self.assertEqual(x, None)
 
+    def test_bug21435(self):
+        # This is a poor test - its only virtue is that it happened to
+        # segfault on Tim's Windows box before the patch for 21435 was
+        # applied.  That's a nasty bug relying on specific pieces of cyclic
+        # trash appearing in exactly the right order in finalize_garbage()'s
+        # input list.
+        # But there's no reliable way to force that order from Python code,
+        # so over time chances are good this test won't really be testing much
+        # of anything anymore.  Still, if it blows up, there's _some_
+        # problem ;-)
+        gc.collect()
+
+        class A:
+            pass
+
+        class B:
+            def __init__(self, x):
+                self.x = x
+
+            def __del__(self):
+                self.attr = None
+
+        def do_work():
+            a = A()
+            b = B(A())
+
+            a.attr = b
+            b.attr = a
+
+        do_work()
+        gc.collect() # this blows up (bad C pointer) when it fails
+
     @cpython_only
     def test_garbage_at_shutdown(self):
         import subprocess

Lib/test/test_importlib/test_api.py

@@ -241,13 +241,13 @@ class ReloadTests:
                                 '__file__': path,
                                 '__cached__': cached,
                                 '__doc__': None,
-                                '__builtins__': __builtins__,
                                 }
                     support.create_empty_file(path)
                     module = self.init.import_module(name)
-                    ns = vars(module)
+                    ns = vars(module).copy()
                     loader = ns.pop('__loader__')
                     spec = ns.pop('__spec__')
+                    ns.pop('__builtins__', None)  # An implementation detail.
                     self.assertEqual(spec.name, name)
                     self.assertEqual(spec.loader, loader)
                     self.assertEqual(loader.path, path)
@@ -263,14 +263,14 @@ class ReloadTests:
                                 '__cached__': cached,
                                 '__path__': [os.path.dirname(init_path)],
                                 '__doc__': None,
-                                '__builtins__': __builtins__,
                                 }
                     os.mkdir(name)
                     os.rename(path, init_path)
                     reloaded = self.init.reload(module)
-                    ns = vars(reloaded)
+                    ns = vars(reloaded).copy()
                     loader = ns.pop('__loader__')
                     spec = ns.pop('__spec__')
+                    ns.pop('__builtins__', None)  # An implementation detail.
                     self.assertEqual(spec.name, name)
                     self.assertEqual(spec.loader, loader)
                     self.assertIs(reloaded, module)
@@ -295,10 +295,11 @@ class ReloadTests:
                     with open(bad_path, 'w') as init_file:
                         init_file.write('eggs = None')
                     module = self.init.import_module(name)
-                    ns = vars(module)
+                    ns = vars(module).copy()
                     loader = ns.pop('__loader__')
                     path = ns.pop('__path__')
                     spec = ns.pop('__spec__')
+                    ns.pop('__builtins__', None)  # An implementation detail.
                     self.assertEqual(spec.name, name)
                     self.assertIs(spec.loader, None)
                     self.assertIsNot(loader, None)
@@ -319,14 +320,14 @@ class ReloadTests:
                                 '__cached__': cached,
                                 '__path__': [os.path.dirname(init_path)],
                                 '__doc__': None,
-                                '__builtins__': __builtins__,
                                 'eggs': None,
                                 }
                     os.rename(bad_path, init_path)
                     reloaded = self.init.reload(module)
-                    ns = vars(reloaded)
+                    ns = vars(reloaded).copy()
                     loader = ns.pop('__loader__')
                     spec = ns.pop('__spec__')
+                    ns.pop('__builtins__', None)  # An implementation detail.
                     self.assertEqual(spec.name, name)
                     self.assertEqual(spec.loader, loader)
                     self.assertIs(reloaded, module)

Lib/test/test_io.py

@@ -2615,6 +2615,38 @@ class TextIOWrapperTest(unittest.TestCase):
         txt.write('5')
         self.assertEqual(b''.join(raw._write_stack), b'123\n45')
 
+    def test_bufio_write_through(self):
+        # Issue #21396: write_through=True doesn't force a flush()
+        # on the underlying binary buffered object.
+        flush_called, write_called = [], []
+        class BufferedWriter(self.BufferedWriter):
+            def flush(self, *args, **kwargs):
+                flush_called.append(True)
+                return super().flush(*args, **kwargs)
+            def write(self, *args, **kwargs):
+                write_called.append(True)
+                return super().write(*args, **kwargs)
+
+        rawio = self.BytesIO()
+        data = b"a"
+        bufio = BufferedWriter(rawio, len(data)*2)
+        textio = self.TextIOWrapper(bufio, encoding='ascii',
+                                    write_through=True)
+        # write to the buffered io but don't overflow the buffer
+        text = data.decode('ascii')
+        textio.write(text)
+
+        # buffer.flush is not called with write_through=True
+        self.assertFalse(flush_called)
+        # buffer.write *is* called with write_through=True
+        self.assertTrue(write_called)
+        self.assertEqual(rawio.getvalue(), b"") # no flush
+
+        write_called = [] # reset
+        textio.write(text * 10) # total content is larger than bufio buffer
+        self.assertTrue(write_called)
+        self.assertEqual(rawio.getvalue(), data * 11) # all flushed
+
     def test_read_nonbytes(self):
         # Issue #17106
         # Crash when underlying read() returns non-bytes

Lib/test/test_long.py

@@ -1235,6 +1235,13 @@ class LongTest(unittest.TestCase):
         for n in map(int, integers):
             self.assertEqual(n, 0)
 
+    def test_shift_bool(self):
+        # Issue #21422: ensure that bool << int and bool >> int return int
+        for value in (True, False):
+            for shift in (0, 2):
+                self.assertEqual(type(value << shift), int)
+                self.assertEqual(type(value >> shift), int)
+
 
 def test_main():
     support.run_unittest(LongTest)

Lib/test/test_re.py

@@ -1223,6 +1223,11 @@ class ReTests(unittest.TestCase):
             pat.scanner(string='abracadabra', pos=3, endpos=10).search().span(),
             (7, 9))
 
+    def test_bug_20998(self):
+        # Issue #20998: Fullmatch of repeated single character pattern
+        # with ignore case.
+        self.assertEqual(re.fullmatch('[a-c]+', 'ABC', re.I).span(), (0, 3))
+
 
 class PatternReprTests(unittest.TestCase):
     def check(self, pattern, expected):

Lib/test/test_signal.py

@@ -454,7 +454,7 @@ class SiginterruptTest(unittest.TestCase):
                 stdout = first_line + stdout
                 exitcode = process.wait()
                 if exitcode not in (2, 3):
-                    raise Exception("Child error (exit code %s): %s"
+                    raise Exception("Child error (exit code %s): %r"
                                     % (exitcode, stdout))
                 return (exitcode == 3)
 

Lib/test/test_subprocess.py

@@ -786,6 +786,7 @@ class ProcessTestCase(BaseTestCase):
                              stdout=subprocess.PIPE,
                              universal_newlines=1)
         p.stdin.write("line1\n")
+        p.stdin.flush()
         self.assertEqual(p.stdout.readline(), "line1\n")
         p.stdin.write("line3\n")
         p.stdin.close()

Lib/urllib/robotparser.py

@@ -7,7 +7,7 @@
     2) PSF license for Python 2.2
 
     The robots.txt Exclusion Protocol is implemented as specified in
-    http://info.webcrawler.com/mak/projects/robots/norobots-rfc.html
+    http://www.robotstxt.org/norobots-rfc.txt
 """
 
 import urllib.parse, urllib.request
@@ -57,7 +57,7 @@ class RobotFileParser:
         except urllib.error.HTTPError as err:
             if err.code in (401, 403):
                 self.disallow_all = True
-            elif err.code >= 400:
+            elif err.code >= 400 and err.code < 500:
                 self.allow_all = True
         else:
             raw = f.read()
@@ -85,6 +85,7 @@ class RobotFileParser:
         state = 0
         entry = Entry()
 
+        self.modified()
         for line in lines:
             if not line:
                 if state == 1:
@@ -129,6 +130,12 @@ class RobotFileParser:
             return False
         if self.allow_all:
             return True
+        # Until the robots.txt file has been read or found not
+        # to exist, we must assume that no url is allowable.
+        # This prevents false positives when a user erronenously
+        # calls can_fetch() before calling read().
+        if not self.last_checked:
+            return False
         # search for given user agent matches
         # the first match counts
         parsed_url = urllib.parse.urlparse(urllib.parse.unquote(url))

Mac/README

@@ -12,6 +12,9 @@ Python on Mac OS X README
 This document provides a quick overview of some Mac OS X specific features in
 the Python distribution.
 
+OS X specific arguments to configure
+====================================
+
 * ``--enable-framework[=DIR]``
 
   If this argument is specified the build will create a Python.framework rather
@@ -121,7 +124,7 @@ on a system running OS X 10.5 or later.  The ``all`` and ``64-bit`` flavors can
 only be built with an 10.5 SDK because ``ppc64`` support was only included with
 OS X 10.5.  Although legacy ``ppc`` support was included with Xcode 3 on OS X
 10.6, it was removed in Xcode 4, versions of which were released on OS X 10.6
-and which is the current standard for OS X 10.7 and 10.8.  To summarize, the
+and which is the standard for OS X 10.7.  To summarize, the
 following combinations of SDKs and universal-archs flavors are available:
 
   * 10.4u SDK with Xcode 2 supports ``32-bit`` only
@@ -134,6 +137,8 @@ following combinations of SDKs and universal-archs flavors are available:
 
   * 10.7 and 10.8 SDKs with Xcode 4 support ``intel`` only
 
+  * 10.8 and 10.9 SDKs with Xcode 5 support ``intel`` only
+
 The makefile for a framework build will also install ``python3.4-32``
 binaries when the universal architecture includes at least one 32-bit
 architecture (that is, for all flavors but ``64-bit``).
@@ -161,7 +166,6 @@ subprocesses also run in 32-bit-mode if the main interpreter does, use
 a ``python3.4-32`` binary and use the value of ``sys.executable`` as the
 ``subprocess`` ``Popen`` executable value.
 
-
 Building and using a framework-based Python on Mac OS X.
 ========================================================
 
@@ -171,7 +175,7 @@ Building and using a framework-based Python on Mac OS X.
 
 The main reason is because you want to create GUI programs in Python. With the
 exception of X11/XDarwin-based GUI toolkits all GUI programs need to be run 
-from a Mac OSX application bundle (".app").
+from a Mac OS X application bundle (".app").
 
 While it is technically possible to create a .app without using frameworks you
 will have to do the work yourself if you really want this.
@@ -196,7 +200,7 @@ Versions/Current and you will see the familiar bin and lib directories.
 3. Do I need extra packages?
 ----------------------------
 
-Yes, probably.  If you want Tkinter support you need to get the OSX AquaTk 
+Yes, probably.  If you want Tkinter support you need to get the OS X AquaTk
 distribution, this is installed by default on Mac OS X 10.4 or later.  Be
 aware, though, that the Cocoa-based AquaTk's supplied starting with OS X
 10.6 have proven to be unstable.  If possible, you should consider
@@ -212,9 +216,9 @@ If you want Cocoa you need to get PyObjC.
 -------------------------------------
 
 This directory contains a Makefile that will create a couple of python-related
-applications (full-blown OSX .app applications, that is) in
+applications (full-blown OS X .app applications, that is) in
 "/Applications/Python <VERSION>", and a hidden helper application Python.app
-inside the Python.framework, and unix tools "python" and "pythonw" into
+inside the Python.framework, and unix tools including "python" into
 /usr/local/bin.  In addition it has a target "installmacsubtree" that installs
 the relevant portions of the Mac subtree into the Python.framework.
 
@@ -252,18 +256,18 @@ What do all these programs do?
 "IDLE.app" is an integrated development environment for Python: editor,
 debugger, etc.
 
-"PythonLauncher.app" is a helper application that will handle things when you
+"Python Launcher.app" is a helper application that will handle things when you
 double-click a .py, .pyc or .pyw file. For the first two it creates a Terminal
 window and runs the scripts with the normal command-line Python. For the
 latter it runs the script in the Python.app interpreter so the script can do
 GUI-things. Keep the ``Option`` key depressed while dragging or double-clicking
 a script to set runtime options. These options can be set persistently
-through PythonLauncher's preferences dialog.
+through Python Launcher's preferences dialog.
 
-The program ``pythonx.x`` runs python scripts from the command line.  Various
+The program ``pythonx.x`` runs python scripts from the command line.
-compatibility aliases are also installed, including ``pythonwx.x`` which
+Previously, various compatibility aliases were also installed, including
-in early releases of Python on OS X was required to run GUI programs.  In
+``pythonwx.x`` which in early releases of Python on OS X was required to run
-current releases, the ``pythonx.x`` and ``pythonwx.x`` commands are identical.
+GUI programs.  As of 3.4.0, the ``pythonwx.x`` aliases are no longer installed.
 
 How do I create a binary distribution?
 ======================================
@@ -308,7 +312,7 @@ The configure script sometimes emits warnings like the one below::
    configure: WARNING: libintl.h:     check for missing prerequisite headers?
    configure: WARNING: libintl.h: see the Autoconf documentation
    configure: WARNING: libintl.h:     section "Present But Cannot Be Compiled"
-   configure: WARNING: libintl.h: proceeding with the preprocessor's result   
+   configure: WARNING: libintl.h: proceeding with the preprocessor's result
    configure: WARNING: libintl.h: in the future, the compiler will take precedence
    configure: WARNING:     ## -------------------------------------- ##
    configure: WARNING:     ## Report this to http://bugs.python.org/ ##

Makefile.pre.in

@@ -1487,6 +1487,7 @@ TAGS::
 
 # Touch generated files
 touch:
+	cd $(srcdir); \
 	hg --config extensions.touch=Tools/hg/hgtouch.py touch -v
 
 # Sanitation targets -- clean leaves libraries, executables and tags

Misc/ACKS

@@ -24,6 +24,7 @@ Jim Ahlstrom
 Farhan Ahmad
 Matthew Ahrens
 Nir Aides
+Akira
 Yaniv Aknin
 Jyrki Alakuijala
 Steve Alexander
@@ -672,6 +673,7 @@ Mads Kiilerich
 Jason Killen
 Jan Kim
 Taek Joo Kim
+Sam Kimbrel
 W. Trevor King
 Paul Kippes
 Steve Kirsch

Misc/NEWS

@@ -10,14 +10,56 @@ Release date: 2014-05-18
 Core and Builtins
 -----------------
 
+- Issue #21418: Fix a crash in the builtin function super() when called without
+  argument and without current frame (ex: embedded Python).
+
+- Issue #21425: Fix flushing of standard streams in the interactive
+  interpreter.
+
+- Issue #21435: In rare cases, when running finalizers on objects in cyclic
+  trash a bad pointer dereference could occur due to a subtle flaw in
+  internal iteration logic.
+
 Library
 -------
 
+- Issue #10744: Fix PEP 3118 format strings on ctypes objects with a nontrivial
+  shape.
+
+- Issue #20998: Fixed re.fullmatch() of repeated single character pattern
+  with ignore case.  Original patch by Matthew Barnett.
+
+- Issue #21075: fileinput.FileInput now reads bytes from standard stream if
+  binary mode is specified.  Patch by Sam Kimbrel.
+
+- Issue #21396: Fix TextIOWrapper(..., write_through=True) to not force a
+  flush() on the underlying binary stream.  Patch by akira.
+
+- Issue #21470: Do a better job seeding the random number generator by
+  using enough bytes to span the full state space of the Mersenne Twister.
+
+- Issue #21398: Fix an unicode error in the pydoc pager when the documentation
+  contains characters not encodable to the stdout encoding.
+
+Tests
+-----
+
+- Issue #17756: Fix test_code test when run from the installed location.
+
+- Issue #17752: Fix distutils tests when run from the installed location.
+
+IDLE
+----
+
+- Issue #18104: Add idlelib/idle_test/htest.py with a few sample tests to begin
+  consolidating and improving human-validated tests of Idle. Change other files
+  as needed to work with htest.  Running the module as __main__ runs all tests.
+
 
 What's New in Python 3.4.1rc1?
 ==============================
 
-Release date: TBA
+Release date: 2014-05-05
 
 Core and Builtins
 -----------------
@@ -54,6 +96,10 @@ Library
 - Issue #21088: Bugfix for curses.window.addch() regression in 3.4.0.
   In porting to Argument Clinic, the first two arguments were reversed.
 
+- Issue #21469:  Reduced the risk of false positives in robotparser by
+  checking to make sure that robots.txt has been read or does not exist
+  prior to returning True in can_fetch().
+
 - Issue #21321: itertools.islice() now releases the reference to the source
   iterator when the slice is exhausted.  Patch by Anton Afanasyev.
 
@@ -153,6 +199,8 @@ Library
 
 - Issue #20884: Don't assume that __file__ is defined on importlib.__init__.
 
+- Issue #21499: Ignore __builtins__ in several test_importlib.test_api tests.
+
 - Issue #20879: Delay the initialization of encoding and decoding tables for
   base32, ascii85 and base85 codecs in the base64 module, and delay the
   initialization of the unquote_to_bytes() table of the urllib.parse module, to
@@ -219,6 +267,8 @@ Extension Modules
 -----------------
 
 - Issue #21276: posixmodule: Don't define USE_XATTRS on KFreeBSD and the Hurd.
+- Issue #21226: Set up modules properly in PyImport_ExecCodeModuleObject
+  (and friends).
 
 IDLE
 ----

Modules/_ctypes/_ctypes.c

@@ -288,6 +288,48 @@ _ctypes_alloc_format_string(const char *prefix, const char *suffix)
     return result;
 }
 
+/*
+  Allocate a memory block for a pep3118 format string, adding
+  the given prefix (if non-null), an additional shape prefix, and a suffix.
+  Returns NULL on failure, with the error indicator set.  If called with
+  a suffix of NULL the error indicator must already be set.
+ */
+char *
+_ctypes_alloc_format_string_with_shape(int ndim, const Py_ssize_t *shape,
+                                       const char *prefix, const char *suffix)
+{
+    char *new_prefix;
+    char *result;
+    char buf[32];
+    int prefix_len;
+    int k;
+
+    prefix_len = 32 * ndim + 3;
+    if (prefix)
+        prefix_len += strlen(prefix);
+    new_prefix = PyMem_Malloc(prefix_len);
+    if (new_prefix == NULL)
+        return NULL;
+    new_prefix[0] = '\0';
+    if (prefix)
+        strcpy(new_prefix, prefix);
+    if (ndim > 0) {
+        /* Add the prefix "(shape[0],shape[1],...,shape[ndim-1])" */
+        strcat(new_prefix, "(");
+        for (k = 0; k < ndim; ++k) {
+            if (k < ndim-1) {
+                sprintf(buf, "%"PY_FORMAT_SIZE_T"d,", shape[k]);
+            } else {
+                sprintf(buf, "%"PY_FORMAT_SIZE_T"d)", shape[k]);
+            }
+            strcat(new_prefix, buf);
+        }
+    }
+    result = _ctypes_alloc_format_string(new_prefix, suffix);
+    PyMem_Free(new_prefix);
+    return result;
+}
+
 /*
   PyCStructType_Type - a meta type/class.  Creating a new class using this one as
   __metaclass__ will call the contructor StructUnionType_new.  It replaces the
@@ -860,14 +902,21 @@ PyCPointerType_new(PyTypeObject *type, PyObject *args, PyObject *kwds)
 
     if (proto) {
         StgDictObject *itemdict = PyType_stgdict(proto);
+        const char *current_format;
         assert(itemdict);
         /* If itemdict->format is NULL, then this is a pointer to an
            incomplete type.  We create a generic format string
            'pointer to bytes' in this case.  XXX Better would be to
            fix the format string later...
         */
-        stgdict->format = _ctypes_alloc_format_string("&",
+        current_format = itemdict->format ? itemdict->format : "B";
-                      itemdict->format ? itemdict->format : "B");
+        if (itemdict->shape != NULL) {
+            /* pointer to an array: the shape needs to be prefixed */
+            stgdict->format = _ctypes_alloc_format_string_with_shape(
+                itemdict->ndim, itemdict->shape, "&", current_format);
+        } else {
+            stgdict->format = _ctypes_alloc_format_string("&", current_format);
+        }
         if (stgdict->format == NULL) {
             Py_DECREF((PyObject *)stgdict);
             return NULL;
@@ -1245,7 +1294,6 @@ PyCArrayType_new(PyTypeObject *type, PyObject *args, PyObject *kwds)
     long length;
     int overflow;
     Py_ssize_t itemsize, itemalign;
-    char buf[32];
 
     /* create the new instance (which is a class,
        since we are a metatype!) */
@@ -1295,13 +1343,7 @@ PyCArrayType_new(PyTypeObject *type, PyObject *args, PyObject *kwds)
     }
 
     assert(itemdict->format);
-    if (itemdict->format[0] == '(') {
+    stgdict->format = _ctypes_alloc_format_string(NULL, itemdict->format);
-        sprintf(buf, "(%ld,", length);
-        stgdict->format = _ctypes_alloc_format_string(buf, itemdict->format+1);
-    } else {
-        sprintf(buf, "(%ld)", length);
-        stgdict->format = _ctypes_alloc_format_string(buf, itemdict->format);
-    }
     if (stgdict->format == NULL)
         goto error;
     stgdict->ndim = itemdict->ndim + 1;

Modules/_ctypes/ctypes.h

@@ -357,6 +357,9 @@ extern void _ctypes_add_traceback(char *, char *, int);
 
 extern PyObject *PyCData_FromBaseObj(PyObject *type, PyObject *base, Py_ssize_t index, char *adr);
 extern char *_ctypes_alloc_format_string(const char *prefix, const char *suffix);
+extern char *_ctypes_alloc_format_string_with_shape(int ndim,
+                                                const Py_ssize_t *shape,
+                                                const char *prefix, const char *suffix);
 
 extern int _ctypes_simple_instance(PyObject *obj);
 

Modules/_ctypes/stgdict.c

@@ -505,7 +505,12 @@ PyCStructUnionType_update_stgdict(PyObject *type, PyObject *fields, int isStruct
             sprintf(buf, "%s:%s:", fieldfmt, fieldname);
 
             ptr = stgdict->format;
-            stgdict->format = _ctypes_alloc_format_string(stgdict->format, buf);
+            if (dict->shape != NULL) {
+                stgdict->format = _ctypes_alloc_format_string_with_shape(
+                    dict->ndim, dict->shape, stgdict->format, buf);
+            } else {
+                stgdict->format = _ctypes_alloc_format_string(stgdict->format, buf);
+            }
             PyMem_Free(ptr);
             PyMem_Free(buf);
 

Modules/_io/textio.c

@@ -1297,7 +1297,7 @@ textiowrapper_write(textio *self, PyObject *args)
     PyObject *b;
     Py_ssize_t textlen;
     int haslf = 0;
-    int needflush = 0;
+    int needflush = 0, text_needflush = 0;
 
     CHECK_INITIALIZED(self);
 
@@ -1331,8 +1331,8 @@ textiowrapper_write(textio *self, PyObject *args)
     }
 
     if (self->write_through)
-        needflush = 1;
+        text_needflush = 1;
-    else if (self->line_buffering &&
+    if (self->line_buffering &&
         (haslf ||
          PyUnicode_FindChar(text, '\r', 0, PyUnicode_GET_LENGTH(text), 1) != -1))
         needflush = 1;
@@ -1363,7 +1363,8 @@ textiowrapper_write(textio *self, PyObject *args)
     }
     self->pending_bytes_count += PyBytes_GET_SIZE(b);
     Py_DECREF(b);
-    if (self->pending_bytes_count > self->chunk_size || needflush) {
+    if (self->pending_bytes_count > self->chunk_size || needflush ||
+        text_needflush) {
         if (_textiowrapper_writeflush(self) < 0)
             return NULL;
     }

Modules/_operator.c

@@ -277,7 +277,7 @@ compare_digest(PyObject *self, PyObject *args)
         Py_buffer view_a;
         Py_buffer view_b;
 
-        if ((PyObject_CheckBuffer(a) == 0) & (PyObject_CheckBuffer(b) == 0)) {
+        if (PyObject_CheckBuffer(a) == 0 && PyObject_CheckBuffer(b) == 0) {
             PyErr_Format(PyExc_TypeError,
                          "unsupported operand types(s) or combination of types: "
                          "'%.100s' and '%.100s'",

Modules/_sre.c

@@ -505,14 +505,14 @@ pattern_dealloc(PatternObject* self)
 }
 
 LOCAL(Py_ssize_t)
-sre_match(SRE_STATE* state, SRE_CODE* pattern)
+sre_match(SRE_STATE* state, SRE_CODE* pattern, int match_all)
 {
     if (state->charsize == 1)
-        return sre_ucs1_match(state, pattern);
+        return sre_ucs1_match(state, pattern, match_all);
     if (state->charsize == 2)
-        return sre_ucs2_match(state, pattern);
+        return sre_ucs2_match(state, pattern, match_all);
     assert(state->charsize == 4);
-    return sre_ucs4_match(state, pattern);
+    return sre_ucs4_match(state, pattern, match_all);
 }
 
 LOCAL(Py_ssize_t)
@@ -576,7 +576,7 @@ pattern_match(PatternObject *self, PyObject *args, PyObject *kwargs)
 
     TRACE(("|%p|%p|MATCH\n", PatternObject_GetCode(self), state.ptr));
 
-    status = sre_match(&state, PatternObject_GetCode(self));
+    status = sre_match(&state, PatternObject_GetCode(self), 0);
 
     TRACE(("|%p|%p|END\n", PatternObject_GetCode(self), state.ptr));
     if (PyErr_Occurred())
@@ -609,12 +609,11 @@ pattern_fullmatch(PatternObject* self, PyObject* args, PyObject* kw)
     if (!string)
         return NULL;
 
-    state.match_all = 1;
     state.ptr = state.start;
 
     TRACE(("|%p|%p|FULLMATCH\n", PatternObject_GetCode(self), state.ptr));
 
-    status = sre_match(&state, PatternObject_GetCode(self));
+    status = sre_match(&state, PatternObject_GetCode(self), 1);
 
     TRACE(("|%p|%p|END\n", PatternObject_GetCode(self), state.ptr));
     if (PyErr_Occurred())
@@ -2572,7 +2571,7 @@ scanner_match(ScannerObject* self, PyObject *unused)
 
     state->ptr = state->start;
 
-    status = sre_match(state, PatternObject_GetCode(self->pattern));
+    status = sre_match(state, PatternObject_GetCode(self->pattern), 0);
     if (PyErr_Occurred())
         return NULL;
 

Modules/faulthandler.c

@@ -144,6 +144,10 @@ faulthandler_get_fileno(PyObject *file, int *p_fd)
             PyErr_SetString(PyExc_RuntimeError, "unable to get sys.stderr");
             return NULL;
         }
+        if (file == Py_None) {
+            PyErr_SetString(PyExc_RuntimeError, "sys.stderr is None");
+            return NULL;
+        }
     }
 
     result = _PyObject_CallMethodId(file, &PyId_fileno, "");

Modules/gcmodule.c

@@ -776,28 +776,40 @@ handle_legacy_finalizers(PyGC_Head *finalizers, PyGC_Head *old)
     return 0;
 }
 
+/* Run first-time finalizers (if any) on all the objects in collectable.
+ * Note that this may remove some (or even all) of the objects from the
+ * list, due to refcounts falling to 0.
+ */
 static void
-finalize_garbage(PyGC_Head *collectable, PyGC_Head *old)
+finalize_garbage(PyGC_Head *collectable)
 {
     destructor finalize;
-    PyGC_Head *gc = collectable->gc.gc_next;
+    PyGC_Head seen;
 
-    for (; gc != collectable; gc = gc->gc.gc_next) {
+    /* While we're going through the loop, `finalize(op)` may cause op, or
+     * other objects, to be reclaimed via refcounts falling to zero.  So
+     * there's little we can rely on about the structure of the input
+     * `collectable` list across iterations.  For safety, we always take the
+     * first object in that list and move it to a temporary `seen` list.
+     * If objects vanish from the `collectable` and `seen` lists we don't
+     * care.
+     */
+    gc_list_init(&seen);
+
+    while (!gc_list_is_empty(collectable)) {
+        PyGC_Head *gc = collectable->gc.gc_next;
         PyObject *op = FROM_GC(gc);
-
+        gc_list_move(gc, &seen);
         if (!_PyGCHead_FINALIZED(gc) &&
-            PyType_HasFeature(Py_TYPE(op), Py_TPFLAGS_HAVE_FINALIZE) &&
+                PyType_HasFeature(Py_TYPE(op), Py_TPFLAGS_HAVE_FINALIZE) &&
-            (finalize = Py_TYPE(op)->tp_finalize) != NULL) {
+                (finalize = Py_TYPE(op)->tp_finalize) != NULL) {
             _PyGCHead_SET_FINALIZED(gc, 1);
             Py_INCREF(op);
             finalize(op);
-            if (Py_REFCNT(op) == 1) {
-                /* op will be destroyed */
-                gc = gc->gc.gc_prev;
-            }
             Py_DECREF(op);
         }
     }
+    gc_list_merge(&seen, collectable);
 }
 
 /* Walk the collectable list and check that they are really unreachable
@@ -1006,7 +1018,7 @@ collect(int generation, Py_ssize_t *n_collected, Py_ssize_t *n_uncollectable,
     m += handle_weakrefs(&unreachable, old);
 
     /* Call tp_finalize on objects which have one. */
-    finalize_garbage(&unreachable, old);
+    finalize_garbage(&unreachable);
 
     if (check_garbage(&unreachable)) {
         revive_garbage(&unreachable);

Modules/sre.h

@@ -86,7 +86,6 @@ typedef struct {
     SRE_REPEAT *repeat;
     /* hooks */
     SRE_TOLOWER_HOOK lower;
-    int match_all;
 } SRE_STATE;
 
 typedef struct {

Modules/sre_lib.h

@@ -173,7 +173,7 @@ SRE(charset)(SRE_CODE* set, SRE_CODE ch)
     }
 }
 
-LOCAL(Py_ssize_t) SRE(match)(SRE_STATE* state, SRE_CODE* pattern);
+LOCAL(Py_ssize_t) SRE(match)(SRE_STATE* state, SRE_CODE* pattern, int match_all);
 
 LOCAL(Py_ssize_t)
 SRE(count)(SRE_STATE* state, SRE_CODE* pattern, Py_ssize_t maxcount)
@@ -259,7 +259,7 @@ SRE(count)(SRE_STATE* state, SRE_CODE* pattern, Py_ssize_t maxcount)
         /* repeated single character pattern */
         TRACE(("|%p|%p|COUNT SUBPATTERN\n", pattern, ptr));
         while ((SRE_CHAR*) state->ptr < end) {
-            i = SRE(match)(state, pattern);
+            i = SRE(match)(state, pattern, 0);
             if (i < 0)
                 return i;
             if (!i)
@@ -490,7 +490,7 @@ typedef struct {
 /* check if string matches the given pattern.  returns <0 for
    error, 0 for failure, and 1 for success */
 LOCAL(Py_ssize_t)
-SRE(match)(SRE_STATE* state, SRE_CODE* pattern)
+SRE(match)(SRE_STATE* state, SRE_CODE* pattern, int match_all)
 {
     SRE_CHAR* end = (SRE_CHAR *)state->end;
     Py_ssize_t alloc_pos, ctx_pos = -1;
@@ -507,7 +507,7 @@ SRE(match)(SRE_STATE* state, SRE_CODE* pattern)
     ctx->last_ctx_pos = -1;
     ctx->jump = JUMP_NONE;
     ctx->pattern = pattern;
-    ctx->match_all = state->match_all;
+    ctx->match_all = match_all;
     ctx_pos = alloc_pos;
 
 entrance:
@@ -739,7 +739,7 @@ entrance:
                 RETURN_FAILURE;
 
             if (ctx->pattern[ctx->pattern[0]] == SRE_OP_SUCCESS &&
-                (!ctx->match_all || ctx->ptr == state->end)) {
+                ctx->ptr == state->end) {
                 /* tail is empty.  we're finished */
                 state->ptr = ctx->ptr;
                 RETURN_SUCCESS;
@@ -824,7 +824,7 @@ entrance:
             }
 
             if (ctx->pattern[ctx->pattern[0]] == SRE_OP_SUCCESS &&
-                (!ctx->match_all || ctx->ptr == state->end)) {
+                (!match_all || ctx->ptr == state->end)) {
                 /* tail is empty.  we're finished */
                 state->ptr = ctx->ptr;
                 RETURN_SUCCESS;
@@ -1269,7 +1269,7 @@ SRE(search)(SRE_STATE* state, SRE_CODE* pattern)
                     state->ptr = ptr - (prefix_len - prefix_skip - 1);
                     if (flags & SRE_INFO_LITERAL)
                         return 1; /* we got all of it */
-                    status = SRE(match)(state, pattern + 2*prefix_skip);
+                    status = SRE(match)(state, pattern + 2*prefix_skip, 0);
                     if (status != 0)
                         return status;
                     /* close but no cigar -- try again */
@@ -1302,7 +1302,7 @@ SRE(search)(SRE_STATE* state, SRE_CODE* pattern)
             state->ptr = ++ptr;
             if (flags & SRE_INFO_LITERAL)
                 return 1; /* we got all of it */
-            status = SRE(match)(state, pattern + 2);
+            status = SRE(match)(state, pattern + 2, 0);
             if (status != 0)
                 break;
         }
@@ -1317,7 +1317,7 @@ SRE(search)(SRE_STATE* state, SRE_CODE* pattern)
             TRACE(("|%p|%p|SEARCH CHARSET\n", pattern, ptr));
             state->start = ptr;
             state->ptr = ptr;
-            status = SRE(match)(state, pattern);
+            status = SRE(match)(state, pattern, 0);
             if (status != 0)
                 break;
             ptr++;
@@ -1327,7 +1327,7 @@ SRE(search)(SRE_STATE* state, SRE_CODE* pattern)
         while (ptr <= end) {
             TRACE(("|%p|%p|SEARCH\n", pattern, ptr));
             state->start = state->ptr = ptr++;
-            status = SRE(match)(state, pattern);
+            status = SRE(match)(state, pattern, 0);
             if (status != 0)
                 break;
         }

Objects/typeobject.c

@@ -6919,9 +6919,16 @@ super_init(PyObject *self, PyObject *args, PyObject *kwds)
     if (type == NULL) {
         /* Call super(), without args -- fill in from __class__
            and first local variable on the stack. */
-        PyFrameObject *f = PyThreadState_GET()->frame;
+        PyFrameObject *f;
-        PyCodeObject *co = f->f_code;
+        PyCodeObject *co;
         Py_ssize_t i, n;
+        f = PyThreadState_GET()->frame;
+        if (f == NULL) {
+            PyErr_SetString(PyExc_RuntimeError,
+                            "super(): no current frame");
+            return -1;
+        }
+        co = f->f_code;
         if (co == NULL) {
             PyErr_SetString(PyExc_RuntimeError,
                             "super(): no code object");

PCbuild/build_ssl.py

@@ -24,6 +24,7 @@
 # python.exe build_ssl.py Release Win32
 
 import os, sys, re, shutil
+import subprocess
 
 # Find all "foo.exe" files on the PATH.
 def find_all_on_path(filename, extras = None):
@@ -46,22 +47,21 @@ def find_all_on_path(filename, extras = None):
 # is available.
 def find_working_perl(perls):
     for perl in perls:
-        fh = os.popen('"%s" -e "use Win32;"' % perl)
+        try:
-        fh.read()
+            subprocess.check_output([perl, "-e", "use Win32;"])
-        rc = fh.close()
+        except subprocess.CalledProcessError:
-        if rc:
             continue
-        return perl
+        else:
-    print("Can not find a suitable PERL:")
+            return perl
+
     if perls:
-        print(" the following perl interpreters were found:")
+        print("The following perl interpreters were found:")
         for p in perls:
             print(" ", p)
         print(" None of these versions appear suitable for building OpenSSL")
     else:
-        print(" NO perl interpreters were found on this machine at all!")
+        print("NO perl interpreters were found on this machine at all!")
     print(" Please install ActivePerl and ensure it appears on your path")
-    return None
 
 # Fetch SSL directory from VC properties
 def get_ssl_dir():

PCbuild/pythoncore.vcxproj

@@ -173,7 +173,7 @@
     </ResourceCompile>
     <PreLinkEvent>
       <Message>Generate build information...</Message>
-      <Command>"$(SolutionDir)make_buildinfo.exe" Release "$(IntDir)"</Command>
+      <Command>"$(SolutionDir)make_buildinfo.exe" Release "$(IntDir)\"</Command>
     </PreLinkEvent>
     <Link>
       <AdditionalDependencies>$(IntDir)getbuildinfo.o;%(AdditionalDependencies)</AdditionalDependencies>
@@ -209,7 +209,7 @@ IF %ERRORLEVEL% NEQ 0 (
     </ResourceCompile>
     <PreLinkEvent>
       <Message>Generate build information...</Message>
-      <Command>"$(SolutionDir)make_buildinfo.exe" Release "$(IntDir)"</Command>
+      <Command>"$(SolutionDir)make_buildinfo.exe" Release "$(IntDir)\"</Command>
     </PreLinkEvent>
     <Link>
       <AdditionalDependencies>$(IntDir)getbuildinfo.o;%(AdditionalDependencies)</AdditionalDependencies>
@@ -314,7 +314,7 @@ IF %ERRORLEVEL% NEQ 0 (
     </ResourceCompile>
     <PreLinkEvent>
       <Message>Generate build information...</Message>
-      <Command>"$(SolutionDir)make_buildinfo.exe" Release "$(IntDir)"</Command>
+      <Command>"$(SolutionDir)make_buildinfo.exe" Release "$(IntDir)\"</Command>
     </PreLinkEvent>
     <Link>
       <AdditionalDependencies>$(IntDir)getbuildinfo.o;%(AdditionalDependencies)</AdditionalDependencies>
@@ -350,7 +350,7 @@ IF %ERRORLEVEL% NEQ 0 (
     </ResourceCompile>
     <PreLinkEvent>
       <Message>Generate build information...</Message>
-      <Command>"$(SolutionDir)make_buildinfo.exe" Release "$(IntDir)"</Command>
+      <Command>"$(SolutionDir)make_buildinfo.exe" Release "$(IntDir)\"</Command>
     </PreLinkEvent>
     <Link>
       <AdditionalDependencies>$(IntDir)getbuildinfo.o;%(AdditionalDependencies)</AdditionalDependencies>
@@ -383,7 +383,7 @@ IF %ERRORLEVEL% NEQ 0 (
     </ResourceCompile>
     <PreLinkEvent>
       <Message>Generate build information...</Message>
-      <Command>"$(SolutionDir)make_buildinfo.exe" Release "$(IntDir)"</Command>
+      <Command>"$(SolutionDir)make_buildinfo.exe" Release "$(IntDir)\"</Command>
     </PreLinkEvent>
     <Link>
       <AdditionalDependencies>$(IntDir)getbuildinfo.o;%(AdditionalDependencies)</AdditionalDependencies>
@@ -419,7 +419,7 @@ IF %ERRORLEVEL% NEQ 0 (
     </ResourceCompile>
     <PreLinkEvent>
       <Message>Generate build information...</Message>
-      <Command>"$(SolutionDir)make_buildinfo.exe" Release "$(IntDir)"</Command>
+      <Command>"$(SolutionDir)make_buildinfo.exe" Release "$(IntDir)\"</Command>
     </PreLinkEvent>
     <Link>
       <AdditionalDependencies>$(IntDir)getbuildinfo.o;%(AdditionalDependencies)</AdditionalDependencies>

Python/condvar.h

@@ -60,7 +60,7 @@
 #include <pthread.h>
 
 #define PyCOND_ADD_MICROSECONDS(tv, interval) \
-do { \
+do { /* TODO: add overflow and truncation checks */ \
     tv.tv_usec += (long) interval; \
     tv.tv_sec += tv.tv_usec / 1000000; \
     tv.tv_usec %= 1000000; \
@@ -89,7 +89,7 @@ do { \
 
 /* return 0 for success, 1 on timeout, -1 on error */
 Py_LOCAL_INLINE(int)
-PyCOND_TIMEDWAIT(PyCOND_T *cond, PyMUTEX_T *mut, long us)
+PyCOND_TIMEDWAIT(PyCOND_T *cond, PyMUTEX_T *mut, PY_LONG_LONG us)
 {
     int r;
     struct timespec ts;
@@ -270,9 +270,9 @@ PyCOND_WAIT(PyCOND_T *cv, PyMUTEX_T *cs)
 }
 
 Py_LOCAL_INLINE(int)
-PyCOND_TIMEDWAIT(PyCOND_T *cv, PyMUTEX_T *cs, long us)
+PyCOND_TIMEDWAIT(PyCOND_T *cv, PyMUTEX_T *cs, PY_LONG_LONG us)
 {
-    return _PyCOND_WAIT_MS(cv, cs, us/1000);
+    return _PyCOND_WAIT_MS(cv, cs, (DWORD)(us/1000));
 }
 
 Py_LOCAL_INLINE(int)
@@ -363,9 +363,9 @@ PyCOND_WAIT(PyCOND_T *cv, PyMUTEX_T *cs)
  * 2 to indicate that we don't know.
  */
 Py_LOCAL_INLINE(int)
-PyCOND_TIMEDWAIT(PyCOND_T *cv, PyMUTEX_T *cs, long us)
+PyCOND_TIMEDWAIT(PyCOND_T *cv, PyMUTEX_T *cs, PY_LONG_LONG us)
 {
-    return SleepConditionVariableSRW(cv, cs, us/1000, 0) ? 2 : -1;
+    return SleepConditionVariableSRW(cv, cs, (DWORD)(us/1000), 0) ? 2 : -1;
 }
 
 Py_LOCAL_INLINE(int)

Python/import.c

@@ -856,7 +856,7 @@ module_dict_for_exec(PyObject *name)
         }
     }
 
-    return d;
+    return d;  /* Return a borrowed reference. */
 }
 
 static PyObject *
@@ -888,33 +888,25 @@ PyObject*
 PyImport_ExecCodeModuleObject(PyObject *name, PyObject *co, PyObject *pathname,
                               PyObject *cpathname)
 {
-    PyObject *d, *v;
+    PyObject *d, *res;
+    PyInterpreterState *interp = PyThreadState_GET()->interp;
+    _Py_IDENTIFIER(_fix_up_module);
 
     d = module_dict_for_exec(name);
     if (d == NULL) {
         return NULL;
     }
 
-    if (pathname != NULL) {
+    if (pathname == NULL) {
-        v = pathname;
+        pathname = ((PyCodeObject *)co)->co_filename;
     }
-    else {
+    res = _PyObject_CallMethodIdObjArgs(interp->importlib,
-        v = ((PyCodeObject *)co)->co_filename;
+                                        &PyId__fix_up_module,
+                                        d, name, pathname, cpathname, NULL);
+    if (res != NULL) {
+        res = exec_code_in_module(name, d, co);
     }
-    Py_INCREF(v);
+    return res;
-    if (PyDict_SetItemString(d, "__file__", v) != 0)
-        PyErr_Clear(); /* Not important enough to report */
-    Py_DECREF(v);
-
-    /* Remember the pyc path name as the __cached__ attribute. */
-    if (cpathname != NULL)
-        v = cpathname;
-    else
-        v = Py_None;
-    if (PyDict_SetItemString(d, "__cached__", v) != 0)
-        PyErr_Clear(); /* Not important enough to report */
-
-    return exec_code_in_module(name, d, co);
 }
 
 

Python/pythonrun.c

@@ -1444,12 +1444,13 @@ PyRun_InteractiveOneObject(FILE *fp, PyObject *filename, PyCompilerFlags *flags)
     d = PyModule_GetDict(m);
     v = run_mod(mod, filename, d, d, flags, arena);
     PyArena_Free(arena);
-    flush_io();
     if (v == NULL) {
         PyErr_Print();
+        flush_io();
         return -1;
     }
     Py_DECREF(v);
+    flush_io();
     return 0;
 }
 

Python/thread_nt.h

@@ -77,7 +77,7 @@ EnterNonRecursiveMutex(PNRMUTEX mutex, DWORD milliseconds)
         /* wait at least until the target */
         DWORD now, target = GetTickCount() + milliseconds;
         while (mutex->locked) {
-            if (PyCOND_TIMEDWAIT(&mutex->cv, &mutex->cs, milliseconds*1000) < 0) {
+            if (PyCOND_TIMEDWAIT(&mutex->cv, &mutex->cs, (PY_LONG_LONG)milliseconds*1000) < 0) {
                 result = WAIT_FAILED;
                 break;
             }

configure

@@ -5605,7 +5605,7 @@ $as_echo_n "checking LDLIBRARY... " >&6; }
 if test "$enable_framework"
 then
   LDLIBRARY='$(PYTHONFRAMEWORKDIR)/Versions/$(VERSION)/$(PYTHONFRAMEWORK)'
-  RUNSHARED=DYLD_FRAMEWORK_PATH="`pwd`:$DYLD_FRAMEWORK_PATH"
+  RUNSHARED=DYLD_FRAMEWORK_PATH=`pwd`${DYLD_FRAMEWORK_PATH:+:${DYLD_FRAMEWORK_PATH}}
   BLDLIBRARY=''
 else
   BLDLIBRARY='$(LDLIBRARY)'
@@ -5625,7 +5625,7 @@ $as_echo "#define Py_ENABLE_SHARED 1" >>confdefs.h
     SunOS*)
 	  LDLIBRARY='libpython$(LDVERSION).so'
 	  BLDLIBRARY='-Wl,-R,$(LIBDIR) -L. -lpython$(LDVERSION)'
-	  RUNSHARED=LD_LIBRARY_PATH=`pwd`:${LD_LIBRARY_PATH}
+	  RUNSHARED=LD_LIBRARY_PATH=`pwd`${LD_LIBRARY_PATH:+:${LD_LIBRARY_PATH}}
 	  INSTSONAME="$LDLIBRARY".$SOVERSION
 	  if test "$with_pydebug" != yes
 	  then
@@ -5635,7 +5635,7 @@ $as_echo "#define Py_ENABLE_SHARED 1" >>confdefs.h
     Linux*|GNU*|NetBSD*|FreeBSD*|DragonFly*|OpenBSD*)
 	  LDLIBRARY='libpython$(LDVERSION).so'
 	  BLDLIBRARY='-L. -lpython$(LDVERSION)'
-	  RUNSHARED=LD_LIBRARY_PATH=`pwd`:${LD_LIBRARY_PATH}
+	  RUNSHARED=LD_LIBRARY_PATH=`pwd`${LD_LIBRARY_PATH:+:${LD_LIBRARY_PATH}}
 	  case $ac_sys_system in
 	      FreeBSD*)
 		SOVERSION=`echo $SOVERSION|cut -d "." -f 1`
@@ -5657,16 +5657,16 @@ $as_echo "#define Py_ENABLE_SHARED 1" >>confdefs.h
 			;;
 	  esac
 	  BLDLIBRARY='-Wl,+b,$(LIBDIR) -L. -lpython$(LDVERSION)'
-	  RUNSHARED=SHLIB_PATH=`pwd`:${SHLIB_PATH}
+	  RUNSHARED=SHLIB_PATH=`pwd`${SHLIB_PATH:+:${SHLIB_PATH}}
 	  ;;
     Darwin*)
     	LDLIBRARY='libpython$(LDVERSION).dylib'
 	BLDLIBRARY='-L. -lpython$(LDVERSION)'
-	RUNSHARED='DYLD_LIBRARY_PATH=`pwd`:${DYLD_LIBRARY_PATH}'
+	RUNSHARED=DYLD_LIBRARY_PATH=`pwd`${DYLD_LIBRARY_PATH:+:${DYLD_LIBRARY_PATH}}
 	;;
     AIX*)
 	LDLIBRARY='libpython$(LDVERSION).so'
-	RUNSHARED=LIBPATH=`pwd`:${LIBPATH}
+	RUNSHARED=LIBPATH=`pwd`${LIBPATH:+:${LIBPATH}}
 	;;
 
   esac

configure.ac

@@ -928,7 +928,7 @@ AC_MSG_CHECKING(LDLIBRARY)
 if test "$enable_framework"
 then
   LDLIBRARY='$(PYTHONFRAMEWORKDIR)/Versions/$(VERSION)/$(PYTHONFRAMEWORK)'
-  RUNSHARED=DYLD_FRAMEWORK_PATH="`pwd`:$DYLD_FRAMEWORK_PATH"
+  RUNSHARED=DYLD_FRAMEWORK_PATH=`pwd`${DYLD_FRAMEWORK_PATH:+:${DYLD_FRAMEWORK_PATH}}
   BLDLIBRARY=''
 else
   BLDLIBRARY='$(LDLIBRARY)'
@@ -946,7 +946,7 @@ if test $enable_shared = "yes"; then
     SunOS*)
 	  LDLIBRARY='libpython$(LDVERSION).so'
 	  BLDLIBRARY='-Wl,-R,$(LIBDIR) -L. -lpython$(LDVERSION)'
-	  RUNSHARED=LD_LIBRARY_PATH=`pwd`:${LD_LIBRARY_PATH}
+	  RUNSHARED=LD_LIBRARY_PATH=`pwd`${LD_LIBRARY_PATH:+:${LD_LIBRARY_PATH}}
 	  INSTSONAME="$LDLIBRARY".$SOVERSION
 	  if test "$with_pydebug" != yes
 	  then
@@ -956,7 +956,7 @@ if test $enable_shared = "yes"; then
     Linux*|GNU*|NetBSD*|FreeBSD*|DragonFly*|OpenBSD*)
 	  LDLIBRARY='libpython$(LDVERSION).so'
 	  BLDLIBRARY='-L. -lpython$(LDVERSION)'
-	  RUNSHARED=LD_LIBRARY_PATH=`pwd`:${LD_LIBRARY_PATH}
+	  RUNSHARED=LD_LIBRARY_PATH=`pwd`${LD_LIBRARY_PATH:+:${LD_LIBRARY_PATH}}
 	  case $ac_sys_system in
 	      FreeBSD*)
 		SOVERSION=`echo $SOVERSION|cut -d "." -f 1`
@@ -978,16 +978,16 @@ if test $enable_shared = "yes"; then
 			;;
 	  esac
 	  BLDLIBRARY='-Wl,+b,$(LIBDIR) -L. -lpython$(LDVERSION)'
-	  RUNSHARED=SHLIB_PATH=`pwd`:${SHLIB_PATH}
+	  RUNSHARED=SHLIB_PATH=`pwd`${SHLIB_PATH:+:${SHLIB_PATH}}
 	  ;;
     Darwin*)
     	LDLIBRARY='libpython$(LDVERSION).dylib'
 	BLDLIBRARY='-L. -lpython$(LDVERSION)'
-	RUNSHARED='DYLD_LIBRARY_PATH=`pwd`:${DYLD_LIBRARY_PATH}'
+	RUNSHARED=DYLD_LIBRARY_PATH=`pwd`${DYLD_LIBRARY_PATH:+:${DYLD_LIBRARY_PATH}}
 	;;
     AIX*)
 	LDLIBRARY='libpython$(LDVERSION).so'
-	RUNSHARED=LIBPATH=`pwd`:${LIBPATH}
+	RUNSHARED=LIBPATH=`pwd`${LIBPATH:+:${LIBPATH}}
 	;;
 
   esac