Commit Graph

23051 Commits

Author SHA1 Message Date
Michael Felt 39afa2d314 bpo-38021: Modify AIX platform_tag so it covers PEP 425 needs (GH-17303)
Provides a richer platform tag for AIX that we expect to be sufficient for PEP 425
binary distribution identification. Any backports to earlier Python versions will be
handled via setuptools.

Patch by Michael Felt.
2019-12-16 00:17:53 +10:00
Xtreak 8289e27393 bpo-36406: Handle namespace packages in doctest (GH-12520) 2019-12-13 10:06:53 -08:00
Lysandros Nikolaou 025a602af7 bpo-39031: Include elif keyword when producing lineno/col-offset info for if_stmt (GH-17582)
When parsing an "elif" node, lineno and col_offset of the node now point to the "elif" keyword and not to its condition, making it consistent with the "if" node.


https://bugs.python.org/issue39031



Automerge-Triggered-By: @pablogsal
2019-12-12 13:40:21 -08:00
Victor Stinner 0d63bacefd
bpo-38614: Use test.support.SHORT_TIMEOUT constant (GH-17566)
Replace hardcoded timeout constants in tests with SHORT_TIMEOUT of
test.support, so it's easier to ajdust this timeout for all tests at
once.

SHORT_TIMEOUT is 30 seconds by default, but it can be longer
depending on --timeout command line option.

The change makes almost all timeouts longer, except
test_reap_children() of test_support which is made 2x shorter:
SHORT_TIMEOUT should be enough. If this test starts to fail,
LONG_TIMEOUT should be used instead.

Uniformize also "from test import support" import in some test files.
2019-12-11 11:30:03 +01:00
Jason R. Coombs b7a0109cd2
bpo-39022, bpo-38594: Sync with importlib_metadata 1.3 (GH-17568)
* bpo-39022, bpo-38594: Sync with importlib_metadata 1.3 including improved docs for custom finders and better serialization support in EntryPoints.

* 📜🤖 Added by blurb_it.

* Correct module reference
2019-12-10 20:05:10 -05:00
Victor Stinner 07871b256c
bpo-38614: Use test.support.LOOPBACK_TIMEOUT constant (GH-17554)
Replace hardcoded timeout constants in tests with LOOPBACK_TIMEOUT of
test.support, so it's easier to ajdust this timeout for all tests at
once.
2019-12-10 20:32:59 +01:00
Pablo Galindo 680068c288
Minor fixes to the NEWS entries (GH-17556) 2019-12-10 16:09:58 +00:00
JohnnyNajera bbc4162baf bpo-38943: Fix IDLE autocomplete window not always appearing (GH-17416)
This has happened on some versions of Ubuntu.
2019-12-09 19:30:01 -05:00
JohnnyNajera 232689b40d bpo-38944: Escape key now closes IDLE completion windows. (GH-17419) 2019-12-09 18:22:16 -05:00
Steve Dower ee17e37356
bpo-39007: Add auditing events to functions in winreg (GH-17541)
Also allows winreg.CloseKey() to accept same types as other functions.
2019-12-09 11:18:12 -08:00
Steve Dower b8cbe74c34
bpo-39008: Require Py_ssize_t for PySys_Audit formats rather than raise a deprecation warning (GH-17540) 2019-12-09 11:05:39 -08:00
Tim Gates c18b805ac6 bpo-39002: Fix simple typo: tranlation -> translation (GH-17517) 2019-12-09 09:42:17 -08:00
Victor Stinner a1a99b4bb7
bpo-20443: No longer make sys.argv[0] absolute for script (GH-17534)
In Python 3.9.0a1, sys.argv[0] was made an asolute path if a filename
was specified on the command line. Revert this change, since most
users expect sys.argv to be unmodified.
2019-12-09 17:34:02 +01:00
Yury Selivanov d219cc4180 bpo-34776: Fix dataclasses to support __future__ "annotations" mode (#9518) 2019-12-09 15:54:20 +01:00
Mark Dickinson bba873e633
bpo-38992: avoid fsum test failure from constant-folding (GH-17513)
* Issue 38992: avoid fsum test failure

* Add NEWS entry
2019-12-09 08:36:34 -06:00
Kyle Stanley ab513a38c9 bpo-37228: Fix loop.create_datagram_endpoint()'s usage of SO_REUSEADDR (#17311) 2019-12-09 15:21:10 +01:00
Victor Stinner 82b4950b5e
bpo-39006: Fix asyncio when the ssl module is missing (GH-17524)
Fix asyncio when the ssl module is missing: only check for
ssl.SSLSocket instance if the ssl module is available.
2019-12-09 15:02:03 +01:00
Victor Stinner 0131aba5ae
bpo-38916: array.array: remove fromstring() and tostring() (GH-17487)
array.array: Remove tostring() and fromstring() methods.  They were
aliases to tobytes() and frombytes(), deprecated since Python 3.2.
2019-12-09 14:09:14 +01:00
Victor Stinner a1838ec259
bpo-38547: Fix test_pty if the process is the session leader (GH-17519)
Fix test_pty: if the process is the session leader, closing the
master file descriptor raises a SIGHUP signal: simply ignore SIGHUP
when running the tests.
2019-12-09 11:57:05 +01:00
Batuhan Taşkaya 109fc2792a bpo-38673: dont switch to ps2 if the line starts with comment or whitespace (GH-17421)
https://bugs.python.org/issue38673
2019-12-08 20:36:27 -08:00
Abhilash Raj 3ae4ea1931
bpo-38708: email: Fix a potential IndexError when parsing Message-ID (GH-17504)
Fix a potential IndexError when passing an empty value to the message-id
parser. Instead, HeaderParseError should be raised.
2019-12-08 17:37:34 -08:00
Abhilash Raj 68157da8b4
bpo-38698: Add a new InvalidMessageID token to email header parser. (GH-17503)
This adds a new InvalidMessageID token to the email header parser which can be
used to represent invalid message-id headers in the parse tree.
2019-12-08 17:35:38 -08:00
Batuhan Taşkaya 526606baf7 bpo-38994: Implement __class_getitem__ for PathLike (GH-17498)
https://bugs.python.org/issue38994
2019-12-08 12:31:15 -08:00
Elena Oat cd90a52983 bpo-38669: patch.object now raises a helpful error (GH17034)
This means a clearer message is now shown when patch.object is called with two string arguments, rather than a class and a string argument.
2019-12-08 20:14:38 +00:00
AMIR 28c91631c2 bpo-38979: fix ContextVar "__class_getitem__" method (GH-17497)
now contextvars.ContextVar "__class_getitem__" method returns ContextVar class, not None. 


https://bugs.python.org/issue38979



Automerge-Triggered-By: @asvetlov
2019-12-08 03:35:59 -08:00
xdegaye 00ada2c1d5 bpo-38852: Set thread stack size to 8 Mb for debug builds on android platforms (GH-17337) 2019-12-08 08:40:14 +01:00
Victor Stinner 6cac113666
bpo-38991: Remove test.support.strip_python_stderr() (GH-17490)
test.support: run_python_until_end(), assert_python_ok() and
assert_python_failure() functions no longer strip whitespaces from
stderr.
2019-12-08 08:38:16 +01:00
Christian Heimes 2b7de6696b bpo-38820: OpenSSL 3.0.0 compatibility. (GH-17190)
test_openssl_version now accepts version 3.0.0.

getpeercert() no longer returns IPv6 addresses with a trailing new line.

Signed-off-by: Christian Heimes <christian@python.org>


https://bugs.python.org/issue38820
2019-12-07 08:59:36 -08:00
Anj-A 4443450fda bpo-38652: Remove provisional note for asyncio.BufferedProtocol (GH-17047)
https://bugs.python.org/issue38652
2019-12-07 04:53:12 -08:00
idomic 892f9e0777 bpo-37404: Raising value error if an SSLSocket is passed to asyncio functions (GH-16457)
https://bugs.python.org/issue37404
2019-12-07 03:52:35 -08:00
Andrew Svetlov 969ae7aca8
Make repr of C accelerated TaskWakeupMethWrapper the same as of pure Python version (GH-17484) 2019-12-07 13:23:21 +02:00
Andrew Svetlov 7ddcd0caa4
bpo-38529: Fix asyncio stream warning (GH-17474) 2019-12-07 13:22:00 +02:00
Batuhan Taşkaya dec367261e bpo-38978: Implement __class_getitem__ for asyncio objects (GH-17491)
https://bugs.python.org/issue38978
2019-12-07 03:05:07 -08:00
Benoit Hudson 723f71abf7 bpo-37931: Fix crash on OSX re-initializing os.environ (GH-15428)
On most platforms, the `environ` symbol is accessible everywhere.

In a dylib on OSX, it's not easily accessible, you need to find it with
_NSGetEnviron.

The code was caching the *value* of environ. But a setenv() can change the value,
leaving garbage at the old value. Fix: don't cache the value of environ, just
read it every time.
2019-12-06 20:15:03 +01:00
Victor Stinner e76ee1a72b
bpo-38982: Fix asyncio PidfdChildWatcher on waitpid() error (GH-17477)
If waitpid() is called elsewhere, waitpid() call fails with
ChildProcessError: use return code 255 in this case, and log a
warning. It ensure that the pidfd file descriptor is closed if this
error occurs.
2019-12-06 16:32:41 +01:00
Mario Corchero b64334cb93 bpo-36820: Break unnecessary cycle in socket.py, codeop.py and dyld.py (GH-13135)
Break cycle generated when saving an exception in socket.py, codeop.py and dyld.py as they keep alive not only the exception but user objects through the ``__traceback__`` attribute.


https://bugs.python.org/issue36820



Automerge-Triggered-By: @pablogsal
2019-12-06 06:27:38 -08:00
wim glenn efefe25443 bpo-27413: json.tool: Add --no-ensure-ascii option. (GH-17472) 2019-12-06 15:44:01 +09:00
Claudiu Popa bb815499af bpo-38698: Prevent UnboundLocalError to pop up in parse_message_id (GH-17277)
parse_message_id() was improperly using a token defined inside an exception
handler, which was raising `UnboundLocalError` on parsing an invalid value.




https://bugs.python.org/issue38698
2019-12-04 19:14:26 -08:00
Victor Stinner 8b787964e0
bpo-38965: Fix faulthandler._stack_overflow() on GCC 10 (GH-17467)
Use the "volatile" keyword to prevent tail call optimization
on any compiler, rather than relying on compiler specific pragma.
2019-12-04 21:10:06 +01:00
serge-sans-paille 7105319ada bpo-38634: Allow non-apple build to cope with libedit (GH-16986)
The readline module now detects if Python is linked to libedit at runtime
on all platforms.  Previously, the check was only done on macOS.

If Python is used as a library by a binary linking to libedit, the linker
resolves the rl_initialize symbol required by the readline module against
libedit instead of libreadline, which leads to a segfault.

Take advantage of the existing supporting code to have readline module being
compatible with both situations.
2019-12-04 17:02:57 +01:00
Inada Naoki 808769f3a4
bpo-33684: json.tool: Use utf-8 for infile and outfile. (GH-17460) 2019-12-04 18:39:31 +09:00
Daniel Himmelstein 03257949bc bpo-29636: Add --(no-)indent arguments to json.tool (GH-345) 2019-12-04 15:15:19 +09:00
Matthew Rollings a62ad4730c bpo-38945: UU Encoding: Don't let newline in filename corrupt the output format (#17418) 2019-12-02 14:25:21 -08:00
Daniel Hillier 8d62df60d8 bpo-37523: Raise ValueError for I/O operations on a closed zipfile.ZipExtFile. (GH-14658)
Raises ValueError when calling the following on a closed zipfile.ZipExtFile: read, readable, seek, seekable, tell.
2019-11-30 10:30:47 +02:00
Steve Dower bea33f5e1d
bpo-38920: Add audit hooks for when sys.excepthook and sys.unraisable hooks are invoked (GH-17392)
Also fixes some potential segfaults in unraisable hook handling.
2019-11-28 08:46:11 -08:00
Tzu-ping Chung d9aa216d49 bpo-38927: Use python -m pip to upgrade venv deps (GH-17403)
I suggest you add `bpo-NNNNN: ` as a prefix for the first commit for future PRs. Thanks!
2019-11-27 20:25:23 +00:00
Tzu-ping Chung 045d4e243d bpo-38928: Fix versionadded for venv's upgrade_deps function (GH-17404) 2019-11-27 20:21:48 +00:00
Inada Naoki ea9835c5d1
bpo-26730: Fix SpooledTemporaryFile data corruption (GH-17400)
SpooledTemporaryFile.rollback() might cause data corruption
when it is in text mode.

Co-Authored-By: Serhiy Storchaka <storchaka@gmail.com>
2019-11-27 22:22:06 +09:00
Bruno P. Kinoshita 9bbcbc9f6d bpo-38688, shutil.copytree: consume iterator and create list of entries to prevent infinite recursion (GH-17098) 2019-11-27 09:10:37 +08:00
Steve Dower c7c01ab1e5
bpo-38922: Raise code.__new__ audit event when code object replace() is called (GH-17394) 2019-11-26 16:27:50 -08:00
HongWeipeng 0b41a922f9 bpo-38045: Improve the performance of _decompose() in enum.py (GH-16483)
* Improve the performance of _decompose() in enum.py

Co-Authored-By: Brandt Bucher <brandtbucher@gmail.com>
2019-11-26 14:36:02 -08:00
Terry Jan Reedy e563a155be bpo-38892: Improve docs for audit event (GH-17361) 2019-11-26 09:07:48 -08:00
Karl Dubost bc441ed7c1 bpo-22377: Fixes documentation for %Z in datetime (GH-16507)
This fixes the issue discussed in https://bugs.python.org/issue22377
and fixes it according to the comments made by Paul Ganssle @pganssle

* It clarifies which values are acceptable in the table
* It extends the note with a clearer information on the valid values


https://bugs.python.org/issue22377
2019-11-26 08:38:41 -08:00
HongWeipeng 036fe85bd3 bpo-27145: small_ints[x] could be returned in long_add and long_sub (GH-15716) 2019-11-26 16:54:49 +09:00
Brandt Bucher 6dd9b64770 bpo-38328: Speed up the creation time of constant list and set display. (GH-17114) 2019-11-26 15:16:53 +09:00
Stefan Behnel c6a7bdb356
bpo-20928: support base-URL and recursive includes in etree.ElementInclude (#5723)
* bpo-20928: bring elementtree's XInclude support en-par with the implementation in lxml by adding support for recursive includes and a base-URL.

* bpo-20928: Support xincluding the same file multiple times, just not recursively.

* bpo-20928: Add 'max_depth' parameter to xinclude that limits the maximum recursion depth to 6 by default.

* Add news entry for updated ElementInclude support
2019-11-25 16:36:25 +01:00
Pablo Galindo 27fc3b6f3f
bpo-38870: Expose a function to unparse an ast object in the ast module (GH-17302)
Add ast.unparse() as a function in the ast module that can be used to unparse an
ast.AST object and produce a string with code that would produce an equivalent ast.AST
object when parsed.
2019-11-24 23:02:40 +00:00
Terry Jan Reedy 6bf644ec82
bpo-38862: IDLE Strip Trailing Whitespace fixes end newlines (GH-17366)
Extra newlines are removed at the end of non-shell files. If the file only has newlines after stripping other trailing whitespace, all are removed, as is done by patchcheck.py.
2019-11-24 16:29:29 -05:00
Claudiu Popa 6f03b236c1 bpo-38876: Raise pickle.UnpicklingError when loading an item from memo for invalid input (GH-17335)
The previous code was raising a `KeyError` for both the Python and C implementation.
This was caused by the specified index of an invalid input which did not exist
in the memo structure, where the pickle stores what objects it has seen.
The malformed input would have caused either a `BINGET` or `LONG_BINGET` load
from the memo, leading to a `KeyError` as the determined index was bogus.

https://bugs.python.org/issue38876



https://bugs.python.org/issue38876
2019-11-24 11:15:08 -08:00
Zac Hatfield-Dodds 665ad3dfa9 Better runtime TypedDict (GH-17214)
This patch enables downstream projects inspecting a TypedDict subclass at runtime to tell which keys are optional.

This is essential for generating test data with Hypothesis or validating inputs with typeguard or pydantic.
2019-11-24 10:48:48 +00:00
Raymond Hettinger 041d8b48a2
bpo-38881: choices() raises ValueError when all weights are zero (GH-17362) 2019-11-23 02:22:13 -08:00
Brett Cannon 84b1ff6560 bpo-38899: virtual environment activation for fish should use `source` (GH-17359)
The previously documented use of `.` is considered deprecated (https://fishshell.com/docs/current/commands.html#source).


https://bugs.python.org/issue38899



Automerge-Triggered-By: @brettcannon
2019-11-22 23:32:27 -08:00
Victor Stinner d68b592dd6
bpo-38896: Remove PyUnicode_ClearFreeList() function (GH-17354)
Remove PyUnicode_ClearFreeList() function: the Unicode free list has
been removed in Python 3.3.
2019-11-23 02:30:32 +01:00
PypeBros 14a89c4798 bpo-38686: fix HTTP Digest handling in request.py (#17045)
* fix HTTP Digest handling in request.py

There is a bug triggered when server replies to a request with `WWW-Authenticate: Digest` where `qop="auth,auth-int"` rather than mere `qop="auth"`. Having both `auth` and `auth-int` is legitimate according to the `qop-options` rule in §3.2.1 of [[https://www.ietf.org/rfc/rfc2617.txt|RFC 2617]]:
>      qop-options       = "qop" "=" <"> 1#qop-value <">
>      qop-value         = "auth" | "auth-int" | token
> **qop-options**: [...] If present, it is a quoted string **of one or more** tokens indicating the "quality of protection" values supported by the server.  The value `"auth"` indicates authentication; the value `"auth-int"` indicates authentication with integrity protection

This is description confirmed by the definition of the [//n//]`#`[//m//]//rule// extended-BNF pattern defined in §2.1 of [[https://www.ietf.org/rfc/rfc2616.txt|RFC 2616]] as 'a comma-separated list of //rule// with at least //n// and at most //m// items'.

When this reply is parsed by `get_authorization`, request.py only tests for identity with `'auth'`, failing to recognize it as one of the supported modes the server announced, and claims that `"qop 'auth,auth-int' is not supported"`.

* 📜🤖 Added by blurb_it.

* bpo-38686 review fix: remember why.

* fix trailing space in Lib/urllib/request.py

Co-Authored-By: Brandt Bucher <brandtbucher@gmail.com>
2019-11-22 15:19:08 -08:00
bcaller 1b779bfb85 bpo-38804: Fix REDoS in http.cookiejar (GH-17157)
The regex http.cookiejar.LOOSE_HTTP_DATE_RE was vulnerable to regular
expression denial of service (REDoS).

LOOSE_HTTP_DATE_RE.match is called when using http.cookiejar.CookieJar
to parse Set-Cookie headers returned by a server.
Processing a response from a malicious HTTP server can lead to extreme
CPU usage and execution will be blocked for a long time.

The regex contained multiple overlapping \s* capture groups.
Ignoring the ?-optional capture groups the regex could be simplified to

    \d+-\w+-\d+(\s*\s*\s*)$

Therefore, a long sequence of spaces can trigger bad performance.

Matching a malicious string such as

    LOOSE_HTTP_DATE_RE.match("1-c-1" + (" " * 2000) + "!")

caused catastrophic backtracking.

The fix removes ambiguity about which \s* should match a particular
space.

You can create a malicious server which responds with Set-Cookie headers
to attack all python programs which access it e.g.

    from http.server import BaseHTTPRequestHandler, HTTPServer

    def make_set_cookie_value(n_spaces):
        spaces = " " * n_spaces
        expiry = f"1-c-1{spaces}!"
        return f"b;Expires={expiry}"

    class Handler(BaseHTTPRequestHandler):
        def do_GET(self):
            self.log_request(204)
            self.send_response_only(204)  # Don't bother sending Server and Date
            n_spaces = (
                int(self.path[1:])  # Can GET e.g. /100 to test shorter sequences
                if len(self.path) > 1 else
                65506  # Max header line length 65536
            )
            value = make_set_cookie_value(n_spaces)
            for i in range(99):  # Not necessary, but we can have up to 100 header lines
                self.send_header("Set-Cookie", value)
            self.end_headers()

    if __name__ == "__main__":
        HTTPServer(("", 44020), Handler).serve_forever()

This server returns 99 Set-Cookie headers. Each has 65506 spaces.
Extracting the cookies will pretty much never complete.

Vulnerable client using the example at the bottom of
https://docs.python.org/3/library/http.cookiejar.html :

    import http.cookiejar, urllib.request
    cj = http.cookiejar.CookieJar()
    opener = urllib.request.build_opener(urllib.request.HTTPCookieProcessor(cj))
    r = opener.open("http://localhost:44020/")

The popular requests library was also vulnerable without any additional
options (as it uses http.cookiejar by default):

    import requests
    requests.get("http://localhost:44020/")

* Regression test for http.cookiejar REDoS

If we regress, this test will take a very long time.

* Improve performance of http.cookiejar.ISO_DATE_RE

A string like

"444444" + (" " * 2000) + "A"

could cause poor performance due to the 2 overlapping \s* groups,
although this is not as serious as the REDoS in LOOSE_HTTP_DATE_RE was.
2019-11-22 15:22:11 +01:00
Siwon Kang 91daa9d722 bpo-38863: Improve is_cgi() in http.server (GH-17312)
is_cgi() function of http.server library does not currently handle a
cgi script if one of the cgi_directories is located at the
sub-directory of given path. Since is_cgi() in CGIHTTPRequestHandler
class separates given path into (dir, rest) based on the first seen
'/', multi-level directories like /sub/dir/cgi-bin/hello.py is divided
into head=/sub, rest=dir/cgi-bin/hello.py then check whether '/sub'
exists in cgi_directories = [..., '/sub/dir/cgi-bin'].
This patch makes the is_cgi() keep expanding dir part to the next '/'
then checking if that expanded path exists in the cgi_directories.

Signed-off-by: Siwon Kang <kkangshawn@gmail.com>





https://bugs.python.org/issue38863
2019-11-22 01:13:05 -08:00
benedwards14 0aca3a3a1e bpo-37838: get_type_hints for wrapped functions with forward reference (GH-17126)
https://bugs.python.org/issue37838
2019-11-21 17:24:58 +00:00
Victor Stinner 3ab479a2d1
bpo-38692: Skip test_posix.test_pidfd_open() on EPERM (GH-17290)
Skip the test_posix.test_pidfd_open() test if os.pidfd_open() fails
with a PermissionError. This situation can happen in a Linux sandbox
using a syscall whitelist which doesn't allow the pidfd_open()
syscall yet (like systemd-nspawn).
2019-11-21 12:54:54 +01:00
Victor Stinner 0127bb1c5c
bpo-38875: test_capi: trashcan tests require cpu resource (GH-17314)
test_capi: trashcan tests now require the test "cpu" resource.
2019-11-21 12:54:02 +01:00
Mark Shannon fee552669f
Produce cleaner bytecode for 'with' and 'async with' by generating separate code for normal and exceptional paths. (#6641)
Remove BEGIN_FINALLY, END_FINALLY, CALL_FINALLY and POP_FINALLY bytecodes. Implement finally blocks by code duplication.
Reimplement frame.lineno setter using line numbers rather than bytecode offsets.
2019-11-21 09:11:43 +00:00
Jason Fried 046442d02b bpo-38857: AsyncMock fix for awaitable values and StopIteration fix [3.8] (GH-17269) 2019-11-20 16:27:51 -08:00
xdegaye 559bad1a70 bpo-38841: Skip asyncio test_create_datagram_endpoint_existing_sock_unix (GH-17294)
on platforms lacking a functional bind() for named unix domain sockets



https://bugs.python.org/issue38841



Automerge-Triggered-By: @asvetlov
2019-11-20 12:02:07 -08:00
Steve Dower de148f263f
bpo-33125: Add support for building and releasing Windows ARM64 packages (GH-16828)
Note that the support is not actually enabled yet, and so we won't be publishing these packages. However, for those who want to build it themselves (even by reusing the Azure Pipelines definition), it's now relatively easy to enable.
2019-11-20 09:30:47 -08:00
Federico Bond be5c79e033 bpo-38821: Fix crash in argparse when using gettext (GH-17192) 2019-11-20 15:29:29 +02:00
Victor Stinner 4dedd0f0dd
bpo-37340: Remove PyMethod_ClearFreeList() and PyCFunction_ClearFreeList() (GH-17284)
Remove PyMethod_ClearFreeList() and PyCFunction_ClearFreeList()
functions: the free lists of bound method objects have been removed.

Remove also _PyMethod_Fini() and _PyCFunction_Fini() functions.
2019-11-20 12:59:12 +01:00
Victor Stinner 7247407c35
bpo-36854: Move _PyRuntimeState.gc to PyInterpreterState (GH-17287)
* Rename _PyGC_InitializeRuntime() to _PyGC_InitState()
* finalize_interp_clear() now also calls _PyGC_Fini() in
  subinterpreters (clear the GC state).
2019-11-20 12:25:50 +01:00
Victor Stinner 488d02a241
bpo-38835: Exclude PyFPE macros from the stable API (GH-17228)
Exclude PyFPE_START_PROTECT() and PyFPE_END_PROTECT() macros of
pyfpe.h from Py_LIMITED_API (stable API).
2019-11-20 12:17:09 +01:00
Terry Jan Reedy b8462477bf
bpo-38636: Fix IDLE tab toggle and file indent width (GH-17008)
These Format menu functions (default shortcuts Alt-T and Alt-U)
were mistakenly disabled in 3.7.5 and 3.8.0.
2019-11-20 01:18:39 -05:00
Benjamin Peterson 7483451577
closes bpo-38712: Add signal.pidfd_send_signal. (GH-17070)
This exposes a Linux-specific syscall for sending a signal to a process
identified by a file descriptor rather than a pid.

For simplicity, we don't support the siginfo_t parameter to the syscall. This
parameter allows implementing a pidfd version of rt_sigqueueinfo(2), which
Python also doesn't support.
2019-11-19 20:39:14 -08:00
Victor Stinner be143ec996
bpo-38835: Don't use PyFPE_START_PROTECT and PyFPE_END_PROTECT (GH-17231)
The PyFPE_START_PROTECT() and PyFPE_END_PROTECT() macros are empty:
they have been doing nothing for the last year  (since commit
735ae8d139), so stop using them.
2019-11-20 02:51:30 +01:00
Łukasz Langa 1c5a71a7dd
Merge tag 'v3.9.0a1' 2019-11-20 02:05:28 +01:00
Pablo Galindo e0cd8aa70a
bpo-37957: Allow regrtest to receive a file with test (and subtests) to ignore (GH-16989)
When building Python in some uncommon platforms there are some known tests that will fail. Right now, the test suite has the ability to ignore entire tests using the -x option and to receive a filter file using the --matchfile filter. The problem with the --matchfile option is that it receives a file with patterns to accept and when you want to ignore a couple of tests and subtests, is too cumbersome to lists ALL tests that are not the ones that you want to accept and he problem with -x is that is not easy to ignore just a subtests that fail and the whole test needs to be ignored.

For these reasons, add a new option to allow to ignore a list of test and subtests for these situations.
2019-11-19 23:46:49 +00:00
Jake Tesler c6b20be85c bpo-38707: Fix for multiprocessing.Process MainThread.native_id (GH-17088)
This PR implements a fix for `multiprocessing.Process` objects; the error occurs when Processes are created using either `fork` or `forkserver` as the `start_method`.

In these instances, the `MainThread` of the newly created `Process` object retains all attributes from its parent's `MainThread` object, including the `native_id` attribute. The resulting behavior is such that the new process' `MainThread` captures an incorrect/outdated `native_id` (the parent's instead of its own). 

This change forces the Process object to update its `native_id` attribute during the bootstrap process.

cc @vstinner





https://bugs.python.org/issue38707



Automerge-Triggered-By: @pitrou
2019-11-19 11:50:12 -08:00
Adam Johnson 892221bfa0 bpo-38839: Fix some unused functions in tests (GH-17189) 2019-11-19 11:45:20 -08:00
Vincent Michel 8e0de2a480 bpo-35409: Ignore GeneratorExit in async_gen_athrow_throw (GH-14755)
Ignore `GeneratorExit` exceptions when throwing an exception into the `aclose` coroutine of an asynchronous generator.





https://bugs.python.org/issue35409
2019-11-19 05:53:52 -08:00
Łukasz Langa fd757083df
Python 3.9.0a1 2019-11-19 12:17:21 +01:00
Batuhan Taşkaya 24555ce2f9 bpo-21767: explicitly mention abc support in functools.singledispatch docs (#17171) 2019-11-19 09:16:46 +01:00
Tomás Farías fe75b62575 bpo-38807: Add os.PathLike to exception message raised by _check_arg_types (#17160) 2019-11-18 21:54:00 -08:00
Steve Dower 00923c6399
bpo-38622: Add missing audit events for ctypes module (GH-17158) 2019-11-18 11:32:46 -08:00
jsnklln e243bae999 bpo-38722: Runpy use io.open_code() (GH-17234)
https://bugs.python.org/issue38722



Automerge-Triggered-By: @taleinat
2019-11-18 11:11:13 -08:00
Tal Einat ee703cbb41 bpo-38809: Windows build scripts use python.exe from virtual envs (GH-17164)
https://bugs.python.org/issue38809
2019-11-18 09:32:25 -08:00
Victor Stinner 04394df74b
bpo-38631: Avoid Py_FatalError() in float.__getformat__() (GH-17232)
Replace Py_FatalError() with a regular RuntimeError exception in
float.__getformat__().
2019-11-18 17:39:48 +01:00
Victor Stinner 59c80889ff
Revert "bpo-38811: Check for presence of os.link method in pathlib. (GH-17170)" (#17219)
This reverts commit 111772fc27.
2019-11-18 12:26:37 +01:00
Zackery Spytz 2bc343417a bpo-36589: Fix the error handling in curses.update_lines_cols(). (GH-12766)
Return None instead of 1.
2019-11-17 19:10:13 +02:00
Toke Høiland-Jørgensen 111772fc27 bpo-38811: Check for presence of os.link method in pathlib. (GH-17170)
Fix also the Path.symplink() method implementation for the case when
symlinks are not supported.
2019-11-17 19:06:38 +02:00
Andrey Doroschenko 645005e947 bpo-38724: Implement subprocess.Popen.__repr__ (GH-17151) 2019-11-17 16:08:31 +02:00
Serhiy Storchaka a0652328a2
bpo-28286: Deprecate opening GzipFile for writing implicitly. (GH-16417)
Always specify the mode argument for writing.
2019-11-16 18:56:57 +02:00
Serhiy Storchaka bd44a7ead9
bpo-38650: Constify PyStructSequence_UnnamedField. (GH-17005)
Make it a constant and referring to a constant string.
2019-11-16 18:55:29 +02:00
Serhiy Storchaka 5fd5cb8d85
bpo-38639: Optimize floor(), ceil() and trunc() for floats. (GH-16991) 2019-11-16 18:00:57 +02:00
Eric Snow 73cdb0c6b2 bpo-38816: Add notes in the C-API docs about fork in subinterpreters. (GH-17176)
The C-API docs are a bit sparse on the interplay between C `fork()` and the CPython runtime.  This change adds some more information on the subject.


https://bugs.python.org/issue38816
2019-11-15 13:28:54 -08:00
Steve Dower abde52cd8e
bpo-38453: Ensure ntpath.realpath correctly resolves relative paths (GH-16967)
Ensure isabs() is always True for \\?\ prefixed paths
Avoid unnecessary usage of readlink() to avoid resolving broken links incorrectly
Ensure shutil tests run in test directory
2019-11-15 09:49:21 -08:00
Phil Connell b22030073b bpo-38778: Document that os.fork is not allowed in subinterpreters (GH-17123)
Small docs update for [bpo-34651](https://bugs.python.org/issue34651).

Other references to fork (e.g. the PyOS.*Fork functions or discussions of fork() when embedding Python) point back to os.fork, so I don't think any other updates are needed.





https://bugs.python.org/issue38778



Automerge-Triggered-By: @ericsnowcurrently
2019-11-15 08:56:03 -08:00
Andrey Doroschenko e8acc865a3 bpo-38351: Modernize email examples from %-formatting to f-strings (GH-17162) 2019-11-15 11:03:46 +02:00
Benjamin Peterson 3ccdd9b180
closes bpo-38692: Add a pidfd child process watcher to asyncio. (GH-17069) 2019-11-13 19:08:50 -08:00
Andrew Svetlov dad6be5ffe bpo-38785: Prevent asyncio from crashing (GH-17144)
if parent `__init__` is not called from a constructor of object derived from `asyncio.Future`



https://bugs.python.org/issue38785
2019-11-13 13:36:46 -08:00
Kirill 61289d4366 bpo-38786: Add parsing of https links to pydoc (GH-17143) 2019-11-13 18:13:52 +02:00
Zackery Spytz 9c2844927d bpo-4630: Add cursor no-blink option for IDLE (GH-16960)
This immediately toggles shell, editor, and output windows, but does not affect other input widgets.
2019-11-13 02:13:33 -05:00
Ilya Kulakov d6d6e2aa02 Add Ilya Kulakov to Misc/ACKS. (GH-17130)
Contributions on bpo-26467 and bpo-29302.
2019-11-12 18:33:04 -08:00
Benjamin Peterson 74fa9f723f
closes bpo-27805: Ignore ESPIPE in initializing seek of append-mode files. (GH-17112)
This change, which follows the behavior of C stdio's fdopen and Python 2's file object, allows pipes to be opened in append mode.
2019-11-12 14:51:34 -08:00
jsnklln d593881505 bpo-38723: Pdb._runscript should use io.open_code() instead of open() (GH-17127)
Co-Authored-By: Brandt Bucher <brandtbucher@gmail.com>
2019-11-12 14:42:47 -08:00
Zackery Spytz c8b53dc3d8 bpo-26353: IDLE adds an unneeded newline when saving a shell window (GH-17103) 2019-11-12 05:54:10 -05:00
Manjusaka 051ff526b5 bpo-38565: add new cache_parameters method for lru_cache (GH-16916) 2019-11-11 23:30:18 -08:00
Brandt Bucher a0ed99bca8 bpo-38438: Simplify argparse "star nargs" usage. (GH-17106) 2019-11-11 12:47:48 -08:00
Raymond Hettinger 84ac437658
bpo-38761: Register WeakSet as a MutableSet (GH-17104) 2019-11-10 20:12:04 -08:00
Inada Naoki 6cbc84fb99
bpo-38613: Optimize set operations of dict keys. (GH-16961) 2019-11-08 00:59:04 +09:00
l0rb 519cb8772a bpo-38716: stop rotating handlers from setting inherited namer and rotator to None (GH-17072) 2019-11-06 21:21:40 +00:00
Benjamin Peterson 5c0c325453 closes bpo-38713: Expose P_PIDFD in os if it's defined. (GH-17071)
https://bugs.python.org/issue38713
2019-11-05 21:58:31 -08:00
Benjamin Peterson 6c4c45efae
bpo-38692: Add os.pidfd_open. (GH-17063) 2019-11-05 19:21:29 -08:00
Jeroen Demeyer bf17d41826 bpo-37645: add new function _PyObject_FunctionStr() (GH-14890)
Additional note: the `method_check_args` function in `Objects/descrobject.c` is written in such a way that it applies to all kinds of descriptors. In particular, a future re-implementation of `wrapper_descriptor` could use that code.

CC @vstinner @encukou 


https://bugs.python.org/issue37645



Automerge-Triggered-By: @encukou
2019-11-05 07:48:04 -08:00
Eddie Elizondo b3966639d2 bpo-35381 Remove all static state from posixmodule (GH-15892)
After #9665, this moves the remaining types in posixmodule to be heap-allocated to make it compatible with PEP384 as well as modifying all the type accessors to fully make the type opaque.

The original PR that got messed up a rebase: https://github.com/python/cpython/pull/10854. All the issues in that commit have now been addressed since https://github.com/python/cpython/pull/11661 got committed.

This change also removes any state from the data segment and onto the module state itself.


https://bugs.python.org/issue35381



Automerge-Triggered-By: @encukou
2019-11-05 07:16:14 -08:00
Benjamin Peterson 62161ce989
closes bpo-37633: Reëxport some function compatibility wrappers for macros in ``pythonrun.h``. (GH-17056) 2019-11-04 21:34:14 -08:00
Victor Stinner f4b1e3d7c6
bpo-38644: Add Py_EnterRecursiveCall() to the limited API (GH-17046)
Provide Py_EnterRecursiveCall() and Py_LeaveRecursiveCall() as
regular functions for the limited API. Previously, there were defined
as macros, but these macros didn't work with the limited API which
cannot access PyThreadState.recursion_depth field.

Remove _Py_CheckRecursionLimit from the stable ABI.

Add Include/cpython/ceval.h header file.
2019-11-04 19:48:34 +01:00
Alexandru Ardelean 6552563b3d bpo-38684: haslib: fix build when Blake2 not enabled in OpenSSL (#17043) 2019-11-04 15:55:56 +01:00
Ram Rachum 8d4fef4ee2 bpo-38422: Clarify docstrings of pathlib suffix(es) (GH-16679)
Whenever I use `path.suffix` I have to check again whether it includes the dot or not. I decided to add it to the docstring so I won't have to keep checking. 


https://bugs.python.org/issue38422



Automerge-Triggered-By: @pitrou
2019-11-02 09:46:24 -07:00
Anthony Sottile b32cb97bce bpo-38312: Add curses.{get,set}_escdelay and curses.{get,set}_tabsize. (GH-16938) 2019-10-31 11:13:48 +02:00
Lucas Cimon b15100fe7d bpo-38586: setting logging.Handler .name property in fileConfig (GH-16918) 2019-10-31 08:06:25 +00:00
Victor Stinner a4ed6ed9f3
bpo-38614: Increase asyncio test_communicate() timeout (GH-16995)
Fix test_communicate() of test_asyncio.test_subprocess: use
support.LONG_TIMEOUT (5 minutes), instead of 1 minute.
2019-10-30 16:00:44 +01:00
Pablo Galindo 6c3e66a34b
bpo-38640: Allow break and continue in always false while loops (GH-16992) 2019-10-30 11:53:26 +00:00
Victor Stinner 24c6258269
bpo-38614: Add timeout constants to test.support (GH-16964)
Add timeout constants to test.support:

* LOOPBACK_TIMEOUT
* INTERNET_TIMEOUT
* SHORT_TIMEOUT
* LONG_TIMEOUT
2019-10-30 12:41:43 +01:00
Serhiy Storchaka 865c3b257f
bpo-28029: Make "".replace("", s, n) returning s for any n != 0. (GH-16981) 2019-10-30 12:03:53 +02:00
Daniel Hillier da6ce58dd5 bpo-36993: Improve error reporting for zipfiles with bad zip64 extra data. (GH-14656) 2019-10-29 09:24:18 +02:00
Steve Dower 0ac6137dd3
bpo-38589: Fixes HTML Help shortcut when Windows is not installed to C drive (GH-16968) 2019-10-28 14:34:25 -07:00
Zackery Spytz edb172a872 bpo-38519: Internal include files missing on Windows (GH-16921) 2019-10-28 10:03:27 -07:00
Victor Stinner e471e72977
bpo-37330: open() no longer accept 'U' in file mode (GH-16959)
open(), io.open(), codecs.open() and fileinput.FileInput no longer
accept "U" ("universal newline") in the file mode. This flag was
deprecated since Python 3.3.
2019-10-28 15:40:08 +01:00
Dong-hee Na 3bfc8e0fcc bpo-38602: Add fcntl.F_OFD_XXXX for fcntlmodule (GH-16956) 2019-10-28 09:31:15 +02:00
Marco Rougeth 85c6f8c65c bpo-38592 Add pt-br switcher to Python Docs website (GH-16924)
* 📜🤖 Added by blurb_it.
2019-10-27 09:30:12 +01:00
Serhiy Storchaka 5c32af7522
bpo-38334: Fix seeking backward on an encrypted zipfile.ZipExtFile. (GH-16937)
Test by Daniel Hillier.
2019-10-27 10:22:14 +02:00
Terry Jan Reedy e3f90b217a
bpo-38598: Do not try to compile IDLE shell or output windows (GH-16939) 2019-10-26 21:15:10 -04:00
Serhiy Storchaka 26ae9f6d3d
bpo-38535: Fix positions for AST nodes for calls without arguments in decorators. (GH-16861) 2019-10-26 16:46:05 +03:00
Zsolt Dollenstein 96b06aefe2 bpo-33348: parse expressions after * and ** in lib2to3 (GH-6586)
These are valid even in python 2.7


https://bugs.python.org/issue33348



Automerge-Triggered-By: @gpshead
2019-10-23 23:19:07 -07:00
Girts a01ba333af bpo-30618: add readlink to pathlib.Path (GH-8285)
This adds a "readlink" method to pathlib.Path objects that calls through
to os.readlink.


https://bugs.python.org/issue30618



Automerge-Triggered-By: @gpshead
2019-10-23 14:18:40 -07:00
Victor Stinner 1b53a24fb4
bpo-34679: ProactorEventLoop only uses set_wakeup_fd() in main thread (GH-16901)
bpo-34679, bpo-38563: asyncio.ProactorEventLoop.close() now only calls
signal.set_wakeup_fd() in the main thread.
2019-10-23 17:25:29 +02:00
Victor Stinner 028f7349a0
bpo-37415: Fix stdatomic.h header check for ICC compiler (GH-16717)
Fix stdatomic.h header check for ICC compiler: the ICC implementation
lacks atomic_uintptr_t type which is needed by Python.

Test:

* atomic_int and atomic_uintptr_t types
* atomic_load_explicit() and atomic_store_explicit()
* memory_order_relaxed and memory_order_seq_cst constants

But don't test ATOMIC_VAR_INIT(): it's not used in Python.
2019-10-22 21:53:50 +02:00
Tyler Kieft f548a3e4a2 bpo-38303: Make audioop extension module PEP-384 compatible (GH-16497)
https://bugs.python.org/issue38303



Automerge-Triggered-By: @encukou
2019-10-22 06:07:03 -07:00
Serhiy Storchaka 10ecbadb79
bpo-31202: Preserve case of literal parts in Path.glob() on Windows. (GH-16860) 2019-10-21 20:37:15 +03:00
Serhiy Storchaka 5bc6a7c06e
bpo-38540: Fix possible leak in PyArg_Parse for "es#" and "et#". (GH-16869) 2019-10-21 11:35:07 +03:00
Dong-hee Na 2eba6ad7bf bpo-38493: Add os.CLD_KILLED and os.CLD_STOPPED. (GH-16821) 2019-10-21 10:01:05 +03:00
Hai Shi 06cd5b6acd bpo-38465: Convert the type of exports counters to Py_ssize_t. (GH-16746) 2019-10-21 09:31:46 +03:00
Raymond Hettinger 58ccd201fa
bpo-36321: Fix misspelled attribute name in namedtuple() (GH-16858) 2019-10-20 10:19:47 -07:00
Dong-hee Na 24dc2f8c56 bpo-38525: Fix a segmentation fault when using reverse iterators of empty dict (GH-16846)
The reverse iterator for empty dictionaries was not handling correctly shared-key dictionaries.
2019-10-19 21:01:08 +01:00
Raymond Hettinger 5eabec022b
bpo-38521: Fix error in NormalDist.__eq__() (GH-16840) 2019-10-18 14:20:35 -07:00
Victor Stinner ecb035cd14
bpo-38502: regrtest uses process groups if available (GH-16829)
test.regrtest now uses process groups in the multiprocessing mode
(-jN command line option) if process groups are available: if
os.setsid() and os.killpg() functions are available.
2019-10-18 15:49:08 +02:00
Tim Graham 5a88d50ff0 bpo-27657: Fix urlparse() with numeric paths (#661)
* bpo-27657: Fix urlparse() with numeric paths

Revert parsing decision from bpo-754016 in favor of the documented
consensus in bpo-16932 of how to treat strings without a // to
designate the netloc.

* bpo-22891: Remove urlsplit() optimization for 'http' prefixed inputs.
2019-10-18 06:07:20 -07:00
Steve Dower fbe3c76c7c bpo-38418: Fixes audit event for os.system to be named 'os.system' (GH-16670)
https://bugs.python.org/issue38418



Automerge-Triggered-By: @zooba
2019-10-18 00:52:15 -07:00
Gregory P. Smith f33c57d5c7
bpo-33604: Raise TypeError on missing hmac arg. (GH-16805)
Also updates the documentation to clarify the situation surrounding
the digestmod parameter that is required despite its position in the
argument list as of 3.8.0 as well as removing old python2 era
references to "binary strings".

We indavertently had this raise ValueError in 3.8.0 for the missing
arg.  This is not considered an API change as no reasonable code would
be catching this missing argument error in order to handle it.
2019-10-17 20:30:42 -07:00
Inada Naoki 9c11029bb4
bpo-37587: json: Use _PyUnicodeWriter when scanning string. (GH-15591) 2019-10-17 16:12:41 +09:00
Steve Dower 7aebbd1182
bpo-38492: Remove pythonw.exe dependency on the Microsoft C++ runtime (GH-16824) 2019-10-16 10:27:17 -07:00
Victor Stinner fab4ef2df0
bpo-35998: Fix test_asyncio.test_start_tls_server_1() (GH-16815)
main() is now responsible to send the ANSWER, rather than
ServerProto. main() now waits until it got the HELLO before sending
the ANSWER over the new transport.

Previously, there was a race condition between main() replacing the
protocol and the protocol sending the ANSWER once it gets the HELLO.

TLSv1.3 was disabled for the test: reenable it.
2019-10-16 02:36:42 +02:00
Joannah Nanjekye 3cd21aa6a1
bpo-38468 : Refactor python-config (#16749)
* refactor python-config

* 📜🤖 Added by blurb_it.

* Update Misc/NEWS.d/next/Build/2019-10-13-16-18-24.bpo-38468.buCO84.rst

Co-Authored-By: Pablo Galindo <Pablogsal@gmail.com>

* Use getvar for pyver

* Update news entry with affected file.

* Update Misc/NEWS.d/next/Build/2019-10-13-16-18-24.bpo-38468.buCO84.rst

Co-Authored-By: Victor Stinner <vstinner@python.org>

* Update 2019-10-13-16-18-24.bpo-38468.buCO84.rst
2019-10-15 11:18:47 -03:00
Julien Danjou 8d59eb1b66 bpo-37961, tracemalloc: add Traceback.total_nframe (GH-15545)
Add a total_nframe field to the traces collected by the tracemalloc module.
This field indicates the original number of frames before it was truncated.
2019-10-15 14:00:16 +02:00
Pablo Galindo f3ef06a7cb
bpo-38478: Correctly handle keyword argument with same name as positional-only parameter (GH-16800) 2019-10-15 12:40:02 +01:00
Victor Stinner eb1dda2b56 bpo-38470: Fix test_compileall.test_compile_dir_maxlevels() (GH-16789)
Fix test_compile_dir_maxlevels() on Windows without long path
support: only create 3 subdirectories instead of between 20 and 100
subdirectories.

Fix also compile_dir() to use the current sys.getrecursionlimit()
value as the default maxlevels value, rather than using
sys.getrecursionlimit() value read at startup.
2019-10-15 11:26:13 +02:00
Vinay Sajip 0b60f64e43
bpo-11410: Standardize and use symbol visibility attributes across POSIX and Windows. (GH-16347) 2019-10-15 08:26:12 +01:00
Stein Karlsen aad2ee0156 bpo-32498: urllib.parse.unquote also accepts bytes (GH-7768) 2019-10-14 13:36:29 +03:00
Pablo Galindo fd5c414880
bpo-38469: Handle named expression scope with global/nonlocal keywords (GH-16755)
The symbol table handing of PEP572's assignment expressions is not resolving correctly the scope of some variables in presence of global/nonlocal keywords in conjunction with comprehensions.
2019-10-14 05:18:05 +01:00
Pablo Galindo 466326dcdf
bpo-38379: Don't block collection of unreachable objects when some objects resurrect (GH-16687)
Currently if any finalizer invoked during garbage collection resurrects any object, the gc gives up and aborts the collection. Although finalizers are assured to only run once per object, this behaviour of the gc can lead to an ever-increasing memory situation if new resurrecting objects are allocated in every new gc collection.

To avoid this, recompute what objects among the unreachable set need to be resurrected and what objects can be safely collected. In this way, resurrecting objects will not block the collection of other objects in the unreachable set.
2019-10-13 16:48:59 +01:00
Samuel Colvin 793cb85437 bpo-38431: Fix __repr__ method of InitVar to work with typing objects. (GH-16702) 2019-10-13 14:45:36 +03:00
Serhiy Storchaka 140a7d1f35
bpo-38378: Rename parameters "out" and "in" of os.sendfile(). (GH-16742)
They conflicted with keyword "in".

Also rename positional-only parameters of private os._fcopyfile()
for consistency.
2019-10-13 11:59:31 +03:00
Gregory P. Smith f3751efb5c
bpo-38417: Add umask support to subprocess (GH-16726)
On POSIX systems, allow the umask to be set in the child process before we exec.
2019-10-12 13:24:56 -07:00
Samuel Colvin 822922af90 bpo-35800: Deprecate smtpd.MailmanProxy (GH-11675)
Since `smtpd.MailmanProxy` is already broken, it is not formally deprecated in 3.9. It will be removed in 3.10.


https://bugs.python.org/issue35800
2019-10-12 10:24:26 -07:00
Abhilash Raj 19a3d87300 bpo-38449: Revert "bpo-22347: Update mimetypes.guess_type to allow oper parsing of URLs (GH-15522)" (GH-16724)
This reverts commit 87bd2071c7.



https://bugs.python.org/issue38449
2019-10-11 22:41:35 -07:00
Ruediger Pluem 2b7dc40b2a bpo-38347: find pathfix for Python scripts whose name contain a '-' (GH-16536)
pathfix.py: Assume all files that end on '.py' are Python scripts when working recursively.
2019-10-11 15:36:50 +02:00
Pablo Galindo 320dd504dd
bpo-38437: Activate GC_DEBUG when PY_DEBUG is set (GH-16707) 2019-10-10 22:45:17 +01:00
Ronan Lamy 7bb14316b8 bpo-38109: Add missing constants to Lib/stat.py (GH-16665)
Add missing stat.S_IFDOOR, stat.S_IFPORT, stat.S_IFWHT,
stat.S_ISDOOR, stat.S_ISPORT, and stat.S_ISWHT values to
the Python implementation of the stat module.
2019-10-10 09:34:46 +02:00
Tim Peters ecbf35f933
bpo-38379: don't claim objects are collected when they aren't (#16658)
* bpo-38379:  when a finalizer resurrects an object,
nothing is actually collected in this run of gc.
Change the stats to relect that truth.
2019-10-09 12:37:30 -05:00
Antonio Gutierrez 594e2edfb5 closes bpo-36161: Use thread-safe ttyname_r instead of ttyname. (GH-14868)
Signed-off-by: Antonio Gutierrez <chibby0ne@gmail.com>
2019-10-08 19:19:48 -07:00
Victor Stinner 0ec618af98
bpo-37531: regrtest ignores output on timeout (GH-16659)
bpo-37531, bpo-38207: On timeout, regrtest no longer attempts to call
`popen.communicate() again: it can hang until all child processes
using stdout and stderr pipes completes. Kill the worker process and
ignores its output.

Reenable test_regrtest.test_multiprocessing_timeout().

bpo-37531: Change also the faulthandler timeout of the main process
from 1 minute to 5 minutes, for Python slowest buildbots.
2019-10-08 18:45:43 +02:00
Pablo Galindo 10cd00a9e3
bpo-38395: Fix ownership in weakref.proxy methods (GH-16632)
The implementation of weakref.proxy's methods call back into the Python
API using a borrowed references of the weakly referenced object
(acquired via PyWeakref_GET_OBJECT). This API call may delete the last
reference to the object (either directly or via GC), leaving a dangling
pointer, which can be subsequently dereferenced.

To fix this, claim a temporary ownership of the referenced object when
calling the appropriate method. Some functions because at the moment they
do not need to access the borrowed referent, but to protect against
future changes to these functions, ownership need to be fixed in
all potentially affected methods.
2019-10-08 16:30:50 +01:00
Victor Stinner 03ab6b4fc6
bpo-38118: Ignore Valgrind false alarm in PyUnicode_Decode() (GH-16651)
Valgrind emits "Conditional jump or move depends on uninitialised
value(s)" false alarms on GCC builtin strcmp() function. The GCC code
is correct.

Valgrind bug: https://bugs.kde.org/show_bug.cgi?id=264936
2019-10-08 15:42:17 +02:00
Serhiy Storchaka 13abda4100
bpo-38405: Make nested subclasses of typing.NamedTuple pickleable. (GH-16641) 2019-10-08 16:29:52 +03:00
Serhiy Storchaka b690a2759e
bpo-36698: IDLE no longer fails when write non-encodable characters to stderr. (GH-16583)
It now escapes them with a backslash, as the regular Python interpreter.
Added the "errors" field to the standard streams.
2019-10-08 14:32:25 +03:00
Serhiy Storchaka d05b000c6b
bpo-38371: Tkinter: deprecate the split() method. (GH-16584) 2019-10-08 14:31:35 +03:00
Antonio Gutierrez 0d3fe8ae49 closes bpo-38402: Check error of primitive crypt/crypt_r. (GH-16599)
Checks also for encryption algorithms methods not supported in different
OSs.

Signed-off-by: Antonio Gutierrez <chibby0ne@gmail.com>
2019-10-07 21:22:17 -07:00
Victor Stinner 1b18455695
bpo-38392: PyObject_GC_Track() validates object in debug mode (GH-16615)
In debug mode, PyObject_GC_Track() now calls tp_traverse() of the
object type to ensure that the object is valid: test that objects
visited by tp_traverse() are valid.

Fix pyexpat.c: only track the parser in the GC once the parser is
fully initialized.
2019-10-08 00:09:31 +02:00
James Abel e310af9e29 bpo-38344: Fix syntax in activate.bat (GH-16533) 2019-10-07 14:07:19 -07:00
Ricardo Bánffy 15ae75d660 bpo-38294: Add list of no-longer-escaped chars to re.escape documentation. (GH-16442)
Prior to 3.7, re.escape escaped many characters that don't have
special meaning in Python, but that use to require escaping in other
tools and languages. This commit aims to make it clear which characters
were, but are no longer escaped.
2019-10-07 23:54:35 +03:00
Victor Stinner 6876257eaa
bpo-36389: _PyObject_CheckConsistency() available in release mode (GH-16612)
bpo-36389, bpo-38376: The _PyObject_CheckConsistency() function is
now also available in release mode. For example, it can be used to
debug a crash in the visit_decref() function of the GC.

Modify the following functions to also work in release mode:

* _PyDict_CheckConsistency()
* _PyObject_CheckConsistency()
* _PyType_CheckConsistency()
* _PyUnicode_CheckConsistency()

Other changes:

* _PyMem_IsPtrFreed(ptr) now also returns 1 if ptr is NULL
  (equals to 0).
* _PyBytesWriter_CheckConsistency() now returns 1 and is only used
  with assert().
* Reorder _PyObject_Dump() to write safe fields first, and only
  attempt to render repr() at the end.
2019-10-07 18:42:01 +02:00
Serhiy Storchaka ef092fe990
bpo-25988: Do not expose abstract collection classes in the collections module. (GH-10596) 2019-10-07 12:10:15 +03:00
Dong-hee Na c38e725d17 bpo-38210: Fix intersection operation with dict view and iterator. (GH-16602) 2019-10-06 14:28:33 +03:00
Andrei Troie 65dcc8a8dc bpo-38332: Catch KeyError from unknown cte in encoded-word. (GH-16503)
KeyError should cause a failure in parsing the encoded word and should be caught and raised as a _InvalidEWError instead.
2019-10-05 09:19:15 -07:00
nde 3faf826e58 bpo-38341: Add SMTPNotSupportedError in the exports of smtplib (#16525)
Add SMTPNotSupportedError in the exports of smtplib

Co-Authored-By: Brandt Bucher <brandtbucher@gmail.com>
2019-10-04 17:30:58 -07:00
Joannah Nanjekye 8855e47d09 bpo-38266: Revert bpo-37878: Make PyThreadState_DeleteCurrent() Internal (GH-16558)
Revert the removal of PyThreadState_DeleteCurrent() with documentation.
2019-10-04 13:35:42 +02:00
Serhiy Storchaka 06cb94bc84
bpo-13153: Use OS native encoding for converting between Python and Tcl. (GH-16545)
On Windows use UTF-16 (or UTF-32 for 32-bit Tcl_UniChar) with the
"surrogatepass" error handler for converting to/from Tcl Unicode objects.

On Linux use UTF-8 with the "surrogateescape" error handler for converting
to/from Tcl String objects.

Converting strings from Tcl to Python and back now never fails
(except MemoryError).
2019-10-04 13:09:52 +03:00
Steve Dower 353fb1ecbf
bpo-38359: Ensures pyw.exe launcher reads correct registry key (GH-16561) 2019-10-03 08:31:21 -07:00
Steve Dower a0e3d27e4e
bpo-38355: Fix ntpath.realpath failing on sys.executable (GH-16551) 2019-10-03 08:31:03 -07:00
Victor Stinner 5e0ea7540f
bpo-37474: Don't call fedisableexcept() on FreeBSD (GH-16515)
On FreeBSD, Python no longer calls fedisableexcept() at startup to
control the floating point control mode. The call became useless
since FreeBSD 6: it became the default mode.
2019-10-01 13:12:29 +02:00
Victor Stinner 3c30a76f3d
bpo-38304: Remove PyConfig.struct_size (GH-16500) (GH-16508)
For now, we'll rely on the fact that the config structures aren't covered by the stable ABI.

We may revisit this in the future if we further explore the idea of offering a stable embedding API.

(cherry picked from commit bdace21b76)
2019-10-01 10:56:37 +02:00
Giampaolo Rodola 94e165096f
bpo-38319: Fix shutil._fastcopy_sendfile(): set sendfile() max block size (GH-16491) 2019-10-01 11:40:54 +08:00
Maxwell A McKinnon cf57cabef8 bpo-32689: Updates shutil.move to allow for Path objects to be used as source arg (GH-15326)
Important work originally done by @emilyemorehouse two years ago and nearly ready to go in.

This bug has affected many people and in some cases has been a dealbreaker to the adoption of the otherwise wonderful pathlib and PEP519. https://stackoverflow.com/questions/33625931/copy-file-with-pathlib-in-python.

This adds the outstanding test request from that PR @vstinner (https://github.com/python/cpython/pull/5393).

Test fails without the change, passes with it, along with every other test in test_shutil.

Some variants were experimented with to make the one line change and the most performant one was picked.


# Added Test for PathLike directory destination, the current fail case

```
Lib/test/test_shutil.py::TestMove::test_move_file_pathlike FAILED                                                               [100%]

============================================================== FAILURES ===============================================================
__________________________________________________ TestMove.test_move_file_pathlike ___________________________________________________

self = <test.test_shutil.TestMove testMethod=test_move_file_pathlike>

    def test_move_file_pathlike(self):
        # Move a file to another location on the same filesystem.
        src = pathlib.Path(self.src_file)
>       self._check_move_file(src, self.dst_dir, self.dst_file)

Lib/test/test_shutil.py:1563:
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
Lib/test/test_shutil.py:1545: in _check_move_file
    shutil.move(src, dst)
/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/shutil.py:562: in move
    real_dst = os.path.join(dst, _basename(src))
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _

path = PosixPath('/var/folders/r2/psq74t5x3nbfzlph8bh2pvdw0000gn/T/tmp9ie0wh9_/foo')

    def _basename(path):
        # A basename() variant which first strips the trailing slash, if present.
        # Thus we always get the last component of the path, even for directories.
        sep = os.path.sep + (os.path.altsep or '')
>       return os.path.basename(path.rstrip(sep))
E       AttributeError: 'PosixPath' object has no attribute 'rstrip'

/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/shutil.py:526: AttributeError
============================================== 1 failed, 102 deselected in 0.30 seconds ===============================================
```

After change:

```
========================================================= test session starts =========================================================
platform darwin -- Python 3.7.4, pytest-5.0.1, py-1.8.0, pluggy-0.12.0 -- /Users/maxwellmckinnon/.venvs/TA3.7/bin/python3.7
cachedir: .pytest_cache
rootdir: /Users/maxwellmckinnon/dev/cpython
plugins: cov-2.7.1, mock-1.10.4
collected 103 items / 102 deselected / 1 selected

Lib/test/test_shutil.py::TestMove::test_move_file_pathlike PASSED                                                               [100%]

============================================== 1 passed, 102 deselected in 0.06 seconds ===============================================
```

Running all the tests in test_shutil.py
```
╰─ pytest Lib/test/test_shutil.py -v
========================================================= test session starts =========================================================
platform darwin -- Python 3.7.4, pytest-5.0.1, py-1.8.0, pluggy-0.12.0 -- /Users/maxwellmckinnon/.venvs/TA3.7/bin/python3.7
cachedir: .pytest_cache
rootdir: /Users/maxwellmckinnon/dev/cpython
plugins: cov-2.7.1, mock-1.10.4
collected 103 items

Lib/test/test_shutil.py::TestShutil::test_chown PASSED                                                                          [  0%]
Lib/test/test_shutil.py::TestShutil::test_copy PASSED                                                                           [  1%]
...
Lib/test/test_shutil.py::TermsizeTests::test_stty_match SKIPPED                                                                 [ 99%]
Lib/test/test_shutil.py::PublicAPITests::test_module_all_attribute PASSED                                                       [100%]

================================================ 96 passed, 7 skipped in 1.25 seconds =================================================
```

# Performance Considerations
Is it considered poor form to get rid of _basename altogether and make use of pathlib in the move function? I'm not sure if the idea is for all these modules to strictly avoid circular dependencies. They are already using os.path which is just as much a citizen in 3.8 as pathlib right?

e.g.

`real_dst = os.path.join(dst, _basename(src))`
becomes
`real_dst = Path(dst) / Path(src).name`

I've looked around and familiarized myself, and I now think importing pathlib here is fine. My only remaining concern is that of performance.

Here's the performance difference for this step. 

```
In [46]: %timeit real_dst = os.path.join("a/b/c", _basename('b/'))
2.71 µs ± 62.6 ns per loop (mean ± std. dev. of 7 runs, 100000 loops each)

In [47]: %timeit real_dst = Path("a/b/c") / Path('b/').name
12.4 µs ± 65.3 ns per loop (mean ± std. dev. of 7 runs, 100000 loops each)
```

Is 10us significant or insignificant compared to the least expensive operation this function will do? I don't know. Let's find out.

```
In [55]: %timeit os.rename('/tmp/a/a.txt', '/tmp/a/b.txt'); os.rename('/tmp/a/b.txt', '/tmp/a/a.txt')
124 µs ± 2.18 µs per loop (mean ± std. dev. of 7 runs, 10000 loops each)
```
62us to rename. 10us seems significant enough that we wouldn't want to favor the Path sugar suggestion. 16% speed decrease from adding the 10us.

What do people think? I was hoping to get to use pathlib.Path here, but I suspect for this low level move, it should be as fast as possible, and 16% is not worth one line of sugary code to me.



https://bugs.python.org/issue32689



Automerge-Triggered-By: @gvanrossum
2019-09-30 19:41:16 -07:00
Neil Schemenauer bcda460baf Clear weakrefs in garbage found by the GC (#16495)
Fix a bug due to the interaction of weakrefs and the cyclic garbage
collector. We must clear any weakrefs in garbage in order to prevent
their callbacks from executing and causing a crash.
2019-09-30 19:06:45 +02:00