Commit Graph

23970 Commits

Author SHA1 Message Date
Vladimir Matveev 1e996c3a3b
bpo-42085: Introduce dedicated entry in PyAsyncMethods for sending values (#22780) 2020-11-10 12:09:55 -08:00
Michal Čihař e59b2deffd
bpo-42014: shutil.rmtree: call onerror with correct function (GH-22585)
The onerror is supposed to be called with failed function, but in this case lstat is wrongly used instead of open.

Not sure if this needs bug or not...

Automerge-Triggered-By: GH:hynek
2020-11-10 08:06:02 -08:00
Andrew Svetlov 42d873c63a
bpo-42183: Fix a stack overflow error for asyncio Task or Future repr() (GH-23020)
The overflow occurs under some circumstances when a task or future
recursively returns itself.

Co-authored-by: Kyle Stanley <aeros167@gmail.com>
2020-11-10 15:58:31 +02:00
Petr Viktorin 0b9c4c6fcf
bpo-42171: Add PEP573-related items to the limited API (GH-23009) 2020-11-10 05:47:31 -08:00
jack1142 bfc6b63102
bpo-36310: Allow pygettext.py to detect calls to gettext in f-strings. (GH-19875)
Adds support to Tools/i18n/pygettext.py for gettext calls in f-strings. This process is done by parsing the f-strings, processing each value, and flagging the ones which contain a gettext call.

Co-authored-by: Batuhan Taskaya <batuhanosmantaskaya@gmail.com>
2020-11-10 01:50:45 +03:00
Tom Gringauz a117167d8d
bpo-41543: contextlib.nullcontext can fill in for an async context manager (GH-21870)
Co-authored-by: Andrew Svetlov <andrew.svetlov@gmail.com>
2020-11-09 14:34:07 +02:00
Ammar Askar 97e8b1eaea
bpo-40624: Add support for the XPath != operator in xml.etree (GH-22147) 2020-11-09 08:02:39 +01:00
kj 4eb41d055e
bpo-42233: Add union type expression support for GenericAlias and fix de-duplicating of GenericAlias (GH-23077) 2020-11-08 20:00:13 -08:00
Ronald Oussoren 23831a7a90
bpo-41754: Ignore NotADirectoryError in invocation of xdg-settings (GH-23075)
It is not clear why this can happen, but several users have mentioned
getting this exception on macOS.
2020-11-08 10:46:55 +01:00
Ronald Oussoren 41761933c1
bpo-41100: Support macOS 11 and Apple Silicon (GH-22855)
Co-authored-by:  Lawrence D’Anna <lawrence_danna@apple.com>

* Add support for macOS 11 and Apple Silicon (aka arm64)
   
  As a side effect of this work use the system copy of libffi on macOS, and remove the vendored copy

* Support building on recent versions of macOS while deploying to older versions

  This allows building installers on macOS 11 while still supporting macOS 10.9.
2020-11-08 10:05:27 +01:00
Erlend Egeberg Aasland 01c6aa43dc
bpo-40077: Convert _queuemodule to use heap types (GH-23136)
@vstinner / @corona10, would you mind reviewing this?
2020-11-07 11:18:37 -08:00
Nick Coghlan 8805a4dad2
bpo-42282: Fold constants inside named expressions (GH-23190)
* The AST optimiser wasn't descending into named expressions, so
  any constant subexpressions weren't being folded at compile time
* Remove "default:" clauses inside the AST optimiser code to reduce the
  risk of similar bugs passing unnoticed in future compiler changes
2020-11-07 22:35:17 +10:00
Julien Palard ee2549c2ba
bpo-41028: Doc: Move switchers to docsbuild-scripts. (GH-20969) 2020-11-07 12:28:31 +01:00
Brett Cannon 825ac38332
bpo-42133: update parts of the stdlib to fall back to `__spec__.loader` when `__loader__` is missing (#22929) 2020-11-06 18:45:56 -08:00
Hai Shi 88c2cfd9ff
bpo-41832: PyType_FromModuleAndSpec() now accepts NULL tp_doc (GH-23123) 2020-11-06 17:04:47 +01:00
Zackery Spytz 91e93794d5
bpo-26389: Allow passing an exception object in the traceback module (GH-22610)
The format_exception(), format_exception_only(), and
print_exception() functions can now take an exception object as a positional-only argument.

Co-Authored-By: Matthias Bussonnier <bussonniermatthias@gmail.com>
2020-11-05 22:18:44 +00:00
Victor Stinner dc42af8fd1
bpo-42260: PyConfig_Read() only parses argv once (GH-23168)
The PyConfig_Read() function now only parses PyConfig.argv arguments
once: PyConfig.parse_argv is set to 2 after arguments are parsed.
Since Python arguments are strippped from PyConfig.argv, parsing
arguments twice would parse the application options as Python
options.

* Rework the PyConfig documentation.
* Fix _testinternalcapi.set_config() error handling.
* SetConfigTests no longer needs parse_argv=0 when restoring the old
  configuration.
2020-11-05 18:58:07 +01:00
vabr-g 4662fa9bfe
bpo-41877 Check for asert, aseert, assrt in mocks (GH-23165)
Currently, a Mock object which is not unsafe will raise an
AttributeError if an attribute with the prefix assert or assret is
accessed on it. This protects against misspellings of real assert
method calls, which lead to tests passing silently even if the tested
code does not satisfy the intended assertion.

Recently a check was done in a large code base (Google) and three
more frequent ways of misspelling assert were found causing harm:
asert, aseert, assrt. These are now added to the existing check.
2020-11-05 09:04:38 -08:00
Victor Stinner 53a03aafd5
bpo-42262: Add Py_NewRef() and Py_XNewRef() (GH-23152)
Added Py_NewRef() and Py_XNewRef() functions to increment the reference
count of an object and return the object.
2020-11-05 15:02:12 +01:00
Pablo Galindo 80449f243b
bpo-42266: Handle monkey-patching descriptors in LOAD_ATTR cache (GH-23157) 2020-11-05 09:23:15 +00:00
Kazantcev Andrey 178695b7ae
bpo-40816 Add AsyncContextDecorator class (GH-20516)
Co-authored-by: Yury Selivanov <yury@edgedb.com>
2020-11-05 10:52:24 +02:00
Victor Stinner af1d64d9f7
bpo-42260: Main init modify sys.flags in-place (GH-23150)
When Py_Initialize() is called twice, the second call now updates
more sys attributes for the configuration, rather than only sys.argv.

* Rename _PySys_InitMain() to _PySys_UpdateConfig().
* _PySys_UpdateConfig() now modifies sys.flags in-place, instead of
  creating a new flags object.
* Remove old commented sys.flags flags (unbuffered and skip_first).
* Add private _PySys_GetObject() function.
* When Py_Initialize(), Py_InitializeFromConfig() and
2020-11-04 17:34:34 +01:00
Victor Stinner 8021875bbc
bpo-1635741: Add PyModule_AddObjectRef() function (GH-23122)
Added PyModule_AddObjectRef() function: similar to
PyModule_AddObjectRef() but don't steal a reference to the value on
success.
2020-11-04 13:59:15 +01:00
Victor Stinner 3529718925
bpo-42236: os.device_encoding() respects UTF-8 Mode (GH-23119)
On Unix, the os.device_encoding() function now returns 'UTF-8' rather
than the device encoding if the Python UTF-8 Mode is enabled.
2020-11-04 11:20:10 +01:00
Mario Corchero 0001a1b69e
bpo-42251: Add gettrace and getprofile to threading (GH-23125)
This allows to retrieve the functions that were set in these two, which might differ from sys.gettrace and sys.getprofile within a thread.
2020-11-04 09:27:43 +00:00
Pablo Galindo a9abbda618 Python 3.10.0a2
-----BEGIN PGP SIGNATURE-----
 
 iQIzBAABCAAdFiEEz9yiRbEEPPKl+Xhl/+h0BBaL2EcFAl+gngEACgkQ/+h0BBaL
 2EfDABAAnQ/bHY40QjoQ+us2vufDj3uvm2S7w0LMM+kgrehhqEoG/FmVtfb8XIvD
 S8FFtSa/VkrVUXvxI/mMpWL78se+Qk9rVQJ8dUZ4SvFJVe1NJdCMNV2Ij263QgJU
 r6bNu42EODDrFAgG9vq5ImND1qnGkF+hKlMHYV3HZhTP4drBEY+TMaB55Vmsp9TH
 eBnfY50y+k8w0hJmsZA1LaVLAXU9WHaWVB90/D22Z9MJmmQF5ofsIFXYj+1IcEkE
 P5CLSCRSzVlpdeNiZWrcBIEK8ap4ZkHvwD8RI6Fu3E/h2TXeiDbp/nCHZDPZMVVk
 OVx3Z4BFTNXE4nmRDIiO5+GqIbgG10LFQKOrOvHQr6inookUaFGwJKwkaEheNr2T
 vcgZMy61OEOeiLqCe6AJ/PuNlQ267zslYucUpNqpQpCu8j691OGEHzv+NNcSRrEq
 T/S4spifDVpFPGLRDkFjJe1OQ33bVZFdJx7guY6WJXVkZILS8vghqQYng55CNzd/
 cdMqb5T6BBRb2SDqJ/qvUhyr7D8YML0pP5pywUUGKTdo84+mW5F/N2Q4g28ik9zv
 pgaehbAwS8jgoPbqH0Tz7qpXiWga9fTbFU47Az07xw5tQFD2+C5fdx+b1h2NBz2q
 exccvIw3Fn8zWikrKbeLgaSAWa2saw25dw0DMes8gG9uHD4+1JA=
 =XAGl
 -----END PGP SIGNATURE-----
gpgsig -----BEGIN PGP SIGNATURE-----
 
 iQIzBAABCAAdFiEEz9yiRbEEPPKl+Xhl/+h0BBaL2EcFAl+hk9UACgkQ/+h0BBaL
 2EdkYxAAs5ujyihMfyoRpU9jV7pNIIbpCHMgV2s74ZBh5jnDPYRfIrGtaE/9HlTp
 PPAScsJvnPwBGNdg8Bs1cLAk/VgJC696ccJXmNugfpxs9CSzhEKQ0oF8PPm+oyqH
 LgU6Dv8dP3+ebOko/vnRPTXbL5gdbvVJvNiV5KUjE6FL+IRl2PHlvUdcEplkPXZ4
 179kgP+8hMCB1Ai1KbMDpO2XIgdkJO+iXgXg9QNDCv2zcyr6OegGYI4cPyB3m/YK
 ZiBTnYK48x/l28Gdi+LISOGa4NNQ7R0j+Qao7YDoqe1yy8Ubj3NrLub+nNnSqnqb
 8PYaFz0sWpJ4I/vjrMaUOJ+hKSwdRZqHF/LqfQBFyBCndmhi6tJNqAzcByqi03HL
 YnXlAYJI9T3rFzBOutfjvbFWkiM2DKAL6dWBIsMeuNEO6MQWTwQHKfidXJyx9zCo
 Vv5QLG5daLFDIgI4qBjJ4POxgl+7eR+5sJBNIVz43KhemiRKLdmFKWmv6moloVB+
 1xoNGmwqz4dzmlAMUj2vJiPuUYl6+P38q1dpAChOXx4hOQZ1ybsmUVUD4S+bLGVQ
 f/VYg6B2I/Fr7Yz9HVYe309abKUSzSeYnW9FkAuTHTmZPjgZHdIJl7euTE0FtNOd
 HQ0paNeC44AoqEnghH35FhFXs+wbedszyc++TltTWn32v0nSVk0=
 =qIu/
 -----END PGP SIGNATURE-----

Merge tag 'v3.10.0a2' into master

Python 3.10.0a2
2020-11-03 17:30:57 +00:00
Serhiy Storchaka 212d32f45c
bpo-42249: Fix writing binary Plist files larger than 4 GiB. (GH-23121) 2020-11-03 16:15:56 +02:00
Erlend Egeberg Aasland 74b4eda98b
bpo-40077: Convert mmap.mmap static type to a heap type (GH-23108) 2020-11-03 10:38:31 +01:00
Pablo Galindo 114ee5dec0
Python 3.10.0a2 2020-11-03 00:02:08 +00:00
Jakub Kulík 9568622c99
bpo-35455: Fix thread_time for Solaris OS (GH-11118) 2020-11-02 23:10:01 +00:00
Victor Stinner 45df61fd2d
bpo-26789: Fix logging.FileHandler._open() at exit (GH-23053)
The logging.FileHandler class now keeps a reference to the builtin
open() function to be able to open or reopen the file during Python
finalization.

Fix errors like:

    Exception ignored in: (...)
    Traceback (most recent call last):
      (...)
      File ".../logging/__init__.py", line 1463, in error
      File ".../logging/__init__.py", line 1577, in _log
      File ".../logging/__init__.py", line 1587, in handle
      File ".../logging/__init__.py", line 1649, in callHandlers
      File ".../logging/__init__.py", line 948, in handle
      File ".../logging/__init__.py", line 1182, in emit
      File ".../logging/__init__.py", line 1171, in _open
    NameError: name 'open' is not defined
2020-11-02 23:17:46 +01:00
Victor Stinner 5cf4782a26
bpo-41796: Make _ast module state per interpreter (GH-23024)
The ast module internal state is now per interpreter.

* Rename "astmodulestate" to "struct ast_state"
* Add pycore_ast.h internal header: the ast_state structure is now
  declared in pycore_ast.h.
* Add PyInterpreterState.ast (struct ast_state)
* Remove get_ast_state()
* Rename get_global_ast_state() to get_ast_state()
* PyAST_obj2mod() now handles get_ast_state() failures
2020-11-02 22:03:28 +01:00
Serhiy Storchaka 34637a0ce2
bpo-42103: Improve validation of Plist files. (GH-22882)
* Prevent some possible DoS attacks via providing invalid Plist files
  with extremely large number of objects or collection sizes.
* Raise InvalidFileException for too large bytes and string size instead of returning garbage.
* Raise InvalidFileException instead of ValueError for specific invalid datetime (NaN).
* Raise InvalidFileException instead of TypeError for non-hashable dict keys.
* Add more tests for invalid Plist files.
2020-11-02 23:01:40 +02:00
Irit Katriel 6fdfcec5b1
bpo-41943: Fix bug where assertLogs doesn't correctly filter messages… (GH-22565)
… by level

@vsajip , @pitrou

Automerge-Triggered-By: GH:vsajip
2020-11-02 11:25:29 -08:00
Jason R. Coombs aca67da4fe
Revert "bpo-37193: remove thread objects which finished process its request (GH-13893)" (GH-23107)
This reverts commit c415590212.
2020-11-02 16:48:56 +00:00
Julien Danjou 64366fa9b3
bpo-41435: Add sys._current_exceptions() function (GH-21689)
This adds a new function named sys._current_exceptions() which is equivalent ot
sys._current_frames() except that it returns the exceptions currently handled
by other threads. It is equivalent to calling sys.exc_info() for each running
thread.
2020-11-02 16:16:25 +02:00
Joongi Kim 6e8dcdaaa4
bpo-41229: Update docs for explicit aclose()-required cases and add contextlib.aclosing() method (GH-21545)
This is a PR to:

 * Add `contextlib.aclosing` which ia analogous to `contextlib.closing` but for async-generators with an explicit test case for [bpo-41229]()
 * Update the docs to describe when we need explicit `aclose()` invocation.

which are motivated by the following issues, articles, and examples:

 * [bpo-41229]()
 * https://github.com/njsmith/async_generator
 * https://vorpus.org/blog/some-thoughts-on-asynchronous-api-design-in-a-post-asyncawait-world/#cleanup-in-generators-and-async-generators
 * https://www.python.org/dev/peps/pep-0533/
 * https://github.com/achimnol/aiotools/blob/ef7bf0cea7af/src/aiotools/context.py#L152

Particuarly regarding [PEP-533](https://www.python.org/dev/peps/pep-0533/), its acceptance (`__aiterclose__()`) would make this little addition of `contextlib.aclosing()` unnecessary for most use cases, but until then this could serve as a good counterpart and analogy to `contextlib.closing()`. The same applies for `contextlib.closing` with `__iterclose__()`.
Also, still there are other use cases, e.g., when working with non-generator objects with `aclose()` methods.
2020-11-02 00:02:48 -08:00
Tal Einat da7bb7b4d7
bpo-40511: Stop unwanted flashing of IDLE calltips (GH-20910)
They were occurring with both repeated 'force-calltip' invocations and by typing parentheses
 in expressions, strings, and comments in the argument code.

Co-authored-by: Terry Jan Reedy <tjreedy@udel.edu>
2020-11-01 22:59:52 -05:00
MARUYAMA Norihiro c415590212
bpo-37193: remove thread objects which finished process its request (GH-13893)
* bpo-37193: remove the thread which finished process request from threads list

* rename variable t to thread.

* don't remove thread from list if it is daemon.

* use lock to protect self._threads.

* use finally block in case of exception from shutdown_request().

* check "not thread.daemon" before lock to avoid holding the lock if it's unnecessary.

* fix the place of _threads_lock.

* separate code to remove a current thread into a function.

* check ValueError when removing thread.

* fix wrong code which all instance shared same lock.

* Extract thread management into a _Threads class to encapsulate atomic operations and separate concerns.

* Replace multiple references of 'block_on_close' with one, avoiding the possibility that 'block_on_close' could change during the course of processing requests. Now, there's exactly one _threads object with behavior fixed for the duration.

* Add docstrings to private classes.

* Add test to ensure that a ThreadingTCPServer can be closed without serving any requests.

* Use _NoThreads as the default value. Fixes AttributeError when server is closed without serving any requests.

* Add blurb

* Add test capturing failure.

Co-authored-by: Jason R. Coombs <jaraco@jaraco.com>
2020-11-01 18:51:04 -05:00
Victor Stinner e662c398d8
bpo-42236: Use UTF-8 encoding if nl_langinfo(CODESET) fails (GH-23086)
If the nl_langinfo(CODESET) function returns an empty string, Python
now uses UTF-8 as the filesystem encoding.

In May 2010 (commit b744ba1d14), I
modified Python to log a warning and use UTF-8 as the filesystem
encoding (instead of None) if nl_langinfo(CODESET) returns an empty
string.

In August 2020 (commit 94908bbc15), I
modified Python startup to fail with a fatal error and a specific
error message if nl_langinfo(CODESET) returns an empty string. The
intent was to prevent guessing the encoding and also investigate user
configuration where this case happens.

In 10 years (2010 to 2020), I saw zero user report about the error
message related to nl_langinfo(CODESET) returning an empty string.

Today, UTF-8 became the defacto standard and it's safe to make the
assumption that the user expects UTF-8. For example,
nl_langinfo(CODESET) can return an empty string on macOS if the
LC_CTYPE locale is not supported, and UTF-8 is the default encoding
on macOS.

While this change is likely to not affect anyone in practice, it
should make UTF-8 lover happy ;-)

Rewrite also the documentation explaining how Python selects the
filesystem encoding and error handler.
2020-11-01 23:07:23 +01:00
kj 1f7dfb277e
bpo-42233: Correctly repr GenericAlias when used with typing module (GH-23081)
Noticed by @serhiy-storchaka in the bpo.  `typing`'s types were not showing the parameterized generic. 
Eg. previously:
```python
>>> typing.Union[dict[str, float], list[int]]
'typing.Union[dict, list]'
```
Now:
```python
>>> typing.Union[dict[str, float], list[int]]
'typing.Union[dict[str, float], list[int]]'
```

Automerge-Triggered-By: GH:gvanrossum
2020-11-01 10:13:38 -08:00
Ronald Oussoren 2165cea548
bpo-29566: binhex.binhex now consitently writes MacOS 9 line endings. (GH-23059)
[bpo-29566]() notes that binhex.binhex uses inconsistent line endings (both Unix and MacOS9 line endings are used). This PR changes this to use the MacOS9 line endings everywhere.
2020-11-01 01:08:48 -08:00
Lysandros Nikolaou 02cdfc93f8
bpo-42218: Correctly handle errors in left-recursive rules (GH-23065)
Left-recursive rules need to check for errors explicitly, since
even if the rule returns NULL, the parsing might continue and lead
to long-distance failures.

Co-authored-by: Pablo Galindo <Pablogsal@gmail.com>
2020-10-31 20:31:41 +02:00
Inada Naoki 43ca084c88
Revert "bpo-42160: tempfile: Reduce overhead of pid check. (GH-22997)"
`_RandomNameSequence` is not true singleton so using `os.register_at_fork` doesn't make sense unlike `random._inst`.

This reverts commit 8e409cebad.
2020-10-31 11:15:38 +09:00
Pablo Galindo 06f8c3328d
bpo-42214: Fix check for NOTEQUAL token in the PEG parser for the barry_as_flufl rule (GH-23048) 2020-10-30 23:48:42 +00:00
Batuhan Taskaya 3af4b58552
bpo-42206: Propagate and raise errors from PyAST_Validate in the parser (GH-23035) 2020-10-30 11:48:41 +00:00
Eric W 8e409cebad
bpo-42160: tempfile: Reduce overhead of pid check. (GH-22997)
The _RandomSequence class in tempfile used to check the current pid every time its rng property was used.
This commit replaces this code with `os.register_at_fork` to reduce the overhead.
2020-10-30 13:56:28 +09:00
Teugea Ioan-Teodor 3317466061
bpo-42061: Document __format__ for IP addresses (GH-23018)
Automerge-Triggered-By: GH:ericvsmith
2020-10-29 15:17:59 -07:00
Yonatan Goldschmidt 350526105f
bpo-42143: Ensure PyFunction_NewWithQualName() can't fail after creating the func object (GH-22953)
func_dealloc() does not handle partially-created objects. Best not to give it any.
2020-10-29 11:58:52 +02:00
Zackery Spytz df59273c7a
bpo-34204: Use pickle.DEFAULT_PROTOCOL in shelve (GH-19639)
Use pickle.DEFAULT_PROTOCOL (currently 5) in shelve instead of a
hardcoded 3.
2020-10-29 02:44:35 -07:00