Commit Graph

46295 Commits

Author SHA1 Message Date
Michal Čihař e59b2deffd
bpo-42014: shutil.rmtree: call onerror with correct function (GH-22585)
The onerror is supposed to be called with failed function, but in this case lstat is wrongly used instead of open.

Not sure if this needs bug or not...

Automerge-Triggered-By: GH:hynek
2020-11-10 08:06:02 -08:00
Andrew Svetlov 42d873c63a
bpo-42183: Fix a stack overflow error for asyncio Task or Future repr() (GH-23020)
The overflow occurs under some circumstances when a task or future
recursively returns itself.

Co-authored-by: Kyle Stanley <aeros167@gmail.com>
2020-11-10 15:58:31 +02:00
Ikko Ashimine 9303b9ecc3
Fix typo in test_array.py (GH-23189) 2020-11-10 18:12:36 +05:30
Victor Stinner 9e1b828265
bpo-42260: Compute the path config in the main init (GH-23211)
The path configuration is now computed in the "main" initialization.
The core initialization no longer computes it.

* Add _PyConfig_Read() function to read the configuration without
  computing the path configuration.
* pyinit_core() no longer computes the path configuration: it is now
  computed by init_interp_main().
* The path configuration output members of PyConfig are now optional:

  * executable
  * base_executable
  * prefix
  * base_prefix
  * exec_prefix
  * base_exec_prefix

* _PySys_UpdateConfig() now skips NULL strings in PyConfig.
* _testembed: Rename test_set_config() to test_init_set_config() for
  consistency with other tests.
2020-11-10 13:21:52 +01:00
jack1142 bfc6b63102
bpo-36310: Allow pygettext.py to detect calls to gettext in f-strings. (GH-19875)
Adds support to Tools/i18n/pygettext.py for gettext calls in f-strings. This process is done by parsing the f-strings, processing each value, and flagging the ones which contain a gettext call.

Co-authored-by: Batuhan Taskaya <batuhanosmantaskaya@gmail.com>
2020-11-10 01:50:45 +03:00
Tom Gringauz a117167d8d
bpo-41543: contextlib.nullcontext can fill in for an async context manager (GH-21870)
Co-authored-by: Andrew Svetlov <andrew.svetlov@gmail.com>
2020-11-09 14:34:07 +02:00
Ammar Askar 97e8b1eaea
bpo-40624: Add support for the XPath != operator in xml.etree (GH-22147) 2020-11-09 08:02:39 +01:00
kj 4eb41d055e
bpo-42233: Add union type expression support for GenericAlias and fix de-duplicating of GenericAlias (GH-23077) 2020-11-08 20:00:13 -08:00
Ronald Oussoren 23831a7a90
bpo-41754: Ignore NotADirectoryError in invocation of xdg-settings (GH-23075)
It is not clear why this can happen, but several users have mentioned
getting this exception on macOS.
2020-11-08 10:46:55 +01:00
Ronald Oussoren 41761933c1
bpo-41100: Support macOS 11 and Apple Silicon (GH-22855)
Co-authored-by:  Lawrence D’Anna <lawrence_danna@apple.com>

* Add support for macOS 11 and Apple Silicon (aka arm64)
   
  As a side effect of this work use the system copy of libffi on macOS, and remove the vendored copy

* Support building on recent versions of macOS while deploying to older versions

  This allows building installers on macOS 11 while still supporting macOS 10.9.
2020-11-08 10:05:27 +01:00
Brett Cannon 825ac38332
bpo-42133: update parts of the stdlib to fall back to `__spec__.loader` when `__loader__` is missing (#22929) 2020-11-06 18:45:56 -08:00
Hai Shi 88c2cfd9ff
bpo-41832: PyType_FromModuleAndSpec() now accepts NULL tp_doc (GH-23123) 2020-11-06 17:04:47 +01:00
Zackery Spytz 91e93794d5
bpo-26389: Allow passing an exception object in the traceback module (GH-22610)
The format_exception(), format_exception_only(), and
print_exception() functions can now take an exception object as a positional-only argument.

Co-Authored-By: Matthias Bussonnier <bussonniermatthias@gmail.com>
2020-11-05 22:18:44 +00:00
Victor Stinner dc42af8fd1
bpo-42260: PyConfig_Read() only parses argv once (GH-23168)
The PyConfig_Read() function now only parses PyConfig.argv arguments
once: PyConfig.parse_argv is set to 2 after arguments are parsed.
Since Python arguments are strippped from PyConfig.argv, parsing
arguments twice would parse the application options as Python
options.

* Rework the PyConfig documentation.
* Fix _testinternalcapi.set_config() error handling.
* SetConfigTests no longer needs parse_argv=0 when restoring the old
  configuration.
2020-11-05 18:58:07 +01:00
Victor Stinner f3cb814315
bpo-42260: Add _PyConfig_FromDict() (GH-23167)
* Rename config_as_dict() to _PyConfig_AsDict().
* Add 'module_search_paths_set' to _PyConfig_AsDict().
* Add _PyConfig_FromDict().
* Add get_config() and set_config() to _testinternalcapi.
* Add config_check_consistency().
2020-11-05 18:12:33 +01:00
vabr-g 4662fa9bfe
bpo-41877 Check for asert, aseert, assrt in mocks (GH-23165)
Currently, a Mock object which is not unsafe will raise an
AttributeError if an attribute with the prefix assert or assret is
accessed on it. This protects against misspellings of real assert
method calls, which lead to tests passing silently even if the tested
code does not satisfy the intended assertion.

Recently a check was done in a large code base (Google) and three
more frequent ways of misspelling assert were found causing harm:
asert, aseert, assrt. These are now added to the existing check.
2020-11-05 09:04:38 -08:00
Pablo Galindo 80449f243b
bpo-42266: Handle monkey-patching descriptors in LOAD_ATTR cache (GH-23157) 2020-11-05 09:23:15 +00:00
Kazantcev Andrey 178695b7ae
bpo-40816 Add AsyncContextDecorator class (GH-20516)
Co-authored-by: Yury Selivanov <yury@edgedb.com>
2020-11-05 10:52:24 +02:00
Victor Stinner 048a35659a
bpo-42260: Add _PyInterpreterState_SetConfig() (GH-23158)
* Inline _PyInterpreterState_SetConfig(): replace it with
  _PyConfig_Copy().
* Add _PyErr_SetFromPyStatus()
* Add _PyInterpreterState_GetConfigCopy()
* Add a new _PyInterpreterState_SetConfig() function.
* Add an unit which gets, modifies, and sets the config.
2020-11-05 00:45:56 +01:00
serge-sans-paille 100964e031
Disable peg generator tests when building with PGO (GH-23141)
Otherwise, when running the testsuite, test_peg_generator tries to compile C
code using the optimized flags and fails because it cannot find the profile
data.
2020-11-04 23:01:08 +00:00
Victor Stinner 3529718925
bpo-42236: os.device_encoding() respects UTF-8 Mode (GH-23119)
On Unix, the os.device_encoding() function now returns 'UTF-8' rather
than the device encoding if the Python UTF-8 Mode is enabled.
2020-11-04 11:20:10 +01:00
Mario Corchero 0001a1b69e
bpo-42251: Add gettrace and getprofile to threading (GH-23125)
This allows to retrieve the functions that were set in these two, which might differ from sys.gettrace and sys.getprofile within a thread.
2020-11-04 09:27:43 +00:00
Pablo Galindo a9abbda618 Python 3.10.0a2
-----BEGIN PGP SIGNATURE-----
 
 iQIzBAABCAAdFiEEz9yiRbEEPPKl+Xhl/+h0BBaL2EcFAl+gngEACgkQ/+h0BBaL
 2EfDABAAnQ/bHY40QjoQ+us2vufDj3uvm2S7w0LMM+kgrehhqEoG/FmVtfb8XIvD
 S8FFtSa/VkrVUXvxI/mMpWL78se+Qk9rVQJ8dUZ4SvFJVe1NJdCMNV2Ij263QgJU
 r6bNu42EODDrFAgG9vq5ImND1qnGkF+hKlMHYV3HZhTP4drBEY+TMaB55Vmsp9TH
 eBnfY50y+k8w0hJmsZA1LaVLAXU9WHaWVB90/D22Z9MJmmQF5ofsIFXYj+1IcEkE
 P5CLSCRSzVlpdeNiZWrcBIEK8ap4ZkHvwD8RI6Fu3E/h2TXeiDbp/nCHZDPZMVVk
 OVx3Z4BFTNXE4nmRDIiO5+GqIbgG10LFQKOrOvHQr6inookUaFGwJKwkaEheNr2T
 vcgZMy61OEOeiLqCe6AJ/PuNlQ267zslYucUpNqpQpCu8j691OGEHzv+NNcSRrEq
 T/S4spifDVpFPGLRDkFjJe1OQ33bVZFdJx7guY6WJXVkZILS8vghqQYng55CNzd/
 cdMqb5T6BBRb2SDqJ/qvUhyr7D8YML0pP5pywUUGKTdo84+mW5F/N2Q4g28ik9zv
 pgaehbAwS8jgoPbqH0Tz7qpXiWga9fTbFU47Az07xw5tQFD2+C5fdx+b1h2NBz2q
 exccvIw3Fn8zWikrKbeLgaSAWa2saw25dw0DMes8gG9uHD4+1JA=
 =XAGl
 -----END PGP SIGNATURE-----
gpgsig -----BEGIN PGP SIGNATURE-----
 
 iQIzBAABCAAdFiEEz9yiRbEEPPKl+Xhl/+h0BBaL2EcFAl+hk9UACgkQ/+h0BBaL
 2EdkYxAAs5ujyihMfyoRpU9jV7pNIIbpCHMgV2s74ZBh5jnDPYRfIrGtaE/9HlTp
 PPAScsJvnPwBGNdg8Bs1cLAk/VgJC696ccJXmNugfpxs9CSzhEKQ0oF8PPm+oyqH
 LgU6Dv8dP3+ebOko/vnRPTXbL5gdbvVJvNiV5KUjE6FL+IRl2PHlvUdcEplkPXZ4
 179kgP+8hMCB1Ai1KbMDpO2XIgdkJO+iXgXg9QNDCv2zcyr6OegGYI4cPyB3m/YK
 ZiBTnYK48x/l28Gdi+LISOGa4NNQ7R0j+Qao7YDoqe1yy8Ubj3NrLub+nNnSqnqb
 8PYaFz0sWpJ4I/vjrMaUOJ+hKSwdRZqHF/LqfQBFyBCndmhi6tJNqAzcByqi03HL
 YnXlAYJI9T3rFzBOutfjvbFWkiM2DKAL6dWBIsMeuNEO6MQWTwQHKfidXJyx9zCo
 Vv5QLG5daLFDIgI4qBjJ4POxgl+7eR+5sJBNIVz43KhemiRKLdmFKWmv6moloVB+
 1xoNGmwqz4dzmlAMUj2vJiPuUYl6+P38q1dpAChOXx4hOQZ1ybsmUVUD4S+bLGVQ
 f/VYg6B2I/Fr7Yz9HVYe309abKUSzSeYnW9FkAuTHTmZPjgZHdIJl7euTE0FtNOd
 HQ0paNeC44AoqEnghH35FhFXs+wbedszyc++TltTWn32v0nSVk0=
 =qIu/
 -----END PGP SIGNATURE-----

Merge tag 'v3.10.0a2' into master

Python 3.10.0a2
2020-11-03 17:30:57 +00:00
Serhiy Storchaka 212d32f45c
bpo-42249: Fix writing binary Plist files larger than 4 GiB. (GH-23121) 2020-11-03 16:15:56 +02:00
Pablo Galindo 114ee5dec0
Python 3.10.0a2 2020-11-03 00:02:08 +00:00
Victor Stinner 45df61fd2d
bpo-26789: Fix logging.FileHandler._open() at exit (GH-23053)
The logging.FileHandler class now keeps a reference to the builtin
open() function to be able to open or reopen the file during Python
finalization.

Fix errors like:

    Exception ignored in: (...)
    Traceback (most recent call last):
      (...)
      File ".../logging/__init__.py", line 1463, in error
      File ".../logging/__init__.py", line 1577, in _log
      File ".../logging/__init__.py", line 1587, in handle
      File ".../logging/__init__.py", line 1649, in callHandlers
      File ".../logging/__init__.py", line 948, in handle
      File ".../logging/__init__.py", line 1182, in emit
      File ".../logging/__init__.py", line 1171, in _open
    NameError: name 'open' is not defined
2020-11-02 23:17:46 +01:00
Serhiy Storchaka 34637a0ce2
bpo-42103: Improve validation of Plist files. (GH-22882)
* Prevent some possible DoS attacks via providing invalid Plist files
  with extremely large number of objects or collection sizes.
* Raise InvalidFileException for too large bytes and string size instead of returning garbage.
* Raise InvalidFileException instead of ValueError for specific invalid datetime (NaN).
* Raise InvalidFileException instead of TypeError for non-hashable dict keys.
* Add more tests for invalid Plist files.
2020-11-02 23:01:40 +02:00
Irit Katriel 6fdfcec5b1
bpo-41943: Fix bug where assertLogs doesn't correctly filter messages… (GH-22565)
… by level

@vsajip , @pitrou

Automerge-Triggered-By: GH:vsajip
2020-11-02 11:25:29 -08:00
Jason R. Coombs aca67da4fe
Revert "bpo-37193: remove thread objects which finished process its request (GH-13893)" (GH-23107)
This reverts commit c415590212.
2020-11-02 16:48:56 +00:00
Lysandros Nikolaou 301822859b
bpo-42224: Fix test_format when locale does not expect number grouping (GH-23067) 2020-11-02 17:27:30 +02:00
Julien Danjou 64366fa9b3
bpo-41435: Add sys._current_exceptions() function (GH-21689)
This adds a new function named sys._current_exceptions() which is equivalent ot
sys._current_frames() except that it returns the exceptions currently handled
by other threads. It is equivalent to calling sys.exc_info() for each running
thread.
2020-11-02 16:16:25 +02:00
Jakub Stasiak 3d86d090dc
bpo-42230: Improve asyncio documentation regarding accepting sets vs iterables (GH-23073)
People call wait() and as_completed() with various non-set iterables,
a list should be the most common but there are others as well[1].

Considering typeshed also documents wait()[2] and as_completed()[3]
as accepting arbitrary iterables I think it's a good idea to document
the status quo better.

[1] https://github.com/aio-libs/aiokafka/pull/672
[2] 620989bac5/stdlib/3/asyncio/tasks.pyi (L161)
[3] 620989bac5/stdlib/3/asyncio/tasks.pyi (L40)
2020-11-02 12:56:35 +02:00
Joongi Kim 6e8dcdaaa4
bpo-41229: Update docs for explicit aclose()-required cases and add contextlib.aclosing() method (GH-21545)
This is a PR to:

 * Add `contextlib.aclosing` which ia analogous to `contextlib.closing` but for async-generators with an explicit test case for [bpo-41229]()
 * Update the docs to describe when we need explicit `aclose()` invocation.

which are motivated by the following issues, articles, and examples:

 * [bpo-41229]()
 * https://github.com/njsmith/async_generator
 * https://vorpus.org/blog/some-thoughts-on-asynchronous-api-design-in-a-post-asyncawait-world/#cleanup-in-generators-and-async-generators
 * https://www.python.org/dev/peps/pep-0533/
 * https://github.com/achimnol/aiotools/blob/ef7bf0cea7af/src/aiotools/context.py#L152

Particuarly regarding [PEP-533](https://www.python.org/dev/peps/pep-0533/), its acceptance (`__aiterclose__()`) would make this little addition of `contextlib.aclosing()` unnecessary for most use cases, but until then this could serve as a good counterpart and analogy to `contextlib.closing()`. The same applies for `contextlib.closing` with `__iterclose__()`.
Also, still there are other use cases, e.g., when working with non-generator objects with `aclose()` methods.
2020-11-02 00:02:48 -08:00
Tal Einat da7bb7b4d7
bpo-40511: Stop unwanted flashing of IDLE calltips (GH-20910)
They were occurring with both repeated 'force-calltip' invocations and by typing parentheses
 in expressions, strings, and comments in the argument code.

Co-authored-by: Terry Jan Reedy <tjreedy@udel.edu>
2020-11-01 22:59:52 -05:00
MARUYAMA Norihiro c415590212
bpo-37193: remove thread objects which finished process its request (GH-13893)
* bpo-37193: remove the thread which finished process request from threads list

* rename variable t to thread.

* don't remove thread from list if it is daemon.

* use lock to protect self._threads.

* use finally block in case of exception from shutdown_request().

* check "not thread.daemon" before lock to avoid holding the lock if it's unnecessary.

* fix the place of _threads_lock.

* separate code to remove a current thread into a function.

* check ValueError when removing thread.

* fix wrong code which all instance shared same lock.

* Extract thread management into a _Threads class to encapsulate atomic operations and separate concerns.

* Replace multiple references of 'block_on_close' with one, avoiding the possibility that 'block_on_close' could change during the course of processing requests. Now, there's exactly one _threads object with behavior fixed for the duration.

* Add docstrings to private classes.

* Add test to ensure that a ThreadingTCPServer can be closed without serving any requests.

* Use _NoThreads as the default value. Fixes AttributeError when server is closed without serving any requests.

* Add blurb

* Add test capturing failure.

Co-authored-by: Jason R. Coombs <jaraco@jaraco.com>
2020-11-01 18:51:04 -05:00
kj 1f7dfb277e
bpo-42233: Correctly repr GenericAlias when used with typing module (GH-23081)
Noticed by @serhiy-storchaka in the bpo.  `typing`'s types were not showing the parameterized generic. 
Eg. previously:
```python
>>> typing.Union[dict[str, float], list[int]]
'typing.Union[dict, list]'
```
Now:
```python
>>> typing.Union[dict[str, float], list[int]]
'typing.Union[dict[str, float], list[int]]'
```

Automerge-Triggered-By: GH:gvanrossum
2020-11-01 10:13:38 -08:00
Inada Naoki d2810054c7
tempfile: Use random.choises() instead of choise() (GH-23068) 2020-11-01 20:02:03 +09:00
Ronald Oussoren 2165cea548
bpo-29566: binhex.binhex now consitently writes MacOS 9 line endings. (GH-23059)
[bpo-29566]() notes that binhex.binhex uses inconsistent line endings (both Unix and MacOS9 line endings are used). This PR changes this to use the MacOS9 line endings everywhere.
2020-11-01 01:08:48 -08:00
Lysandros Nikolaou 02cdfc93f8
bpo-42218: Correctly handle errors in left-recursive rules (GH-23065)
Left-recursive rules need to check for errors explicitly, since
even if the rule returns NULL, the parsing might continue and lead
to long-distance failures.

Co-authored-by: Pablo Galindo <Pablogsal@gmail.com>
2020-10-31 20:31:41 +02:00
Inada Naoki 43ca084c88
Revert "bpo-42160: tempfile: Reduce overhead of pid check. (GH-22997)"
`_RandomNameSequence` is not true singleton so using `os.register_at_fork` doesn't make sense unlike `random._inst`.

This reverts commit 8e409cebad.
2020-10-31 11:15:38 +09:00
Victor Stinner b62bdf71ea
bpo-42208: Add _locale._get_locale_encoding() (GH-23052)
* Add a new _locale._get_locale_encoding() function to get the
  current locale encoding.
* Modify locale.getpreferredencoding() to use it.
* Remove the _bootlocale module.
2020-10-31 01:32:11 +01:00
Pablo Galindo 06f8c3328d
bpo-42214: Fix check for NOTEQUAL token in the PEG parser for the barry_as_flufl rule (GH-23048) 2020-10-30 23:48:42 +00:00
Akashkumar D Khunt d6238ba82d
bpo-42172: Correct typo for test_socket.py (GH-23013) 2020-10-30 15:03:51 +09:00
Eric W 8e409cebad
bpo-42160: tempfile: Reduce overhead of pid check. (GH-22997)
The _RandomSequence class in tempfile used to check the current pid every time its rng property was used.
This commit replaces this code with `os.register_at_fork` to reduce the overhead.
2020-10-30 13:56:28 +09:00
Victor Stinner 5776663675
bpo-42029: Remove IRIX code (GH-23023)
IRIX code was slowy removed in Python 2.4 (--with-sgi-dl), Python 3.3
(Irix threads), and Python 3.7.
2020-10-29 15:16:23 +01:00
Zackery Spytz df59273c7a
bpo-34204: Use pickle.DEFAULT_PROTOCOL in shelve (GH-19639)
Use pickle.DEFAULT_PROTOCOL (currently 5) in shelve instead of a
hardcoded 3.
2020-10-29 02:44:35 -07:00
Lysandros Nikolaou 15acc4eaba
bpo-41659: Disallow curly brace directly after primary (GH-22996) 2020-10-27 20:54:20 +02:00
Georges Toth 303aac8c56
bpo-30681: Support invalid date format or value in email Date header (GH-22090)
I am re-submitting an older PR which was abandoned but is still relevant, #10783 by @timb07.

The issue being solved () is still relevant. The original PR #10783 was closed as
the final request changes were not applied and since abandoned.

In this new PR I have re-used the original patch plus applied both comments from the review, by @maxking and @pganssle.


For reference, here is the original PR description:
In email.utils.parsedate_to_datetime(), a failure to parse the date, or invalid date components (such as hour outside 0..23) raises an exception. Document this behaviour, and add tests to test_email/test_utils.py to confirm this behaviour.

In email.headerregistry.DateHeader.parse(), check when parsedate_to_datetime() raises an exception and add a new defect InvalidDateDefect; preserve the invalid value as the string value of the header, but set the datetime attribute to None.

Add tests to test_email/test_headerregistry.py to confirm this behaviour; also added test to test_email/test_inversion.py to confirm emails with such defective date headers round trip successfully.

This pull request incorporates feedback gratefully received from @bitdancer, @brettcannon, @Mariatta and @warsaw, and replaces the earlier PR #2254.

Automerge-Triggered-By: GH:warsaw
2020-10-26 17:31:06 -07:00
Lisa Roach 8374d2ee15
bpo-39101: Fixes BaseException hang in IsolatedAsyncioTestCase. (GH-22654) 2020-10-26 09:28:17 -07:00
Alexey Izbyshev c0590c0033
bpo-42146: Fix memory leak in subprocess.Popen() in case of uid/gid overflow (GH-22966)
Fix memory leak in subprocess.Popen() in case of uid/gid overflow

Also add a test that would catch this leak with `--huntrleaks`.

Alas, the test for `extra_groups` also exposes an inconsistency
in our error reporting: we use a custom ValueError for `extra_groups`,
but propagate OverflowError for `user` and `group`.
2020-10-25 17:09:32 -07:00