Commit Graph

46271 Commits

Author SHA1 Message Date
Serhiy Storchaka 212d32f45c
bpo-42249: Fix writing binary Plist files larger than 4 GiB. (GH-23121) 2020-11-03 16:15:56 +02:00
Victor Stinner 45df61fd2d
bpo-26789: Fix logging.FileHandler._open() at exit (GH-23053)
The logging.FileHandler class now keeps a reference to the builtin
open() function to be able to open or reopen the file during Python
finalization.

Fix errors like:

    Exception ignored in: (...)
    Traceback (most recent call last):
      (...)
      File ".../logging/__init__.py", line 1463, in error
      File ".../logging/__init__.py", line 1577, in _log
      File ".../logging/__init__.py", line 1587, in handle
      File ".../logging/__init__.py", line 1649, in callHandlers
      File ".../logging/__init__.py", line 948, in handle
      File ".../logging/__init__.py", line 1182, in emit
      File ".../logging/__init__.py", line 1171, in _open
    NameError: name 'open' is not defined
2020-11-02 23:17:46 +01:00
Serhiy Storchaka 34637a0ce2
bpo-42103: Improve validation of Plist files. (GH-22882)
* Prevent some possible DoS attacks via providing invalid Plist files
  with extremely large number of objects or collection sizes.
* Raise InvalidFileException for too large bytes and string size instead of returning garbage.
* Raise InvalidFileException instead of ValueError for specific invalid datetime (NaN).
* Raise InvalidFileException instead of TypeError for non-hashable dict keys.
* Add more tests for invalid Plist files.
2020-11-02 23:01:40 +02:00
Irit Katriel 6fdfcec5b1
bpo-41943: Fix bug where assertLogs doesn't correctly filter messages… (GH-22565)
… by level

@vsajip , @pitrou

Automerge-Triggered-By: GH:vsajip
2020-11-02 11:25:29 -08:00
Jason R. Coombs aca67da4fe
Revert "bpo-37193: remove thread objects which finished process its request (GH-13893)" (GH-23107)
This reverts commit c415590212.
2020-11-02 16:48:56 +00:00
Lysandros Nikolaou 301822859b
bpo-42224: Fix test_format when locale does not expect number grouping (GH-23067) 2020-11-02 17:27:30 +02:00
Julien Danjou 64366fa9b3
bpo-41435: Add sys._current_exceptions() function (GH-21689)
This adds a new function named sys._current_exceptions() which is equivalent ot
sys._current_frames() except that it returns the exceptions currently handled
by other threads. It is equivalent to calling sys.exc_info() for each running
thread.
2020-11-02 16:16:25 +02:00
Jakub Stasiak 3d86d090dc
bpo-42230: Improve asyncio documentation regarding accepting sets vs iterables (GH-23073)
People call wait() and as_completed() with various non-set iterables,
a list should be the most common but there are others as well[1].

Considering typeshed also documents wait()[2] and as_completed()[3]
as accepting arbitrary iterables I think it's a good idea to document
the status quo better.

[1] https://github.com/aio-libs/aiokafka/pull/672
[2] 620989bac5/stdlib/3/asyncio/tasks.pyi (L161)
[3] 620989bac5/stdlib/3/asyncio/tasks.pyi (L40)
2020-11-02 12:56:35 +02:00
Joongi Kim 6e8dcdaaa4
bpo-41229: Update docs for explicit aclose()-required cases and add contextlib.aclosing() method (GH-21545)
This is a PR to:

 * Add `contextlib.aclosing` which ia analogous to `contextlib.closing` but for async-generators with an explicit test case for [bpo-41229]()
 * Update the docs to describe when we need explicit `aclose()` invocation.

which are motivated by the following issues, articles, and examples:

 * [bpo-41229]()
 * https://github.com/njsmith/async_generator
 * https://vorpus.org/blog/some-thoughts-on-asynchronous-api-design-in-a-post-asyncawait-world/#cleanup-in-generators-and-async-generators
 * https://www.python.org/dev/peps/pep-0533/
 * https://github.com/achimnol/aiotools/blob/ef7bf0cea7af/src/aiotools/context.py#L152

Particuarly regarding [PEP-533](https://www.python.org/dev/peps/pep-0533/), its acceptance (`__aiterclose__()`) would make this little addition of `contextlib.aclosing()` unnecessary for most use cases, but until then this could serve as a good counterpart and analogy to `contextlib.closing()`. The same applies for `contextlib.closing` with `__iterclose__()`.
Also, still there are other use cases, e.g., when working with non-generator objects with `aclose()` methods.
2020-11-02 00:02:48 -08:00
Tal Einat da7bb7b4d7
bpo-40511: Stop unwanted flashing of IDLE calltips (GH-20910)
They were occurring with both repeated 'force-calltip' invocations and by typing parentheses
 in expressions, strings, and comments in the argument code.

Co-authored-by: Terry Jan Reedy <tjreedy@udel.edu>
2020-11-01 22:59:52 -05:00
MARUYAMA Norihiro c415590212
bpo-37193: remove thread objects which finished process its request (GH-13893)
* bpo-37193: remove the thread which finished process request from threads list

* rename variable t to thread.

* don't remove thread from list if it is daemon.

* use lock to protect self._threads.

* use finally block in case of exception from shutdown_request().

* check "not thread.daemon" before lock to avoid holding the lock if it's unnecessary.

* fix the place of _threads_lock.

* separate code to remove a current thread into a function.

* check ValueError when removing thread.

* fix wrong code which all instance shared same lock.

* Extract thread management into a _Threads class to encapsulate atomic operations and separate concerns.

* Replace multiple references of 'block_on_close' with one, avoiding the possibility that 'block_on_close' could change during the course of processing requests. Now, there's exactly one _threads object with behavior fixed for the duration.

* Add docstrings to private classes.

* Add test to ensure that a ThreadingTCPServer can be closed without serving any requests.

* Use _NoThreads as the default value. Fixes AttributeError when server is closed without serving any requests.

* Add blurb

* Add test capturing failure.

Co-authored-by: Jason R. Coombs <jaraco@jaraco.com>
2020-11-01 18:51:04 -05:00
kj 1f7dfb277e
bpo-42233: Correctly repr GenericAlias when used with typing module (GH-23081)
Noticed by @serhiy-storchaka in the bpo.  `typing`'s types were not showing the parameterized generic. 
Eg. previously:
```python
>>> typing.Union[dict[str, float], list[int]]
'typing.Union[dict, list]'
```
Now:
```python
>>> typing.Union[dict[str, float], list[int]]
'typing.Union[dict[str, float], list[int]]'
```

Automerge-Triggered-By: GH:gvanrossum
2020-11-01 10:13:38 -08:00
Inada Naoki d2810054c7
tempfile: Use random.choises() instead of choise() (GH-23068) 2020-11-01 20:02:03 +09:00
Ronald Oussoren 2165cea548
bpo-29566: binhex.binhex now consitently writes MacOS 9 line endings. (GH-23059)
[bpo-29566]() notes that binhex.binhex uses inconsistent line endings (both Unix and MacOS9 line endings are used). This PR changes this to use the MacOS9 line endings everywhere.
2020-11-01 01:08:48 -08:00
Lysandros Nikolaou 02cdfc93f8
bpo-42218: Correctly handle errors in left-recursive rules (GH-23065)
Left-recursive rules need to check for errors explicitly, since
even if the rule returns NULL, the parsing might continue and lead
to long-distance failures.

Co-authored-by: Pablo Galindo <Pablogsal@gmail.com>
2020-10-31 20:31:41 +02:00
Inada Naoki 43ca084c88
Revert "bpo-42160: tempfile: Reduce overhead of pid check. (GH-22997)"
`_RandomNameSequence` is not true singleton so using `os.register_at_fork` doesn't make sense unlike `random._inst`.

This reverts commit 8e409cebad.
2020-10-31 11:15:38 +09:00
Victor Stinner b62bdf71ea
bpo-42208: Add _locale._get_locale_encoding() (GH-23052)
* Add a new _locale._get_locale_encoding() function to get the
  current locale encoding.
* Modify locale.getpreferredencoding() to use it.
* Remove the _bootlocale module.
2020-10-31 01:32:11 +01:00
Pablo Galindo 06f8c3328d
bpo-42214: Fix check for NOTEQUAL token in the PEG parser for the barry_as_flufl rule (GH-23048) 2020-10-30 23:48:42 +00:00
Akashkumar D Khunt d6238ba82d
bpo-42172: Correct typo for test_socket.py (GH-23013) 2020-10-30 15:03:51 +09:00
Eric W 8e409cebad
bpo-42160: tempfile: Reduce overhead of pid check. (GH-22997)
The _RandomSequence class in tempfile used to check the current pid every time its rng property was used.
This commit replaces this code with `os.register_at_fork` to reduce the overhead.
2020-10-30 13:56:28 +09:00
Victor Stinner 5776663675
bpo-42029: Remove IRIX code (GH-23023)
IRIX code was slowy removed in Python 2.4 (--with-sgi-dl), Python 3.3
(Irix threads), and Python 3.7.
2020-10-29 15:16:23 +01:00
Zackery Spytz df59273c7a
bpo-34204: Use pickle.DEFAULT_PROTOCOL in shelve (GH-19639)
Use pickle.DEFAULT_PROTOCOL (currently 5) in shelve instead of a
hardcoded 3.
2020-10-29 02:44:35 -07:00
Lysandros Nikolaou 15acc4eaba
bpo-41659: Disallow curly brace directly after primary (GH-22996) 2020-10-27 20:54:20 +02:00
Georges Toth 303aac8c56
bpo-30681: Support invalid date format or value in email Date header (GH-22090)
I am re-submitting an older PR which was abandoned but is still relevant, #10783 by @timb07.

The issue being solved () is still relevant. The original PR #10783 was closed as
the final request changes were not applied and since abandoned.

In this new PR I have re-used the original patch plus applied both comments from the review, by @maxking and @pganssle.


For reference, here is the original PR description:
In email.utils.parsedate_to_datetime(), a failure to parse the date, or invalid date components (such as hour outside 0..23) raises an exception. Document this behaviour, and add tests to test_email/test_utils.py to confirm this behaviour.

In email.headerregistry.DateHeader.parse(), check when parsedate_to_datetime() raises an exception and add a new defect InvalidDateDefect; preserve the invalid value as the string value of the header, but set the datetime attribute to None.

Add tests to test_email/test_headerregistry.py to confirm this behaviour; also added test to test_email/test_inversion.py to confirm emails with such defective date headers round trip successfully.

This pull request incorporates feedback gratefully received from @bitdancer, @brettcannon, @Mariatta and @warsaw, and replaces the earlier PR #2254.

Automerge-Triggered-By: GH:warsaw
2020-10-26 17:31:06 -07:00
Lisa Roach 8374d2ee15
bpo-39101: Fixes BaseException hang in IsolatedAsyncioTestCase. (GH-22654) 2020-10-26 09:28:17 -07:00
Alexey Izbyshev c0590c0033
bpo-42146: Fix memory leak in subprocess.Popen() in case of uid/gid overflow (GH-22966)
Fix memory leak in subprocess.Popen() in case of uid/gid overflow

Also add a test that would catch this leak with `--huntrleaks`.

Alas, the test for `extra_groups` also exposes an inconsistency
in our error reporting: we use a custom ValueError for `extra_groups`,
but propagate OverflowError for `user` and `group`.
2020-10-25 17:09:32 -07:00
Jason R. Coombs d1a0a960ee
bpo-42043: Add support for zipfile.Path subclasses (#22716)
* bpo-42043: Add support for zipfile.Path inheritance as introduced in zipp 3.2.0.

* Add blurb.
2020-10-25 14:45:05 -04:00
Hai Shi 14cdc215aa
bpo-41919: Avoid resource leak in test_io (GH-22973)
Co-authored-by: Pablo Galindo <Pablogsal@gmail.com>
2020-10-25 18:38:33 +00:00
Jason R. Coombs df8d4c83a6
bpo-41490: ``path`` and ``contents`` to aggressively close handles (#22915)
* bpo-41490: ``path`` method to aggressively close handles

* Add blurb

* In ZipReader.contents, eagerly evaluate the contents to release references to the zipfile.

* Instead use _ensure_sequence to ensure any iterable from a reader is eagerly converted to a list if it's not already a sequence.
2020-10-25 14:21:46 -04:00
Mark Roseman 5df6c99cb4
bpo-33987: Add master ttk Frame to IDLE search dialogs (GH-22942) 2020-10-24 23:14:02 -04:00
Mark Roseman c579ad14d3
bpo-33987: Use master ttk Frame for IDLE config dialog (GH-22943) 2020-10-24 19:45:00 -04:00
Mark Roseman e53e54425d
bpo-33987: Use ttk Label on IDLE statusbar (GH-22941) 2020-10-24 19:32:34 -04:00
Serhiy Storchaka 8cd1dbae32
bpo-41052: Fix pickling heap types implemented in C with protocols 0 and 1 (GH-22870) 2020-10-24 21:14:23 +03:00
Jacob Neil Taylor 16ee68da6e
bpo-38976: Add support for HTTP Only flag in MozillaCookieJar (#17471)
Add support for HTTP Only flag in MozillaCookieJar

Co-authored-by: blurb-it[bot] <43283697+blurb-it[bot]@users.noreply.github.com>
2020-10-23 15:48:55 -07:00
Christopher Marchfelder da6f098188
bpo-40592: shutil.which will not return None anymore if ; is the last char in PATHEXT (GH-20088)
shutil.which will not return None anymore for empty str in PATHEXT
Empty PATHEXT will now be defaulted to _WIN_DEFAULT_PATHEXT
2020-10-23 11:08:24 +01:00
Eric Snow 345cd37abe
bpo-36876: Fix the C analyzer tool. (GH-22841)
The original tool wasn't working right and it was simpler to create a new one, partially re-using some of the old code. At this point the tool runs properly on the master. (Try: ./python Tools/c-analyzer/c-analyzer.py analyze.)  It take ~40 seconds on my machine to analyze the full CPython code base.

Note that we'll need to iron out some OS-specific stuff (e.g. preprocessor). We're okay though since this tool isn't used yet in our workflow. We will also need to verify the analysis results in detail before activating the check in CI, though I'm pretty sure it's close.

https://bugs.python.org/issue36876
2020-10-22 18:42:51 -06:00
Cheryl Sabella facb522d44
bpo-33987: IDLE: Use ttk Frame on doc window and statusbar (GH-11433) 2020-10-22 15:14:35 -04:00
Mark Shannon b52432cb8c
bpo-42057: Add regression test to master. (GH-22893) 2020-10-23 00:42:26 +09:00
Batuhan Taskaya c7437e2c02
bpo-41747: Ensure all dataclass methods uses their parents' qualname (GH-22155)
* bpo-41747: Ensure all dataclass methods uses their parents' qualname

Co-authored-by: Serhiy Storchaka <storchaka@gmail.com>
2020-10-21 09:49:22 -04:00
Максим 5f22741340
bpo-23706: Add newline parameter to pathlib.Path.write_text (GH-22420) (GH-22420)
* Add _newline_ parameter to `pathlib.Path.write_text()`
* Update documentation of `pathlib.Path.write_text()`
* Add test case for `pathlib.Path.write_text()` calls with _newline_ parameter passed

Automerge-Triggered-By: GH:methane
2020-10-20 19:08:19 -07:00
Terry Jan Reedy a460d45063
Update idlelib/help.html to current Sphinx output (GH-22833)
idle.rst is unchanged
2020-10-20 20:56:01 -04:00
Gregory P. Smith 786addd9d0
bpo-41586: Attempt to make the pipesize tests more robust. (GH-22839)
Several buildbots are failing on these, likely due to an inability to
set the pipe size to the desired test value.
2020-10-20 17:37:20 -07:00
Serhiy Storchaka 1d3469988e
bpo-38144: Re-add accidentally removed audition for glob. (GH-22805) 2020-10-20 19:45:38 +03:00
Andrey Doroschenko ec42789e6e
bpo-39693: mention KeyError in tarfile extractfile documentation (GH-18639)
Co-authored-by: Andrey Darascheka <andrei.daraschenka@leverx.com>
2020-10-20 10:05:01 -04:00
Miro Hrončok faddc7449d
bpo-38439: Add 256px IDLE icon to the .ico, drop gifs from it (GH-19648) 2020-10-20 13:21:08 +01:00
TIGirardi f2312037e3
bpo-38324: Fix test__locale.py Windows failures (GH-20529)
Use wide-char _W_* fields of lconv structure on Windows
Remove "ps_AF" from test__locale.known_numerics on Windows
2020-10-20 12:39:52 +01:00
Ronald Oussoren 3185267400
bpo-41491: plistlib: accept hexadecimal integer values in xml plist files (GH-22764) 2020-10-20 09:26:33 +02:00
Raymond Hettinger 871934d4cf
bpo-4356: Add key function support to the bisect module (GH-20556) 2020-10-19 22:04:01 -07:00
Justin Turner Arthur de73d432bb
bpo-38912: fix close before connect callback in test_asyncio SSL tests (GH-22691)
Reduces the rate at which the ENV CHANGED failure occurs in test_asyncio SSL tests (due to unclosed transport), but does not 100% resolve it.
2020-10-19 21:18:57 -04:00
Ruben Vorderman 23c0fb8edd
bpo-41586: Add pipesize parameter to subprocess & F_GETPIPE_SZ and F_SETPIPE_SZ to fcntl. (GH-21921)
* Add F_SETPIPE_SZ and F_GETPIPE_SZ to fcntl module
* Add pipesize parameter for subprocess.Popen class

This will allow the user to control the size of the pipes.
On linux the default is 64K. When a pipe is full it blocks for writing.
When a pipe is empty it blocks for reading. On processes that are
very fast this can lead to a lot of wasted CPU cycles. On a typical
Linux system the max pipe size is 1024K which is much better.
For high performance-oriented libraries such as xopen it is nice to
be able to set the pipe size.

The workaround without this feature is to use my_popen_process.stdout.fileno() in
conjuction with fcntl and 1031 (value of F_SETPIPE_SZ) to acquire this behavior.
2020-10-19 16:30:02 -07:00