# Improve asyncio.wait function
The original code creates the futures set two times.
We can create this set before, avoiding the second creation.
This new behaviour [breaks the aiokafka library](https://github.com/aio-libs/aiokafka/pull/672), because it gives an iterator to that function, so the second iteration become empty.
Automerge-Triggered-By: GH:1st1
The onerror is supposed to be called with failed function, but in this case lstat is wrongly used instead of open.
Not sure if this needs bug or not...
Automerge-Triggered-By: GH:hynek
The path configuration is now computed in the "main" initialization.
The core initialization no longer computes it.
* Add _PyConfig_Read() function to read the configuration without
computing the path configuration.
* pyinit_core() no longer computes the path configuration: it is now
computed by init_interp_main().
* The path configuration output members of PyConfig are now optional:
* executable
* base_executable
* prefix
* base_prefix
* exec_prefix
* base_exec_prefix
* _PySys_UpdateConfig() now skips NULL strings in PyConfig.
* _testembed: Rename test_set_config() to test_init_set_config() for
consistency with other tests.
Adds support to Tools/i18n/pygettext.py for gettext calls in f-strings. This process is done by parsing the f-strings, processing each value, and flagging the ones which contain a gettext call.
Co-authored-by: Batuhan Taskaya <batuhanosmantaskaya@gmail.com>
Co-authored-by: Lawrence D’Anna <lawrence_danna@apple.com>
* Add support for macOS 11 and Apple Silicon (aka arm64)
As a side effect of this work use the system copy of libffi on macOS, and remove the vendored copy
* Support building on recent versions of macOS while deploying to older versions
This allows building installers on macOS 11 while still supporting macOS 10.9.
The format_exception(), format_exception_only(), and
print_exception() functions can now take an exception object as a positional-only argument.
Co-Authored-By: Matthias Bussonnier <bussonniermatthias@gmail.com>
The PyConfig_Read() function now only parses PyConfig.argv arguments
once: PyConfig.parse_argv is set to 2 after arguments are parsed.
Since Python arguments are strippped from PyConfig.argv, parsing
arguments twice would parse the application options as Python
options.
* Rework the PyConfig documentation.
* Fix _testinternalcapi.set_config() error handling.
* SetConfigTests no longer needs parse_argv=0 when restoring the old
configuration.
Currently, a Mock object which is not unsafe will raise an
AttributeError if an attribute with the prefix assert or assret is
accessed on it. This protects against misspellings of real assert
method calls, which lead to tests passing silently even if the tested
code does not satisfy the intended assertion.
Recently a check was done in a large code base (Google) and three
more frequent ways of misspelling assert were found causing harm:
asert, aseert, assrt. These are now added to the existing check.
* Inline _PyInterpreterState_SetConfig(): replace it with
_PyConfig_Copy().
* Add _PyErr_SetFromPyStatus()
* Add _PyInterpreterState_GetConfigCopy()
* Add a new _PyInterpreterState_SetConfig() function.
* Add an unit which gets, modifies, and sets the config.
Otherwise, when running the testsuite, test_peg_generator tries to compile C
code using the optimized flags and fails because it cannot find the profile
data.
The logging.FileHandler class now keeps a reference to the builtin
open() function to be able to open or reopen the file during Python
finalization.
Fix errors like:
Exception ignored in: (...)
Traceback (most recent call last):
(...)
File ".../logging/__init__.py", line 1463, in error
File ".../logging/__init__.py", line 1577, in _log
File ".../logging/__init__.py", line 1587, in handle
File ".../logging/__init__.py", line 1649, in callHandlers
File ".../logging/__init__.py", line 948, in handle
File ".../logging/__init__.py", line 1182, in emit
File ".../logging/__init__.py", line 1171, in _open
NameError: name 'open' is not defined
* Prevent some possible DoS attacks via providing invalid Plist files
with extremely large number of objects or collection sizes.
* Raise InvalidFileException for too large bytes and string size instead of returning garbage.
* Raise InvalidFileException instead of ValueError for specific invalid datetime (NaN).
* Raise InvalidFileException instead of TypeError for non-hashable dict keys.
* Add more tests for invalid Plist files.
This adds a new function named sys._current_exceptions() which is equivalent ot
sys._current_frames() except that it returns the exceptions currently handled
by other threads. It is equivalent to calling sys.exc_info() for each running
thread.
They were occurring with both repeated 'force-calltip' invocations and by typing parentheses
in expressions, strings, and comments in the argument code.
Co-authored-by: Terry Jan Reedy <tjreedy@udel.edu>
* bpo-37193: remove the thread which finished process request from threads list
* rename variable t to thread.
* don't remove thread from list if it is daemon.
* use lock to protect self._threads.
* use finally block in case of exception from shutdown_request().
* check "not thread.daemon" before lock to avoid holding the lock if it's unnecessary.
* fix the place of _threads_lock.
* separate code to remove a current thread into a function.
* check ValueError when removing thread.
* fix wrong code which all instance shared same lock.
* Extract thread management into a _Threads class to encapsulate atomic operations and separate concerns.
* Replace multiple references of 'block_on_close' with one, avoiding the possibility that 'block_on_close' could change during the course of processing requests. Now, there's exactly one _threads object with behavior fixed for the duration.
* Add docstrings to private classes.
* Add test to ensure that a ThreadingTCPServer can be closed without serving any requests.
* Use _NoThreads as the default value. Fixes AttributeError when server is closed without serving any requests.
* Add blurb
* Add test capturing failure.
Co-authored-by: Jason R. Coombs <jaraco@jaraco.com>
[bpo-29566]() notes that binhex.binhex uses inconsistent line endings (both Unix and MacOS9 line endings are used). This PR changes this to use the MacOS9 line endings everywhere.
Left-recursive rules need to check for errors explicitly, since
even if the rule returns NULL, the parsing might continue and lead
to long-distance failures.
Co-authored-by: Pablo Galindo <Pablogsal@gmail.com>
* Add a new _locale._get_locale_encoding() function to get the
current locale encoding.
* Modify locale.getpreferredencoding() to use it.
* Remove the _bootlocale module.
The _RandomSequence class in tempfile used to check the current pid every time its rng property was used.
This commit replaces this code with `os.register_at_fork` to reduce the overhead.
I am re-submitting an older PR which was abandoned but is still relevant, #10783 by @timb07.
The issue being solved () is still relevant. The original PR #10783 was closed as
the final request changes were not applied and since abandoned.
In this new PR I have re-used the original patch plus applied both comments from the review, by @maxking and @pganssle.
For reference, here is the original PR description:
In email.utils.parsedate_to_datetime(), a failure to parse the date, or invalid date components (such as hour outside 0..23) raises an exception. Document this behaviour, and add tests to test_email/test_utils.py to confirm this behaviour.
In email.headerregistry.DateHeader.parse(), check when parsedate_to_datetime() raises an exception and add a new defect InvalidDateDefect; preserve the invalid value as the string value of the header, but set the datetime attribute to None.
Add tests to test_email/test_headerregistry.py to confirm this behaviour; also added test to test_email/test_inversion.py to confirm emails with such defective date headers round trip successfully.
This pull request incorporates feedback gratefully received from @bitdancer, @brettcannon, @Mariatta and @warsaw, and replaces the earlier PR #2254.
Automerge-Triggered-By: GH:warsaw
Fix memory leak in subprocess.Popen() in case of uid/gid overflow
Also add a test that would catch this leak with `--huntrleaks`.
Alas, the test for `extra_groups` also exposes an inconsistency
in our error reporting: we use a custom ValueError for `extra_groups`,
but propagate OverflowError for `user` and `group`.
* bpo-41490: ``path`` method to aggressively close handles
* Add blurb
* In ZipReader.contents, eagerly evaluate the contents to release references to the zipfile.
* Instead use _ensure_sequence to ensure any iterable from a reader is eagerly converted to a list if it's not already a sequence.
The original tool wasn't working right and it was simpler to create a new one, partially re-using some of the old code. At this point the tool runs properly on the master. (Try: ./python Tools/c-analyzer/c-analyzer.py analyze.) It take ~40 seconds on my machine to analyze the full CPython code base.
Note that we'll need to iron out some OS-specific stuff (e.g. preprocessor). We're okay though since this tool isn't used yet in our workflow. We will also need to verify the analysis results in detail before activating the check in CI, though I'm pretty sure it's close.
https://bugs.python.org/issue36876
* Add _newline_ parameter to `pathlib.Path.write_text()`
* Update documentation of `pathlib.Path.write_text()`
* Add test case for `pathlib.Path.write_text()` calls with _newline_ parameter passed
Automerge-Triggered-By: GH:methane
* Add F_SETPIPE_SZ and F_GETPIPE_SZ to fcntl module
* Add pipesize parameter for subprocess.Popen class
This will allow the user to control the size of the pipes.
On linux the default is 64K. When a pipe is full it blocks for writing.
When a pipe is empty it blocks for reading. On processes that are
very fast this can lead to a lot of wasted CPU cycles. On a typical
Linux system the max pipe size is 1024K which is much better.
For high performance-oriented libraries such as xopen it is nice to
be able to set the pipe size.
The workaround without this feature is to use my_popen_process.stdout.fileno() in
conjuction with fcntl and 1031 (value of F_SETPIPE_SZ) to acquire this behavior.
This PR replaces #1977. The reason for the replacement is two-fold.
The fix itself is different is that if the CTE header doesn't exist in the original message, it is inserted. This is important because the new CTE could be quoted-printable whereas the original is implicit 8bit.
Also the tests are different. The test_nonascii_as_string_without_cte test in #1977 doesn't actually test the issue in that it passes without the fix. The test_nonascii_as_string_without_content_type_and_cte test is improved here, and even though it doesn't fail without the fix, it is included for completeness.
Automerge-Triggered-By: @warsaw
There are two different `SimpleQueue` types imported (from `multiprocessing.queues` and `queue`) in `Lib/test/test_genericalias.py`, the second one shadowing the first one, making the first one not actually tested. Fix by using different names.
Automerge-Triggered-By: @gvanrossum
Remove complex special methods __int__, __float__, __floordiv__,
__mod__, __divmod__, __rfloordiv__, __rmod__ and __rdivmod__
which always raised a TypeError.
This fixes the test failure with Tk 6.8.10 which is caused by changes to how Tk rounds the `from`, `to` and `tickinterval` arguments. This PR uses `noconv` if the patchlevel is greater than or equal to 8.6.10 (credit to Serhiy for this idea as it is much simpler than what I previously proposed).
Going into more detail for those who want it, the Tk change was made in [commit 591f68c](591f68cb38) and means that the arguments listed above are rounded relative to the value of `from`. However, when rounding the `from` argument ([line 623](591f68cb38/generic/tkScale.c (L623))), it is rounded relative to itself (i.e. rounding `0`) and therefore the assigned value for `from` is always what is given (no matter what values of `from` and `resolution`).
Automerge-Triggered-By: @pablogsal
This special marker annotation is intended to help in distinguishing
proper PEP 484-compliant type aliases from regular top-level variable
assignments.
The hard part was making all the tests pass; there are some subtle issues here, because apparently the future import wasn't tested very thoroughly in previous Python versions.
For example, `inspect.signature()` returned type objects normally (except for forward references), but strings with the future import. We changed it to try and return type objects by calling `typing.get_type_hints()`, but fall back on returning strings if that function fails (which it may do if there are future references in the annotations that require passing in a specific namespace to resolve).
Similarly to GH-22566, those tests called eval() on content received via
HTTP in test_named_sequences_full. This likely isn't exploitable because
unicodedata.lookup(seqname) is called before self.checkletter(seqname,
None) - thus any string which isn't a valid unicode character name
wouldn't ever reach the checkletter method.
Still, it's probably better to be safe than sorry.
Enable recursion checks which were disabled when get __bases__ of
non-type objects in issubclass() and isinstance() and when intern
strings. It fixes a stack overflow when getting __bases__ leads
to infinite recursion.
Originally recursion checks was disabled for PyDict_GetItem() which
silences all errors including the one raised in case of detected
recursion and can return incorrect result. But now the code uses
PyDict_GetItemWithError() and PyDict_SetDefault() instead.
* bpo-26680: Adds support for int.is_integer() for compatibility with float.is_integer().
The int.is_integer() method always returns True.
* bpo-26680: Adds a test to ensure that False.is_integer() and True.is_integer() are always True.
* bpo-26680: Adds Real.is_integer() with a trivial implementation using conversion to int.
This default implementation is intended to reduce the workload for subclass
implementers. It is not robust in the presence of infinities or NaNs and
may have suboptimal performance for other types.
* bpo-26680: Adds Rational.is_integer which returns True if the denominator is one.
This implementation assumes the Rational is represented in it's
lowest form, as required by the class docstring.
* bpo-26680: Adds Integral.is_integer which always returns True.
* bpo-26680: Adds tests for Fraction.is_integer called as an instance method.
The tests for the Rational abstract base class use an unbound
method to sidestep the inability to directly instantiate Rational.
These tests check that everything works correct as an instance method.
* bpo-26680: Updates documentation for Real.is_integer and built-ins int and float.
The call x.is_integer() is now listed in the table of operations
which apply to all numeric types except complex, with a reference
to the full documentation for Real.is_integer(). Mention of
is_integer() has been removed from the section 'Additional Methods
on Float'.
The documentation for Real.is_integer() describes its purpose, and
mentions that it should be overridden for performance reasons, or
to handle special values like NaN.
* bpo-26680: Adds Decimal.is_integer to the Python and C implementations.
The C implementation of Decimal already implements and uses
mpd_isinteger internally, we just expose the existing function to
Python.
The Python implementation uses internal conversion to integer
using to_integral_value().
In both cases, the corresponding context methods are also
implemented.
Tests and documentation are included.
* bpo-26680: Updates the ACKS file.
* bpo-26680: NEWS entries for int, the numeric ABCs and Decimal.
Co-authored-by: Robert Smallshire <rob@sixty-north.com>
This commit reverts commit ac0333e1e1 as the original links are working again and they provide extended features such as comments and alternative versions.
* When the parameters argument is a list, correctly handle the case
of changing it during iteration.
* When the parameters argument is a custom sequence, no longer
override an exception raised in ``__len__()``.
EnumMeta double-checks that `__repr__`, `__str__`, `__format__`, and `__reduce_ex__` are not the same as `object`'s, and replaces them if they are -- even if that replacement was intentionally done in the Enum being constructed. This patch fixes that.
Automerge-Triggered-By: @ethanfurman
Partially revert commit ac46eb4ad6662cf6d771b20d8963658b2186c48c:
"bpo-38113: Update the Python-ast.c generator to PEP384 (gh-15957)".
Using a module state per module instance is causing subtle practical
problems.
For example, the Mercurial project replaces the __import__() function
to implement lazy import, whereas Python expected that "import _ast"
always return a fully initialized _ast module.
Add _PyAST_Fini() to clear the state at exit.
The _ast module has no state (set _astmodule.m_size to 0). Remove
astmodule_traverse(), astmodule_clear() and astmodule_free()
functions.
tarfile writes full path to FNAME field of GZIP format instead of just basename if user specified absolute path. Some archive viewers may process file incorrectly. Also it creates security issue because anyone can know structure of directories on system and know username or other personal information.
RFC1952 says about FNAME:
This is the original name of the file being compressed, with any directory components removed.
So tarfile must remove directory names from FNAME and write only basename of file.
Automerge-Triggered-By: @jaraco
Currently, empty sequences in gather rules make the conditional for
gather rules fail as empty sequences evaluate as "False". We need to
explicitly check for "None" (the failure condition) to avoid false
negatives.
Simply closing the event loop isn't enough to avoid warnings. If we
don't also shut down the event loop's default executor, it sometimes
logs a "dangling thread" warning.
Follow-up to GH-22017
* bpo-41696: Fix handling of debug mode in asyncio.run
This allows PYTHONASYNCIODEBUG or -X dev to enable asyncio debug mode
when using asyncio.run
* 📜🤖 Added by blurb_it.
Co-authored-by: hauntsaninja <>
Co-authored-by: blurb-it[bot] <43283697+blurb-it[bot]@users.noreply.github.com>
When allocating MemoryError classes, there is some logic to use
pre-allocated instances in a freelist only if the type that is being
allocated is not a subclass of MemoryError. Unfortunately in the
destructor this logic is not present so the freelist is altered even
with subclasses of MemoryError.
Stopping and restarting a proactor event loop on windows can lead to
spurious errors logged (ConnectionResetError while reading from the
self pipe). This fixes the issue by ensuring that we don't attempt
to start multiple copies of the self-pipe reading loop.
Currently, if `asyncio.wait_for()` itself is cancelled it will always
raise `CancelledError` regardless if the underlying task is still
running. This is similar to a race with the timeout, which is handled
already.
When I was fixing bpo-32751 back in GH-7216 I missed the case when
*timeout* is zero or negative. This takes care of that.
Props to @aaliddell for noticing the inconsistency.
asyncio.AbstractEventLoop.run_in_executor should be a method that returns an asyncio Future, not an async method.
This matches the concrete implementations, and the documentation better.
Prior to this change, attempting to subclass the C implementation of
zoneinfo.ZoneInfo gave the following error:
TypeError: unbound method ZoneInfo.__init_subclass__() needs an argument
https://bugs.python.org/issue41025
The http.cookiejar module has is_blocked() and blocked_domains()
methods, so "blocklist" term sounds better than "denylist" in this
module.
Replace also denylisted with denied in test___all__.
test_run method test_fatal_error failed when run twice, as with
python -m test -m test_fatal_error test_idle test_idle
because func.called was not reinitialized to 0.
This bug caused a failure on a refleak buildbot.
Mark Shannon
Batuhan Taskaya
Diogo Dutra
Vladimir Matveev
Michal Čihař
Andrew Svetlov
Ikko Ashimine
Victor Stinner
jack1142
Tom Gringauz
Ammar Askar
kj
Ronald Oussoren
Brett Cannon
Hai Shi
Zackery Spytz
vabr-g
Pablo Galindo
Kazantcev Andrey
serge-sans-paille
Mario Corchero
Serhiy Storchaka
Irit Katriel
Jason R. Coombs
Lysandros Nikolaou
Julien Danjou
Jakub Stasiak
Joongi Kim
Tal Einat
MARUYAMA Norihiro
Inada Naoki
Akashkumar D Khunt
Eric W
Georges Toth
Lisa Roach
Alexey Izbyshev
Mark Roseman
Jacob Neil Taylor
Christopher Marchfelder
Eric Snow
Cheryl Sabella
Максим
Terry Jan Reedy
Gregory P. Smith
Andrey Doroschenko
Miro Hrončok
TIGirardi
Raymond Hettinger
Justin Turner Arthur
Ruben Vorderman
Mark Sapiro
Barry Warsaw
Bar Harel
Anthony Sottile
scaramallion
Max Bernstein
Necdet Can Atesman
Brandt Bucher
Anatoliy Platonov
Ned Deily
linchiwei123
Saiyang Gou
E-Paine
Mikhail Golubev
Ben Avrahami
Florian Bruhin
Hansraj Das
Ram Rachum
scoder
Robert Smallshire
Dennis Sweeney
Jan Mazur
Andre Delfino
Eric Larson
Ethan Furman
Bas van Beek
Thomas Grainger
Angelin BOOZ
Berker Peksag
Peter McCormick
idomic
Patrick Reader
Batuhan Taskaya
Maggie Moss
Graham Bleaney
Erlend Egeberg Aasland
Artem Bulgakov
Jakub Kulík
Steve Dower
Dong-hee Na
Ben Darnell
Shantanu
Benjamin Peterson
han-solo
Marek Madejski
Vinay Sharma
Elvis Pranskevichus
Cleber Rosa
James Weaver
Allen
Stefan Krah
Paul Ganssle
Rishav Kundu
Filipe Laíns
Konge
Nathan M
Eric L. Frederich
Hans Petter Jansson
Karthikeyan Singaravelan
pxinwr
Akuli
Guido van Rossum