* regrtest: Add --cleanup option to remove "test_python_*" directories
of previous failed test jobs.
* Add "make cleantest" to run "python3 -m test --cleanup".
(cherry picked from commit 47fbc4e45b)
… as proposed in PEP 572; key is now evaluated before value.
https://bugs.python.org/issue35224
(cherry picked from commit c8a35417db)
Co-authored-by: Jörn Heissler <joernheissler@users.noreply.github.com>
test_gdb no longer fails if it gets an "unexpected" message on
stderr: it now ignores stderr. The purpose of test_gdb is to test
that python-gdb.py commands work as expected, not to test gdb.
(cherry picked from commit e56a123fd0)
Co-authored-by: Victor Stinner <vstinner@redhat.com>
Tab now moves focus across and down for Help Source and Custom Run.
(cherry picked from commit 54cf2e0780)
Co-authored-by: Terry Jan Reedy <tjreedy@udel.edu>
For datetime.datetime.strptime(), the leading zero for some two-digit formats is optional.
This adds a footnote to the strftime/strptime documentation to reflect this fact, and adds some tests to ensure that it is true.
bpo-34903
(cherry picked from commit 6b9c204ee7)
Co-authored-by: Mike Gleen <mike.gleen@gmail.com>
The initialize options are 1) add command line options, which are appended to sys.argv as if passed on a real command line, and 2) skip the shell restart. The customization dialog is accessed by a new entry on the Run menu.
(cherry picked from commit 201bc2d18b)
Co-authored-by: Cheryl Sabella <cheryl.sabella@gmail.com>
Many PyRun_XXX() functions like PyRun_String() were no longer
exported in libpython38.dll by mistake. Export them again to fix the
ABI compatibiliy.
(cherry picked from commit 343ed0ffe0)
Co-authored-by: Victor Stinner <vstinner@redhat.com>
Measure required height by quickly maximizing once per screen.
A search for a better method failed.
(cherry picked from commit 5bff3c86ab)
Co-authored-by: Tal Einat <taleinat+github@gmail.com>
On Windows, os.dup() no longer creates an inheritable fd when handling a
character file.
(cherry picked from commit 28fca0c422)
Co-authored-by: Zackery Spytz <zspytz@gmail.com>
uuid could try fallback methods that had no chance of working on a particular
platform, and this could cause spurious test failures, as well as degraded
performance as fallback options were tried and failed.
This fixes both the uuid module and its test's SkipUnless logic to use a
prefiltered list of techniques that may at least potentially work on that platform.
Patch by Michael Felt (aixtools).
(cherry picked from commit 3a1d50e7e5)
Co-authored-by: Michael Felt <aixtools@users.noreply.github.com>
Join the thread to prevent leaking a running thread and leaking a
reference.
Cleanup also the test:
* asyncioWindowsProactorEventLoopPolicy became the default policy,
there is no need to set it manually.
* Only start the thread once the loop is running.
* Use a shorter sleep in the thread (100 ms rather than 1 sec).
* Use close_loop(loop) rather than loop.close().
* Use longer variable names.
(cherry picked from commit 07559450b2)
Co-authored-by: Victor Stinner <vstinner@redhat.com>
Fix a regression introduced by af8646c805 that was causing code of the form:
if True and False:
do_something()
to be optimized incorrectly, eliminating the block.
(cherry picked from commit 05f8318655)
Co-authored-by: Pablo Galindo <Pablogsal@gmail.com>
The peephole optimizer was not optimizing correctly bytecode after negative deltas were introduced. This is due to the fact that some special values (255) were being searched for in both instruction pointer delta and line number deltas.
(cherry picked from commit 3498c642f4)
Co-authored-by: Pablo Galindo <Pablogsal@gmail.com>
The __exit__() method of test.support.catch_unraisable_exception
context manager now ignores unraisable exception raised when clearing
self.unraisable attribute.
(cherry picked from commit 6d22cc8e90)
Co-authored-by: Victor Stinner <vstinner@redhat.com>
Calling setlocale(LC_CTYPE, "") on a system where GetACP() returns CP_UTF8 results in empty strings in _tzname[].
This causes time.tzname to be an empty string.
I have reported the bug to the UCRT team and will follow up, but it will take some time get a fix into production.
In the meantime one possible workaround is to temporarily change the locale by calling setlocale(LC_CTYPE, "C") before calling _tzset and restore the current locale after if the GetACP() == CP_UTF8 or CP_UTF7
@zooba
https://bugs.python.org/issue36779
(cherry picked from commit b4c7defe58)
Co-authored-by: Paul Monson <paulmon@users.noreply.github.com>
Allow pure Python implementation of pickle to work
even when the C _pickle module is unavailable.
Fix test_pickle when _pickle is missing: declare PyPicklerHookTests
outside "if has_c_implementation:" block.
(cherry picked from commit 63ab4ba07b)
Co-authored-by: Victor Stinner <vstinner@redhat.com>
* bpo-36402: Fix threading._shutdown() race condition (GH-13948)
Fix a race condition at Python shutdown when waiting for threads.
Wait until the Python thread state of all non-daemon threads get
deleted (join all non-daemon threads), rather than just wait until
Python threads complete.
* Add threading._shutdown_locks: set of Thread._tstate_lock locks
of non-daemon threads used by _shutdown() to wait until all Python
thread states get deleted. See Thread._set_tstate_lock().
* Add also threading._shutdown_locks_lock to protect access to
threading._shutdown_locks.
* Add test_finalization_shutdown() test.
(cherry picked from commit 468e5fec8a)
* bpo-36402: Fix threading.Thread._stop() (GH-14047)
Remove the _tstate_lock from _shutdown_locks, don't remove None.
(cherry picked from commit 6f75c87375)
* bpo-18748: Fix _pyio.IOBase destructor (closed case) (GH-13952)
_pyio.IOBase destructor now does nothing if getting the closed
attribute fails to better mimick _io.IOBase finalizer.
(cherry picked from commit 4f6f7c5a61)
* bpo-37223: test_io: silence destructor errors (GH-13954)
Implement also MockNonBlockWriterIO.seek() method.
(cherry picked from commit b589cef9c4)
* bpo-37223, test_io: silence last 'Exception ignored in:' (GH-14029)
Use catch_unraisable_exception() to ignore 'Exception ignored in:'
error when the internal BufferedWriter of the BufferedRWPair is
destroyed. The C implementation doesn't give access to the
internal BufferedWriter, so just ignore the warning instead.
(cherry picked from commit 913fa1c824)
(A single int is still allowed, but undocumented.)
https://bugs.python.org/issue35766
(cherry picked from commit 10b55c1643)
Co-authored-by: Guido van Rossum <guido@python.org>
* [bpo-36520](https://bugs.python.org/issue36520): reset the encoded word offset when starting a new
line during an email header folding operation
* 📜🤖 Added by blurb_it.
* [bpo-36520](https://bugs.python.org/issue36520): add an additional test case, and provide descriptive
comments for the test_folding_with_utf8_encoding_* tests
* [bpo-36520](https://bugs.python.org/issue36520): fix whitespace issue
* [bpo-36520](https://bugs.python.org/issue36520): changes per reviewer request -- remove extraneous
backslashes; add whitespace between terminating quotes and
line-continuation backslashes; use "bpo-" instead of
"issue GH-" in comments
(cherry picked from commit f6713e84af)
Co-authored-by: websurfer5 <49998481+websurfer5@users.noreply.github.com>
https://bugs.python.org/issue36520
If internal tasks weak set is changed by another thread during iteration.
https://bugs.python.org/issue36607
(cherry picked from commit 65aa64fae8)
Co-authored-by: Andrew Svetlov <andrew.svetlov@gmail.com>
Signed-off-by: Christian Heimes <christian@python.org>
https://bugs.python.org/issue37215
(cherry picked from commit 8a8b59c979)
Co-authored-by: Christian Heimes <christian@python.org>
Fix an unintended ValueError from :func:`subprocess.run` when checking for
conflicting `input` and `stdin` or `capture_output` and `stdout` or `stderr` args
when they were explicitly provided but with `None` values within a passed in
`**kwargs` dict rather than as passed directly by name.
(cherry picked from commit 8cc605acdd)
Co-authored-by: Rémi Lapeyre <remi.lapeyre@henki.fr>
Currently, inspect.getfile(str) will report nonsense:
```pytb
>>> inspect.getfile(str)
TypeError: <module 'builtins' (built-in)> is a built-in class
```
This fixes that
https://bugs.python.org/issue37173
(cherry picked from commit d407d2a726)
Co-authored-by: Philipp A <flying-sheep@web.de>
There is a possibility that someone (like me) accidentally will omit parentheses with `FileType` arguments after `FileType`, and parser will contain wrong file until someone will try to use it.
Example:
```python
parser = argparse.ArgumentParser()
parser.add_argument('-x', type=argparse.FileType)
```
https://bugs.python.org/issue37150
(cherry picked from commit 03d5831a2d)
Co-authored-by: zygocephalus <grrrr@protonmail.com>
Replace two Python function calls with a single one to ensure that no
memory allocation is done between the invalid object is created and
when _PyObject_IsFreed() is called.
(cherry picked from commit 3bf0f3ad20)
This avoids the search dialogs being hidden behind the editor window.
(cherry picked from commit 554450fb4e)
Co-authored-by: Tal Einat <taleinat@gmail.com>
Add it to the end of the first line if there is room. Tests were reworked.
(cherry picked from commit 949fe976d5)
Co-authored-by: Terry Jan Reedy <tjreedy@udel.edu>
* bpo-30835: email: Fix AttributeError when parsing invalid Content-Transfer-Encoding
Parsing an email containing a multipart Content-Type, along with a
Content-Transfer-Encoding containing an invalid (non-ASCII-decodable) byte
will fail. email.feedparser.FeedParser._parsegen() gets the header and
attempts to convert it to lowercase before comparing it with the accepted
encodings, but as the header contains an invalid byte, it's returned as a
Header object rather than a str.
Cast the Content-Transfer-Encoding header to a str to avoid this.
Found using the AFL fuzzer.
Reported-by: Daniel Axtens <dja@axtens.net>
Signed-off-by: Andrew Donnellan <andrew@donnellan.id.au>
* Add email and NEWS entry for the bugfix.
* bpo-35805: Add parser for Message-ID header.
This parser is based on the definition of Identification Fields from RFC 5322
Sec 3.6.4.
This should also prevent folding of Message-ID header using RFC 2047 encoded
words and hence fix bpo-35805.
* Prevent folding of non-ascii message-id headers.
* Add fold method to MsgID token to prevent folding.
Add BaseEventLoop.wait_executor_on_close attribute: true by default.
loop.close() now waits for the default executor to finish by default.
Set loop.wait_executor_on_close attribute to False to not wait for
the executor.
Modify test_coroutines, test_cprofile, test_generators, test_raise,
test_ssl and test_yield_from to use
support.catch_unraisable_exception() rather than
support.captured_stderr().
test_thread: remove test_save_exception_state_on_error() which is now
updated. test_unraisable_exception() checks that sys.unraisablehook()
is called to handle _thread.start_new_thread() exception.
test_cprofile now rely on unittest for test discovery: replace
support.run_unittest() with unittest.main().
* bpo-37014: Update docstring and Documentation of fileinput.FileInput()
* Explain the behavior of fileinput.FileInput() when reading stdin.
* Update blurb.
* bpo-37014: Fix typo in the docstring and documentation.
Fixed QueueListener in order to avoid random deadlocks.
Unable to add regression tests atm due to time constraints, will add it in a bit.
Regarding implementation, although it's nested, it does not cause performance issues whatsoever, and does not call task_done() in case of an exception (which is the right thing to do IMHO).
https://bugs.python.org/issue36813
Adds a new option in trace that allows tracing runnable modules. It is
exposed as `--module module_name` as `-m` is already in use for another
argument.
This adds a vector of "search fingers" so that usable_arenas can be kept in sorted order (by number of free pools) via constant-time operations instead of linear search.
This should reduce worst-case time for reclaiming a great many objects from O(A**2) to O(A), where A is the number of arenas. See bpo-37029.
The ssl module now can dump key material to a keylog file and trace TLS
protocol messages with a tracing callback. The default and stdlib
contexts also support SSLKEYLOGFILE env var.
The msg_callback and related enums are private members. The feature
is designed for internal debugging and not for end users.
Signed-off-by: Christian Heimes <christian@python.org>
This is an old feature request that appears from time to time. After a year of experimenting with various introspection capabilities in `typing_inspect` on PyPI, I propose to add these two most commonly used functions: `get_origin()` and `get_args()`. These are essentially thin public wrappers around private APIs: `__origin__` and `__args__`.
As discussed in the issue and on the typing tracker, exposing some public helpers instead of `__origin__` and `__args__` directly will give us more flexibility if we will decide to update the internal representation, while still maintaining backwards compatibility.
The implementation is very simple an is essentially a copy from `typing_inspect` with one exception: `ClassVar` was special-cased in `typing_inspect`, but I think this special-casing doesn't really help and only makes things more complicated.
Bump the removal to 3.9, indicate collections.abc available since 3.3,
replace version-changed directive to deprecated-removed.
https://bugs.python.org/issue36953
It is now allowed to add new fields at the end of the PyTypeObject struct without having to allocate a dedicated compatibility flag in tp_flags.
This will reduce the risk of running out of bits in the 32-bit tp_flags value.
* bpo-26836: Add os.memfd_create()
* Use the glibc wrapper for memfd_create()
Co-Authored-By: Christian Heimes <christian@python.org>
* Fix deletions caused by autoreconf.
* Use MFD_CLOEXEC as the default value for *flags*.
* Add memset_s to configure.ac.
* Revert memset_s changes.
* Apply the requested changes.
* Tweak the docs.
* bpo-22385: Support output separators in hex methods.
Also in binascii.hexlify aka b2a_hex.
The underlying implementation behind all hex generation in CPython uses the
same pystrhex.c implementation. This adds support to bytes, bytearray,
and memoryview objects.
The binascii module functions exist rather than being slated for deprecation
because they return bytes rather than requiring an intermediate step through a
str object.
This change was inspired by MicroPython which supports sep in its binascii
implementation (and does not yet support the .hex methods).
https://bugs.python.org/issue22385
Add explicit `asyncSetUp` and `asyncTearDown` methods.
The rest is the same as for #13228
`AsyncTestCase` create a loop instance for every test for the sake of test isolation.
Sometimes a loop shared between all tests can speed up tests execution time a lot but it requires control of closed resources after every test finish. Basically, it requires nested supervisors support that was discussed with @1st1 many times. Sorry, asyncio supervisors have no chance to land on Python 3.8.
The PR intentionally does not provide API for changing the used event loop or getting the test loop: use `asyncio.set_event_loop_policy()` and `asyncio.get_event_loop()` instead.
The PR adds four overridable methods to base `unittest.TestCase` class:
```
def _callSetUp(self):
self.setUp()
def _callTestMethod(self, method):
method()
def _callTearDown(self):
self.tearDown()
def _callCleanup(self, function, /, *args, **kwargs):
function(*args, **kwargs)
```
It allows using asyncio facilities with minimal influence on the unittest code.
The last but not least: the PR respects contextvars. The context variable installed by `asyncSetUp` is available on test, `tearDown` and a coroutine scheduled by `addCleanup`.
https://bugs.python.org/issue32972
_thread.start_new_thread() now logs uncaught exception raised by the
function using sys.unraisablehook(), rather than sys.excepthook(), so
the hook gets access to the function which raised the exception.
I tried to get rid of the `_ProtocolMeta`, but unfortunately it didn'y work. My idea to return a generic alias from `@runtime_checkable` made runtime protocols unpickleable. I am not sure what is worse (a custom metaclass or having some classes unpickleable), so I decided to stick with the status quo (since there were no complains so far). So essentially this is a copy of the implementation in `typing_extensions` with two modifications:
* Rename `@runtime` to `@runtime_checkable` (plus corresponding updates).
* Allow protocols that extend `collections.abc.Iterable` etc.
It has been documented as deprecated and to be removed in 3.8;
From a comment on another thread – which I can't find ; leave get_coro_wrapper() for now, but always return `None`.
https://bugs.python.org/issue36933
Return a coroutine while patching async functions with a decorator.
Co-authored-by: Andrew Svetlov <andrew.svetlov@gmail.com>
https://bugs.python.org/issue36996
Fix destructor _pyio.BytesIO and _pyio.TextIOWrapper: initialize
their _buffer attribute as soon as possible (in the class body),
because it's used by __del__() which calls close().
Add a new threading.excepthook() function which handles uncaught
Thread.run() exception. It can be overridden to control how uncaught
exceptions are handled.
threading.ExceptHookArgs is not documented on purpose: it should not
be used directly.
* threading.excepthook() and threading.ExceptHookArgs.
* Add _PyErr_Display(): similar to PyErr_Display(), but accept a
'file' parameter.
* Add _thread._excepthook(): C implementation of the exception hook
calling _PyErr_Display().
* Add _thread._ExceptHookArgs: structseq type.
* Add threading._invoke_excepthook_wrapper() which handles the gory
details to ensure that everything remains alive during Python
shutdown.
* Add unit tests.
When using the "=" debug functionality of f-strings, use another Constant node (or a merged constant node) instead of adding expr_text to the FormattedValue node.
This will address the common mistake many asyncio users make:
an "except Exception" clause breaking Tasks cancellation.
In addition to this change, we stop inheriting asyncio.TimeoutError
and asyncio.InvalidStateError from their concurrent.futures.*
counterparts. There's no point for these exceptions to share the
inheritance chain.
In 3.9 we'll focus on implementing supervisors and cancel scopes,
which should allow better handling of all exceptions, including
SystemExit and KeyboardInterrupt
* No type cache for types with specialized mro, invalidation is hard.
* FIX: Don't disable method cache custom types that do not implement mro().
* fixing implem.
* Avoid storing error flags, also decref.
* news entry
* Clear as soon as we're getting an error.
* FIX: Reference leak.
The implementation is straightforward, it just mimics `ClassVar` (since the latter is also a name/access qualifier, not really a type). Also it is essentially copied from `typing_extensions`.
Update PyObject_CallMethodObjArgs and _PyObject_CallMethodIdObjArgs
to use _PyObject_GetMethod to avoid creating a bound method object
in many cases.
On a microbenchmark of PyObject_CallMethodObjArgs calling a method on
an interpreted Python class, this optimization resulted in a 1.7x
speedup.
It is also possible to link against a library or executable with a
statically linked libpython, but not both with the same DLL. In fact
building a statically linked python is currently broken on Cygwin
for other (related) reasons.
The same problem applies to other POSIX-like layers over Windows
(MinGW, MSYS) but Python's build system does not seem to attempt
to support those platforms at the moment.
Previously, it was hard to tell whether a function should be awaited. It was also incorrect (per PEP 484) to put this in the type hint for coroutine functions. Added this info to the output of builtins.help and pydoc.
https://bugs.python.org/issue36045
To embed Python into an application, a new --embed option must be
passed to "python3-config --libs --embed" to get "-lpython3.8" (link
the application to libpython). To support both 3.8 and older, try
"python3-config --libs --embed" first and fallback to "python3-config
--libs" (without --embed) if the previous command fails.
Add a pkg-config "python-3.8-embed" module to embed Python into an
application: "pkg-config python-3.8-embed --libs" includes
"-lpython3.8". To support both 3.8 and older, try "pkg-config
python-X.Y-embed --libs" first and fallback to "pkg-config python-X.Y
--libs" (without --embed) if the previous command fails (replace
"X.Y" with the Python version).
On the other hand, "pkg-config python3.8 --libs" no longer contains
"-lpython3.8". C extensions must not be linked to libpython (except
on Android, case handled by the script); this change is backward
incompatible on purpose.
"make install" now also installs "python-3.8-embed.pc".
PyErr_WriteUnraisable() now creates a traceback object if there is no
current traceback. Moreover, call PyErr_NormalizeException() and
PyException_SetTraceback() to normalize the exception value. Ignore
silently any error.
* Copy test_exceptions.test_unraisable() to
test_sys.UnraisableHookTest().
* Use catch_unraisable_exception() in test_coroutines,
test_exceptions, test_generators.
* Fixes issue 24882
* Add news file entry for change.
* Change test_concurrent_futures.ThreadPoolShutdownTest
Adjust the shutdown test so that, after submitting three jobs
to the executor, the test checks for less than three threads,
instead of looking for exactly three threads.
If idle threads are being recycled properly, then we should have
less than three threads.
* Switched idle count to semaphor, Updated tests
As suggested by reviewer tomMoral, swapped lock-protected counter
with a semaphore to track the number of unused threads.
Adjusted test_threads_terminate to wait for completiton of the
previous future before submitting a new one (and checking the
number of threads used).
Also added a new test to confirm the thread pool can be saturated.
* Updates tests as requested by pitrou.
* Correct minor whitespace error.
* Make test_saturation faster
Wrap the callback call within the `add_done_callback` function within concurrent.futures, in order to behave in an identical manner to callbacks added to a running future are triggered once it has completed.
This disallows things like `# type: ignoreé`, which seems wrong.
Also switch to using Py_ISALNUM for the alnum check, for consistency
with other code (and maybe correctness re: locale issues?).
https://bugs.python.org/issue36878
CVE-2019-9948: Avoid file reading as disallowing the unnecessary URL
scheme in URLopener().open() and URLopener().retrieve()
of urllib.request.
Co-Authored-By: SH <push0ebp@gmail.com>
GH-13238 made extra text after a # type: ignore accepted by the parser.
This finishes the job and actually plumbs the extra text through the
parser and makes it available in the AST.
In order to support typing checks calling hex(), oct() and bin() on user-defined classes, a SupportIndex protocol is required. The ability to check these at runtime would be good to add for completeness sake. This is pretty much just a copy of SupportsInt with the names tweaked.
Add new sys.unraisablehook() function which can be overridden to
control how "unraisable exceptions" are handled. It is called when an
exception has occurred but there is no way for Python to handle it.
For example, when a destructor raises an exception or during garbage
collection (gc.collect()).
Changes:
* Add an internal UnraisableHookArgs type used to pass arguments to
sys.unraisablehook.
* Add _PyErr_WriteUnraisableDefaultHook().
* The default hook now ignores exception on writing the traceback.
* test_sys now uses unittest.main() to automatically discover tests:
remove test_main().
* Add _PyErr_Init().
* Fix PyErr_WriteUnraisable(): hold a strong reference to sys.stderr
while using it
* bpo-36929: Modify io/re tests to allow for missing mod name
For a vanishingly small number of internal types, CPython sets the
tp_name slot to mod_name.type_name, either in the PyTypeObject or the
PyType_Spec. There are a few minor places where this surfaces:
* Custom repr functions for those types (some of which ignore the
tp_name in favor of using a string literal, such as _io.TextIOWrapper)
* Pickling error messages
The test suite only tests the former. This commit modifies the test
suite to allow Python implementations to omit the module prefix.
https://bugs.python.org/issue36929
The mmap module set MAP_SHARED flag when map anonymous memory, however VxWorks
only support MAP_PRIVATE when map anonymous memory, this commit clear MAP_SHARED
and set MAP_PRIVATE.
Victor Stinner
Miss Islington (bot)
Steve Dower
Serhiy Storchaka
Andrew Svetlov
Abhilash Raj
Petr Viktorin
Zackery Spytz
Łukasz Langa
Christian Heimes
David Carlier
Inada Naoki
Ned Deily
Pablo Galindo
Augusto Hack
Matthias Bussonnier
Michele Angrisano
Cheryl Sabella
Mark Dickinson
Anthony Sottile
Eric Snow
Sergey Fedoseev
Bar Harel
Yash Aggarwal
Mario Corchero
Tim Peters
Ammar Askar
Akshay Sharma
Ivan Levkivskyi
Matthias Bussonnier
xdegaye
Alex Grönholm
Giampaolo Rodola
Ying Wang
Antoine Pitrou
Gregory P. Smith
Jeroen Demeyer
Bo Bayles
依云
Guido van Rossum
Dino Viehland
Francisco Facioni
Joannah Nanjekye
Mario Corchero
Xtreak
sbstp
Eric V. Smith
Yury Selivanov
Rémi Lapeyre
Raymond Hettinger
Chih-Hsuan Yen
Julien Palard
Michael J. Sullivan
Terry Jan Reedy
Jason R. Coombs
Berker Peksag
E. M. Bray
Dan Rose
Matěj Cepl
Steve Dower
Damien Nadé
Batuhan Taşkaya
Sean
Sam Martin
Victor Stinner
Jake Tesler
Jelle Zijlstra
Paul Dagnelie
Marcel Plch
Jörg Stucke
Max Bernstein
Lihua Zhao
pxinwr
Erik Janssens
Thomas Moreau