Before this change we end up duplicating effort and throwing away data in FrozenImporter.find_spec(). Now we do the work once in find_spec() and the only thing we do in FrozenImporter.exec_module() is turn the raw frozen data into a code object and then exec it.
We've added _imp.find_frozen(), add an arg to _imp.get_frozen_object(), and updated FrozenImporter. We've also moved some code around to reduce duplication, get a little more consistency in outcomes, and be more efficient.
Note that this change is mostly necessary if we want to set __file__ on frozen stdlib modules. (See https://bugs.python.org/issue21736.)
https://bugs.python.org/issue45324
* bpo-44594: fix (Async)ExitStack handling of __context__
Make enter_context(foo()) / enter_async_context(foo()) equivalent to
`[async] with foo()` regarding __context__ when an exception is raised.
Previously exceptions would be caught and re-raised with the wrong
context when explicitly overriding __context__ with None.
Add a PID to names of POSIX shared memory objects to allow
running multiprocessing tests (test_multiprocessing_fork,
test_multiprocessing_spawn, etc) in parallel.
I've added a number of test-only modules. Some of those cases are covered by the recently frozen stdlib modules (and some will be once we add encodings back in). However, I figured we'd play it safe by having a set of modules guaranteed to be there during tests.
https://bugs.python.org/issue45020
* Work correctly if an additional fresh module imports other
additional fresh module which imports a blocked module.
* Raises ImportError if the specified module cannot be imported
while all additional fresh modules are successfully imported.
* Support blocking packages.
* Always restore the import state of fresh and blocked modules
and their submodules.
* Fix test_decimal and test_xml_etree which depended on an undesired
side effect of import_fresh_module().
Add the _PyTime_AsTimespec_clamp() function: similar to
_PyTime_AsTimespec(), but clamp to _PyTime_t min/max and don't raise
an exception.
PyThread_acquire_lock_timed() now uses _PyTime_AsTimespec_clamp() to
remove the Py_UNREACHABLE() code path.
* Add _PyTime_AsTime_t() function.
* Add PY_TIME_T_MIN and PY_TIME_T_MAX constants.
* Replace _PyTime_AsTimeval_noraise() with _PyTime_AsTimeval_clamp().
* Add pytime_divide_round_up() function.
* Fix integer overflow in pytime_divide().
* Add pytime_divmod() function.
Removed extra comma in comment that indicates state of a `Barrier` as it was confusing and breaking the flow while reading.
Co-authored-by: Priyank <5903604+cpriyank@users.noreply.github.com>
* during tarfile parsing, a zlib error indicates invalid data
* tarfile.open now raises a descriptive exception from the zlib error
* this makes it clear to the user that they may be trying to open a
corrupted tar file
During runtime startup we figure out the stdlib dir but currently throw that information away. This change preserves it and exposes it via PyConfig.stdlib_dir, _Py_GetStdlibDir(), and sys._stdlib_dir.
https://bugs.python.org/issue45211
Fix the threading._shutdown() function when the threading module was
imported first from a thread different than the main thread: no
longer log an error at Python exit.
This accomplishes 2 things:
* consolidates some common code between getpath.c and getpathp.c
* makes the helpers available to code in other files
FWIW, the signature of the join_relfile() function (in fileutils.c) intentionally mirrors that of Windows' PathCchCombineEx().
Note that this change is mostly moving code around. No behavior is meant to change.
https://bugs.python.org/issue45211
Fix a race condition in the Thread.join() method of the threading
module. If the function is interrupted by a signal and the signal
handler raises an exception, make sure that the thread remains in a
consistent state to prevent a deadlock.
Having `operator.call(obj, arg)` mean `type(obj).__call__(obj, arg)` is
consistent with the other dunder operators. The semantics with `*args,
**kwargs` then follow naturally from the single-arg semantics.
Modify managed_connect() helper to support in-memory databases. Use it
for the regression tests added in GH-27844.
Automerge-Triggered-By: GH:pablogsal
Instead of explicitly enumerate test classes for run_unittest()
use the unittest ability to discover tests. This also makes these
tests discoverable and runnable with unittest.
load_tests() can be used for dynamic generating tests and adding
doctests. setUpModule(), tearDownModule() and addModuleCleanup()
can be used for running code before and after all module tests.
TestResult methods addFailure(), addError(), addSkip() and
addSubTest() are now called immediately after raising an exception
in test or finishing a subtest. Previously they were called only
after finishing the test clean up.
unittest.TestCase.debug() raises now a SkipTest if the class or
the test method are decorated with the skipping decorator.
Previously it only raised a SkipTest if the test method was decorated
with other decorator in addition to the skipping decorator, or
if SkipTest was explicitly raised in the test or setup methods.
This can occur when the zip file gets deleted, you call zipimport.zipimporter.invalidate_cache(), and then try to use zipimport.zipimporter.find_spec() (i.e. you left the zip file path on sys.path).
`os.path.realpath()` already exists in all our supported Python versions.
There's no longer a need to check if it exists or not.
Co-authored-by: Łukasz Langa <lukasz@langa.pl>
* "fail*" and "assert*" aliases of TestCase methods.
* Broken from start TestCase method assertDictContainsSubset().
* Ignored TestLoader.loadTestsFromModule() parameter use_load_tests.
* Old alias _TextTestResult of TextTestResult.
Previously, test classes ISOTPTest, J1939Test, BasicUDPLITETest and
UDPLITETimeoutTest were not included in the list of tests and
were not run by regrtest.
tearDown() is not called if setUp() raises an exception
(including SkipTest). addCleanup() should be used for guaranteed
execution of the cleanup code.
Here's one more small cleanup that should have been in PR gh-28319. We eliminate stdout side-effects from importing the frozen __hello__ module, and update tests accordingly. We also move the module's source file into Lib/ from Toos/freeze/flag.py.
https://bugs.python.org/issue45019
Make test_compileall quiet: test_year_2038_mtime_compilation() and
test_larger_than_32_bit_times() of test_compileall no longer log
"Compiling ..." messages to stdout.
Doing this provides significant performance gains for runtime startup (~15% with all the imported modules frozen). We don't yet freeze all the imported modules because there are a few hiccups in the build systems we need to sort out first. (See bpo-45186 and bpo-45188.)
Note that in PR GH-28320 we added a command-line flag (-X frozen_modules=[on|off]) that allows users to opt out of (or into) using frozen modules. The default is still "off" but we will change it to "on" as soon as we can do it in a way that does not cause contributors pain.
https://bugs.python.org/issue45020
The line that should not have been needed on macOS tk 8.6.8 but was,
should not be a problem on Ubuntu, but is. It is not needed on macOS
tk 8.6.11, installed with 3.10. Disable it but leave it for
now in case some system needs it.
Currently we freeze several modules into the runtime. For each of these modules it is essential to bootstrapping the runtime that they be frozen. Any other stdlib module that we later freeze into the runtime is not essential. We can just as well import from the .py file. This PR lets users explicitly choose which should be used, with the new "-X frozen_modules=[on|off]" CLI flag. The default is "off" for now.
https://bugs.python.org/issue45020
There are a few things I missed in gh-27980. This is a follow-up that will make subsequent PRs cleaner. It includes fixes to tests and tools that reference the frozen modules.
https://bugs.python.org/issue45019
In the configparser module, these have been deprecated since Python 3.2:
* the SafeConfigParser class,
* the filename property of the ParsingError class,
* the readfp method of the ConfigParser class,
* Constructors of subclasses of some buitin classes (e.g. tuple, list,
frozenset) no longer accept arbitrary keyword arguments.
* Subclass of set can now define a __new__() method with additional
keyword parameters without overriding also __init__().
* Calling guess_all_extensions() with strict=False potentially
mutated types_map_inv.
* Mutating the result of guess_all_extensions() mutated types_map_inv.
* In default mode output separate characters for skipped and failed subtests.
* In verbose mode output separate lines (including description) for skipped
and failed subtests.
* In verbose mode output test description for errors in test cleanup.
Due to significant security concerns, the reuse_address parameter of
asyncio.loop.create_datagram_endpoint, deprecated in Python 3.9, is
now removed. This is because of the behavior of the socket option
SO_REUSEADDR in UDP.
Co-authored-by: Łukasz Langa <lukasz@langa.pl>
Remove deprecated __getitem__ methods of xml.dom.pulldom.DOMEventStream,
wsgiref.util.FileWrapper and fileinput.FileInput, deprecated since Python 3.9.
* Remove commented deprecation of ctypes.c_buffer.
* Remove references to ctypes.c_string which doesn't exist.
* Remove StringTestCase: it only had skipped test methods.
open(), io.open(), codecs.open() and fileinput.FileInput no longer
accept "U" ("universal newline") in the file mode. This flag was
deprecated since Python 3.3.
The binhex module, deprecated in Python 3.9, is now removed. The
following binascii functions, deprecated in Python 3.9, are now also
removed:
* a2b_hqx(), b2a_hqx();
* rlecode_hqx(), rledecode_hqx().
The binascii.crc_hqx() function remains available.
libregrtest now clears the type cache later to reduce the risk of
false alarm when checking for reference leaks. Previously, the type
cache was cleared too early and libregrtest raised a false alarm
about reference leaks under very specific conditions.
Move also support.gc_collect() outside clear/cleanup functions to
make the garbage collection more explicit.
Co-authored-by: Irit Katriel <1055913+iritkatriel@users.noreply.github.com>
* Functions registered with addModuleCleanup() were not called unless
the user defines tearDownModule() in their test module.
* Functions registered with addClassCleanup() were not called if
tearDownClass is set to None.
* Buffering in TestResult did not work with functions registered
with addClassCleanup() and addModuleCleanup().
* Errors in functions registered with addClassCleanup() and
addModuleCleanup() were not handled correctly in buffered and
debug modes.
* Errors in setUpModule() and functions registered with
addModuleCleanup() were reported in wrong order.
* And several lesser bugs.
Previously it returned None if the test class or method was
decorated with a skipping decorator.
Co-authored-by: Iman Tabrizian <iman.tabrizian@gmail.com>
PyPy and potentially other implementations have different or no
contraints on the number of blocks that can be statically nested. move
the test that checks for this behaviour into a unit test and mark it as
CPython-only.
Fixes:
Traceback (most recent call last):
File "/home/graingert/projects/asyncio-demo/demo.py", line 36, in <module>
sys.exit(main())
File "/home/graingert/projects/asyncio-demo/demo.py", line 30, in main
test_all_tasks_threading()
File "/home/graingert/projects/asyncio-demo/demo.py", line 24, in test_all_tasks_threading
results.append(f.result())
File "/usr/lib/python3.10/concurrent/futures/_base.py", line 438, in result
return self.__get_result()
File "/usr/lib/python3.10/concurrent/futures/_base.py", line 390, in __get_result
raise self._exception
File "/usr/lib/python3.10/concurrent/futures/thread.py", line 52, in run
result = self.fn(*self.args, **self.kwargs)
File "/usr/lib/python3.10/asyncio/runners.py", line 47, in run
_cancel_all_tasks(loop)
File "/usr/lib/python3.10/asyncio/runners.py", line 56, in _cancel_all_tasks
to_cancel = tasks.all_tasks(loop)
File "/usr/lib/python3.10/asyncio/tasks.py", line 53, in all_tasks
tasks = list(_all_tasks)
File "/usr/lib/python3.10/_weakrefset.py", line 60, in __iter__
with _IterationGuard(self):
File "/usr/lib/python3.10/_weakrefset.py", line 33, in __exit__
w._commit_removals()
File "/usr/lib/python3.10/_weakrefset.py", line 57, in _commit_removals
discard(l.pop())
IndexError: pop from empty list
Also fixes:
Exception ignored in: weakref callback <function WeakKeyDictionary.__init__.<locals>.remove at 0x00007fe82245d2e0>
Traceback (most recent call last):
File "/usr/lib/pypy3/lib-python/3/weakref.py", line 390, in remove
del self.data[k]
KeyError: <weakref at 0x00007fe76e8d8180; dead>
Exception ignored in: weakref callback <function WeakKeyDictionary.__init__.<locals>.remove at 0x00007fe82245d2e0>
Traceback (most recent call last):
File "/usr/lib/pypy3/lib-python/3/weakref.py", line 390, in remove
del self.data[k]
KeyError: <weakref at 0x00007fe76e8d81a0; dead>
Exception ignored in: weakref callback <function WeakKeyDictionary.__init__.<locals>.remove at 0x00007fe82245d2e0>
Traceback (most recent call last):
File "/usr/lib/pypy3/lib-python/3/weakref.py", line 390, in remove
del self.data[k]
KeyError: <weakref at 0x000056548f1e24a0; dead>
See: https://github.com/agronholm/anyio/issues/362#issuecomment-904424310
See also: https://bugs.python.org/issue29519
Co-authored-by: Łukasz Langa <lukasz@langa.pl>
Various date parsing utilities in the email module, such as
email.utils.parsedate(), are supposed to gracefully handle invalid
input, typically by raising an appropriate exception or by returning
None.
The internal email._parseaddr._parsedate_tz() helper used by some of
these date parsing routines tries to be robust against malformed input,
but unfortunately it can still crash ungracefully when a non-empty but
whitespace-only input is passed. This manifests as an unexpected
IndexError.
In practice, this can happen when parsing an email with only a newline
inside a ‘Date:’ header, which unfortunately happens occasionally in the
real world.
Here's a minimal example:
$ python
Python 3.9.6 (default, Jun 30 2021, 10:22:16)
[GCC 11.1.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import email.utils
>>> email.utils.parsedate('foo')
>>> email.utils.parsedate(' ')
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
File "/usr/lib/python3.9/email/_parseaddr.py", line 176, in parsedate
t = parsedate_tz(data)
File "/usr/lib/python3.9/email/_parseaddr.py", line 50, in parsedate_tz
res = _parsedate_tz(data)
File "/usr/lib/python3.9/email/_parseaddr.py", line 72, in _parsedate_tz
if data[0].endswith(',') or data[0].lower() in _daynames:
IndexError: list index out of range
The fix is rather straight-forward: guard against empty lists, after
splitting on whitespace, but before accessing the first element.
* Fix typo in __repr__ code
* Add more tests for global int flag reprs
* use last module if multi-module string
- when an enum's `__module__` contains several module names, only
use the last one
Co-authored-by: Łukasz Langa <lukasz@langa.pl>
Co-authored-by: Ethan Furman <ethan@stoneleaf.us>
* Use binascii.a2b_base64 to validate b64decode input.
This change leads to exception messages changes (mostly).
* Added more information to docstring of b64decode
* Added a reference to binascii.a2b_base64 in the docs
Method stopTestRun() is now always called in pair with method startTestRun()
for TestResult objects implicitly created in TestCase.run().
Previously it was not called for test methods and classes decorated with
a skipping decorator.
Additional improvements:
- messages which were compiled regular expressions aren't unpacked back into
strings for unmatched warnings;
- removed unnecessary "if tokens:" check (there's one before the for loop);
- took `endswith` calculation out of the for loop.
While the comment said 'We don't bother resizing localspluskinds',
this would cause .replace() to crash when it happened.
(Also types.CodeType(), but testing that is tedious, and this tests all
code paths.)
Pablo Galindo
Eric Snow
Hai Shi
Jeremy Kloth
Christian Heimes
John Belmonte
Serhiy Storchaka
Raymond Hettinger
Nikita Sobolev
Victor Stinner
Łukasz Langa
Jack DeVries
Terry Jan Reedy
Antony Lee
Alex Vig
Daniel Hillier
Jason Fried
Alexander Böhn
Mohamad Mansour
andrei kulakov
Erlend Egeberg Aasland
nullptr
Rémi Lapeyre
Adam Schwalm
Brett Cannon
Daniel Hahler
Omer Katz
speedrun-program
Barry Warsaw
Hubert Badocha
Konstantin Popov
Irit Katriel
junyixie
Benjamin Peterson
Hugo van Kemenade
giovanniwijaya
Henry-Joseph Audéoud
Yurii Karabas
Tzu-ping Chung
Brandt Bucher
Dong-hee Na
Batuhan Taskaya
Ruben Vorderman
Inada Naoki
Miguel Brito
E-Paine
Carl Friedrich Bolz-Tereick
Thomas Grainger
chilaxan
Mark Shannon
wouter bolsterlee
Gregory Anders
Ken Jin
Karthikeyan Singaravelan
Ammar Askar
Idan Moral
Mark Dickinson
Jakub Kulík
Maximilian Hils
Guido van Rossum