Also, reword the What's New messages: this doesn't change the limited API, it only brings the Py_LIMITED_API macro closer to the ideal of only allowing the limited API.
Automerge-Triggered-By: GH:encukou
Added non parallel-safe :func:`~contextlib.chdir` context manager to change
the current working directory and then restore it on exit. Simple wrapper
around :func:`~os.chdir`.
Signed-off-by: Filipe Laíns <lains@riseup.net>
Co-authored-by: Łukasz Langa <lukasz@langa.pl>
In Python 3.8 and 3.9, stacking `@functools.singledispatchmethod` on top of
`@classmethod` or `@staticmethod` caused an exception to be raised if the
method was registered using type-annotations rather than
`@method.register(int)`. This was not caught by unit tests, however, as the
tests only tested the `@method.register(int)` way of registering additional
implementations. The bug is no longer present in Python 3.10+, but
`test_functools.py` is still lacking regression tests for these cases. This
commit adds these test cases.
There are two errors that this commit fixes:
* The parser was not correctly computing the offset and the string
source for E_LINECONT errors due to the incorrect usage of strtok().
* The parser was not correctly unwinding the call stack when a tokenizer
exception happened in rules involving optionals ('?', [...]) as we
always make them return valid results by using the comma operator. We
need to check first if we don't have an error before continuing.
I considered only falling back when both were 0, but that still seems
wrong, and the highly popular rich[1] library does it this way, so I
thought we should probably inherit that behavior.
[1] https://github.com/willmcgugan/rich
Signed-off-by: Filipe Laíns <lains@riseup.net>
Co-authored-by: Łukasz Langa <lukasz@langa.pl>
The Python 3.11 limited C API no longer includes stdlib.h, stdio.h,
string.h and errno.h.
* Exclude Py_MEMCPY() from Python 3.11 limited C API.
* xxlimited C extension is now built with Python 3.11 limited C API.
Split header files to move the non-limited API to Include/cpython/:
* Include/warnings.h => Include/cpython/warnings.h
* Include/weakrefobject.h => Include/cpython/weakrefobject.h
Exclude PyWeakref_GET_OBJECT() from the limited C API. It never
worked since the PyWeakReference structure is opaque in the limited C
API.
Move _PyWarnings_Init() and _PyErr_WarnUnawaitedCoroutine() to the
internal C API.
* bpo-45516: add protocol description to the Traversable documentation
Signed-off-by: Filipe Laíns <lains@riseup.net>
* Update Doc/library/importlib.rst
Co-authored-by: Jason R. Coombs <jaraco@jaraco.com>
* Update Lib/importlib/abc.py
* Update Doc/library/importlib.rst
Co-authored-by: Jason R. Coombs <jaraco@jaraco.com>
Co-authored-by: Jason R. Coombs <jaraco@jaraco.com>
The default was "off". Switching it to "on" means users get the benefit of frozen stdlib modules without having to do anything. There's a special-case for running-in-source-tree, so contributors don't get surprised when their stdlib changes don't get used.
https://bugs.python.org/issue45020
Remove fallbacks for missing round(), copysign() and hypot() in
Python/pymath.c. Python now requires these functions to build.
These fallbacks were needed on Visual Studio 2012 and older. They are
no longer needed since Visual Stuido 2013. Python is now built with
Visual Studio 2017 or newer since Python 3.6.
Add PyThreadState_EnterTracing() and PyThreadState_LeaveTracing()
functions to the limited C API to suspend and resume tracing and
profiling.
Add an unit test on the PyThreadState C API to _testcapi.
Add also internal _PyThreadState_DisableTracing() and
_PyThreadState_ResetTracing().
Move classobject.h, context.h, genobject.h and longintrepr.h header
files from Include/ to Include/cpython/.
Remove redundant "#ifndef Py_LIMITED_API" in context.h.
Remove explicit #include "longintrepr.h" in C files. It's not needed,
Python.h already includes it.
Remove two functions from the limited C API:
* PyMarshal_WriteLongToFile()
* PyMarshal_WriteObjectToFile()
The PEP 384 excludes functions expecting "FILE*" from the stable ABI.
Remove also the Py_MARSHAL_VERSION macro from the limited C API.
Currently frozen modules do not have __file__ set. In their spec, origin is set to "frozen" and they are marked as not having a location. (Similarly, for frozen packages __path__ is set to an empty list.) However, for frozen stdlib modules we are able to extrapolate __file__ as long as we can determine the stdlib directory at runtime. (We now do so since gh-28586.) Having __file__ set is helpful for a number of reasons. Likewise, having a non-empty __path__ means we can import submodules of a frozen package from the filesystem (e.g. we could partially freeze the encodings module).
This change sets __file__ (and adds to __path__) for frozen stdlib modules. It uses sys._stdlibdir (from gh-28586) and the frozen module alias information (from gh-28655). All that work is done in FrozenImporter (in Lib/importlib/_bootstrap.py).
Also, if a frozen module is imported before importlib is bootstrapped (during interpreter initialization) then we fix up that module and its spec during the importlib bootstrapping step (i.e. imporlib._bootstrap._setup()) to match what gets set by FrozenImporter, including setting the file info (if the stdlib dir is known). To facilitate this, modules imported using PyImport_ImportFrozenModule() have __origname__ set using the frozen module alias info. __origname__ is popped off during importlib bootstrap.
(To be clear, even with this change the new code to set __file__ during fixups in imporlib._bootstrap._setup() doesn't actually get triggered yet. This is because sys._stdlibdir hasn't been set yet in interpreter initialization at the point importlib is bootstrapped. However, we do fix up such modules at that point to otherwise match the result of importing through FrozenImporter, just not the __file__ and __path__ parts. Doing so will require changes in the order in which things happen during interpreter initialization. That can be addressed separately. Once it is, the file-related fixup code from this PR will kick in.)
Here are things this change does not do:
* set __file__ for non-stdlib modules (no way of knowing the parent dir)
* set __file__ if the stdlib dir is not known (nor assume the expense of finding it)
* relatedly, set __file__ if the stdlib is in a zip file
* verify that the filename set to __file__ actually exists (too expensive)
* update __path__ for frozen packages that alias a non-package (since there is no package dir)
Other things this change skips, but we may do later:
* set __file__ on modules imported using PyImport_ImportFrozenModule()
* set co_filename when we unmarshal the frozen code object while importing the module (e.g. in FrozenImporter.exec_module()) -- this would allow tracebacks to show source lines
* implement FrozenImporter.get_filename() and FrozenImporter.get_source()
https://bugs.python.org/issue21736
Creating an Enum exhibited quadratic behavior based on the number of members in three places:
- `EnumDict._member_names`: a list searched with each new member's name
- member creation: a `for` loop checking each existing member to see if new member was a duplicate
- `auto()` values: a list of all previous values in enum was copied before being sent to `_generate_next_value()`
Two of those issues have been resolved:
- `_EnumDict._member_names` is now a dictionary so lookups are fast
- member creation tries a fast value lookup before falling back to the slower `for` loop lookup
The third issue still remains, as `_generate_next_value_()` can be user-overridden and could corrupt the last values list if it were not copied.
They support now splitting escape sequences between input chunks.
Add the third parameter "final" in codecs.raw_unicode_escape_decode().
It is True by default to match the former behavior.
They support now splitting escape sequences between input chunks.
Add the third parameter "final" in codecs.unicode_escape_decode().
It is True by default to match the former behavior.
Building Python now requires a C99 <math.h> header file providing
isinf(), isnan() and isfinite() functions.
Remove the Py_FORCE_DOUBLE() macro. It was used by the
Py_IS_INFINITY() macro.
Changes:
* Remove Py_IS_NAN(), Py_IS_INFINITY() and Py_IS_FINITE()
in PC/pyconfig.h.
* Remove the _Py_force_double() function.
* configure no longer checks if math.h defines isinf(), isnan() and
isfinite().
At import time, the xmlrpc.client module uses different date formats to
test strftime so it can format years with 4 digits consistently.
Depending on the underlying C library and its strftime implementation
some of these calls can result in ValueErrors, blocking the
xmlrpc.client module from being imported.
This commit changes the behavior of this bit of code to react to
ValueError exceptions, treating the format that caused them as an
non-viable option.
Move Include/pystrhex.h to Include/internal/pycore_strhex.h.
The header file only contains private functions.
The following C extensions are now built with Py_BUILD_CORE_MODULE
macro defined to get access to the internal C API:
* _blake2
* _hashopenssl
* _md5
* _sha1
* _sha3
* _ssl
* binascii
* Never change types' cached keys. It could invalidate inline attribute objects.
* Lazily create object dictionaries.
* Update specialization of LOAD/STORE_ATTR.
* Don't update shared keys version for deletion of value.
* Update gdb support to handle instance values.
* Rename SPLIT_KEYS opcodes to INSTANCE_VALUE.
Change the configure logic to function properly on macOS when the compiler
outputs a platform triplet for option --print-multiarch.
Co-authored-by: Ned Deily <nad@python.org>
Remove the following math macros using the errno variable:
* Py_ADJUST_ERANGE1()
* Py_ADJUST_ERANGE2()
* Py_OVERFLOWED()
* Py_SET_ERANGE_IF_OVERFLOW()
* Py_SET_ERRNO_ON_MATH_ERROR()
Create pycore_pymath.h internal header file.
Rename Py_ADJUST_ERANGE1() and Py_ADJUST_ERANGE2() to
_Py_ADJUST_ERANGE1() and _Py_ADJUST_ERANGE2(), and convert these
macros to static inline functions.
Move the following macros to pycore_pymath.h:
* _Py_IntegralTypeSigned()
* _Py_IntegralTypeMax()
* _Py_IntegralTypeMin()
* _Py_InIntegralTypeRange()
When libregrtest spawns a worker process, stderr is now written into
stdout to keep messages order. Use a single pipe for stdout and
stderr, rather than two pipes. Previously, messages were out of order
which made analysis of buildbot logs harder
Fix test_name_error_suggestions_do_not_trigger_for_too_many_locals()
of test_exceptions if a directory name contains "a1" (like
"Python-3.11.0a1"): use a stricter regular expression.
* Support HTTP response status code 308 in urllib.
HTTP response status code 308 is defined in https://tools.ietf.org/html/rfc7538 to be the permanent redirect variant of 307 (temporary redirect).
* Update documentation to include http_error_308()
* Add blurb for bpo-40321 fix
Co-authored-by: Roland Crosby <roland@rolandcrosby.com>
In the list of generated frozen modules at the top of Tools/scripts/freeze_modules.py, you will find that some of the modules have a different name than the module (or .py file) that is actually frozen. Let's call each case an "alias". Aliases do not come into play until we get to the (generated) list of modules in Python/frozen.c. (The tool for freezing modules, Programs/_freeze_module, is only concerned with the source file, not the module it will be used for.)
Knowledge of which frozen modules are aliases (and the identity of the original module) normally isn't important. However, this information is valuable when we go to set __file__ on frozen stdlib modules. This change updates Tools/scripts/freeze_modules.py to map aliases to the original module name (or None if not a stdlib module) in Python/frozen.c. We also add a helper function in Python/import.c to look up a frozen module's alias and add the result of that function to the frozen info returned from find_frozen().
https://bugs.python.org/issue45020
Before this change we end up duplicating effort and throwing away data in FrozenImporter.find_spec(). Now we do the work once in find_spec() and the only thing we do in FrozenImporter.exec_module() is turn the raw frozen data into a code object and then exec it.
We've added _imp.find_frozen(), add an arg to _imp.get_frozen_object(), and updated FrozenImporter. We've also moved some code around to reduce duplication, get a little more consistency in outcomes, and be more efficient.
Note that this change is mostly necessary if we want to set __file__ on frozen stdlib modules. (See https://bugs.python.org/issue21736.)
https://bugs.python.org/issue45324
* bpo-44594: fix (Async)ExitStack handling of __context__
Make enter_context(foo()) / enter_async_context(foo()) equivalent to
`[async] with foo()` regarding __context__ when an exception is raised.
Previously exceptions would be caught and re-raised with the wrong
context when explicitly overriding __context__ with None.
WaitForSingleObject() accepts timeout in milliseconds in the range
[0; 0xFFFFFFFE] (DWORD type). INFINITE value (0xFFFFFFFF) means no
timeout. 0xFFFFFFFE milliseconds is around 49.7 days.
PY_TIMEOUT_MAX is (0xFFFFFFFE * 1000) milliseconds on Windows, around
49.7 days.
Partially revert commit 37b8294d62.
On Unix, if the sem_clockwait() function is available in the C
library (glibc 2.30 and newer), the threading.Lock.acquire() method
now uses the monotonic clock (time.CLOCK_MONOTONIC) for the timeout,
rather than using the system clock (time.CLOCK_REALTIME), to not be
affected by system clock changes.
configure now checks if the sem_clockwait() function is available.
* Work correctly if an additional fresh module imports other
additional fresh module which imports a blocked module.
* Raises ImportError if the specified module cannot be imported
while all additional fresh modules are successfully imported.
* Support blocking packages.
* Always restore the import state of fresh and blocked modules
and their submodules.
* Fix test_decimal and test_xml_etree which depended on an undesired
side effect of import_fresh_module().
PyThread_acquire_lock_timed() now clamps the timeout into the
[_PyTime_MIN; _PyTime_MAX] range (_PyTime_t type) if it is too large,
rather than calling Py_FatalError() which aborts the process.
PyThread_acquire_lock_timed() no longer uses
MICROSECONDS_TO_TIMESPEC() to compute sem_timedwait() argument, but
_PyTime_GetSystemClock() and _PyTime_AsTimespec_truncate().
Fix _thread.TIMEOUT_MAX value on Windows: the maximum timeout is
0x7FFFFFFF milliseconds (around 24.9 days), not 0xFFFFFFFF
milliseconds (around 49.7 days).
Set PY_TIMEOUT_MAX to 0x7FFFFFFF milliseconds, rather than 0xFFFFFFFF
milliseconds.
Fix PY_TIMEOUT_MAX overflow test: replace (us >= PY_TIMEOUT_MAX) with
(us > PY_TIMEOUT_MAX).
* during tarfile parsing, a zlib error indicates invalid data
* tarfile.open now raises a descriptive exception from the zlib error
* this makes it clear to the user that they may be trying to open a
corrupted tar file
Fix the threading._shutdown() function when the threading module was
imported first from a thread different than the main thread: no
longer log an error at Python exit.
Fix a race condition in the Thread.join() method of the threading
module. If the function is interrupted by a signal and the signal
handler raises an exception, make sure that the thread remains in a
consistent state to prevent a deadlock.
Having `operator.call(obj, arg)` mean `type(obj).__call__(obj, arg)` is
consistent with the other dunder operators. The semantics with `*args,
**kwargs` then follow naturally from the single-arg semantics.
On Windows, time.sleep() now uses a waitable timer which has a
resolution of 100 ns (10^-7 sec). Previously, it had a solution of 1
ms (10^-3 sec).
* On Windows, time.sleep() now calls PyErr_CheckSignals() before
resetting the SIGINT event.
* Add _PyTime_As100Nanoseconds() function.
* Complete and update time.sleep() documentation.
Co-authored-by: Livius <egyszeregy@freemail.hu>
Detect refcount bugs in C extensions when the empty Unicode string
singleton is destroyed by mistake.
* Move forward declarations to the top of unicodeobject.c.
* Simplifiy unicode_is_singleton().
TestResult methods addFailure(), addError(), addSkip() and
addSubTest() are now called immediately after raising an exception
in test or finishing a subtest. Previously they were called only
after finishing the test clean up.
unittest.TestCase.debug() raises now a SkipTest if the class or
the test method are decorated with the skipping decorator.
Previously it only raised a SkipTest if the test method was decorated
with other decorator in addition to the skipping decorator, or
if SkipTest was explicitly raised in the test or setup methods.
This can occur when the zip file gets deleted, you call zipimport.zipimporter.invalidate_cache(), and then try to use zipimport.zipimporter.find_spec() (i.e. you left the zip file path on sys.path).
Add the Py_ALWAYS_INLINE macro to ask the compiler to always inline a
static inline function. The compiler can ignore it and decides to not
inline the function.
* "fail*" and "assert*" aliases of TestCase methods.
* Broken from start TestCase method assertDictContainsSubset().
* Ignored TestLoader.loadTestsFromModule() parameter use_load_tests.
* Old alias _TextTestResult of TextTestResult.
This will enable us to drop the frozen module header files from the repository.
It does currently cause many source files to be built twice, which just takes more time. For whoever comes to fix this in the future, the files shared between freeze_module and pythoncore should be put into a static library that is consumed by both.
Doing this provides significant performance gains for runtime startup (~15% with all the imported modules frozen). We don't yet freeze all the imported modules because there are a few hiccups in the build systems we need to sort out first. (See bpo-45186 and bpo-45188.)
Note that in PR GH-28320 we added a command-line flag (-X frozen_modules=[on|off]) that allows users to opt out of (or into) using frozen modules. The default is still "off" but we will change it to "on" as soon as we can do it in a way that does not cause contributors pain.
https://bugs.python.org/issue45020
Currently we freeze several modules into the runtime. For each of these modules it is essential to bootstrapping the runtime that they be frozen. Any other stdlib module that we later freeze into the runtime is not essential. We can just as well import from the .py file. This PR lets users explicitly choose which should be used, with the new "-X frozen_modules=[on|off]" CLI flag. The default is "off" for now.
https://bugs.python.org/issue45020
In the configparser module, these have been deprecated since Python 3.2:
* the SafeConfigParser class,
* the filename property of the ParsingError class,
* the readfp method of the ConfigParser class,
In Unix operating systems, time.sleep() now uses the clock_nanosleep() function,
if available, which allows to sleep for an interval specified with nanosecond precision.
Co-authored-by: Victor Stinner <vstinner@python.org>
* Constructors of subclasses of some buitin classes (e.g. tuple, list,
frozenset) no longer accept arbitrary keyword arguments.
* Subclass of set can now define a __new__() method with additional
keyword parameters without overriding also __init__().
* Calling guess_all_extensions() with strict=False potentially
mutated types_map_inv.
* Mutating the result of guess_all_extensions() mutated types_map_inv.
linkage issues mainly for shared libs and missing system library,
also little nit into the signal extension as strsignal returns
a constant in this platform.
* In default mode output separate characters for skipped and failed subtests.
* In verbose mode output separate lines (including description) for skipped
and failed subtests.
* In verbose mode output test description for errors in test cleanup.
* issue45067 - Fix _curses compilation in CentOS 7. Verify the version of ncurses
for extended color support feature usage.
The function extended_color_content was introduced in 2017. The ncurses-devel
package in CentOS 7 had a older version ncurses resulted in compilation error.
For compiling ncurses with extended color support, we verify the version of the
ncurses library.
Co-authored-by: blurb-it[bot] <43283697+blurb-it[bot]@users.noreply.github.com>
Co-authored-by: Serhiy Storchaka <storchaka@gmail.com>
Release the GIL while performing isatty() system calls on arbitrary
file descriptors. In particular, this affects os.isatty(),
os.device_encoding() and io.TextIOWrapper. By extension,
io.open() in text mode is also affected.
Due to significant security concerns, the reuse_address parameter of
asyncio.loop.create_datagram_endpoint, deprecated in Python 3.9, is
now removed. This is because of the behavior of the socket option
SO_REUSEADDR in UDP.
Co-authored-by: Łukasz Langa <lukasz@langa.pl>
Remove deprecated __getitem__ methods of xml.dom.pulldom.DOMEventStream,
wsgiref.util.FileWrapper and fileinput.FileInput, deprecated since Python 3.9.
Convert the Py_TYPE() and Py_SIZE() macros to static inline
functions. The Py_SET_TYPE() and Py_SET_SIZE() functions must now be
used to set an object type and size.
The deallocator function of the BaseException type now uses the
trashcan mecanism to prevent stack overflow. For example, when a
RecursionError instance is raised, it can be linked to another
RecursionError through the __context__ attribute or the __traceback__
attribute, and then a chain of exceptions is created. When the chain
is destroyed, nested deallocator function calls can crash with a
stack overflow if the chain is too long compared to the available
stack memory.
Fix PyAiter_Check to only check for the `__anext__` presense (not for
`__aiter__`). Rename `PyAiter_Check()` to `PyAIter_Check()`,
`PyObject_GetAiter()` -> `PyObject_GetAIter()`.
Co-authored-by: Pablo Galindo Salgado <Pablogsal@gmail.com>
For list.sort(), replace our ad hoc merge ordering strategy with the principled, elegant,
and provably near-optimal one from Munro and Wild's "powersort".
open(), io.open(), codecs.open() and fileinput.FileInput no longer
accept "U" ("universal newline") in the file mode. This flag was
deprecated since Python 3.3.
The binhex module, deprecated in Python 3.9, is now removed. The
following binascii functions, deprecated in Python 3.9, are now also
removed:
* a2b_hqx(), b2a_hqx();
* rlecode_hqx(), rledecode_hqx().
The binascii.crc_hqx() function remains available.
libregrtest now clears the type cache later to reduce the risk of
false alarm when checking for reference leaks. Previously, the type
cache was cleared too early and libregrtest raised a false alarm
about reference leaks under very specific conditions.
Move also support.gc_collect() outside clear/cleanup functions to
make the garbage collection more explicit.
Co-authored-by: Irit Katriel <1055913+iritkatriel@users.noreply.github.com>
Frozen modules must be added to several files in order to work properly. Before this change this had to be done manually. Here we add a tool to generate the relevant lines in those files instead. This helps us avoid mistakes and omissions.
https://bugs.python.org/issue45019
Christian Heimes
Mark Shannon
Petr Viktorin
Ken Jin
Filipe Laíns
Alex Waygood
Pablo Galindo Salgado
Paul Ganssle
Erlend Egeberg Aasland
Victor Stinner
Inada Naoki
Filipe Laíns
andrei kulakov
Benjamin Szőke
Terry Jan Reedy
Eric Snow
Raymond Hettinger
Dong-hee Na
Graham Inggs
Carl Friedrich Bolz-Tereick
Serhiy Storchaka
Dennis Sweeney
rtobar
Ben Hoyt
David Bohman
180909
Mike Gilbert
Josephine-Marie
Joongi Kim
180909
Julien Palard
Matthias Reichl
Steve Dower
Ammar Askar
Christian Clauss
Łukasz Langa
Jochem Schulenklopper
Illia Volochii
Hai Shi
John Belmonte
TAGAMI Yukihiro
native-api
AngstyDuck
Nikita Sobolev
Jack DeVries
Antony Lee
Daniel Hillier
Jason Fried
Alexander Böhn
Mohamad Mansour
scoder
nullptr
Irit Katriel
Rémi Lapeyre
Adam Schwalm
Brett Cannon
speedrun-program
Barry Warsaw
Benjamin Peterson
Hugo van Kemenade
David CARLIER
Henry-Joseph Audéoud
Senthil Kumaran
Vincent Michel
Florin Spătar
Yurii Karabas
Stanisław Skonieczny
Yury Selivanov
Tzu-ping Chung
Tim Peters
Ruben Vorderman