This includes adding what should be a relatively temporary
`Modules/_decimal/windows/mpdecimal.h` shim to choose between `mpdecimal32vc.h`
or `mpdecimal64vc.h` based on which of `CONFIG_64` or `CONFIG_32` is defined.
* bpo-27578: Fix inspect.getsource() on empty file
For modules from empty files, `inspect.getsource()` now
returns an empty string, and `inspect.getsourcelines()` returns
a list of one empty string, fixing the expected invariant.
As indicated by `exec('')`, empty strings are valid Python
source code.
Co-authored-by: Oleg Iarygin <oleg@arhadthedev.net>
There is a race between when `Thread._tstate_lock` is released[^1] in `Thread._wait_for_tstate_lock()`
and when `Thread._stop()` asserts[^2] that it is unlocked. Consider the following execution
involving threads A, B, and C:
1. A starts.
2. B joins A, blocking on its `_tstate_lock`.
3. C joins A, blocking on its `_tstate_lock`.
4. A finishes and releases its `_tstate_lock`.
5. B acquires A's `_tstate_lock` in `_wait_for_tstate_lock()`, releases it, but is swapped
out before calling `_stop()`.
6. C is scheduled, acquires A's `_tstate_lock` in `_wait_for_tstate_lock()` but is swapped
out before releasing it.
7. B is scheduled, calls `_stop()`, which asserts that A's `_tstate_lock` is not held.
However, C holds it, so the assertion fails.
The race can be reproduced[^3] by inserting sleeps at the appropriate points in
the threading code. To do so, run the `repro_join_race.py` from the linked repo.
There are two main parts to this PR:
1. `_tstate_lock` is replaced with an event that is attached to `PyThreadState`.
The event is set by the runtime prior to the thread being cleared (in the same
place that `_tstate_lock` was released). `Thread.join()` blocks waiting for the
event to be set.
2. `_PyInterpreterState_WaitForThreads()` provides the ability to wait for all
non-daemon threads to exit. To do so, an `is_daemon` predicate was added to
`PyThreadState`. This field is set each time a thread is created. `threading._shutdown()`
now calls into `_PyInterpreterState_WaitForThreads()` instead of waiting on
`_tstate_lock`s.
[^1]: 441affc9e7/Lib/threading.py (L1201)
[^2]: 441affc9e7/Lib/threading.py (L1115)
[^3]: 8194653279
---------
Co-authored-by: blurb-it[bot] <43283697+blurb-it[bot]@users.noreply.github.com>
Co-authored-by: Antoine Pitrou <antoine@python.org>
Change automatically generated tkinter.Checkbutton widget names to
avoid collisions with automatically generated tkinter.ttk.Checkbutton
widget names within the same parent widget.
* Restore support of None and other false values.
* Raise TypeError for non-zero integers and non-empty sequences.
The regressions were introduced in gh-74668
(bdba8ef42b).
Any capitalization of "xn--" should be acceptable for the ACE prefix
(see https://tools.ietf.org/html/rfc3490#section-5).
Co-authored-by: Pepijn de Vos <pepijndevos@gmail.com>
Co-authored-by: Erlend E. Aasland <erlend@python.org>
Co-authored-by: Petr Viktorin <encukou@gmail.com>
Use the NtQueryInformationProcess system call to efficiently retrieve the parent process ID in a single step, rather than using the process snapshots API which retrieves large amounts of unnecessary information and is more prone to failure (since it makes heap allocations).
Includes a fallback to the original win32_getppid implementation in case the unstable API appears to return strange results.
On Windows, time.monotonic() now uses the QueryPerformanceCounter()
clock to have a resolution better than 1 us, instead of the
gGetTickCount64() clock which has a resolution of 15.6 ms.
* GH-116554: Relax list.sort()'s notion of "descending" run
Rewrote `count_run()` so that sub-runs of equal elements no longer end a descending run. Both ascending and descending runs can have arbitrarily many sub-runs of arbitrarily many equal elements now. This is tricky, because we only use ``<`` comparisons, so checking for equality doesn't come "for free". Surprisingly, it turned out there's a very cheap (one comparison) way to determine whether an ascending run consisted of all-equal elements. That sealed the deal.
In addition, after a descending run is reversed in-place, we now go on to see whether it can be extended by an ascending run that just happens to be adjacent. This succeeds in finding at least one additional element to append about half the time, and so appears to more than repay its cost (the savings come from getting to skip a binary search, when a short run is artificially forced to length MIINRUN later, for each new element `count_run()` can add to the initial run).
While these have been in the back of my mind for years, a question on StackOverflow pushed it to action:
https://stackoverflow.com/questions/78108792/
They were wondering why it took about 4x longer to sort a list like:
[999_999, 999_999, ..., 2, 2, 1, 1, 0, 0]
than "similar" lists. Of course that runs very much faster after this patch.
Co-authored-by: Alex Waygood <Alex.Waygood@Gmail.com>
Co-authored-by: Pieter Eendebak <pieter.eendebak@gmail.com>
gh-116307: Create a new import helper 'isolated modules' and use that instead of 'Clean Import' to ensure that tests from importlib_resources don't leave modules in sys.modules.
* and fix global flag repr
* Update Misc/NEWS.d/next/Library/2024-03-11-12-11-10.gh-issue-116600.FcNBy_.rst
Co-authored-by: Kirill Podoprigora <kirill.bast9@mail.ru>
These give applications the option of more forcefully terminating client
connections for asyncio servers. Useful when terminating a service and
there is limited time to wait for clients to finish up their work.
In free-threaded builds, running with `PYTHON_GIL=0` will now disable the
GIL. Follow-up issues track work to re-enable the GIL when loading an
incompatible extension, and to disable the GIL by default.
In order to support re-enabling the GIL at runtime, all GIL-related data
structures are initialized as usual, and disabling the GIL simply sets a flag
that causes `take_gil()` and `drop_gil()` to return early.
* set default return value of functional types as _mock_return_value
* added test of wrapping child attributes
* added backward compatibility with explicit return
* added docs on the order of precedence
* added test to check default return_value
This follows in the footsteps of #21586
This speeds up import uuid by about 6ms on Linux.
Before:
```
λ hyperfine -w 4 "./python -c 'import uuid'"
Benchmark 1: ./python -c 'import uuid'
Time (mean ± σ): 20.4 ms ± 0.4 ms [User: 16.7 ms, System: 3.8 ms]
Range (min … max): 19.6 ms … 21.8 ms 136 runs
```
After:
```
λ hyperfine -w 4 "./python -c 'import uuid'"
Benchmark 1: ./python -c 'import uuid'
Time (mean ± σ): 14.5 ms ± 0.3 ms [User: 11.5 ms, System: 3.2 ms]
Range (min … max): 13.9 ms … 16.0 ms 175 runs
```
This adds `VERIFY_X509_STRICT` to make the default
SSL context perform stricter (per RFC 5280) validation, as well
as `VERIFY_X509_PARTIAL_CHAIN` to enforce more standards-compliant
path-building behavior.
As part of this changeset, I had to tweak `make_ssl_certs.py`
slightly to emit 5280-conforming CA certs. This changeset includes
the regenerated certificates after that change.
Signed-off-by: William Woodruff <william@yossarian.net>
Co-authored-by: Victor Stinner <vstinner@python.org>
* Increment PyExpat_CAPI_MAGIC due to SetReparseDeferralEnabled addition.
This is a followup to git commit
6a95676bb5 from Github PR #115623.
* RESTify news API list.
Improve algorithm for computing which rolled-over log files to delete
in logging.TimedRotatingFileHandler. It is now reliable for handlers
without namer and with arbitrary deterministic namer that leaves
the datetime part in the file name unmodified.
If awailable, enable -fstrict-overflow for libmpdec. Also
shut off false positive warnings (-Warray-bounds).
The later was backported from mpdecimal-4.0.0.
This makes the asyncio REPL (`python -m asyncio`) more usable
and similar to the regular REPL.
This exposes register_readline() as a top-level function in site.py,
but it's intentionally undocumented.
Co-authored-by: Carol Willing <carolcode@willingconsulting.com>
Co-authored-by: Itamar Oren <itamarost@gmail.com>
* Do not overwrite already rolled over files. It happened at midnight or
during the DST change and caused the loss of data.
* computeRollover() now always return the timestamp larger than the
specified time.
* Fix computation of the rollover time during the DST change.
Support callables with the __call__() method and types with
__new__() and __init__() methods set to class methods, static
methods, bound methods, partial functions, and other types of
methods and descriptors.
Add tests for numerous types of callables and descriptors.
Allow controlling Expat >=2.6.0 reparse deferral (CVE-2023-52425) by adding five new methods:
- `xml.etree.ElementTree.XMLParser.flush`
- `xml.etree.ElementTree.XMLPullParser.flush`
- `xml.parsers.expat.xmlparser.GetReparseDeferralEnabled`
- `xml.parsers.expat.xmlparser.SetReparseDeferralEnabled`
- `xml.sax.expatreader.ExpatParser.flush`
Based on the "flush" idea from https://github.com/python/cpython/pull/115138#issuecomment-1932444270 .
### Notes
- Please treat as a security fix related to CVE-2023-52425.
Includes code suggested-by: Snild Dolkow <snild@sony.com>
and by core dev Serhiy Storchaka.
This change is part of the work on PEP-738: Adding Android as a
supported platform.
* Remove the "1.0" suffix from libpython's filename on Android, which
would prevent Gradle from packaging it into an app.
* Simplify the build command in the Makefile so that libpython always
gets given an SONAME with the `-Wl-h` argument, even if the SONAME is
identical to the actual filename.
* Disable a number of functions on Android which can be compiled and
linked against, but always fail at runtime. As a result, the native
_multiprocessing module is no longer built for Android.
* gh-115390 (bee7bb331) added some pre-determined results to the
configure script for things that can't be autodetected when
cross-compiling; this change adds Android to these where appropriate.
* Add a couple more pre-determined results for Android, and making them
cover iOS as well. This means the --enable-ipv6 configure option will
no longer be required on either platform.
Use of a proxy is intended to defer DNS for the hosts to the proxy itself, rather than a potential for information leak of the host doing DNS resolution itself for any reason. Proxy bypass lists are strictly name based. Most implementations of proxy support agree.
Nothing else in Python generally logs the contents of variables, so this
can be very unexpected for developers and could leak sensitive
information in to terminals and log files.
In some cases we might cause a StreamWriter to stay alive even when the
application has dropped all references to it. This prevents us from
doing automatical cleanup, and complaining that the StreamWriter wasn't
properly closed.
Fortunately, the extra reference was never actually used for anything so
we can just drop it.
Instead of showing a dot for each iteration, show:
- '.' for zero (on negative) leaks
- number of leaks for 1-9
- 'X' if there are more leaks
This allows more rapid iteration: when bisecting, I don't need
to wait for the final report to see if the test still leaks.
Also, show the full result if there are any non-zero entries.
This shows negative entries, for the unfortunate cases where
a reference is created and cleaned up in different runs.
Test *failure* is still determined by the existing heuristic.
Setting the __class__ attribute of a lazy-loading module to ModuleType enables other threads to attempt to access attributes before the loading is complete. Now that is protected by a lock.
The platform standard on macOS is to show a proxy icon for open
files in the titlebar of Windows. Make sure IDLE matches this
behaviour.
Don't use both the long and short names in the window title.
The behaviour of other editors (such as Text Editor) is to show
only the short name with the proxy icon.
Co-authored-by: Terry Jan Reedy <tjreedy@udel.edu>
Part of the work on PEP 738: Adding Android as a supported platform.
* Rename the LIBPYTHON variable to MODULE_LDFLAGS, to more accurately
reflect its purpose.
* Edit makesetup to use MODULE_LDFLAGS when linking extension modules.
* Edit the Makefile so that extension modules depend on libpython on
Android and Cygwin.
* Restore `-fPIC` on Android. It was removed several years ago with a
note that the toolchain used it automatically, but this is no longer
the case. Omitting it causes all linker commands to fail with an error
like `relocation R_AARCH64_ADR_PREL_PG_HI21 cannot be used against
symbol '_Py_FalseStruct'; recompile with -fPIC`.
Reproducer depends on terminal size - the traceback occurs when there's
an option long enough so the usage line doesn't fit the terminal width.
Option order is also important for reproducibility.
Excluding empty groups (with all options suppressed) from inserts
fixes the problem.
This builds on https://github.com/python/cpython/pull/106807, which adds
a return code to ResourceTracker, to make future debugging easier.
Testing this “in situ” proved difficult, since the global ResourceTracker is
involved in test infrastructure. So, the tests here create a new instance and
feed it fake data.
---------
Co-authored-by: Yonatan Bitton <yonatan.bitton@perception-point.io>
Co-authored-by: Yonatan Bitton <bityob@gmail.com>
Co-authored-by: Antoine Pitrou <antoine@python.org>
Restore support of such combination, disabled in gh-113796.
csv.writer() now quotes empty fields if delimiter is a space and
skipinitialspace is true and raises exception if quoting is not possible.
This change adds an `eval_breaker` field to `PyThreadState`. The primary
motivation is for performance in free-threaded builds: with thread-local eval
breakers, we can stop a specific thread (e.g., for an async exception) without
interrupting other threads.
The source of truth for the global instrumentation version is stored in the
`instrumentation_version` field in PyInterpreterState. Threads usually read the
version from their local `eval_breaker`, where it continues to be colocated
with the eval breaker bits.
Keep the old private _PyCFunctionFastWithKeywords name (Python 3.7)
as an alias to the new public name PyCFunctionFastWithKeywords
(Python 3.13a4).
_PyCFunctionWithKeywords doesn't exist in Python 3.13a3, whereas
_PyCFunctionFastWithKeywords was removed in Python 3.13a4.
The charset name "Windows-31J" is registered in the IANA Charset Registry[1]
and is implemented in Python as the cp932 codec.
[1] https://www.iana.org/assignments/charset-reg/windows-31J
Signed-off-by: Masayuki Moriyama <masayuki.moriyama@miraclelinux.com>
* test.bisect_cmd now exit with code 0 on success, and code 1 on
failure. Before, it was the opposite.
* test.bisect_cmd now runs the test worker process with
-X faulthandler.
* regrtest RunTests: Add create_python_cmd() and bisect_cmd()
methods.
Fix the exceptions raised by posixpath.commonpath
Raise ValueError, not IndexError when passed an empty iterable. Raise
TypeError, not ValueError when passed None.
lseek() always returns 0 for character pseudo-devices like
`/dev/urandom` (for other non-regular files, e.g. `/dev/stdin`, it
always returns -1, to which CPython reacts by raising appropriate
exceptions). They are thus technically seekable despite not having seek
semantics.
When calling read() on e.g. an instance of `io.BufferedReader` that
wraps such a file, `BufferedReader` reads ahead, filling its buffer,
creating a discrepancy between the number of bytes read and the internal
`tell()` always returning 0, which previously resulted in e.g.
`BufferedReader.tell()` or `BufferedReader.seek()` being able to return
positions < 0 even though these are supposed to be always >= 0.
Invariably keep the return value non-negative by returning
max(former_return_value, 0) instead, and add some corresponding tests.
This change essentially replaces usage of `%1` with `%~1`, which removes
quotes, if any. Without this change, the if statements fail due to
the quotes mangling the syntax.
Additionally, this change works around comma being treated as a parameter
delimiter in test.bat by escaping commas at time of parsing. Tested
combinations of rt and regrtest arguments, all seems to work as before
but now you can specify commas in arguments like "-uall,extralargefile".
* gh-114572: Fix locking in cert_store_stats and get_ca_certs
cert_store_stats and get_ca_certs query the SSLContext's X509_STORE with
X509_STORE_get0_objects, but reading the result requires a lock. See
https://github.com/openssl/openssl/pull/23224 for details.
Instead, use X509_STORE_get1_objects, newly added in that PR.
X509_STORE_get1_objects does not exist in current OpenSSLs, but we can
polyfill it with X509_STORE_lock and X509_STORE_unlock.
* Work around const-correctness problem
* Add missing X509_STORE_get1_objects failure check
* Add blurb
* bpo-38364: unwrap partialmethods just like we unwrap partials
The inspect.isgeneratorfunction, inspect.iscoroutinefunction and inspect.isasyncgenfunction already unwrap functools.partial objects, this patch adds support for partialmethod objects as well.
Also: Rename _partialmethod to __partialmethod__.
Since we're checking this attribute on arbitrary function-like objects,
we should use the namespace reserved for core Python.
---------
Co-authored-by: Petr Viktorin <encukou@gmail.com>
Use critical sections to make deque methods that operate on mutable
state thread-safe when the GIL is disabled. This is mostly accomplished
by using the @critical_section Argument Clinic directive, though there
are a few places where this was not possible and critical sections had
to be manually acquired/released.
Add PythonFinalizationError exception. This exception derived from
RuntimeError is raised when an operation is blocked during the Python
finalization.
The following functions now raise PythonFinalizationError, instead of
RuntimeError:
* _thread.start_new_thread()
* subprocess.Popen
* os.fork()
* os.fork1()
* os.forkpty()
Morever, _winapi.Overlapped finalizer now logs an unraisable
PythonFinalizationError, instead of an unraisable RuntimeError.
We add _winapi.BatchedWaitForMultipleObjects to wait for larger numbers of handles.
This is an internal module, hence undocumented, and should be used with caution.
Check the docstring for info before using BatchedWaitForMultipleObjects.
Part of the PEP 730 work to add iOS support.
This change lays the groundwork for introducing iOS/tvOS/watchOS
frameworks; it includes the structural refactoring needed so that iOS
branches can be added into in a subsequent PR.
Summary of changes:
* Updates config.sub to the 2024-01-01 release. This is the "as
released" version of config.sub.
* Adds a RESSRCDIR variable to allow sharing of macOS and iOS Makefile
steps.
* Adds an INSTALLTARGETS variable so platforms can customise which
targets are actually installed. This will be used to exclude certain
targets (e.g., binaries, manfiles) from iOS framework installs.
* Adds a PYTHONFRAMEWORKINSTALLNAMEPREFIX variable; this is used as
the install name for the library. This is needed to allow for iOS
frameworks to specify an @rpath-based install name.
* Evaluates MACHDEP earlier in the configure process so that
ac_sys_system is available.
* Modifies _PYTHON_HOST_PLATFORM evaluation for cross-platform builds
so that the CPU architecture is differentiated from the host
identifier. This will be used to generate a _PYTHON_HOST_PLATFORM
definition that includes ABI information, not just CPU architecture.
* Differentiates between SOABI_PLATFORM and PLATFORM_TRIPLET.
SOABI_PLATFORM is used in binary module names, and includes the ABI,
but not the OS or CPU architecture (e.g.,
math.cpython-313-iphonesimulator.dylib). PLATFORM_TRIPLET is used
as the sys._multiarch value, and on iOS will contains the ABI and
architecture (e.g., iphoneos-arm64). This differentiation hasn't
historically been needed because while macOS is a multiarch platform,
it uses a bare darwin as PLATFORM_TRIPLE.
* Removes the use of the deprecated -Wl,-single_module flag when
compiling macOS frameworks.
* Some whitespace normalisation where there was a mix of spaces and tabs
in a single block.
When replace() method is called on a subclass of datetime, date or time,
properly call derived constructor. Previously, only the base class's
constructor was called.
Also, make sure to pass non-zero fold values when creating subclasses in
various methods. Previously, fold was silently ignored.
Immediate merits:
* eliminate complex workarounds for 'z' format support
(NOTE: mpdecimal recently added 'z' support, so this becomes
efficient in the long term.)
* fix 'z' format memory leak
* fix 'z' format applied to 'F'
* fix missing '#' format support
Suggested and prototyped by Stefan Krah.
Fixes gh-114563, gh-91060
Co-authored-by: Stefan Krah <skrah@bytereef.org>
* Class methods no longer have "method of builtins.type instance" note.
* Corresponding notes are now added for class and unbound methods.
* Method and function aliases now have references to the module or the
class where the origin was defined if it differs from the current.
* Bound methods are now listed in the static methods section.
* Methods of builtin classes are now supported as well as methods of
Python classes.
Now the special comparison methods like `__eq__` and `__lt__` return
NotImplemented if one of comparands is date and other is datetime
instead of ignoring the time part and the time zone or forcefully
return "not equal" or raise TypeError.
It makes comparison of date and datetime subclasses more symmetric
and allows to change the default behavior by overriding
the special comparison methods in subclasses.
It is now the same as if date and datetime was independent classes.
Setters for members with an unsigned integer type now support
the same range of valid values for objects that has a __index__()
method as for int.
Previously, Py_T_UINT, Py_T_ULONG and Py_T_ULLONG did not support
objects that has a __index__() method larger than LONG_MAX.
Py_T_ULLONG did not support negative ints. Now it supports them and
emits a RuntimeWarning.
By default, it preserves an inconsistent behavior of older Python
versions: packs the count into a 1-tuple if only one or none
options are specified (including 'update'), returns None instead of 0.
Except that setting wantobjects to 0 no longer affects the result.
Add a new parameter return_ints: specifying return_ints=True makes
Text.count() always returning the single count as an integer
instead of a 1-tuple or None.
Avoid race conditions in the creation of directories during concurrent
extraction in tarfile and zipfile.
Co-authored-by: Samantha Hughes <shughes-uk@users.noreply.github.com>
Co-authored-by: Peder Bergebakken Sundt <pbsds@hotmail.com>
When expanding and filtering paths for a `**` wildcard segment, build an `re.Pattern` object from the subsequent pattern parts, rather than the entire pattern, and match against the `os.DirEntry` object prior to instantiating a path object. Also skip compiling a pattern when expanding a `*` wildcard segment.
Victor Stinner
Zachary Ware
kernc
AN Long
John Sloboda
mpage
Serhiy Storchaka
Donghee Na
Tian Gao
Zackery Spytz
vxiiduu
Jason R. Coombs
Nikita Sobolev
Brett Cannon
Tim Peters
Thomas Wouters
Pablo Galindo Salgado
Mehdi Drissi
Christopher Chavez
Guido van Rossum
Ethan Furman
Pierre Ossman (ThinLinc team)
Malcolm Smith
Brett Simmers
Sergey B Kirpichev
infohash
Russell Keith-Magee
Shantanu
William Woodruff
Jason Zhang
Ken Jin
Itamar Oren
Prince Roshan
Tobias Rautenkranz
Yuriy Chernyshov
pan324
Gregory P. Smith
Erlend E. Aasland
Marco Trevisan
Rémi Lapeyre
Steve Dower
Sebastian Pipping
Seth Michael Larson
Weii Wang
Jelle Zijlstra
Miguel Brito
Petr Viktorin
Jérémie Detrey
Kien Dang
Emmanuel Arias
Alex Waygood
Raymond Hettinger
Furkan Onder
Matan Perelman
Jay Ting
Barney Gale
Chris Markiewicz
Kirill Podoprigora
Ronald Oussoren
Irit Katriel
Gabriele Catania
Daniel Mach
Frank Hoffmann
Victor Westerhuis
Sam Gross
Eugene Toder
Alexander Shadchin
Masayuki Moriyama
Jamie Phan
Sebastian Rittau
Hugo van Kemenade
Brian Schubert
wookie184
6t8k
Thomas Weißschuh
Derek Higgins
Ammar Askar
Łukasz Langa
David Benjamin
monkeyman192
Martijn Pieters
David Hewitt
kcatss
Ezio Melotti
John Belmonte
Brandt Bucher
Soumendra Ganguly
Mike Zimin