Change the default multiprocessing start method away from fork to forkserver or spawn on the remaining platforms where it was fork. See the issue for context. This makes the default far more thread safe (other than for people spawning threads at import time... - don't do that!).
Co-authored-by: blurb-it[bot] <43283697+blurb-it[bot]@users.noreply.github.com>
Co-authored-by: Hugo van Kemenade <1324225+hugovk@users.noreply.github.com>
* Reduce the number of iterations and the number of threads so a
whole test file takes less than a minute.
* Refactor test_racing_iter_extend() to remove two levels of
indentation.
* test_monitoring() uses a sleep of 100 ms instead of 1 second.
* Add NEWS.d entry
* Allow ISO-8601 24:00 alternative to midnight on datetime.time.fromisoformat()
* Allow ISO-8601 24:00 alternative to midnight on datetime.datetime.fromisoformat()
* Add NEWS.d entry
* Improve error message when hour is 24 and minute/second/microsecond is not 0
* Add tests for 24:00 fromisoformat
* Remove duplicate call to days_in_month() by storing in variable
* Add Python implementation
* Fix Lint
* Fix differing error msg in datetime.fromisoformat implementations when 24hrs has non-zero time component(s)
* Fix using time components inside tzinfo in Python implementation
* Don't parse tzinfo in C implementation when invalid iso midnight
* Remove duplicated variable in datetime test assertion line
* Add self to acknowledgements
* Remove duplicate NEWS entry
* Linting
* Add missing test case for when wrapping the year makes it invalid (too large)
* Lib/test/certdata: do not hardcode reference cert data into tests
The script was simply printing the reference data and asking
users to update it by hand into the test suites. This can
be easily improved by writing the data into files and
having the test cases load the files.
* make_ssl_certs: make it possible to pass in expiration dates from command line
Note that in this commit, the defaults are same as they were,
so if nothing is specified the script works as before.
---------
Signed-off-by: Alexander Kanavin <alex.kanavin@gmail.com>
Often, ForwardRefs represent a single simple name. In that case, we
can avoid going through the overhead of creating AST nodes and code
objects and calling eval(): we can simply look up the name directly
in the relevant namespaces.
Co-authored-by: Victor Stinner <vstinner@python.org>
typing.get_origin() does what we need here, without reaching into
typing internals. This shouldn't change any behavior (so I am going
to skip news), but it sets a good example for other users introspecting
typing objects.
* Detect source file encoding.
* Use the "replace" error handler even for UTF-8 (default) encoding.
* Remove the BOM.
* Fix detection of too long lines if they contain NUL.
* Return the head rather than the tail for truncated long lines.
This allows to use positional argument with nargs='*' and without default
in mutually exclusive group and improves error message about required
arguments.
Arguments with the value identical to the default value (e.g. booleans,
small integers, empty or 1-character strings) are no longer considered
"not present".
- If setting `_fields_` fails, e.g. with AttributeError, don't set the attribute in `__dict__`
- Document the “finalization” behaviour
- Beef up tests: add `getattr`, test Union as well as Structure
- Put common functionality in a common function
Co-authored-by: Peter Bierma <zintensitydev@gmail.com>
In case of usage a long command along with max_help_position more than
the length of the command, the command's help was incorrectly started
on the new line.
Co-authored-by: Pavel Ditenbir <pavel.ditenbir@gmail.com>
We were sometimes passing None as the globals argument to eval(), which makes it
inherit the globals from the calling scope. Instead, ensure that globals is always
non-None. The test was passing accidentally because I passed "annotationlib" as a
module object; fix that. Also document the parameters to ForwardRef() and remove
two unused private ones.
Co-authored-by: Alex Waygood <Alex.Waygood@Gmail.com>
https://github.com/python/cpython/issues/88496 replaced text.update with text.update_idletasks in colorizer.py and outwin.py to fix test failures on macOS. While theoretically correct, the result was Shell freezing when receiving continuous short strings to print. Test: `while 1: 1`.
The guess is that there is no idle time in which to do the screen update. Reverting the change in one of the files,
outwin, fixes the issue. Colorizer runs ever 1/20 second and seems to work fine.
When running test-outwin on macOS, alias 'update'
to 'update_idletasks on the text used for testing.
Add a helper function that checks whether the test suite is running
inside a systemd-nspawn container, and skip the few tests failing
with `--suppress-sync=true` in that case. The tests are failing because
`--suppress-sync=true` stubs out `fsync()`, `fdatasync()` and `msync()`
calls, and therefore they always return success without checking for
invalid arguments.
Call `os.open(__file__, os.O_RDONLY | os.O_SYNC)` and check the errno to
detect whether `--suppress-sync=true` is actually used, and skip
the tests only in that scenario.
Debian (and derivatives) provide a /usr/bin/pager binary, managed by the
alternatives system, that always points to an available pager utility.
Allow _pyrepl to use it, to follow system policy.
This is a very trivial change, from a patch that Debian has been
carrying since 2.7 era. Seems appropriate to upstream.
https://bugs.debian.org/799555
Multiple places in the I/O stack optimize common cases by using the
information from stat. Currently individual members are extracted from
the stat and stored into the fileio struct. Refactor the code to store
the whole stat struct instead.
Parallels the changes to _io. The `stat` Python object doesn't allow
changing members, so rather than modifying estimated_size, just clear
the value.
pyrepl: Support Del, PgUp, and PgDn on TERM=vt100
From Fedora's /etc/inputrc:
"\e[5~": history-search-backward
"\e[6~": history-search-forward
"\e[3~": delete-char
Fixes https://github.com/python/cpython/issues/124027
Use a `_PyStackRef` and defer the reference to `f_executable` when
possible. This avoids some reference count contention in the common case
of executing the same code object from multiple threads concurrently in
the free-threaded build.
* gh-116608: Apply style and compatibility changes from importlib_metadata.
* gh-121735: Ensure module-adjacent resources are loadable from a zipfile.
* gh-121735: Allow all modules to be processed by the ZipReader.
* Add blurb
* Remove update-zips script, unneeded.
* Remove unnecessary references to removed static fixtures.
* Remove zipdata fixtures, unused.
In gh-121602, I applied a fix to a builtin types initialization bug.
That fix made sense in the context of some broader future changes,
but introduced a little bit of extra complexity. That fix has turned
out to be incomplete for some of the builtin types we haven't
been testing. I found that out while improving the tests.
A while back, @markshannon suggested a simpler fix that doesn't
have that problem, which I've already applied to 3.12 and 3.13.
I'm switching to that here. Given the potential long-term
benefits of the more complex (but still incomplete) approach,
I'll circle back to it in the future, particularly after I've improved
the tests so no corner cases slip through the cracks.
(This is effectively a "forward-port" of 716c677 from 3.13.)
* Raise PicklingError instead of UnicodeEncodeError, ValueError
and AttributeError in both implementations.
* Chain the original exception to the pickle-specific one as __context__.
* Include the error message of ImportError and some AttributeError in
the PicklingError error message.
* Unify error messages between Python and C implementations.
* Refer to documented __reduce__ and __newobj__ callables instead of
internal methods (e.g. save_reduce()) or pickle opcodes (e.g. NEWOBJ).
* Include more details in error messages (what expected, what got).
* Avoid including a potentially long repr of an arbitrary object in
error messages.
This switches the main pyrepl event loop to always be non-blocking so that it
can listen to incoming interruptions from other threads.
This also resolves invalid display of exceptions from other threads
(gh-123178).
This also fixes freezes with pasting and an active input hook.
Use a `PyMutex` to avoid the race in mutex initialization. Use relaxed
atomics to avoid the data race on reading `_PyOS_ReadlineTState` when
checking for re-entrant calls.
Setting dev_mode to 1 in an isolated configuration now enables also
faulthandler.
Moreover, setting "module_search_paths" option with
PyInitConfig_SetStrList() now sets "module_search_paths_set" to 1.
Improve import time of `socket` by writing `socket.errorTab`
as a constant and lazy import modules.
Co-authored-by: Pieter Eendebak <pieter.eendebak@gmail.com>
Co-authored-by: Bénédikt Tran <10796600+picnixz@users.noreply.github.com>
Co-authored-by: Gregory P. Smith <greg@krypto.org>
Co-authored-by: Hugo van Kemenade <1324225+hugovk@users.noreply.github.com>
Co-authored-by: Shantanu <12621235+hauntsaninja@users.noreply.github.com>
Increases the multiprocessing connection buffer size from 8k to 64k for efficiency, without overallocating.
Co-authored-by: Bénédikt Tran <10796600+picnixz@users.noreply.github.com>
Co-authored-by: Victor Stinner <vstinner@python.org>
Add PyConfig_Get(), PyConfig_GetInt(), PyConfig_Set() and
PyConfig_Names() functions to get and set the current runtime Python
configuration.
Add visibility and "sys spec" to config and preconfig specifications.
_PyConfig_AsDict() now converts PyConfig.xoptions as a dictionary.
Co-authored-by: Bénédikt Tran <10796600+picnixz@users.noreply.github.com>
Co-authored-by: blurb-it[bot] <43283697+blurb-it[bot]@users.noreply.github.com>
Co-authored-by: Hugo van Kemenade <1324225+hugovk@users.noreply.github.com>
* Remove backtracking when parsing tarfile headers
* Rewrite PAX header parsing to be stricter
* Optimize parsing of GNU extended sparse headers v0.0
Co-authored-by: Kirill Podoprigora <kirill.bast9@mail.ru>
Co-authored-by: Gregory P. Smith <greg@krypto.org>
* urljoin() with relative reference "?" sets empty query and removes fragment.
* Preserve empty components (authority, params, query, fragment) in urljoin().
* Preserve empty components (authority, params, query) in urldefrag().
Also refactor the code and get rid of double _coerce_args() and
_coerce_result() calls in urljoin(), urldefrag(), urlparse() and
urlunparse().
When checking if the registering browser is the "OS preferred browser", do not use a substring search - that makes no sense: one can have a preferred browser that looks like a super-string of a known browser, e.g. "firefox-nightly" vs "firefox".
https://github.com/python/cpython/issues/108172 explains in more detail, and lays out a potential better future enhancement for this case of just using xdg-open. We'll go with this for now.
---------
Co-authored-by: Alex Waygood <Alex.Waygood@Gmail.com>
See https://github.com/python/cpython/issues/121313 for analysis, but this greatly reduces memory overallocation and overhead when multiprocessing is sending non-small data over its pipes between processes.
This checks are redundant in normal circumstances and can only work if
the extension registry was intentionally broken.
* The Python implementation now raises exception for the extension code
with false boolean value.
* Simplify the C code. RuntimeError is now raised in explicit checks.
* Add many tests.
The `zip_next` function uses a common optimization technique for methods
that generate tuples. The iterator maintains an internal reference to
the returned tuple. When the method is called again, it checks if the
internal tuple's reference count is 1. If so, the tuple can be reused.
However, this approach is not safe under the free-threading build:
after checking the reference count, another thread may perform the same
check and also reuse the tuple. This can result in a double decref on
the items of the replaced tuple and a double incref (memory leak) on
the items of the tuple being set.
This adds a function, `_PyObject_IsUniquelyReferenced` that
encapsulates the stricter logic necessary for the free-threaded build:
the internal tuple must be owned by the current thread, have a local
refcount of one, and a shared refcount of zero.
* Make `weakref.WeakSet` safe against concurrent mutations while it is being iterated.
`_IterationGuard` is no longer used for `WeakSet`, it now relies on copying the underlying set which is an atomic operation while iterating so that it can be modified by other threads.
Remove *ignore* and *on_error* arguments from `pathlib.Path.copy[_into]()`,
because these arguments are under-designed. Specifically:
- *ignore* is appropriated from `shutil.copytree()`, but it's not clear
how it should apply when the user copies a non-directory. We've changed
the callback signature from the `shutil` version, but I'm not confident
the new signature is as good as it can be.
- *on_error* is a generalisation of `shutil.copytree()`'s error handling,
which is to accumulate exceptions and raise a single `shutil.Error` at
the end. It's not obvious which solution is better.
Additionally, this arguments may be challenging to implement in future user
subclasses of `PathBase`, which might utilise a native recursive copying
method.
Per feedback from Paul Moore on GH-123158, it's better to defer making
`Path.delete()` public than ship it with under-designed error handling
capabilities.
We leave a remnant `_delete()` method, which is used by `move()`. Any
functionality not needed by `move()` is deleted.
These two methods accept an *existing* directory path, onto which we join
the source path's base name to form the final target path.
A possible alternative implementation is to check for directories in
`copy()` and `move()` and adjust the target path, which is done in several
`shutil` functions. This behaviour is helpful in a shell context, but
less so in a stored program that explicitly specifies destinations. For
example, a user that calls `Path('foo.py').copy('bar.py')` might not
imagine that `bar.py/foo.py` would be created, but under the alternative
implementation this will happen if `bar.py` is an existing directory.
When display lines above the cursor come from the cache, the first line
to not come from the cache may be a wrapped line, starting half way
through a logical line in the buffer. Detect and handle this case to
avoid accidentally drawing a stray prompt in the middle of a logical
line.
Add a `Path.move()` method that moves a file or directory tree, and returns a new `Path` instance pointing to the target.
This method is similar to `shutil.move()`, except that it doesn't accept a *copy_function* argument, and it doesn't check whether the destination is an existing directory.
* pass the original string error message from the ftplib error to URLError()
* Update request.py
Change error string for ftp error to be consistent with other errors reported for ftp
* Add NEWS entry for change to urllib.request for ftp errors.
* Track the change in the ftp error message in the test.
Check that the current default heap is initialized in
`_mi_os_get_aligned_hint` and `mi_os_claim_huge_pages`.
The mimalloc function `_mi_os_get_aligned_hint` assumes that there is an
initialized default heap. This is true for our main thread, but not for
background threads. The problematic code path is usually called during
initialization (i.e., `Py_Initialize`), but it may also be called if the
program allocates large amounts of memory in total.
The crash only affected the free-threaded build.
`Path.read_bytes()` is used to read a whole file. buffering /
BufferedIO is focused around making small, possibly interleaved,
read/write efficient which doesn't add value in this case.
On my Mac, running the benchmark:
```python
import pyperf
from pathlib import Path
def read_all(all_paths):
for p in all_paths:
p.read_bytes()
def read_file(path_obj):
path_obj.read_bytes()
all_rst = list(Path("Doc").glob("**/*.rst"))
all_py = list(Path(".").glob("**/*.py"))
assert all_rst, "Should have found rst files"
assert all_py, "Should have found python source files"
runner = pyperf.Runner()
runner.bench_func("read_file_small", read_file, Path("Doc/howto/clinic.rst"))
runner.bench_func("read_file_large", read_file, Path("Doc/c-api/typeobj.rst"))
```
before:
```python
.....................
read_file_small: Mean +- std dev: 6.80 us +- 0.07 us
.....................
read_file_large: Mean +- std dev: 10.8 us +- 0.2 us
````
after:
```python
.....................
read_file_small: Mean +- std dev: 5.67 us +- 0.05 us
.....................
read_file_large: Mean +- std dev: 9.77 us +- 0.52 us
```
Adds a script for running the test suite on Android emulator devices. Starting
with a fresh install of the Android Commandline tools; the script manages
installing other requirements, starting the emulator (if required), and
retrieving results from that emulator.
This replaces the existing hashlib Blake2 module with a single implementation that uses HACL\*'s Blake2b/Blake2s implementations. We added support for all the modes exposed by the Python API, including tree hashing, leaf nodes, and so on. We ported and merged all of these changes upstream in HACL\*, added test vectors based on Python's existing implementation, and exposed everything needed for hashlib.
This was joint work done with @R1kM.
See the PR for much discussion and benchmarking details. TL;DR: On many systems, 8-50% faster (!) than `libb2`, on some systems it appeared 10-20% slower than `libb2`.
As of 529a160 (gh-118204), building with HAVE_DYNAMIC_LOADING stopped working. This is a minimal fix just to get builds working again. There are actually a number of long-standing deficiencies with HAVE_DYNAMIC_LOADING builds that need to be resolved separately.
Replace `umask(0)` with `umask(0o002)` so the created files are not
world-writable, and replace `umask(0o022)` with `umask(0o026)` to check
that permissions for 'others' can still be set.
Co-authored-by: Adam Turner <9087854+AA-Turner@users.noreply.github.com>
Co-authored-by: Hugo van Kemenade <1324225+hugovk@users.noreply.github.com>
Co-authored-by: Alyssa Coghlan <ncoghlan@gmail.com>
There were a still a number of gaps in the tests, including not looking
at all the builtin types and not checking wrappers in subinterpreters
that weren't in the main interpreter. This fixes all that.
I considered incorporating the names of the PyTypeObject fields
(a la gh-122866), but figured doing so doesn't add much value.
Rename `pathlib.Path.copy()` to `_copy_file()` (i.e. make it private.)
Rename `pathlib.Path.copytree()` to `copy()`, and add support for copying
non-directories. This simplifies the interface for users, and nicely
complements the upcoming `move()` and `delete()` methods (which will also
accept any type of file.)
Co-authored-by: Adam Turner <9087854+AA-Turner@users.noreply.github.com>
This reverts commit dcc028d924 and
commit 6c54e5d721.
Keep the deprecated logging warn() method in Python 3.13.
Co-authored-by: Gregory P. Smith <greg@krypto.org>
Co-authored-by: Hugo van Kemenade <1324225+hugovk@users.noreply.github.com>
* Parameters after the var-positional parameter are now keyword-only
instead of positional-or-keyword.
* Correctly calculate min_kw_only.
* Raise errors for invalid combinations of the var-positional parameter
with "*", "/" and deprecation markers.
Return -1 and set an exception on error; return 0 if the iterator is
exhausted, and return 1 if the next item was fetched successfully.
Prefer this API to PyIter_Next(), which requires the caller to use
PyErr_Occurred() to differentiate between iterator exhaustion and errors.
Co-authered-by: Irit Katriel <iritkatriel@yahoo.com>
Fix _PyArg_UnpackKeywordsWithVararg for the case when argument for
positional-or-keyword parameter is passed by keyword.
There was only one such case in the stdlib -- the TypeVar constructor.
Frames of methods in code and codeop modules was show with non-default
sys.excepthook.
Save correct tracebacks in sys.last_traceback and update __traceback__
attribute of sys.last_value and sys.last_exc.
Rename `pathlib.Path.rmtree()` to `delete()`, and add support for deleting
non-directories. This simplifies the interface for users, and nicely
complements the upcoming `move()` and `copy()` methods (which will also
accept any type of file.)
Fix PyEval_GetLocals() to avoid SystemError ("bad argument to
internal function"). Don't redefine the 'ret' variable in the if
block.
Add an unit test on PyEval_GetLocals().
The free-threaded build partially stores heap type reference counts in
distributed manner in per-thread arrays. This avoids reference count
contention when creating or destroying instances.
Co-authored-by: Ken Jin <kenjin@python.org>
Modifies the handling of stdout/stderr redirection on Android to accomodate
the rate and buffer size limits imposed by Android's logging infrastructure.
As per C11 DR#471, ctanh (0 + i NaN) and ctanh (0 + i Inf) should return
0 + i NaN (with "invalid" exception in the second case). This has
corresponding implications for ctan(z), as its errors and special cases
are handled as if the operation is implemented by -i*ctanh(i*z).
This patch fixes cmath's code to do same.
Glibs patch: https://sourceware.org/git/?p=glibc.git;a=commitdiff;h=d15e83c5f5231d971472b5ffc9219d54056ca0f1
As per C11 DR#471 (adjusted resolution accepted for C17), cacosh (0 +
iNaN) should return NaN ± i pi/2, not NaN + iNaN. This patch
fixes cmath's code to do same.
The `PyStructSequence` destructor would crash if it was deallocated after
its type's dictionary was cleared by the GC, because it couldn't compute
the "real size" of the instance. This could occur with relatively
straightforward code in the free-threaded build or with a reference
cycle involving the type in the default build, due to differing orders
in which `tp_clear()` was called.
Account for the non-sequence fields in `tp_basicsize` and use that,
along with `Py_SIZE()`, to compute the "real" size of a
`PyStructSequence` in the dealloc function. This avoids the accesses to
the type's dictionary during dealloc, which were unsafe.
On recent versions of macOS (sometime between Catalina and Sonoma 14.5), the default Hovertip foreground color changed from black to white, thereby matching the background. This might be a matter of matching the white foreground of the dark-mode text. The unreadable result is shown here (#120083 (comment)).
The foreground and background colors were made parameters so we can pass different colors for future additional hovertips in IDLE.
---------
Co-authored-by: Terry Jan Reedy <tjreedy@udel.edu>
This flag was added as an escape hatch in gh-91401 and backported to
Python 3.10. The flag broke at some point between its addition and now.
As there is currently no publicly known environments that require this,
remove it rather than work on fixing it.
This leaves the flag in the subprocess module to not break code which
may have used / checked the flag itself.
discussion: https://discuss.python.org/t/subprocess-use-vfork-escape-hatch-broken-fix-or-remove/56915/2
Currently, idle-dev@python.org and idle-dev mailing list
serve to collect spam (90+%). Change About IDLE to direct
discussions to discuss.python.org. Users are already
doing so.
## Encode header parts that contain newlines
Per RFC 2047:
> [...] these encoding schemes allow the
> encoding of arbitrary octet values, mail readers that implement this
> decoding should also ensure that display of the decoded data on the
> recipient's terminal will not cause unwanted side-effects
It seems that the "quoted-word" scheme is a valid way to include
a newline character in a header value, just like we already allow
undecodable bytes or control characters.
They do need to be properly quoted when serialized to text, though.
## Verify that email headers are well-formed
This should fail for custom fold() implementations that aren't careful
about newlines.
Co-authored-by: Bas Bloemsaat <bas@bloemsaat.org>
Co-authored-by: Serhiy Storchaka <storchaka@gmail.com>
Raymond Hettinger
sobolevn
Malcolm Smith
Jelle Zijlstra
Erlend E. Aasland
Gregory P. Smith
Dino Viehland
Bénédikt Tran
Tian Gao
Peter Bierma
Victor Stinner
Ron Frederick
dgpb
larryhastings
Nice Zombies
TizzySaurus
Alexander Kanavin
Zachary Ware
Emily Morehouse
Irit Katriel
Eric V. Smith
Jakub Kulík
Jérémie Detrey
Payton
Serhiy Storchaka
Jason Fried
Donghee Na
Petr Viktorin
Alex Waygood
Savannah Ostrowski
Terry Jan Reedy
Michał Górny
Y5
Brian Schubert
Stefano Rivera
Sergey B Kirpichev
Jacek
Furkan Onder
Tomas R
Cody Maloney
Sam James
neonene
Loïc Estève
Wenzel Jakob
Benjamin Peterson
Yngve Mardal Moe
Miro Hrončok
Jason R. Coombs
Sam Gross
Mark Shannon
Anders Kaseorg
Xie Yanbo
Tushar Sadhwani
Eric Snow
algonell
Adam Turner
Seth Michael Larson
Łukasz Langa
Nadeshiko Manju
aorcajo
Arnon Yaari
nkinnan
Bar Harel
Wulian
Damien
devdanzin
Yorik Hansen
Inada Naoki
abstractee
Barney Gale
Daniel Hollas
Yoda
Aarni Koskela
Luka
Oded Arbel
Alexander P.
Bogdan Romanyuk
Alexandr Mitin
Pieter Eendebak
Kumar Aditya
Kirill Podoprigora
Matt Wozniski
Pablo Galindo Salgado
CF Bolz-Tereick
Shantanu
Kevin Evans
blhsing
mpage
Jeremy Hylton
James
Pedro Lacerda
Mateusz Nowak
Eddie Elizondo
Jonathan Protzenko
Trey Hunner
Nico Mexis
Hugo van Kemenade
Ali Hamdan
pukkandan
Pradyun Gedam
Виталий Дмитриев
John Riggles
Jonathon Vandezande
jianghuyiyuan
Russell Keith-Magee