* Detect source file encoding.
* Use the "replace" error handler even for UTF-8 (default) encoding.
* Remove the BOM.
* Fix detection of too long lines if they contain NUL.
* Return the head rather than the tail for truncated long lines.
This allows to use positional argument with nargs='*' and without default
in mutually exclusive group and improves error message about required
arguments.
Arguments with the value identical to the default value (e.g. booleans,
small integers, empty or 1-character strings) are no longer considered
"not present".
- If setting `_fields_` fails, e.g. with AttributeError, don't set the attribute in `__dict__`
- Document the “finalization” behaviour
- Beef up tests: add `getattr`, test Union as well as Structure
- Put common functionality in a common function
Co-authored-by: Peter Bierma <zintensitydev@gmail.com>
In case of usage a long command along with max_help_position more than
the length of the command, the command's help was incorrectly started
on the new line.
Co-authored-by: Pavel Ditenbir <pavel.ditenbir@gmail.com>
We were sometimes passing None as the globals argument to eval(), which makes it
inherit the globals from the calling scope. Instead, ensure that globals is always
non-None. The test was passing accidentally because I passed "annotationlib" as a
module object; fix that. Also document the parameters to ForwardRef() and remove
two unused private ones.
Co-authored-by: Alex Waygood <Alex.Waygood@Gmail.com>
https://github.com/python/cpython/issues/88496 replaced text.update with text.update_idletasks in colorizer.py and outwin.py to fix test failures on macOS. While theoretically correct, the result was Shell freezing when receiving continuous short strings to print. Test: `while 1: 1`.
The guess is that there is no idle time in which to do the screen update. Reverting the change in one of the files,
outwin, fixes the issue. Colorizer runs ever 1/20 second and seems to work fine.
When running test-outwin on macOS, alias 'update'
to 'update_idletasks on the text used for testing.
Add a helper function that checks whether the test suite is running
inside a systemd-nspawn container, and skip the few tests failing
with `--suppress-sync=true` in that case. The tests are failing because
`--suppress-sync=true` stubs out `fsync()`, `fdatasync()` and `msync()`
calls, and therefore they always return success without checking for
invalid arguments.
Call `os.open(__file__, os.O_RDONLY | os.O_SYNC)` and check the errno to
detect whether `--suppress-sync=true` is actually used, and skip
the tests only in that scenario.
Debian (and derivatives) provide a /usr/bin/pager binary, managed by the
alternatives system, that always points to an available pager utility.
Allow _pyrepl to use it, to follow system policy.
This is a very trivial change, from a patch that Debian has been
carrying since 2.7 era. Seems appropriate to upstream.
https://bugs.debian.org/799555
Multiple places in the I/O stack optimize common cases by using the
information from stat. Currently individual members are extracted from
the stat and stored into the fileio struct. Refactor the code to store
the whole stat struct instead.
Parallels the changes to _io. The `stat` Python object doesn't allow
changing members, so rather than modifying estimated_size, just clear
the value.
pyrepl: Support Del, PgUp, and PgDn on TERM=vt100
From Fedora's /etc/inputrc:
"\e[5~": history-search-backward
"\e[6~": history-search-forward
"\e[3~": delete-char
Fixes https://github.com/python/cpython/issues/124027
Use a `_PyStackRef` and defer the reference to `f_executable` when
possible. This avoids some reference count contention in the common case
of executing the same code object from multiple threads concurrently in
the free-threaded build.
* gh-116608: Apply style and compatibility changes from importlib_metadata.
* gh-121735: Ensure module-adjacent resources are loadable from a zipfile.
* gh-121735: Allow all modules to be processed by the ZipReader.
* Add blurb
* Remove update-zips script, unneeded.
* Remove unnecessary references to removed static fixtures.
* Remove zipdata fixtures, unused.
In gh-121602, I applied a fix to a builtin types initialization bug.
That fix made sense in the context of some broader future changes,
but introduced a little bit of extra complexity. That fix has turned
out to be incomplete for some of the builtin types we haven't
been testing. I found that out while improving the tests.
A while back, @markshannon suggested a simpler fix that doesn't
have that problem, which I've already applied to 3.12 and 3.13.
I'm switching to that here. Given the potential long-term
benefits of the more complex (but still incomplete) approach,
I'll circle back to it in the future, particularly after I've improved
the tests so no corner cases slip through the cracks.
(This is effectively a "forward-port" of 716c677 from 3.13.)
* Raise PicklingError instead of UnicodeEncodeError, ValueError
and AttributeError in both implementations.
* Chain the original exception to the pickle-specific one as __context__.
* Include the error message of ImportError and some AttributeError in
the PicklingError error message.
* Unify error messages between Python and C implementations.
* Refer to documented __reduce__ and __newobj__ callables instead of
internal methods (e.g. save_reduce()) or pickle opcodes (e.g. NEWOBJ).
* Include more details in error messages (what expected, what got).
* Avoid including a potentially long repr of an arbitrary object in
error messages.
This switches the main pyrepl event loop to always be non-blocking so that it
can listen to incoming interruptions from other threads.
This also resolves invalid display of exceptions from other threads
(gh-123178).
This also fixes freezes with pasting and an active input hook.
Use a `PyMutex` to avoid the race in mutex initialization. Use relaxed
atomics to avoid the data race on reading `_PyOS_ReadlineTState` when
checking for re-entrant calls.
Setting dev_mode to 1 in an isolated configuration now enables also
faulthandler.
Moreover, setting "module_search_paths" option with
PyInitConfig_SetStrList() now sets "module_search_paths_set" to 1.
Improve import time of `socket` by writing `socket.errorTab`
as a constant and lazy import modules.
Co-authored-by: Pieter Eendebak <pieter.eendebak@gmail.com>
Co-authored-by: Bénédikt Tran <10796600+picnixz@users.noreply.github.com>
Co-authored-by: Gregory P. Smith <greg@krypto.org>
Co-authored-by: Hugo van Kemenade <1324225+hugovk@users.noreply.github.com>
Co-authored-by: Shantanu <12621235+hauntsaninja@users.noreply.github.com>
Increases the multiprocessing connection buffer size from 8k to 64k for efficiency, without overallocating.
Co-authored-by: Bénédikt Tran <10796600+picnixz@users.noreply.github.com>
Co-authored-by: Victor Stinner <vstinner@python.org>
Add PyConfig_Get(), PyConfig_GetInt(), PyConfig_Set() and
PyConfig_Names() functions to get and set the current runtime Python
configuration.
Add visibility and "sys spec" to config and preconfig specifications.
_PyConfig_AsDict() now converts PyConfig.xoptions as a dictionary.
Co-authored-by: Bénédikt Tran <10796600+picnixz@users.noreply.github.com>
Co-authored-by: blurb-it[bot] <43283697+blurb-it[bot]@users.noreply.github.com>
Co-authored-by: Hugo van Kemenade <1324225+hugovk@users.noreply.github.com>
* Remove backtracking when parsing tarfile headers
* Rewrite PAX header parsing to be stricter
* Optimize parsing of GNU extended sparse headers v0.0
Co-authored-by: Kirill Podoprigora <kirill.bast9@mail.ru>
Co-authored-by: Gregory P. Smith <greg@krypto.org>
* urljoin() with relative reference "?" sets empty query and removes fragment.
* Preserve empty components (authority, params, query, fragment) in urljoin().
* Preserve empty components (authority, params, query) in urldefrag().
Also refactor the code and get rid of double _coerce_args() and
_coerce_result() calls in urljoin(), urldefrag(), urlparse() and
urlunparse().
When checking if the registering browser is the "OS preferred browser", do not use a substring search - that makes no sense: one can have a preferred browser that looks like a super-string of a known browser, e.g. "firefox-nightly" vs "firefox".
https://github.com/python/cpython/issues/108172 explains in more detail, and lays out a potential better future enhancement for this case of just using xdg-open. We'll go with this for now.
---------
Co-authored-by: Alex Waygood <Alex.Waygood@Gmail.com>
See https://github.com/python/cpython/issues/121313 for analysis, but this greatly reduces memory overallocation and overhead when multiprocessing is sending non-small data over its pipes between processes.
This checks are redundant in normal circumstances and can only work if
the extension registry was intentionally broken.
* The Python implementation now raises exception for the extension code
with false boolean value.
* Simplify the C code. RuntimeError is now raised in explicit checks.
* Add many tests.
The `zip_next` function uses a common optimization technique for methods
that generate tuples. The iterator maintains an internal reference to
the returned tuple. When the method is called again, it checks if the
internal tuple's reference count is 1. If so, the tuple can be reused.
However, this approach is not safe under the free-threading build:
after checking the reference count, another thread may perform the same
check and also reuse the tuple. This can result in a double decref on
the items of the replaced tuple and a double incref (memory leak) on
the items of the tuple being set.
This adds a function, `_PyObject_IsUniquelyReferenced` that
encapsulates the stricter logic necessary for the free-threaded build:
the internal tuple must be owned by the current thread, have a local
refcount of one, and a shared refcount of zero.
* Make `weakref.WeakSet` safe against concurrent mutations while it is being iterated.
`_IterationGuard` is no longer used for `WeakSet`, it now relies on copying the underlying set which is an atomic operation while iterating so that it can be modified by other threads.
Remove *ignore* and *on_error* arguments from `pathlib.Path.copy[_into]()`,
because these arguments are under-designed. Specifically:
- *ignore* is appropriated from `shutil.copytree()`, but it's not clear
how it should apply when the user copies a non-directory. We've changed
the callback signature from the `shutil` version, but I'm not confident
the new signature is as good as it can be.
- *on_error* is a generalisation of `shutil.copytree()`'s error handling,
which is to accumulate exceptions and raise a single `shutil.Error` at
the end. It's not obvious which solution is better.
Additionally, this arguments may be challenging to implement in future user
subclasses of `PathBase`, which might utilise a native recursive copying
method.
Per feedback from Paul Moore on GH-123158, it's better to defer making
`Path.delete()` public than ship it with under-designed error handling
capabilities.
We leave a remnant `_delete()` method, which is used by `move()`. Any
functionality not needed by `move()` is deleted.
These two methods accept an *existing* directory path, onto which we join
the source path's base name to form the final target path.
A possible alternative implementation is to check for directories in
`copy()` and `move()` and adjust the target path, which is done in several
`shutil` functions. This behaviour is helpful in a shell context, but
less so in a stored program that explicitly specifies destinations. For
example, a user that calls `Path('foo.py').copy('bar.py')` might not
imagine that `bar.py/foo.py` would be created, but under the alternative
implementation this will happen if `bar.py` is an existing directory.
When display lines above the cursor come from the cache, the first line
to not come from the cache may be a wrapped line, starting half way
through a logical line in the buffer. Detect and handle this case to
avoid accidentally drawing a stray prompt in the middle of a logical
line.
Add a `Path.move()` method that moves a file or directory tree, and returns a new `Path` instance pointing to the target.
This method is similar to `shutil.move()`, except that it doesn't accept a *copy_function* argument, and it doesn't check whether the destination is an existing directory.
* pass the original string error message from the ftplib error to URLError()
* Update request.py
Change error string for ftp error to be consistent with other errors reported for ftp
* Add NEWS entry for change to urllib.request for ftp errors.
* Track the change in the ftp error message in the test.
Check that the current default heap is initialized in
`_mi_os_get_aligned_hint` and `mi_os_claim_huge_pages`.
The mimalloc function `_mi_os_get_aligned_hint` assumes that there is an
initialized default heap. This is true for our main thread, but not for
background threads. The problematic code path is usually called during
initialization (i.e., `Py_Initialize`), but it may also be called if the
program allocates large amounts of memory in total.
The crash only affected the free-threaded build.
`Path.read_bytes()` is used to read a whole file. buffering /
BufferedIO is focused around making small, possibly interleaved,
read/write efficient which doesn't add value in this case.
On my Mac, running the benchmark:
```python
import pyperf
from pathlib import Path
def read_all(all_paths):
for p in all_paths:
p.read_bytes()
def read_file(path_obj):
path_obj.read_bytes()
all_rst = list(Path("Doc").glob("**/*.rst"))
all_py = list(Path(".").glob("**/*.py"))
assert all_rst, "Should have found rst files"
assert all_py, "Should have found python source files"
runner = pyperf.Runner()
runner.bench_func("read_file_small", read_file, Path("Doc/howto/clinic.rst"))
runner.bench_func("read_file_large", read_file, Path("Doc/c-api/typeobj.rst"))
```
before:
```python
.....................
read_file_small: Mean +- std dev: 6.80 us +- 0.07 us
.....................
read_file_large: Mean +- std dev: 10.8 us +- 0.2 us
````
after:
```python
.....................
read_file_small: Mean +- std dev: 5.67 us +- 0.05 us
.....................
read_file_large: Mean +- std dev: 9.77 us +- 0.52 us
```
Adds a script for running the test suite on Android emulator devices. Starting
with a fresh install of the Android Commandline tools; the script manages
installing other requirements, starting the emulator (if required), and
retrieving results from that emulator.
This replaces the existing hashlib Blake2 module with a single implementation that uses HACL\*'s Blake2b/Blake2s implementations. We added support for all the modes exposed by the Python API, including tree hashing, leaf nodes, and so on. We ported and merged all of these changes upstream in HACL\*, added test vectors based on Python's existing implementation, and exposed everything needed for hashlib.
This was joint work done with @R1kM.
See the PR for much discussion and benchmarking details. TL;DR: On many systems, 8-50% faster (!) than `libb2`, on some systems it appeared 10-20% slower than `libb2`.
As of 529a160 (gh-118204), building with HAVE_DYNAMIC_LOADING stopped working. This is a minimal fix just to get builds working again. There are actually a number of long-standing deficiencies with HAVE_DYNAMIC_LOADING builds that need to be resolved separately.
Replace `umask(0)` with `umask(0o002)` so the created files are not
world-writable, and replace `umask(0o022)` with `umask(0o026)` to check
that permissions for 'others' can still be set.
Co-authored-by: Adam Turner <9087854+AA-Turner@users.noreply.github.com>
Co-authored-by: Hugo van Kemenade <1324225+hugovk@users.noreply.github.com>
Co-authored-by: Alyssa Coghlan <ncoghlan@gmail.com>
There were a still a number of gaps in the tests, including not looking
at all the builtin types and not checking wrappers in subinterpreters
that weren't in the main interpreter. This fixes all that.
I considered incorporating the names of the PyTypeObject fields
(a la gh-122866), but figured doing so doesn't add much value.
Rename `pathlib.Path.copy()` to `_copy_file()` (i.e. make it private.)
Rename `pathlib.Path.copytree()` to `copy()`, and add support for copying
non-directories. This simplifies the interface for users, and nicely
complements the upcoming `move()` and `delete()` methods (which will also
accept any type of file.)
Co-authored-by: Adam Turner <9087854+AA-Turner@users.noreply.github.com>
This reverts commit dcc028d924 and
commit 6c54e5d721.
Keep the deprecated logging warn() method in Python 3.13.
Co-authored-by: Gregory P. Smith <greg@krypto.org>
Co-authored-by: Hugo van Kemenade <1324225+hugovk@users.noreply.github.com>
* Parameters after the var-positional parameter are now keyword-only
instead of positional-or-keyword.
* Correctly calculate min_kw_only.
* Raise errors for invalid combinations of the var-positional parameter
with "*", "/" and deprecation markers.
Return -1 and set an exception on error; return 0 if the iterator is
exhausted, and return 1 if the next item was fetched successfully.
Prefer this API to PyIter_Next(), which requires the caller to use
PyErr_Occurred() to differentiate between iterator exhaustion and errors.
Co-authered-by: Irit Katriel <iritkatriel@yahoo.com>
Fix _PyArg_UnpackKeywordsWithVararg for the case when argument for
positional-or-keyword parameter is passed by keyword.
There was only one such case in the stdlib -- the TypeVar constructor.
Frames of methods in code and codeop modules was show with non-default
sys.excepthook.
Save correct tracebacks in sys.last_traceback and update __traceback__
attribute of sys.last_value and sys.last_exc.
Rename `pathlib.Path.rmtree()` to `delete()`, and add support for deleting
non-directories. This simplifies the interface for users, and nicely
complements the upcoming `move()` and `copy()` methods (which will also
accept any type of file.)
Fix PyEval_GetLocals() to avoid SystemError ("bad argument to
internal function"). Don't redefine the 'ret' variable in the if
block.
Add an unit test on PyEval_GetLocals().
The free-threaded build partially stores heap type reference counts in
distributed manner in per-thread arrays. This avoids reference count
contention when creating or destroying instances.
Co-authored-by: Ken Jin <kenjin@python.org>
Modifies the handling of stdout/stderr redirection on Android to accomodate
the rate and buffer size limits imposed by Android's logging infrastructure.
As per C11 DR#471, ctanh (0 + i NaN) and ctanh (0 + i Inf) should return
0 + i NaN (with "invalid" exception in the second case). This has
corresponding implications for ctan(z), as its errors and special cases
are handled as if the operation is implemented by -i*ctanh(i*z).
This patch fixes cmath's code to do same.
Glibs patch: https://sourceware.org/git/?p=glibc.git;a=commitdiff;h=d15e83c5f5231d971472b5ffc9219d54056ca0f1
As per C11 DR#471 (adjusted resolution accepted for C17), cacosh (0 +
iNaN) should return NaN ± i pi/2, not NaN + iNaN. This patch
fixes cmath's code to do same.
The `PyStructSequence` destructor would crash if it was deallocated after
its type's dictionary was cleared by the GC, because it couldn't compute
the "real size" of the instance. This could occur with relatively
straightforward code in the free-threaded build or with a reference
cycle involving the type in the default build, due to differing orders
in which `tp_clear()` was called.
Account for the non-sequence fields in `tp_basicsize` and use that,
along with `Py_SIZE()`, to compute the "real" size of a
`PyStructSequence` in the dealloc function. This avoids the accesses to
the type's dictionary during dealloc, which were unsafe.
On recent versions of macOS (sometime between Catalina and Sonoma 14.5), the default Hovertip foreground color changed from black to white, thereby matching the background. This might be a matter of matching the white foreground of the dark-mode text. The unreadable result is shown here (#120083 (comment)).
The foreground and background colors were made parameters so we can pass different colors for future additional hovertips in IDLE.
---------
Co-authored-by: Terry Jan Reedy <tjreedy@udel.edu>
This flag was added as an escape hatch in gh-91401 and backported to
Python 3.10. The flag broke at some point between its addition and now.
As there is currently no publicly known environments that require this,
remove it rather than work on fixing it.
This leaves the flag in the subprocess module to not break code which
may have used / checked the flag itself.
discussion: https://discuss.python.org/t/subprocess-use-vfork-escape-hatch-broken-fix-or-remove/56915/2
Currently, idle-dev@python.org and idle-dev mailing list
serve to collect spam (90+%). Change About IDLE to direct
discussions to discuss.python.org. Users are already
doing so.
## Encode header parts that contain newlines
Per RFC 2047:
> [...] these encoding schemes allow the
> encoding of arbitrary octet values, mail readers that implement this
> decoding should also ensure that display of the decoded data on the
> recipient's terminal will not cause unwanted side-effects
It seems that the "quoted-word" scheme is a valid way to include
a newline character in a header value, just like we already allow
undecodable bytes or control characters.
They do need to be properly quoted when serialized to text, though.
## Verify that email headers are well-formed
This should fail for custom fold() implementations that aren't careful
about newlines.
Co-authored-by: Bas Bloemsaat <bas@bloemsaat.org>
Co-authored-by: Serhiy Storchaka <storchaka@gmail.com>
* gh-122188: Move magic number to its own file
* Add versionadded directive
* Do work in C
* Integrate launcher.c
* Make _pyc_magic_number private
* Remove metadata
* Move sys.implementation -> _imp
* Modernize comment
* Move _RAW_MAGIC_NUMBER to the C side as well
* _pyc_magic_number -> pyc_magic_number
* Remove unused import
* Update docs
* Apply suggestions from code review
Co-authored-by: Eric Snow <ericsnowcurrently@gmail.com>
* Fix typo in tests
---------
Co-authored-by: Eric Snow <ericsnowcurrently@gmail.com>
* Authenticate socket connection for `socket.socketpair()` fallback when the platform does not have a native `socketpair` C API. We authenticate in-process using `getsocketname` and `getpeername` (thanks to Nathaniel J Smith for that suggestion).
Co-authored-by: Gregory P. Smith <greg@krypto.org>
The tests were only checking cases where the slot wrapper was present in the initial case. They were missing when the slot wrapper was added in the additional initializations. This fixes that.
* Use compensated summation for complex sums with floating-point items.
This amends #121176.
* sum() specializations for floats and complexes now use
PyLong_AsDouble() instead of PyLong_AsLongAndOverflow() and
compensated summation as well.
Co-authored-by: Tomas R <tomas.roun8@gmail.com>
Co-authored-by: Scott Odle <scott@sjodle.com>
Co-authored-by: Bénédikt Tran <10796600+picnixz@users.noreply.github.com>
Co-authored-by: Petr Viktorin <encukou@gmail.com>
Serializing objects with complex __qualname__ (such as unbound methods and
nested classes) by name no longer involves serializing parent objects by value
in pickle protocols < 4.
TSan doesn't fully recognize the synchronization via I/O, so ensure that
socket name is retrieved earlier and use a different socket for sending
the "STOP" message.
In the process of speeding up readall, A number of related tests
(ex. large file tests in test_zipfile) found problems with the
change I was making. This adds I/O tests to specifically test these
cases to help ensure they don't regress and hopefully make debugging
easier.
This is part of the improvements from
https://github.com/python/cpython/pull/121593#issuecomment-2222261986
Serhiy Storchaka
Victor Stinner
Jason Fried
Donghee Na
Petr Viktorin
Bénédikt Tran
Alex Waygood
Savannah Ostrowski
Jelle Zijlstra
Terry Jan Reedy
Michał Górny
Y5
Brian Schubert
sobolevn
Jakub Kulík
Stefano Rivera
Sergey B Kirpichev
Jacek
Furkan Onder
Tomas R
Cody Maloney
Sam James
neonene
Loïc Estève
Wenzel Jakob
Benjamin Peterson
Irit Katriel
Malcolm Smith
Yngve Mardal Moe
Miro Hrončok
Jason R. Coombs
Sam Gross
Mark Shannon
Anders Kaseorg
Xie Yanbo
Tushar Sadhwani
Raymond Hettinger
Eric Snow
algonell
Adam Turner
Seth Michael Larson
Łukasz Langa
Nadeshiko Manju
aorcajo
Arnon Yaari
Peter Bierma
nkinnan
Bar Harel
Nice Zombies
Wulian
Damien
devdanzin
Yorik Hansen
Inada Naoki
abstractee
Barney Gale
Daniel Hollas
Yoda
Aarni Koskela
Luka
Oded Arbel
Alexander P.
Bogdan Romanyuk
Alexandr Mitin
Pieter Eendebak
Kumar Aditya
Kirill Podoprigora
Matt Wozniski
Pablo Galindo Salgado
CF Bolz-Tereick
Shantanu
Kevin Evans
blhsing
mpage
Jeremy Hylton
James
Pedro Lacerda
Mateusz Nowak
Eddie Elizondo
Jonathan Protzenko
Trey Hunner
Nico Mexis
Tian Gao
Erlend E. Aasland
Hugo van Kemenade
Ali Hamdan
pukkandan
Pradyun Gedam
Виталий Дмитриев
John Riggles
Jonathon Vandezande
jianghuyiyuan
Russell Keith-Magee
Michael Droettboom
Dino Viehland
Clinton
WilliamRoyNelson
AN Long
Mikołaj Kuranowski