* pass the original string error message from the ftplib error to URLError()
* Update request.py
Change error string for ftp error to be consistent with other errors reported for ftp
* Add NEWS entry for change to urllib.request for ftp errors.
* Track the change in the ftp error message in the test.
Check that the current default heap is initialized in
`_mi_os_get_aligned_hint` and `mi_os_claim_huge_pages`.
The mimalloc function `_mi_os_get_aligned_hint` assumes that there is an
initialized default heap. This is true for our main thread, but not for
background threads. The problematic code path is usually called during
initialization (i.e., `Py_Initialize`), but it may also be called if the
program allocates large amounts of memory in total.
The crash only affected the free-threaded build.
`Path.read_bytes()` is used to read a whole file. buffering /
BufferedIO is focused around making small, possibly interleaved,
read/write efficient which doesn't add value in this case.
On my Mac, running the benchmark:
```python
import pyperf
from pathlib import Path
def read_all(all_paths):
for p in all_paths:
p.read_bytes()
def read_file(path_obj):
path_obj.read_bytes()
all_rst = list(Path("Doc").glob("**/*.rst"))
all_py = list(Path(".").glob("**/*.py"))
assert all_rst, "Should have found rst files"
assert all_py, "Should have found python source files"
runner = pyperf.Runner()
runner.bench_func("read_file_small", read_file, Path("Doc/howto/clinic.rst"))
runner.bench_func("read_file_large", read_file, Path("Doc/c-api/typeobj.rst"))
```
before:
```python
.....................
read_file_small: Mean +- std dev: 6.80 us +- 0.07 us
.....................
read_file_large: Mean +- std dev: 10.8 us +- 0.2 us
````
after:
```python
.....................
read_file_small: Mean +- std dev: 5.67 us +- 0.05 us
.....................
read_file_large: Mean +- std dev: 9.77 us +- 0.52 us
```
Adds a script for running the test suite on Android emulator devices. Starting
with a fresh install of the Android Commandline tools; the script manages
installing other requirements, starting the emulator (if required), and
retrieving results from that emulator.
This replaces the existing hashlib Blake2 module with a single implementation that uses HACL\*'s Blake2b/Blake2s implementations. We added support for all the modes exposed by the Python API, including tree hashing, leaf nodes, and so on. We ported and merged all of these changes upstream in HACL\*, added test vectors based on Python's existing implementation, and exposed everything needed for hashlib.
This was joint work done with @R1kM.
See the PR for much discussion and benchmarking details. TL;DR: On many systems, 8-50% faster (!) than `libb2`, on some systems it appeared 10-20% slower than `libb2`.
As of 529a160 (gh-118204), building with HAVE_DYNAMIC_LOADING stopped working. This is a minimal fix just to get builds working again. There are actually a number of long-standing deficiencies with HAVE_DYNAMIC_LOADING builds that need to be resolved separately.
Replace `umask(0)` with `umask(0o002)` so the created files are not
world-writable, and replace `umask(0o022)` with `umask(0o026)` to check
that permissions for 'others' can still be set.
Co-authored-by: Adam Turner <9087854+AA-Turner@users.noreply.github.com>
Co-authored-by: Hugo van Kemenade <1324225+hugovk@users.noreply.github.com>
Co-authored-by: Alyssa Coghlan <ncoghlan@gmail.com>
There were a still a number of gaps in the tests, including not looking
at all the builtin types and not checking wrappers in subinterpreters
that weren't in the main interpreter. This fixes all that.
I considered incorporating the names of the PyTypeObject fields
(a la gh-122866), but figured doing so doesn't add much value.
Rename `pathlib.Path.copy()` to `_copy_file()` (i.e. make it private.)
Rename `pathlib.Path.copytree()` to `copy()`, and add support for copying
non-directories. This simplifies the interface for users, and nicely
complements the upcoming `move()` and `delete()` methods (which will also
accept any type of file.)
Co-authored-by: Adam Turner <9087854+AA-Turner@users.noreply.github.com>
This reverts commit dcc028d924 and
commit 6c54e5d721.
Keep the deprecated logging warn() method in Python 3.13.
Co-authored-by: Gregory P. Smith <greg@krypto.org>
Co-authored-by: Hugo van Kemenade <1324225+hugovk@users.noreply.github.com>
* Parameters after the var-positional parameter are now keyword-only
instead of positional-or-keyword.
* Correctly calculate min_kw_only.
* Raise errors for invalid combinations of the var-positional parameter
with "*", "/" and deprecation markers.
Return -1 and set an exception on error; return 0 if the iterator is
exhausted, and return 1 if the next item was fetched successfully.
Prefer this API to PyIter_Next(), which requires the caller to use
PyErr_Occurred() to differentiate between iterator exhaustion and errors.
Co-authered-by: Irit Katriel <iritkatriel@yahoo.com>
Fix _PyArg_UnpackKeywordsWithVararg for the case when argument for
positional-or-keyword parameter is passed by keyword.
There was only one such case in the stdlib -- the TypeVar constructor.
Frames of methods in code and codeop modules was show with non-default
sys.excepthook.
Save correct tracebacks in sys.last_traceback and update __traceback__
attribute of sys.last_value and sys.last_exc.
Rename `pathlib.Path.rmtree()` to `delete()`, and add support for deleting
non-directories. This simplifies the interface for users, and nicely
complements the upcoming `move()` and `copy()` methods (which will also
accept any type of file.)
Fix PyEval_GetLocals() to avoid SystemError ("bad argument to
internal function"). Don't redefine the 'ret' variable in the if
block.
Add an unit test on PyEval_GetLocals().
The free-threaded build partially stores heap type reference counts in
distributed manner in per-thread arrays. This avoids reference count
contention when creating or destroying instances.
Co-authored-by: Ken Jin <kenjin@python.org>
Modifies the handling of stdout/stderr redirection on Android to accomodate
the rate and buffer size limits imposed by Android's logging infrastructure.
As per C11 DR#471, ctanh (0 + i NaN) and ctanh (0 + i Inf) should return
0 + i NaN (with "invalid" exception in the second case). This has
corresponding implications for ctan(z), as its errors and special cases
are handled as if the operation is implemented by -i*ctanh(i*z).
This patch fixes cmath's code to do same.
Glibs patch: https://sourceware.org/git/?p=glibc.git;a=commitdiff;h=d15e83c5f5231d971472b5ffc9219d54056ca0f1
As per C11 DR#471 (adjusted resolution accepted for C17), cacosh (0 +
iNaN) should return NaN ± i pi/2, not NaN + iNaN. This patch
fixes cmath's code to do same.
The `PyStructSequence` destructor would crash if it was deallocated after
its type's dictionary was cleared by the GC, because it couldn't compute
the "real size" of the instance. This could occur with relatively
straightforward code in the free-threaded build or with a reference
cycle involving the type in the default build, due to differing orders
in which `tp_clear()` was called.
Account for the non-sequence fields in `tp_basicsize` and use that,
along with `Py_SIZE()`, to compute the "real" size of a
`PyStructSequence` in the dealloc function. This avoids the accesses to
the type's dictionary during dealloc, which were unsafe.
On recent versions of macOS (sometime between Catalina and Sonoma 14.5), the default Hovertip foreground color changed from black to white, thereby matching the background. This might be a matter of matching the white foreground of the dark-mode text. The unreadable result is shown here (#120083 (comment)).
The foreground and background colors were made parameters so we can pass different colors for future additional hovertips in IDLE.
---------
Co-authored-by: Terry Jan Reedy <tjreedy@udel.edu>
This flag was added as an escape hatch in gh-91401 and backported to
Python 3.10. The flag broke at some point between its addition and now.
As there is currently no publicly known environments that require this,
remove it rather than work on fixing it.
This leaves the flag in the subprocess module to not break code which
may have used / checked the flag itself.
discussion: https://discuss.python.org/t/subprocess-use-vfork-escape-hatch-broken-fix-or-remove/56915/2
Currently, idle-dev@python.org and idle-dev mailing list
serve to collect spam (90+%). Change About IDLE to direct
discussions to discuss.python.org. Users are already
doing so.
## Encode header parts that contain newlines
Per RFC 2047:
> [...] these encoding schemes allow the
> encoding of arbitrary octet values, mail readers that implement this
> decoding should also ensure that display of the decoded data on the
> recipient's terminal will not cause unwanted side-effects
It seems that the "quoted-word" scheme is a valid way to include
a newline character in a header value, just like we already allow
undecodable bytes or control characters.
They do need to be properly quoted when serialized to text, though.
## Verify that email headers are well-formed
This should fail for custom fold() implementations that aren't careful
about newlines.
Co-authored-by: Bas Bloemsaat <bas@bloemsaat.org>
Co-authored-by: Serhiy Storchaka <storchaka@gmail.com>
* gh-122188: Move magic number to its own file
* Add versionadded directive
* Do work in C
* Integrate launcher.c
* Make _pyc_magic_number private
* Remove metadata
* Move sys.implementation -> _imp
* Modernize comment
* Move _RAW_MAGIC_NUMBER to the C side as well
* _pyc_magic_number -> pyc_magic_number
* Remove unused import
* Update docs
* Apply suggestions from code review
Co-authored-by: Eric Snow <ericsnowcurrently@gmail.com>
* Fix typo in tests
---------
Co-authored-by: Eric Snow <ericsnowcurrently@gmail.com>
* Authenticate socket connection for `socket.socketpair()` fallback when the platform does not have a native `socketpair` C API. We authenticate in-process using `getsocketname` and `getpeername` (thanks to Nathaniel J Smith for that suggestion).
Co-authored-by: Gregory P. Smith <greg@krypto.org>
The tests were only checking cases where the slot wrapper was present in the initial case. They were missing when the slot wrapper was added in the additional initializations. This fixes that.
* Use compensated summation for complex sums with floating-point items.
This amends #121176.
* sum() specializations for floats and complexes now use
PyLong_AsDouble() instead of PyLong_AsLongAndOverflow() and
compensated summation as well.
Co-authored-by: Tomas R <tomas.roun8@gmail.com>
Co-authored-by: Scott Odle <scott@sjodle.com>
Co-authored-by: Bénédikt Tran <10796600+picnixz@users.noreply.github.com>
Co-authored-by: Petr Viktorin <encukou@gmail.com>
Serializing objects with complex __qualname__ (such as unbound methods and
nested classes) by name no longer involves serializing parent objects by value
in pickle protocols < 4.
TSan doesn't fully recognize the synchronization via I/O, so ensure that
socket name is retrieved earlier and use a different socket for sending
the "STOP" message.
In the process of speeding up readall, A number of related tests
(ex. large file tests in test_zipfile) found problems with the
change I was making. This adds I/O tests to specifically test these
cases to help ensure they don't regress and hopefully make debugging
easier.
This is part of the improvements from
https://github.com/python/cpython/pull/121593#issuecomment-2222261986
This fixes the flakiness in:
* test_inspect_keeps_globals_from_inspected_file
* test_inspect_keeps_globals_from_inspected_module
The output already includes newlines. Adding newlines for every entry in
the output list introduces non-determinism because it added '\n' in
places where stdout is flushed or some buffer becomes full.
The regex also needed to be updated because pyrepl includes control
characters -- the visible output on each line doesn't immediately follow
a newline character.
Co-authored-by: Łukasz Langa <lukasz@langa.pl>
Add *preserve_metadata* keyword-only argument to `pathlib.Path.copytree()`,
defaulting to false. When set to true, we copy timestamps, permissions,
extended attributes and flags where available, like `shutil.copystat()`.
Add a `Path.rmtree()` method that removes an entire directory tree, like
`shutil.rmtree()`. The signature of the optional *on_error* argument
matches the `Path.walk()` argument of the same name, but differs from the
*onexc* and *onerror* arguments to `shutil.rmtree()`. Consistency within
pathlib is probably more important.
In the private pathlib ABCs, we add an implementation based on `walk()`.
Co-authored-by: Bénédikt Tran <10796600+picnixz@users.noreply.github.com>
* Reject uop definitions that declare values as 'unused' that are already cached by prior uops
* Track which variables are defined and only load from memory when needed
* Support explicit `flush` in macro definitions.
* Make sure stack is flushed in where needed.
Problem occurred when attribute xyz could not be pickled.
Since this is not trivial to selectively fix, block all
attributes (other than 'width'). IDLE does not access them
and they are private implementation details.
console.compile with the "single" param throws an exception when
there are multiple statements, never allowing to adding newlines
to a pasted code block (gh-121610)
This add a few extra checks to allow extending when in an indented
block, and tests for a few examples
Co-authored-by: Łukasz Langa <lukasz@langa.pl>
Any cross-interpreter mechanism for passing objects between interpreters must be very careful to respect isolation, even when the object is effectively immutable (e.g. int, str). Here this especially relates to when an interpreter sends one of its objects, and then is destroyed while the inter-interpreter machinery (e.g. queue) still holds a reference to the object.
When I added interpreters.Queue, I dealt with that case (using an atexit hook) by silently removing all items from the queue that were added by the finalizing interpreter.
Later, while working on concurrent.futures.InterpreterPoolExecutor (gh-116430), I noticed it was somewhat surprising when items were silently removed from the queue when the originating interpreter was destroyed. (See my comment on that PR.)
It took me a little while to realize what was going on. I expect that users, which much less context than I have, would experience the same pain.
My approach, here, to improving the situation is to give users three options:
1. return a singleton (interpreters.queues.UNBOUND) from Queue.get() in place of each removed item
2. raise an exception (interpreters.queues.ItemInterpreterDestroyed) from Queue.get() in place of each removed item
3. existing behavior: silently remove each item (i.e. Queue.get() skips each one)
The default will now be (1), but users can still explicitly opt in any of them, including to the silent removal behavior.
The behavior for each item may be set with the corresponding Queue.put() call. and a queue-wide default may be set when the queue is created. (This is the same as I did for "synconly".)
They are alternate constructors which only accept numbers
(including objects with special methods __float__, __complex__
and __index__), but not strings.
It is our general practice to make new optional parameters keyword-only,
even if the existing parameters are all positional-or-keyword. Passing
this parameter as positional would look confusing and could be error-prone
if additional parameters are added in the future.
On POSIX systems, excluding macOS framework installs, the lib directory
for the free-threaded build now includes a "t" suffix to avoid conflicts
with a co-located default build installation.
When builtin static types are initialized for a subinterpreter, various "tp" slots have already been inherited (for the main interpreter). This was interfering with the logic in add_operators() (in Objects/typeobject.c), causing a wrapper to get created when it shouldn't. This change fixes that by preserving the original data from the static type struct and checking that.
On heavily loaded machines, the subprocess may finish its sleep before
the parent process manages to synchronize with it via a failed lock.
This leads to errors like:
Exception: failed to sync child in 300.3 sec
Use pipes instead to mutually synchronize between parent and child.
The ProcessPoolForkserver combined with resource_tracker starts a thread
after forking, which is not supported by TSan.
Also skip test_multiprocessing_fork for the same reason
The change in gh-118157 (b2cd54a) should have also updated clear_singlephase_extension() but didn't. We fix that here. Note that clear_singlephase_extension() (AKA _PyImport_ClearExtension()) is only used in tests.
Add *preserve_metadata* keyword-only argument to `pathlib.Path.copy()`, defaulting to false. When set to true, we copy timestamps, permissions, extended attributes and flags where available, like `shutil.copystat()`. The argument has no effect on Windows, where metadata is always copied.
Internally (in the pathlib ABCs), path types gain `_readable_metadata` and `_writable_metadata` attributes. These sets of strings describe what kinds of metadata can be retrieved and stored. We take an intersection of `source._readable_metadata` and `target._writable_metadata` to minimise reads/writes. A new `_read_metadata()` method accepts a set of metadata keys and returns a dict with those keys, and a new `_write_metadata()` method accepts a dict of metadata. We *might* make these public in future, but it's hard to justify while the ABCs are still private.
Make error message for index() methods consistent
Remove the repr of the searched value (which can be arbitrary large)
from ValueError messages for list.index(), range.index(), deque.index(),
deque.remove() and ShareableList.index(). Make the error messages
consistent with error messages for other index() and remove()
methods.
This reduces the system call count of a simple program[0] that reads all
the `.rst` files in Doc by over 10% (5706 -> 4734 system calls on my
linux system, 5813 -> 4875 on my macOS)
This reduces the number of `fstat()` calls always and seek calls most
the time. Stat was always called twice, once at open (to error early on
directories), and a second time to get the size of the file to be able
to read the whole file in one read. Now the size is cached with the
first call.
The code keeps an optimization that if the user had previously read a
lot of data, the current position is subtracted from the number of bytes
to read. That is somewhat expensive so only do it on larger files,
otherwise just try and read the extra bytes and resize the PyBytes as
needeed.
I built a little test program to validate the behavior + assumptions
around relative costs and then ran it under `strace` to get a log of the
system calls. Full samples below[1].
After the changes, this is everything in one `filename.read_text()`:
```python3
openat(AT_FDCWD, "cpython/Doc/howto/clinic.rst", O_RDONLY|O_CLOEXEC) = 3`
fstat(3, {st_mode=S_IFREG|0644, st_size=343, ...}) = 0`
ioctl(3, TCGETS, 0x7ffdfac04b40) = -1 ENOTTY (Inappropriate ioctl for device)
lseek(3, 0, SEEK_CUR) = 0
read(3, ":orphan:\n\n.. This page is retain"..., 344) = 343
read(3, "", 1) = 0
close(3) = 0
```
This does make some tradeoffs
1. If the file size changes between open() and readall(), this will
still get all the data but might have more read calls.
2. I experimented with avoiding the stat + cached result for small files
in general, but on my dev workstation at least that tended to reduce
performance compared to using the fstat().
[0]
```python3
from pathlib import Path
nlines = []
for filename in Path("cpython/Doc").glob("**/*.rst"):
nlines.append(len(filename.read_text()))
```
[1]
Before small file:
```
openat(AT_FDCWD, "cpython/Doc/howto/clinic.rst", O_RDONLY|O_CLOEXEC) = 3
fstat(3, {st_mode=S_IFREG|0644, st_size=343, ...}) = 0
ioctl(3, TCGETS, 0x7ffe52525930) = -1 ENOTTY (Inappropriate ioctl for device)
lseek(3, 0, SEEK_CUR) = 0
lseek(3, 0, SEEK_CUR) = 0
fstat(3, {st_mode=S_IFREG|0644, st_size=343, ...}) = 0
read(3, ":orphan:\n\n.. This page is retain"..., 344) = 343
read(3, "", 1) = 0
close(3) = 0
```
After small file:
```
openat(AT_FDCWD, "cpython/Doc/howto/clinic.rst", O_RDONLY|O_CLOEXEC) = 3
fstat(3, {st_mode=S_IFREG|0644, st_size=343, ...}) = 0
ioctl(3, TCGETS, 0x7ffdfac04b40) = -1 ENOTTY (Inappropriate ioctl for device)
lseek(3, 0, SEEK_CUR) = 0
read(3, ":orphan:\n\n.. This page is retain"..., 344) = 343
read(3, "", 1) = 0
close(3) = 0
```
Before large file:
```
openat(AT_FDCWD, "cpython/Doc/c-api/typeobj.rst", O_RDONLY|O_CLOEXEC) = 3
fstat(3, {st_mode=S_IFREG|0644, st_size=133104, ...}) = 0
ioctl(3, TCGETS, 0x7ffe52525930) = -1 ENOTTY (Inappropriate ioctl for device)
lseek(3, 0, SEEK_CUR) = 0
lseek(3, 0, SEEK_CUR) = 0
fstat(3, {st_mode=S_IFREG|0644, st_size=133104, ...}) = 0
read(3, ".. highlight:: c\n\n.. _type-struc"..., 133105) = 133104
read(3, "", 1) = 0
close(3) = 0
```
After large file:
```
openat(AT_FDCWD, "cpython/Doc/c-api/typeobj.rst", O_RDONLY|O_CLOEXEC) = 3
fstat(3, {st_mode=S_IFREG|0644, st_size=133104, ...}) = 0
ioctl(3, TCGETS, 0x7ffdfac04b40) = -1 ENOTTY (Inappropriate ioctl for device)
lseek(3, 0, SEEK_CUR) = 0
lseek(3, 0, SEEK_CUR) = 0
read(3, ".. highlight:: c\n\n.. _type-struc"..., 133105) = 133104
read(3, "", 1) = 0
close(3) = 0
```
Co-authored-by: Shantanu <12621235+hauntsaninja@users.noreply.github.com>
Co-authored-by: Erlend E. Aasland <erlend.aasland@protonmail.com>
Co-authored-by: Victor Stinner <vstinner@python.org>
As noted in gh-117983, the import importlib.util can be triggered at
interpreter startup under some circumstances, so adding threading makes
it a potentially obligatory load.
Lazy loading is not used in the stdlib, so this removes an unnecessary
load for the majority of users and slightly increases the cost of the
first lazily loaded module.
An obligatory threading load breaks gevent, which monkeypatches the
stdlib. Although unsupported, there doesn't seem to be an offsetting
benefit to breaking their use case.
For reference, here are benchmarks for the current main branch:
```
❯ hyperfine -w 8 './python -c "import importlib.util"'
Benchmark 1: ./python -c "import importlib.util"
Time (mean ± σ): 9.7 ms ± 0.7 ms [User: 7.7 ms, System: 1.8 ms]
Range (min … max): 8.4 ms … 13.1 ms 313 runs
```
And with this patch:
```
❯ hyperfine -w 8 './python -c "import importlib.util"'
Benchmark 1: ./python -c "import importlib.util"
Time (mean ± σ): 8.4 ms ± 0.7 ms [User: 6.8 ms, System: 1.4 ms]
Range (min … max): 7.2 ms … 11.7 ms 352 runs
```
Compare to:
```
❯ hyperfine -w 8 './python -c pass'
Benchmark 1: ./python -c pass
Time (mean ± σ): 7.6 ms ± 0.6 ms [User: 5.9 ms, System: 1.6 ms]
Range (min … max): 6.7 ms … 11.3 ms 390 runs
```
This roughly halves the import time of importlib.util.
This amends 6988ff02a5: memory allocation for
stginfo->ffi_type_pointer.elements in PyCSimpleType_init() should be
more generic (perhaps someday fmt->pffi_type->elements will be not a
two-elements array).
It should finally resolve#61103.
Co-authored-by: Victor Stinner <vstinner@python.org>
Co-authored-by: Bénédikt Tran <10796600+picnixz@users.noreply.github.com>
Check for `ERROR_INVALID_PARAMETER` when calling `_winapi.CopyFile2()` and
raise `UnsupportedOperation`. In `Path.copy()`, handle this exception and
fall back to the `PathBase.copy()` implementation.
* Move get_signal_name() from test.libregrtest to test.support.
* Use get_signal_name() in support.script_helper.
* support.script_helper now decodes stdout and stderr from UTF-8,
instead of ASCII, if a command failed.
When creating the JUnit XML file, regrtest now escapes characters
which are invalid in XML, such as the chr(27) control character used
in ANSI escape sequences.
asyncio earlier relied on subprocess module to send signals to the process, this has some drawbacks one being that subprocess module unnecessarily calls waitpid on child processes and hence it races with asyncio implementation which internally uses child watchers. To mitigate this, now asyncio sends signals directly to the process without going through the subprocess on non windows systems. On Windows it fallbacks to subprocess module handling but on windows there are no child watchers so this issue doesn't exists altogether.
In some cases, previously computed as (nan+nanj), we could
recover meaningful component values in the result, see
e.g. the C11, Annex G.5.2, routine _Cdivd().
* parse_intermixed_args() now raises ArgumentError instead of calling
error() if exit_on_error is false.
* Internal code now always raises ArgumentError instead of calling
error(). It is then caught at the higher level and error() is called if
exit_on_error is true.
On Windows, test_cext and test_cppext now pass /WX flag to the MSC
compiler to treat all compiler warnings as errors. In verbose mode,
these tests now log the compiler commands to help debugging.
Change Py_BUILD_ASSERT_EXPR implementation on Windows to avoid a
compiler warning about an unnamed structure.
The check for whether the log file is a real file is expensive on NFS
filesystems. This commit reorders the rollover condition checking to
not do the file type check if the expected file size is less than the
rotation threshold.
Co-authored-by: Oleg Iarygin <oleg@arhadthedev.net>
This PR sets up tagged pointers for CPython.
The general idea is to create a separate struct _PyStackRef for everything on the evaluation stack to store the bits. This forces the C compiler to warn us if we try to cast things or pull things out of the struct directly.
Only for free threading: We tag the low bit if something is deferred - that means we skip incref and decref operations on it. This behavior may change in the future if Mark's plans to defer all objects in the interpreter loop pans out.
This implies a strict stack reference discipline is required. ALL incref and decref operations on stackrefs must use the stackref variants. It is unsafe to untag something then do normal incref/decref ops on it.
The new incref and decref variants are called dup and close. They mimic a "handle" API operating on these stackrefs.
Please read Include/internal/pycore_stackref.h for more information!
---------
Co-authored-by: Mark Shannon <9448417+markshannon@users.noreply.github.com>
PyUnicode_FromFormat() no longer produces the ending \ufffd
character for truncated C string when use precision with %s and %V.
It now truncates the string before the start of truncated multibyte sequences.
The integer part of the timestamp can be rounded up, while the millisecond
calculation truncates, causing the log timestamp to be wrong by up to 999 ms
(affected roughly 1 in 8 million timestamps).
Add `pathlib.Path.copytree()` method, which recursively copies one
directory to another.
This differs from `shutil.copytree()` in the following respects:
1. Our method has a *follow_symlinks* argument, whereas shutil's has a
*symlinks* argument with an inverted meaning.
2. Our method lacks something like a *copy_function* argument. It always
uses `Path.copy()` to copy files.
3. Our method lacks something like a *ignore_dangling_symlinks* argument.
Instead, users can filter out danging symlinks with *ignore*, or
ignore exceptions with *on_error*
4. Our *ignore* argument is a callable that accepts a single path object,
whereas shutil's accepts a path and a list of child filenames.
5. We add an *on_error* argument, which is a callable that accepts
an `OSError` instance. (`Path.walk()` also accepts such a callable).
Co-authored-by: Nice Zombies <nineteendo19d0@gmail.com>
* linked list
* add tail optmiization to linked list
* wip
* wip
* wip
* more fixes
* finally it works
* add tests
* remove weakreflist
* add some comments
* reduce code duplication in _asynciomodule.c
* address some review comments
* add invariants about the state of the linked list
* add better explanation
* clinic regen
* reorder branches for better branch prediction
* Update Modules/_asynciomodule.c
* Apply suggestions from code review
Co-authored-by: Itamar Oren <itamarost@gmail.com>
* fix capturing of eager tasks
* add comment to task finalization
* fix tests and couple c implmentation to c task
improved linked-list logic and more comments
* fix test
---------
Co-authored-by: Itamar Oren <itamarost@gmail.com>
The tests are now passed with the current version of Tcl/Tk under
development (8.7b1+ and 9.0b3+).
The following changes were also made to make the tests more flexible:
* Helper methods like checkParam() now interpret the expected error message
as a regular expression instead of a literal.
* Add support of new arguments in checkEnumParam():
- allow_empty=True skips testing with empty string;
- fullname= specifies the name for error message if it differs from the
option name;
- sort=True sorts values for error message.
* Add support of the allow_empty argument in checkReliefParam():
allow_empty=True adds an empty string to the list of accepted values.
* Attributes _clip_highlightthickness, _clip_pad and _clip_borderwidth
specify how negative values of options -highlightthickness, -padx, -pady
and -borderwidth are handled.
* Use global variables for some common error messages.
Co-authored-by: Terry Jan Reedy <tjreedy@udel.edu>
* Add an InternalDocs file describing how interning should work and how to use it.
* Add internal functions to *explicitly* request what kind of interning is done:
- `_PyUnicode_InternMortal`
- `_PyUnicode_InternImmortal`
- `_PyUnicode_InternStatic`
* Switch uses of `PyUnicode_InternInPlace` to those.
* Disallow using `_Py_SetImmortal` on strings directly.
You should use `_PyUnicode_InternImmortal` instead:
- Strings should be interned before immortalization, otherwise you're possibly
interning a immortalizing copy.
- `_Py_SetImmortal` doesn't handle the `SSTATE_INTERNED_MORTAL` to
`SSTATE_INTERNED_IMMORTAL` update, and those flags can't be changed in
backports, as they are now part of public API and version-specific ABI.
* Add private `_only_immortal` argument for `sys.getunicodeinternedsize`, used in refleak test machinery.
* Make sure the statically allocated string singletons are unique. This means these sets are now disjoint:
- `_Py_ID`
- `_Py_STR` (including the empty string)
- one-character latin-1 singletons
Now, when you intern a singleton, that exact singleton will be interned.
* Add a `_Py_LATIN1_CHR` macro, use it instead of `_Py_ID`/`_Py_STR` for one-character latin-1 singletons everywhere (including Clinic).
* Intern `_Py_STR` singletons at startup.
* For free-threaded builds, intern `_Py_LATIN1_CHR` singletons at startup.
* Beef up the tests. Cover internal details (marked with `@cpython_only`).
* Add lots of assertions
Co-Authored-By: Eric Snow <ericsnowcurrently@gmail.com>
Remove SafeChildWatcher, FastChildWatcher and MultiLoopChildWatcher from asyncio. These child watchers have been deprecated since Python 3.12. The tests are also removed and some more tests will be added after the rewrite of child watchers.
If the child process takes longer than SHORT_TIMEOUT seconds to
complete, kill the process but then wait until it completes with no
timeout to not leak child processes.
Add support for not following symlinks in `pathlib.Path.copy()`.
On Windows we add the `COPY_FILE_COPY_SYMLINK` flag is following symlinks is disabled. If the source is symlink to a directory, this call will fail with `ERROR_ACCESS_DENIED`. In this case we add `COPY_FILE_DIRECTORY` to the flags and retry. This can fail on old Windowses, which we note in the docs.
No news as `copy()` was only just added.
In preparation for the addition of `PathBase.rmtree()`, implement
`DummyPath.unlink()` and `rmdir()`, and move corresponding tests into
`test_pathlib_abc` so they're run against `DummyPath`.
Preparatory work for moving `_rmtree_unsafe()` and `_rmtree_safe_fd()` to
`pathlib._os` so that they can be used from both `shutil` and `pathlib`.
Move implementation-specific setup from `rmtree()` into the safe/unsafe
functions, and give them the same signature `(path, dir_fd, onexc)`.
In the tests, mock `os.open` rather than `_rmtree_safe_fd()` to ensure the
FD-based walk is used, and replace a couple references to
`shutil._use_fd_functions` with `shutil.rmtree.avoids_symlink_attacks`
(which has the same value).
No change of behaviour.
This exposes `PyUnstable_Object_ClearWeakRefsNoCallbacks` as an unstable
C-API function to provide a thread-safe mechanism for clearing weakrefs
without executing callbacks.
Some C-API extensions need to clear weakrefs without calling callbacks,
such as after running finalizers like we do in subtype_dealloc.
Previously they could use `_PyWeakref_ClearRef` on each weakref, but
that's not thread-safe in the free-threaded build.
Co-authored-by: Petr Viktorin <encukou@gmail.com>
The `inspect.ismethoddescriptor()` function did not check for the lack of
`__delete__()` and, consequently, erroneously returned True when applied
to *data* descriptors with only `__get__()` and `__delete__()` defined.
Co-authored-by: Bénédikt Tran <10796600+picnixz@users.noreply.github.com>
Co-authored-by: Alyssa Coghlan <ncoghlan@gmail.com>
Mocking only works if sys.modules['pydoc'] and pydoc are the same,
but some pydoc functions reload the module and change sys.modules.
Ensure that sys.modules['pydoc'] is always restored after the corresponding
tests.
This matches the output behavior in 3.10 and earlier; the optimization in 3.11 allowed the zlib library's "os" value to be filled in instead in the circumstance when mtime was 0. this keeps things consistent.
In gh-120009 I used an atexit hook to finalize the _datetime module's static types at interpreter shutdown. However, atexit hooks are executed very early in finalization, which is a problem in the few cases where a subclass of one of those static types is still alive until the final GC collection. The static builtin types don't have this probably because they are finalized toward the end, after the final GC collection. To avoid the problem for _datetime, I have applied a similar approach here.
Also, credit goes to @mgorny and @neonene for the new tests.
FYI, I would have liked to take a slightly cleaner approach with managed static types, but wanted to get a smaller fix in first for the sake of backporting. I'll circle back to the cleaner approach with a future change on the main branch.
Add __all__ to the following modules:
importlib.machinery, importlib.util and xml.sax.
Add also "# noqa: F401" in collections.abc,
subprocess and xml.sax.
* Sort __all__; remove collections.abc.__all__; remove private names
* Add tests
Add a `Path.copy()` method that copies the content of one file to another.
This method is similar to `shutil.copyfile()` but differs in the following ways:
- Uses `fcntl.FICLONE` where available (see GH-81338)
- Uses `os.copy_file_range` where available (see GH-81340)
- Uses `_winapi.CopyFile2` where available, even though this copies more metadata than the other implementations. This makes `WindowsPath.copy()` more similar to `shutil.copy2()`.
The method is presently _less_ specified than the `shutil` functions to allow OS-specific optimizations that might copy more or less metadata.
Incorporates code from GH-81338 and GH-93152.
Co-authored-by: Eryk Sun <eryksun@gmail.com>
Tools such as ruff can ignore "imported but unused" warnings if a
line ends with "# noqa: F401". It avoids the temptation to remove
an import which is used effectively.
Remove wheeldata from both sides of the `assertEqual`, so that we're
*actually* ignoring it from the test set.
This test is only making assertions about the source tree, no code is
being executed that would do anything different based on the value of
`WHEEL_PKG_DIR`.
The _strptime module object was cached in a static local variable (in the datetime.strptime() implementation). That's a problem when it crosses isolation boundaries, such as reinitializing the runtme or between interpreters. This change fixes the problem by dropping the static variable, instead always relying on the normal sys.modules cache (via PyImport_Import()).
* Remove pyrepl's optimization for self-insert
This will be replaced by a less specialized optimization.
* Use line-buffering when pyrepl echoes pastes
Previously echoing was totally suppressed until the entire command had
been pasted and the terminal ended paste mode, but this gives the user
no feedback to indicate that an operation is in progress. Drawing
something to the screen once per line strikes a balance between
perceived responsiveness and performance.
* Remove dead code from pyrepl
`msg_at_bottom` is always true.
* Speed up pyrepl's screen rendering computation
The Reader in pyrepl doesn't hold a complete representation of the
screen area being drawn as persistent state. Instead, it recomputes it,
on each keypress. This is fast enough for a few hundred bytes, but
incredibly slow as the input buffer grows into the kilobytes (likely
because of pasting).
Rather than making some expensive and expansive changes to the repl's
internal representation of the screen, add some caching: remember some
data from one refresh to the next about what was drawn to the screen
and, if we don't find anything that has invalidated the results that
were computed last time around, reuse them. To keep this caching as
simple as possible, all we'll do is look for lines in the buffer that
were above the cursor the last time we were asked to update the screen,
and that are still above the cursor now. We assume that nothing can
affect a line that comes before both the old and new cursor location
without us being informed. Based on this assumption, we can reuse old
lines, which drastically speeds up the overwhelmingly common case where
the user is typing near the end of the buffer.
* Speed up pyrepl prompt drawing
Cache the `can_colorize()` call rather than repeatedly recomputing it.
This call looks up an environment variable, and is called once per
character typed at the REPL. The environment variable lookup shows up as
a hot spot when profiling, and we don't expect this to change while the
REPL is running.
* Speed up pasting multiple lines into the REPL
Previously, we were checking whether the command should be accepted each
time a line break was encountered, but that's not the expected behavior.
In bracketed paste mode, we expect everything pasted to be part of
a single block of code, and encountering a newline shouldn't behave like
a user pressing <Enter> to execute a command. The user should always
have a chance to review the pasted command before running it.
* Use a read buffer for input in pyrepl
Previously we were reading one byte at a time, which causes much slower
IO than necessary. Instead, read in chunks, processing previously read
data before asking for more.
* Optimize finding width of a single character
`wlen` finds the width of a multi-character string by adding up the
width of each character, and then subtracting the width of any escape
sequences. It's often called for single character strings, however,
which can't possibly contain escape sequences. Optimize for that case.
* Optimize disp_str for ASCII characters
Since every ASCII character is known to display as single width, we can
avoid not only the Unicode data lookup in `disp_str` but also the one
hidden in `str_width` for them.
* Speed up cursor movements in long pyrepl commands
When the current pyrepl command buffer contains many lines, scrolling up
becomes slow. We have optimizations in place to reuse lines above the
cursor position from one refresh to the next, but don't currently try to
reuse lines below the cursor position in the same way, so we wind up
with quadratic behavior where all lines of the buffer below the cursor
are recomputed each time the cursor moves up another line.
Optimize this by only computing one screen's worth of lines beyond the
cursor position. Any lines beyond that can't possibly be shown by the
console, and bounding this makes scrolling up have linear time
complexity instead.
---------
Signed-off-by: Matt Wozniski <mwozniski@bloomberg.net>
Co-authored-by: Pablo Galindo <pablogsal@gmail.com>
In order to patch flask.g e.g. as in #84982, that
proxies getattr must not be invoked. For that,
mock must not try to read from the original
object. In some cases that is unavoidable, e.g.
when doing autospec. However, patch("flask.g",
new_callable=MagicMock) should be entirely safe.
If the Helper() class was initialized with an output, the topics, keywords
and symbols help still use the pager instead of the output.
Change the behavior so the output is used if available while keeping the
previous behavior if no output was configured.
The tests were accidentally disabled by 2da0dc0, which didn't handle classes correctly.
I considered updating no_rerun() to support classes, but the way test_datetime.py works would have made things fairly messy. Plus, it looks like the refleaks we had encountered before have been resolved.
In `glob._Globber`, move pathlib-specific methods to `pathlib._abc.PathGlobber` and replace them with abstract methods. Rename `glob._Globber` to `glob._GlobberBase`. As a result, the `glob` module is no longer befouled by code that can only ever apply to pathlib.
No change of behaviour.
Adjust DeprecationWarning when testing element truth values in ElementTree, we're planning to go with the more natural True return rather than a disruptive harder to code around exception raise, and are deferring the behavior change for a few more releases.
Add private `posixpath._realpath()` function, which is a generic version of `realpath()` that can be parameterised with string tokens (`sep`, `curdir`, `pardir`) and query functions (`getcwd`, `lstat`, `readlink`). Also add support for limiting the number of symlink traversals.
In the private `pathlib._abc.PathBase` class, call `posixpath._realpath()` and remove our re-implementation of the same algorithm.
No change to any public APIs, either in `posixpath` or `pathlib`.
Co-authored-by: Nice Zombies <nineteendo19d0@gmail.com>
Some time strings that contain fractional hours or minutes are permitted
by ISO 8601, but such strings are very unlikely to be intentional. The
current parser does not parse such strings correctly or raise an error.
This change raises a ValueError when hours or minutes contain a decimal mark.
Co-authored-by: blurb-it[bot] <43283697+blurb-it[bot]@users.noreply.github.com>
This matches the default GIL switch interval. It greatly speeds up the
free-threaded build: previously, it spent nearly all its time in
`gc.collect()`.
The `test_imaplib` was taking 40+ minutes in the refleak build bots because
the tests waiting on a client `self._setup()` was creating a client that
prevented progress until its connection timed out, which scaled with the
global timeout.
We should set `connect=False` for the tests that don't want `_setup()` to
create a client.
Co-authored-by: Serhiy Storchaka <storchaka@gmail.com>
The process is expected to time out. In the refleak builds,
`support.SHORT_TIMEOUT` is often five minutes and we run the tests six
times, so test_signal was taking >30 minutes.
The free-threaded build currently immortalizes objects that use deferred
reference counting (see gh-117783). This typically happens once the
first non-main thread is created, but the behavior can be suppressed for
tests, in subinterpreters, or during a compile() call.
This fixes a race condition involving the tracking of whether the
behavior is suppressed.
Sergey B Kirpichev
Barney Gale
Kevin Evans
blhsing
Mark Shannon
Bar Harel
Donghee Na
Bénédikt Tran
mpage
Irit Katriel
Wulian
Serhiy Storchaka
Peter Bierma
Jeremy Hylton
James
Pablo Galindo Salgado
Pedro Lacerda
sobolevn
CF Bolz-Tereick
Sam Gross
Kirill Podoprigora
Cody Maloney
Mateusz Nowak
Malcolm Smith
Jason R. Coombs
Eddie Elizondo
Jonathan Protzenko
Eric Snow
Trey Hunner
Jelle Zijlstra
Xie Yanbo
Victor Stinner
Nico Mexis
Petr Viktorin
neonene
Tian Gao
Erlend E. Aasland
Hugo van Kemenade
Ali Hamdan
pukkandan
Pradyun Gedam
Виталий Дмитриев
John Riggles
Jonathon Vandezande
jianghuyiyuan
Russell Keith-Magee
Terry Jan Reedy
Michael Droettboom
Łukasz Langa
Dino Viehland
Clinton
Seth Michael Larson
WilliamRoyNelson
AN Long
Mikołaj Kuranowski
Sebastian Rittau
HarryLHW
Bradley Reynolds
Karolina Surma
Rodrigo Oliveira
Matthieu Caneill
Gregor
Alex Waygood
Michael
Andreas Stocker
Lysandros Nikolaou
Marta Gómez Macías
saucoide
Milan Oberkirch
Rodrigo Girão Serrão
Dominic H
Kumar Aditya
Tomas R
Bas Bloemsaat
Zachary Ware
Ken Jin
Alyssa Coghlan
Chris Markiewicz
Max Muoto
Steve Dower
Craig Robson
Janek Nouvertné
devdanzin
Itamar Oren
Wulian233
Nice Zombies
Jan Kaliszewski
Daniele Parmeggiani
Steele Farnsworth
Idan Kapustian
Ruben Vorderman
Raymond Hettinger
Stefano Rivera
Eugene Triguba
Matt Wozniski
Robert Collins
Pieter Eendebak
E. M. Bray
Carl Meyer
Saul Shanabrook
Enrico Tröger
Xi Ruoyao
Jacob Walls
benchatt
Vinay Sajip
Kaundur
Daniel Hollas
Radislav Chugunov