* Fix test_concurrent_futures to actually test what it says.
Many ProcessPoolExecutor based tests were ignoring the mp_context
and using the default instead. This meant we lacked proper test
coverage of all of them.
Also removes the old _prime_executor() worker delay seeding code
as it appears to have no point and causes 20-30 seconds extra
latency on this already long test. It also interfered with some
of the refactoring to fix the above to not needlessly create their
own executor when setUp has already created an appropriate one.
* Don't import the name from multiprocessing directly to avoid confusion.
* 📜🤖 Added by blurb_it.
Co-authored-by: blurb-it[bot] <43283697+blurb-it[bot]@users.noreply.github.com>
Based on suggestions by Guido van Rossum, Spencer Brown, and Alex Waygood.
Co-authored-by: Alex Waygood <Alex.Waygood@Gmail.com>
Co-authored-by: Guido van Rossum <gvanrossum@gmail.com>
Co-authored-by: Ken Jin <kenjin4096@gmail.com>
* fix the comparison of character and integer by using ord()
* 📜🤖 Added by blurb_it.
Co-authored-by: blurb-it[bot] <43283697+blurb-it[bot]@users.noreply.github.com>
* revise the first paragraph of docs for os.path
* add a mention of `os.PathLike` protocol
* remove warnings rendered irrelevant by :pep:`383` and :pep:`529`
Co-authored-by: Jelle Zijlstra <jelle.zijlstra@gmail.com>
Fix#91487
When transferring a small file, e.g. 256 KiB, the speed of this PR is comparable. However, if a large file, e.g. 65536 KiB, is transferred, asyncio UDP will be over 100 times faster than the original. The speed is presumably significantly faster if a larger file is transferred, e.g. 1048576 KiB.
Automerge-Triggered-By: GH:gpshead
The existing event loop `start_tls()` method is not sufficient for
connections using the streams API. The existing StreamReader works
because the new transport passes received data to the original protocol.
The StreamWriter must then write data to the new transport, and the
StreamReaderProtocol must be updated to close the new transport
correctly.
The new StreamWriter `start_tls()` updates itself and the reader
protocol to the new SSL transport.
Co-authored-by: Ian Good <icgood@gmail.com>
Fix an uninitialized bool in exception print context.
`struct exception_print_context.need_close` was uninitialized.
Found by oss-fuzz in a test case running under the undefined behavior sanitizer.
https://oss-fuzz.com/testcase-detail/6217746058182656
```
Python/pythonrun.c:1241:28: runtime error: load of value 253, which is not a valid value for type 'bool'
#0 0xbf2203 in print_chained cpython3/Python/pythonrun.c:1241:28
#1 0xbea4bb in print_exception_cause_and_context cpython3/Python/pythonrun.c:1320:19
#2 0xbea4bb in print_exception_recursive cpython3/Python/pythonrun.c:1470:13
#3 0xbe9e39 in _PyErr_Display cpython3/Python/pythonrun.c:1517:9
```
Pretty obvious what the ommission was upon code inspection.
* Deprecate imghdr
Co-authored-by: Hugo van Kemenade <hugovk@users.noreply.github.com>
* Update Doc/whatsnew/3.11.rst
Co-authored-by: Hugo van Kemenade <hugovk@users.noreply.github.com>
* Inline `imghdr` into `email.mime.image`
Co-authored-by: Hugo van Kemenade <hugovk@users.noreply.github.com>
Co-authored-by: Barry Warsaw <barry@python.org>
The left-hand side expression of the if-check can be converted to a
constant by the compiler, but the addition on the right-hand side is
performed during runtime.
Move the addition from the right-hand side to the left-hand side by
turning it into a subtraction there. Since the values are known to
be large enough to not turn negative, this is a safe operation.
Prevents a very unlikely integer overflow on 32 bit systems.
Fixes GH-91421.
I talked to @davidfstr and I offered to implement the runtime part of PEP 655
to make sure we can get it in before the feature freeze. We're going to defer
the documentation to a separate PR, because it can wait until after the feature
freeze.
The runtime implementation conveniently already exists in typing-extensions,
so I largely copied that.
Co-authored-by: David Foster <david@dafoster.net>
* Move the code for generating Modules/_sre/sre_constants.h from
Lib/re/_constants.py into a separate script
Tools/scripts/generate_sre_constants.py.
* Add target `regen-sre` in the makefile.
* Make target `regen-all` depending on `regen-sre`.
_Py_closerange() currently assumes that close_range() closes
all file descriptors even if it returns an error (other than ENOSYS).
This assumption can be wrong on Linux if a seccomp sandbox denies
the underlying syscall, pretending that it returns EPERM or EACCES.
In this case _Py_closerange() won't close any descriptors at all,
which in the worst case can be a security issue.
Fix this by falling back to other methods in case of any close_range()
error. Note that fallbacks will not be triggered on any problems with
closing individual file descriptors because close_range() is documented
to ignore such errors on both Linux[1] and FreeBSD[2].
[1] https://man7.org/linux/man-pages/man2/close_range.2.html
[2] https://www.freebsd.org/cgi/man.cgi?query=close_range&sektion=2
Remove the Include/code.h header file. C extensions should only
include the main <Python.h> header file.
Python.h includes directly Include/cpython/code.h instead.
Copying and pickling instances of subclasses of builtin types
bytearray, set, frozenset, collections.OrderedDict, collections.deque,
weakref.WeakSet, and datetime.tzinfo now copies and pickles instance attributes
implemented as slots.
The side effect of this bug was that venv environments directly
used the main interpreter instead of the intermediate stub executable,
which can cause problems when a script uses system APIs that
require the use of an application bundle.
Move the private _PyFrameEvalFunction type, and private
_PyInterpreterState_GetEvalFrameFunc() and
_PyInterpreterState_SetEvalFrameFunc() functions to the internal C
API. The _PyFrameEvalFunction callback function type now uses the
_PyInterpreterFrame type which is part of the internal C API.
Update the _PyFrameEvalFunction documentation.
Move the private undocumented _PyEval_EvalFrameDefault() function to
the internal C API. The function now uses the _PyInterpreterFrame
type which is part of the internal C API.
bpo-47151: Fallback to fork when vfork fails in subprocess. An OS kernel can specifically decide to disallow vfork() in a process. No need for that to prevent us from launching subprocesses.
Update documentation to note that in some circumstances,
KeyboardInterrupt may cause code to enter an inconsistent state. Also
document sample workaround to avoid KeyboardInterrupt, if needed.
In rare cases, capturing group could get wrong result.
Regular expression engines in Perl and Java have similar bugs.
The new behavior now matches the behavior of more modern
RE engines: in the regex module and in PHP, Ruby and Node.js.
* `PyFrame_FastToLocalsWithError` and `PyFrame_LocalsToFast` are no longer called during profile and tracing.
(Contributed by Fabio Zadrozny)
* Make accesses to a frame's `f_locals` safe from C code, not relying on calls to `PyFrame_FastToLocals` or `PyFrame_LocalsToFast`.
* Document new `PyFrame_GetLocals` C-API function.
Shared extension on Emscripten now have suffix
``.cpython-311-wasm32-emscripten.so`` (JS loader) and
``.cpython-311-wasm32-emscripten.wasm`` (WebAssembly code).
After a long deliberation we ended up feeling that the message argument for Future.cancel(), added in 3.9, was a bad idea, so we're deprecating it in 3.11 and plan to remove it in 3.13.
- Add requires_fork and requires_subprocess to more tests
- Skip extension import tests if dlopen is not available
- Don't assume that _testcapi is a shared extension
- Skip a lot of socket tests that don't work on Emscripten
- Skip mmap tests, mmap emulation is incomplete
- venv does not work yet
- Cannot get libc from executable
The "entire" test suite is now passing on Emscripten with EMSDK from git head (91 suites are skipped).
* Add the metadata_encoding parameter in the zipfile.ZipFile constructor.
* Add the --metadata-encoding option in the zipfile CLI.
Co-authored-by: Stephen J. Turnbull <stephen@xemacs.org>
* Moves the bytecode to the end of the corresponding PyCodeObject, and quickens it in-place.
* Removes the almost-always-unused co_varnames, co_freevars, and co_cellvars member caches
* _PyOpcode_Deopt is a new mapping from all opcodes to their un-quickened forms.
* _PyOpcode_InlineCacheEntries is renamed to _PyOpcode_Caches
* _Py_IncrementCountAndMaybeQuicken is renamed to _PyCode_Warmup
* _Py_Quicken is renamed to _PyCode_Quicken
* _co_quickened is renamed to _co_code_adaptive (and is now a read-only memoryview).
* Do not emit unused nonzero opargs anymore in the compiler.
Remove the private undocumented function
_PyEval_GetCoroutineOriginTrackingDepth() from the C API. Call the
public sys.get_coroutine_origin_tracking_depth() function instead.
Change the internal function
_PyEval_SetCoroutineOriginTrackingDepth():
* Remove the 'tstate' parameter;
* Add return value and raises an exception if depth is negative;
* No longer export the function: call the public
sys.set_coroutine_origin_tracking_depth() function instead.
Uniformize also function declarations in pycore_ceval.h.
Remove the following private undocumented functions from the C API:
* _PyEval_GetAsyncGenFirstiter()
* _PyEval_GetAsyncGenFinalizer()
* _PyEval_SetAsyncGenFirstiter()
* _PyEval_SetAsyncGenFinalizer()
Call the public sys.get_asyncgen_hooks() and sys.set_asyncgen_hooks()
functions instead.
When compiled with `USE_ZLIB_CRC32` defined (`configure` sets this on POSIX systems), `binascii.crc32(...)` failed to compute the correct value when the input data was >= 4GiB. Because the zlib crc32 API is limited to a 32-bit length.
This lines it up with the `zlib.crc32(...)` implementation that doesn't have that flaw.
**Performance:** This also adopts the same GIL releasing for larger inputs logic that `zlib.crc32` has, and causes the Windows build to always use zlib's crc32 instead of our slow C code as zlib is a required build dependency on Windows.
The `_SharedFile` tracks its own virtual position into the file as
`self._pos` and updates it after reading or seeking. `tell()` should
return this position instead of calling into the underlying file object,
since if multiple `_SharedFile` instances are being used concurrently on
the same file, another one may have moved the real file position.
Additionally, calling into the underlying `tell` may expose thread
safety issues in the underlying file object because it was called
without taking the lock.
Clarifies a versionchanged note on crc32 & adler32 docs that the workaround is only needed for Python 2 and earlier.
Also cleans up an unnecessary intermediate variable in the implementation.
Authored-By: Ma Lin / animalize
Co-authored-by: Gregory P. Smith <greg@krypto.org>
Define *posix_venv* and *nt_venv* sysconfig installation schemes
to be used for bootstrapping new virtual environments.
Add *venv* sysconfig installation scheme to get the appropriate one of the above.
The schemes are identical to the pre-existing
*posix_prefix* and *nt* install schemes.
The venv module now uses the *venv* scheme to create new virtual environments
instead of hardcoding the paths depending only on the platform. Downstream
Python distributors customizing the *posix_prefix* or *nt* install
scheme in a way that is not compatible with the install scheme used in
virtual environments are encouraged not to customize the *venv* schemes.
When Python itself runs in a virtual environment,
sysconfig.get_default_scheme and
sysconfig.get_preferred_scheme with `key="prefix"` returns
*venv*.
* Improve exception compliance with PEP 249
* Raise InterfaceError instead of ProgrammingError for SQLITE_MISUSE.
If SQLITE_MISUSE is raised, it is a sqlite3 module bug. Users of the
sqlite3 module are not responsible for using the SQLite C API correctly.
* Don't overwrite BufferError with ValueError when conversion to BLOB fails.
* Raise ProgrammingError instead of Warning if user tries to execute() more
than one SQL statement.
* Raise ProgrammingError instead of ValueError if an SQL query contains null characters.
* Make sure `_pysqlite_set_result` raises an exception if it returns -1.
wasm32-emscripten does not support exceptfds and requires NULL. Python
now passes NULL instead of a fdset pointer when the input list is empty.
This works fine on all platforms and might even be a tiny bit faster.
Specifically, prepare for starring of tuples via a new genericalias iter type. GenericAlias also partially supports the iterator protocol after this change.
Co-authored-by: Jelle Zijlstra <jelle.zijlstra@gmail.com>
Co-authored-by: Kumar Aditya <59607654+kumaraditya303@users.noreply.github.com>
Co-authored-by: Ken Jin <28750310+Fidget-Spinner@users.noreply.github.com>
Add new functions to pack and unpack C double (serialize and
deserialize):
* PyFloat_Pack2(), PyFloat_Pack4(), PyFloat_Pack8()
* PyFloat_Unpack2(), PyFloat_Unpack4(), PyFloat_Unpack8()
Document these functions and add unit tests.
Rename private functions and move them from the internal C API
to the public C API:
* _PyFloat_Pack2() => PyFloat_Pack2()
* _PyFloat_Pack4() => PyFloat_Pack4()
* _PyFloat_Pack8() => PyFloat_Pack8()
* _PyFloat_Unpack2() => PyFloat_Unpack2()
* _PyFloat_Unpack4() => PyFloat_Unpack4()
* _PyFloat_Unpack8() => PyFloat_Unpack8()
Replace the "unsigned char*" type with "char*" which is more common
and easy to use.
In Linux kernel 5.14 one can dynamically request size of altstacksize
based on hardware capabilities with getauxval(AT_MINSIGSTKSZ).
This changes allows for Python extension's request to Linux kernel
to use AMX_TILE instruction set on Sapphire Rapids Xeon processor
to succeed, unblocking use of the ISA in frameworks.
Introduced HAVE_LINUX_AUXVEC_H in configure.ac and pyconfig.h.in
Used cpython_autoconf:269 docker container to generate configure.
Example:
async with asyncio.timeout(5):
await some_task()
Will interrupt the await and raise TimeoutError if some_task() takes longer than 5 seconds.
Co-authored-by: Guido van Rossum <guido@python.org>
Irit Katriel
Gregory P. Smith
Sebastian Rittau
Shantanu
Jelle Zijlstra
Yu Liu
Jack DeVries
Erlend Egeberg Aasland
msoxzw
Brett Cannon
Brandt Bucher
Oleg Iarygin
Hood Chatham
Inada Naoki
Pieter Eendebak
David Foster
Pablo Galindo Salgado
Dustin Rodrigues
Tobias Stoeckmann
Serhiy Storchaka
Ilya Leoshkevich
Dennis Sweeney
Mark Shannon
Adrian Garcia Badaracco
John Belmonte
Alexey Izbyshev
Petr Viktorin
Christian Heimes
Victor Stinner
Steve Dower
Ken Jin
Sam Ezeh
Hugo van Kemenade
Matthieu Dartiailh
Ronald Oussoren
rtobar
yyyyyyyan
180909
Jason R. Coombs
Ma Lin
Michał D
Zachary Ware
Andrew Svetlov
Dong-hee Na
Daniël van Noord
Dave Goncalves
benfogle
Vincent Bernat
Maciej Górski
ty
Kumar Aditya
Jouke Witteveen
Matthew Rahtz
Duprat
Russel Webber
penguin_wwy
Kevin Mehall
Arie Bovenberg
Miro Hrončok
Bader Zaidan
David CARLIER
Carlos Damazio
Mariusz Felisiak
Maximilian Hils
Michael P. Nitowski
Ned Deily
Crowthebird
Pradyun Gedam
Alex Grönholm
Gregory Beauregard
Oleksandr Pavlyk