The side effect of this bug was that venv environments directly
used the main interpreter instead of the intermediate stub executable,
which can cause problems when a script uses system APIs that
require the use of an application bundle.
In rare cases, capturing group could get wrong result.
Regular expression engines in Perl and Java have similar bugs.
The new behavior now matches the behavior of more modern
RE engines: in the regex module and in PHP, Ruby and Node.js.
After a long deliberation we ended up feeling that the message argument for Future.cancel(), added in 3.9, was a bad idea, so we're deprecating it in 3.11 and plan to remove it in 3.13.
- Add requires_fork and requires_subprocess to more tests
- Skip extension import tests if dlopen is not available
- Don't assume that _testcapi is a shared extension
- Skip a lot of socket tests that don't work on Emscripten
- Skip mmap tests, mmap emulation is incomplete
- venv does not work yet
- Cannot get libc from executable
The "entire" test suite is now passing on Emscripten with EMSDK from git head (91 suites are skipped).
* Add the metadata_encoding parameter in the zipfile.ZipFile constructor.
* Add the --metadata-encoding option in the zipfile CLI.
Co-authored-by: Stephen J. Turnbull <stephen@xemacs.org>
Replace the child process `typeperf.exe` with a daemon thread that reads the performance counters directly. This prevents the issues that arise from inherited handles in grandchild processes (see issue37531 for discussion).
We only use the load tracker when running tests in multiprocess mode. This prevents inadvertent interactions with tests expecting a single threaded environment. Displaying load is really only helpful for buildbots running in multiprocess mode anyway.
* Moves the bytecode to the end of the corresponding PyCodeObject, and quickens it in-place.
* Removes the almost-always-unused co_varnames, co_freevars, and co_cellvars member caches
* _PyOpcode_Deopt is a new mapping from all opcodes to their un-quickened forms.
* _PyOpcode_InlineCacheEntries is renamed to _PyOpcode_Caches
* _Py_IncrementCountAndMaybeQuicken is renamed to _PyCode_Warmup
* _Py_Quicken is renamed to _PyCode_Quicken
* _co_quickened is renamed to _co_code_adaptive (and is now a read-only memoryview).
* Do not emit unused nonzero opargs anymore in the compiler.
When compiled with `USE_ZLIB_CRC32` defined (`configure` sets this on POSIX systems), `binascii.crc32(...)` failed to compute the correct value when the input data was >= 4GiB. Because the zlib crc32 API is limited to a 32-bit length.
This lines it up with the `zlib.crc32(...)` implementation that doesn't have that flaw.
**Performance:** This also adopts the same GIL releasing for larger inputs logic that `zlib.crc32` has, and causes the Windows build to always use zlib's crc32 instead of our slow C code as zlib is a required build dependency on Windows.
Define *posix_venv* and *nt_venv* sysconfig installation schemes
to be used for bootstrapping new virtual environments.
Add *venv* sysconfig installation scheme to get the appropriate one of the above.
The schemes are identical to the pre-existing
*posix_prefix* and *nt* install schemes.
The venv module now uses the *venv* scheme to create new virtual environments
instead of hardcoding the paths depending only on the platform. Downstream
Python distributors customizing the *posix_prefix* or *nt* install
scheme in a way that is not compatible with the install scheme used in
virtual environments are encouraged not to customize the *venv* schemes.
When Python itself runs in a virtual environment,
sysconfig.get_default_scheme and
sysconfig.get_preferred_scheme with `key="prefix"` returns
*venv*.
Previously it tested that that the actual output contains every non-whitespace
character from the expected output (ignoring order and repetitions).
Now it will test that the actual output contains the same lines as the expected
output, in the same order, ignoring indentation and empty lines.
* Improve exception compliance with PEP 249
* Raise InterfaceError instead of ProgrammingError for SQLITE_MISUSE.
If SQLITE_MISUSE is raised, it is a sqlite3 module bug. Users of the
sqlite3 module are not responsible for using the SQLite C API correctly.
* Don't overwrite BufferError with ValueError when conversion to BLOB fails.
* Raise ProgrammingError instead of Warning if user tries to execute() more
than one SQL statement.
* Raise ProgrammingError instead of ValueError if an SQL query contains null characters.
* Make sure `_pysqlite_set_result` raises an exception if it returns -1.
wasm32-emscripten does not support exceptfds and requires NULL. Python
now passes NULL instead of a fdset pointer when the input list is empty.
This works fine on all platforms and might even be a tiny bit faster.
Specifically, prepare for starring of tuples via a new genericalias iter type. GenericAlias also partially supports the iterator protocol after this change.
Co-authored-by: Jelle Zijlstra <jelle.zijlstra@gmail.com>
Co-authored-by: Kumar Aditya <59607654+kumaraditya303@users.noreply.github.com>
Co-authored-by: Ken Jin <28750310+Fidget-Spinner@users.noreply.github.com>
Add new functions to pack and unpack C double (serialize and
deserialize):
* PyFloat_Pack2(), PyFloat_Pack4(), PyFloat_Pack8()
* PyFloat_Unpack2(), PyFloat_Unpack4(), PyFloat_Unpack8()
Document these functions and add unit tests.
Rename private functions and move them from the internal C API
to the public C API:
* _PyFloat_Pack2() => PyFloat_Pack2()
* _PyFloat_Pack4() => PyFloat_Pack4()
* _PyFloat_Pack8() => PyFloat_Pack8()
* _PyFloat_Unpack2() => PyFloat_Unpack2()
* _PyFloat_Unpack4() => PyFloat_Unpack4()
* _PyFloat_Unpack8() => PyFloat_Unpack8()
Replace the "unsigned char*" type with "char*" which is more common
and easy to use.
Add methods __typing_subst__() in TypeVar and ParamSpec.
Simplify code by using more object-oriented approach, especially
the C code for types.GenericAlias and the Python code for
collections.abc.Callable.
GH-30297 removed a duplicate `from test import support` statement from `test_asyncio.test_sslproto`. However, in between that PR being filed and it being merged, GH-31275 removed the _other_ `from test import support` statement. This means that `support` is now undefined in `test_asyncio.test_sslproto`, causing the CI to fail on all platforms for all PRS.
Example:
async with asyncio.timeout(5):
await some_task()
Will interrupt the await and raise TimeoutError if some_task() takes longer than 5 seconds.
Co-authored-by: Guido van Rossum <guido@python.org>
GH-26091 added the _typevar_types and _paramspec_tvars instance
variables to _GenericAlias. However, they were not propagated
consistently. This commit addresses the most prominent deficiency
identified in bpo-46581 (namely their absence from
_GenericAlias.copy_with), but there could be others.
Co-authored-by: Ken Jin <28750310+Fidget-Spinner@users.noreply.github.com>
Co-authored-by: Serhiy Storchaka <storchaka@gmail.com>
- fd inheritance can't be modified because Emscripten doesn't support subprocesses anyway.
- setpriority always fails
- geteuid no longer causes problems with latest emsdk
- umask is a stub
- geteuid / getuid always return 0, but process cannot chown to random uid.
- getgroups always fails.
- geteuid and getegid always return 0 (root), which confuse tarfile and
tests.
- hardlinks (link, linkat) always fails.
- non-encodable file names are not supported by NODERAWFS layer.
- mark more tests with dependency on subprocess and multiprocessing.
Mocking does not work if the module fails to import.
This adds a new standard library module, `tomllib`, for parsing TOML.
The implementation is based on Tomli (https://github.com/hukkin/tomli).
## Steps taken (converting `tomli` to `tomllib`)
- Move everything in `tomli:src/tomli` to `Lib/tomllib`. Exclude `py.typed`.
- Remove `__version__ = ...` line from `Lib/tomllib/__init__.py`
- Move everything in `tomli:tests` to `Lib/test/test_tomllib`. Exclude the following test data dirs recursively:
- `tomli:tests/data/invalid/_external/`
- `tomli:tests/data/valid/_external/`
- Create `Lib/test/test_tomllib/__main__.py`:
```python
import unittest
from . import load_tests
unittest.main()
```
- Add the following to `Lib/test/test_tomllib/__init__.py`:
```python
import os
from test.support import load_package_tests
def load_tests(*args):
return load_package_tests(os.path.dirname(__file__), *args)
```
Also change `import tomli as tomllib` to `import tomllib`.
- In `cpython/Lib/tomllib/_parser.py` replace `__fp` with `fp` and `__s` with
`s`. Add the `/` to `load` and `loads` function signatures.
- Run `make regen-stdlib-module-names`
- Create `Doc/library/tomllib.rst` and reference it in `Doc/library/fileformats.rst`
Co-authored-by: Alex Waygood <alex.waygood@gmail.com>
Co-authored-by: Nikita Sobolev <mail@sobolevn.me>
Co-authored-by: Ken Jin <28750310+Fidget-Spinner@users.noreply.github.com>
This removes discrepancy between list["int"] and List["int"].
Co-authored-by: Jelle Zijlstra <jelle.zijlstra@gmail.com>
Co-authored-by: Alex Waygood <Alex.Waygood@Gmail.com>
When an exception is created in a nested call to PyObject_GetAttr, any
external calls will override the context information of the
AttributeError that we have already placed in the most internal call.
This will cause the suggestions we create to nor work properly as the
attribute name and object that we will be using are the incorrect ones.
To avoid this, we need to check first if these attributes are already
set and bail out if that's the case.
Also made modes containing 'a' or 'x' act the same as a mode containing 'w' when argument is '-'
(so 'a'/'x' return sys.stdout like 'w', and 'ab'/'xb' return sys.stdout.buffer like 'wb').
If Python is built with UBSan, test_hashlib skips tests on the _sha3
extension which currently has undefined behaviors.
This change allows to run test_hashlib to check for new UBSan regression,
but the known _sha3 undefined behavior must be fixed.
Skip tests on ASAN and/or MSAN builds:
* multiprocessing tests
* test___all__
* test_concurrent_futures
* test_decimal
* test_peg_generator
* test_tools
The ASAN job of GitHub Actions no longer excludes these tests.
Also from the _asyncio C accelerator module,
and adjust one test that the change caused to fail.
For more discussion see the discussion starting here:
https://github.com/python/cpython/pull/31394#issuecomment-1053545331
(Basically, @asvetlov proposed to return False from cancel()
when there is already a pending cancellation, and I went along,
even though it wasn't necessary for the task group implementation,
and @agronholm has come up with a counterexample that fails
because of this change. So now I'm changing it back to the old
semantics (but still bumping the counter) until we can have a
proper discussion about this.)
For threads, and for multiprocessing, it's always been the case that ``args=list`` works fine when passed to ``Process()`` or ``Thread()``, and such code is common in the wild. But, according to the docs, only a tuple can be used. This brings the docs into synch with reality.
Doc changes by Charlie Zhao.
Co-authored-by: Tim Peters <tim.peters@gmail.com>
Remove the undocumented private float.__set_format__() method,
previously known as float.__set_format__() in Python 3.7. Its
docstring said: "You probably don't want to use this function. It
exists mainly to be used in Python's test suite."
Fix a bug in urllib.request.HTTPPasswordMgr.find_user_password() and
urllib.request.HTTPPasswordMgrWithPriorAuth.is_authenticated() which
allowed to bypass authorization. For example, access to URI "example.org/foobar"
was allowed if the user was authorized for URI "example.org/foo".
The locale.getdefaultlocale() function is deprecated and will be
removed in Python 3.13. Use locale.setlocale(),
locale.getpreferredencoding(False) and locale.getlocale() functions
instead.
Fix parsing a numeric literal immediately (without spaces) followed by
"not in" keywords, like in "1not in x". Now the parser only emits
a warning, not a syntax error.
Curly brackets were never allowed in namespace URIs
according to RFC 3986, and so-called namespace-validating
XML parsers have the right to reject them a invalid URIs.
libexpat >=2.4.5 has become strcter in that regard due to
related security issues; with ET.XML instantiating a
namespace-aware parser under the hood, this test has no
future in CPython.
References:
- https://datatracker.ietf.org/doc/html/rfc3968
- https://www.w3.org/TR/xml-names/
Also, test_minidom.py: Support Expat >=2.4.5
There are several changes:
1. We now don't explicitly check for any base / sub types, because new name check covers it
2. I've also checked that `no_type_check` do not modify foreign functions. It was the same as with `type`s
3. I've also covered `except TypeError` in `no_type_check` with a simple test case, it was not covered at all
4. I also felt like adding `lambda` test is a good idea: because `lambda` is a bit of both in class bodies: a function and an assignment
<!-- issue-number: [bpo-46571](https://bugs.python.org/issue46571) -->
https://bugs.python.org/issue46571
<!-- /issue-number -->
* Remove task group names (for now)
We're not sure that they are needed, and once in the code
we would never be able to get rid of them.
Yury wrote:
> Ideally, there should be a way for someone to build a "trace"
> of taskgroups/task leading to the current running task.
> We could do that using contextvars, but I'm not sure we should
> do that in 3.11.
* Pass optional name on to task in create_task()
* Remove a bunch of unused stuff
* Add exception for uninstantiated interpolation (configparser)
The current feedback when users try to pass an uninstantiated
interpolation into a ConfigParser is an error message that does not help
users solve the problem. This current error of `TypeError: before_set()
missing 1 required positional argument: 'value'` does not display until
the parser is used, which usually results in the assumption that
instantiation of the parser was done correctly. The new exception of
InterpolationTypeError, will be raised on the line where the
ConfigParser is instantiated. This will result in users see the line
that has the error in their backtrace for faster debugging.
There have been a number of bugs created in the issue tracker, which
could have been addressed by:
https://bugs.python.org/issue26831 and https://bugs.python.org/issue26469
* 📜🤖 Added by blurb_it.
* Replace custom Error with TypeError
Per feedback from @iritkatriel, the custom InterpolationTypeError has
been dropped in favour of a TypeError with a custom message, and the
unittests have been expanded.
* More verbose message
Co-authored-by: blurb-it[bot] <43283697+blurb-it[bot]@users.noreply.github.com>
Co-authored-by: Łukasz Langa <lukasz@langa.pl>
The `module` parameter carries semantic information about the forward ref.
Forward refs are different if they refer to different module even if they
have the same name. This affects the `__eq__`, `__repr__` and `__hash__` methods.
Co-authored-by: Andreas Hangauer <andreas.hangauer@siemens.com>
Co-authored-by: Alex Waygood <Alex.Waygood@Gmail.com>
Co-authored-by: Ken Jin <28750310+Fidget-Spinner@users.noreply.github.com>
asyncio/taskgroups.py is an adaptation of taskgroup.py from EdgeDb, with the following key changes:
- Allow creating new tasks as long as the last task hasn't finished
- Raise [Base]ExceptionGroup (directly) rather than TaskGroupError deriving from MultiError
- Instead of monkey-patching the parent task's cancel() method,
add a new public API to Task
The Task class has a new internal flag, `_cancel_requested`, which is set when `.cancel()` is called successfully. The `.cancelling()` method returns the value of this flag. Further `.cancel()` calls while this flag is set return False. To reset this flag, call `.uncancel()`.
Thus, a Task that catches and ignores `CancelledError` should call `.uncancel()` if it wants to be cancellable again; until it does so, it is deemed to be busy with uninterruptible cleanup.
This new Task API helps solve the problem where TaskGroup needs to distinguish between whether the parent task being cancelled "from the outside" vs. "from inside".
Co-authored-by: Yury Selivanov <yury@edgedb.com>
Co-authored-by: Andrew Svetlov <andrew.svetlov@gmail.com>
The module parameter carries semantic information about the forward ref.
Show to the user that forward refs with same argument but different
module are different.
Co-authored-by: Andreas Hangauer <andreas.hangauer@siemens.com>
Co-authored-by: blurb-it[bot] <43283697+blurb-it[bot]@users.noreply.github.com>
Ensure strong references are acquired whenever using `set_next()`. Added randomized test cases for `__eq__` methods that sometimes mutate sets when called.
Confirmed with @jaraco that this indeed needs a fix.
A question that came up while I was digging into the code: I think `SelectableGroups` could similarly use `__slots__ = ()`, since its purpose seems only for convenience around `dict`, not to have attributes of its own.
Automerge-Triggered-By: GH:jaraco
Numeric fields of type float, notably mtime, can't be represented
exactly in the ustar header, so the pax header is used. But it is
helpful to set them to the nearest int (i.e. second rather than
nanosecond precision mtimes) in the ustar header as well, for the
benefit of unarchivers that don't understand the pax header.
Add test for tarfile.TarInfo.create_pax_header to confirm correct
behaviour.
In [Lib/test/support/import_helper.py](https://github.com/python/cpython/blob/master/Lib/test/support/import_helper.py), the function `make_legacy_pyc` makes a call to `os.rename` which can fail when the source and target live on different devices. This happens (for example) when `PYTHONPYCACHEPREFIX` is set to a directory anywhere on disk, while a ramdisk is mounted on `/tmp` (the latter of which is the default on various Linux distros). Replacing `os.rename` with `shutil.move` fixes this.
Automerge-Triggered-By: GH:brettcannon
The calendar.LocaleTextCalendar and calendar.LocaleHTMLCalendar
classes module now use locale.getlocale(), instead of using
locale.getdefaultlocale(), if no locale is specified.
skip_if_buggy_ucrt_strfptime() of test.support now uses
locale.getpreferredencoding(False) instead of
locale.getdefaultlocale() to get the Windows code page.
encodings registers the _alias_mbcs() codec search function before
the search_function() codec search function. Previously, the
_alias_mbcs() was never used.
Fix the test_codecs.test_mbcs_alias() test: use the current ANSI code
page, not a fake ANSI code page number.
Remove the test_site.test_aliasing_mbcs() test: the alias is now
implemented in the encodings module, no longer in the site module.
It was added as part of #29222 to avoid running freeze tool tests on the
buildbots but the logic was wrong so it did not skip tests on typical posix
setup buildbots where the worker is launched from cron via an @reboot task and
thus have no USER environment variable. This uses the canonical
`getpass.getuser()` API rather than rolling its own attempt.
* Revert "bpo-45173 Remove configparser deprecations"
This reverts commit df2284bc41.
* bpo-45173: Note these configparser deprecations will be removed in 3.12
Per Pitrou:
> The original intent for the “accessor” thing was to have a variant that did all accesses under a filesystem tree in a race condition-free way using openat and friends. It turned out to be much too hairy to actually implement, so was entirely abandoned, but the accessor abstraction was left there.
https://discuss.python.org/t/make-pathlib-extensible/3428/2
Accessors are:
- Lacking any internal purpose - '_NormalAccessor' is the only implementation
- Lacking any firm conceptual difference to `Path` objects themselves (inc. subclasses)
- Non-public, i.e. underscore prefixed - '_Accessor' and '_NormalAccessor'
- Unofficially used to implement customized `Path` objects, but once once [bpo-24132]() is addressed there will be a supported route for that.
This patch preserves all existing behaviour.
Fix test_gdb.test_pycfunction() for Python built with clang -Og.
Tolerate inlined functions in the gdb traceback.
When _testcapimodule.c is built by clang -Og, _null_to_none() is
inlined in meth_varargs() and so gdb returns _null_to_none() as
the frame #1. If it's not inlined, meth_varargs() is the frame #1.
musl libc and gnu libc are not ABI compatible so we need set different
SOABI for musl and not simply assume that all linux is linux-gnu.
Replace linux-gnu with the detected os for the build from config.guess
for linux-musl*.
* Add PRECALL_FUNCTION opcode.
* Move 'call shape' varaibles into struct.
* Replace CALL_NO_KW and CALL_KW with KW_NAMES and CALL instructions.
* Specialize for builtin methods taking using the METH_FASTCALL | METH_KEYWORDS protocol.
* Allow kwnames for specialized calls to builtin types.
* Specialize calls to tuple(arg) and str(arg).
* Substitution with a list of types returns now a tuple of types.
* Substitution with Concatenate returns now a Concatenate with
concatenated lists of arguments.
* Substitution with Ellipsis is not supported.
Previously this didn't matter because there weren't any valid code paths
that could trigger a type check with a special form, but after the bug
fix for `Annotated` wrapping special forms it's now possible to annotate
something like `Annotated['ClassVar[int]', (3, 4)]`. This change would
also be needed for proposed future changes, such as allowing `ClassVar`
and `Final` to nest each other in dataclasses.
Brett Cannon
Sam Ezeh
Hugo van Kemenade
Jelle Zijlstra
Erlend Egeberg Aasland
Matthieu Dartiailh
Irit Katriel
Ronald Oussoren
Matthew Rahtz
Shantanu
180909
Inada Naoki
Jason R. Coombs
Ma Lin
Christian Heimes
Serhiy Storchaka
Jeremy Kloth
Andrew Svetlov
Mark Shannon
Steve Dower
Oleg Iarygin
Maciej Górski
Jouke Witteveen
Duprat
Daniël van Noord
Pablo Galindo Salgado
Tim Peters
Brandt Bucher
Gregory P. Smith
Arie Bovenberg
Miro Hrončok
Bader Zaidan
Maximilian Hils
Kumar Aditya
Alex Grönholm
andrei kulakov
Gregory Beauregard
Victor Stinner
Alex Waygood
Nikita Sobolev
Matt Bogosian
Taneli Hukkinen
James Hilton-Balfe
Niklas Rosenstein
Jacob Walls
Dennis Sweeney
MojoVampire
Ken Jin
Guido van Rossum
Jannis Vajen
Tin Tvrtković
Charlie Zhao
Eric V. Smith
Thomas Grainger
Sebastian Pipping
Saul Shanabrook
Patrick Reader
Brian Faherty
aha79
97littleleaf11
Hakan Çelik
Alex-Blade
Zackery Spytz
Petr Viktorin
Brandt Bucher
Joshua Root
Andrzej Mateja
Barney Gale
Jason Wilkes
Manish Kumar ⛄
Natanael Copa
Russel Webber
Karolina Surma
Kinshuk Dua