Fetch CONFIG_ARGS from the original source directory, instead of from
the copied source tree. When "make clean" is executed in the copied
source tree, the build directory is cleared and the configure argument
lookup fails. However, the original source directory still contains this
information.
gh-86618 assumed a-b-c = a-(b+c) = a-d where d = b+d.
For floats 2.0, 1.0, and 0.9999999999999999, this assumption
is false. The net change of 1.1102230246251565e-16 to 0.0
results in division by 0. Revert the replacement. Add test.
* Convert PyObject_DelAttr() and PyObject_DelAttrString() macros to
functions.
* Add PyObject_DelAttr() and PyObject_DelAttrString() functions to
the stable ABI.
* Replace PyObject_SetAttr(obj, name, NULL) with
PyObject_DelAttr(obj, name).
Detect email address parsing errors and return empty tuple to indicate the parsing error (old API). This fixes or at least ameliorates CVE-2023-27043.
---------
Co-authored-by: Gregory P. Smith <greg@krypto.org>
Fixes a hang in multiprocessing process pool executor when a child process crashes and code could otherwise block on writing to the pipe. See GH-94777 for more details.
Some items remained uninitialized if _sre.template() was called with invalid
indices. Then attempt to clear them in the destructor led to dereferencing
of uninitialized pointer.
Prevent `multiprocessing.spawn` from failing to *import* in environments
where `sys.executable` is `None`. This regressed in 3.11 with the addition
of support for path-like objects in multiprocessing.
Adds a test decorator to have tests only run when part of test_multiprocessing_spawn to `_test_multiprocessing.py` so we can start to avoid re-running the same not-global-state specific test in all 3 modes when there is no need.
We match paths using the `_lines` attribute, which is derived from the
path's string representation. The bug arises because an empty path's string
representation is `'.'` (not `''`), which is matched by the `'*'` wildcard.
mock: Add `ThreadingMock` class
Add a new class that allows to wait for a call to happen by using
`Event` objects. This mock class can be used to test and validate
expectations of multithreading code.
It uses two attributes for events to distinguish calls with any argument
and calls with specific arguments.
The calls with specific arguments need a lock to prevent two calls in
parallel from creating the same event twice.
The timeout is configured at class and constructor level to allow users
to set a timeout, we considered passing it as an argument to the
function but it could collide with a function parameter. Alternatively
we also considered passing it as positional only but from an API
caller perspective it was unclear what the first number meant on the
function call, think `mock.wait_until_called(1, "arg1", "arg2")`, where
1 is the timeout.
Lastly we also considered adding the new attributes to magic mock
directly rather than having a custom mock class for multi threading
scenarios, but we preferred to have specialised class that can be
composed if necessary. Additionally, having added it to `MagicMock`
directly would have resulted in `AsyncMock` having this logic, which
would not work as expected, since when if user "waits" on a
coroutine does not have the same meaning as waiting on a standard
call.
Co-authored-by: Karthikeyan Singaravelan <tir.karthi@gmail.com>
If the output arg to Helper() is a stream rather than the default None, which means 'page to stdout', the ImportError from pydoc.resolve is currently not caught in pydoc.doc. The same error is caught when output is None.
---------
Co-authored-by: Terry Jan Reedy <tjreedy@udel.edu>
Remove the "cpython/pytime.h" header file: it only contained private
functions. Move functions to the internal pycore_time.h header file.
Move tests from _testcapi to _testinternalcapi. Rename also test
methods to have the same name than tested C functions.
No longer export these functions:
* _PyTime_Add()
* _PyTime_As100Nanoseconds()
* _PyTime_FromMicrosecondsClamp()
* _PyTime_FromTimespec()
* _PyTime_FromTimeval()
* _PyTime_GetPerfCounterWithInfo()
* _PyTime_MulDiv()
Command Prompt (CMD Shell) and older versions of PowerShell
require double quotes and single quotes inside the string.
This form also works on linux and macOS.
* EOFError no longer overrides other errors such as MemoryError or OSError at
the start of the object.
* Raise more relevant error when the NULL object occurs as a code object
component.
* Minimize an overhead of calling PyErr_Occurred().
When running the Python test suite with -jN option, if a worker stdout
cannot be decoded from the locale encoding report a failed testn so the
exitcode is non-zero.
PyTuple_SET_ITEM() and PyList_SET_ITEM() now check the index argument
with an assertion if Python is built in debug mode or is built with
assertions.
* list_extend() and _PyList_AppendTakeRef() now set the list size
before calling PyList_SET_ITEM().
* PyStructSequence_GetItem() and PyStructSequence_SetItem() now check
the index argument: must be lesser than REAL_SIZE(op).
* PyStructSequence_GET_ITEM() and PyStructSequence_SET_ITEM() are now
aliases to PyStructSequence_GetItem() and
PyStructSequence_SetItem().
Added a new, experimental, tracing optimizer and interpreter (a.k.a. "tier 2"). This currently pessimizes, so don't use yet -- this is infrastructure so we can experiment with optimizing passes. To enable it, pass ``-Xuops`` or set ``PYTHONUOPS=1``. To get debug output, set ``PYTHONUOPSDEBUG=N`` where ``N`` is a debug level (0-4, where 0 is no debug output and 4 is excessively verbose).
All of this code is likely to change dramatically before the 3.13 feature freeze. But this is a first step.
Brings `pathlib.Path.is_dir()` and `in line with `os.DirEntry.is_dir()`, which
will be important for implementing generic path walking and globbing.
Likewise `is_file()`.
Remove old aliases which were kept backwards compatibility with
Python 3.8:
* _PyObject_CallMethodNoArgs()
* _PyObject_CallMethodOneArg()
* _PyObject_CallOneArg()
* _PyObject_FastCallDict()
* _PyObject_Vectorcall()
* _PyObject_VectorcallMethod()
* _PyVectorcall_Function()
Update code which used these aliases to use new names.
This new exception type is raised instead of `NotImplementedError` when
a path operation is not supported. It can be raised from `Path.readlink()`,
`symlink_to()`, `hardlink_to()`, `owner()` and `group()`. In a future
version of pathlib, it will be raised by `AbstractPath` for these methods
and others, such as `AbstractPath.mkdir()` and `unlink()`.
* Add tests on PyImport_AddModuleRef(), PyImport_AddModule() and
PyImport_AddModuleObject().
* pythonrun.c: Replace Py_XNewRef(PyImport_AddModule(name)) with
PyImport_AddModuleRef(name).
Deprecate two methods of creating typing.TypedDict classes with 0 fields using the functional syntax: `TD = TypedDict("TD")` and `TD = TypedDict("TD", None)`. Both will be disallowed in Python 3.15. To create a TypedDict class with 0 fields, either use `class TD(TypedDict): pass` or `TD = TypedDict("TD", {})`.
* Add table describing possible executable classes for out-of-process debuggers.
* Remove shim code object creation code as it is no longer needed.
* Make lltrace a bit more robust w.r.t. non-standard frames.
Deprecate creating a typing.NamedTuple class using keyword arguments to denote the fields (`NT = NamedTuple("NT", x=int, y=str)`). This will be disallowed in Python 3.15. Use the class-based syntax or the functional syntax instead.
Two methods of creating `NamedTuple` classes with 0 fields using the functional syntax are also deprecated, and will be disallowed in Python 3.15: `NT = NamedTuple("NT")` and `NT = NamedTuple("NT", None)`. To create a `NamedTuple` class with 0 fields, either use `class NT(NamedTuple): pass` or `NT = NamedTuple("NT", [])`.
* GH-104554: Add RTSPS support to `urllib/parse.py`
RTSPS is the permanent scheme defined in
https://www.iana.org/assignments/uri-schemes/uri-schemes.xhtml
alongside RTSP and RTSPU schemes.
* 📜🤖 Added by blurb_it.
---------
Co-authored-by: blurb-it[bot] <43283697+blurb-it[bot]@users.noreply.github.com>
For a while now, pending calls only run in the main thread (in the main interpreter). This PR changes things to allow any thread run a pending call, unless the pending call was explicitly added for the main thread to run.
Adds start_offset, cache_offset, end_offset, baseopcode,
baseopname, jump_target and oparg to dis.Instruction.
Also slightly improves the disassembly output by allowing
opnames to overflow into the space reserved for opargs.
Fix bugs where exceptions could end up being overwritten
because of deferred error handling.
Co-authored-by: Erlend E. Aasland <erlend.aasland@protonmail.com>
When inverting a Flag member (or boundary STRICT), only consider other canonical flags; when inverting an IntFlag member (or boundary KEEP), also consider aliases.
* support inspect.iscoroutinefunction in create_autospec(async_def)
* test create_autospec with inspect.iscoroutine and inspect.iscoroutinefunction
* test when create_autospec functions check their signature
The original name wasn't as clear as it could have been. This change includes the following:
* rename the function
* change the default value for "disable_check" to False
* add clues to the docstring that folks should probably not use the function
---------
Co-authored-by: Kirill Podoprigora <kirill.bast9@mail.ru>
This ensures that `commoninstall` is completed before `bininstall` is
started when parallel builds are used (`make -j install`), and so the
`python3` symlink is only installed after all standard library modules
are installed.
Co-authored-by: Zachary Ware <zachary.ware@gmail.com>
This commit introduces a 'walk-and-match' strategy for handling glob patterns that include a non-terminal `**` wildcard, such as `**/*.py`. For this example, the previous implementation recursively walked directories using `os.scandir()` when it expanded the `**` component, and then **scanned those same directories again** when expanded the `*.py` component. This is wasteful.
In the new implementation, any components following a `**` wildcard are used to build a `re.Pattern` object, which is used to filter the results of the recursive walk. A pattern like `**/*.py` uses half the number of `os.scandir()` calls; a pattern like `**/*/*.py` a third, etc.
This new algorithm does not apply if either:
1. The *follow_symlinks* argument is set to `None` (its default), or
2. The pattern contains `..` components.
In these cases we fall back to the old implementation.
This commit also replaces selector classes with selector functions. These generators directly yield results rather calling through to their successors. A new internal `Path._glob()` method takes care to chain these generators together, which simplifies the lazy algorithm and slightly improves performance. It should also be easier to understand and maintain.
Disallow thread creation and fork at interpreter finalization.
in the following functions, check if interpreter is finalizing and raise `RuntimeError` with appropriate message:
* `_thread.start_new_thread` and thus `threading`
* `posix.fork`
* `posix.fork1`
* `posix.forkpty`
* `_posixsubprocess.fork_exec` when a `preexec_fn=` is supplied.
---------
Co-authored-by: blurb-it[bot] <43283697+blurb-it[bot]@users.noreply.github.com>
Co-authored-by: Gregory P. Smith <greg@krypto.org>
Upgrade builds to OpenSSL 1.1.1u.
This OpenSSL version addresses a pile if less-urgent CVEs since 1.1.1t.
The Mac/BuildScript/build-installer.py was already updated.
Also updates _ssl_data_111.h from OpenSSL 1.1.1u, _ssl_data_300.h from 3.0.9, and adds a new _ssl_data_31.h file from 3.1.1 along with the ssl.c code to use it.
Manual edits to the _ssl_data_300.h file prevent it from removing any existing definitions in case those exist in some peoples builds and were important (avoiding regressions during backporting).
backports of this prior to 3.12 will not include the openssl 3.1 header.
Remove functions in the C API:
* PyEval_AcquireLock()
* PyEval_ReleaseLock()
* PyEval_InitThreads()
* PyEval_ThreadsInitialized()
But keep these functions in the stable ABI.
Mention "make regen-limited-abi" in "make regen-all".
Remove the following old functions to configure the Python
initialization, deprecated in Python 3.11:
* PySys_AddWarnOptionUnicode()
* PySys_AddWarnOption()
* PySys_AddXOption()
* PySys_HasWarnOptions()
* PySys_SetArgvEx()
* PySys_SetArgv()
* PySys_SetPath()
* Py_SetPath()
* Py_SetProgramName()
* Py_SetPythonHome()
* Py_SetStandardStreamEncoding()
* _Py_SetProgramFullPath()
Most of these functions are kept in the stable ABI, except:
* Py_SetStandardStreamEncoding()
* _Py_SetProgramFullPath()
Update Doc/extending/embedding.rst and Doc/extending/extending.rst to
use the new PyConfig API.
_testembed.c:
* check_stdio_details() now sets stdio_encoding and stdio_errors
of PyConfig.
* Add definitions of functions removed from the API but kept in the
stable ABI.
* test_init_from_config() and test_init_read_set() now use
PyConfig_SetString() instead of PyConfig_SetBytesString().
Remove _Py_ClearStandardStreamEncoding() internal function.
Deprecate the old Py_UNICODE and PY_UNICODE_TYPE types in the C API:
use wchar_t instead.
Replace Py_UNICODE with wchar_t in multiple C files.
Co-authored-by: Inada Naoki <songofacandy@gmail.com>
In gh-103912 we added tp_bases and tp_mro to each PyInterpreterState.types.builtins entry. However, doing so ignored the fact that both PyTypeObject fields are public API, and not documented as internal (as opposed to tp_subclasses). We address that here by reverting back to shared objects, making them immortal in the process.
`PurePath.match()` now handles the `**` wildcard as in `Path.glob()`, i.e. it matches any number of path segments.
We now compile a `re.Pattern` object for the entire pattern. This is made more difficult by `fnmatch` not treating directory separators as special when evaluating wildcards (`*`, `?`, etc), and so we arrange the path parts onto separate *lines* in a string, and ensure we don't set `re.DOTALL`.
Co-authored-by: Hugo van Kemenade <hugovk@users.noreply.github.com>
Co-authored-by: Alex Waygood <Alex.Waygood@Gmail.com>
Add a keyword-only *follow_symlinks* parameter to `pathlib.Path.glob()` and`rglob()`.
When *follow_symlinks* is `None` (the default), these methods follow symlinks except when evaluating "`**`" wildcards. When set to true or false, symlinks are always or never followed, respectively.
For backwards compatibility, accept backslashes as path separators in
`PurePosixPath` if an instance of `PureWindowsPath` is supplied.
This restores behaviour from Python 3.11.
Co-authored-by: Gregory P. Smith <greg@krypto.org>
Use `str.lower()` rather than `ntpath.normcase()` to normalize case of
Windows paths. This restores behaviour from Python 3.11.
Co-authored-by: Gregory P. Smith <greg@krypto.org>
* Remove the Lib/test/imghdrdata/ directory.
* Copy 5 pictures (gif, png, ppm, pgm, xbm) from removed
Lib/test/imghdrdata/ to a new Lib/test/tkinterdata/ directory.
* Update Sphinx from 4.5 to 6.2 in Doc/requirements.txt.
Reduced _idle_semaphore.release calls in concurrent.futures.thread._worker
_idle_semaphore.release() is now only called if only work_queue is empty.
---------
Co-authored-by: blurb-it[bot] <43283697+blurb-it[bot]@users.noreply.github.com>
Co-authored-by: Shantanu <12621235+hauntsaninja@users.noreply.github.com>
On Linux where the `subprocess` module can use the `vfork` syscall for
faster spawning, prevent the parent process from blocking other threads
by dropping the GIL while it waits for the vfork'ed child process `exec`
outcome. This prevents spawning a binary from a slow filesystem from
blocking the rest of the application.
Fixes#104372.
* socket_helper.transient_internet() no longer imports nntplib to
catch nntplib.NNTPTemporaryError.
* ssltests.py no longer runs test_nntplib.
* "make quicktest" no longer runs test_nntplib.
* WASM: remove nntplib from OMIT_NETWORKING_FILES.
* Remove mentions to nntplib in the email documentation.
Class editor.IndentSearcher contains all editor references to tokenize module.
Module io tokenize reference cover those other modules.
Co-authored-by: Jelle Zijlstra <jelle.zijlstra@gmail.com>
In Python 3.8 and prior, `pathlib.Path.__exit__()` marked a path as closed;
some subsequent attempts to perform I/O would raise an IOError. This
functionality was never documented, and had the effect of making `Path`
objects mutable, contrary to PEP 428. In Python 3.9 we made `__exit__()` a
no-op, and in 3.11 `__enter__()` began raising deprecation warnings. Here
we remove both methods.
This matches the GIL releasing behavior of our existing `_hashopenssl`
module, extending it to the HACL* built-ins.
Includes adding comments to better describe the ENTER/LEAVE macros
purpose and explain the lock strategy in both existing and new code.
Apply BOLT optimizations to libpython for shared builds. Most of the C
code is in libpython so it is critical to apply BOLT there fully realize
BOLT benefits.
This change also reworks how BOLT instrumentation is applied. It
effectively removes the readelf based logic added in gh-101525 and
replaces it with a mechanism that saves a copy of the pre-bolt binary
and restores that copy when necessary. This allows us to perform BOLT
optimizations without having to manually delete the output binary to
force a new bolt run.
Also:
- add a clean-bolt target for purging BOLT files and hook that up to the
clean target
- .gitignore BOLT related files
Before and after this refactor, `make` will no-op after a previous run.
Both versions should also share common make DAG deficiencies where
targets fail to trigger as often as they need to or can trigger
prematurely in certain scenarios. e.g. after this change you may need to
`rm profile-bolt-stamp` to force a BOLT run because there aren't
appropriate non-phony targets for BOLT's make target to depend on.
To make it easier to iterate on custom BOLT settings, the flags to pass
to instrumentation and application are now defined in configure and can
be overridden by passing BOLT_INSTRUMENT_FLAGS and BOLT_APPLY_FLAGS.
Fix a race condition in the internal `multiprocessing.process` cleanup
logic that could manifest as an unintended `AttributeError` when calling
`BaseProcess.close()`.
---------
Co-authored-by: Oleg Iarygin <oleg@arhadthedev.net>
Co-authored-by: Gregory P. Smith <greg@krypto.org>
* Support for conversion specifiers o (octal) and X (uppercase hexadecimal).
* Support for length modifiers j (intmax_t) and t (ptrdiff_t).
* Length modifiers are now applied to all integer conversions.
* Support for wchar_t C strings (%ls and %lV).
* Support for variable width and precision (*).
* Support for flag - (left alignment).
This commit replaces the Python implementation of the tokenize module with an implementation
that reuses the real C tokenizer via a private extension module. The tokenize module now implements
a compatibility layer that transforms tokens from the C tokenizer into Python tokenize tokens for backward
compatibility.
As the C tokenizer does not emit some tokens that the Python tokenizer provides (such as comments and non-semantic newlines), a new special mode has been added to the C tokenizer mode that currently is only used via
the extension module that exposes it to the Python layer. This new mode forces the C tokenizer to emit these new extra tokens and add the appropriate metadata that is needed to match the old Python implementation.
Co-authored-by: Pablo Galindo <pablogsal@gmail.com>
bpo-17258: `multiprocessing` now supports stronger HMAC algorithms for inter-process connection authentication rather than only HMAC-MD5.
Signed-off-by: Christian Heimes <christian@python.org>
gpshead: I Reworked to be more robust while keeping the idea.
The protocol modification idea remains, but we now take advantage of the
message length as an indicator of legacy vs modern protocol version. No
more regular expression usage. We now default to HMAC-SHA256, but do so
in a way that will be compatible when communicating with older clients
or older servers. No protocol transition period is needed.
More integration tests to verify these claims remain true are required. I'm
unaware of anyone depending on multiprocessing connections between
different Python versions.
---------
Signed-off-by: Christian Heimes <christian@python.org>
Co-authored-by: Gregory P. Smith [Google] <greg@krypto.org>
This PR updates `math.nextafter` to add a new `steps` argument. The behaviour is as though `math.nextafter` had been called `steps` times in succession.
---------
Co-authored-by: Mark Dickinson <mdickinson@enthought.com>
New additions to the tty library. Functions added: cfmakeraw(), and cfmakecbreak(). The
functions setcbreak() and setraw() now return original termios to save an extra tcgetattr() call.
---------
Signed-off-by: Soumendra Ganguly <soumendraganguly@gmail.com>
Co-authored-by: blurb-it[bot] <43283697+blurb-it[bot]@users.noreply.github.com>
Co-authored-by: Gregory P. Smith [Google LLC] <greg@krypto.org>
This adds a number of PRIO_DARWIN_* constants to the os module for use with os.setpriority.
---------
Co-authored-by: blurb-it[bot] <43283697+blurb-it[bot]@users.noreply.github.com>
Move all of the Python C API calls into the parent process up front
instead of doing PyLong_AsLong and PyErr_Occurred and PyTuple_GET from
the post-fork/vfork child process.
Much of this was long overdue. We shouldn't have been using PyTuple and
PyLong APIs within all of these low level functions anyways.
This fixes several ways file descriptors could be leaked from `subprocess.Popen` constructor during error conditions by opening them later and using a context manager "fds to close" registration scheme to ensure they get closed before returning.
---------
Co-authored-by: Gregory P. Smith [Google] <greg@krypto.org>
Fix Zip64 extensions not being properly applied in some cases:
Fixes an issue where adding a small file to a `ZipFile`
object while forcing zip64 extensions causes an extra Zip64 record to be
added to the zip, but doesn't update the `min_version` or file sizes in
the primary central directory header.
Also fixed an edge case in checking if zip64 extensions are required:
This fixes an issue where if data requiring zip64 extensions was added
to an unseekable stream without specifying `force_zip64=True`, zip64
extensions would not be used and a RuntimeError would not be raised when
closing the file (even though the size would be known at that point).
This would result in successfully writing corrupt zip files.
Deciding if zip64 extensions are required outside of the `FileHeader`
function means that both `FileHeader` and `_ZipWriteFile` will always be
in sync. Previously, the `FileHeader` function could enable zip64
extensions without propagating that decision to the `_ZipWriteFile`
class, which would then not correctly write the data descriptor record
or check for errors on close.
If anyone is actually using `ZipInfo.FileHeader` as a public API without
explicitly passing True or False in for zip64, their own code may still be
susceptible to that kind of bug unless they make a similar change to
where the zip64 decision happens.
Fixes#103861
---------
Co-authored-by: Gregory P. Smith <greg@krypto.org>
Add http.client.HTTPConnection method get_proxy_response_headers() - this is a followup to https://github.com/python/cpython/pull/26152 which added it as a non-public attribute. This way we don't pre-compute a headers dictionary that most users will never access. The new method is properly public and documented and triggers full proxy header parsing into a dict only when actually called.
---------
Co-authored-by: blurb-it[bot] <43283697+blurb-it[bot]@users.noreply.github.com>
Co-authored-by: Gregory P. Smith <greg@krypto.org>
This implements PEP 695, Type Parameter Syntax. It adds support for:
- Generic functions (def func[T](): ...)
- Generic classes (class X[T](): ...)
- Type aliases (type X = ...)
- New scoping when the new syntax is used within a class body
- Compiler and interpreter changes to support the new syntax and scoping rules
Co-authored-by: Marc Mueller <30130371+cdce8p@users.noreply.github.com>
Co-authored-by: Eric Traut <eric@traut.com>
Co-authored-by: Larry Hastings <larry@hastings.org>
Co-authored-by: Alex Waygood <Alex.Waygood@Gmail.com>
By convention make targets that don't refer to a file have a dependency
on the fake .PHONY target/file. This ensures that these targets are
always evaluated because there is no rule to create a .PHONY file
and that will force make to think the rule is out of date and needs
to be rebuilt.
This commit consistently associates virtual targets with .PHONY by
declaring the .PHONY dependency immediately above the make rule. This
should avoid race conditions and avoidable rebuilds across multiple make
invocations.
* Pickle the `name` and `args` attributes of AttributeError when present.
Co-authored-by: Gregory P. Smith <greg@krypto.org>
Co-authored-by: Erlend E. Aasland <erlend.aasland@protonmail.com>
Reject HTTP requests with invalid http/x.y version numbers: x or y being non-digits or too-long.
---------
Co-authored-by: Oleg Iarygin <oleg@arhadthedev.net>
Co-authored-by: Gregory P. Smith <greg@krypto.org>
These are stubs to be used for adding hypothesis (https://hypothesis.readthedocs.io/en/latest/) tests to the standard library.
When the tests are run in an environment where `hypothesis` and its various dependencies are not installed, the stubs will turn any tests with examples into simple parameterized tests and any tests without examples are skipped.
It also adds hypothesis tests for the `zoneinfo` module, and a Github Actions workflow to run the hypothesis tests as a non-required CI job.
The full hypothesis interface is not stubbed out — missing stubs can be added as necessary.
Co-authored-by: Zac Hatfield-Dodds <zac.hatfield.dodds@gmail.com>
When monitoring LINE events, instrument all instructions that can have a predecessor on a different line.
Then check that the a new line has been hit in the instrumentation code.
This brings the behavior closer to that of 3.11, simplifying implementation and porting of tools.
Use the unused keyword param in the converter to explicitly
mark an argument as unused:
/*[clinic input]
SomeBaseClass.stubmethod
flag: bool(unused=True)
[clinic start generated code]*/